Running multiple agents
Experimental
#

This guide is applicable to Dagster Cloud.

Each Dagster Cloud full deployment (e.g., prod) needs to have at least one agent running. Since the agent is lightweight and can tolerate short downtimes such as during an upgrade, a single agent is adequate for most use cases.

If you want to distribute runs across multiple Kubernetes clusters, ECS clusters, EC2 instances, or other environments, you can run multiple agents. For example, one in each cluster, instance, etc.

It's recommended to only use multiple agents of the same type (e.g. multiple Kubernetes agents). Submitting runs to a specific agent isn't currently supported.

Prerequisites#

To complete the steps in this guide, you'll need:

  • dagster-cloud version 0.13.0 or later for both agents and job code.
  • To use a Kubernetes, Docker, or local agent. These agent types can support running multiple agents.

Step 1: Set up the agent#

Follow the setup guide for your agent:

After the agent is up and running, move on to the next step.

Step 2: Enable the agent_replicas setting#

In this step, you'll enable the agent_replicas option to turn on running multiple agents. You can also choose to label the agent, making it easily identifiable on Dagster Cloud's Status page.

Follow the instructions for your agent:

DOCKER AGENTS

Add the following to the dagster.yaml file:

agent_replicas:
  enabled: true

dagster_cloud_api:
  # <your other config>
  agent_label: "My agent" # optional
KUBERNETES AGENTS

Add the following options to your Helm command:

helm upgrade \
    ...
    --set agentReplicas.enabled=true \
    --set dagsterCloud.agentLabel="My agent" # optional, only supported on 0.13.14 and later

Or if you're using a values.yaml file:

agentReplicas:
  enabled: true

dagsterCloud:
  agentLabel: "My agent" # optional, only supported on 0.13.14 and later
LOCAL AGENTS

Add the following to the dagster.yaml file:

agent_replicas:
  enabled: true

dagster_cloud_api:
  # <your other config>
  agent_label: "My agent" # optional