Managing multiple code locations with Components
This feature is still in development and might change in patch releases. It’s not production-ready, and the documentation may also evolve. Stay tuned for updates.
You can use a deployment directory to manage multiple code locations within a single coherent directory structure.
A deployment directory contains a root pyproject.toml containing deployment-level settings and a code_locations directory containing one or more code locations. A deployment does not define a Python environment by default. Instead, Python environments are defined per code location.
Scaffold a new deployment
To scaffold a new deployment, run:
dg scaffold deployment my-deployment
Creating a Dagster deployment at my-deployment.
Scaffolded files for Dagster project in my-deployment.
This will create a new directory called my-deployment. Let's look at the structure:
cd my-deployment && tree
.
├── code_locations
└── pyproject.toml
2 directories, 1 file
Importantly, the pyproject.toml file contains an is_deployment setting marking this directory as a deployment:
[tool.dg]
is_deployment = true
Add a code location to the deployment
To add a code location to the deployment, run:
dg scaffold code-location code-location-1
Creating a Dagster code location at /.../my-deployment/code_locations/code-location-1.
Scaffolded files for Dagster project in /.../my-deployment/code_locations/code-location-1.
...
This will create a new directory code-location-1 within the code_locations.
It will also setup a new uv-managed Python environment for the code location. Let's have a look:
tree
.
├── code_locations
│ └── code-location-1
│ ├── code_location_1
│ │ ├── __init__.py
│ │ ├── components
│ │ ├── definitions.py
│ │ └── lib
│ │ └── __init__.py
│ ├── code_location_1_tests
│ │ └── __init__.py
│ ├── pyproject.toml
│ └── uv.lock
└── pyproject.toml
...
code-location-1 also contains a virtual environment directory .venv that is not shown above. This environment is managed by uv and its contents are specified in the uv.lock file.
The code-location-1 directory contains a pyproject.toml file that defines
it as a code location and component library:
...
[tool.dagster]
module_name = "code_location_1.definitions"
code_location_name = "code-location-1"
[tool.dg]
is_code_location = true
is_component_lib = true
...
Let's enter this directory and search for registered component types:
cd code_locations/code-location-1 && dg list component-type
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Component Type ┃ Summary ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ definitions@dagster_components │ Wraps an arbitrary set of │
│ │ Dagster definitions. │
│ pipes_subprocess_script_collection@dagster_components │ Assets that wrap Python │
│ │ scripts executed with │
│ │ Dagster's │
│ │ PipesSubprocessClient. │
└───────────────────────────────────────────────────────┴────────────────────────────────┘
This is the default set of component types available in every new code location. We can add to it by installing dagster-components[sling]:
uv add 'dagster-components[sling]'
Due to a bug in sling package metadata, if you are on a macOS machine with Apple Silicon you may also need to run uv add sling_mac_arm64.
And now we have a new available component:
dg list component-type
Using /.../my-deployment/code_locations/code-location-1/.venv/bin/dagster-components
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Component Type ┃ Summary ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ definitions@dagster_components │ Wraps an arbitrary set of │
│ │ Dagster definitions. │
│ pipes_subprocess_script_collection@dagster_components │ Assets that wrap Python │
│ │ scripts executed with │
│ │ Dagster's │
│ │ PipesSubprocessClient. │
│ sling_replication_collection@dagster_components │ Expose one or more Sling │
│ │ replications to Dagster as │
│ │ assets. │
└───────────────────────────────────────────────────────┴────────────────────────────────┘
Add a second code location to the deployment
As stated above, environments are scoped per code location. dg commands will only use the environment of code-location-1 when we are inside the code-location-1 directory.
Let's create another code location to demonstrate this:
cd ../.. && dg scaffold code-location code-location-2
Creating a Dagster code location at /.../my-deployment/code_locations/code-location-2.
Scaffolded files for Dagster project in /.../my-deployment/code_locations/code-location-2.
...
Now we have two code locations. We can list them with:
dg list code-location
code-location-1
code-location-2
And finally, let's check the available component types in code-location-2:
cd code_locations/code-location-2 && dg list component-type
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Component Type ┃ Summary ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ definitions@dagster_components │ Wraps an arbitrary set of │
│ │ Dagster definitions. │
│ pipes_subprocess_script_collection@dagster_components │ Assets that wrap Python │
│ │ scripts executed with │
│ │ Dagster's │
│ │ PipesSubprocessClient. │
└───────────────────────────────────────────────────────┴────────────────────────────────┘
As you can see, we are back to only the default list of component types. This is because we are now using the environment of code-location-2, in which we have not installed dagster-components[sling].
Load code locations with dg
For a final step, let's load up our two code locations with dg dev. dg dev will automatically recognize the code locations in your deployment and launch them in their respective environments. Let's run dg dev back in the
deployment root directory and load up the Dagster UI in your browser:
cd ../.. && dg dev
