Skip to main content

Dagster & Looker (Component)

Beta feature

This feature is considered in a beta stage. It is still being tested and may change. For more information, see the API lifecycle stages documentation.

The dagster-looker library provides a LookerComponent which can be used to easily represent Looker dashboards, explores, and Persistent Derived Tables (PDTs) as assets in Dagster.

info

LookerComponent is a state-backed component, which fetches and caches Looker instance metadata. For information on managing component state, see Configuring state-backed components.

1. Prepare a Dagster project​

To begin, you'll need a Dagster project. You can use an existing components-ready project or create a new one:

uvx create-dagster project my-project && cd my-project/src

Activate the project virtual environment:

source ../.venv/bin/activate

Finally, add the dagster-looker library to the project:

uv add dagster-looker

2. Scaffold a Looker component definition​

Now that you have a Dagster project, you can scaffold a Looker component definition:

dg scaffold defs dagster_looker.LookerComponent looker_ingest
Creating defs at /.../my-project/src/my_project/defs/looker_ingest.

The dg scaffold defs call will generate a defs.yaml file:

tree my_project/defs
my_project/defs
├── __init__.py
└── looker_ingest
    └── defs.yaml

2 directories, 2 files

3. Configure your Looker instance​

Update the defs.yaml file with your Looker instance connection details. You'll need to provide your base URL, client ID, and client secret. For more information on creating API credentials, see the Looker API documentation.

my_project/defs/looker_ingest/defs.yaml
type: dagster_looker.LookerComponent

attributes:
  looker_resource:
    base_url: "{{ env.LOOKER_BASE_URL }}"
    client_id: "{{ env.LOOKER_CLIENT_ID }}"
    client_secret: "{{ env.LOOKER_CLIENT_SECRET }}"
dg list defs
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions                                                                         ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets  │ ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┓ │
│         │ ┃ Key                  ┃ Group   ┃ Deps                 ┃ Kinds     ┃ Description ┃ │
│         │ ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━┩ │
│         │ │ my_dashboard_1       │ default │ my_model::my_explore │ dashboard │             │ │
│         │ │                      │         │                      │ looker    │             │ │
│         │ ├──────────────────────┼─────────┼──────────────────────┼───────────┼─────────────┤ │
│         │ │ my_model::my_explore │ default │ view/my_view         │ explore   │             │ │
│         │ │                      │         │                      │ looker    │             │ │
│         │ ├──────────────────────┼─────────┼──────────────────────┼───────────┼─────────────┤ │
│         │ │ view/my_view         │ default │                      │           │             │ │
│         │ └──────────────────────┴─────────┴──────────────────────┴───────────┴─────────────┘ │
└─────────┴─────────────────────────────────────────────────────────────────────────────────────┘

4. Filter Looker content​

You can filter which Looker dashboards and explores are loaded using the looker_filter key. For example, you can load only dashboards from specific folders and only explores used in those dashboards:

my_project/defs/looker_ingest/defs.yaml
type: dagster_looker.LookerComponent

attributes:
  looker_resource:
    base_url: "{{ env.LOOKER_BASE_URL }}"
    client_id: "{{ env.LOOKER_CLIENT_ID }}"
    client_secret: "{{ env.LOOKER_CLIENT_SECRET }}"
  looker_filter:
    dashboard_folders:
      - ["Shared"]
    only_fetch_explores_used_in_dashboards: true

5. Configure PDT Builds​

You can configure the LookerComponent to create executable assets for Looker Persistent Derived Tables (PDTs). This allows Dagster to orchestrate the materialization of these tables.

To enable this, add the pdt_builds key to your configuration. You must specify the model_name and view_name for each PDT you wish to build.

You can also specify a resource_key if your Looker resource has a custom name (defaults to "looker").

type: dagster_looker.LookerComponent
attributes:
  looker_resource:
    base_url: "{{ env.LOOKER_BASE_URL }}"
    client_id: "{{ env.LOOKER_CLIENT_ID }}"
    client_secret: "{{ env.LOOKER_CLIENT_SECRET }}"
  resource_key: "looker"
  pdt_builds:
    - model_name: my_model
      view_name: my_pdt_view
      force_rebuild: "true"
    - model_name: sales_model
      view_name: monthly_report_pdt
      workspace: dev

## 6. Customize Looker asset metadata

You can customize the metadata and grouping of Looker assets using the `translation` key:

<CodeExample
  path="docs_snippets/docs_snippets/guides/components/integrations/looker-component/9-customized-component.yaml"
  title="my_project/defs/looker_ingest/defs.yaml"
  language="yaml"
/>

<WideContent maxSize={1100}>
  <CliInvocationExample path="docs_snippets/docs_snippets/guides/components/integrations/looker-component/10-list-defs.txt" />
</WideContent>

### Customize specific data types

You may also specify distinct translation behavior for specific data types. For example, you can add a tag to all dashboards:

<CodeExample
  path="docs_snippets/docs_snippets/guides/components/integrations/looker-component/11-customized-dashboard-translation.yaml"
  title="my_project/defs/looker_ingest/defs.yaml"
  language="yaml"
/>

<WideContent maxSize={1100}>
  <CliInvocationExample path="docs_snippets/docs_snippets/guides/components/integrations/looker-component/12-list-defs.txt" />
</WideContent>