What does running dbt in production mean?
Running dbt in production simply means setting up a system to run a dbt job on a schedule, rather than running dbt commands manually from the command line. These production dbt jobs should create the tables and views that your business intelligence tools and end users query. Before continuing, make sure you understand dbt's approach to managing environments.
dbt commands in production
We've written a guide for the dbt commands we run in production, over on Discourse.
As well as setting up a schedule, there are a number of other things you should consider when setting up dbt to run in production, such as:
- The complexity involved in creating a new dbt job, or editing an existing one.
- Setting up notifications if a step within your job returns an error code (e.g. a model cannot be built, or a test fails).
- Accessing logs to help debug any issues.
- Pulling the latest version of your git repo before running dbt (i.e. continuous deployment).
- Running your dbt project before merging code into master (i.e. continuous integration).
- Allowing access for team members that need to collaborate on your dbt project.
Ways to run dbt in production
Using dbt Cloud
We've built dbt Cloud from the ground up to empower data teams to easily run dbt in production. With dbt Cloud, you can:
- run your jobs on a schedule
- view logs for any historical invocation of dbt
- configure error notifications
- render your project's documentation
If you're interested in giving dbt Cloud a spin, you can sign up for a forever free account here.
If your organization is using Airflow, there are a number of ways you can run your dbt jobs, including:
- Using this dbt-cloud-plugin. This plugin gives you the best of both worlds -- deep integration of dbt into your existing data stack, along with all of the benefits of dbt Cloud.
- Invoking dbt through the BashOperator. In this case, be sure to install dbt into a virtual environment to avoid issues with conflicting dependencies between Airflow and dbt.
- Installing the airflow-dbt python package. This package uses Airflow's operator and hook concept — the source code can be found on github.
Alternatively, you can use the supported ShellTask to execute dbt commands through the shell.
If your organization is using Dagster, you can use the dagster_dbt library to integrate dbt commands into your pipelines. Execution through both the dbt CLI and the dbt RPC server is supported, and metadata from dbt runs is automatically aggregated. Check out the example pipeline for details.
Using an automation server
Automation servers, like CodeDeploy, GitLab CI/CD (video), Bamboo and Jenkins, can be used to schedule bash commands for dbt. They also provide a UI to view logging to the command line, and integrate with your git repository.
Cron is a decent way to schedule bash commands. However, while it may seem like an easy route to schedule a job, writing code to take care of all of the additional features associated with a production deployment often makes this route more complex compared to other options listed here.