dbt CLI: Create a project

Now that we've successfully run our sample query in BigQuery, and chosen the way we want to develop, we can create a dbt project! In this step, we'll create a starter project with example models, before we build our own models.

info

These are the instructions for developing a project using the dbt CLI. If you're developing in dbt Cloud, follow the instructions here.

Create a repository

We're going to use GitHub as our git provider for this tutorial, but you can use any git provider. If you don't yet have a GitHub account, create one now.

1. Create a new GitHub repository here named dbt-tutorial.
2. Click Create repository (without .gitignore and without a license).

Create a GitHub repo

Create a project

The dbt CLI comes with a command to help you scaffold a dbt project. To create your dbt project:

1. Ensure dbt is installed by running dbt --version:

$ dbt --version

info

dbt should have been installed as part of the Setting Up part of the tutorial. If it was not installed, please follow the installation instructions.

2. Run the init command:

$ dbt init jaffle-shop

3. cd into your project:

$ cd jaffle-shop

You can use pwd to confirm that you are in the right spot.

3. Open your project (i.e. the directory you just created) in a code editor like Atom or VSCode. You should see a directory structure with .sql and .yml files that were generated by the init command.

The starter project in a code editor

4. Update the following values in the dbt_project.yml file:

dbt_project.yml

name: jaffle_shop # this normally says my_new_package

...

profile: jaffle_shop # this normally says default

...

models:
  jaffle_shop: #this normally says my_new_package. It should match the value for `name:`
    ...

Connect to BigQuery

When developing locally, dbt connects to your data warehouse using a profile — a yaml file with all the connection details to your warehouse.

1. Create a file in the ~/.dbt/ directory named profiles.yml.
2. Move your BigQuery keyfile into this directory.
3. Copy the following into the file — make sure you update the values where indicated.

profiles.yml

jaffle_shop: # this needs to match the profile: in your dbt_project.yml file
  target: dev
  outputs:
    dev:
      type: bigquery
      method: service-account
      keyfile: /Users/claire/.dbt/dbt-user-creds.json # replace this with the full path to your keyfile
      project: grand-highway-265418 # Replace this with your project id
      dataset: dbt_alice # Replace this with dbt_your_name, e.g. dbt_bob
      threads: 1
      timeout_seconds: 300
      location: US
      priority: interactive

4. Execute the debug command from your project to confirm that you can successfully connect

$ dbt debug

Confirm that the last line of the output is Connection test: OK connection ok.

A successful dbt debug command

FAQs

 My data team uses a different data warehouse. What should my profiles.yml file look like for my warehouse?

The structure of a profile looks different on each warehouse. Check out the supported databases page, and navigate to the Profile Setup section for your warehouse.

 Why are profiles stored outside of my project?

Profiles are stored separately to dbt projects to avoid checking credentials into version control. Database credentials are extremely sensitive information and should never be checked into version control.

 What should I name my profile?

We typically use a company name for a profile name, and then use targets to differentiate between dev and prod. Check out the docs on managing environments for more information.

 What should I name my target?

We typically use targets to differentiate between development and production runs of dbt, naming the targets dev and prod respectively. Check out the docs on managing environments for more information.

 Can I use environment variables in my profile?

Yes! Check out the docs on environment variables for more information.

Perform your first dbt run

Our sample project has some example models in it. We're going to check that we can run them to confirm everything is in order.

1. Execute the run command to build the example models:

$ dbt run

You should have an ouput that looks like this:

A successful dbt run command

Commit your changes

We need to commit our changes so that our repository has up-to-date code.

1. Link the GitHub repository you created to your dbt project by running the following commands. Make sure you use the correct git URL for your repository.

$ git init
$ git branch -M main
$ git add .
$ git commit -m "Create a dbt project"
$ git remote add origin https://github.com/USERNAME/dbt-tutorial.git
$ git push -u origin main

info

If this is your first time using git, it's worth taking some time to understand the basics.