Run Airflow and dbt pipelines on Snowflake locally with LocalStack

Introduction

dbt on Snowflake can be orchestrated in many ways: CI pipelines, scheduled jobs, or workflow tools like Apache Airflow. When teams do use Airflow with Astronomer Cosmos (which turns dbt projects into Airflow task groups), validating the full DAG against Snowflake often means pointing at a shared cloud account or waiting on CI. That feedback loop is slow for iterative SQL work.

You can run Airflow locally against Snowflake Cloud, but testing the full path (including an MWAA-style environment, Cosmos task groups, and Snowflake transformations) without touching shared cloud resources is harder.

LocalStack’s Snowflake emulator and MWAA (Managed Workflows for Apache Airflow) emulator let you run that stack on a single machine. In this tutorial, we’ll set up a local Airflow environment with a dbt project that seeds hotel booking data into Snowflake, transforms it through dbt models, and queries the results.

How does our sample application work?

The sample app models a hotel booking system. Three CSV seed files (two booking tables and a customer table) get loaded into Snowflake, transformed through dbt models, and queried at the end to verify the output.

Sample app layout

Orchestration Layer

Amazon MWAA (Managed Workflows for Apache Airflow) handles the orchestration, which LocalStack emulates locally. When you call mwaa create-environment, LocalStack spins up an Airflow container on your machine, wires it to a local S3 bucket, and installs the Python dependencies.

The dbt project builds five models on top of the seed data:

combined_bookings unions the two booking CSVs using dbt_utils.union_relations
customer selects fields from the customers seed
prepped_data joins customers with their bookings
hotel_count_by_day counts bookings per hotel per day
thirty_day_avg_cost calculates a 30-day rolling average cost using a window function

Astronomer Cosmos converts this dbt project into native Airflow tasks. Each seed becomes a seed task, each model becomes a run task, and dbt’s dependency graph maps to Airflow task dependencies.

The DAG defines a DbtTaskGroup pointing at the dbt project directory. The operator_args={"install_deps": True} flag tells Cosmos to run dbt deps before each task, installing the dbt_utils package from packages.yml.

Connection setup

The Snowflake connection is defined via an AIRFLOW_CONN_SNOWFLAKE_LOCAL environment variable in the DAG file. Airflow registers connections from environment variables following the AIRFLOW_CONN_<CONN_ID> naming convention, creating a connection called snowflake_local:

os.environ["AIRFLOW_CONN_SNOWFLAKE_LOCAL"] = json.dumps({
    "conn_type": "snowflake",
    "login": "test",
    "password": "test",
    "extra": {
        "host": "snowflake.localhost.localstack.cloud",
        "port": 4566,
        "account": "test",
        "database": "test",
        "schema": "public",
    },
})

The default SnowflakeUserPasswordProfileMapping in Cosmos doesn’t map host and port from the connection’s extra field, so the DAG patches the mapping at module level:

SnowflakeUserPasswordProfileMapping.airflow_param_mapping["host"] = "extra.host"
SnowflakeUserPasswordProfileMapping.airflow_param_mapping["port"] = "extra.port"

This passes the emulator’s hostname and port through to dbt when Cosmos generates the profile.

Task graph

After the dbt task group finishes, an EmptyOperator acts as a sync point before the final query_result_data task. That task opens a Snowflake connection and runs SELECT * FROM transform.PREPPED_DATA, printing all 14 rows of joined booking data. The DAG uses max_active_tasks=1 to run tasks sequentially and avoid concurrent CREATE TABLE collisions in the Snowflake emulator.

With that explanation out of the way, let’s get started!

Prerequisites

Docker
localstack CLI with a valid LOCALSTACK_AUTH_TOKEN (available with a LocalStack license)
awslocal CLI
Python 3 and curl

Step 1: Start LocalStack for Snowflake emulator

The Snowflake emulator and the MWAA container need to communicate over a shared Docker network. Create the network and start LocalStack:

docker network create --attachable --subnet 172.20.0.0/24 localstack

DOCKER_FLAGS='-e SF_LOG=trace --network localstack --name=localhost.localstack.cloud --network-alias=snowflake.localhost.localstack.cloud' \
DEBUG=1 \
localstack start -s snowflake -d

Wait for LocalStack to be ready:

localstack wait -t 120

Verify the Snowflake emulator is running:

curl -d '{}' snowflake.localhost.localstack.cloud:4566/session

The expected output is:

{"success": true}

Step 2: Clone the sample application

Clone the repository and navigate to the project directory:

git clone https://github.com/localstack-samples/localstack-snowflake-samples.git
cd localstack-snowflake-samples/airflow-dbt-transformation

The project layout:

airflow-dbt-transformation/
├── airflow_dag.py            # Airflow DAG definition with Cosmos + Snowflake
├── dbt_project.yml           # dbt project configuration
├── packages.yml              # dbt package dependencies (dbt_utils)
├── requirements.txt          # Python packages for the Airflow container
├── Makefile                  # Automation targets for init/deploy
├── macros/
│   └── custom_demo_macros.sql  # Schema name override for dbt
├── models/
│   ├── transform/            # Core transformation models
│   │   ├── combined_bookings.sql
│   │   ├── customer.sql
│   │   └── prepped_data.sql
│   └── analysis/             # Analytical models
│       ├── hotel_count_by_day.sql
│       └── thirty_day_avg_cost.sql
└── seeds/                    # Raw CSV data
    ├── bookings_1.csv
    ├── bookings_2.csv
    └── customers.csv

Step 3: Create the MWAA environment

Create an S3 bucket and upload the requirements.txt:

awslocal s3 mb s3://snowflake-airflow
awslocal s3 cp requirements.txt s3://snowflake-airflow/

Create the MWAA environment:

awslocal mwaa create-environment \
  --dag-s3-path /dags \
  --execution-role-arn arn:aws:iam::000000000000:role/airflow-role \
  --network-configuration {} \
  --source-bucket-arn arn:aws:s3:::snowflake-airflow \
  --requirements-s3-path /requirements.txt \
  --airflow-configuration-options agent.code=007,agent.name=bond \
  --name my-mwaa-env \
  --endpoint-url http://localhost.localstack.cloud:4566

The expected output is:

{
    "Arn": "arn:aws:airflow:us-east-1:000000000000:environment/my-mwaa-env"
}

LocalStack spins up an Airflow container in the background and installs the Python dependencies from requirements.txt (including astronomer-cosmos, dbt-snowflake, and the Snowflake connector). Check that the environment is ready:

awslocal mwaa get-environment --name my-mwaa-env --query 'Environment.[Status,WebserverUrl]' --output text

The expected output is:

AVAILABLE  http://localhost.localstack.cloud:4510

Note down the Airflow webserver URL, as the port number can vary between runs.

Step 4: Deploy the DAG and dbt project

Upload the dbt project files and the DAG to the S3 bucket:

awslocal s3 cp packages.yml s3://snowflake-airflow/dags/
awslocal s3 cp dbt_project.yml s3://snowflake-airflow/dags/
awslocal s3 cp --recursive models s3://snowflake-airflow/dags/models
awslocal s3 cp --recursive seeds s3://snowflake-airflow/dags/seeds
awslocal s3 cp --recursive macros s3://snowflake-airflow/dags/macros
awslocal s3 cp airflow_dag.py s3://snowflake-airflow/dags/

The MWAA emulator syncs files from S3 into the Airflow container at roughly 30-second intervals. Wait about 30-40 seconds, then verify the files:

docker exec localstack-mwaa-000000000000-us-east-1-my-mwaa-env ls /opt/airflow/dags/

You should see airflow_dag.py, dbt_project.yml, packages.yml, and the macros/, models/, seeds/ directories. If any are missing, wait another 30 seconds and check again.

The Python dependencies (astronomer-cosmos, dbt-snowflake) get installed from the requirements.txt uploaded earlier. This can take 1-2 minutes on a first run. Verify the install:

docker exec localstack-mwaa-000000000000-us-east-1-my-mwaa-env bash -c "pip show astronomer-cosmos" 2>/dev/null | head -2

The expected output is:

Name: astronomer-cosmos
Version: 1.13.1

If nothing is returned, the install is still in progress. Give it another minute.

Step 5: Trigger a DAG run

Restart the Airflow services so the DAG processor picks up the new code:

docker exec localstack-mwaa-000000000000-us-east-1-my-mwaa-env bash -c "supervisorctl restart all"

airflow_scheduler: stopped
airflow_dag_processor: stopped
airflow_triggerer: stopped
airflow_api_server: stopped
airflow_api_server: started
airflow_dag_processor: started
airflow_scheduler: started
airflow_triggerer: started

Wait about 5-10 seconds for the DAG processor, then open the Airflow webserver URL (from Step 3) in your browser. Log in with localstack / localstack as the username and password.

The DAGs page should show the dbt_snowpark DAG. Click on it to open the DAG detail page. Toggle the pause switch to unpause it, then click Trigger DAG to start a manual run.

Airflow DAGs page

The pipeline takes about 60-90 seconds. The execution order is:

Seed tasks: bookings_1_seed, bookings_2_seed, customers_seed load CSVs into Snowflake
Transform models: customer_run and combined_bookings_run, then prepped_data_run joins their output
Analysis models: hotel_count_by_day_run and thirty_day_avg_cost_run operate on the prepped data
Intermediate: an EmptyOperator sync point
Query task: query_result_data queries transform.PREPPED_DATA and prints the results

When the run finishes, all 10 tasks should show green in the grid. Click on any task to inspect its logs. The query_result_data task log shows all 14 rows of joined booking data that the pipeline produced.

Airflow DAG detail page

The graph view shows the full dependency chain from seeds through transforms to analysis and the final query. You can trigger additional runs to verify the pipeline is idempotent.

Airflow DAG graph view

Summary

This setup gives you a fully local Airflow + dbt + Snowflake environment. The MWAA emulator handles Airflow, the Snowflake emulator handles the data warehouse, and Cosmos wires dbt into Airflow’s task system. You can iterate on SQL models, change DAG configuration, and re-deploy in under two minutes.

The sample application is available on GitHub. You can find more about LocalStack’s Snowflake emulator in the documentation and about MWAA support in the MWAA docs.

Run Airflow and dbt pipelines on Snowflake locally with LocalStack

Introduction

How does our sample application work?

Orchestration Layer

Connection setup

Task graph

Prerequisites

Step 1: Start LocalStack for Snowflake emulator

Step 2: Clone the sample application

Step 3: Create the MWAA environment

Step 4: Deploy the DAG and dbt project

Step 5: Trigger a DAG run

Summary

About the Author

Launch yourself in the world of local cloud development

Introduction

How does our sample application work?

Orchestration Layer

Connection setup

Task graph

Prerequisites

Step 1: Start LocalStack for Snowflake emulator

Step 2: Clone the sample application

Step 3: Create the MWAA environment

Step 4: Deploy the DAG and dbt project

Step 5: Trigger a DAG run

Summary

About the Author

Subscribe to our Newsletter

Success!

Related Articles

Run AWS Glue ETL Jobs with Snowflake Locally Using LocalStack

Running CDK Integration Tests Locally with LocalStack

Build and Test AWS Lambda Extensions Locally with LocalStack

Launch yourself in the world of local cloud development