Setup

Setup Your Environment

Get your Daana CLI environment up and running in 5 minutes.

Overview

You'll set up three things:

  1. Daana CLI - Download the binary for your platform
  2. Olist Quickstart Project - Complete tutorial package with databases, data, and examples
  3. Daana Framework - Install the transformation engine

What You'll Get

After this 5-minute setup, you'll have:

  • ✅ Daana CLI installed and working
  • ✅ Two PostgreSQL databases running (internaldb + customerdb)
  • ✅ Real e-commerce data loaded (100,000+ Olist orders)
  • ✅ Complete business model (CUSTOMER and ORDER entities)
  • ✅ Daana framework installed and ready

Step 1: Download Daana CLI

For Beta Testers

The Daana team will provide you with a direct download link for the binary. Once you have the binary:

# macOS/Linux: Extract and move to PATH
tar -xzf daana-cli_*_<OS>_<ARCH>.tar.gz
sudo mv daana-cli /usr/local/bin/
 
# Verify installation
daana-cli --version

Note: If you don't have the download link yet, contact your Daana representative. The repository is private during beta.

✅ Checkpoint: You should see version information (e.g., v0.5.7 or later)

For Contributors

If you're building from source:

git clone --recurse-submodules https://github.com/daana-code/daana-cli.git
cd daana-cli
task setup
task build

Step 2: Create Tutorial Project

Create a new project with the Olist tutorial example:

# Create project with Olist example data
daana-cli init --example olist
 
# This creates: olist-quickstart/ directory with:
#   - Docker Compose for databases
#   - Real Olist e-commerce dataset (100k+ orders)
#   - Pre-built business model (CUSTOMER and ORDER entities)
#   - Configuration files ready to use
 
# Navigate into the project
cd olist-quickstart

✅ Checkpoint: You should see a new olist-quickstart/ directory with docker-compose.yml, model.yaml, connections.yaml, and workflow.yaml

Note: The init command will create .gitignore for you, which already includes *.bak to keep backup files out of version control.

Step 3: Start Databases

Daana CLI uses two PostgreSQL databases:

  • internaldb (port 5434): Daana's internal database - stores your models, mappings, and workflows
  • customerdb (port 5432): Your data warehouse - contains source data and transformed business entities

The tutorial project includes a docker-compose.yml file that sets up both databases:

# From olist-quickstart/ directory
docker-compose up -d
 
# Wait ~30 seconds for databases to start
docker-compose ps
 
# You should see 2 containers running:
#   daana-customerdb (port 5432)
#   daana-internaldb (port 5434)

✅ Checkpoint: Run docker ps - you should see 2 containers running

Verify Databases Are Ready

# Check database health
docker-compose ps
 
# Both should show "Up" status

Step 4: Load Olist Sample Data

The init command created the Olist dataset file in data/olist-sample-data.sql.gz (24MB compressed). Now let's load this real Brazilian e-commerce data into the database:

# From olist-quickstart/ directory
gunzip -c data/olist-sample-data.sql.gz | docker exec -i daana-customerdb psql -U dev -d customerdb
 
# This takes ~30 seconds
 
# Verify data loaded successfully
docker exec -it daana-customerdb psql -U dev -d customerdb -c "SELECT COUNT(*) FROM stage.olist_orders_dataset;"
 
# Should show: ~99,441 orders

What just happened: The data file was already included when you ran init --example olist. This step loads it from your local data/ directory into the PostgreSQL database.

✅ Checkpoint: You should see "count" with ~99,441 rows

Explore the Source Data (Optional)

# Connect to customerdb
docker exec -it daana-customerdb psql -U dev -d customerdb
 
# Inside psql, explore the tables:
\dt stage.*                                           # List all tables in stage schema
SELECT * FROM stage.olist_orders_dataset LIMIT 5;    # See sample orders
SELECT * FROM stage.olist_customers_dataset LIMIT 5; # See sample customers
\q                                                    # Exit

Step 5: Install Daana Framework

Install the Daana transformation framework into both databases:

# From olist-quickstart/ directory
daana-cli install --connections connections.yaml --profile dev

This command:

  • ✅ Installs Daana framework tables in internaldb (port 5434)
  • ✅ Installs transformation infrastructure in customerdb (port 5432, using the "dev" profile from connections.yaml)

You'll see installation progress and "Installation completed successfully" at the end.

✅ Checkpoint: Installation completes without errors

Note: If installation fails, see troubleshooting below.

Verify Your Setup

Let's verify everything is working:

1. Check Daana CLI

daana-cli --version

Should show: daana-cli version <Version /> (or later)

2. Check Databases

docker-compose ps
# Should show 2 containers running

3. Check internaldb (Daana's internal database)

docker exec -it daana-internaldb psql -U dev -d internaldb
 
# Inside psql, check for Daana tables
\dt
 
# You should see tables like: workflow, model_definition, entity_mapping, etc.
# Exit with: \q

4. Check Customer Database (your data warehouse)

docker exec -it daana-customerdb psql -U dev -d customerdb
 
# Inside psql, check schemas
\dn
 
# You should see: public (with Olist tables) and daana_dw (empty, ready for transformed data)
 
# Check source data
SELECT COUNT(*) FROM stage.olist_orders_dataset;    # ~99,441 orders
SELECT COUNT(*) FROM stage.olist_customers_dataset; # ~99,441 customers
 
# Exit with: \q

What You Have Now

🎉 Success! You now have:

  • ✅ Daana CLI installed and working
  • ✅ Two databases running:
    • internaldb (port 5434): Daana's internal database
    • customerdb (port 5432): Your data warehouse with Olist data
  • ✅ Real e-commerce data: 100,000+ orders, 99,000+ customers
  • ✅ Tutorial project structure with all configuration files
  • ✅ Daana framework installed in both databases
  • ✅ Complete business model with CUSTOMER and ORDER entities (model.yaml)

Understanding the Architecture

Why Two Databases?

Think of it like this:

  • internaldb (Daana's brain): Stores your blueprints (models, mappings, workflows). When you run daana-cli materialize model, it goes here.

  • customerdb (Your warehouse): This is where the magic happens:

    • Source data lives in stage schema (Olist: stage.olist_customers_dataset, stage.olist_orders_dataset, etc.)
    • Transformed data goes in the daana_dw schema
    • Your BI tools will query daana_dw tables

You configure both databases in connections.yaml and select which profile to use (e.g., dev, prod) when running commands.

What's in the Tutorial Project?

olist-quickstart/
├── docker-compose.yml              # Database setup
├── connections.yaml                # Database connection profiles
├── model.yaml                      # Complete business model (CUSTOMER and ORDER)
├── workflow.yaml                   # Workflow configuration
├── mappings/                       # You'll create mapping files here (generated in tutorial)
└── data/
    └── olist-sample-data.sql.gz   # Real e-commerce data (25 MB compressed)

Next Steps

Ready to transform some real data? Continue to the Tutorial → where you'll:

  • Explore the Olist source data
  • Understand the business model
  • Map raw data to business entities
  • Learn to iterate on your model without losing work
  • Execute the transformation
  • See clean business entities in daana_dw schema!

Troubleshooting

"Cannot connect to Docker daemon"

Make sure Docker Desktop is running:

  • macOS: Check menu bar for Docker icon
  • Windows: Check system tray for Docker icon
  • Linux: Run sudo systemctl start docker

"Port already in use"

If ports 5432 or 5434 are already in use:

# Check what's using the ports
lsof -i :5432
lsof -i :5434
 
# Stop any existing PostgreSQL
# macOS: brew services stop postgresql
# Linux: sudo systemctl stop postgresql

Installation fails with "connection refused"

Wait a bit longer for databases to start:

# Check if databases are ready
docker logs daana-internaldb
docker logs daana-customerdb
 
# Look for: "database system is ready to accept connections"

Installation fails

If installation fails, restart the databases:

docker-compose down -v
docker-compose up -d
 
# Wait ~30 seconds, then try again
daana-cli install --connections connections.yaml --profile dev

Data didn't load

# Reload the Olist data
gunzip -c data/olist-sample-data.sql.gz | docker exec -i daana-customerdb psql -U dev -d customerdb
 
# Verify tables exist
docker exec -it daana-customerdb psql -U dev -d customerdb -c "\dt"

Reset Everything

If you want to start fresh:

# From olist-quickstart/ directory
 
# Stop and remove databases (⚠️ This deletes all data!)
docker-compose down -v
 
# Start databases again
docker-compose up -d
 
# Reload Olist data
gunzip -c data/olist-sample-data.sql.gz | docker exec -i daana-customerdb psql -U dev -d customerdb
 
# Reinstall framework
daana-cli install --connections connections.yaml --profile dev

Need Help?

Everything set up? Let's start transforming data! Head to the Tutorial →

PrerequisitesTutorial