CLI Reference¶

QuickETL provides a command-line interface for running and managing pipelines.

Quick Reference¶

Command	Description
`run`	Execute a pipeline
`validate`	Validate configuration
`workflow`	Run and manage multi-pipeline workflows
`init`	Create new project or pipeline
`info`	Display version and backend info
`schema`	Output JSON schema for IDE support

Global Options¶

quicketl --version    # Show version
quicketl --help       # Show help

Exit Codes¶

Code	Meaning
0	Success
1	Error (validation, execution, etc.)

run¶

Execute a pipeline from a YAML configuration file.

Usage¶

quicketl run <config_file> [options]

Options¶

Option	Short	Description
`--engine`	`-e`	Override compute engine
`--var`	`-v`	Set variable (KEY=VALUE), can be repeated
`--dry-run`		Execute without writing output
`--fail-on-checks`		Fail on quality check failure (default)
`--no-fail-on-checks`		Continue despite check failures
`--verbose`	`-V`	Enable verbose logging
`--json`	`-j`	Output result as JSON

Examples¶

# Basic run
quicketl run pipeline.yml

# With variables
quicketl run pipeline.yml --var DATE=2025-01-15
quicketl run pipeline.yml --var DATE=2025-01-15 --var REGION=north

# Override engine
quicketl run pipeline.yml --engine polars
quicketl run pipeline.yml --engine spark

# Dry run (no output written)
quicketl run pipeline.yml --dry-run

# Continue on check failure
quicketl run pipeline.yml --no-fail-on-checks

# JSON output (for scripting)
quicketl run pipeline.yml --json

Output¶

Running pipeline: sales_etl
  Engine: duckdb

╭─────────────── Pipeline: sales_etl ───────────────╮
│ SUCCESS                                           │
╰───────────────────── Duration: 245.3ms ───────────╯

Steps
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┓
┃ Step           ┃ Type    ┃ Status ┃ Duration ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━┩
│ read_source    │ file    │ OK     │   45.2ms │
│ transform_0    │ filter  │ OK     │    0.3ms │
│ quality_checks │ checks  │ OK     │   12.4ms │
│ write_sink     │ file    │ OK     │    8.1ms │
└────────────────┴─────────┴────────┴──────────┘

Quality Checks: PASSED (2/2)
Rows processed: 1000 → Rows written: 950

JSON Output¶

{
  "pipeline_name": "sales_etl",
  "status": "SUCCESS",
  "duration_ms": 245.3,
  "rows_processed": 1000,
  "rows_written": 950,
  "checks_passed": 3,
  "checks_failed": 0
}

validate¶

Validate a pipeline configuration without executing it.

Usage¶

quicketl validate <config_file> [options]

Options¶

Option	Short	Description
`--verbose`	`-v`	Show detailed configuration

Examples¶

# Basic validation
quicketl validate pipeline.yml

# Verbose output
quicketl validate pipeline.yml --verbose

Output¶

Valid configuration:

Configuration is valid

Pipeline: sales_etl
  Engine: duckdb
  Source: file (data/sales.parquet)
  Transforms: 3
  Checks: 2
  Sink: file (output/results.parquet)

Invalid configuration:

Configuration is invalid

Errors:
  - transforms -> 0 -> op: Input should be 'select', 'filter', ...
  - sink: Field required

CI/CD Usage¶

# .github/workflows/validate.yml
- name: Validate pipelines
  run: |
    for f in pipelines/*.yml; do
      quicketl validate "$f"
    done

workflow¶

Run and manage multi-pipeline workflows with dependency management and parallel execution.

Subcommands¶

Subcommand	Description
`run`	Execute a workflow
`validate`	Validate workflow configuration
`info`	Show workflow structure and execution order
`generate`	Generate Airflow DAG or Prefect flow

workflow run¶

Execute a workflow from a YAML configuration file.

quicketl workflow run <config_file> [options]

Options¶

Option	Short	Description
`--var`	`-v`	Set variable (KEY=VALUE), can be repeated
`--dry-run`		Execute without writing output
`--workers`	`-w`	Maximum parallel workers for parallel stages
`--verbose`	`-V`	Enable verbose logging
`--json`	`-j`	Output result as JSON

Examples¶

# Basic run
quicketl workflow run workflows/medallion.yml

# With variables
quicketl workflow run workflows/etl.yml --var DATE=2025-01-15

# Dry run
quicketl workflow run workflows/etl.yml --dry-run

# Limit parallel workers
quicketl workflow run workflows/etl.yml --workers 4

# JSON output for scripting
quicketl workflow run workflows/etl.yml --json

Output¶

Running workflow: medallion_etl
  Stages: 3
  Pipelines: 9

╭────────────────────── Workflow: medallion_etl ──────────────────────╮
│ SUCCESS                                                              │
╰───────────────────────── Duration: 1643.7ms ─────────────────────────╯
                        Stages
┏━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Stage        ┃ Status ┃ Pipelines ┃ Duration ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━┩
│ bronze       │ OK     │       4/4 │ 1314.2ms │
│ silver_clean │ OK     │       3/3 │  205.9ms │
│ silver_agg   │ OK     │       2/2 │  122.8ms │
└──────────────┴────────┴───────────┴──────────┘
Pipelines: 9/9 succeeded

workflow validate¶

Validate a workflow configuration without executing it.

quicketl workflow validate <config_file> [options]

Options¶

Option	Short	Description
`--verbose`	`-V`	Show detailed workflow structure

Examples¶

# Basic validation
quicketl workflow validate workflows/medallion.yml

# Show structure
quicketl workflow validate workflows/medallion.yml --verbose

Validation checks:

YAML syntax is valid
All required fields are present
Stage dependencies are valid (no cycles, no missing deps)
All referenced pipeline files exist

workflow info¶

Display workflow structure and execution order.

quicketl workflow info <config_file>

Example¶

quicketl workflow info workflows/medallion.yml

Output:

Workflow: medallion_etl
  Bronze -> Silver medallion architecture pipeline

Execution Order:
  1. bronze
  2. silver_clean, silver_agg
  3. gold

                        Stages
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ Stage        ┃ Pipelines ┃ Parallel ┃ Depends On       ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ bronze       │         4 │ Yes      │ -                │
│ silver_clean │         3 │ Yes      │ bronze           │
│ silver_agg   │         2 │ No       │ silver_clean     │
│ gold         │         2 │ No       │ silver_agg       │
└──────────────┴───────────┴──────────┴──────────────────┘

Variables:
  DATA_DIR: ./data

workflow generate¶

Generate orchestration code from a workflow configuration.

quicketl workflow generate <config_file> [options]

Options¶

Option	Short	Description
`--target`	`-t`	Target orchestrator: `airflow` or `prefect` (default: airflow)
`--output`	`-o`	Output file path (prints to stdout if not specified)
`--dag-id`		DAG/flow ID (defaults to workflow name)
`--schedule`	`-s`	Cron schedule (Airflow only)

Examples¶

# Generate Airflow DAG to stdout
quicketl workflow generate workflows/etl.yml --target airflow

# Save to file with schedule
quicketl workflow generate workflows/etl.yml --target airflow -o dags/etl_dag.py --schedule "0 0 * * *"

# Generate Prefect flow
quicketl workflow generate workflows/etl.yml --target prefect -o flows/etl_flow.py

# Custom DAG ID
quicketl workflow generate workflows/etl.yml --target airflow --dag-id daily_etl_v2

See Workflow Documentation for complete workflow configuration reference.

init¶

Initialize QuickETL in the current directory or create a new project.

Usage¶

quicketl init [name] [options]

Arguments¶

Argument	Required	Description
`name`	No	Project name. If omitted, initializes in current directory

Options¶

Option	Short	Description
`--pipeline`	`-p`	Create pipeline file only (requires name)
`--output`	`-o`	Output directory (default: current)
`--force`	`-f`	Overwrite existing files

Examples¶

# Initialize in current directory (existing project)
quicketl init

# Create new project in subdirectory
quicketl init my_project
cd my_project

# Create pipeline file only
quicketl init my_pipeline -p

# Specify output directory
quicketl init my_project -o ./projects/

In Current Directory¶

When run without a name, quicketl init adds QuickETL structure to your existing project:

your_project/
├── pipelines/
│   └── sample.yml      # Working sample pipeline
├── data/
│   ├── sales.csv       # Sample data
│   └── output/         # Pipeline outputs
└── .env                # Created only if not present

Existing files (README.md, .gitignore, etc.) are preserved.

New Project Structure¶

When run with a name, creates a complete project:

my_project/
├── pipelines/
│   └── sample.yml      # Working sample pipeline
├── data/
│   ├── sales.csv       # Sample data
│   └── output/         # Pipeline outputs
├── scripts/            # Custom Python scripts
├── README.md
├── .env
└── .gitignore

The sample pipeline is immediately runnable:

quicketl run pipelines/sample.yml

info¶

Display QuickETL version and backend information.

Usage¶

quicketl info [options]

Options¶

Option	Short	Description
`--backends`	`-b`	Show available backends
`--check`	`-c`	Check backend availability

Examples¶

# Version info
quicketl info

# List backends with availability check
quicketl info --backends --check

Output¶

QuickETL v0.1.0
Python 3.12.0

With --backends --check:

Available Backends
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Backend   ┃ Name           ┃ Status         ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ duckdb    │ DuckDB         │ OK             │
│ polars    │ Polars         │ OK             │
│ spark     │ Apache Spark   │ Not installed  │
│ snowflake │ Snowflake      │ Not installed  │
└───────────┴────────────────┴────────────────┘

schema¶

Output JSON schema for pipeline configuration (for IDE autocompletion).

Usage¶

quicketl schema [options]

Options¶

Option	Short	Description
`--output`	`-o`	Output file path (default: stdout)
`--indent`	`-i`	JSON indentation level (default: 2)

Examples¶

# Output to stdout
quicketl schema

# Save to file
quicketl schema -o .quicketl-schema.json

VS Code Integration¶

quicketl schema -o .quicketl-schema.json

Then in .vscode/settings.json:

{
  "yaml.schemas": {
    ".quicketl-schema.json": ["pipelines/*.yml"]
  }
}

Shell Completion¶

Enable tab completion for commands and options:

# Bash
quicketl --install-completion bash

# Zsh
quicketl --install-completion zsh

# Fish
quicketl --install-completion fish

Environment Variables¶

The CLI respects environment variables for configuration:

export DATABASE_URL=postgresql://localhost/db
export SNOWFLAKE_ACCOUNT=xy12345.us-east-1
quicketl run pipeline.yml

See Environment Variables for the full list.