Daedalus

Appearance

Getting Started ​

This page walks you from a fresh checkout to your first training run.

Prerequisites ​

  • Python 3.13+ — the project requires >=3.13 (pinned in .python-version at the repo root).
  • uv — the package and project manager used for every command below.

Install ​

Install the project and its dev dependencies (pytest, ruff):

bash
uv sync --dev

This installs the core engine. Both optional backends import lazily, so a plain import daedalus never requires them — install an extra only when you actually use that backend.

Optional extras ​

bash
# Unlocks catalog.table.SnowflakeSource:
#   key-pair JWT auth + Arrow query pushdown
uv sync --extra snowflake
bash
# Unlocks the MLflowTracker (mlflow-skinny) projection of training runs
uv sync --extra mlflow
  • snowflake — pulls in snowflake-connector-python + cryptography so a data source can be backed by Snowflake (snowflake://<account> with key-pair JWT auth). Without it, the Snowflake source stays unimportable but the rest of Daedalus works.
  • mlflow — pulls in mlflow-skinny so each TrainingRunRecord (feature service @ version, git SHA, config paths) can be projected to MLflow. Tracking is off by default (a no-op NullTracker); enable it by setting TrainingRunConfig.tracking.kind="mlflow".

Point at your data ​

Daedalus ingests data directly from your sources, read in place — there is no download or copy step. You don't stage data into the project first; you point a feature view's source at where the data already lives. A source is a config-only choice of one of four backends:

  • Postgres (read-only)
  • S3 Hive-partitioned Parquet (s3:// / r2:// / local)
  • DuckLake tables
  • Snowflake (read-only)

For example, a view backed by hive-partitioned parquet in object storage:

yaml
# feature_views/feed_events.yaml
name: feed_events
entities:
  - user
  - artwork
source:
  name: feed_events_parquet
  path: "s3://pixai-features/feed"     # read in place — recursive hive glob
  s3_key_id: "${S3_KEY_ID}"            # credentials via ${ENV}, never inline
  s3_secret: "${S3_SECRET}"
  s3_region: "${S3_REGION}"
  timestamp_field: event_timestamp
features:
  - name: event
    dtype: VARCHAR

A local parquet directory works the same way — it is just one direct source example, not a required staging step. See the Data Sources Overview for the dispatch table and the per-backend recipes (Postgres, S3 parquet, DuckLake, Snowflake), and inject every credential as a ${ENV} reference (never inline a secret).

First commands ​

Inspect the feature catalog ​

Start by browsing what features exist — no Python required:

bash
# List every feature view with a one-line summary
uv run daeda catalog list

# Print a view's metadata + schema as JSON
uv run daeda catalog show user_profile

Run a training pipeline ​

Training is a single canonical path (Core v1): daeda pipeline train <service> compiles the feature service into one operator DAG and runs it through the executor — one pipeline, no separate stages.

bash
# Full run over the month range configured in settings.yaml
uv run daeda pipeline train pixai_feed_relevance

Useful flags for iterating:

bash
# Narrow to a single day (YYYY-MM-DD)
uv run daeda pipeline train pixai_feed_relevance --target-date 2026-05-15

# Run just one engine sub-range (source-side operators / enrichment operators)
uv run daeda pipeline train pixai_feed_relevance --only source --target-date 2026-05-15
uv run daeda pipeline train pixai_feed_relevance --only enrich --target-date 2026-05-15

The processed month range (feed_start / feed_end) and other knobs are config-driven — see Configuration. The available service is pixai_feed_relevance (the shared feed-relevance dataset — DSSM, XGBoost, and other rankers each select the columns they need).

Enrich needs a local Lance store

The enrichment operators pool embeddings out of a Lance store (data/pixai-relevance-v2-embeddings by default) that must sit on a local filesystem — Lance's atomic-rename requirement rules out GPFS / NFS. It is the one artifact Daedalus cannot read in place, so set it up first: Local Lance embedding store covers the download and daeda store build. --only source does not need it.

Running tests ​

Run the full suite:

bash
uv run pytest

Tests are auto-marked by directory (see conftest.py). Run a subset by marker:

bash
uv run pytest -m unit              # fast unit tests
uv run pytest -m integration       # integration tests (slow)
uv run pytest -m cli               # CLI behavior tests
uv run pytest -m "unit and not slow"

Or target a single file or test:

bash
uv run pytest tests/catalog/test_loader.py
uv run pytest tests/catalog/test_loader.py::test_load_feature_view_from_yaml

Lint and format with ruff (line length 88):

bash
uv run ruff check .
uv run ruff format .

Next steps ​