Daedalus

Appearance

What is Daedalus? ​

Daedalus is the feature platform behind PixAI's recommendation system. It turns declarative YAML feature definitions into point-in-time-correct training datasets, and wraps the whole thing in a lean operational layer (orchestration, an HTTP control plane, agent interop, and a CLI) over a single shared engine.

The name is the project's north star: a small set of well-fitted parts — a catalog, an engine, a pipeline, and a thin platform skin — that compose into something that flies.

The big picture ​

Daedalus is built as four layers, each one feeding the next:

  1. Declarative YAML feature catalog — Feature views live as YAML in feature_views/*.yaml and are loaded by FeatureCatalog.from_yaml_dir(). Each view carries its schema, source metadata, and the canonical entity join keys (user → user_id, artwork → artwork_id). Adding a feature is a YAML edit, not a code change. See Feature Catalog.

  2. DuckDB / Polars rolling-aggregation engine — A point-in-time aggregation engine (aggregate_pit_table) with two interchangeable, bit-for-bit equivalent implementations: a DuckDB windowed list-aggregation path (default) and a Polars rolling().agg() path kept for A/B testing and fallback. You pick one via the compute.engine field in a service's settings.yaml. See Compute Engine.

  3. One operator pipeline — A feature service compiles into a single engine-tagged operator DAG that the executor runs end to end — there are no separate "skeleton" and "enrich" stages, just operators. Source-side operators do the feature joins + point-in-time rolling aggregations; a pythonic operator appends avg-pooled artwork embeddings via Ray Data over a Lance store. Each operator is optimized at its own (sub-operator) level. See Operator Pipeline.

  4. A lean platform layer — The same in-process training API is exposed through four surfaces, with no feature logic re-derived:

    • Dagster orchestration (daedalus.definitions) — each training dataset is a daily partitioned graph-backed asset.
    • A FastAPI JSON-RPC control plane (daeda serve-api).
    • An MCP server (daeda mcp) for external agents.
    • The daeda agent CLI (config, lineage, materialize-day).

    See Platform Surfaces.

Core v1: one canonical training path ​

Training is intentionally a single entry point — daeda pipeline train <service> — referred to throughout the docs as Core v1. It compiles the feature service into one operator DAG and runs it through the executor. The standalone daeda skeleton / daeda enrich commands were removed in the Core-v1 lean cut (last dual-path tag: v0.7.1); output is byte-identical to that older pair.

bash
# Compile a service to an editable operator DAG (inspect / hand-tune)
daeda pipeline compile pixai_feed_relevance

# Run the one pipeline (the canonical path)
daeda pipeline train pixai_feed_relevance

Who consumes Daedalus ​

Daedalus produces the training datasets for PixAI's recommender models. Feature services are the unit of consumption — each one resolves a set of feature views into a typed training schema:

  • pixai_feed_relevance — the canonical, model-neutral For-You feed training dataset, shared across the DSSM retrieval tower, the XGBoost reranker, and other downstream models. Each consumer selects the subset of columns it needs from this one contract.

Additional consumers (Node2Vec graph embeddings, search) are planned and will be added as new feature services without changes to the engine.

Declarative by design

New features and new consumers are almost always a YAML change. The engine, pipeline, and platform layers stay fixed — you describe what you want in feature_views/ and feature_services/, and Daedalus compiles and runs it.

Next steps ​

  • Getting Started — install Daedalus, link your data, and run your first training pipeline.
  • Architecture Overview — a deeper tour of the catalog, engine, pipeline, and platform layers.