Overview
Daedalus is a feature platform built around one idea: a feature service is a declarative contract that compiles to one pipeline — a DAG of engine-tagged operators — which the executor runs directly against your data sources. There is no copy step and no separate "skeleton" and "enrich" stages: just operators, each optimized at its own level.
One pipeline — an operator DAG. A feature service compiles to a single operator DAG. Each operator carries a declared engine, and optimization happens per operator (sub-operator level) — there is no global query planner.
Interact
Hover a layer for its description, and click an engine chip to highlight which operators run on it. Nodes link to the relevant section.
The layers
1. Ingestion — direct sources
Daedalus reads data in place. A feature view points its source at one of four backends and DuckDB reads it directly — there is no download/copy stage:
- Postgres (read-only)
- S3 Hive-partitioned Parquet
- DuckLake tables
- Snowflake (read-only)
See Data Sources.
2. Feature Catalog
YAML feature views resolve to Arrow-typed columns; a feature service is the column-level contract a model consumes. The service is what gets compiled. See Feature Catalog.
3. One pipeline — an operator DAG
A feature service compiles (daeda pipeline compile) into a single operator DAG. Each operator declares the engine it runs on. The vocabulary:
| Operator | Engine | Role |
|---|---|---|
Scan / Filter / Project | sql | read + project a source relation |
Join | sql | enrich the spine with static columns |
PointInTimeJoin | sql | strict-< as-of lookups |
RollingAggregate | sql | windowed point-in-time aggregations |
ArrowUdfTransform | sql_arrow_udf | a vectorized Arrow UDF over a SQL column |
RayUdfTransform | pythonic | distributed Ray/Lance embedding compute |
Sink | sql | write the result |
See Operator Pipeline and Training → Operators.
4. Per-operator optimization
Optimization is sub-operator level — each operator self-optimizes through its declared engine; there is no global query planner:
sql— composed into DuckDB SQL and pushed down to the source (the source optimizes its own scan/filter/join).sql_arrow_udf— SQL produces a column, then a vectorized Arrow UDF (e.g.avg_pool) runs over Arrow batches.pythonic— distributed Ray actors over Arrow batches and the Lance embedding store.
Across all of them, daedalus owns only three levers: lazy evaluation, work partitioning, batch streaming — Arrow-native throughout (see Compute Engine and Arrow-Native Design).
5. Platform surfaces
The executor is wrapped by surfaces that all reuse one in-process API — Dagster, a JSON-RPC control plane, an MCP server, the CLI, and an optional MLflow projection. See Platform Surfaces.
Design rule: no query optimizer
Sources (DuckLake / Snowflake / Postgres, and DuckDB over them) optimize the source-side query themselves. Daedalus deliberately has no LogicalPlan→PhysicalPlan planner and no cost model — it composes pushdown-friendly SQL for the source layer and owns only the post-source compute layer (Arrow UDF + Ray UDF), governed by lazy / partition / stream. This keeps the pipeline a thin, predictable, per-operator-optimized DAG.
Where to go next
- Feature Catalog — how views and services resolve.
- Operator Pipeline — the compiler and the DAG.
- Compute Engine — DuckDB & Polars rolling aggregation.
- Arrow-Native Design — the data-format contract.
- Training → Overview — running the pipeline.