Overview

A training dataset in Daedalus is produced by one pipeline — a single DAG of engine-tagged operators compiled from a feature service. There is no "skeleton stage" and no "enrich stage": the source-side feature joins, the point-in-time rolling aggregations, and the embedding enrichment are all just operators in that one DAG, each tagged with the engine it runs on.

daeda pipeline train pixai_feed_relevance

This is the sole training entry point (Core v1). It compiles the feature service feature_services/pixai_feed_relevance/definition.yaml into an operator DAG and runs that DAG straight through the executor.

Single entry point

The standalone daeda skeleton, daeda enrich, and daeda enrich-shards commands were removed at v0.7.1 (the Core-v1 lean cut). daeda pipeline train runs the same proven engine entry points the old commands wrapped, so its output is byte-identical to the old dual path.

One pipeline, not stages

A feature service is a declarative column-level contract. daeda pipeline compile maps that contract onto the operator vocabulary and emits one topologically ordered DAG:

events Scan
  → per-referenced-view (Scan + Join)   static feature columns
  → RollingAggregate                    windowed point-in-time rollups
  → PointInTimeJoin lookups             cumulative dislike / recent-published
  → Project assembly                    curation, age cols, output ordering
  → RayUdfTransform enrichment          avg-pooled artwork embeddings
  → Sink                                day-partitioned parquet

Each operator carries a declared engine tag — sql, sql_arrow_udf, or pythonic — and self-optimizes at its own level. There is no global query planner: sql operators push down into the source, sql_arrow_udf runs a vectorized Arrow UDF, and pythonic runs distributed Ray actors over the Lance embedding store. See Operators & Optimization for the full operator-by-operator walkthrough and Operator Pipeline for the compiler/executor architecture.

Contract-driven

The DAG is derived from the service's declared columns: a PointInTimeJoin is emitted only for declared lookback columns, and a RayUdfTransform only for declared embedding columns. A service with neither gets no PIT-join and no Ray UDF operator at all.

Compile: emit an editable DAG

Inspect or hand-tune the compiled DAG before running it. compile writes <output-dir>/<service>.generated.yaml (default output dir config/pipelines/) and echoes the YAML to stdout:

daeda pipeline compile pixai_feed_relevance            # write + echo the generated DAG
daeda pipeline compile pixai_feed_relevance --no-write # echo only, write nothing

Engine tags in the emitted YAML are defaults the author can override. To pin an operator to a different engine, hand-edit <output-dir>/<service>.yaml; that file takes precedence over the regenerated <service>.generated.yaml when present.

Train: run the pipeline

# Full configured month range
daeda pipeline train pixai_feed_relevance

# A single target day
daeda pipeline train pixai_feed_relevance --target-date 2026-05-15

# Only the source-side operators / only the enrichment operators (resumable)
daeda pipeline train pixai_feed_relevance --only source --target-date 2026-05-15
daeda pipeline train pixai_feed_relevance --only enrich --target-date 2026-05-15

# Concurrent target days (needs CPU / memory / DuckDB-spill headroom)
daeda pipeline train pixai_feed_relevance --day-workers 4

See Execution & Tuning for every flag, the day-partition model, and the Ray envelope that bounds the pythonic enrichment operators.

Output layout

The pipeline writes under output_root (default data/training_output/, or data/mewtant/training_output/ per the documented production default):

Path	Produced by	Description
dt=YYYY-MM-DD/part-N-0.parquet	source-side operators (Sink)	Per-day Zstd parquet partitions, sorted by event_timestamp
<root>_enriched/dt=YYYY-MM-DD/*.parquet	enrichment operator (RayUdfTransform)	Sharded, chronologically contiguous parquet with the embedding columns (image_embedding, like_artwork_avg_embeds) appended

The enrichment operator skips days already present in its output directory, so a failed run is safe to re-run (day-atomic, resumable).

Next

• Operators & Optimization — the operator vocabulary and the end-to-end DAG walkthrough, with per-operator (sub-operator) optimization.
• Execution & Tuning — running the pipeline: flags, day parallelism, the Ray envelope, and the config files.
• Operator Pipeline — the compiler and the executor that turn the contract into a runnable DAG.