Daedalus

Appearance

Execution & Tuning ​

daeda pipeline train <service> compiles a feature service and runs its operator DAG through the executor. This page covers the run flags, day partitioning and parallelism, the Ray envelope that bounds the pythonic enrichment operators, and the config files that drive a run.

bash
daeda pipeline train pixai_feed_relevance

For the one-pipeline mental model see Overview; for the operator-by-operator breakdown see Operators & Optimization.

daeda pipeline train flags ​

FlagDefaultDescription
SERVICE(required)Feature service name (e.g. pixai_feed_relevance).
--feature-views-dirfeature_viewsDirectory of feature view YAML.
--feature-services-dirfeature_servicesDirectory of feature service YAML.
--runtime-config-pathfeature_services/pixai_feed_relevance/settings.yamlTraining runtime config.
--aggregation-config-pathconfig/training/aggregations.yamlAggregation + embedding-store config.
--output-root(from config)Override the output root.
--target-date(all days)Narrow a run to a single YYYY-MM-DD day.
--day-workers(from config, 1)Concurrent target days.
--only source|enrich|sql|sql-arrow-udf|pythonic(full DAG)Run only the selected engine units.
--ray-num-cpus16Ray logical CPUs for the pythonic enrichment operators.

--only runs a sub-range of the DAG ​

--only does not select "stages" — it selects a contiguous sub-range of the one operator DAG by engine unit:

  • --only source runs only the source-side operators — every sql / sql_arrow_udf operator (Scan / Filter / Project / Join / PointInTimeJoin / RollingAggregate / Sink). It stops before the RayUdfTransform enrichment operators. (--only sql / --only sql-arrow-udf narrow further to a single engine.)
  • --only enrich (alias --only pythonic) runs only the enrichment operators — the pythonic RayUdfTransform(s) — over an already-written source-side output.

A run with no --only executes the full DAG: source-side operators first, then the enrichment operators (only if the pipeline declares a RayUdfTransform). Both sub-ranges are resumable, so a failed run is safe to re-run.

bash
# Full DAG, single day
daeda pipeline train pixai_feed_relevance --target-date 2026-05-15

# Source-side operators only, then enrichment operators only
daeda pipeline train pixai_feed_relevance --only source --target-date 2026-05-15
daeda pipeline train pixai_feed_relevance --only enrich --target-date 2026-05-15

# Override config + output locations, size the Ray envelope
daeda pipeline train pixai_feed_relevance \
    --runtime-config-path feature_services/pixai_feed_relevance/settings.yaml \
    --aggregation-config-path config/training/aggregations.yaml \
    --output-root /path/to/output \
    --ray-num-cpus 16

Day partitioning & parallelism ​

The pipeline iterates every day from feed_start through feed_end (inclusive), processing each as an independent unit and writing one dt=YYYY-MM-DD/part-N-0.parquet partition per day. --target-date narrows a run to a single day; the month range and chunk sizes are config-driven in feature_services/pixai_feed_relevance/settings.yaml.

--day-workers N (or day_workers in the service's settings.yaml) processes N target days concurrently. The default is 1; raise it only when the host has enough CPU, memory, and DuckDB spill bandwidth for N independent daily pipelines. The source-side operators are memory-bounded per day — each major operator is materialized as its own DuckDB COPY stage so memory is released between the spine, rolling, lookup, and assembly work.

The Ray envelope for pythonic operators ​

The RayUdfTransform enrichment operators run under the 16c / 64 GiBRayEnvelope. A singleton local Ray cluster is sized to that budget; the actor pool auto-sizes to ~80% of the worker-CPU budget so Ray Data read/write tasks aren't starved.

ResourceDefaultNotes
Ray logical CPUs (--ray-num-cpus)16Total CPUs exposed to the local Ray node.
Ray memory budget64 GiBTotal, including object store + system reservation.
Object store20 GiBRay object store size.
Ray system reservation1 CPU / 4 GiBReserved for Ray system processes.
Worker CPU budgetray-num-cpus − 1Actor pool auto-sizes to ~80% of this.

Key behaviors:

  • --preload-day-embeddings (default on) — the day's needed FP16 embedding subset is loaded once on the driver and shared with actors via the object store. A single day touches only ~470K distinct artworks, so this stays under 1 GiB.
  • Chronological sharded output — output is sharded parquet under dt=YYYY-MM-DD/ with Ray Data preserve_order on, so each shard is a contiguous chronological event_timestamp range (no serial single-file merge).
  • Day-atomic & resumable — days are processed sequentially for memory isolation; an already-enriched day is skipped, so a failed run resumes cleanly.

Read-only cgroup pods

Ray's enable_resource_isolation needs a writable cgroup v2. On read-only-cgroup devpods it is off by default — enforce the envelope with the external memory governor (misc/bench/memcap.py) instead.

The Lance embedding pool is cumulative (current pinned: data/pixai-relevance-v2-embeddings, ~33.0M rows / 72 GB), and image_embedding lives only on the enrichment side — it is never read by the source-side operators.

Config files ​

A run is driven by two layered config files (Pydantic Settings with YAML sources):

FileDrives
feature_services/pixai_feed_relevance/settings.yamlChunk sizes, feed_start / feed_end, feature_refs, day_workers, compute.engine (the rolling_engine for RollingAggregate).
config/training/aggregations.yamlaggregation_specs, enrichment_specs, embedding_stores, pit_lookups.

feature_refs selects which views the source-side Scan/Join operators load (all columns in each view's features are loaded, not just the referenced one). The output schema is resolved from the feature-service contract (feature_services/<service>/definition.yaml) by the training adapter — not listed in settings.yaml. Each contract column given as {name, default} is filled with the specified literal (e.g. NULL) when absent — so the schema is consistent even for a --only source run that hasn't appended embeddings yet.

Next ​