Daedalus

Appearance

Local Lance embedding store ​

Daedalus reads every other source in place — parquet on S3, DuckLake tables, Postgres — with no copy or download step. The embedding pool is the one exception, and it is a deliberate one:

  • LanceEmbeddingStore is local-path only. It does not read s3://.
  • Lance's commit path relies on atomic rename, which GPFS / NFS do not provide. A store built on a network mount appears to work, then corrupts under concurrent Ray actors.

So the enrichment unit needs a Lance store on a local filesystem (ext4 / xfs / overlay) before it can run. The source-side unit does not — image_embedding lives only in the enrichment operators, so --only source never touches the pool. If your host lacks the disk, run source there and enrich where the pool is.

Size the disk first

The pinned production pool (data/pixai-relevance-v2-embeddings, the fp16 2026-01-01..2026-07-17 window, ~33.0M rows) is 72 GB on disk, built from ~154 GB of raw hive-partitioned vectors. Budget for both if you download the full source, plus headroom — the builder refuses to start below 85 GiB free.

Step 1 — download the hive-partitioned vectors ​

The raw vectors live under an S3 prefix, hive-partitioned year=YYYY/month=M/day=D (month and day are unpadded integers — month=1, not month=01).

bash
# Credentials: MEWTANT_AWS_* from your .env — see .env.example
export AWS_ACCESS_KEY_ID="$MEWTANT_AWS_ACCESS_KEY_ID" \
       AWS_SECRET_ACCESS_KEY="$MEWTANT_AWS_SECRET_ACCESS_KEY" \
       AWS_REGION="$MEWTANT_REGION"

# s5cmd is much faster than aws-cli here (parallel by default).
s5cmd cp 's3://pixai-rec-sys/infra/siglip2_vectors/year=2026/*' \
         /data/<you>/infra/siglip2_vectors/year=2026/

The download target may live on GPFS — only the Lance store itself must be local. Downloading a single year is usually enough; scope it to match the window your dataset covers.

To check what you are about to pull before pulling it:

bash
s5cmd ls 's3://pixai-rec-sys/infra/siglip2_vectors/'
uv run python misc/embedding-kv-store-benchmark/inspect_hive_parquet.py \
  /data/<you>/infra/siglip2_vectors/year=2026     # metadata only, decodes nothing

Step 2 — build the store ​

The store is declared, not passed in. embedding_stores in config/training/aggregations.yaml carries the path, id / embedding columns, dimension, dtype, batch size, and the source_filter that scopes the pool — so a rebuild reproduces the same pool rather than whatever flags were typed last time.

bash
# What is declared, and what is already built?
uv run daeda store list

# Build it from the parquet you just downloaded.
uv run daeda store build artwork_embeddings \
  --source /data/<you>/infra/siglip2_vectors \
  --duckdb-threads 16 --duckdb-memory-limit 32GiB

--source overrides the declared source_parquet for a host whose layout differs; omit it to use the declaration as-is. Other useful flags:

FlagWhy
--source-filter "year = 2026 AND (month*100+day) BETWEEN 101 AND 717"Override the declared window. Pass '' to build every partition.
--store-path <dir>Build somewhere other than the declared path.
--rebuildReplace an existing store. Without it, an existing store is an error, not a silent no-op.
--allow-network-filesystemSkip the filesystem check. Only if you know the path is local despite its fstype.

The build streams id → FixedSizeList<float16>[1152] through DuckDB in batch_size chunks and writes a checksummed ID-index sidecar plus manifest.json, so later opens are O(1) instead of re-scanning IDs.

A single day builds in seconds, which makes it a cheap way to prove the whole path works before committing to the full pool:

bash
uv run daeda store build artwork_embeddings \
  --source /data/<you>/infra/siglip2_vectors \
  --source-filter "year = 2026 AND month = 1 AND day = 1" \
  --store-path /tmp/lance-smoke
# built 'artwork_embeddings': 166462 rows, dim 1152, 0.36 GiB

Step 3 — run the pipeline ​

With the store in place, the enrichment unit resolves it by name from the same declaration:

bash
uv run daeda pipeline train pixai_feed_relevance --only source --target-date 2026-07-15
daeda pipeline train pixai_feed_relevance --only enrich --target-date 2026-07-15

Do not run the enrich unit under uv run

Ray's raylet re-execs the launcher to spawn its workers, from a working directory with no .env — the workers then crash. uv run can also reinstall .venv mid-flight and break a running job. Activate the venv and call daeda directly for enrichment.

If you only need to exercise the code path ​

You do not need production embeddings to test enrichment. A deterministic synthetic store is enough:

bash
uv run python misc/bench/seed_synthetic_lance.py --help