Eventual-Inc/Daft

Daft

Performance History

Latest Results

docs: fix remaining Slack links missed in first pass Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

everettVT/slack-link-audit

4 minutes ago

refactor: extract partition_ref_id bit ops into named helpers Per review: input_id_from_ref / partition_idx_from_ref instead of inline shifts and masks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

colin/flight-shuffle-coordinator-memory

34 minutes ago

feat(checkpoint): distributed observability counters (#7026) ## What Surfaces checkpoint progress on the dashboard for distributed (Flotilla) checkpoint writes, via three run-total counters: - **`keys_staged`** — on the `StageCheckpointKeys` source operator (source keys recorded for dedup) - **`files_staged`** — on the write sink (staged file-metadata blobs) - **`checkpoints_sealed`** — on the write sink (successful seals) ## Why A checkpointed write previously gave no observability — there was no way to see from the dashboard whether/how much a run staged or sealed. These counters make checkpoint activity visible alongside the existing `rows.written` / throughput stats. ## How The counters ride the existing `StatSnapshot` aggregation path rather than introducing a new mechanism: - Each worker's `RuntimeStats` records the counter and includes it in its `StatSnapshot` (`StageCheckpointKeysSnapshot`, and two new fields on `WriteSnapshot`). - The distributed pipeline node's `handle_worker_node_stats` sums the snapshot fields into driver-side meter counters and re-emits them via `export_snapshot`. - Non-checkpoint operators report zero via default no-op `RuntimeStats` trait methods, so nothing changes for ordinary writes. The source op also now records `rows.in/out` (previously it ignored its stats handle, so the node showed empty throughput); row tallies are owned by the intermediate-op runner to avoid double-counting. ## Testing Verified end-to-end on a local Flotilla run (Ray runner, mock S3): a paired `read_parquet(checkpoint=...)` → `write_deltalake(checkpoint=...)` job shows `keys_staged=200`, `files_staged=1`, `checkpoints_sealed=1` on the dashboard, cross-checked against the staged objects in the store. A recovery rerun (same key) correctly stages zero. ## Scope Counters only. Driver-side lifecycle *events* (commit / recovery) and the dashboard-server UI for them are intentionally left as follow-ups. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

main

44 minutes ago

feat(checkpoint): add distributed observability counters Surface checkpoint progress on the dashboard for distributed (Flotilla) runs via worker->driver counter aggregation: - keys_staged on the StageCheckpointKeys source operator - files_staged and checkpoints_sealed on the write sink Each worker's RuntimeStats builds a StatSnapshot; the distributed pipeline node's handle_worker_node_stats sums the new fields into driver-side meter counters and re-exports them. Non-checkpoint operators report zero via default no-op RuntimeStats methods. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rohit/feature/checkpoint-metrics

2 hours ago

refactor(checkpoint): bincode-encoded versioned FileMetadata blobs (#7001) ## Summary - Replaces the hand-rolled length-prefix encoding (`[count][fmt_len][fmt_json][data_len][data]*`) for `FileMetadata` batches with `bincode` over a versioned tagged enum: `FileMetadataBatch::V1 { files: Vec<FileMetadata> }`. - Future schema evolution: add a `V2(...)` variant to the envelope; do not modify `V1`. Old readers fail loudly on unknown variants (correct default for forward-compat). - Net `-26` lines; encode/decode reduce from ~70 to ~20 lines. ## Test plan - [ ] Existing test suite green (22 unit + 17 integration tests in `daft-checkpoint`). - [ ] New encoding tests: round-trip, unknown-variant hard-error (simulates a future writer), truncated input, random garbage. - [ ] `cargo clippy -p daft-checkpoint --all-features --tests` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

main

2 hours ago

fixes

chris/query-runner-metadata

2 hours ago

ci: rerun PR checks

jackylee-ch:codex-feat-iceberg-branch-tag-reads

4 hours ago

fix(mcap): support http urls through fsspec handler

RitwijParmar:codex/daft-http-mcap-fsspec-handler

4 hours ago

Latest Branches

docs: standardize Slack links to use daft.ai/slack#7066

25 minutes ago

70e5f6b

everettVT/slack-link-audit

fix(flight-shuffle): reduce coordinator memory to O(map_tasks + partitions)#7056

1 hour ago

3cb4e55

colin/flight-shuffle-coordinator-memory

feat(checkpoint): distributed observability counters#7026

5 days ago

06176aa

rohit/feature/checkpoint-metrics

Home Terms Privacy Docs