Distributed ingest pipeline re-architecture

The problem

A critical ingest path written for single-node processing was running out of room: memory pressure as the source grew, and throughput that capped everything downstream.

Approach

Rewrite the pipeline as native distributed operations rather than wrapping the old code, and put production services around it for the read path and the asynchronous workload.

Outcome

Ingest that scales with the data instead of fighting it.