Pandas → Spark-native pipeline re-architecture
The problem
A critical ingestion pipeline had been written in pandas and was hitting a hard wall: vertical scaling on a single executor, memory pressure as the source data grew, and an operational model that required ever-larger boxes to keep up. Throughput had become a constraint on downstream analytics, and the on-call team was tired of OOM incidents.
Approach
Spark-native rewrite, not a wrapper
The wrong way to do this is to wrap pandas code in spark.createDataFrame and hope. I rewrote the pipeline as Spark-native operations: native joins instead of merge loops, window functions instead of group-apply, broadcast hints where cardinality justified them, predicate pushdown into the storage layer, and skew-aware partitioning on the join keys that mattered.
MongoDB / Azure environment
The pipeline ran across a MongoDB source-of-record into an Azure-based analytics tier. That meant tuning the MongoDB connector for partitioned reads, sizing executors for the Azure VM family in use, and being careful about the boundary where Spark output landed back into transactional stores.
Production microservices around the pipeline
Alongside the core pipeline I built and maintained the surrounding production microservices:
- FastAPI endpoints for the synchronous read-path — typed, validated, OpenAPI-documented, instrumented for latency and error budgets.
- Celery task orchestration for the asynchronous workload — fan-out jobs, retries, dead-letter queues, idempotent task design so retries didn't double-write.
- Python/PySpark services with the usual production hygiene: structured logging, health checks, graceful shutdown, dependency pinning.
Outcome
Significant throughput gains and — more importantly — horizontal scalability: the pipeline now grows with the data instead of fighting it. The OOM class of incident went away, and downstream analytics were no longer rate-limited by ingest.
Stack
PySpark · MongoDB Spark connector · Azure · FastAPI · Celery · Redis (broker) · Pytest · structured logging.