Spark + Ray on EKS
Two engines, the same 858M rows, built from scratch
Wanted to see what the engines underneath actually look like without any of the platform automation I'd been working on top of, so I stood both of them up myself from vanilla open-source packages.
Wanted to see what the engines underneath actually look like without any of the platform automation I'd been working on top of, so I stood both of them up myself from vanilla open-source packages and went down the question of when you'd reach for one over the other. Built Spark and Ray on a self-managed Kubernetes cluster from scratch, including the S3, networking, auth, container, and observability setup around them. From there I wrote equivalent ML pipelines in Spark MLlib and Ray + XGBoost and benchmarked them across 858M rows, comparing runtime, memory, and model performance. Most of the project turned into understanding what each framework quietly does for you: Spark was much more forgiving once a workload outgrew memory, while scaling Ray meant going deeper into distributed XGBoost, streaming data, and managing memory myself.
Kubernetes · Spark · Ray · XGBoost · AWS · Docker · Prometheus · Grafana

