Why I stopped guessing at Spark and dbt config values

I've spent more than a decade building data pipelines, and the part nobody warns you about isn't the pipeline logic. It's the tuning. Executor memory, shuffle partitions, cluster size, thread counts. You pick numbers, ship it, and a few weeks later something breaks in a way that's obviously tuning-related but not obviously what to change. The pattern repeats enough times that you start recognizing it before you've even opened the logs. Job's slow, thousands of tiny shuffle tasks, someone way overestimated the partition count. Job dies on OOM, memory's set for last quarter's data volume, nobody updated it since. Cloud bill jumps, a cluster's been sized for peak load and just sits there mostly idle the other 20 hours a day. Every senior data engineer has this pattern-matching running in their head. It's tribal knowledge, and it lives in one or two people's heads on most teams, which means it doesn't scale and it definitely doesn't survive someone leaving. So I built a small tool to make that pattern-matching explicit instead of tribal: it reads your pipeline's config alongside its actual run metrics and tells you what's likely wrong, with the reasoning shown, not just a suggested number. Why rules instead of a model The obvious move in 2026 is to reach for an ML model. I didn't, and it wasn't because I don't think ML has a place here eventually. It's that for this specific problem, a handful of threshold rules already gets you most of the value, and they're something you can actually audit. If a rule fires, I can point at the exact condition and the exact number: average heap usage 28%, peak 47%, five runs, no OOM errors, therefore memory's over-provisioned, shrink it by roughly a fifth. That's checkable. You can look at your own metrics and see whether the reasoning holds. A model's confidence score doesn't give you that, and for something that's about to change a production config, I want the person approving it to be able to say "yes, I see why" rather than "the model said so." There are seven rules right now: memory over/under-provisioned, shuffle partition sizing, cluster instance count over/under-provisioned, and two for dbt (thread count and tasks-per-thread). Each one is a pure function: current config plus recent run metrics in, a recommendation or nothing out. No side effects, no state, fully unit tested. Nothing applies itself This was a deliberate line, not a limitation I plan to remove later. Every recommendation requires a human click before anything changes. I don't trust an agent to touch a production Spark config unattended, and I don't think you should either, no matter how good the underlying reasoning is. The tool's job is to surface the diff and the reasoning behind it. Approving it is still yours. What's actually real today, and what isn't dbt is the one framework with a genuine integration. Upload your actual target/run_results.json and you get recommendations computed from your real run history, not a fixture. Spark's rules exist and are fully tested, but there's no live reader yet, no Spark REST API or event-log ingestion, so it works on sample data today rather than your own cluster. I'd rather say that plainly than let the pitch imply more than the tool does. Try it There's a live dashboard with three real sample pipelines you can click into right now, no signup, no upload required to look around: opti-pipe.onrender.com. If you run dbt, you can upload your own run_results.json and get recommendations against your actual data. I'd genuinely like to hear from anyone who tries it against a real pipeline, especially if a recommendation is wrong. That's the fastest way I'll find the next rule worth adding.