A CPU profile found the cause, not the timing
An internal analytics dashboard froze for up to two minutes and restarted. Within a day the freeze dropped from 83 to 28 seconds, and the follow-up plan removes it.
Context. The dashboard the business uses for daily decisions computes metrics over ~250k records across 24 months. One day it started freezing for a minute or two and restarting. The team spent a full day on a tool that kept falling over.
What went wrong in the response. The first suspect was a feature merged minutes before the first errors. It was reverted and called the cause without being reproduced. A second AI session "confirmed" it from the same logs. The freezes continued after the revert.
What I did.
- Took a CPU profile in production. 73 of 100 seconds went to work repeated for each of 24 months that did not depend on the month. Garbage collection took another 37.
- Fixed the core issue the same day: that work now runs once per build. 83 → 28 seconds, and the container stays up.
- Proposed two structural fixes: move the computation to its own worker with its own cache, and store closed months in the database with a formula version. The second removes ~90% of the work on every build.
- Added process guards: a merge freeze during incidents, a performance test at production data volume, and alerts on long stalls.
Takeaway. Two AI agents reading the same logs are one piece of evidence, not two. The profile should be the first step, not the last.