Before Explaining the Spike, I Had to Check the Question

The button told the parent to mark takeoff and landing. The algorithm needed a little more of the video than that.

In JumpOnion, the figure-skating app I build, that mismatch was enough to break the analysis. A parent could follow the interface exactly and remove the frames the measurement depended on. The system would still return a number. Working harder on the algorithm would not restore evidence the interface had told someone to cut away.

That experience helped shape unclosed, my entry in the OpenSearch Agent Skills Hackathon. It received the 2026 Grand Prize. The project starts with a question I had learned to ask while building products: what has to be true before this answer means anything?

The problem existed before the hackathon

My family and I use the products I build. IvyBloom helps organize my children's learning; JumpOnion grew out of our figure-skating needs. Using them means I encounter what a demonstration can leave out: the input a real person supplies, the report they act on, and the cost of getting either wrong.

I had seen another version in IvyBloom. A heuristic used repeated “not learned” responses to exclude concepts from an analysis. Some of those concepts were material the learner still needed to work on.

The report could improve by becoming less complete. The difficulty had not gone away; it had been removed from the report's field of view.

In one product, the input was missing something. In the other, a decision about what counted as evidence hid something important. Both experiences made me less interested in how convincing the final explanation sounded.

When I discussed the OpenSearch project with my AI collaborator, I challenged the starting point. If someone says, “Latency spiked; find the cause,” an agent can accept the spike as an established fact and spend the rest of the investigation explaining it. It may do excellent work on a question nobody checked.

I wanted the investigation to show its unfinished work as well: the alternatives it checked, the ones it ruled out, and the ones it never reached.

Two alarms can look alike

The project uses controlled request-log fixtures. Some contain a genuine regression. Others make a comparison look alarming because the measurement is not sufficient to support it.

A very high tail-latency reading from a few requests and a high reading from a substantial sample can both look impressive on a screen. The numbers alone do not tell the investigator whether the comparison is sound.

So unclosed checks the premise before building an explanation on it. It examines the measurement, the sample, the window, and the population being compared. A rejected comparison does not declare the service healthy. It says that this particular claim needs better support.

The distinction matters. A slow request may still deserve investigation even when it cannot establish the original claim about the whole window.

The slowest endpoint might have been slow all along

One of the most useful corrections involved the baseline.

A checkout request that writes to a database will often take longer than a health check. Comparing their raw times can keep selecting checkout as suspicious, including on an ordinary day.

Here is an illustration, not a result from the project:

Checkout is still the slower group. The health check has moved much further from its usual level. Neither row, by itself, tells us whether that movement exceeds normal variation.

The probe therefore starts by subtracting each group's own historical median, measured outside the focus window. It asks how that group moved from where it normally sits.

Then it needs to ask how surprising the movement is. Different groups have different amounts of ordinary variation. Mixing them into one pool would give each group a borrowed idea of normal.

The reference calculation redraws samples from each group's own data. It also accounts for the fact that the probe chooses the largest movement from several candidates. The biggest value in a chart should not get the benefit of pretending it was the only value we checked.

This was a correction earned through testing. The recorded evaluation caught the earlier probe treating a standing difference as a new change. The published implementation also names groups that lack enough history instead of borrowing a baseline for them.

There is a separate question I would still ask an operator: does this meet your users' requirements? A service can be consistently slow and still be unacceptable. Its own history helps investigate change; it does not set the standard of service the business should accept.

Leave the unfinished branches visible

Once the observation survives the checks, unclosed builds a record of the investigation within a declared set of hypotheses.

It keeps queries, evidence, and branch states together. An unvisited branch remains unvisited. A suggestive explanation without a measured contribution remains incomplete. Before the record can close, its measured explanations have to account for the effect within the project's stated tolerance.

That is a limited meaning of closure. Logs and correlations do not establish causation. The tool does not turn a plausible deployment story into proof that the deployment caused the incident.

For me, the useful output is the record another person can inspect and continue. “Here is what we checked, and here is what is still missing” is a better handoff than a polished paragraph that hides both.

The comparison did not prove what I hoped

The repository includes an agent comparison with and without the skill. On the part that could be scored, both arms answered the premise question correctly in all six runs.

That does not demonstrate an accuracy advantage for unclosed. Another case exposed a problem with the evaluation contract itself: it combined a real slowdown with an overstated early report, so a binary label could reward either interpretation. It was excluded from the scored comparison rather than used to improve the headline.

The fixtures are synthetic. Captures run against a real OpenSearch instance are still controlled experiments, not customer incidents. The project also publishes detection limits and false-confirmation risks. I cannot turn those tests into a claim that it reduces production incident time.

Winning the Grand Prize gives me a reason to put the work in front of more practitioners. The next useful evidence would come from a bounded evaluation of whether the investigation record helps people identify gaps and continue each other's work.

After OpenSearchCon · September 26, 2026

I use AI to help implement the systems I design. My responsibility is to define the question, decide what evidence is enough for a claim, and stay involved when a clean result does not match the problem.

I bring the same questions to a popular open-source project. What makes its idea work? Which part solves a problem I actually have? Can I use it directly, does it need adaptation, or does it show that my existing architecture should change? The original idea deserves credit; my choice still needs evidence from the task it is supposed to improve.

IvyBloom and JumpOnion gave me repeated practice at that responsibility. They are the origin of this project, not claimed deployments of it.

The OpenSearchCon talk is now complete. The question behind it remains: what can this investigation support, and what evidence is still missing? The public project and recorded evaluations below let readers inspect that question beyond the talk. The presentation is not evidence of a production deployment or an accuracy advantage.

The answer I want to be able to explain is simple: here is what this investigation can support, and here is what you would need before saying more.

Read the project and its recorded evaluations, including the cases that did not support an advantage. For the product experience behind it, read the JumpOnion button-label incident.