A few minutes before the Hackathon Winners Showcase, I was talking with Lisa Briggs and Bobby Mohammed, who were about to introduce the three of us winners on stage. They asked how I’d found the OpenSearch Agent Skills Hackathon. I gave them the honest answer: my agents told me this competition was right for me.
I’m Ciki Zeng, an independent builder and a parent. I design software with AI agents. I decide what a system should do, where it’s allowed to be wrong, and how I’ll know when it is. The agents write much of the code. I check the result.
How my agent found it
One of the agents I run reads hackathon listings every morning and compares them with what I actually build. On July 29 it put this hackathon near the top. Its reasoning was simple: the challenge was to write agent skills, and writing skills for my own agents is something I already do every day. The next morning it ranked the hackathon first.
The morning after that, the same report didn’t mention it at all. Nothing had errored. The recommendation was simply gone, and a recommendation that quietly disappears looks exactly like “nothing changed.” I fixed the routine so that anything it has already flagged must show up again or be marked as not rechecked. That night I registered.
I didn’t know yet that this would turn out to be the whole theme of my hackathon.
A perfect score on an exam with no questions
The hackathon offered example problems like “why did latency spike at 2 a.m.?” My strength is asking the question behind the question. After building through a lot of real problems in my own products, my first reaction was to question this one.
The spike is being treated as a fact. The agent is asked to explain a result everyone assumes is real. But that result might be like a perfect score on an exam that never had any questions on it. An agent will usually start from the angle you give it and work downward, and it may stop halfway once it finds an answer that sounds right.
My second thought was that a complete chain of cause and effect is not always findable. A chain of reasoning is. Think of it like a tree: at every branch, record which path you took, what it returned, whether it confirmed anything, and what you haven’t checked yet. Then anyone can see whether the investigation really closed.
That became unclosed, an agent skill that reads OpenSearch logs and checks whether an investigation’s conclusion has enough evidence behind it. It runs three checks before it lets a conclusion close.
Check the premise
Is the reported anomaly real, or could the time window or the measurement be misleading us?
Track the alternatives
Which explanations have evidence, which are ruled out, and which were never checked?
Check the numbers
Does the explanation account for the size of the change we actually saw?
The report names the evidence it needs next. The case stays open.
Closing means the scoped checks were met. It does not mean the production problem is fixed, and it does not prove causation.
What building it taught me
The most useful lessons came from the places where I was wrong.
The same window can be two different datasets.An alert looks at a time window while it is still filling up. An audit looks at the same window later, when it’s complete. In one test case, the alert saw 40 requests, the audit saw 200, and the verdict flipped. A time field always has a value, and the value always looks reasonable, so you have to ask which moment it describes before you trust it.
The slowest thing is not always the thing that changed. A checkout request that writes to a database is slower than a health check on an ordinary day. An early version of the skill kept treating that standing difference as a new problem. Testing caught it, and now each group is compared with its own history and its own normal variation.
My evaluation needed checking too.I compared an agent with and without the skill. On the questions that could be scored, both answered correctly in all six runs, so I can’t claim an accuracy advantage. One test case turned out to be ambiguous: it could reward either answer. I excluded it from the score instead of using it to improve the headline, and I published the comparison along with its limits.
Tests should fail when the rules break. I deliberately broke the rules the skill depends on, one at a time, to see whether the tests would catch each break. A green test suite only means something if it can turn red.
unclosed won the Grand Prize. That is how I ended up in San Jose for my first OpenSearchCon, September 22–24.
Day one: a room full of people who run things at scale
My first morning started with breakfast with Ritz Hemnani and Abdulkadir Dalga, who were presenting Booking.com’s migration to OpenSearch later in the conference. Then the CERN keynote put the scale of the room in perspective: 139 clusters, 4.3 trillion documents, and 1.4 PB of data.
It was a room full of people who run search and observability at enormous scale. I brought a different kind of experience: building for real families, where a wrong conclusion lands on someone I know. That turned out to be a useful lens in almost every conversation I had.
The session that mattered most to me was “From Alert to Answer: Accelerating Anomaly Investigation with OpenSearch Agent Skills” by Brian Graf and Seema Saharan. It was about helping agents get from an alert to an answer faster, and it was candid about failure modes and what changes in production. That is very close to the problem unclosed works on. unclosed asks what that answer is actually standing on. I started to see it less as a standalone skill and more as a possible trust layer: a check that runs before an answer is handed to a person.
Afterward I talked with Brian about whether unclosed could plug into an investigation pipeline like theirs.
Day two: ten minutes on stage
At the Winners Showcase, I didn’t start with logs. I started with two products I build, dogfood myself, and test with real users in the market.
In IvyBloom, a learning system my children use, two cards showed the same title, both marked complete, both four out of four. It’s easy to read that as the same learning result. The records said otherwise: one skill was Grade 8 and the other was Grade 5. In another case, an audit had attached a learner’s latest score to an earlier attempt, so a 2/4 looked like a 4/4. Nothing crashed. The data was all there. The way we joined it changed the story.
In JumpOnion, my figure-skating jump analysis tool, I’ve watched AI confidently find fault with a world champion’s triple Axel. Drawing pose points on a skater is not a diagnosis.
I also added a few things that weren’t on my slides: that the system I build with existed before this project, that every case study in it is a mistake I actually made, and that I care about the quality of what my agents produce because the only way to connect them to real business is trust.
The line I wanted to leave with the room was this: before AI explains why, it should show whether what happened is real, and what evidence is still missing.
After my talk, I went straight to the unconference Kris Freedain was hosting. I got to hear other people’s fantastic ideas and the projects they were building. That was the part of the conference that felt most like a community.
Day three: “rarely see anyone use AI this way”
At breakfast on the last morning, two engineers told me they rarely see anyone use AI the way I do. I’ve thought about that sentence a lot since. I don’t use AI as a search box. I’ve spent this year building something closer to a partner: a system that remembers what I’ve done, with evidence, and that I’ve learned to govern instead of just trust. I wrote about that on September 30:
I asked for “another me.” Here’s what it took to trust it. →
The same habit in other rooms
What surprised me most was how many speakers arrived at the same habit from completely different directions.
The talk that stayed with me most was Booking.com’s migration of its autocomplete search, partly because I’d had breakfast with two of its speakers on the first morning. Before moving users over, they listed every feature the old system had and decided which ones mattered. They mirrored real queries to the new system as shadow traffic, with no impact on users, and compared its results against the old one. Every daily index had to pass a validation check before it could go live. Then they moved traffic in slices. One line from their slides stuck with me: “Optimize for non-inferior business metrics, not 100% technical parity.”
I had done a much smaller version of this in JumpOnion. When I changed the pose model, I was genuinely afraid of changing the data and results for the families already using it. So the first thing I did was freeze the existing outputs as a reference. Then I ran the old and new versions on the same set of real skating videos and compared them before letting the new one in. Even then, the new signal could only show observations until it was strong enough to support a diagnosis. Booking.com works at a scale far larger than mine, and that is exactly why it’s worth learning from early. Every time I change my architecture, it’s because real families rely on the result. It has to be correct, and it has to be restrained about what it claims.
“Your Agent Is a Distributed System,” by Harishankar Menderkar and Sarat Chandra Ventrapragada, made the same point from the agent side. In their reproducible demo, an agent wrote “remediation complete” on every one of 500 incidents. When they asked the platform itself, 104 of those incidents had never stopped erroring. An agent’s report of success is a claim. The independent evidence is something else.
500 / 500
“remediation complete”
104
never stopped erroring
Different teams, different stacks, one shared habit: treat “done” as a claim to verify, not a result to believe.
What I’m taking home
Before the conference, I thought this problem was narrow, something I cared about because I build for two very demanding users at home. Now I think it’s one of the most common questions in the building.
I’ve also started using OpenSearch for my own work. On the last day of the conference, I set up a local OpenSearch instance to search my own build history: the conversations and notes behind everything I’ve made this year. While I was waiting for my ride that afternoon, I realized why that matters to me: I don’t want the road I’ve walked to be buried. Now I can find it again.
That is also the direction of my next work: combining OpenSearch with long-term memory for agents, so that what an agent remembers can be searched, traced back to its evidence, and trusted.
I want to keep contributing. The trust-layer idea is where I’d like to go next, and I’d love to explore it with people who work on investigation workflows every day. The code for unclosed is public at github.com/simpleciki/unclosed.
Thank you to Lisa Briggs, Bobby Mohammed, and the Linux Foundation events team for making the Winners Showcase run so smoothly, to Kris for pulling me into the conversation, and to the OpenSearch Project for the hackathon that brought me here. If your team has seen a convincing report with missing evidence behind it, I’d love to hear about it.