My Science Questions Passed Every Check. They Mostly Tested Subtraction.

The question needed the data table. That was what we had asked for.

Subtract the first row from the fourth row, and you had the answer. You could do it without knowing the science idea the lesson was meant to teach.

The question passed the criterion. The criterion had lost the point of the lesson.

This happened while I was expanding science material in IvyBloom, the learning app I build for my children. The workflow had checks for the structure of each card and statistics about the questions. The batch looked acceptable through those checks. A separate review found that much of the work was table reading and arithmetic with very little need for the unit's science concepts.

How a sensible requirement drifted

An earlier problem was that questions did not really depend on the data in front of the learner. A table could be present as decoration while the answer came from somewhere else.

So the next criterion sounded reasonable: most questions should require the table.

My AI collaborator proposed it, and the generated questions met it. The missing test was whether needing the table also meant needing the science.

Those are different requirements. Reading a value is one task. Explaining what the value means using a scientific idea is another. Depending on the lesson, both can be useful. But a page full of the first cannot be counted as evidence that we delivered the second.

The mistake became clearer when stated that way. We had replaced a difficult educational judgment with a condition that was easier to check.

Someone had to do the task

I asked for an independent review of the learning task itself. An AI collaborator found that many questions could be answered as arithmetic even though the cards had cleared our existing checks.

The review also found a proposed correct answer that expressed the misconception the unit was meant to correct. A complete-looking card could still teach the wrong thing.

I did not discover each of those defects personally. I set the requirement for that review and remained responsible for what could enter the learning workflow. My AI collaborators authored, checked, and corrected the material. Naming those jobs accurately is part of explaining how the product gets built.

The green checks were answering smaller questions

Formatting and completeness checks still matter. But they cannot tell me whether a child needs the science idea or whether an explanation is right.

Each check can be doing its job while the overall task remains unfinished.

What changed after the review

For the science cards, I changed the acceptance question: a correct response should depend on the science idea being taught, not only the numbers shown beside it.

We held the affected batches for correction. The next material improved under review but still had issues; revising a criterion was not itself proof that the content was ready.

I also had to keep the role of these cards clear. Teaching material with a few checks for understanding should not quietly become a claim that the system has measured a child's mastery. The amount and quality of evidence needed for that claim are different.

Improving quality has a delivery cost too

A later review found that raising the quality bar could make an existing scheduled lesson unavailable when a learner opened it.

My requirement was simple: improving the material must not silently take away the learning already scheduled for the child.

The proposed change was revised to preserve access while the quality issue remained in view. A local check opened the next lesson; the issue was caught before a child encountered that failure.

For me, the decision was about what the child would receive while the material was corrected. A stricter rule is only an improvement if it does not quietly remove a lesson.

The question I now want answered

The experience changed the question I ask of a green result: what did it actually establish for the learner?

Here the checks established that the task used the table. They had not established that it used the science. I held the affected content rather than treating the green result as proof of educational quality.

That decision does not demonstrate improved learning outcomes. It describes a review that found a mismatch and a limit on what I was willing to present as ready.

My children do not experience a validation pass. They experience the question on the screen. That is where I need the lesson to make sense.

See the IvyBloom project and the case-study library for other bounded records of what changed.