Our Model Learned Its Teachers’ Mistake, and Our Ruler Couldn’t See It

We are building a system that translates manga and webtoon pages automatically. The first link in the chain looks simple: find the text on the page. But “text” is not one thing. There is the dialogue inside a speech bubble. There is text outside the bubbles, like a narration box or a character’s inner voice. And there are sound effects that are part of the drawing: “BOOM”, “KRAAK”, the giant letters drawn as a sword swings.

The distinction is not cosmetic. Bubble text is erased and replaced with the translation. A sound effect is the artwork itself; touching it means damaging the art. So it is not enough for the model to say “there is text here”. It also has to say correctly what kind of text it is.

This post is about a mistake we could not see for months while trying to get that right.

The symptom: translations stamped on the artwork

We trained our own detection model. It scored well on the test set. We shipped it. Then we started seeing strange things on translated pages:

• Some sound effects had a translation printed on top of them, while the effect itself was still in place.
• In some places the same text had been typeset twice, one on top of the other.

Our first reflex was “the model is weak, it needs more data and longer training”. But the test scores did not say that. The model did well on the test. The problem was somewhere else.

The discovery: two boxes on the same pixels, two different labels

When we opened the production outputs one by one, we saw the pattern. The model was drawing two boxes around the same text. The boxes overlapped almost exactly, but one was labelled “sound effect” and the other “text outside a bubble”.

What followed was a chain accident:

• The “sound effect” copy followed its rule and left the art alone.
• The “text” copy was translated and printed on top of the same spot.

Our “never touch a sound effect” rule had been breached. A single object was processed twice.

We counted how common it was: across 379 production pages we found 1,064 twin boxes. 73% of the pages had at least one. This was not an exception; it was a trait.

Why we missed it: the ruler ignored the class

The really annoying question was this: how did such a common mistake never show up in the test?

The answer was in our evaluation script. The test matched each predicted box to a ground-truth box by position. If the box was in the right place it counted as correct; its label was never checked. A model that drew two boxes in the same place was never penalised by this ruler. The extra box sometimes even helped it.

Worse, the test set itself was dirty. Even a few dozen of the “right answers” contained the same twin boxes. So the ruler was bent, and so was the sample we measured with it.

Root cause: a teacher’s mistake is inherited

So where had the model learned this behaviour?

We were training the model generation by generation:

1. We trained the first generation only on a small “gold” set approved by hand by people.
2. We grew the dataset with the labels it produced.
3. We trained the next generation on that larger set.

This approach, pseudo-labelling or a teacher-student setup, is a well-known way to produce a lot of data with little human effort.

When we split the labels by source and counted, the picture became clear:

About 90% of the second generation’s training data were machine labels. The teacher model had a small, systematic habit: when unsure, give the same text two labels at once. The student learned it not as a mistake but as the right answer. And it made the habit even stronger.

The fix: ruler first, then data, model last

Our reflex could again have been “retrain”. But every new model measured with a bent ruler would be judged with the same blindness. We reversed the order:

1. We fixed the ruler. Evaluation now checks the class. A box in the right place with the wrong label counts as an error. Extra boxes on the same object are reported separately.
2. We cleaned the test set. First we corrected the “right answers” themselves.
3. We stamped every label with its origin. Every box now carries which generation or which human approval it came from. When a generation’s mistake surfaces, its labels can be removed surgically without touching the rest of the dataset.
4. We showed conflicting pairs to a judge. Two boxes on the same spot were shown to AI judges that can read the image. The loser was rejected. Regions nobody could be sure about were masked as “ignore” areas during training.
5. We retrained from a clean start. We did not rent paid GPUs until the dataset’s cleanliness was proven with numbers. In the final round, where we inspected random samples one by one, 40 out of 40 were clean.

Second surprise: the medicine turned into poison once the disease was gone

The new model arrived. We measured its raw output with the fixed ruler: class-correct score 79.3%. Nice.

Then we measured it the way it runs in production, after the post-processing steps that handle its output: 65.7%. Recall for text outside bubbles had dropped from 38% to 11%.

Why? Months earlier we had written a post-processing rule to clean up the old model’s twin boxes. It dropped one of any two overlapping boxes according to a fixed priority order. For the old model, that was medicine. The new model no longer produced twins. But the rule was still there, and now it was dropping correct detections too.

We rewrote the rule from “predefined priority” to “the box the model is more confident about wins”. The end-to-end score rose to 78.3%, so the raw model’s performance was kept almost without loss. Only then did we ship it.

Lessons

1. Ruler first. Before improving a model, know exactly what the tool measuring it sees and what it does not. Our ruler could not see our most common mistake, by design.
2. A teacher’s mistake is inherited. Pseudo-labelling is a powerful tool. But the teacher model’s systematic mistakes pass to the student as “right answers” and grow there. Keep machine labels separate from human labels and record where they came from. One day you will have to pull them out.
3. Measure end to end. A model can be good on its own. What the user sees is the model plus every rule around it. Measure them together, not separately.
4. Old patches are debt. A rule written to cover a problem becomes a problem itself once the original problem is gone. When the model changes, re-test not only the model but every patch around it.
5. Do not burn money before the data is proven. Training takes hours and costs money. Every training run on dirty data ends with a model that has learned the wrong thing better.

Finding this mistake took far longer than a training run. But once we found it, we had more than a better model: we had an evaluation setup that finally lets us see when we are actually getting better.