What the score counts

What this lesson is about

There is a way of measuring how good an extraction is. It lives in the code rather than in a claim. It compares a reading against a hand-written correct answer, field by field. Knowing what it counts is useful. Knowing what it leaves out is more useful, because that is where you are the only check.

The measuring stick was built before the thing it measures

The benchmark is code you can run offline. No key, no network, no spend. Its own note gives the reason it was built first: so the extractor was never shipped as a black box.

It has three parts. A gold corpus of three hand-written correct answers: a heavyweight tee, a piqué polo and a pullover hoodie. They are written as typed objects, so a typing mistake in the gold answer is a build error. A scorer. And a mock extractor, which copies each gold answer and then puts in a short list of deliberate, documented mistakes: a dropped bill row, a near-miss number, a swapped measurement, a missed soft field, an invented row. A mock that returned the gold answer unchanged would score full marks and prove nothing.

How a field is graded

Both the correct answer and the reading are flattened into one flat set of values. Each value is tagged with its section. Then the two sets are compared.

• A value in the gold answer that the reading matched is a true positive.
• A value in the gold answer that the reading missed, or got wrong, is a false negative.
• A value the reading produced that is wrong, or that is not in the gold answer at all, is a false positive.

So a value that is present in both and does not match costs two things at once, one of each. That is the standard treatment, and it is worth holding on to, because it is what makes an invented row so expensive.

Rows are lined up by a key before their cells are compared. A colourway by its name, a bill line by material plus placement, a measurement by its point of measure. Each key is tidied up first. So a row whose key is misread does not partly match a gold row. It counts as a gold row missed and a candidate row invented, both.

Text is matched by tidying it up and then comparing exactly: lower-cased, punctuation stripped, extra spaces collapsed. A Pantone code written as 19-4024 TCX and one written as 19 4024 tcx are the same value.

The tolerance, and the error it lets through

Numbers are not compared exactly. A number passes when the gap between it and the correct value is no more than the larger of two things: two per cent of that value, or 0.01.

That rule is sensible for its purpose. It forgives a value written as one point one against one point one zero, and it refuses a half against a third. But work out where it sits on a garment measurement, because the answer is not obvious.

A one-centimetre error passes whenever two per cent of the correct value is at least one centimetre. Two per cent of fifty is exactly one. So a value of fifty read as fifty-one passes, and a value of forty-nine read as fifty fails. A one-centimetre misreading is only caught below fifty centimetres.

Now put a real garment against it. Children's pyjama chest widths sit well under fifty, so the scorer is strict there. An adult body length of seventy allows 1.4 centimetres of drift, and a graded spec commonly carries a tolerance band tighter than that. So a cell can pass the benchmark while carrying an error the buyer's own tolerance would reject.

This is not a defect in the scorer. It is a benchmark tolerance, not a garment tolerance, and the two were never the same thing. It is here because people read a high measurement score and conclude the numbers are safe to cut from.

What the number actually is

Running the benchmark offline, today, gives this. F1 is the standard way of combining two measures into one score. Recall is how much of what was really there you found. Precision is how much of what you found was right.

Read what that is before you quote it. It is the mock extractor, whose errors were chosen by hand, against three packs that were also written by hand. It is a test that the scorer works, and a shape for what a real report will look like. It is not a measurement of the live reader. And it is certainly not a measurement of the live reader on your buyer's files.

The design document is careful about this too. It says to track the score by format group: clean digital, scanned or photographed, handwritten margins. And it says never to report a group's average without saying how many documents it came from, because a hundred per cent on two documents is noise.

One dropped row and one invented row cost the same

Here is the finding worth carrying out of this lesson, and it comes straight out of the run above.

The tee's gold answer has a care-label line on the bill. The mock drops it. That line carries eight values: its material, its placement, its consumption, its unit, its wastage, its supplier, its unit cost and its currency. The bill section of the tee scores exactly eight false negatives.

The hoodie's mock invents a drawcord line that is not in the gold answer. It carries the same eight kinds of value, and the bill section of the hoodie scores exactly eight false positives.

To the score they are the same size of mistake, and the F1 arithmetic is right to say so. To the person on the review screen they are nothing alike.

A dropped row is findable by counting. The pack has thirteen components and the screen has twelve, and any reviewer who compares the two columns will notice.

An invented row is not findable by counting, because it makes the total look right or generous. It reads as data. It then flows into everything downstream. A material is created in the library for it. It becomes a line on the cost sheet. It becomes demand to purchase.

So the two failures deserve different amounts of your attention, in the opposite order to how they feel. That difference is invisible in a single F1 figure. That is why the section report always prints precision and recall separately beside it.

What the score does not look at

The scorer names five sections: header, colourways, bill of materials, construction, measurements. The extraction the model returns is bigger than that, and the difference is not small.

Half of the header is outside the number. Three fields on every bill line are outside it, including the fabric weight and the composition that the model is explicitly told to always fill on a fabric line. And six whole lists the extractor can return are outside it entirely. Those are everything the sorting rules in lesson two were about.

There is a reason, and it is in the schema's own opening comment. The schema is described as provisional, and as the benchmark's contract rather than the database's. It is deliberately richer than the tables it was written against. The sorted sections came later, when sorting was added, and the scorer was not widened with them.

None of that makes the number dishonest. It makes it narrower than its name, which is a different thing. And it tells you where the score stops standing in for you.

Help me score one extraction against the source pack the way a field-level scorer does, so I learn where this pairing of buyer and reader actually fails.

I will give you two lists for one section at a time: what the pack says, and what the extraction produced.

For each section, sort every value into one of four buckets and count them: matched, wrong, missing entirely, and present in the extraction but not in the pack. Show the counts before any percentage.

Then tell me two things separately, rather than one combined score. How much of what was really there did we capture? And how much of what we captured was right? Say which of the two is worse for this section, and what that implies about the source document.

Treat a row whose name is misread as both a miss and an invention. Explain why that is the honest treatment when everything downstream is joined to the name.

When you compare numbers, ask me what tolerance I want to apply before you judge any of them. And tell me what a percentage-based tolerance does at small values against large ones.

Then finish with the part I care about. Which of these errors would a person notice by comparing the two documents, and which would slide past? Rank the ones that would slide past first, and for each one say what habit would have caught it.

Do not give me an overall score for the extraction. I want the shape of the failures, not a grade.