We built a score to catch generic content. Its first casualty was our own spec.

Most content scoring tools are marking their own homework. We know because we nearly shipped one.

This is the story of the Non-Commodity Score (NCS): a pre-publish gate that scores a page 0 to 100 on how far it sits from generic, commoditised content. It is also the story of how the v0 design failed its own test, and what we changed before a line of production code got written.

The idea

The premise is simple. Search engines and AI answer engines are drowning in content that says nothing new. If a page adds no information the top ten results do not already contain, cites no first-hand experience, carries no real data, and connects to nothing in the knowledge graph, it is a commodity. Commodities do not get cited in AI Overviews and do not hold rankings.

So we sketched a composite score built from four vectors:

1. Information gain: does the page add material the ranking set does not already have?
2. Experiential evidence: the first E of E-E-A-T. Did someone actually do the thing?
3. Empirical telemetry: hard numbers, engineering units, real data tables.
4. Entity connectivity: is the content grounded in the global knowledge graph via schema and authoritative links?

The v0 document weighted these 40/20/20/20, set pass and reject thresholds at 70 and 40, wrapped it in a Python class, and wired it to a WordPress webhook. It looked precise. It read like a shipped product.

Then we audited it

Before building, we ran the v0 spec through the same adversarial review we apply to client work. It did not survive contact.

Finding 1: the flagship metric measured the wrong thing. The 40 percent weighted “information gain” was defined as cosine distance from the centroid of the top ten ranking pages. Think about what maximises distance from a topic centroid: off-topic content. Gibberish. A page about garden furniture scored into a query about power transformers would post a spectacular information gain number. There was no relevance floor at all. Real information gain is additive and on-topic: it is what you say that the corpus has not already said, not how far you drift from the subject.

Finding 2: the code did not implement the theory. The embedding model specified was spaCy’s en_core_web_sm, which has no usable document vectors. It cannot produce the embeddings the divergence maths needs. Worse, when no comparison corpus existed, the divergence function silently returned 1.0. That is 40 points, auto-maxed, in the default path. The entity check was a binary: more than ten named entities scored 1.0, otherwise 0.5, without ever inspecting a hyperlink or a schema block. A guaranteed floor of 10 points. Net effect: an empty-ish page with no corpus and a dozen entity mentions started at roughly 60 out of 100. The tool designed to catch content that fakes substance was faking substance.

Finding 3: pseudo-precision. The 40/20/20/20 weights and the 70/40 thresholds were asserted, not fitted. Nothing traced to the Content Warehouse leak, DOJ testimony, or any labelled outcome data. Shipping invented constants under a confident label is exactly the failure mode the tool exists to police.

Finding 4: experience measured by pronoun counting. The experiential vector counted first-person pronouns. That is trivially gamed by sprinkling “we” and “I” through boilerplate, and it penalises the precise evidence you want to reward. “The field team measured a 4.2 percent loss at 33 kV” contains no first-person pronoun and is worth more than a paragraph of “in our experience”.

What we changed

The rebuild kept the four-vector skeleton and replaced every measurement.

Information gain became additive novelty with a relevance floor. Chunk the target and the top-N corpus into passages, embed with a real model (all-mpnet-base-v2 class, not a tagger’s side output). Each target passage must clear a relevance threshold to the topic centroid (default tau of 0.35) before it can score. Novelty is 1 minus the similarity to the closest passage the corpus already contains. Off-topic passages score nothing. And if no corpus exists, the score returns null and the composite renormalises over the remaining vectors. It is never auto-maxed again.

Experience is now graded by an LLM judge that must show its working. The rubric scores first-hand actions, specific conditions (when, where, what equipment, what settings), and outcomes reported as numbers rather than adjectives. Crucially, it returns the evidence spans that justify the grade, so every score is falsifiable by a human reading the same passages. The pronoun count survives only as a cheap pre-filter deciding whether the judge is worth calling.

Telemetry and connectivity now parse what is actually on the page. Real HTML and JSON-LD parsing against a domain-configurable unit lexicon, because a power engineering client’s units (kV, MVA, Hz, percent loss) are not a groundworks client’s units (t, m³, kN, MPa). Connectivity requires the rendered DOM, since JSON-LD is frequently JS-injected and raw HTML undercounts it, and it distinguishes entities that merely declare a sameAs from those whose identifiers actually resolve at Wikidata, ORCID, or the entity’s official domain.

And the honest part: the score ships uncalibrated, and says so. Weights and thresholds are parameters, not constants. They get fitted against a labelled set of pages with observed outcomes (cited in AI Overviews, holds ranking, neither) and the fit reports its AUC, so the score’s discriminating power is measured rather than asserted. Until a fitted model clears an agreed AUC floor, every output carries the label “uncalibrated: directional only” and the tool is a diagnostic profile, not a pass/fail gate.

The uncomfortable lesson

The v0 document was, by its own eventual standard, commodity content. Confident numbers with no provenance. Claimed measurements the code never made. Precision as a styling choice rather than a property of the evidence.

That is worth sitting with, because it is not a quirk of one internal spec. It is the default state of the content scoring genre. Formulas with unexplained weights, thresholds with no fitted basis, and “AI-powered” checks that reduce to counting pronouns are everywhere, and they survive because the output looks like measurement.

The fix is not a better formula. It is a different posture: every sub-score carries a provenance tag, every grade carries the evidence spans that produced it, missing data is reported as missing rather than silently scored, and no hard gate exists until the model has demonstrated, with a number, that it can discriminate.

Where this goes next

NCS is now specced to the Telperion independence standard: its own schema, its own I/O contract, a hard cost cap checked before any external call fires. The next build step is the calibration harness, because a labelled corpus is the only thing standing between a directional diagnostic and a gate you can trust at publish time.

We will publish the calibration results, including the AUC, whatever they turn out to be. If the score cannot discriminate, you will read that here too. That is rather the point.

Laurelin Labs is the innovation lab behind Telperion Insights. We prototype in the open, publish the teardowns, and productise what survives. If you want the NCS diagnostic run against your own catalogue when the pilot opens, join the mailing list.

By Piers Butler, founder of Laurelin Labs. Explore the frameworks behind the work, or read about the Four Pillars this score serves.