Journal · scoring
Building a quality score without vanity metrics
A quality score is useful when it changes a conversation. It is harmful when it becomes a new vanity number with a more expensive name. The Event Quality Charter clinic exists because teams keep smuggling page views back in through the side door.
Start from jobs, not from the schema
List the jobs your app actually serves this quarter. Not the full event dictionary — the jobs. For a wallet: pay a bill, check a balance, send money, freeze a card. For grocery: find an item, substitute, pay, track a rider.
Only events that help you see those jobs may enter the score. Screen views of marketing carousels do not. “App opened” does not. “Feature highlighted impression” almost never does.
Allow friction events; distrust volume events
Error shown, OTP resent, search revised, filter cleared, payment method switched — these belong. They describe struggle and recovery. Counts of “content viewed” usually describe inventory, not quality.
If an event fires because a designer added a tooltip, it will inflate after every release. That is vanity with a timestamp. Ban it from the score even if it stays in the warehouse for other work.
Write admission rules
Our charter template has four gates. An event must (1) map to a job or a known failure of that job, (2) fire at a stable moment in the UI, (3) be joinable to a session id, and (4) have an owner who will notice if it breaks. No owner, no admission.
Revisit the charter when a new journey ships. Do not let the score silently absorb every new event “just in case.” That is how indices rot.
Publish the limitation with the number
If you cannot see cross-app payment completion, the score under-counts success. If replay coverage is iOS-heavy, Android struggle is under-sampled. Put those sentences next to the weekly number. Executives can handle footnotes. They cannot handle a surprise when support tickets diverge.
A score without vanity is quieter. It moves slowly. That is a feature. If yours jumps every time marketing runs a campaign, you are still measuring attention, not quality.