A judge for messy data
Everyone has piles of data they can't read. Summaries don't help, because you can't count, compare or check them. This is how I made sense of five years of my own meetings with a judge instead of a reader: a tool that sorts every piece of text against definitions you write and never writes a word itself. And how to do the same with your pile.
Your pile
Everyone has one. Meeting notes, a support queue, a folder of interview transcripts, a year of Slack, a document library nobody has opened since the reorg. Too big to read and too messy to search.
Mine was 2,470 recorded meetings, five years of my working life. I wanted an honest picture of it: where the time went, what kind of work it was, what I decided and what I promised. I wanted that picture to be something I could check and explore, not simply something I consumed.
The usual move is to hand the pile to an AI agent and ask for summaries of different types. I did that and got pages of fluent prose.
Then I noticed what I couldn't do with it: ask the agent to be accountable for its answers.
- I couldn't quantify any of it.
- I couldn't compare one quarter, team or source with another.
- I couldn't audit whether the agent was making sense or producing gibberish.
- I couldn't check where its conclusions came from or why it reached them.
I turned a pile I couldn't read into a smaller pile I couldn't trust.
Jev, by typesafe, works differently from other agents. It doesn't turn the whole pile into its own summary. It interprets pieces of my data to answer the questions I write, and it gives a confidence with each answer.
This is the story of how I used it to make sense of my working life since 2021.
A judge, not a reader
1Jev answers one question about one piece of text, from your list
Jev, made by typesafe, is a judging service. You give it one piece of text and a rubric, and it returns a verdict: which of your labels fits the text, and how sure it is about each one. It never writes a sentence.
A rubric is four things. A question. The thing the question is asked about, which I'll call the unit. A list of labels, each with a definition. And the lines between labels: where two of them could both claim the same text, a sentence saying which one wins.
Here's a small one, so the words mean something. Question: "What is this support ticket about?" Unit: one ticket. Labels: billing ("charges, invoices, refunds"), access ("can't log in, lost password"), bug ("something that used to work doesn't"), none of these. A line between labels: "a refund request for a feature that broke is a bug, not billing."
Give Jev a ticket and that rubric, and it returns something like: bug 0.84, billing 0.12, access 0.03, none 0.01. Those four numbers are one judgement split across your labels, so they add up to 1. The label with the highest number becomes the tag. That's the whole mechanism, and everything in this post is built on it.
2You write the rubric before you can ask anything
That's backwards from an LLM, where you ask first and get prose back. With Jev, nothing happens until the rubric exists.
It's extra work up front, and it is the work: writing the rubric forces you to say exactly what you want to understand about your data, in words precise enough that a judge can apply them the same way ten thousand times.
The payoff is what the rubric buys you. Once it exists, you can run it over far more text than you could ever read, because each verdict takes a fraction of a second and the whole five years cost less than a budget conversation. And you can run it again every time you change your mind about the labels. That second part matters more than it sounds, and I'll come back to it.
3What a real verdict looks like
Here is one from my project. The unit is a ten-minute piece of a meeting, which I'll call a stretch. (Why ten minutes, and why a piece rather than the whole meeting, is in the next section.) This stretch comes from a planning call in August 2026 about a dinner series we were hosting, and it opens with me asking: "Why should someone show up? What are they going to get out of it?"
I asked Jev two questions about it. First, a broad one: which family of work is this? My nine families are things like market, clients, product, leadership, with the full list further down. Jev's top answer was market, with a confidence of 0.54. Then a narrower one, within that family: which area of market work is it? Two of the labels, with their definitions, were:
- Marketing: positioning, campaigns, landing pages, outreach plans, events that build an audience.
- Editorial: newsletters, podcasts, articles, thought leadership. Not this: material made to sell, which is marketing.
Which area of market work is happening in this stretch?
tag: marketing
Marketing 0.77, editorial 0.23, the other labels near zero. Tag: marketing. You can see why from the opening line: deciding why people should come to an event is positioning, which the marketing definition names.
Two things about those numbers matter for everything that follows.
First, the labels and their definitions are the whole game. Jev applied the distinction I wrote. The 0.23 on editorial is there because dinner events sit close to content in my definitions, and if two labels blur, the blur is mine to fix.
Second, the confidence is a number, so you can act on it. A verdict of 0.77 against 0.23 is one you can keep. A verdict of 0.51 against 0.49 is one you should look at, and I'll call those unsure verdicts. For the broad family question I set a bar at 0.6: this stretch's 0.54 was under it, so its ten minutes were shared between its top two families in proportion to the confidences, mostly market, a slice content, rather than all going to the winner. Nothing gets forced into a box it only half fits, and the unsure verdicts become, later in this post, your most useful to-do list.
4Why that's different from asking an LLM
An LLM writes an answer. Jev picks one from the list you gave it, so it can't drift, improvise, or invent a label you didn't offer.
An LLM's "I'm fairly sure" is a phrase. Jev's confidence is a number, so you can set a bar against it and sort every verdict into keep or look at.
The same rubric gives answers you can compare: across thousands of items, and across runs on different days.
And it's fast and cheap enough that re-running everything is routine rather than a budget conversation.
So the hard part is writing a rubric that fits your data, and you won't get that right the first time. Finding it is a loop: write a rubric, run it, read what comes back, change it. Three decisions have to be made before the loop can start, and they're next.
Three decisions before the loop starts
I got each of these wrong at least once, and each mistake cost me a round.
1Pick your unit: the thing that gets one verdict
The unit decides what your totals mean. If the unit is a whole meeting, you can count meetings by kind. If it's a ten-minute stretch, you can count minutes by kind, and that's a different answer.
"Where did my time go?" can't be answered at the meeting level, because meetings wander. A call starts on product, drifts into hiring and ends on pricing. One label for the whole meeting would blur all of that, and I'd be counting the meeting's title rather than its contents.
So the unit became a stretch. A script cuts each transcript where the speaker changes, into pieces of five to fifteen minutes. Then Jev gets one yes-or-no question per piece, "is this the same topic as the piece before it?", and the script merges neighbours where the answer is yes. Ten minutes is the trade-off: long enough to contain one topic, short enough that a meeting's topic changes fall between stretches rather than inside them. Every stretch keeps its start and end time, so any number I produce later can be traced to the moment it came from.
2Know what's in your pile
"Messy" means mixed as well as big, and a mixed pile distorts every total, because items that shouldn't be there still get a label and still get counted.
Mine turned out to contain more than meetings. Talks I'd given, YouTube transcripts and other material had been pulled in alongside the recordings, and I didn't know that when I wrote my first rubric. The judge told me, in a way I'll show in the moves below, before I noticed any other way.
3Sample before you scale
Don't run your first rubric over the whole pile. Run it over a random sample.
Mine was 200 meetings out of 2,470, picked at random, then checked so that every company I'd worked with and every one of the five years was represented. The only rule about size is that the sample has to be big enough for each label's share to mean something, and spread enough to look like the pile. For you that might be fifty items or a thousand.
A sample does three things a full run can't. It's fast enough to repeat, so you'll iterate instead of defending your first attempt. It's small enough to hand-check, so you can read a few verdicts per label and see whether the judge understood you. And it's cheap enough to throw away, which is what you want when the first rubric is wrong, because it will be.
Then keep that same sample fixed. Every rubric version runs over the same items, so when the numbers move, it's the rubric that moved them.
A note on the counts in this post, so they don't look like they drift. Claude tagged 36 meetings under my first list, and 29 of those were compared with Jev. The second list was tested on a larger slice of 459 meetings. Everything from the third list onward ran on the fixed 200-meeting sample, and then on all 2,470.
With those three in place, here is where my rubric ended up. Then the three questions that got it there.
The rubric I ended up with
The four parts of a rubric, each one for my case.
My first list didn't ask this question. It asked a different one, and the answers to that one are what led me here.
The four lists that got me here
Your data has a shape of labels that fits it best, based on what you want to understand about it. You won't know that shape at the start.
You find it by looping. Write a rubric. Run it on the sample.
Then read the totals first: each label's share of the sample, added up before you look at any single verdict. A label claiming a share you know to be false is obvious in a total and invisible in a hundred individual verdicts. Then read the unsure verdicts.
Then ask two questions: was Jev confident, and do the labels make sense? When either answer is no, rephrase.
Some of the rephrasing is finding better words for what you meant. Some of it is realising what you actually want to know. Both happened to me, in the same loop, and that's the point of this section: refining the rubric for Jev refined the question for me.
Four lists in two weeks, three different questions. Each entry says what I saw, and so what I asked next.
The liststrategist · operator · advisor · learner · facilitator · negotiator · product owner · architect and builder · observer · design critic
This was a list of how I show up in a room, written from memory. Claude tagged 36 meetings with it, and kept running into the same gaps: four roles missing, which were peer and thought partner, systems designer, user researcher, and plain social time.
What I saw: a role isn't a kind of work. A hiring call and a pricing call could both be "strategist", so roles couldn't be added up into "where did my time go". And before I could count anything by kind, I needed to know which company and which project each meeting belonged to, because the folders they'd been filed in were often wrong, and the picture I wanted was per company.
So I asked: which company and project is this about, and what would it be on a CV?
The listthe company the meeting was about, judged from its content rather than its folder → the project within the company → a job-posting title such as “Head of Product”
What I saw: this version proved the judge could be trusted with judgement calls at scale. On 95% of meetings its answer matched the company I'd filed the meeting under, and when I read the misses, several were my filing that was wrong, not Jev. But a CV title is for presenting work, not counting it. I still couldn't say what kind of work filled a given afternoon, because the unit was still the whole meeting.
So I asked: what kind of work filled each ten minutes?
The listnine families → 23 areas, plus the leadership lens
Two things changed at once. The unit became the stretch, for the reasons in the previous section. And the labels became kinds of work rather than roles or titles, so that they could be added up.
What I saw: on the 200-meeting sample, two labels claimed shares of my time that I knew were false. The totals had caught the rubric out, and the fix for each was one sentence. Move 6 below tells that story, because the way I found it is the move.
So I asked: the same question, with the definitions corrected.
The listthe same nine families and 23 areas, with editorial redefined by who the material is for, and the leadership lens asking only about my own actions
What I saw: same 200 meetings, re-run; both false shares fell to where they belonged, and the answers I cared about, where my time went by quarter, stopped moving between versions. Then the full run: 2,470 meetings, 12,603 stretches, about five minutes.
Four lists sounds like trial and error. Each round used one or more of a small set of moves, though, and the moves are the part you can take to any pile.
Eight moves that got the list right
In the order I'd suggest trying them. Two terms come up throughout: the unsure pile is the verdicts where the top two labels are close, and the "none" pile is the items where Jev chose "none of these". Both are piles the judge sorts for you.
- 1Let Claude suggest labels, let Jev prove they earn their place
- 2A “none of these labels” pile shows what your list forgot
- 3Two labels that keep competing over the same data need a line between them
- 4When Jev and an LLM disagree, a label is missing
- 5Test several wordings on the same sample, keep what wins
- 6If a result feels wrong, say so and make the judge investigate
- 7After every rubric change, re-run the same sample and compare
- 8Watch for diminishing returns, and decide when to stop
1Let Claude suggest labels, let Jev prove they earn their place
Start here. It's the most concrete thing you can do, and it gets the list moving.
By the third list, Claude's job was no longer to tag. Its job was to read the piles Jev had already sorted, the unsure pile and the "none" pile, and propose what was missing. It came back with suggestions: a label for pairing sessions, a label for designing systems rather than building them, a label for user research that had been scored as passive observing.
Then Jev checked each proposal the only way that matters. I added the suggested label to the rubric and re-ran the same sample. If the new label took a real share of the sample, and took it from "none of these" or from the unsure pile, it stayed. If it only took share from a neighbouring label that was already working, it went.
That division held for the whole project. The LLM reads and proposes, because that needs words. The judge measures whether the proposal is true, because that needs the same answer across hundreds of items.
2A “none of these labels” pile shows what your list forgot
Every question I gave Jev ended with the same option: none of these.
It's tempting to leave it out, because it feels like giving the judge an excuse. It's the opposite. Without it, every item has to land somewhere on your list, and the ones that don't fit get quietly forced into the nearest label. You never find out.
With it, the misfits collect in one place. On the second list, "none of these" was chosen for 10 of the 386 meetings that were eligible for a CV title, under 3%, and that low number told me the title list was close to complete. On the first list, the "none" pile and the notes Claude left while tagging pointed at the same four gaps, which is how peer and thought partner, systems designer, user researcher and social time got onto the list.
The rate matters as much as the pile. A rising "none" rate for one kind of item is the earliest signal you'll get that your data contains something your rubric hasn't imagined yet.
3Two labels that keep competing over the same data need a line between them
Jev returns a confidence for every label, so each verdict has a second place: the label with the second-highest number. Second place tells you which other label was competing for the same item.
On the 200-meeting sample, stretches tagged marketing often had editorial in second place, and stretches tagged editorial had marketing right behind. Even in the final run, 224 of the 12,603 stretches have those two labels in first and second place. On 132 of them the gap between the two confidences is under 0.3, and on some it's as narrow as 0.51 against 0.49.
That pattern means one thing. The two definitions overlap, and the judge is doing its honest best to choose between options that, as written, both fit.
The competition came from pitch decks and sales scripts. My editorial definition said "newsletters, podcasts, articles". My marketing definition said "positioning, campaigns, landing pages". A script for a sales video is arguably both. The fix was one sentence in the editorial definition: "Not this: material made to sell, which is marketing." Once the line was drawn by who the material is for, most of the competition stopped.
Some of it didn't, and that's fine. An item where the judge is still torn gets shared rather than forced, as described in the verdict section, and second place is how I know which items those are.
4When Jev and an LLM disagree, a label is missing
This was the first thing I did with Jev, before trusting it with anything.
Claude had tagged 36 meetings with the first list. For 29 of them, from the first two weeks of October, I asked Jev the same question with the same roles and compared the two sets of answers. They agreed on 17. The 12 disagreements were where the learning was.
One week stood out. It was a week of personal career conversations, and Claude had labelled almost every one of them "learner". Jev wouldn't. It spread the same meetings across strategist, negotiator and operator, with low confidence on most of them.
Both judges were telling me the same thing in different ways. Claude had stretched "learner" to cover meetings that didn't fit anywhere. Jev had refused to stretch, and its low confidence said the list didn't contain the right answer. What was actually happening in those meetings was me thinking out loud with people I trust, and the list had no label for that. That's how "peer and thought partner" got onto the list, and the same gap turned out to explain developer pairing sessions and friend check-ins that had been forced into other roles.
5Test several wordings on the same sample, keep what wins
Because a sample run takes seconds, wording stops being a matter of taste and becomes something you can test.
On the second list, I ran three versions of the same definitions over the same 459 meetings. The first version pushed internal team meetings into "advisory" and content work into "creative operations", two of the CV titles. The second sharpened those two definitions and nothing else. The third asked for the title in one step instead of asking for the family of title first and the title second.
Then I compared the totals. The sharper wording alone cut the false "creative operations" tag from 19 meetings to one or two. The one-step and two-step versions gave the same title 98% of the time when both were above my confidence bar, so I kept the simpler one. Three experiments, same sample, a few minutes each.
6If a result feels wrong, say so and make the judge investigate
My first sample run on the third list said 18% of my time went to editorial content. No number on the page told me that was wrong. I did. I know roughly where my weeks go, and a fifth of them don't go to newsletters.
That instinct is a tool, and it's most useful early, while you're still finding the broad shape of the labels and you know the data better than the judge does. Its job is to pull on a thread that doesn't feel right, not to overrule the judge.
In practice that means saying to whoever runs the rubric, a person or an agent: "That doesn't feel right. Show me how you got there." Then reading the stretches behind the number.
That one question led straight to two things I hadn't known. Pitch decks and sales scripts I'd worked on were being counted as content, which was a rubric problem, and move 3 is the fix. And the pile held talks I'd given and YouTube transcripts that weren't meetings at all, which was a data problem: those items need marking so they stay out of the meeting-time totals, and that clean-up is part of the next round. One wrong number exposed both.
The same run showed the leadership lens on 40% of all minutes. Nobody leads 40% of the time. When I reread the question, the reason was in the wording: it asked whether leadership was happening in the stretch, not whether I was the one doing it, so every time a colleague set direction it was credited to me. The fix was a correction you can read in the rubric, "is Adrian himself clearly exercising one of these skills", and the number fell because the definition changed, not because I nudged it until it looked right.
Prepare to be surprised in both directions. Sometimes the judge is right and your memory is wrong, which is its own lesson about your data. Sometimes your gut is right and the model has misread something. The only way to find out what's happening is to investigate rather than accept blindly. Don't assume the agent is always correct.
7After every rubric change, re-run the same sample and compare
Every change to the definitions got a version number, and the same 200 meetings were run under the old version and the new one.
That sounds like bookkeeping. It's the only honest way to know whether a change helped. The two false shares from move 6 are the proof. After the "not this" note, editorial fell to 5.5% of minutes. After the lens was scoped to my own actions, it fell to 15%. Same meetings, same unit, two rubrics, side by side.
Without the fixed sample, I'd have been comparing one run on one set of meetings with another run on a different set, and any difference could have been the meetings. It also meant a change could be undone. If a new version had made something worse, I'd have seen it in the same table and gone back.
8Watch for diminishing returns, and decide when to stop
You can run this loop for a long time. Each round makes the rubric a little better, and at some point the gains stop being worth the rounds.
The sign to watch is a confidence ceiling: the point where the unsure share stops dropping no matter how you reword. After my last round, Jev was still unsure which family 41% of stretches belonged to, so their minutes were shared.
I'll say plainly that that is a bar I no longer accept. For a rubric I'd rely on, I want Jev confident on something like 85 to 90% of items, with the rest marked unsure, and finding out whether this data allows that is my next round.
I stopped where I did because the answers I cared about had stopped moving. Where my time went by quarter, what I decided, what I promised: those were the same from one version to the next, and another round of wording wouldn't have changed them. That's a real signal, and it's the one to use. Just set the bar higher than I did.
Part of this is accepting what kind of data you have. Transcripts of people talking are qualitative. They'll never label perfectly, and chasing the last few points of confidence is chasing something the data doesn't contain. What you want is strong, directional confidence on the questions you actually asked, with the unsure cases marked as unsure.
When the ceiling won't move and the answers still feel wrong, that's a different signal. The problem is the structure rather than the wording: the unit, the families, or the question itself needs rethinking, and that's a bigger decision than another round of the loop.
That's how the list got good. What follows is what a judged pile gives you that a summarised one can't.
What a judged pile gives you
1You can keep rethinking, because re-running is cheap
When the totals exposed the editorial problem, the fix was one rewritten label. Then I re-ran all five years: about 34,000 verdicts, since every stretch gets the family question and the lens, and all but the "none" stretches get the area question too. It took about five minutes.
When changing your mind costs five minutes, you change it as often as the data tells you to. That's how the rubric gets good, and it's why the extra work of writing a rubric pays for itself.
2Every claim can be checked back to its source
I gave Jev one more yes-or-no per stretch: is an explicit decision being made here? It said yes on 555 stretches. Only those went to Claude, which pulled out 2,005 decisions, written as pointers to the exact lines in the transcript rather than as copied text, so nothing could be paraphrased into existence.
Then Jev came back as a checker, with another yes-or-no: does the passage at those lines actually support this decision? 1,630 passed. The other 375 were dropped before I ever saw them.
Every decision that remains links to the moment it was said. That turns AI output into something you can audit rather than something you take on trust.
3It knows when it's unsure, so the sure cases can run themselves
When I compared Jev's verdicts with Claude's on the first list, they agreed on 9 of the 10 meetings where Jev's confidence in its top label was above 0.7, and on far fewer below that.
That's the property that lets you automate. The confident cases run themselves, the unsure ones go to a person, and the unsure pile doubles as the list of where your rubric needs work next.
Think big: what you can build once the pile is labelled
Once every stretch of every meeting carries a verdict, questions that used to mean reading become questions that mean counting. Where did the quarter go? What did we decide, and did it stick? What did I promise, and did it land?
These are the four things I built on the labels, and each has an obvious cousin on someone else's pile.
Before the recipe, the cases where none of this applies.
Where it doesn't fit
A judge isn't a writer, so don't ask it to draft the strategy memo.
It judges one piece of text at a time, so a question like "has our culture changed since the reorg?" needs a different order: count with the judge first (how many stretches per month are tagged conflict, or praise), then let an LLM read the counts and interpret them.
For a dozen interview notes, skip all of this and just ask Claude.
And if your labels are fuzzy and overlapping, Jev will give you confident nonsense, because it will apply your fuzzy definitions consistently. The totals will look plausible and mean nothing. That's one more reason the rubric is the work, and why the unsure pile and the "none" pile matter: they're where fuzzy definitions show themselves.
If it does fit your pile, here is the mindset, then the recipe.
Working with Jev: the big picture
- The rubric is the work. Decide what you want to know before you ask anything.
- Your data has a shape of labels that fits it. You're searching for that shape, not just labelling.
- Expect the first version to be wrong, and treat that as the method.
- Judge first, write second. The judge makes the calls; the LLM writes only where words are needed.
- Trust comes from checkability. Every verdict has a confidence, and every claim traces back to its source.
- Size the unit to the question. What you count decides what your totals mean.
- Know what's in your pile. A mixed pile distorts every total.
- Sample before you scale. A random, spread-out slice you can re-run in seconds beats a full run you can only afford once.
- Use your gut to pull on threads, not to overrule the evidence.
- Give every label a "not this" note wherever it borders another. (editorial and marketing, move 3)
- Always include "none of these", and watch how often it fires. (the four missing roles, move 2)
- Read the totals before reading any single verdict. (the 18% that couldn't be right, move 6)
- Keep the confident verdicts, work the unsure pile. (9 of 10 right above 0.7)
- Run two judges and read only where they disagree. (the "learner" week, move 4)
- Keep the sample fixed; change the wording, bump the version, re-run. (move 7)
- Let the judge gate what the LLM reads, then let it check what the LLM wrote. (2,005 decisions, 1,630 kept)
- When a result feels wrong, say so and investigate. (move 6)
- Stop when a couple of rounds leave your answers where they were, and set the bar higher than I did. (move 8)
Try it yourself
- Pick one pile, look at what's actually in it, and choose the unit you'll count. The ticket, the ten-minute stretch, the paragraph, the pull request.
- Write the question you want answered, then a short first list of labels with definitions, and a "not this" note wherever two labels border each other. Start small; my first list had ten labels and the final one has 23. For last quarter's meeting notes: "Sales: calls with prospects. Not this: check-ins with existing clients, which are client work."
- Pull a random sample, spread across whatever varies most, and save the list.
- Run the rubric over the sample. Jev is an API, so you'll need an account with typesafe and a short script, or an agent in Claude Code, that sends each item with your rubric, collects the verdicts, and adds up each label's share. If you don't write code, this is the step to hand to someone who does, or to an agent. A technical companion to this post, written for an agent to read, will carry a copyable version; until it exists, the moves above are the brief to give whoever runs it.
- Read the totals first, then the unsure pile. Fix the wording, give the rubric a new version number, and re-run on the same sample. Expect several rounds.
- Stop when the answers you care about stop moving between rounds, then scale up. Move 8 is the stopping rule.
The rubric is the work. The judge is how you find out if it's right.
The figures come from the run logs, pilot results and tagged files of this project, October 20 to November 2, 2026. A hand-check of 60 stretches against my own judgement is still pending, and the 41% unsure share is the figure I most want to move. The thresholds and shares are specific to my data; measure your own.