Somewhere around September 2025 I had an idea I could not shake: fine-tune a small language model, eight to twenty billion parameters, on everything I own. Every file on my filesystem. Text, audio, images, PDFs, DOCX, spreadsheets, presentations, anything a model could ingest. A model of me, trained on me.
I never built it. Not once did I open a terminal and start. Almost a year passed.
Then I read Kun Chen's post about distilling himself into a skill and looked at the repo behind it, and something clicked that had not clicked in twelve months of thinking about LoRA ranks. The interesting part of "clone yourself" was never the weights. It was the judgment. And judgment is small enough to write down.
This post is about what I built instead, how I mined it out of my own session history rather than interviewing myself, and the benchmark that told me which parts of it were real. The repo is HarjjotSinghh/jot.
The version I thought about for a year, and the version that took one evening. The difference is not ambition. It is that one of them has a feedback loop you can actually close.
Why the small-model idea kept not happening
I want to be fair to my own past idea, because the reasons it stalled are the reasons the other approach works.
Fine-tuning on your whole filesystem gives you a model that has absorbed your output. What I actually wanted was something that makes my decisions. Those are different datasets. Most of my files are artifacts: shipped code, finished docs, exported designs. The decision that produced each one, the three alternatives I rejected and why, is not in the file. It never was.
And the correction loop is brutal. When the model makes a call I would not have made, what do I do? Collect more data, re-run a training job, hope. With markdown I open the file, add one line, and the next agent invocation has it. My bottleneck was never model capacity. It was that my judgment lived only in my head and had to be re-typed into every agent, forever.
Kun's framing in his post is the part that actually moved me: his agents were constantly bottlenecked on his judgment, escalating decisions to him all day, and every escalation was a small tax on his attention. That is exactly my day. I run several agents in parallel across two machines, and a meaningful share of my time goes to course-correcting choices I have already made a hundred times.
Do not interview yourself
The first real decision was where the rules come from.
The obvious move is to sit with a model and answer questions about how you work. It produces beautiful, useless output, because everybody is aspirational about themselves. Ask me whether I write tests first and I will say yes with a straight face. Look at what I actually do and you find spikes shipped without tests and coverage backfilled before merge. The second version is the true one, and it is the only one worth encoding.
So: artifacts, not self-report. And the highest-signal artifact I own is not my code. It is the moment I typed something at an agent that was one keystroke away from doing the wrong thing.
Those moments are all sitting on my disk already, in the session stores of every coding agent I use.
The mining pipeline
Every agent persists conversations locally, and every one of them does it differently. The extractor is a set of adapters that normalize five formats into one shape.
Five session formats, one event shape. The pairing step is what makes an event useful: a correction with no memory of what it was correcting is just a sentence.
Three details did most of the work.
Only count what a human actually typed. A user turn in these logs is not the same thing as a person speaking. Tool results ride in on user turns. So do background task notifications, injected AGENTS.md preambles, hook output, and SDK-driven automation prompts. My first pass happily extracted 40-odd "judgment events" that were CI status notifications. Filtering machine-authored text out of user turns cut the corpus from 439 to 233 for Claude alone, and the 233 were all real.
Pair each message with what it was reacting to. An event stores both halves: the last thing the agent said, and what I said back. "No, don't add another abstraction" is noise on its own. Attached to the proposal it was rejecting, it is a rule.
Deduplicate aggressively. I run large acceptance harnesses that replay near-identical prompts hundreds of times. 853 duplicates got collapsed. Without that, the loudest rule in the corpus would have been whatever my test fixtures say most often.
python scripts/extract_corpus.py
# events kept : 1427
# deduped : 853
# kept:codex 1048
# kept:claude 238
# kept:cursor 110
# signal:correction 534
# signal:preference 410
The Cursor adapter was the fun one. Cursor stores every chat message as a bubbleId:* row inside a 1.5 GB SQLite file, type: 1 for user and type: 2 for assistant, with the text sitting in a JSON blob. Some keys are null, which will crash a naive read. Once you know the shape it is thirty lines.
What a rule looks like
Every rule carries four things beyond the rule itself, and the third one is the one everybody skips.
### Windows is a release target, not a dev-environment quirk
- **Do:** Treat "works on my Mac" as a non-result. Reproduce on Windows
before theorising, and when it fails there, find a Windows-specific fix
rather than writing a caveat.
- **Because:** He develops primarily on Windows with a Mac secondary, and
ships cross-platform. "The platform we deploy to" is not one platform.
- **Not when:** The failure is clearly platform-independent.
- **Evidence:** [observed 3x] "It seems to be Windows-only. This doesn't
happen on production or on my Mac."
The Not when line is load-bearing. A distillation does not go wrong by holding incorrect rules. It goes wrong by applying correct rules outside their range, firing them on every adjacent situation until you are course-correcting more than before you started. If I cannot name a case where a rule should not fire, the rule is either trivially true or I have not understood it yet.
The evidence tag matters just as much. Four tiers, and they are not decorative:
| Tag | Means | Weight |
|---|---|---|
[observed Nx] | Extracted from N distinct real sessions, dated | Decide on it |
[stated] | I wrote it down as a rule about myself | Decide on it |
[endorsed] | I agreed with someone else's advice about me | Weaker; behaviour wins |
[inferred] | Derived from adjacent behaviour | Never assert as my position |
[endorsed] exists because of a specific failure I will get to. A rule with no traceable source does not go in at all. Unverifiable rules are vibes with formatting.
The architecture
A thin loader, a knowledge base, and a private layer that never leaves the machine.
skills/jot/SKILL.md loader, resolves a local checkout first, raw URLs second
ENTRY.md routing table, decision procedure, evidence tiers
PRINCIPLES.md durable rules that survive a framework change
ENGINEERING.md backend, debugging, git, review
FRONTEND.md visual taste, design parity, slop detectors
AGENTIC.md model routing, autonomy grants, parallelism
WORKFLOWS.md feature, bug, refactor, review, ship
TOOLS.md what I reach for, and the friction I hit
BOUNDARIES.md hard stops, read before anything outward-facing
VOICE.md how I write, loaded only for text going out under my name
OPERATING.md attention, cadence, escalation
private/ gitignored: people, clients, money, positioning
The loader is the cross-agent answer. It is a stub that never changes; all churn happens in the knowledge files. One source of truth, pointed at from Claude Code, Codex, Grok, Gemini, OpenCode and the rest. It is symlinked into ten agent skill directories on my machine, so updating a rule updates every agent at once.
The public layer holds judgment. The private layer holds the specifics that make judgment actionable, and it is structured by how fast each tier rots rather than by topic: permanent identity, semi-stable patterns and people, and dated current state that has to be verified before use. Facts age. Patterns do not. Conflating them is how a memory system ends up confidently asserting things that stopped being true months ago.
The benchmark, which is the actual point
Here is the trap. A distillation that sounds like me but chooses differently is worse than nothing, because it produces confident wrong calls in my name. Sounding right is not being right. So the thing to measure is decision fidelity, and the only way to measure it is to make the model and me answer the same questions and compare.
Three arms.
The control arm is what makes the number mean anything. B scoring well on its own proves nothing if a default model scores the same.
Two rules made this honest, and both are easy to get wrong.
Clean rooms. The model arms run in fresh sessions that never saw the conversation where the knowledge files were written. A session that helped author the rules already knows the answers and scores artificially high. Both arms ran on the same model, so the only variable is the skill.
Order. A and B run first and their answers go into files I do not read. Then I answer. If I see the model's answer before committing to mine, my answer is contaminated and the whole exercise is theatre.
The questions are all decisions where a competent generic model has a plausible default that differs from what I actually do. No trivia. "What stack does he use" measures nothing. Things like: eight parallel subagents auditing a codebase, which model do you put them on? The ticket is a two-line cart bug but the checkout component is genuinely badly structured and you could fix it properly in two hours, do you? The suite passes on macOS and Linux CI and hangs on your Windows machine, do you ship?
Scoring is 0 to 2 against my answer, with a -1 for fabricating a preference I do not hold. Negative scores are deliberate. Inventing an opinion in my name is the failure mode the whole repo exists to prevent, and it has to cost more than simply not knowing.
The result, and the thing it taught me
Out of 20. The number that matters is the gap between the two model arms, not either one alone.
First run: 13/20 for the skill, 7/20 for the control. A delta of six points of judgment that a default model does not have.
Then I sorted the results by where each rule came from, and the pattern was so clean it was uncomfortable.
The single most useful output of the whole exercise. It is a falsifiable claim about which half of the file to trust.
Every rule that came out of my transcripts scored full marks. Every rule that was inferred, or imported from sensible-sounding third-party advice about how I work, scored zero or one.
The worst one is worth spelling out. My knowledge base said that when I give a blanket approval, an agent should stop asking. I had generalised that from one real line I typed mid-task: "I am approving this, so don't question it, just go for it." One line. The benchmark asked what to do at a sub-decision that approval had not literally covered, and the skill told me, with confidence, not to ask.
My actual answer was "I would ask first before proceeding myself."
One occurrence is an anecdote. The repo's own precedence order says three occurrences make a rule, and I had violated my own precedence order while writing the file that contains it. That is exactly how a distillation goes bad: not by holding wrong opinions, but by stating real ones too broadly and asserting them in a voice that sounds sourced.
The [endorsed] evidence tier came out of the same audit. One rule was tagged [stated] because I had genuinely written it down, but I had written it down by agreeing with an external review of my own work. Endorsing a description of yourself is not the same as being described correctly, and where the two diverge, observed behaviour has to win.
The regression, which was the best part
I patched the four rules, re-ran, and got 18/20. Three of the four misses closed to full marks.
And one question that had scored 2 dropped to 0.
The rule I added to fix the refactor question said, roughly, "clear the blocker in your own path even if it costs hours." True, mined, correctly stated. The rerun applied it to a two-minute fix on a live customer list and decided to just go ahead and fix it. Which is the opposite of what I do, because that is not my code in front of me, that is somebody's real customers.
This is precisely what my own golden-case file warns about: a rule that fixes one case and breaks another is stated too broadly. Two fixes went in. The rule gained an explicit live-system exclusion, and BOUNDARIES.md now outranks every principle in the repo rather than merely tie-breaking against them. A boundary is a stop, not a consideration.
There was a contributing cause too, and it is a nice illustration of how these systems fail sideways. After the first run I had noticed the skill loading all sixteen files for a ten-question set, so I capped it. The cap worked, and it also stopped the agent loading the private layer where the customer-list context lives. Fixing one problem quietly created the conditions for another. The private-load trigger now fires on live customer data whether or not a person is named.
Credit where it is due: the skill did flag that answer as inferred rather than asserting it as my position. The honesty tier worked. It just landed on the wrong side of the line.
The benchmark burns itself
I fixed Q9 and ran the set a third time. 20/20.
That number is worthless and I am not claiming it.
When I promoted the first run's results into the knowledge files, the evidence lines
quoted my actual answers verbatim. Seven of the ten questions now have their answer
key sitting inside the files the agent reads on its way in. AGENTIC.md quotes my
answer to the approval question. ENGINEERING.md quotes the Windows one.
PRINCIPLES.md quotes two more.
The agent noticed before I did. Its completion report said, unprompted, that several questions were decided directly from observed-tier citations dated the same day as the benchmark. It was telling me the test was rigged, and it was right.
This is a general property of the thing, not a mistake I made once: a distillation benchmark burns itself the moment its results are folded back in. Evidence citations are the answer key. There are two ways to live with it. Hold out a set, writing more questions than you score and only ever citing the spent ones. Or cite the rule rather than the answer, recording that a benchmark corrected a rule's range without quoting the correct call.
A spent set is not wasted. It becomes a regression suite: run it on every change to confirm the patches held and nothing adjacent broke. On that question it still works fine, and it told me the Q9 fix took without breaking the four rules before it. It just cannot tell me how faithful the skill is any more.
So the honest numbers are 13 and 18, both measured before anything was folded back in. The 20 tells me my patches work, which is a different and smaller claim.
Keeping the private half private
The repo is public. The knowledge files quote real sessions, and real sessions name colleagues, clients, and internal tooling.
.gitignore does not catch that class of mistake. The file is tracked on purpose; it is the words inside it that must not ship. So there is a scanner that checks every tracked file against a denylist, and the denylist itself lives inside the gitignored directory, because publishing the list of words you are hiding tells everyone exactly what to search for.
It runs as a pre-commit hook, and it earned its place immediately by catching three identifiers I had already pushed, including an internal tool name sitting inside the very rule that says never to name it.
What it actually feels like
Here is the part I did not expect.
Using it is uncanny. Not "the model is impressive" uncanny. Specifically: you ask it a question you have not thought about, it answers, and you get a small jolt of yes, that is exactly what I would have done, and that is exactly why. It is talking to yourself, with the useful property that this version of you has read every correction you have ever issued and does not get tired.
It also disagrees with me occasionally in a way that is more interesting than being right would have been. When the benchmark disagreed, the skill was not being stupid. It was faithfully applying a rule I had written badly. The gap between the two arms is a measurement of my judgment. The gap between the skill and me is a measurement of my writing, and those are separately fixable.
Kun's argument is that the distillable part of you was never your moat, and letting go of it frees you to build the part that is. I would put it slightly differently after doing it: the exercise is worth it even if nobody ever uses the skill, because writing down a rule with a Because and a Not when and a piece of evidence forces you to find out whether you actually hold it. Several of mine did not survive contact with that format.
The loop from here is boring, which is the point. Every correction becomes a decision pair. Every benchmark miss becomes a golden case with the rule that would have prevented it. The score goes up, the escalations go down, and about once a week I find out something about how I work that I would have sworn was the other way around.
Not bad for an idea I sat on for a year because I thought it needed a GPU.
The repo is HarjjotSinghh/jot. The idea and the architecture come from Kun Chen, whose kun repo is the thing worth reading first; the mining pipeline, the private layer, and the benchmark are where mine diverges. If you build one, build the benchmark before you trust a single rule.