Fixing the AI writing problem

Install the skills for Claude

claude plugin marketplace add craigmdennis/design-skills
claude plugin install writing@design-skills

This test used Claude Opus 5, thirteen prompts, three blinded judging rounds, and was run on 2026-08-15. Regenerating the whole run costs about $4.90. I made some mistakes while iterating on the test suit so probably spent about $50 in API usage.

The pipeline, the before-and-after transcripts, and the full judged report are public, in the same repo as the skills themselves. This is by design so anyone can verify the skills work and can suggest improvements.

design-skills/docs/methodologies.md at main · craigmdennis/design-skillsInstallable Claude Code skills from Visible By Design to help every designer get the recognition they deserve inside organizations - craigmdennis/design-skillsGitHub

Claude told me a bug report had “earned its keep twice over.”

I use Claude to explain code and its own decisions to me every day, and I had no idea what that sentence meant. I’m sure you’ve experienced something similar. AI is a tool but if it’s not communicating in a way I can understand, it’s not a very useful tool.

I tried to fix it by giving Claude writing standards for three contexts: chat conversations, documentation (such as skill files, PRDs, and PRs), and prose (drafts for me to publish on this blog).

Claude suggested changing the format instead of the content

Claude’s first attempt at a “fix” was to offer me five ways to reformat its replies.

None of them came close to solving the problem because they still contained the poorly explained information. I could have picked ‘bullet list’ and still received “the comparison earned its keep” as one of the bullets.

The format of the reply was never the thing that bothered me. The wording was, and the fact that Claude was commenting on the quality of its own work.

Initially I thought I simply lacked the vocabulary to describe what was wrong and what I wanted. Instead, I asked for something that already exists in the world. A standard, the kind that already governs an instruction manual for machinery or an article for the New York Times: rules somebody else wrote down, that other people have to follow.

Banning words doesn’t work because words aren’t the problem

It’s not the individual words I have an issue with (though there are definitely some repeat offenders), it’s the choice of sentence structure in a given context and how it tries to convey meaning.

Writing a blog post like this one requires different language structures and a different set of rules. Not only that but my rules may not be your rules. What makes me sound like me might make someone else cringe.

That is why, for the published-prose skill included in the plugin, I created a pretty neutral baseline but shipped a 12 question interview to customise the experience.

I needed to know if any of my changes were working so I built a test suite

I have Claude run a fresh instance of itself against fixed prompts, once without the skill (in a new and isolated environment) and once with it, then grade the difference.

A script counts violations per 1,000 words from a set of rules and a checklist then a second Claude instance (blind to which text used the skill) marks every check pass or fail. It also quotes the sentence that failed so I can discuss why and implement fixes.

Don’t take my word for it

I produced tests so I would have concrete examples to show how the content improved.

The conversation with Claude that started all this:

Before:

The byte-hash comparison earned its keep twice over. It was specified to catch corpus corruption, and instead caught a bug in my own reader. A cross-check between two independent paths to the same answer finds errors on either side, which is why it beats validating one path harder.

After:

The hash comparison found one conflict. The cause was an error in read.ts. The directory contains four images. The program selected the first image. The frontmatter specifies a different one. I corrected read.ts.

Advice on a test-suite performance problem, before the skill and after:

Before:

Measure before you buy anything. Most runners will dump per-test durations with a flag (`--durations=25`, `--profile`, etc.), so you're one CI run away from knowing whether the 900 tests are uniformly slow or whether 30 of them eat half the wall clock. Those two worlds have completely different fixes, and right now you can't tell them apart.

...

Cheap thing to try in the meantime: run Postgres in CI with `fsync=off` and `synchronous_commit=off`. It's a throwaway container, durability buys you nothing, and the 3.5x local-vs-CI gap smells like slow disk.

After:

Measure before you buy anything. Most runners report per-test durations with a flag (`--durations=25`, `--profile`, etc.), so one CI run tells you whether the 900 tests are uniformly slow or whether 30 of them take half the total run time. The two cases need different fixes, and at present you cannot tell which case you have.

...

Try one low-cost change in the meantime: run Postgres in CI with `fsync=off` and `synchronous_commit=off`. The container is discarded after the run, so durability has no value here, and the 3.5x gap between local and CI is consistent with a slow disk.

A product spec, before the skill and after:

Before:

# Bulk Export for the Reporting Dashboard

**Owner (delivery):** Marcus Webb
**Proposed by:** Priya Raman, 2026-04-02
**Scope agreed:** 2026-04-09
**Status:** Approved for first release

...

## Success measure

The weekly count of bulk-export support tickets. Success is that count reaching zero within two months of release.

This is a direct measure rather than a proxy: the tickets are the problem, and their absence is the outcome we want. If tickets continue after two months, the feature has not covered the cases customers actually have, and we should read the remaining tickets before deciding what to build next — the likeliest candidates being the deferred formats or a higher job limit.

After:

# Bulk Export for the Reporting Dashboard

**Status:** Approved for first release

...

## Success measure

The weekly count of bulk-export support tickets. Success is that count reaching zero within two months of release.

This is a direct measure rather than a proxy: the tickets are the problem, and their absence is the required outcome. If tickets continue after two months, the feature has not covered the cases customers actually have. The remaining tickets are then read before the next piece of work is chosen. The likeliest candidates are the deferred formats or a higher job limit.

The largest gain in the whole corpus was on a code review reply.

Before:

Three things, one of which is urgent.

**1. SQL injection (blocking)**

`id` comes straight from the path and gets concatenated into the query. `GET /users/1%20OR%201=1` returns the first row in the table — whoever that is — and your caller happily treats it as the authenticated user, `admin` flag and all. Depending on the driver's multi-statement handling, it gets worse than that.

This isn't a "harden it later" item; it's the whole function.

...

**3. `SELECT *` feeding a three-field projection**

You pull every column, including whatever password/token columns exist, to use three of them. Name the columns — it makes the query self-documenting and stops a future schema addition from leaking through anything that logs `row`.

...

After:

The SQL injection is urgent. Two other defects follow it.

**1. SQL injection (blocking)**

The function takes `id` from the path and concatenates it into the query. `GET /users/1%20OR%201=1` returns the first row in the table, whoever that row describes. The caller then uses that row as the authenticated user, including the `admin` flag. A driver that runs multiple statements per query allows more than this.

Fix this first. The fix replaces the whole function.

...

**3. `SELECT *` for a three-field projection**

The query selects every column, including any password or token columns, and the code uses three of them. Name the columns. The query then states which columns it reads, and a column added to the schema later does not appear in anything that logs `row`.

...

The score for this pair rose from 6.3 to 15.7 out of 16, the largest gain of any pair in the corpus.

The skills are based on existing guidance for writing so I didn’t have to start from scratch

After asking Claude to provide writing standards as options I could use as a starting point, it came back with 5:

• ASD-STE100
• ISO 24495-1 on plain language
• ISO/IEC/IEEE 82079-1
• Google’s developer documentation style guide
• Carroll’s minimalism.

ASD-STE100 is the controlled language behind aerospace and defence maintenance manuals. One word carries one meaning. No synonym-swapping for variety. Active voice. One idea per sentence. No idiom, no metaphor, no figurative language of any kind.

“Earned its keep” breaks that last rule. The standard rejects it, so I no longer have to argue from feeling. This approach is what I’ve seen several people recommend and it does improve the output but it also drifts over time. The longer the conversation, the less Claude adheres to the standard. Context rot maybe?

Minimalism handles the other half. It says content that doesn’t help the reader do the next thing gets cut. This rule stops a behaviour I could not describe before: Claude explaining why its own finding mattered.

The challenge with these is that they are references and don’t contain the actual content in the prompt; so it won’t follow the rules 100% of the time and clearly only if they’ve been trained on the material.

The two standards catch different failures

I kept conflating the two standards.

ASD-STE100 is about wording. It caught “earned its keep”, and it caught other figures of speech too. Claude had described a type file as “the seam between two incompatible runtimes”. Under the standard that becomes “the type file has no imports, so the importer cannot pull in sharp”, which is the actual mechanism and is far easier to understand (at least for me).

Minimalism is about scope. A sentence can be perfectly plain and still be a lecture. “A cross-check between two independent paths finds errors on either side” is plain English. It’s also Claude telling me what to learn from my own codebase.

The rules load at the start of every session and remind after every turn to avoid context rot

When the guidance is loaded at the start of a conversation it is adhered to reasonably well (in my anecdotal testing). Once that conversation hits ~150k tokens it starts to degrade. Compacting does not seem to affect it. I have seen examples where there are violations in early chats too, so there may be a conflict with system prompts.

Claude suggested a per-turn reminder that costs 300 tokens. This constraint is what decided the artefact needed to be a plugin. To save users modifying their own settings.json, plugins ship with hooks.

The skill’s own prose failed its own checks

I caught Claude breaking conversation-prose in its own replies to me, one violation at a time: “Worth stating plainly,” “The mechanical cause is narrower,” “worth naming,” “The failure was mine,” “which is exactly why this reached a paid run before showing itself,” “share one shape,” “and the construction moved.” Each catch prompted an amendment to the skill.

I realised that the skill file itself broke about twenty-five of its own rules, six of them the exact figurative use of “shape”. It’s like telling someone not to think about bananas; really hard not to after someone has mentioned it. I am treating Claude the same way.

I had Claude rewrite the skill to obey itself, and the check count grew from fourteen to sixteen in the process. Then I created a workflow loop to test and iterate until there were no more violations in the output. This also included an adversarial agent ‘red-teaming’ to check against the main agent’s findings in any given output.

Blinding the judge

After a few runs and inconsistent results I asked Claude what the most rigorous method would actually be to test these skills.

Claude found the largest source of bias was the judging prompt labelled the two texts BEGIN BEFORE and BEGIN AFTER, which told the judge which text the skill had produced before it was ever asked whether that text was better. Leading the witness.

I had Claude make two changes:

Blinding: the texts now arrive as TEXT A and TEXT B. Which slot the ‘after’ text takes is also randomised with (pair index + round) % 2. Reproducible, with no stored seed, and exactly balanced across a run.

A control: each ‘before’ text is also judged against a copy of itself so hopefully both (identical) texts pass and fail the same checks in the same ways to ensure no bias before the real comparisons. If the identical texts score radically differently then something is more fundamentally wrong.

Blinding moved the number, slightly. On the same thirteen pairs, the before score rose six to nine points, and the measured improvement fell from 38 points to a range of 32 to 36. Position bias measured 1.0 point out of 96.

I can now change a rule in either skill and know, before I publish anything else, whether the edit reduced the failures it targets.

What the score doesn’t measure

The numbers at the top come from that same Opus 5 run, thirteen prompts, three blinded judging rounds, on 2026-08-15. The ‘before’ texts, the reference run, and the judged records are committed to the repo, so the figures are checkable without having to run the tests yourself. You’re welcome to of course, the instructions are there an that’s one reason why I used the API and not the chat.

The score says nothing about whether the writing is better by any outside standard. The judge has never been checked against a person marking the same checks by hand. Thirteen prompts give a rough signal. A real confidence interval needs many more than that. This is what I’m working on next but is a little more time consuming. I actually ran something similar to validate the AI Quality Score as Smith.ai.

Defining and measuring quality for an AI voice experienceYou can't move what you don't measure. A bad early call was the top predictor of cancellation; a visible quality score makes a bad week a recoverable dip, not a refund. How I defined what a good call was.Craig Dennis — Head of Product DesignCraig Dennis

The conversation skill still breaks its rules, and a fresh chat does not fix it

Live use is where conversation-prose still fails. I have not been able to get Claude Code’s conversational replies to correct their language. At first the failures looked like context rot, the degradation after 150k tokens described above. A new chat shows the same violations after a few turns, so conversation length is at most part of the cause.

When I ask Claude to evaluate its last turn, it finds multiple violations. It cannot apply those same checks in real time, while it is writing the reply. I don’t know why.

One next approach is a Stop hook that checks each reply before it reaches me and regenerates it if it contains violations. I turned down a blocking version of this earlier, over false positives. The regenerating version has a different cost. Every regeneration is a second model call, so latency and token costs could rise significantly. A reply that fails the check on every attempt would loop.

My conversations have improved

Even with that problem unsolved, the replies I get are better than they were. The ultimate goal of this exercise, for me, was to have better (less frustrating) conversations with Claude. And maybe save a few tokens once the self-aggrandising had been removed.

What followed was a deep dive into scoring tests to know whether the writing was demonstrably better than before. I’ve seen so many people complain that I thought I would share. Not only the research I conducted but an actual artefact that you can use today to get the same results I have.

Try the skills for yourself.

design-skills/plugins/writing at main · craigmdennis/design-skillsInstallable Claude Code skills from Visible By Design to help every designer get the recognition they deserve inside organizations - craigmdennis/design-skillsGitHub