pencil ai

pencil harness

Your model writes.
Pencil decides when it’s done

Pencil Harness is the world’s first agent harness built around a System‑1 decision model.[4] It answers the quick judgements, so your model only has to write.

Read the essay

Works with Claude, GPT, Gemini, Qwen, Llama, DeepSeek and Mistral, or any endpoint that speaks the OpenAI API.

Fast and slow

Most of an agent’s judgements are quick ones.

Pull your hand off a hot pan. You did not think about it. A reflex answered, and you moved. Now work out how to split a restaurant bill four ways with a tip. That takes careful thought, and you would not want a reflex doing it.

People use both and switch without noticing.[1] Agents mostly have one. In many agent loops, every judgement goes to the same large model that writes the code. Is the task finished? Is this the right tool? Should we stop? Each answer is generated word by word, and each costs time and tokens.

Pencil gives the agent a reflex. A small decision model answers those questions in one quick pass and returns a probability for each possible answer. Your model is called when there is something to write, or when the reflex says it is not sure.

  • Your model, generating
  • Decision model, one pass
  • Tool run
Same task, two harnesses. Upper line in each row: a typical harness. Lower line: Pencil. Illustrative timing, not a measurement. The point is where the time goes: in a typical harness every judgement is a long generated call; with Pencil judgements are instant and only writing takes time.
Text version of this figure

Two lanes run the same task, fixing a failing date test, in six steps. In the first lane, a typical harness, the model generates a decision at each of four judgement steps, and each one takes a long time. In the second lane, Pencil, the decision model answers those four steps at once, and only the writing step takes noticeable time. The Pencil lane finishes first. The timing is illustrative.

What gets decided

A decision is a question with a short list of answers.

Each card shows one question, the situation it was asked about, and how the decision model answered. Every question can answer “unclear”, and a low-confidence answer is sent up instead of acted on.

Question

What kind of task is this?

Situation

You write: “The date picker shows yesterday for users in Tokyo. Can you look?”

Answers

Fix a bug 86%
Explain code 7%
Add a feature 5%
Unclear 2%

Settled Pencil takes the bug-fix path.

Question

How hard is this?

Situation

You write: “Rename getUser to fetchUser everywhere.”

Answers

Easy 91%
Medium 7%
Hard 1%
Unclear 1%

Settled A short plan, no extra deliberation.

Question

What kind of edit is this?

Situation

A document and the request: “Make the second paragraph shorter and drop the jargon.”

Answers

Rewrite 78%
Proofread 12%
New section 4%
Unclear 6%

Sent to the model The leading answer is below the bar for this question.

Illustrative. Situations and probabilities are examples we wrote to show the mechanism. The dashed line on each card is the bar an answer must clear to settle.

How Pencil works

Every judgement climbs a ladder and stops at the first rung that is sure.

The rungs, bottom to top: a plain rule, the decision model, your big model, and you. Cheap and certain comes first. Careful and expensive comes last. If any rung is unsure, the answer goes up, never sideways.

1

A request arrives: your model has just edited a file and the loop needs to know whether the task is finished.

Left and right arrows step through. P pauses.
One decision, followed from request to answer. Illustrative.
Text version of this figure

Routine decision: A request asks whether the task is finished. A rule checks that tests pass. The decision model answers done with high probability, which clears the bar for that answer. Your model is not called. The loop stops and the decision is recorded. The numbers are illustrative.

Escalation: A repeated failure triggers a rule, which cannot decide. The decision model’s leading answer is below its bar, so the question goes to your model with the failed attempts. If that is also unsure, the question goes to you. The numbers are illustrative.

Earning trust: A decision point goes through five states. It starts off, then runs in shadow next to existing logic, then goes live once the evidence supports it. Live decisions keep being checked, and a decision point that drifts is demoted back to shadow automatically.

Calibrated, not confident

A probability is useful only if it is right as often as it says.

A forecast that says 70% chance of rain should be followed by rain on about seven days in ten. Models are often sure of themselves in a way they have not earned. We do not take a model’s word for its own confidence.

Each kind of decision is calibrated on labelled examples, so that “90%” means about nine in ten. Each answer then gets its own bar. An answer whose mistake would do harm, such as approving a risky command, gets a stricter bar than one whose mistake costs a retry.

Move the bar

Settled by System 1: 0% Sent up the ladder: 0% Wrong answers let through: 0
85%

Drag, or use the arrow keys. Decisions above the bar settle on their own. The rest go up the ladder.

  • Right
  • Wrong

Raise the bar and fewer mistakes get through, but more questions cost a model call. Lower it and the reverse. The right bar depends on what a mistake costs.

Illustrative data. 100 made-up decisions, not a measurement.

Built for code and documents

The two kinds of work where a wrong edit is expensive.

  • Exact-once edits

    An edit applies only when the text it targets appears exactly once.

  • No stale writes

    If the file changed after the model read it, the write is refused.

  • Collateral check

    We hash the file before and after, so unrelated changes are caught.

  • Undo journal

    Every change is journaled, so you can roll back any step.

  • Checks before opinions

    Tests and linters run before any model is asked for a view.

  • Edits as suggestions

    Changes can arrive as suggestions you accept, instead of already made.

Safe by construction

The limits live in code, where a clever prompt cannot talk them away.

  • A safety floor

    Some rules sit below every setting, and no setting can lower them.

  • Rule of two

    Untrusted input plus private data never causes an outside effect without a person.

  • Text is data

    Web pages and files are read as text, never followed as instructions.

  • Unclear means careful

    When an answer is unclear, Pencil takes the safer side.

  • Stricter bars for harm

    Answers whose mistakes would do damage need more confidence to settle.

  • Judges cannot clear failures

    A model judge can add a failure to a check but never remove one.

  • Evidence you can inspect

    Every decision is stored with enough detail to replay it.

  • Held-out evaluation

    We score the harness on tasks we never tune against.

Any model, open or closed

Pencil talks to your model through one standard client.

Pencil speaks the OpenAI API, so any model behind a compatible endpoint can write for it. Differences between models, such as how they format tool calls, live in a small config file per model. Before a model is used, we run 22 checks against its endpoint. If one fails, you see which and why.

Pencil is built for small context windows, 16K and up, so small and local models work well too.

Model familyTypeWhere it runs
ClaudeClosedHosted
GPTClosedHosted
GeminiClosedHosted
QwenOpenHosted or local
LlamaOpenHosted or local
DeepSeekOpenHosted or local
MistralOpen and closedHosted or local

Any of these connects through an OpenAI-compatible endpoint. The tested list, with the model versions and endpoint results, is published with the beta.

How we’ll measure it

We have no results to show yet. Here is what we will measure and against what.

We are pre-product, so we have no benchmark results of our own and we will not borrow someone else’s to look better. What we can do is say now how we will judge ourselves. Each row below is a number we will report, measured against the same model running in a standard harness. The method comes first: the tasks, how they are scored, which model versions ran, and how you can run them yourself.

What we measureAgainstResults
Decisions settled without the big modelSame model, standard harnessPublishing with the beta
Errors let throughSame model, standard harnessPublishing with the beta
Latency per decisionSame model, standard harnessPublishing with the beta
Cost per taskSame model, standard harnessPublishing with the beta

A plan, not results. No figure in this table has been measured yet.

One thing others have measured

In October 2026, Artificial Analysis introduced a Decision Index. On classification and tool-calling benchmarks (WildGuardTest and When2Call), leading decision models scored comparably to or above frontier language models run at low reasoning effort. Calibration is still imperfect overall, and best on When2Call.[2] We cite it as evidence about the category. It says nothing about Pencil’s own numbers.

Get early access

We are letting teams in a few at a time.

Tell us what you build and which models you run. We will write back when there is a place for you.

Email us

Request early access

Four short fields. We read every request.

Optional: which models you run today.

We use your details only to reply about early access. See the Privacy notice.

FAQ

A small, fast model that does not write. You give it text and a list of questions, and it returns a probability for each answer in one pass. The name comes from Kahneman’s fast, automatic mode of thought.[1]

No. Your model still writes the code, the edits and the drafts. Pencil answers the yes/no and pick-one questions around the writing, and calls your model when it is unsure.

Any model behind an OpenAI-compatible endpoint, open or closed, hosted or local. The table above lists the families we plan to support first. We run 22 checks on an endpoint before using it.

It says so. Every question can answer “unclear”, and an answer below its bar goes up the ladder to your model, and then to you. Unsure never turns into a quiet guess.

We calibrate each kind of decision on labelled examples and check that “90%” is right about nine times in ten. A new decision point runs in shadow first and goes live only when the evidence supports it. If it drifts, it is demoted automatically.

Not yet. We are pre-launch and letting teams in a few at a time. Request early access and we will write when there is room.

Notes

  1. Daniel Kahneman, Thinking, Fast and Slow, Farrar, Straus and Giroux, 2011. https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow
  2. Artificial Analysis, “Artificial Analysis Decision Index”, 13 October 2026. https://artificialanalysis.ai/articles/artificial-analysis-decision-index
  3. Lindenbauer et al., “The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management”, arXiv 2508.21433. https://arxiv.org/abs/2508.21433
  4. “World’s first”: among the agent harnesses we studied in October 2026 (OpenCode, Pi, Hermes Agent, DeepSeek Harness and Codex CLI), some offer typed decision calls, but none runs a calibrated decision model inside the loop by default. Our own survey.