Docs
The concepts behind Pencil Harness.
Full documentation arrives with early access. Until then, this page explains the ideas the product is built on, so you know what you would be adopting. There is no API reference yet, and we would rather leave it out than invent one.
Decision ladder
Every judgement in Pencil climbs four rungs: a deterministic rule, the decision model, your big model, and you. The agent stops at the first rung that is sure. Rules are cheapest and run first. The decision model answers in one pass with no generated words. Your model is called when the decision model is unsure or the question is open-ended. You are the last rung, and some questions always end there. Every question may answer “unclear”, and unclear is treated as a reason to go up, never as permission to guess.
Decision types
A decision is a closed question: yes or no, pick one of N, or a score on a scale. Pencil groups them into families. Routing asks what kind of task this is and how hard. Loop control asks whether the work is done, should continue, is stuck or needs you. Tool checks ask whether a tool choice fits and whether an edit is aimed at the right place. Other families cover edits, grounding, safety, context and errors. Questions about the same text are asked together, in one request.
Calibration and thresholds
A probability only helps if it matches reality. We calibrate each decision type on labelled examples, so that an answer given 90% is right about nine times in ten. Each answer then gets its own threshold. Where a mistake would cause harm, such as approving a risky command, the threshold is stricter. An answer under its threshold does not settle. It moves up the ladder. Thresholds and calibration will be visible and adjustable in the product, within the limits of the safety floor.
Shadow to live
A decision point does not start in charge. It begins off. Then it runs in shadow, answering beside your existing logic while changing nothing, so you can compare. It goes live only when the evidence supports it. After that, its answers keep being checked against outcomes, and if they drift it is demoted back to shadow automatically. Each decision is recorded with its probabilities, threshold, deciding rung and model version, so any run can be replayed.
Bring your own model
Pencil uses one OpenAI-compatible client, so any model behind a compatible endpoint can write for it, open or closed, hosted or local. Per-model quirks, such as tool-call formats, sit in a config file for that model. Before a model is used, Pencil runs 22 checks against the endpoint and tells you which ones fail. The harness is designed for context windows as small as 16K, which keeps small and local models practical.
Waitlist
Documentation opens with early access. or write to hello@pencilton.com.