Agent Grader

The rubric

10 criteria

The ten questions

Every one of these is a question a builder should want answered about their own system — and every one is a second-user question: the walls it probes appear the day someone besides the builder needs the system to work for them. Nothing here is hidden from the grading; read it all first if you like.

  1. 01 # Talk-back & adaptation

    Can users reply to an agent, tell it how they like things done, and have it actually adapt?

    • 0 ptReplies go nowhere. People answer the bot anyway.
    • 1 ptReplies continue a conversation, but nothing durably changes.
    • 2 ptThey ask me, and I edit the prompt files.
    • 3 ptAn instruction durably changes that person's agent from the next run on.

    # why it matters

    This is what people expect from anything calling itself an agent in 2026, from their own daily usage. People dismiss AI that doesn't listen to them, and non-adoption follows no matter how good the prompts are — users read a system whose replies go nowhere as email from a robot, and they stop reading it. More than any other question, this one decides adoption, because it is the ownership question.

    # what good looks like

    A reply to a skill run is a new prompt against full context. An instruction — "shorter," "lead with the dollar figure," "don't include the demo pipeline in my morning briefing" — updates the skill or agent definition, the agent confirms what changed, and every subsequent run reflects it, with no settings panel, no ticket, no developer. Something a person can coach and name is something they own, and ownership converts a tool people tolerate into a system people feed.

  2. 02 # Authorship

    Who writes the skill prompts?

    • 0 ptI do. Every prompt, hand-written from my own head.
    • 1 ptThe team drafts with a chatbot's help, but from generic context.
    • 2 ptA model drafts them from some company docs I feed it.
    • 3 ptA model writes them from our full history and strategy, per person, previewed before deploy.

    # why it matters

    The pride in hand-authorship is earned, and the hand-authorship is also the bottleneck: every new skill costs the one person who can write them, which is why DIY systems plateau around a dozen skills. Ask yourself how the coaching prompt differs between your best rep and your worst — with one prompt for everyone, it doesn't. A builder writing from their own head is competing against a system that reads three years of the company's actual conversations first.

    # what good looks like

    Describe the initiative, and a model drafts every agent identity and skill prompt grounded in the actual workspace — the pipeline, the methodology, each rep's deals and call history — previewed before deploy, at a per-person specificity no hand-maintained prompt library reaches. The folder you already built doesn't get thrown away; it becomes the authoring environment, version-controlled, with production as the deployment target.

  3. 03 # Governance

    Can you choose which skills users can modify and which must run one way?

    • 0 ptThe concept doesn't exist. One prompt file, the same for everyone.
    • 1 ptEverything is locked. Only I can change anything.
    • 1 ptEverything is open. Anyone can change anything.
    • 3 ptPer-skill modes — managed, template, personal — that I set and can change over time.

    # why it matters

    Salesforce writeback, forecast methodology, and anything compliance-adjacent must run exactly one way; coaching and briefings must be personal. Lock everything and you lose adoption; open everything and you lose control — both are failure modes, and a one-mode system gets total governance by accident while presenting it as policy. A control plane that cannot express this distinction is not a control plane.

    # what good looks like

    Every skill carries a deployment mode in the data model itself: managed (versioned, synced, admin-only — update the source and every copy updates), template (deployed as a starting point each user forks), or personal (owned entirely by the rep and their agent). The admin sets the dial per skill and moves it over time, with workspace-level instructions above it all that no individual can override.

  4. 04 # Harvest

    When one person's agent gets great, can everyone benefit?

    • 0 ptEveryone runs the same skills. There's no variation to harvest.
    • 1 ptI'd have to go read everyone's setups by hand.
    • 2 ptI occasionally copy a good idea into the shared prompts.
    • 3 ptI can map over every variant, spot the winners, and roll them out fleet-wide.

    # why it matters

    This is the water-cooler moment: one rep's emails are suddenly great because they coached their agent into a step-function improvement. In a same-skills-for-everyone rig, that improvement is trapped with that rep and the system's ceiling is its median user — you optimized for manageability by giving up the discovery mechanism. With harvest, anyone's local maximum becomes the team's new floor.

    # what good looks like

    Personalized drift is R&D, and the control plane is what lets you harvest it. An admin maps over every personalized variant, sees what the best performers changed, and rolls it out fleet-wide as a managed or suggested update — while respecting everyone's personal layer. That harvesting is the day-to-day work of running a fleet, not an archaeology project.

  5. 05 # Measured outcomes

    How much time have you given your team back? Are their calendars different than six months ago?

    • 0 ptNothing measured. Reps say they like it, and I have some Slack screenshots.
    • 1 ptMy own job changed completely. Nobody else's has.
    • 2 ptI can point to a few measured wins for specific people or teams.
    • 3 ptOutcomes are instrumented across the team — time, engagement, behavior change.

    # why it matters

    A system with no measured outcome is a project, and projects get cut. It also reveals whether you have eval at all: if you can't measure the time saved or the behavior changed, you can't measure quality drift either — and the most common DIY outcome is that the builder's job transformed while everyone else's stayed the same.

    # what good looks like

    Engagement and outcome are instrumented natively — every skill run, read, reply, and action taken — with a visible artifact behind them: calendar blocks for CRM data entry go to zero, and mornings start with reviewed pipeline instead of tab-loading. The measurement layer is what lets you claim the result credibly instead of anecdotally.

  6. 06 # Compounding

    Does using your skills make them better?

    • 0 ptThe prompts are byte-identical to day one, except where I've edited them.
    • 1 ptIt gets better when the model gets better. The system itself doesn't improve.
    • 2 ptI periodically revise the prompts by hand based on what I hear.
    • 3 ptUsage itself improves it: memory compounds, replies tune skills, winning variants spread.

    # why it matters

    This is the compounding question, and compounding is the entire economic argument for the agent-native model. A flat system is a cost; a compounding one is an asset that widens the gap every day. If a thousand runs from now the prompts are byte-identical, what you built is a scheduler with good prompts.

    # what good looks like

    Three mechanisms stack. Every interaction enriches shared memory, so memory compounds. Every user reply tunes that person's skills, so personalization compounds. And eval over the resulting drift harvests the winners for everyone, so the organization compounds.

  7. 07 # Eval

    How do you know what's working and what isn't?

    • 0 ptI don't, really.
    • 1 ptI ask people. They tell me in Slack.
    • 2 ptThe bot self-reports — a weekly email of what it couldn't answer.
    • 3 ptRun-level history plus engagement per skill, per agent, per user, all queryable.

    # why it matters

    Without run-level and engagement-level introspection, skill quality degrades silently as the business moves, and you find out about drift from the damage. Asking people in Slack is anecdote collection routed through the one person who is already the bottleneck — and RevOps, of all functions, should refuse to run on beliefs.

    # what good looks like

    Every run, thread, and notification is inspectable — output, delivery, and whether the human read it, clicked it, or replied — per skill, per agent, per user, across the whole fleet. "Which of my deployed skills got ignored last week, and why" is a five-minute question answered in natural language, and run history doubles as an eval corpus a scheduled job can score against a rubric.

  8. 08 # Feedback visibility

    Can you see how user feedback is changing the skills?

    • 0 ptThere's no per-user adaptation, so there's nothing to see.
    • 1 ptFeedback disappears into files. I have no view over it.
    • 2 ptI could reconstruct it from git if I had to.
    • 3 ptSkills are versioned, diffable objects. The feedback is itself a dataset.

    # why it matters

    Feedback you can't observe is feedback you can't learn from. Even a builder who hacks per-user prompt patches has no view over the patches — the feedback disappears into files — and the loop isn't closed until someone, or something, can read all of it.

    # what good looks like

    Skill definitions and agent identities are readable, versioned objects. An admin can diff any skill over time, snapshot fleet state before a deploy, and see exactly how each person's feedback reshaped their variant. The feedback layer is a dataset you can query, not sediment in a repo.

  9. 09 # Loops

    Does the system keep getting better if you leave for two months?

    • 0 ptSkills run on schedule. No run changes the next run.
    • 1 ptSome runs write state somewhere, but nothing reads it back.
    • 2 ptA loop or two exists, and I maintain it by hand.
    • 3 ptDurable state, chained skills, and eval — it keeps improving whether I'm here or not.

    # why it matters

    GTM reality is non-stationary: messaging that worked in Q1 dies in Q3, and a static system coaches this quarter's reps against last quarter's playbook. A skill that runs nightly is a repeated action, not a loop — a loop requires the run to change state that changes the next run. For most DIY rigs, the improvement rate is exactly the free hours of one talented person, and it stops the day that person stops tending it.

    # what good looks like

    Durable state that agents read and write, skills that chain, and eval over the results make loops a first-class pattern. The flagship: one skill mines every call for what's actually working and updates a living rubric, a second coaches each rep against it targeted to their specific gaps, a third watches whether behavior changes and adjusts. Nobody manually updates the doc, and the human's role rises from maintainer to conductor — taste, standards, deciding what becomes policy.

  10. 10 # The keyboard test

    Do your salespeople still open Salesforce and punch data in with a keyboard?

    • 0 ptYes. Daily, for everything.
    • 1 ptSome writebacks exist. Reps still do most of the entry.
    • 2 ptAgents maintain most of the record. Humans review.
    • 3 ptData entry is effectively zero. Agents write under each user's own permissions.

    # why it matters

    This question exposes the ceiling of the whole approach. A DIY layer on top of the old system of record leaves the old human contract intact — people type structured data in so dashboards can come out — which means the bot's intelligence is capped by what tired humans typed into dropdowns. A perfect pipeline export, every dropdown filled, still has almost no signal in it.

    # what good looks like

    Agents maintain the record from what actually happened — calls, emails, commitments — at a granularity keyboard entry never reached, and where the old CRM must stay the system of record, they write to it under each user's own permissions instead of a god-mode service account. No amount of orchestration-layer cleverness fixes the substrate underneath it; the control plane and the memory layer are two halves of one system, and the DIY path has to build both.