The short answer
Make an AI agent more dependable by defining its expected behavior, testing exceptions, restricting its tools, reviewing consequential actions, and learning from production runs. A workflow demonstrator reports that a revised Monday brief flagged a shortage of one cleaner for the following Thursday.1 A trading-agent creator reports fixing a memory issue during an experiment involving $500 of real money over about two months.2
What founders did
7 real examples, each linked to the moment in the source. Results are self-reported.
- YouTube
Mitchell Troyanovsky
Described Basis's behavior spec for stating important intended agent behavior and checking relevant runs against it.
- YouTubeDemonstrated total: 6100
Mohammad Rameez Imdad
The demonstrated calculation returned 6100.
Demonstrated AI extracting order items and code calculating the total from product prices.
- YouTube
Sharbel A.
Described per-app permissions and kept a connected payment app read-only.
- YouTube
Cole Medin
Built a workflow dashboard that surfaces decisions requiring human attention.
- YouTube
AI Master
The following Monday's shorter brief flagged a shortage of one cleaner for the following Thursday.
Demonstrated revising a business brief to prioritize staffing warnings and omit routine email listings.
- YouTube
AI LABS
Demonstrated turning feature requirements into eval files and requiring review of those evals before building.
- YouTubeMemory issue required a fix
The Tech Girl
The creator reports that a memory issue required a fix and that the agent sometimes did nothing.
Described a trading-agent experiment with tool connections, memory, instructions, testing, and scheduling.
Your playbook
Use this when
Use this playbook when you have an AI workflow to deploy or improve and can inspect its inputs, outputs, and tool permissions.
You’ll leave with
Build an agent reliability worksheet containing a behavior spec, test cases, permission boundaries, approval points, and a run-review plan.
Specify the behavior before editing the prompt
- Do
In the reliability worksheet, identify the workflow's trigger, source records, required output, forbidden actions, and human handoff. Mark calculations or fixed rules that should run outside the model.
- Make
Add a behavior spec and an exact-operation boundary to the worksheet.
- Check
If a reviewer cannot judge a run against the spec, clarify it before changing the prompt.
Why this step is in the plan
Mitchell Troyanovsky says Basis developed a behavior spec to identify the important intended behaviors before writing agent context.3 Mohammad Rameez Imdad's order demo assigns AI the interpretation of a chat and leaves pricing from product data to code; the demonstrated calculation returns 6100.4
Test ordinary behavior and exceptions
- Do
Add an ordinary input, missing data, an unsupported request, and a tool failure to the worksheet. State the expected result or handoff for each, run the agent, and record what happened.
- Make
Add a case ledger with expected behavior, actual behavior, and pass or fail.
- Check
If a bad run passes, tighten the check; if a case fails, repair and rerun it before release.
Why this step is in the plan
Fence tools and external actions
- Do
List every tool and credential the workflow can use. Assign the narrowest access suitable for its tested behavior, block consequential actions it need not take, and name an approver for exceptions.
- Make
Add a permission map, blocked actions, and approval owners to the worksheet.
- Check
If a forbidden action can execute without approval, narrow access before proceeding.
Why this step is in the plan
Review before scheduling or release
- Do
Assign a reviewer to the test definitions and consequential outputs. Inspect varied manual runs, record edits and rejections, and decide which tested branch, if any, may be scheduled or released.
- Make
Add review ownership, approval points, and a branch-specific release decision.
- Check
If behavior still surprises the reviewer, the process is unsettled, or a gate is bypassable, keep the workflow under manual review.
Why this step is in the plan
Log runs and repair what fails
- Do
For released runs, retain the input reference, agent decision, tool action, approval outcome, and error. Inspect exceptions and rejected outputs; add reproducible failures to the case ledger before changing instructions or widening permissions.
- Make
Add a run-review log, regression cases, and the next permission decision.
- Check
If a failure cannot be traced and replayed, improve the record and hold the current permission boundary.
Why this step is in the plan
A release path for an AI agent
- Before you allow
Expected behavior is unclear
Add this controlSpecify the input, output, exact operations, and handoff
Continue only ifA reviewer can distinguish an acceptable run from an unacceptable one.
- Before you allow
The ordinary case works
Add this controlTest missing data, unsupported requests, and tool failures
Continue only ifEach failure produces an identifiable repair or a defined handoff.
- Before you allow
The agent can affect customers, money, or data
Add this controlRestrict permissions and gate consequential actions
Continue only ifA blocked action cannot execute without its assigned approval.
- Before you allow
The workflow is ready for release review
Add this controlInspect varied manual runs, then log released runs
Continue only ifReviewers understand its behavior, the process is settled, and exceptions remain traceable.
Release boundary
Keep consequential actions behind review unless the relevant tests pass and the approval gate works.
Expansion boundary
Expand only a tested branch or permission; keep other boundaries in place.
Make it yoursAgent reliability worksheet
A fill-in template for your own test
Workflow and trigger: [task and event] Expected behavior: [observable result] Exact operation: [code or fixed rule] Ordinary case: [input and expected result] Exception: [input and expected handoff] Forbidden action: [action and permission block] Approval owner: [person and decision] Run record: [input reference, decision, tool action, review outcome]
When this works
- Use this release path when outputs can be inspected and consequential actions can be blocked or sent to a reviewer.
When to avoid
- Do not grant unattended write access when you cannot test relevant failures, trace actions, or enforce a handoff.
Related questions
How do I tell whether my agent calls the right skill?
IBM Technology explains that a skill's description should say both what it does and when the agent should use it; a description that merely says it generates reports is too vague.9 Add cases where the skill should run and cases where it should not.
What should I measure besides task completion?
Greg Isenberg warns that an agent optimizing for passing tests might rewrite them, or close support tickets without solving them.10 Pair completion with a check for the harmful shortcut it could reward.
Can I compare agents with the same test cases?
Lenny Rachitsky's newsletter reports that unequal connectors prevented an agent in a personal-agent comparison from completing several work tasks.11 Record each agent's available tools and permissions beside its results before comparing them.
