“Oh You’re Right”

Is Not A Review: AI That Proves What It Checked

Calvin Hendryx-Parker, CTO

Whit Morriss, Sr. Architect

Six Feet Up

LinkedIn Live

What is this about, who is it for

Where we are going (today)

The root of the problem

Creating determinism

with stochastic systems.

quality input
constraint
dialectic
visibility

Operational work, technical or more business orient is almost alway non-deterministic.
LLMs are also non-deterministic
How can we take problems that we know do not have deterministic solutions and create meaningful value with non-deterministic tools?

If you work in risk & compliance, your job is often wading through ream of uncertainty to try to determine the best possible decision. An LLM may not provide perfect answer, but it can help you pinpoint issues and opportunities faster.

If we can narrow the frame and focus of the LLM, it can uncover important information faster, giving us the opportunity to take more time to dig deeper or respond or maybe just move on.

Anything that can remove complexity and uncertainty safely provides value. There is something similar with using LLMs, which while they can manage large of amount of data quickly, they inherently are non-deterministic.

The problem

Claude, a new document and me makes 3.

Let’s see this in action…

Tragedy of The Commons Context

Context is a Workspace, not a Warehouse.

Agents

“Agents are models using tools in a loop”
–Hannah Moran, Anthropic

The Tool For Job

a context buffer w/ LLM connection
local tools: skills, subagents, mcp, hooks
local resources: flat files, dbs, scripts, sockets, etc

What’s in the box?

Memory & Skills

From search to assessment

Let’s Orchestrate

Bonus: Auditing

How do we talk to the sqllite and chroma? How do we load our data into them rather than just having it swished into the context?

Introduce the concept of skills and why they are so good for quickly prototyping tasks and processes.

they start super simple as Markdown documents
scale up to complex systems

Orchestration is a big word.

What does it mean Wikipedia has 3 definitions for “Orchestration”: for music, for computers and for games. In all three cases, you could make the argument orchestration is an act of using rules to coordinate and manage a group of actors to achieve something meaningful.

For our purposes, orchestration is organizing more than one actor (meat agent or digital) to do something useful.

Search is great, but I have reviewed many documents like this, I have an idea about what I care about and would like to get some automated recon before I have to dig into the doc myself. I also want to delegate as much of this digging to my agents.

I want the LLM to give me feedback, but I want to be sure that feedback is useful, focused, and actionable.

To do this, we are going to use an idea from AI testing, the eval. When testing LLMs, we assume that our responses are nondeterministic, so we must create a sort of bracketing of test questions and an assessment of answers.

We will use an eval skill we have created where we give our skill a criterion file, the skill uses our memory in the dbs, and then returns answers.

The questions constrain and focus the agent’s return; our databases constrain the data acted upon.

Every action is logged: what was loaded, what was searched, what was evaluated, by whom
In regulated industries this is compliance. For everyone else, it’s trust
Engineering parallel: this is your CI log — you can trace any finding back to the evidence
The audit trail is append-only. Agents can’t cover their tracks

see: demo

Your Future (could be) Agentic

Talk about repo and playbook, how to install with claude code or cowork.

Talk about how these tools are cheap and easy to make with LLMs. Built for purpose software is now a reality. You can create skill and agents and arrange them to get real work done quickly w/ some care and patience.

Increasing determinism is the key, by constraint, good inputs, and deploying the brownian ratchet of iterative critique and improvement. These process apply to full software development as well. Testing, CI/CD, specification building, PR workflows, etc are just more formalized ways to create more determinism.

We’ve talked about how you can easily get started with agents, but in the greater world, agentic orchestration is being used to emulate whole software orgs (ala gastown) or opensource ecosystems (ala wasteland) or communities of agents like moltbook. Understanding how to use agents locally will help you deal with the every growing landscape of agent-driven compute we are increasingly living in.

Closing remarks – - thank you for coming on this speedy ride through building an agent workflow - reinforce takeaways - don’t trust context, manage the data you want the agent to use explicitly - use skills and agents to increase determinism so you get good results from the LLM - use skills and agents to capture your workflows so you can automate more of your toil