Harness & AgentOps
Guardrails, routing, evaluation, observability and hand-over, released like software.
- Channels and identity
- Gateway and routing
- Guardrails and policy
- Knowledge and tools
- Evaluation
- Observability and cost
- Hand-over to people
The harness around every agent
The model is one layer. We design, build and run the others from the first sprint.
- 01Channels and identityChat, portal, ITSM and API entry points; every user and agent signs in.KeycloakSSOper-role access
- 02Gateway and routingOne LLM gateway routes each task, applies budgets and logs every call.LiteLLMKongbudgets per agent
- 03Guardrails and policyInput and output checks; a policy decision on every tool call.Guardrails AIOPApolicy as code
- 04Knowledge and toolsHybrid retrieval over governed data; tools behind an MCP gateway.QdrantNeo4jMCP gateway
- 05EvaluationFixed test sets and regression checks decide whether a version ships.test setsregressionfail-closed
- 06Observability and costTraces per step, service levels per agent, token cost per team.OpenTelemetryPrometheusGrafana
- 07Hand-over to peopleLow confidence or an approval step: a person takes over with full context.approval stepscontext-preserving hand-over
AgentOps: deploy, evaluate, monitor, improve
An agent changes whenever a prompt, tool, model or document changes. We release it like software.
Build
Prompts, tools, policies and agent flows versioned in Git.
Tools agents can call, through MCP
Systems exposed as MCP tools behind one gateway: every call has an identity, a policy check and a log line.
Controls in the harness
Each control has a place in the harness and a failure it prevents.
| Control | What it prevents | Harness layer |
|---|---|---|
| LLM gateway with routing (LiteLLM) | One model for every task; spend nobody sees | Gateway and routing |
| Input and output guardrails (Guardrails AI) | Unsafe or off-policy answers reaching users | Guardrails and policy |
| Policy on tool calls (OPA) | Agents calling tools they should not | Guardrails and policy |
| MCP gateway with single sign-on (Keycloak) | Tool calls without identity or audit | Knowledge and tools |
| Relevance check before answering | Confident answers to out-of-scope questions | Knowledge and tools |
| Fail-closed gate and human sign-off | Untested versions reaching production | Evaluation |
Agents draft, route, query and fix. Engineers set the policies, approve releases and own the outcome.
Who does what in an agent release
People decide, tools enforce, agents take the repetitive work.
- Pick the use case and its measure
- Decide which actions need approval
- Policies written as code
- Review prompts, tools and code
- Everything versioned in Git
- Secrets and identity wired in
- Coding assistants draft code and tests
- Read failures, decide fixes
- Test sets on every change
- Fail-closed gate
- The agent answers the fixed set
- Approve the release
- Deploy with a way back
- Take hand-overs
- Approve sensitive actions
- Traces, service levels, cost per agent
- Alerts
- Handle routine requests and signals
- Turn incidents into rules and knowledge
- New cases added to the test set
- Propose fixes for review
Questions this raises
The questions this part raises, and their answers.
All model calls go through one gateway that records tokens per agent and team, with budgets and alerts. Routine steps go to cheaper models.
Cost sits on the same dashboard as quality and latency.