Governance

Can humans observe, control, and evolve agent behavior safely?

As agents gain autonomy, the need for human oversight increases rather than decreases. Governance covers the mechanisms that let humans control what agents can do, observe what they have done, and evolve the system safely over time. This is the dimension that makes all the others safe to use.

Key principles

  • Support scoped authentication and agent-specific identities
  • Provide approval workflows, dry runs, and previews for consequential actions
  • Enable traceability from prompt to side effect
  • Implement cost controls, rate limits, and circuit breakers
  • Signal versioning, deprecation, and migration paths machines can detect

Safety and policy

Agents need their own identity and access controls, separate from the humans who deploy them.

Authentication and authorization

  • Scoped authentication - grant agents exactly the permissions they need and no more
  • Agent-specific API keys or tokens - audit, rate-limit, and revoke agent access independently
  • Agent identities - distinguish agent traffic from human traffic in your systems

Abuse protections

  • Rate limits calibrated for agent traffic - automated access patterns look different from human usage
  • Usage quotas - constrain total consumption
  • Anomaly detection - flag unusual patterns in agent behavior

Human observability

Humans need to see what agents are doing. When an agent makes a change, it should be possible to trace that change back through the tool call, the agent's reasoning, and the original prompt or trigger.

Traceability

  • Correlation IDs - link related requests across systems
  • Request logging - record what agents did and when
  • Audit trails - reconstruct exactly what happened when something goes wrong

Human-in-the-loop workflows

  • Dry runs - let agents preview what an operation would do without executing it
  • Approval workflows - queue consequential actions for human review before execution
  • Previews - show the expected outcome of an operation before committing

These patterns let teams grant agents broad capabilities while maintaining control over irreversible operations.

Resource control

Autonomous agents can consume resources quickly if unconstrained.

  • Cost controls and budgeting - let operators set spending limits
  • Per-identity rate limits - configurable per agent, not just per API key
  • Budget ceilings - prevent runaway costs from agent loops or errors
  • Circuit breakers - when an agent enters a failure loop, help it stop rather than continue burning resources

Provide clear signals when rate limits are approached or exceeded, including guidance on when the agent can retry.

Change management

APIs evolve, and agents need to evolve with them. The more you can signal about changes, the more smoothly agents can transition across API versions.

  • Versioning strategies - explicit and machine-readable
  • Deprecation signals - agents and their operators can detect upcoming changes
  • Sunset headers - standard mechanism for communicating end-of-life timelines
  • Migration guidance - help agents adapt before breakage occurs
  • Machine-readable changelogs - let agents assess whether an update affects their operations
  • Compatibility guarantees - explicit contracts about what will and will not change