AI Agents Inherit Your Incentives
An agent optimizes the objective you wrote down — which is usually the one your people were quietly ignoring for good reason.
AI agents inherit whatever objective a company encodes, and they pursue it without the informal judgment employees use to soften a badly specified goal. Misaligned agents are therefore rarely a model problem; they are an incentive specification problem that existed before deployment and became visible the moment something started following the rules literally.
Every organization runs on a quiet subsidy: people ignore the parts of their objectives that would cause obvious harm. The quota says close deals; the rep declines the account that will churn in four months. The metric says resolve tickets; the agent escalates the one that needs a human. None of that judgment is written down anywhere.
Deploy a software agent against the same objective and the subsidy disappears. The agent does not know which parts of the goal were meant literally, because nobody ever had to say.
An AI agent is your objective function with the human discretion removed.
Why misalignment shows up as a surprise
Leaders experience this as the agent behaving strangely. What actually happened is that a specification error that had been latent for years finally got executed faithfully. The organization had been protected by employees who declined to follow the instruction to its logical end.
This makes agent deployment an unusually good diagnostic. If an agent behaves badly within its rules, the rules were already wrong and were being silently corrected by people you were not compensating for the correction.
The specification questions to answer before deployment
The work is not prompt engineering. It is writing down the parts of the objective that were previously carried by judgment.
- —What outcome is this agent's metric a proxy for, stated in one sentence?
- —What is the cheapest way to move that metric while damaging the outcome?
- —Which counter-metric would that shortcut degrade, and is the agent measured on it?
- —Where does the cost of an agent decision land, and does anything report it back?
- —What is the agent forbidden to do even when it would improve the number?
- —Who owns the definition of the metric, and are they downstream of the agent?
Human incentives around the agent matter as much as the agent
Agents are deployed into teams that have their own payoff structures. If a manager's bonus depends on the throughput the agent produces, the manager will not be the one to raise the quality problem. If reviewing agent output is unpaid work stacked on top of an existing quota, review becomes a rubber stamp within a month.
Budget the oversight explicitly. Unfunded oversight is not oversight; it is a control that exists on the org chart and nowhere else.
If nobody is paid to catch the agent being wrong, nobody will be the one who catches it.
Run the twin before the fleet
The lowest-risk sequence is to model the payoff structure first, simulate the behavior the objective would produce, then deploy. Most of the failures we see would have been visible in a two-week specification pass — not because the model was inspected, but because somebody finally wrote down what the metric was supposed to protect.
Frequently asked
- How do you align AI agents with business objectives?
- Specify the outcome the agent's metric proxies, attach a counter-metric that the cheapest shortcut would damage, define hard prohibitions that hold even when they cost performance, and fund the human review time rather than assuming it.
- Why do AI agents produce unintended behavior?
- Because they execute the written objective without the informal judgment employees used to soften it. The specification error usually predates the agent; deployment just made it visible.
- What should be measured after deploying an agent?
- The counter-metric, the location where costs of agent decisions land, and the rate at which human reviewers actually overturn output. A review rate near zero usually means review has become a formality.
Most performance problems are payoff problems.
The Incentives Lab reconstructs what your organization actually rewards from evidence people cannot manage — promotions, calendars, budget shifts, attrition, and what happens after a bad quarter. Start with the free diagnostic, or talk to us about an audit.
New essay every Wednesday. Subscribe via RSS.
Aaron Bare
Aaron Bare is a strategist, Wall Street Journal-bestselling author, and the founder of The Incentives Lab. He writes and advises on incentive design inside organizations — why culture is the residue of what a company rewards, how KPIs quietly go perverse, and how AI systems inherit the incentives their designers set.
More from Aaron BareGo deeper
More essays
Anatomy of a Two-Week Incentive Sprint
Ten working days from a behavior nobody can explain to a structural change with a counter-metric attached.
Goodhart's Law in the Real World
When a measure becomes a target, it stops being a good measure. Three field cases and the paired-metric fix.
What a Cultural Performance Audit Actually Finds
Not a sentiment score. A reconstruction of what your organization rewards, built from evidence nobody can manage for the survey.