The model is innocent. The objective function is the crime scene.
Reward hacking at training speed is reward hacking at civilizational speed.
Root-Out Framework →AI Incentives · Diagnostic
An 8-question diagnostic surfaces the hidden incentive your AI rollout is rewarding inside your org — usually not the one your roadmap claims. Four minutes. Members unlock the structural fix.
The catalog below is the concept library. These four lessons teach how alignment, adoption, and reward design actually play out in real organizations. Hover any underlined term to see the definition.
Most enterprise AI failures get blamed on data quality, model accuracy, or change management. Underneath all three is the same problem: the AI's objective was specified narrowly, the humans around it had a different objective, and the two diverged predictably.
Alignment, in practice, is incentive design wearing a math costume.
When an AI is rewarded for a proxy — clicks, watch time, "helpfulness" scores — it will eventually find a way to maximize the proxy in ways its designers never intended. This is reward hacking, and it is the same phenomenon as the Hanoi rat bounty, only faster and more literal.
Specify behaviors, not just outcomes. Test for shortcuts before deploying, not after.
Every employee who hears "AI will replace" will sabotage adoption — quietly, plausibly, effectively. They will cite data quality, compliance, or change fatigue. The framing is not PR. It is the literal incentive structure of the rollout.
If usage threatens the user's job, usage will not happen.
The strongest AI deployments are designed as collaboration patterns, not handoffs. The human owns judgment under uncertainty; the AI owns volume under known rules; the interface between them is built so that both get rewarded by the same outcomes.
Misaligned loops produce moral hazard on both sides — humans rubber-stamping AI output, AI optimizing for what humans approve.
What the model is actually optimizing.
"Models do exactly what they're told. The damage lives in the spec."
Open full entry →Engagement-maximizing recommenders that produce radicalization.
Strategy is downstream of objective design.
Treat objective function definition as the most important AI design decision.
The model achieves the goal as stated, not as intended.
"AI is a wish-granting genie. Be careful what you wish for."
Open full entry →RL agents exploiting environment bugs to maximize reward.
Every AI failure is mostly a specification failure.
Adversarial review of specs. Red-teaming. Multi-objective constraints.
Maximizing the reward signal in unintended ways.
"Show me how you measure success and I'll show you how I'll cheat."
Open full entry →Boats spinning in circles to collect score in CoastRunners.
Internal AI deployments often hack their own KPIs.
Pair every reward with a counter-metric. Audit for drift.
Optimizing a proxy of the goal degrades the actual goal.
"The metric is not the mission."
Open full entry →Click-through rates that drive worse user experience.
AI rolled out against proxy KPIs that diverge from real value.
Continuously validate proxies against true business outcomes.
The trained model develops its own internal optimizer.
"You trained one optimizer. You got two."
Open full entry →Theoretical concern in AI safety; observed glimmers in large models.
Why complex AI systems are harder to govern than they look.
Interpretability tooling. Constrained training procedures.
Outer: the spec matches our intent. Inner: the model actually pursues the spec.
"Two ways to be misaligned. Both common."
Open full entry →A model that 'wants' something different from what it was trained to want.
Why alignment is harder than guardrails.
Design for both. Monitor for drift on each.
Reinforcement learning from human feedback.
"We taught the model what we wanted by clicking thumbs."
Open full entry →How modern LLMs were tuned to follow instructions.
Quality of feedback shapes quality of model.
Diverse, calibrated, well-incentivized feedback labor.
Models trained to follow a written set of principles.
"Bill of rights for the bot."
Open full entry →Anthropic's Claude training methodology.
Customization frameworks for enterprise deployments.
Write your enterprise constitution before deploying agents at scale.
Confident outputs that are factually wrong.
"Confidence and accuracy are different variables. Models can max either."
Open full entry →LLMs inventing citations, case law, or factual claims.
Major source of enterprise AI risk.
Retrieval grounding. Citation requirements. User training.
Generate based on retrieved documents, not just trained weights.
"Memory the model didn't have to memorize."
Open full entry →Enterprise chatbots grounded in internal docs.
Standard pattern for enterprise LLM deployment.
Quality of retrieval matters more than quality of generation.
Specialized training on domain data.
"The model already knows English. You teach it your dialect."
Open full entry →Domain-specific LLMs for legal, medical, financial work.
When generic models aren't good enough.
Fine-tune for tone and format. Use RAG for facts.
Models learning from examples in the prompt.
"Demonstration is faster than retraining."
Open full entry →Few-shot prompting.
Often the right answer before reaching for fine-tuning.
Master prompt engineering before investing in fine-tuning.
Malicious instructions hidden in user input or retrieved content.
"Your agent's loyalty just got rewritten by a webpage."
Open full entry →Indirect prompt injection through documents an agent reads.
Major security risk for agent deployments.
Treat all input as adversarial. Output validation. Sandboxing.
Bypassing model safety constraints.
"Every safety system has a creative attack surface."
Open full entry →Role-playing exploits that bypass content policies.
Risk in user-facing AI products.
Defense in depth. Continuous red-teaming. Don't rely on model alignment alone.
Performance degradation as real-world data shifts.
"The model that worked last quarter doesn't work this one."
Open full entry →Fraud-detection models that degrade as fraud patterns evolve.
ML operations is half AI strategy.
Continuous monitoring. Retraining schedules. Drift detection.
Underlying data distribution changes over time.
"The world moves. Your model stays still."
Open full entry →COVID broke many production models overnight.
Production AI requires ongoing data observability.
Statistical monitoring of input distributions.
The relationship between inputs and outputs changes.
"What 'good' means moves; the model doesn't notice."
Open full entry →Customer behavior shifts post-pandemic.
Quiet performance erosion.
Monitor outcomes, not just inputs and outputs.
AI bolted onto existing workflows to look forward-leaning.
"We added AI. Somewhere. The board loved it."
Open full entry →Chatbots launched without integration, escalation, or measurement.
Investment without thesis. Theater without value.
Define the business outcome before deploying the tool.
AI pilots that succeed and never scale.
"Death by a thousand pilots."
Open full entry →Enterprises with dozens of pilots and zero production deployments.
Common AI failure mode.
Design pilots with scaling criteria from day one.
Employees using unauthorized AI tools to get work done.
"Your real AI strategy is whatever your team's actually using."
Open full entry →Personal ChatGPT, Claude, and Copilot subscriptions across the org.
Governance gap and competitive intelligence loss.
Sanction great tools. Provide better internal alternatives. Train, don't punish.
AI doesn't just replace tasks — it threatens identities.
"Resistance isn't fear of change. It's fear of irrelevance."
Open full entry →Senior contributors resisting tools that compress junior-level work.
Most AI adoption resistance is status threat in disguise.
Reframe roles to elevate, not replace. Identity-protective change management.
Individual speed gains hide collective quality decline.
"Everyone moves faster. The product gets worse."
Open full entry →AI-generated code that's faster to write and harder to maintain.
Local optimization producing global degradation.
Measure outcomes, not throughput.
Foundational skills erode through AI offloading.
"The junior engineer who never debugs becomes the senior who can't."
Open full entry →GPS use measurably reduces spatial reasoning over time.
Long-term capability risk that's hard to see quarter to quarter.
Deliberate practice of foundational skills. Constraints on AI use during learning.
Agents deployed before anyone owns the consequences.
"We shipped the agent. Nobody shipped the accountability."
Open full entry →Customer-facing agents with unclear escalation and liability paths.
Risk profile that scales faster than oversight.
Write the accountability chain before deploying the system.
Written rules about how AI may be used internally.
"Without a policy, every team writes their own. Inconsistently."
Open full entry →Use policies covering PII, IP, sensitive data, attribution.
Necessary infrastructure for scaled adoption.
Write it. Train on it. Audit against it.
Documentation of model purpose, performance, limitations, and risks.
"Nutrition labels for AI."
Open full entry →Standard practice from major model providers.
Required for responsible enterprise deployment.
Maintain model cards for every production system.
Inventory of models, data, tools, and dependencies in an AI system.
"You can't govern what you can't list."
Open full entry →Increasingly required by regulators and enterprise procurement.
Supply chain transparency for AI systems.
Maintain AI-BOMs. Update them. Use them for risk review.
Human review at critical AI decision points.
"The autopilot is fine. Until it isn't."
Open full entry →Medical AI requiring physician confirmation.
Risk reduction for consequential decisions.
Design where the human adds real signal, not where they rubber-stamp.
Human oversight without per-decision review.
"Watch the system, not every action."
Open full entry →AI trading systems with circuit breakers.
Scaling pattern for high-volume agent work.
Effective monitoring matters more than illusory per-decision approval.
Adding noise to data to protect individual privacy.
"Useful patterns without identifiable people."
Open full entry →Apple's iOS data collection.
Privacy-preserving analytics and model training.
Use where regulatory or trust pressure justifies the accuracy tradeoff.
Training models across devices without centralizing data.
"The model travels. The data stays home."
Open full entry →Mobile keyboard prediction training.
Privacy-by-design pattern.
Consider when data centralization is the bottleneck or risk.
Generated training data that mimics real data.
"Real data is expensive. Synthetic is fast. Both have their lies."
Open full entry →Augmentation for rare classes in fraud detection.
Useful — and risky — pattern for training data scarcity.
Test models on real holdout data. Don't trust synthetic for validation.
Matching trust in a system to its actual reliability.
"Too much trust ships errors. Too little wastes capability."
Open full entry →Users either over-trusting or rejecting AI outputs.
Adoption strategy and risk management.
Surface confidence intervals. Train users on appropriate skepticism.
Quantifying uncertainty in model outputs.
"An answer without uncertainty is a guess wearing a confidence costume."
Open full entry →Probabilistic forecasting with quantified ranges.
Decision quality under model uncertainty.
Require uncertainty quantification on every model output that drives decisions.
Treating AI as more humanlike than it is.
"It said 'I think' and we believed it."
Open full entry →Users attributing intent, emotion, and ethics to LLMs.
Risk of misreading model behavior.
Interface design that resists misleading anthropomorphic cues.
Reduced vigilance with automated systems.
"Human-in-the-loop is fine until the human isn't."
Open full entry →Aviation, manufacturing, and now AI agents.
Predictable human factor in oversight systems.
Designed friction. Periodic manual exercises. Active monitoring.
Discounting algorithmic advice even when superior.
"We forgive humans for errors and punish algorithms for the same ones."
Open full entry →Even when algorithms outperform, users prefer humans.
Adoption failure of high-quality AI.
Display calibration data. Allow user adjustment. Build trust gradually.
Augmentation strategy vs. substitution strategy.
"Coach increases the median. Replacement raises the floor."
Open full entry →Doctors with AI outperform AI alone in many domains.
Strategic positioning of AI in workflows.
Choose deliberately. Don't drift between them.
Augment when judgment matters. Automate when scale matters.
"Two strategies. Often confused. Different outcomes."
Open full entry →Customer service tier-1 vs. customer service complex case.
Strategic AI deployment pattern.
Audit each workflow: augment or automate? Choose explicitly.
AI system that takes actions to achieve goals, often across tools.
"Chat is reactive. Agents are autonomous. The risk profile is different."
Open full entry →Customer support agents, coding agents, research agents.
Major emerging category.
Sandbox first. Define guardrails. Tight observability.
When the agent acts, who's responsible?
"'The algorithm did it' is not a legal defense."
Open full entry →Autonomous trading. Autonomous customer commitments.
Liability frameworks lagging deployment.
Map liability before deploying. Insurance. Audit trails.
Categorizing AI use cases by risk level.
"Not every model needs the same governance."
Open full entry →EU AI Act risk categorization.
Governance proportional to risk.
Tier your AI portfolio. Govern accordingly.
Comprehensive AI regulation in the EU.
"The first major regulatory regime. Setting global precedent."
Open full entry →Risk-tiered, prohibited-use-defined regulation.
Cross-border AI strategy implications.
Build compliance for EU; you'll get most jurisdictions free.
U.S. voluntary AI risk management framework.
"Voluntary frameworks become the de facto standard once they're cited in contracts."
Open full entry →Standard reference for U.S. enterprise AI governance.
Mapping AI risk in U.S. regulatory environment.
Adopt NIST AI RMF as your governance baseline.
Logging of AI inputs, outputs, and decisions.
"If you can't trace it, you can't defend it."
Open full entry →Required for many high-stakes deployments.
Discovery readiness and incident response.
Log everything. Make logs queryable. Preserve them.
Why did it produce this? vs. How does it work?
"Different questions. Different methods. Both useful."
Open full entry →SHAP values vs. mechanistic interpretability.
Compliance, user trust, debugging.
Match the tool to the question.
Adversarial testing of AI systems.
"Pay someone to break it before users do."
Open full entry →Standard practice for major model launches.
Pre-deployment risk reduction.
Internal and external red-teaming. Continuous, not one-off.
Systematic testing of model quality, safety, and capability.
"If you don't have evals, you have vibes."
Open full entry →Domain-specific eval suites for every production model.
AI quality engineering.
Treat evals as core engineering practice, not an afterthought.
Evaluation suites that no longer reflect real-world conditions.
"Your benchmarks aged out of relevance."
Open full entry →Benchmark scores rising while real usage quality declines.
Quality erosion masked by stale measurement.
Periodically refresh evals against current production data.
Real cost includes data, ops, monitoring, governance, training.
"The model is the cheapest part."
Open full entry →Enterprise deployments where ops costs exceed model costs many-fold.
Budget surprises predictable.
Estimate TCO honestly before committing. 5x your initial estimate.
GPU access and pricing shape what's feasible.
"Strategy moves with the GPU curve."
Open full entry →Inference cost-per-token shaping deployment patterns.
Strategic infrastructure decisions.
Monitor compute cost trajectories. Plan against multiple scenarios.
Training is a one-time cost. Inference is forever.
"Cheap to train, expensive to serve. Pick your poison."
Open full entry →Why model size matters at scale.
Architecture choice and deployment strategy.
Right-size the model for the inference budget, not the training budget.
Train a smaller model to imitate a larger one.
"Compress the wisdom. Keep the speed."
Open full entry →Edge deployments of distilled LLMs.
Cost and latency optimization.
Distillation often beats fine-tuning at scale.
Where the model runs shapes privacy, latency, cost, and capability.
"Three different tradeoffs. Pick two."
Open full entry →Mobile AI vs. data-center AI.
Deployment architecture decision.
Choose deliberately per use case.
Cross-functional governance body for AI decisions.
"Without it, every team is its own AI policy department."
Open full entry →Standard pattern for enterprises with serious AI portfolios.
Governance infrastructure.
Compose for diversity: legal, technical, ethics, business, frontline.
Stages of organizational AI capability.
"Pilot ≠ production ≠ platform ≠ pervasive."
Open full entry →Standard maturity frameworks from major analysts.
Strategy phasing and investment sequencing.
Benchmark honestly. Don't claim a stage you haven't reached.
Scarce AI talent commands market-distorting compensation.
"If you can't pay it, you'd better partner."
Open full entry →ML engineer comp at the top of the market.
Strategic hiring decisions.
Build vs. partner vs. buy decisions made deliberately.
Teams built around AI from day one vs. teams adding it to existing workflows.
"Augmented teams adopt. Native teams reinvent."
Open full entry →Insurgent startups vs. incumbent enterprises.
Competitive dynamics shifting.
Some workflows demand native rebuild. Some accept augmentation. Decide per case.
Investing in workforce transition vs. workforce change.
"Replacement is fast and expensive. Reskilling is slow and expensive."
Open full entry →Major corporate transformation programs.
Talent strategy under AI change.
Plan honestly. Both/and. Communicate transparently.
AI investment outpacing measurable productivity gains.
"We see AI everywhere except in the productivity statistics."
Open full entry →Echoes of the 1980s computer productivity paradox.
Patience and measurement matters.
Set realistic productivity expectations and timelines.
Teams under-reporting AI capability to protect comp or status.
"Quiet quitting goes both ways."
Open full entry →Salespeople not adopting tools that reveal pipeline health.
Predictable when adoption threatens individual interests.
Align personal upside with adoption upside.
Pre-deployment analysis of who loses what.
"Resistance you didn't see coming will see you coming."
Open full entry →Stakeholder analysis adapted for AI rollouts.
Change management foundation.
Before deploying: map the loss landscape. Plan compensation.
Redesigning roles around AI capability.
"Bolting AI onto existing jobs leaves both half-broken."
Open full entry →Customer service roles fundamentally redesigned around AI augmentation.
Where productivity gains actually materialize.
Don't add AI to old jobs. Design new ones.
Designing processes from scratch around AI capability.
"The right question is what we'd build if AI had always existed."
Open full entry →Greenfield deployments that out-perform retrofitted ones.
Major strategic opportunity for incumbents.
Run parallel design exercises: existing-workflow + AI-first.
Innovators → early adopters → majority → laggards, AI-specific.
"Stage one bought tools. Stage two will redesign work."
Open full entry →Diffusion patterns observable across industries.
Strategy phasing.
Know which curve segment you're targeting. Pitch accordingly.
AI capability outpacing organizational ability to use it.
"The model can. The org can't. Yet."
Open full entry →Most enterprises sitting on capabilities they haven't integrated.
Strategic opportunity hiding in plain sight.
Capability audits. Use-case backlogs. Adoption infrastructure investment.
Strategic choice on AI capability sourcing.
"Three answers. Different bets. Pick wrong and you wear it."
Open full entry →Foundation model decisions across enterprises.
Capital allocation and strategic positioning.
Decide per layer of the stack. Don't make one decision for all of AI.
AI strategy = decisions about which capabilities to build and where.
"AI strategy is not a tools strategy. It's an org strategy."
Open full entry →Companies treating AI as procurement vs. as transformation.
Fundamental framing of AI investment.
Lead with capability questions, not vendor questions.
Concentration risk on a single AI provider.
"Diversify before you wish you had."
Open full entry →Enterprises with everything riding on one foundation model.
Strategic risk management.
Multi-model architectures. Abstraction layers. Portability assumptions.
A handful of providers shape the entire AI economy.
"Watch the moats forming. They're forming fast."
Open full entry →OpenAI, Anthropic, Google, Meta, plus a few more.
Long-term strategic positioning.
Plan for both worlds: continued concentration and surprise decentralization.
Open-weight vs. API-only models.
"Different bets on the future of AI economics."
Open full entry →Llama, Mistral, DeepSeek vs. GPT-class APIs.
Cost, control, capability tradeoffs.
Use both. Different use cases want different answers.
How much delay the user experience tolerates.
"If it doesn't feel fast, it isn't valuable enough."
Open full entry →Real-time vs. async AI experiences.
Architecture and model choice driver.
Set latency budgets early. They constrain feasible architectures.
Operating economics shaped by per-token pricing.
"Strategy bends to wherever the compute curve goes."
Open full entry →Why model choice matters at scale.
Margin discipline in AI products.
Right-size models per query. Cache aggressively.
Cache common prompt prefixes to reduce cost and latency.
"Don't pay twice for the same setup."
Open full entry →Common system prompts cached across calls.
Cost optimization in production deployments.
Architect for cache reuse.
Multiple specialized agents coordinating on tasks.
"More agents, more orchestration, more breakage."
Open full entry →Research, coding, and analysis agent ensembles.
Emerging deployment pattern.
Orchestration is harder than the agents themselves.
Models calling external functions and APIs.
"Models without tools talk. Models with tools act."
Open full entry →Function calling, code execution, web search.
Foundation of agent capability.
Sandbox tools. Audit calls. Rate limit.
How much input the model can process at once.
"Bigger context, more capability, more cost."
Open full entry →1M token context windows in newer models.
Architecture decisions for document-heavy work.
Don't over-index on long context. Retrieval often beats stuffing.
Vector representation of content for similarity and search.
"The math behind 'this looks like that.'"
Open full entry →Semantic search, RAG retrieval.
Foundation of modern AI search and similarity.
Choose embedding models deliberately. They're not all equivalent.
Database optimized for high-dimensional similarity search.
"Search by meaning, not by keyword."
Open full entry →Pinecone, Weaviate, pgvector.
Infrastructure for RAG and similarity systems.
Don't over-build. Often pgvector is enough.
Tracing AI components for risk and compliance.
"Provenance is the new audit trail."
Open full entry →EU AI Act requirements.
Compliance and supply chain risk.
Maintain AI-BOM continuously. Audit periodically.
Tracking where training and inference data came from.
"If you can't trace the data, you can't trust the model."
Open full entry →Standard requirement in regulated industries.
Compliance, quality, debugging.
Engineer lineage tracking from day one.
Cryptographic tracking of content origin.
"Watermarks for the synthetic media era."
Open full entry →Coalition for Content Provenance and Authenticity.
Trust infrastructure for an AI-generated content world.
Adopt where applicable. Watch evolving standards.
Systematic skew in model behavior across groups.
"The model learned what the data taught it. The data taught it our history."
Open full entry →Hiring tools, lending models, recommendation systems.
Legal, reputational, and effectiveness risk.
Bias audits. Diverse data. Continuous monitoring.
Quantitative measures of model behavior across groups.
"Several mathematical definitions of fairness. They conflict."
Open full entry →Demographic parity, equalized odds, calibration.
Trade-offs require explicit choices.
Pick fairness criteria deliberately. Document the choice.
Testing model behavior on hypothetical alternate inputs.
"What if the same applicant were male/female/named differently?"
Open full entry →Standard practice for fairness auditing.
Bias detection.
Run counterfactuals on consequential models continuously.