Scientific Definition
Evaluation suites that no longer reflect real-world conditions.
Plain-English Definition
Evaluation suites that no longer reflect real-world conditions.
Feynman Explanation
Your benchmarks aged out of relevance.
Core Principle
Evaluation suites that no longer reflect real-world conditions.
Mechanisms
Pending editorial review.
Pending editorial review.
Pending editorial review.
Pending editorial review.
Pending editorial review.
Pending editorial review.
Pending editorial review.
Inputs (Triggers)
Pending editorial review.
Outputs (Behaviors)
Pending editorial review.
Behavioral Signature
Your benchmarks aged out of relevance.
Examples
- Benchmark scores rising while real usage quality declines.
- Quality erosion masked by stale measurement.
Pending editorial review.
Famous Experiments
Pending editorial review.
Design Principles
- Periodically refresh evals against current production data.
Measurement Approaches
Pending editorial review.
Evidence
Pending editorial review.
Pending editorial review.
The Perverse Incentive Lens™
How this behavior is exploited — and how to redesign around it.
- Periodically refresh evals against current production data.
Pending editorial review.
Pending editorial review.
Interactive Mini Network
Click any neighbor to re-center the graph and follow the threads of connection.
Knowledge Graph Neighbors
Auto-linked to the rest of the Human Behavior Taxonomy by family, domain, dimension, and shared keywords.
The relationship between inputs and outputs changes.
Underlying data distribution changes over time.
Performance degradation as real-world data shifts.
AI system that takes actions to achieve goals, often across tools.
Designing processes from scratch around AI capability.
How much input the model can process at once.
Vector representation of content for similarity and search.
Systematic testing of model quality, safety, and capability.
Models learning from examples in the prompt.
How much delay the user experience tolerates.
Train a smaller model to imitate a larger one.
Multiple specialized agents coordinating on tasks.