A model gets picked in a Slack thread on a Tuesday. Someone shares a benchmark screenshot, two engineers agree, and the API key goes into the config. Six months later that model is answering customer questions, the engineer who chose it has moved teams, and nobody can explain the decision.
That gap is what model governance is meant to close. Not a policy PDF, and not an ethics committee, but a documented sequence that turns “which model” into a decision anyone can reconstruct later. Most teams can name the model they shipped. Far fewer can produce the document that says why it won, who signed off, and what would send the decision back for review.
What follows is that sequence: what model governance actually covers, why informal selection fails at scale, a four-step approval process, a scorecard you can copy, and a maturity checklist for locating where your team currently sits.
What model governance actually means
Model governance is the practice of deciding which models your organization may use, on what evidence, and under whose authority. It covers the review before a model goes live and the review that happens after, and the output is a record rather than an opinion.
The scope is narrower than most governance programs assume. It is not about writing acceptable-use policy. It is about a specific model, in a specific use case, judged against fixed criteria, with a name attached to the approval.
It helps to separate three things that constantly overlap. AI governance is the outer ring: policy, ethics, regulatory posture, and how the company decides what it will and will not build. Model governance sits inside that ring and answers one question repeatedly: is this specific model approved for this specific job, and does that approval still hold. MLOps is the machinery underneath, the deployment pipelines, versioning, rollback, and monitoring that make a governance decision enforceable but do not make the decision.
Model choice looks like an engineering call because the inputs are technical: latency, context window, tool-calling reliability. But the consequences land outside engineering. A model that hallucinates on a policy question creates legal exposure. A model that routes data to a new provider creates a security review. A model that costs several times more per call changes the product’s unit economics. The evidence is technical, the accountability is not, and the two need to meet in one document.
Why informal model selection breaks down at scale
One team picking one model informally is fine. The failure appears when ten teams do it independently over eighteen months and the organization discovers it has no idea what it is running. It shows up in three recognizable shapes.
Shadow AI is any model adopted outside a review. It usually starts well, a team needs a summarizer, someone drops in a key, the feature ships. The problem is cumulative: each unreviewed adoption is a data path nobody mapped, a cost line nobody forecast, and a provider nobody vetted. By the time someone asks for a model inventory, building one is an archaeology project, and because every team chose on different criteria, you cannot rank the models, retire the weak ones, or negotiate as one buyer.
Orphaned models are ones running in production with no living owner, because the person who chose it left or moved on. They fail quietly: nobody watches the failure rate, nobody notices when the provider ships a new version, and nobody remembers the original acceptance threshold. The model keeps answering until something breaks loudly enough to trace back.
Approval theater is the worst outcome, because it looks like governance. A form gets filled in, a director signs, a ticket closes, and nothing in that record tells the next person how the decision was made. The test is simple: if a different team faced the same choice next quarter, could they reuse your approval as a template. If not, the sign-off documented compliance with a process, not the decision.
The four-step approval process
A repeatable process has four steps, and each one produces an artifact the next step depends on, which is what stops the sequence collapsing back into a Slack thread.
Step 1: Intake and use-case tiering. A short form filled in before anyone benchmarks anything: what is the use case, what data does it touch, who sees the output, and what happens if the output is wrong. Tiering follows from those answers. A model drafting internal notes is low risk. A model generating text a customer reads, or one touching regulated data, is high risk and gets the full review. Tiering earns its keep by cutting work, because without it every request gets the same heavyweight review, teams route around it, and you are back to shadow AI.
Step 2: Evidence-based comparison. This is where most processes go soft, comparing two candidates on vibes and a public leaderboard rank. Instead, run both on the same task-specific test set and record four things: output quality on your data, safety behavior, cost per call, and latency. Public benchmarks tell you what to shortlist; your own data decides. The criteria have to be identical across candidates, because a comparison where model A saw a hundred prompts and model B saw twelve is a preference with numbers attached.
Step 3: Guardrail and safety review before sign-off. Quality scores tell you how the model behaves on a good day. Guardrail review tells you how it behaves under pressure: prompt injection, jailbreak phrasings, requests that would leak personal data. Run the same adversarial set against every candidate and record a pass rate, and do it before sign-off, because a model that fails an injection test at the top tier is disqualified regardless of how well it writes.
Step 4: Documented owner and re-approval triggers. The record names one person accountable for the model in production, not a team and not a committee, one person with the authority to pull it. It also lists what invalidates the approval: the provider ships a new version, the use case changes, volume crosses a threshold, or a quality metric drops below the level the approval was based on. Without triggers, an approval is permanent by default, which is how a model approved for an internal pilot ends up serving customers two years later on the original review.
Who decides, not just who attends
Governance discussions default to who is in the room. The more useful question is who decides and what everyone else contributes, written down before the first review rather than argued during it.
Engineering brings the evidence: eval results, latency, failure modes seen in testing. They own the measurements, not the verdict. Legal and compliance flag where data may be processed, what disclosures the use case requires, and which regulations apply. Security reviews the data path and the provider’s posture. The business owner states what the model is for and what an acceptable failure rate looks like in their context. Each of those is an input with a clear boundary, and the failure mode is letting any one of them veto informally through delay.
A single accountable owner beats a committee vote, because a vote spreads accountability until it disappears, and when a model misbehaves the board that approved it months ago is not the group that answers for it. A named owner is reachable and has an incentive to keep the monitoring honest. Others advise, one person owns. This also fixes speed, since committee scheduling is a common reason approvals take weeks, and a slow process is one teams learn to skip.
What repeatable looks like: the scorecard
Repeatable means the next decision uses the same form, criteria, and thresholds as the last one, and the scorecard is what makes that concrete. Every candidate for a given tier gets scored on the same dimensions against pass/fail thresholds set before testing starts. Setting the thresholds first matters more than the numbers, because deciding the acceptable hallucination rate after seeing the results is a justification, not a test.
Dimension What it measures Pass/fail threshold Accuracy / groundedness Output correctness against your reference data Set per tier before testing Hallucination rate Share of responses with unsupported claims Below the tier ceiling Safety / guardrail pass rate Behavior against injection, jailbreak, PII prompts No top-tier failures Cost per 1K tokens Blended input and output cost at expected volume Within budget envelope Latency p95 response time on representative prompts Under product requirement Compliance flags Data residency, retention, disclosure obligations Zero unresolved flags
Once the dimensions and thresholds are stable, a review becomes filling in the candidate columns rather than relitigating what matters. And whatever you scored at approval becomes your monitoring set, so you are not designing a second measurement system after launch. If groundedness was the deciding column, groundedness is what you watch on live traffic, which gives the re-approval trigger something real to fire on.
Where your team sits: the maturity checklist
Most teams find themselves split across the columns below, strong on inventory and weak on re-approval. That split is the useful output: find the row furthest to the left and fix that one first.
The gap between the middle columns is the important one. Documented governance produces a file. Operationalized governance produces a file that keeps updating itself.
Keeping the process from becoming paperwork
Every governance process decays the same way. The evidence-gathering gets expensive, so people reuse last quarter’s numbers, and the form outlives the substance behind it. The fix is making the evidence cheap to regenerate. If re-running the full scorecard against a new model version takes an afternoon of manual testing, it will not happen. If it runs on a schedule and posts results, re-approval becomes a review of fresh data.
Two things make that possible. The scorecard dimensions are all measurable programmatically: groundedness and hallucination rate from evaluation runs against a reference set, guardrail pass rate from an adversarial suite, cost and latency from traces. Treat that suite the way you treat tests, running it at intake and again whenever a trigger fires, so the output is a generated scorecard rather than a typed one. And trace production behavior, because re-approval based on memory is guesswork while re-approval based on traces is a review of what the model actually did. Tracing also closes the orphaned-model problem, since a model with live traces has a visible footprint and shows up in the inventory whether or not anyone remembers deploying it.
How this maps to the frameworks
The four steps are a concrete version of what the NIST AI Risk Management Framework calls govern, map, measure, and manage, and they produce the risk-tiered, documented control that ISO/IEC 42001 asks an AI management system to maintain. Neither standard mandates this exact shape, but both require the outcome it produces: a documented, evidence-based, owner-attributed decision an auditor can review.
Both of those are voluntary. The EU AI Act is not, and since 2 August 2026 the bulk of its obligations apply, including the regime for high-risk systems. It sorts AI into tiers, prohibited practices, high-risk systems, limited-risk systems carrying transparency duties, and general-purpose models, and attaches different duties to each. If any of your users or deployments touch the EU, the tier you assign at intake stops being only a workload decision and starts determining which legal obligations attach, so it is worth mapping your internal tiers onto the Act’s categories at intake rather than running two parallel schemes.
Where this lands
Model governance fails when the choice lives in someone’s head instead of a document, and shadow AI, orphaned models, and approval theater are all versions of the same missing artifact. The four steps fix it: tier the use case at intake, compare candidates on identical criteria, review guardrails before sign-off, and name an owner with explicit re-approval triggers.
Run the maturity checklist against your own setup, find the row sitting furthest left, and make the scorecard evidence cheap enough to regenerate, because a scorecard nobody can refresh becomes paperwork within a couple of quarters. The goal is not more process. It is that the next person who asks why you are running this model gets an answer built from evidence rather than recollection. The longer version of this guide includes the full scorecard and role-mapping detail.