How to Build a Supplier Scorecard for AI Vertical Drama Production Partners

The platform that selects production partners by impression — "they delivered well last time," "they were responsive," "the content looked good" — is selecting on incomplete data. Impression captures the most recent commission and weights it against a general sense of the relationship. It does not capture the pattern across multiple commissions, the genre-specific performance differential, or the budget tier alignment that makes one production partner a better match for a specific commission than another equally familiar partner.

The supplier scorecard converts production partner performance from impression to evidence. A production partner who has delivered 12 commissions for the platform has 12 data points across first-pass acceptance rate, milestone delivery reliability, quality tier alignment, escalation frequency, and revision efficiency. That data is more accurate than the impression the content executive holds from the last commission they paid attention to.

This post covers the complete supplier scorecard framework: the five performance dimensions, how to collect data for each, how to weight the dimensions for different commission types, and how to use the scorecard to make commission matching decisions that improve as data accumulates.

The Five Scorecard Dimensions

Dimension 1: First-Pass Acceptance Rate

What it measures: The percentage of episode outputs submitted by the production partner that pass all five quality criteria without requiring a revision instruction.

How to collect it: The generation log for each commission records the quality criteria result for every submitted output — pass or fail for each of the five markers. First-pass acceptance rate is the number of first-submission passes divided by the total number of outputs submitted across all commissions from this production partner.

Why it matters: First-pass acceptance rate is the most direct measure of the production partner's generation quality discipline. A high first-pass rate means the production partner is conducting rigorous self-review before submission — their internal quality standard matches the platform's quality criteria. A low first-pass rate means the production partner is submitting outputs they have not adequately reviewed, generating revision cycles that extend production timelines and consume creative director time.

Scoring benchmark: Above 80%: strong. 65% to 80%: acceptable. Below 65%: below standard — triggers a production process review before the next commission assignment.

Dimension 2: Milestone Delivery Reliability

What it measures: The percentage of agreed milestone dates that the production partner meets or beats across all commissions.

How to collect it: The production calendar records the agreed milestone date and the actual delivery date for every milestone in every commission. Milestone delivery reliability is the number of on-time or early milestones divided by the total number of milestones agreed across all commissions.

Why it matters: Milestone delivery reliability is the most direct predictor of whether the production partner will deliver the next commission on schedule. A production partner with 92% on-time milestone delivery is a production partner the platform can schedule against reliably. A production partner with 68% on-time delivery is a production partner the platform needs to build buffer into every schedule for — which affects the platform's content calendar and the downstream distribution timeline.

Scoring benchmark: Above 90%: strong. 75% to 90%: acceptable. Below 75%: below standard — requires mandatory schedule buffer on all future commissions and a production process review before the next assignment.

Dimension 3: Quality Tier Alignment

What it measures: Whether the delivered content quality matches the expected quality standard for the agreed per-minute production budget.

How to collect it: Each commission's delivery review records the quality tier the delivered content achieves (Tier 1, Tier 2, or Tier 3 as defined in the quality and price-tier framework) and the production budget tier the commission was priced at. Quality tier alignment is the match between the two — did the content deliver at, above, or below the expected quality for the agreed budget?

Why it matters: Quality tier alignment tells the platform whether its production spend is producing the expected content quality. A production partner consistently delivering above the expected quality tier at the agreed budget is a production partner whose content is underpriced — a valuable relationship to expand. A production partner consistently delivering below the expected quality tier at the agreed budget is a production partner who is overcharging or underperforming — a relationship to reprice or discontinue.

Scoring benchmark: At or above expected tier on 85% or more of commissions: strong. At or above expected tier on 65% to 85% of commissions: acceptable. Below expected tier on more than 35% of commissions: below standard — requires repricing negotiation or production quality improvement plan.

Dimension 4: Escalation Frequency

What it measures: How often the production partner's commissions require formal escalation — a notification that a quality failure or milestone failure has exceeded the standard revision and recovery procedure.

How to collect it: The production management log records every escalation event across all commissions from the production partner: the trigger, the escalation date, the resolution procedure, and the time to resolution.

Why it matters: Escalation events are the most resource-intensive production management events. Each escalation requires creative director involvement, potentially production partner replacement evaluation, and administrative overhead that the standard production workflow does not. A production partner who requires escalation on 40% of their commissions is a production partner whose management overhead cost is significantly higher than their nominal management fee implies.

Scoring benchmark: Zero escalation events: strong. One escalation event per 10 commissions or fewer: acceptable. More than one per 10 commissions: below standard — triggers a supplier risk review and may require reduced commission allocation until the pattern is addressed.

Dimension 5: Revision Efficiency

What it measures: How quickly and accurately the production partner completes revision instructions once issued.

How to collect it: The revision log records the revision instruction date, the resubmission date, and whether the resubmission addressed the specific failure identified in the revision instruction or required a second revision cycle.

Why it matters: A production partner who receives a specific revision instruction and resubmits an output that addresses the specific failure demonstrates production discipline — they read the instruction, understood the failure, and corrected it precisely. A production partner who resubmits an output that misses the specific failure — generating a new version without correcting the identified problem — demonstrates a revision process gap that will compound across every future commission.

Scoring benchmark: Above 90% of revisions addressed correctly on first resubmission: strong. 75% to 90%: acceptable. Below 75%: below standard — requires specific revision process coaching or reduced commission allocation.

The Scorecard Weighting by Commission Type

Not all five dimensions are equally important for all commission types. The weighting should reflect what matters most for the specific commission being matched.

For a premiere commission at a tier-1 platform acquisition target (ReelShort, DramaBox):

First-pass acceptance rate: 35% of the matching decision. Quality tier alignment: 35%. Milestone delivery reliability: 20%. Revision efficiency: 10%. Escalation frequency: scored separately — any escalation history is a red flag for tier-1 platform acquisition commissions regardless of the overall score.

For a volume commission at a tier-2 platform (standard professional quality, 20-episode series):

Milestone delivery reliability: 35% of the matching decision. First-pass acceptance rate: 30%. Revision efficiency: 20%. Quality tier alignment: 10%. Escalation frequency: 5%.

For a genre-specific commission (thriller, revenge arc, or male-protagonist content):

Add a genre performance sub-dimension alongside the five main dimensions: has the production partner previously delivered commissions in this genre, and if so, what was their first-pass acceptance rate and quality tier alignment specifically for that genre? A production partner with a 78% overall first-pass acceptance rate but a 91% first-pass acceptance rate on thriller content is a stronger match for a thriller commission than a production partner with an 85% overall rate but no thriller history.

Building the Scorecard Infrastructure

The supplier scorecard is only as useful as the data it is built from. The data collection infrastructure must be in place before the first commission with a production partner — not retrospectively assembled from memory after several commissions have been completed.

The minimum infrastructure:

A shared spreadsheet (Google Sheets or equivalent) with one row per commission and columns for each scorecard dimension's raw data: output count, first-pass pass count, milestone count, on-time milestone count, quality tier delivered, quality tier expected, escalation event count, revision count, and accurate revision resubmission count.

The production coordinator maintains the spreadsheet, populating each column from the generation log and the production management log for each commission. The scorecard dimension scores are calculated automatically from the raw data columns.

A scorecard summary sheet that aggregates each production partner's raw data across all commissions and produces the five dimension scores and the overall weighted score for each commission type weighting.

The scorecard is reviewed at the end of every commission's delivery acceptance and at the quarterly account review. Production partners whose scores fall below any dimension's benchmark threshold are flagged for the production process review specified in that dimension's scoring benchmark.

Using the Scorecard for Commission Matching

The scorecard's primary commercial function is commission matching — selecting the production partner from the qualified network whose scorecard profile best fits the specific commission's requirements.

The matching process: identify the commission type (premiere vs volume, genre category). Apply the appropriate dimension weighting. Sort the qualified production partners by weighted score. Select the highest-scoring production partner with available capacity for the commission's timeline.

A production partner with the highest overall weighted score but insufficient capacity for the commission's timeline is not the correct match. The scorecard ranks within the capacity-available subset, not across the full network.

The scorecard also identifies expansion opportunities: production partners whose quality tier alignment consistently exceeds expectations are candidates for budget tier advancement — commissioning them at the next budget tier up to test whether their quality scales proportionately. Production partners whose scorecard falls below standard thresholds are candidates for performance improvement plans or de-qualification from the network.

Axis AI Studios Perspective

The supplier scorecard framework described in this post is the foundation of AXIS Management's production partner network intelligence. Every production partner in the AXIS Management network has a live scorecard updated after every commission. Commission matching decisions at every tier are made against scorecard data rather than relationship impression.

For platforms who want to implement supplier scorecard infrastructure for their own direct production relationships, the data collection framework described in this post can be built in Google Sheets with the production coordinator maintaining it alongside the generation log. The investment is administrative discipline at the commission level, not a software development project.

For platforms who want the scorecard intelligence without building the infrastructure internally, AXIS Management's commission matching decisions are made against the accumulated scorecard data across the full production network — data that a single platform's direct commission history cannot replicate at equivalent depth.

Reach out at business@axisaistudios.com for commissioning conversations or for platforms who want to discuss how AXIS Management's supplier scorecard data is applied to their specific commission matching.

FAQ

How Many Commissions Does a Production Partner Need Before the Scorecard Data Is Statistically Useful?

Three commissions is the minimum for the scorecard to differentiate between systematic performance patterns and single-commission variance. A production partner with one commission at 85% first-pass acceptance rate may be performing at their normal standard or may have had an unusually strong production session. A production partner with three commissions consistently above 85% is demonstrating a systematic production quality standard. Below three commissions, the scorecard should be used as a weak signal rather than a definitive ranking.

Should the Scorecard Be Shared With Production Partners?

Yes, selectively. Sharing a production partner's own scorecard data — their own first-pass acceptance rate, their own milestone delivery rate — creates transparency that typically improves subsequent performance. Production partners who see that their revision efficiency is below the benchmark threshold and understand what the benchmark measures have specific improvement targets rather than vague quality feedback. Sharing the comparative rankings across production partners is not recommended — the comparative data is the platform's or managed service's proprietary network intelligence, not the production partners' shared information.

What Should a Platform Do When a Production Partner Falls Below the Scorecard Threshold?

The scorecard threshold breach triggers a defined response rather than an ad hoc decision. Each dimension's scoring benchmark specifies the response: below 65% first-pass acceptance triggers a production process review before the next assignment; more than one escalation event per 10 commissions triggers a supplier risk review; below 75% milestone delivery reliability requires mandatory schedule buffer on all future commissions. These responses are documented in the managed service agreement or the platform's direct production management policy so they are applied consistently rather than negotiated case by case.

Further Reading

For the production data collection that feeds the supplier scorecard's raw data columns, the guide to what production data platforms should be collecting from every AI series commissioned covers every data category, how to collect it from the generation log and production calendar, and why each data point predicts future performance.

For the quality tier framework that the Quality Tier Alignment dimension is based on, the quality and price-tier framework covers the three production tiers, the tier-specific quality criteria, and how quality tier is assessed at each delivery review.

For the due diligence process that supplier qualification precedes scorecard data accumulation, the guide to what to look for in an AI vertical drama production partner covers the pre-qualification checklist that a production partner passes before they enter the network and begin accumulating scorecard data.