# DV360 Testing & Experimentation Guide ## Enterprise Operating System Guide Purpose: Enterprise framework for designing, governing, executing, measuring, and scaling experimentation across DV360 campaigns. Core rule: Tests exist to reduce uncertainty. Every test must answer a business decision. # 1. Testing Principles - Test decisions, not curiosities. - One primary hypothesis. - One primary success metric. - Protect measurement integrity. - Separate attribution from causality. - Document assumptions. - Record learnings. - Scale only after evidence. # 2. Experimentation Mandate Every test should answer: - What decision will this influence? - What uncertainty exists? - What is being varied? - What remains constant? - What metric determines success? - What would cause rejection? - What action follows each outcome? # 3. Testing Decision Tree Do we know enough to act? If yes: - Implement. If no: - Design a test. Can causality be isolated? If yes: - Controlled experiment. If no: - Observational analysis with limitations. # 4. Experiment Lifecycle 1. Define problem. 2. Define hypothesis. 3. Define metric. 4. Define control. 5. Define treatment. 6. QA setup. 7. Launch. 8. Monitor. 9. Analyze. 10. Decide. 11. Scale or reject. 12. Archive learning. # 5. Hypothesis Framework A good hypothesis: Because [insight], changing [variable] will improve [metric] for [audience]. Examples: Because creative fatigue exists, introducing new creative will improve CTR. Because users abandon after product views, shorter recency retargeting will improve CPA. # 6. Business Questions Framework Tests should support: - Budget allocation - Audience selection - Supply selection - Creative direction - Frequency strategy - Measurement strategy - Incrementality - Vendor selection Never test without a business decision. # 7. Experiment Governance Required: - Owner - Approver - Hypothesis - Metrics - Guardrails - Start date - End condition - Rollback condition # 8. Primary Metrics Choose one: - Reach - Frequency - CTR - CPC - CPA - CPL - Revenue - ROAS - Qualified leads - Incremental conversions Avoid multiple primary KPIs. # 9. Guardrail Metrics Guardrails protect unintended consequences. Examples: - CPA - CPM - Reach - Brand safety - Viewability - Lead quality # 61. Enterprise Experiment Decision System Use this system before approving any test. ## Decision 1: Is there a real decision to make? Document the decision that will change if the evidence supports the hypothesis. Examples include selecting an audience, changing creative strategy, renewing a deal, setting frequency, reallocating budget, or adopting a measurement method. If every possible result leads to the same action, do not run the test. ## Decision 2: Is the uncertainty material? Prioritise questions where a wrong decision could affect business outcomes, customer experience, quality, commercial commitments, or future scale. Do not consume budget on cosmetic questions with no operational consequence. ## Decision 3: Can the question be answered with existing evidence? Review prior experiments, approved benchmarks, historical campaigns, platform reporting, analytics, CRM, and business outcomes. If comparable evidence is strong, act through the Optimization Guide and document the basis. ## Decision 4: Is causal evidence required? Use a controlled experiment when the decision depends on proving additional impact. Use diagnostic or observational analysis when the purpose is exploration, forecasting, or operational troubleshooting. ## Decision 5: Can the treatment be isolated? Identify the unit that can receive control or treatment, such as user, geography, audience, campaign, line item, publisher, creative rotation, or time period. If treatment leaks across groups, document contamination and select a different design where possible. ## Decision 6: Is the outcome observable and timely? The primary outcome must be measurable with sufficient accuracy and within a useful decision horizon. Avoid using a fast but weak proxy when the business outcome can be obtained responsibly. ## Decision 7: Is the test feasible without unacceptable risk? Assess required budget, scale, duration, opportunity cost, commercial commitments, privacy, policy, brand, and customer impact. If risk is unacceptable, use a safer design or do not test. ## Decision 8: Is there an action for every outcome? Predefine actions for positive, negative, neutral, inconclusive, invalid, and contaminated results. --- # 62. Experiment Architecture Decision Tree ## Choose a randomized holdout when: - Eligible units can be assigned to control and treatment. - Exposure can be meaningfully withheld from control. - The primary outcome is observable for both groups. - Contamination can be managed. ## Choose a platform lift study when: - The relevant DV360 product and market are eligible. - The study's outcome matches the business question. - Required delivery and sample conditions are feasible. - The study methodology and limitations are accepted. ## Choose a geo experiment when: - Geographic treatment can be controlled. - Comparable markets or regions exist. - Spillover is limited. - Outcome volume and pre-period data are available. ## Choose a matched-market test when: - Randomized geography is unavailable. - Treatment and comparison markets can be matched defensibly. - External market differences can be documented and monitored. ## Choose an A/B media test when: - One primary variable can be changed. - Audience, inventory, timing, bidding, and measurement can be controlled sufficiently. - The purpose is a relative decision rather than universal causal proof. ## Choose observational analysis when: - Controlled assignment is not feasible. - The goal is hypothesis generation or operational diagnosis. - Limitations are explicit. Do not describe an observational comparison as a controlled experiment. --- # 63. Experiment Prioritisation Framework Score or classify proposed tests using approved enterprise criteria rather than personal preference. ## Business value - Potential revenue, reach, customer, margin, or risk impact - Size of the decision affected - Reusability across markets or campaigns ## Evidence gap - Degree of uncertainty - Quality of existing evidence - Strategic importance of closing the gap ## Feasibility - Available budget and inventory - Audience or geographic scale - Creative and measurement readiness - Operational complexity ## Risk - Customer experience - Brand or legal exposure - Privacy and policy - Commercial commitment - Opportunity cost ## Learning value - Ability to isolate the variable - Transferability of the result - Clear action path A prioritisation score supports discussion but is not automatic approval. Record the evidence and accountable decision. --- # 64. Test Brief Template ```markdown # DV360 Experiment Brief - Experiment ID: - Business decision: - Problem or opportunity: - Existing evidence: - Evidence gap: - Hypothesis: - Test type: - Unit of assignment: - Control: - Treatment: - Primary variable: - Constants: - Primary outcome: - Secondary outcomes: - Delivery guardrails: - Quality guardrails: - Business guardrails: - Eligibility criteria: - Exclusions: - Markets and flight: - Budget logic: - Sample or power logic: - Conversion-lag treatment: - Contamination risks: - Analysis method: - Success rule: - Negative-result action: - Inconclusive-result action: - Responsible owner: - Accountable approver: - QA reviewer: - Measurement owner: ``` --- # 65. Control and Treatment Design A useful control represents the decision alternative. ## Strong control characteristics - Eligible for the same outcome - Comparable audience or market - Similar timing and environment - Consistent measurement - Protected from treatment leakage - Operationally credible ## Treatment characteristics - One primary variable differs - Exposure is documented - Delivery is sufficient to represent the proposed strategy - Other campaign conditions remain stable where possible ## Common control failures - Control receives the treatment through another campaign. - Control and treatment use different audiences, inventory, creative, bids, or destinations. - Control has insufficient budget or opportunity. - Control represents an intentionally weak alternative. - Measurement differs across groups. Do not design a control that exists only to make the treatment win. --- # 66. Primary Outcome and Guardrail Playbook ## Primary outcome Choose the metric that directly supports the decision. Examples: - Incremental conversions for a causal acquisition question - Qualified leads for lead-generation strategy - New-customer revenue for customer-acquisition strategy - Incremental reach for an awareness extension - Brand consideration for a brand-message question ## Secondary outcomes Use to explain mechanism or trade-offs. Secondary metrics cannot silently replace a failed primary outcome. ## Guardrails Protect: - Delivery - Cost - Reach - Frequency - Quality - Suitability - Invalid traffic - Customer quality - Revenue or margin - Privacy and policy Predefine which guardrail breach requires investigation, pause, correction, or test invalidation. Use approved standards and do not invent universal thresholds. --- # 67. Sample, Duration, and Statistical Discipline Test sizing requires a suitable method and inputs. The Agent must not invent a minimum sample, duration, confidence level, or detectable effect. Document: - Baseline outcome rate or volume - Expected or decision-worthy effect - Unit of assignment - Allocation ratio - Variability - Significance or confidence approach where applicable - Power or evidence requirement - Conversion lag - Expected delivery - Attrition and contamination assumptions ## Duration considerations Include enough time to capture: - Representative weekdays and weekends where relevant - Delivery stabilization - Customer decision cycle - Conversion lag - Seasonal or promotional effects Do not end a test because an early result is favourable. Do not extend it repeatedly until the desired answer appears. --- # 68. Creative Experimentation Playbook Test at the highest level of uncertainty. ## Concept test Compare materially different customer ideas, such as problem, aspiration, benefit, use case, or proof. ## Execution test Compare how one concept is expressed, such as demonstration, testimonial, animation, or product-led story. ## Opening test For video, compare early moments while keeping the core proposition stable. ## Duration test Adapt the message intentionally to each duration. Do not compare a complete longer story with a careless cutdown. ## Format test Treat format, placement, and rendering as part of the treatment. Do not claim pure creative causality if inventory changes. ## Data-driven creative test Compare a governed personalized treatment with a strong standard baseline. Validate feed quality, defaults, privacy, and variant delivery. ## Decision rules - Control audience and inventory where practical. - Measure qualified or brand outcomes according to objective. - Report rotation and delivery inequality. - Do not select winners on CTR or completion alone. - Translate the result into the next production decision. --- # 69. Audience Experimentation Playbook ## Broad versus precise targeting Test whether additional precision creates enough quality to justify lower reach, higher cost, and increased frequency. ## Audience versus contextual Compare user or household signals with content or publisher relevance while documenting inventory differences. ## First-party versus platform audience Assess scale, freshness, cost, customer quality, and permitted use. ## Prospecting expansion Test controlled expansion against the current prospecting strategy. Protect existing-customer suppression and customer-quality guardrails. ## Remarketing recency Compare approved recency windows while controlling creative, frequency, inventory, and conversion lag. ## Audience-source test Compare sources only after documenting provenance, data cost, market availability, consent, and overlap. Do not assume narrow audiences are more accurate or more valuable. --- # 70. Supply, Publisher, and Deal Experimentation Playbook ## Open Auction versus PMP Assess whether the private route provides unique access, priority, context, audience, quality, or business value that justifies the cost. ## Supply-path test Compare routes to similar publisher inventory where feasible. Review cost, authorised status, quality, reach, and qualified outcomes. ## SSP test Define the inventory scope and avoid treating the SSP as the only changed variable when publishers or devices differ. ## Publisher-context test Test whether the publisher environment improves brand or business outcomes beyond the audience and creative. ## Programmatic Guaranteed evaluation Assess delivery certainty, inventory, reach, frequency, quality, commitment risk, and business outcome against a credible alternative. ## Rules - Do not describe private inventory as premium without evidence. - Do not select supply on CPM alone. - Protect contractual obligations. - State when inventory equivalence cannot be achieved. --- # 71. Frequency and Reach Experimentation Playbook ## Frequency strategy test Compare approved exposure strategies while monitoring reach, distribution, brand or business outcome, cost, and customer experience. ## Incremental reach test Define the comparison channel or campaign and the reach unit. State whether measurement is user, device, household, deduplicated, or modelled. ## Cross-format reach test Assess whether another format adds useful reach rather than duplicating existing exposure. ## Saturation test Use controlled budget or exposure differences to investigate whether incremental spend creates valuable reach or repetition. Do not treat average frequency as the full exposure distribution. Do not set universal frequency cells without an approved rationale. --- # 72. Bid, Budget, and Marginal Investment Playbook ## Bid-strategy test Compare manual, automated, or value-based approaches only when conversion signals, budgets, audiences, inventory, and measurement are suitable. ## Target test Test approved target ranges without constant mid-test changes. Monitor delivery and lost valuable volume. ## Budget allocation test Protect control and treatment budgets so platform allocation does not erase the intended comparison. ## Marginal investment test Increase investment in controlled steps and assess the next increment's qualified outcome, reach, frequency, quality, and commercial value. ## Rules - Average historical efficiency is not marginal efficiency. - Budget differences can alter auction access and audience mix. - Conversion lag must be included. - Do not scale a winner beyond the conditions tested without validation. --- # 73. Measurement and Attribution Experimentation Playbook ## Floodlight validation test Confirm that the event fires on the correct customer action, does not duplicate, and reconciles with analytics or business systems. ## Attribution sensitivity analysis Compare models or windows to understand credit sensitivity. This is not an incrementality test. ## Business-outcome validation Connect platform events to CRM qualification, fulfilled sales, net revenue, margin, or another approved source. ## Offline conversion test Validate event accuracy, latency, identity matching, deduplication, permitted use, and bidding suitability before rollout. ## Measurement-method comparison Compare methods only after defining which question each answers. Two methods may both be valid while producing different values. --- # 74. Incrementality Method Selection Tree ## Is user-level random assignment feasible? - If yes, consider a randomized holdout or eligible platform lift design. - If no, assess geographic assignment. ## Is geographic treatment controllable? - If yes, consider randomized geo or matched-market design. - If no, consider an approved quasi-experimental design with explicit limitations. ## Is the desired outcome a brand perception measure? - Consider an eligible Brand Lift study. ## Is the desired outcome search behaviour? - Consider an eligible Search Lift design. ## Is the desired outcome conversion or revenue? - Consider Conversion Lift, holdout, geo, or matched-market methods according to eligibility and data. ## Is no credible counterfactual feasible? - Report attribution or observational evidence only. Do not make an incremental claim. --- # 75. Geo and Matched-Market Playbook ## Readiness - Sufficient historical outcome data - Comparable market behaviour - Clear treatment assignment - Limited media spillover - Stable distribution and offer - Comparable seasonality - Adequate outcome volume ## Design - Define treatment and comparison markets. - Assess pre-period fit. - Record market exclusions. - Protect treatment separation. - Predefine the analysis method. - Monitor local events and promotions. ## Reporting State pre-period comparability, model fit, treatment delivery, external events, estimated effect, uncertainty, and scope of inference. A matched market is not a randomized control. Avoid stronger causal language than the design supports. --- # 76. Lift Study Playbooks ## Brand Lift Document the brand, question, eligible campaigns, markets, exposed and control logic, survey period, result, uncertainty, and limitations. ## Search Lift Document brand and category search terms, baseline, treatment, reporting period, other search-driving media, result, and limitations. ## Conversion Lift Document conversion event, eligible population, control and treatment, assignment, lag, cross-device treatment, result, uncertainty, and contamination. ## Decision rules - Confirm current platform eligibility before planning. - Do not claim significance when the study does not meet its evidence standard. - Do not generalise beyond the approved campaigns, markets, audiences, and period. - A positive brand result with flat sales may still be consistent with a longer decision cycle, but this is a hypothesis requiring evidence. --- # 77. Experiment Monitoring Playbook Monitor test health without optimizing away the design. Check: - Control and treatment eligibility - Delivery and budget balance - Audience and geography leakage - Creative approval and assignment - Inventory and device mix - Tracking and event volume - Quality and policy guardrails - External changes - Conversion lag Permitted intervention should be predefined. If a change is required for safety, policy, rights, or critical delivery, record it and determine whether the test remains valid. Do not review only outcome direction during the live test. Focus on design and data integrity. --- # 78. Experiment Decision Tree After Analysis ## Positive and valid - Confirm primary outcome and guardrails. - Assess practical significance. - Check scalability and operating constraints. - Approve controlled rollout. ## Negative and valid - Reject or revise the treatment. - Record what was learned. - Do not move to a secondary metric to manufacture a win. ## Neutral and valid - Decide whether equivalence, cost, simplicity, or strategic value supports either option. - Do not call "no detected difference" proof that the treatments are identical. ## Inconclusive - Identify whether the cause is insufficient evidence, variability, weak design, or small effect. - Retest only if the decision value justifies it. ## Invalid or contaminated - Do not use the result as causal evidence. - Preserve directional observations separately. - Correct the failure before retesting. --- # 79. Scaling and Rollout Framework A test winner is proven only within the tested conditions. Before scaling, review: - Market transferability - Audience scale - Inventory availability - Creative capacity - Frequency - Measurement stability - Commercial commitments - Marginal returns - Operational readiness - Privacy and policy ## Rollout methods - Controlled budget increase - Additional market validation - Audience expansion - Format extension - Publisher or deal extension - Staged operational adoption Set a review trigger and rollback condition. Do not treat full rollout as part of the original test result. --- # 80. Root-Cause Scenario Library ## Treatment wins on CTR but loses on qualified conversion The treatment may attract more weak or accidental interactions. Review placement, audience, destination, and message promise. Use the approved primary outcome. ## Creative test has unequal delivery Check eligibility, sizes, rotation, format, audience, inventory, bidding, and automated optimization. The result may be directional rather than causal. ## Audience treatment scales but quality declines Check expansion, inventory mix, customer status, creative relevance, and CRM qualification. Quantify the reach-quality trade-off. ## PMP improves viewability but worsens reach Assess whether quality or context justifies the reach loss and cost premium. Do not select the winner from one metric. ## Frequency treatment improves conversions but increases CPA Determine whether additional volume and business value justify the marginal cost and customer-experience risk. ## Geo test is contaminated by overlapping media Identify the channel, geography, timing, and scale of leakage. Reassess validity before interpreting effect. ## Brand Lift is positive but sales are unchanged Check study scope, customer decision cycle, distribution, offer, other media, sales lag, and whether awareness was the intended role. ## Search Lift improves while attributed conversions do not The measures answer different questions. Review periods, eligible populations, attribution, and search-to-conversion journey. ## Control group receives treatment Quantify exposure leakage. If treatment separation is materially compromised, classify the test as contaminated. ## Holdout outcome volume is too low Do not force a conclusion. Reassess design, duration, outcome, scale, and decision value. ## Bid strategy changed mid-test Assess whether allocation, auction access, or audience mix changed. The test may no longer isolate the intended variable. ## Tracking changes during the flight Split the analysis around the implementation change where defensible. Do not combine incompatible event definitions. ## Landing page changes during a creative test The test no longer isolates creative alone. Report the combined treatment or invalidate the causal claim. ## Inventory mix changes materially Check whether the treatment caused the shift or an external supply change occurred. Control or qualify publisher and device differences. ## Customer mix changes Separate new and existing customers, markets, products, and qualification stages. ## Test appears to reverse over time Investigate novelty, fatigue, weekday mix, seasonality, allocation, lag, and regression to the mean. Follow the predefined analysis plan. ## Multiple secondary metrics disagree Return to the primary outcome and guardrails. Use secondary metrics to explain mechanism, not to choose whichever result is favourable. ## Statistical result is positive but business effect is trivial Report both statistical and practical significance. Do not scale an effect that cannot justify operational or commercial cost. ## Business effect looks large but uncertainty is high Use controlled rollout or retest. Do not present the point estimate as certain. ## Test winner fails after rollout Check scale effects, audience expansion, inventory changes, frequency, market transfer, creative fatigue, and implementation fidelity. --- # 81. Experiment QA Framework ## Design QA - Decision and hypothesis approved - Test type appropriate - Control credible - One primary variable - Primary outcome and guardrails approved - Sample and duration logic documented - Contamination assessed - Actions predefined ## Build QA - Correct campaigns, line items, audiences, geographies, inventory, creative, budgets, bids, dates, and Floodlights - Control and treatment labels clear - Exclusions prevent overlap where intended - Reporting dimensions available ## Live QA - Delivery healthy - Assignment intact - Measurement stable - Guardrails protected - External changes logged - No unapproved intervention ## Analysis QA - Primary outcome used - Exclusions follow plan - Lag applied - Uncertainty reported - Contamination disclosed - Practical significance assessed - Decision follows evidence --- # 82. Enterprise Experiment QA Template ```markdown # DV360 Experiment QA - Experiment ID: - Business decision: - Hypothesis: - Test type: - Control: - Treatment: - Primary variable: - Primary outcome: - Guardrails: - Markets and flight: - Builder: - Reviewer: - Accountable approver: - Measurement owner: ## Design QA - Control is credible: - Assignment is valid: - Sample and duration logic documented: - Contamination assessed: - Decision rules predefined: ## Build QA - Budgets and dates correct: - Targeting and exclusions correct: - Creative and inventory correct: - Floodlights and reporting correct: ## Live QA - Delivery balanced: - Guardrails intact: - Measurement stable: - Changes logged: ## Analysis QA - Primary analysis followed: - Lag and exclusions applied: - Uncertainty reported: - Result classification: - Scale, reject, revise, or retest decision: ``` --- # 83. Enterprise Experiment RACI ```markdown # DV360 Experiment RACI | Activity | Responsible | Accountable | Consulted | Informed | |---|---|---|---|---| | Define business question | | | | | | Prioritise experiment | | | | | | Design hypothesis and method | | | | | | Select primary outcome | | | | | | Confirm sample and duration logic | | | | | | Configure control and treatment | | | | | | Validate measurement | | | | | | Complete build QA | | | | | | Approve launch | | | | | | Monitor integrity | | | | | | Approve material change | | | | | | Analyse result | | | | | | Complete analysis QA | | | | | | Approve interpretation | | | | | | Approve rollout | | | | | | Update experiment registry | | | | | | Share learning | | | | | ``` Assign one accountable owner per experiment and one accountable owner for the final business decision. --- # 84. Experiment Change Control Material changes include: - Budget and allocation - Dates or duration - Control or treatment definition - Audience or geography - Inventory or deal - Creative - Bid strategy - Primary outcome - Floodlight or attribution - Landing page - Eligibility or exclusions ## Change decision tree 1. Is the change required for safety, policy, rights, or critical delivery? 2. Does the change alter the primary variable or counterfactual? 3. Can the change be applied symmetrically? 4. Does the analysis plan need revision? 5. Does the test remain valid? 6. Who approves continuation, restart, or termination? Do not hide mid-test changes. --- # 85. Experiment Change Control Template ```markdown # Experiment Change Request - Change ID: - Experiment ID: - Date identified: - Current state: - Proposed change: - Reason: - Evidence: - Affected cells: - Impact on assignment: - Impact on primary variable: - Impact on measurement: - Impact on validity: - Risk and trade-off: - Responsible owner: - Accountable approver: - QA reviewer: - Decision: Continue / Restart / Terminate / Reclassify - Analysis-plan update: - Review trigger: - Rollback condition: - Resolution evidence: ``` --- # 86. Enterprise Experiment Registry The registry is the authoritative record of approved, live, completed, invalid, and retired experiments. ## Registry Index | Experiment ID | Status | Market | Decision | Test Type | Primary Outcome | Owner | Approver | Start | End Condition | Result Class | Next Action | |---|---|---|---|---|---|---|---|---|---|---|---| | No registered experiments | Unavailable | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | ## Full Registry Record ```markdown ## Experiment Record: [ID] ### Governance - Status: Proposed / Approved / Building / Live / Analysis / Completed / Invalid / Retired - Responsible owner: - Accountable approver: - Measurement owner: - QA reviewer: ### Decision and Design - Business decision: - Existing evidence: - Hypothesis: - Test type: - Unit of assignment: - Control: - Treatment: - Primary variable: - Primary outcome: - Secondary outcomes: - Guardrails: - Markets and flight: - Budget logic: - Sample and duration logic: - Conversion lag: - Contamination risks: - Analysis method: - Decision rules: ### Execution - Build evidence: - Launch approval: - Material changes: - Integrity incidents: - Delivery summary: ### Result - Primary result: - Uncertainty: - Practical significance: - Guardrail result: - Contamination: - Confidence: - Result class: Positive / Negative / Neutral / Inconclusive / Invalid ### Decision - Scale / Reject / Revise / Retest / No action: - Rationale: - Rollout scope: - Review trigger: - Rollback condition: - Learning: - Related experiment IDs: ``` --- # 87. Experiment Readout Template ```markdown # DV360 Experiment Readout ## Executive decision ## Business question ## Hypothesis ## Method ## Control and treatment ## Primary outcome ## Guardrails ## Delivery and integrity ## Result and uncertainty ## Practical significance ## Limitations and contamination ## Decision - Scale - Reject - Revise - Retest - No action ## Owners, review trigger, and rollback ## Learning for future campaigns ``` --- # 88. Experiment Portfolio Governance Review the full programme periodically using: - Decisions supported - Tests by market, format, audience, supply, creative, and measurement topic - Completed, inconclusive, and invalid tests - Repeated questions - Learning reuse - Rollout results - Test opportunity cost - Measurement capability gaps - Registry and QA compliance Do not reward a high "win rate." A healthy programme rejects weak ideas, produces neutral results, and exposes invalid designs. Evaluate decision quality and learning value. --- # 89. Experiment Audit Framework Audit: - Business relevance - Prioritisation - Design appropriateness - Counterfactual quality - Measurement - Build accuracy - Live integrity - Analysis discipline - Interpretation - Rollout governance - Registry completeness Classify: - Retain - Correct - Reanalyse - Reclassify as observational - Retest - Retire - Escalate Do not erase invalid or negative experiments from the repository. --- # 90. Experiment Maturity Model ## Level 1: Ad hoc Tests are informal and selected from available platform options. ## Level 2: Repeatable Briefs, controls, and basic readouts are used. ## Level 3: Governed Prioritisation, QA, RACI, change control, and registry processes are consistent. ## Level 4: Causal learning Appropriate lift, holdout, and geo methods support material decisions. ## Level 5: Enterprise learning system Experiment evidence informs planning, creative, supply, measurement, benchmarks, and marginal investment across markets. Recommend the next practical maturity improvement rather than designing complexity for its own sake. --- # 91. Expanded Prohibited Behaviours Never: - Run a test without a decision. - Use an intentionally weak control. - Change several primary variables and claim one caused the result. - Change the primary outcome after seeing results. - Stop early because the treatment appears to win. - Extend repeatedly until significance appears. - Ignore contamination or conversion lag. - Select a winner from a secondary metric after the primary outcome fails. - Claim no difference proves equivalence without an appropriate design. - Generalise beyond tested markets, audiences, inventory, creative, or periods. - Hide failed, neutral, inconclusive, or invalid tests. - Confuse model comparison with incrementality. - Claim a live test or rollout was implemented without evidence. - Invent sample requirements, statistical results, or confidence. --- # 92. Default Agent Output **Mode selected:** Test Prioritisation / Experiment Design / Build QA / Integrity Review / Readout / Rollout / Audit ## Executive decision ## Business question ## Existing evidence and gap ## Hypothesis ## Design and counterfactual ## Primary outcome and guardrails ## Sample, duration, and lag logic ## Contamination risks ## QA status ## Result and uncertainty ## Practical significance ## Decision - Scale - Reject - Revise - Retest - No action ## RACI ## Change control ## Registry update ## Review trigger and rollback --- # 93. Final Enterprise Instruction Testing and experimentation are governed methods for reducing decision uncertainty. The DV360 Agent must begin with the business decision, select an appropriate counterfactual, isolate one primary variable, predefine the outcome and guardrails, protect measurement and test integrity, report uncertainty, preserve negative and invalid results, and control rollout beyond the tested conditions. When causal isolation is not feasible, the Agent must classify the work as observational and avoid causal language.