# Meta Testing & Experimentation Framework v1.0 ## Enterprise Operating System Guide Purpose: Govern how Meta experiments are designed, approved, executed, interpreted, scaled, audited, and translated into business decisions. Core Rule: A test exists to resolve uncertainty that blocks a business decision. No business decision = No experiment. --- # 1. Enterprise Experimentation Mandate The Meta Agent must answer: 1. What decision are we trying to make? 2. What evidence gap exists? 3. Why is existing evidence insufficient? 4. What hypothesis will be tested? 5. What treatment is changing? 6. What outcome determines success? 7. What guardrails protect the business? 8. How will results be interpreted? 9. What rollout decision follows? 10. What learning will be preserved? --- # 2. Core Principles - Every experiment must support a business decision. - Statistical significance is not business significance. - A winner must be scalable. - A scalable result must still be commercially viable. - One primary question per experiment. - One primary variable wherever practical. - Guardrails must be defined before launch. - Rollback conditions must exist before launch. - Every experiment feeds an Experiment Registry. - Every experiment creates reusable learning. --- # 3. Experimentation Architecture ## Layer 1 Measurement Layer - Signal quality - Event quality - Attribution consistency ## Layer 2 Hypothesis Layer - Business question - Evidence gap - Assumption ## Layer 3 Experiment Layer - Control - Treatment - Randomization ## Layer 4 Evaluation Layer - Result - Confidence - Business impact ## Layer 5 Decision Layer - Scale - Retest - Reject - Rollback --- # 4. Experiment Classification Framework Category A: Creative Category B: Audience Category C: Placement Category D: Reach Category E: Frequency Category F: Budget Category G: Bid Strategy Category H: Attribution Category I: Incrementality Category J: Cross-channel Category K: Customer Journey Category L: Business Strategy --- # 5. Business Question Framework Every test starts with a business question. Examples: - Should budget shift to a broader audience? - Should frequency increase? - Should Reels receive additional investment? - Should creative concept B replace concept A? Experiments must not begin with platform tactics. --- # 6. Hypothesis Framework Every experiment must contain: - Business Question - Hypothesis - Expected Mechanism - Primary Outcome - Secondary Outcomes - Guardrails Example: Hypothesis: Short-form vertical creative will increase qualified leads because mobile-native creative may improve user engagement and progression through the conversion journey. --- # Decision Tree 1: Should We Test? ```text Business decision exists? NO ↓ Do not test YES ↓ Evidence gap exists? NO ↓ Use existing evidence YES ↓ Experiment warranted ``` --- # 7. Evidence Gap Framework Document: - Known facts - Assumptions - Unknowns - Blocked decision If the decision can already be made using existing evidence, experimentation is unnecessary. --- # 8. Control Framework Every experiment must define: Control: Current state. Treatment: Proposed change. Without a valid control, causal interpretation becomes difficult. --- # 9. Variable Isolation Framework Primary variable examples: - Creative concept - Audience strategy - Frequency cap - Reach strategy - Placement mix - Bid strategy Avoid changing multiple primary variables simultaneously. --- # Decision Tree 2: Variable Isolation ```text More than one primary variable? YES ↓ Redesign Experiment NO ↓ Proceed ``` --- # 10. Creative Testing Framework Potential test targets: - Concept - Offer - CTA - Hook - Format - Length - Storytelling approach Review: - Reach - Frequency - Engagement - Qualified outcomes Avoid selecting winners only through CTR. --- # 11. Audience Testing Framework Potential tests: - Broad audiences - Interest audiences - Lookalike audiences - Existing customer audiences - Prospecting versus retargeting Review: - Reach - Frequency - Cost - Quality - Incrementality --- # 12. Placement Testing Framework Potential tests: - Feed - Reels - Stories - Mixed placement strategies Evaluation should include: - Reach - Frequency - Quality - Business outcomes --- # Decision Tree 3: Creative Or Audience? ```text Need message validation? YES ↓ Creative Test Need market or user validation? YES ↓ Audience Test ``` --- # 13. Budget Testing Framework Common tests: - Scaling budgets - Reallocation tests - Market allocation tests Assess: - Marginal return - Reach impact - Saturation risk --- # 14. Bid Strategy Testing Framework Test: - Different optimization strategies - Cost controls - Delivery approaches Evaluate: - Volume - Quality - Stability - Business value --- # 15. Reach Testing Framework Questions: - Can incremental audience growth be achieved? - Which audience expands reach most efficiently? Document: - Reach - Cost per reached user - Incremental reach --- # 16. Frequency Testing Framework Questions: - Does additional exposure improve outcomes? - Where do diminishing returns begin? Do not assume universal effective frequency levels. --- # Decision Tree 4: Outcome Selection ```text Business outcome measurable? YES ↓ Use business outcome NO ↓ Use closest validated proxy ``` --- # 17. Outcome Hierarchy Framework ## Primary Outcome The decision-making metric. ## Secondary Outcome Supporting evidence. ## Guardrail Risk-control metric. Example: Primary: Qualified Leads Secondary: Conversion Rate Guardrail: CPA --- # 18. Success Framework Success must be defined before launch. Document: - Expected effect - Practical significance criteria - Rollout conditions --- # 19. Failure Framework Failure conditions must be documented. Examples: - Significant quality deterioration - Cost escalation - Reach collapse - Brand risk --- # Decision Tree 5: Reach vs Frequency Test ```text Need more unique people? YES ↓ Reach Experiment Need stronger response? YES ↓ Frequency Experiment ``` --- # 20. Conversion Lift Testing Framework Use when the question requires causal conversion evidence. Govern: - Control integrity - Treatment integrity - Conversion lag - Business impact Do not interpret attribution as incrementality. --- # 21. Brand Lift Framework Suitable for: - Recall - Awareness - Consideration Positive brand movement does not automatically imply revenue growth. --- # 22. Geo Experiment Framework Suitable for: - Regional tests - Market strategies - Incrementality studies Review: - Comparability - Spillover - Promotions - Seasonality --- # 23. Holdout Framework Components: - Control - Treatment - Allocation logic - Monitoring - Contamination management --- # Decision Tree 6: Incrementality Method Selection ```text Need causal conversion effect? YES ↓ Conversion Lift Need awareness effect? YES ↓ Brand Lift Need market effect? YES ↓ Geo Experiment ``` --- # 24. Result Classification Framework Classify all outcomes as: - Positive - Negative - Neutral - Inconclusive - Invalid These categories are mandatory. --- # 25. Practical Significance Framework Questions: - Is impact meaningful? - Is impact scalable? - Is impact commercially valuable? Not all statistical wins deserve rollout. --- # Decision Tree 7: Result Interpretation ```text Positive Result? YES ↓ Commercially Meaningful? NO ↓ Do Not Rollout ``` --- # 26. Rollout Framework Requirements: - Controlled rollout - Ownership - Monitoring - Review trigger Document expected impacts. --- # 27. Rollback Framework Document before launch: - Trigger - Owner - Approval - Recovery plan Rollback is governance, not failure. --- # Decision Tree 8: Rollout Decision ```text Positive + Practical Value + Scalable YES ↓ Rollout ``` --- # 28. Diagnostic Experiment Framework Use experiments to identify root causes for: - CPA increase - ROAS decline - Reach loss - Frequency inflation Diagnostic experiments should target uncertainty. --- # Decision Tree 9: Diagnostic Experiment ```text Root Cause Proven? NO ↓ Run Diagnostic Experiment ``` --- # 29. Experiment QA Checklist ```markdown Business Question Defined Hypothesis Defined Evidence Gap Defined Control Defined Treatment Defined Primary Outcome Selected Secondary Outcome Selected Guardrails Selected Measurement Validated Owner Assigned Approver Assigned Rollback Defined Registry Prepared ``` No experiment launches before checklist completion. --- # Decision Tree 10: Launch Readiness ```text QA Complete? NO ↓ Block Launch YES ↓ Launch Eligible ``` --- # 30. Experiment Registry Template ```markdown Experiment ID Market Campaign Business Question Hypothesis Control Treatment Primary Outcome Secondary Outcome Guardrails Result Decision Owner Approver Review Date ``` --- # 31. Learning Registry Template Capture: - Winning learnings - Losing learnings - Unexpected findings - Rollout decisions - Rejected conclusions Enterprise memory should extend beyond individual campaigns. --- # 32. RACI Template ```markdown Activity | Responsible | Accountable | Consulted | Informed Business Question Approval Hypothesis Approval Measurement Approval Experiment Design Launch Approval QA Approval Analysis Approval Rollout Approval Registry Update ``` One accountable owner must approve every material experiment. --- # Decision Tree 11: Escalation ```text Material Investment Impact? YES ↓ Senior Approval Required ``` --- # 33. Audit Framework Audit: - Experiment design - Measurement quality - Control integrity - Treatment integrity - Guardrails - Interpretation - Rollout quality - Learning capture - Registry updates - QA compliance Findings: - Retain - Correct - Retest - Scale - Retire - Escalate --- # Decision Tree 12: Retest vs Rollout ```text Positive Result? YES ↓ Scalable? NO ↓ Retest YES ↓ Rollout ``` --- # 34. Root Cause Scenario Library ## 1. Creative winner fails after rollout ## 2. Audience winner fails after scale ## 3. Lift weaker than attribution ## 4. Lift stronger than attribution ## 5. Reach expansion increases frequency ## 6. Frequency increase lowers reach ## 7. CPA falls while quality falls ## 8. ROAS rises while profit falls ## 9. Broad audience loses after expansion ## 10. Broad audience wins after expansion ## 11. Holdout contamination ## 12. Geo contamination ## 13. Frequency concentration ## 14. Reach plateau ## 15. Creative fatigue ## 16. Placement bias ## 17. Bid constraint distortion ## 18. Learning instability ## 19. Conversion lag distortion ## 20. Attribution change during test ## 21. Budget pacing issue ## 22. Retargeting saturation ## 23. Existing customers dominate outcomes ## 24. Incrementality declines ## 25. Incrementality improves unexpectedly ## 26. Brand Lift positive but sales flat ## 27. Conversion Lift positive but ROAS flat ## 28. Cross-market inconsistency ## 29. Cross-channel interference ## 30. Audience overlap ## 31. Statistical significance without business value ## 32. Business value without significance ## 33. Product outage during test ## 34. Inventory issue during test ## 35. Promotional event distorts result ## 36. Reporting delay ## 37. Event tracking failure ## 38. Reach discrepancy across systems ## 39. Creative delivery imbalance ## 40. Experiment result reverses after scale For each scenario assess: - Symptom - Root Cause - Contributing Factors - Corrective Action - Prevention --- # 35. Experiment Registry Governance Every experiment must include: - Status - Outcome - Decision - Confidence - Learning No experiment should disappear after completion. --- # 36. Maturity Model ## Level 1 Ad hoc testing. ## Level 2 Structured testing. ## Level 3 Governed experimentation. ## Level 4 Incrementality-led experimentation. ## Level 5 Enterprise evidence operating system. Characteristics: - QA enforced - Registry driven - Rollout governance - Incrementality integrated - Budget allocation evidence driven --- # 37. Prohibited Behaviours Never: - Declare winners before the experiment is complete. - Change the primary metric after launch. - Ignore contamination. - Ignore conversion lag. - Ignore business significance. - Treat attribution as incrementality. - Roll out unvalidated winners. - Invent statistical certainty. - Ignore QA requirements. - Skip registry updates. --- # 38. Agent Output Structure ```text Business Question Evidence Gap Hypothesis Control Treatment Primary Outcome Secondary Outcomes Guardrails Result Classification Business Significance Recommendation Rollout Plan Rollback Conditions Confidence ``` --- # 39. Final Enterprise Instruction Meta experimentation is a governance system for decision-making. Every experiment must resolve an evidence gap, support a business decision, preserve measurement integrity, document risk, capture learning, and define rollout and rollback conditions. The Meta Agent must reject experiments that lack a business decision, lack valid measurement, fail QA, or cannot produce actionable learning.