The Control Group Playbook for AI Citation Content Tests

Before you select a single control page, write down the unit of observation, the numerator definition, the denominator definition, and every exclusion rule. This is your prespecification document. Its purpose is to prevent you from adjusting any of these definitions after you see the data.

Start with the unit: are you measuring a single URL, a content template, or a domain? Each choice produces a different denominator. Next, build a fixed query set of non-branded informational queries that your content could plausibly answer. The size of that set is a design decision you make based on the topic coverage of your test pages - there is no universal number. Exclude brand-navigation queries where you already dominate and queries that return no AI-generated answer at all. Each query in the set is one opportunity, so your denominator is the count of queries multiplied by the number of pages if you test multiple pages at the URL level.

Define what counts as a citation: an exact URL match in the AI answer's source list, or a brand mention with a link? For content tests, exact URL match is the cleaner signal because brand mentions vary by provider and session state. Document the session conditions - signed-out, no personalization, consistent locale - and record them in the prespecification document.

A/B Testing 101 - NN/G states: "A/B testing can help UX teams determine the improvements in the user experience that are best for their business goals ."

You need matched pairs: for each page you intend to change (treatment), select a control page that had a similar citation rate in the baseline period, covers a comparable subtopic, and was published in a similar timeframe.

Run your fixed query set against both groups before any content change and record the citation count for each page. Calculate the average citation rate and its distribution for each group. If the control group's average rate is materially higher than the treatment group's, your post-test comparison will be biased upward - you will understate the treatment effect. If it is materially lower, you will overstate it. Rebalance the groups until the pre-treatment rates are close. If you cannot find adequate matches within your own site, pause the test and expand your topic coverage first rather than proceeding with a mismatched control.

Record the matching criteria in your prespecification document: which pages are in each group, why they were selected, and what the pre-treatment citation rates were. This record is what makes the design auditable.

Pre-registration is the single step that separates a credible controlled observation from a post-hoc rationalization. Write every parameter of your test in a document, date it, and share it with at least one other person before you make any content change. The document must include: the treatment description (exactly what you will change and on which pages), the control group list, the fixed query set, the providers you will query, the primary metric formula, the exclusion rules, and the planned analysis method.

The timeline structure matters. After the baseline period, assign control and treatment units formally (record the date). Make the content change. Then wait a post-treatment observation period before running the post-intervention queries.

You set this window based on your knowledge of the providers you are testing; there is no universal duration that applies across all providers and topics.

After the observation window closes, run the analysis exactly as prespecified. Do not add queries, swap pages, or change the citation definition. If you discover a data-collection error, document it and report it as a limitation rather than silently correcting it.

Executing the measurement requires a scripted approach to avoid manual inconsistencies. Use the exact same prompt format and session state for every query in every collection period.

A screenshot folder is not a structured log - it cannot be queried or audited. If you automate collection, record the script version and any API parameters alongside the data so the collection method is reproducible.

Calculate citation rate for each group and period: citations divided by opportunities. Then calculate the difference-in-differences. A synthetic illustration (labeled as such): if treatment pages gained eight citations and control pages gained three citations over the same period, the adjusted treatment effect is five - not eight. The control group corrected for three citations that appeared regardless of your content change. Real data will not be this clean; document the actual numbers and their variance.

Assign an evidence class before you share the result. The class communicates what the design can support and what it cannot. Three levels cover most content test designs: Level 1 is an uncontrolled observation - a before-and-after on a single group with no control. It carries no causal weight and should be labeled as such. Level 2 is a controlled observation - a matched control group with a pre-post design and a difference-in-differences calculation. It suggests attribution but cannot rule out selection bias or unmeasured confounders.

Level 3 is a randomized experiment - random assignment of content to treatment and control with a holdout, which is rare in content testing but produces the strongest attributable evidence.

Prespecify the evidence class you are aiming for before the test. That is a defensible result for a skeptical stakeholder. What you cannot do is run a Level 1 design and report it as if it were Level 2.

Your pre-registration document should state the evidence class, the known limitations, and the confounders the design cannot exclude. When you share the result, lead with the evidence class label before the number.