A category-page experiment removed GIF-preview thumbnails for one arm and kept them for the other. Reviewing the readout, an ML engineer asked two questions about how to analyze it properly. Both sounded like the same kind of question — "can we narrow the analysis down to the users who matter?" One of them had to be rejected outright. The other turned out to be encouragement design, and answering it properly required not narrowing anything down at all.
The Experiment, Briefly
Control kept GIF previews on the category page as normal. Treatment removed them entirely, replacing every card with a static image...