Small Exploratory Dataset: How Much Do ChatGPT/Gemini “best X” Recommendations Change When You Ask The Same Question 3x? (n=10 Questions, All Responses + Code Released)

We ran a small experiment and we’re releasing everything so people can poke holes in it or rerun it.

We asked ChatGPT (web search) and Gemini (grounding) ten “best X for a small business” questions, 3 times each, same day. For each answer we extracted the set of businesses it recommended, then measured overlap across the 3 identical runs (mean pairwise Jaccard).

Result: ~69.5% overlap overall, so roughly a third of the recommended businesses change between identical asks. ChatGPT held steadier (87%) than Gemini (52%). The top 1-3 names stayed locked every run; the lower slots rotated.

Caveats up front: it’s 10 questions, 3 runs, 2 engines, one day. Directional, not definitive, no confidence intervals. We’re releasing it precisely because it’s small, so anyone can rerun with more questions. Extraction was validated (all 290 names appear verbatim in their source answers). Conducted by a company (Pressfront) with an interest in the topic, which is exactly why the full method and every response are public.

Raw JSON, CSVs, the analysis script, and a preprint (DOI: 10.5281/zenodo.22738861) are in the repo, CC BY 4.0.

https://github.com/jjoseph18/ai-recommendation-consistency