Been trying to build a dataset of tweets for a research project (analyzing discourse patterns around specific topics) and the official X API is basically unusable unless you want to drop $5k+/month for reasonable limits.
I’ve tried a few different approaches:
- Official API → rate limits killed me immediately
- Manual scraping → got my IP banned within a day
- Some random npm packages → half of them are broken now
Found a breakdown comparing different methods and it actually explained why most DIY scrapers fail (anti-bot stuff has gotten way more aggressive lately). Makes sense why so many tools just stopped working after Elon’s changes.
Anyone here working with Twitter data regularly? What’s actually reliable right now? Need something that can pull ~50k tweets/day without constant babysitting.
Not trying to do anything shady – just need public tweet text, timestamps, and basic engagement metrics for academic analysis.
submitted by /u/Technical_Fee4829
[link] [comments]
