we run a small fund and we built a rudimentary system that tracks total headcount over time for the companies we watch. what we actually wanted was headcount by department over time, because total headcount moves too slowly to tell you anything and department level is where the signal is.
we never put much time into it because scraping is a pain, and we do not want to become a data provider. i want the team spending its time on the analysis, not on deciding which proxy to buy this month.
so we looked at buying instead. one vendor has this pattern where you fetch ids first and then collect the data in a second call, which strains our infra and works out around 8 cents a returned company. another returns great data at roughly the same price. most of the rest dont have enough filters to be worth it. and a flat file means we host it ourselves, which is a whole feature build.
where did you draw the build versus buy line on this? and has anybody found department level headcount over time as an actual field rather than a crawl
submitted by /u/_marlowe_
[link] [comments]
