Partner sends over an export last month. 63 columns, maybe 20 named something a human could read. Rest are val_b, metric_14, one that’s just “flag3.” No ID matching anything in our warehouse, no data dictionary, guy who built it left in 2023.<\/p>\n
Two days gone. Almost none of it pipeline work. Just me squinting at distributions going “this one’s between 0 and 1 and 40% null so it’s probably a rate of something,” then hand-checking matches to see if I was making it up. Which I was, twice.<\/p>\n
So what do you actually do here:<\/p>\n
Profile everything up front, or start joining and work backwards from the wreckage? I go back and forth. Profiling is probably right but I get impatient.<\/p>\n
At what point do you just email whoever sent the file? Held out way too long on this one, mostly pride.<\/p>\n
Anything better than fuzzy matching on a few fields and hoping, when there’s no shared key at all?<\/p>\n
That last one is where I’m stuck. Not the matching, the knowing when to stop trusting it. A 60% confident match is worse than none because it doesn’t fail loudly, it quietly wires two wrong things together and you find out four dashboards later when someone asks why the numbers look weird.<\/p>\n
Curious if anyone has a real system or if it’s vibes and coffee for everyone.<\/p>\n<\/div>\n
submitted by \/u\/donewitheverything26 <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
