Hey folks, senior data engineer and dlthub cofounder here (dlt = oss python library for data integration)<\/p>\n
Most datasets are behind rest APIS. We created a system by which you can vibe-code a rest api connector (python dict based, looks like config, easy to review) including llm context, a debug app and easy ways to explore your data. <\/p>\n
We describe it as our “LLM native” workflow. Your end product is a resilient, self healing production grade pipeline. We created 8800+ contexts to facilitate this generation but it also works without them to a lesser degree. Our next step is we will generate running code, early next year.<\/p>\n
Blog tutorial with video: https:\/\/dlthub.com\/blog\/workspace-video-tutorial<\/a><\/p>\n And once you created this pipeline you can access it via what we call dataset interface https:\/\/dlthub.com\/docs\/general-usage\/dataset-access\/dataset<\/a> which is a runtime agnostic way to query your data (meaning we spin up a duckdb on the fly if you load to files, but if you load to a db we use that) <\/p>\n More education opportunities from us (data engineering courses): https:\/\/dlthub.learnworlds.com\/<\/a><\/p>\n hope this was useful, feedback welcome<\/p>\n<\/div>\n submitted by \/u\/Thinker_Assignment <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
