{"id":40158,"date":"2026-04-04T09:44:04","date_gmt":"2026-04-04T07:44:04","guid":{"rendered":"https:\/\/www.graviton.at\/letterswaplibrary\/scaling-a-rag-based-ai-for-student-wellness-how-to-ethically-scrape-curate-500-academic-papers-for-a-white-box-social-science-project\/"},"modified":"2026-04-04T09:44:04","modified_gmt":"2026-04-04T07:44:04","slug":"scaling-a-rag-based-ai-for-student-wellness-how-to-ethically-scrape-curate-500-academic-papers-for-a-white-box-social-science-project","status":"publish","type":"post","link":"https:\/\/www.graviton.at\/letterswaplibrary\/scaling-a-rag-based-ai-for-student-wellness-how-to-ethically-scrape-curate-500-academic-papers-for-a-white-box-social-science-project\/","title":{"rendered":"Scaling A RAG-based AI For Student Wellness: How To Ethically Scrape & Curate 500+ Academic Papers For A “White Box” Social Science Project?"},"content":{"rendered":"

<\/p>\n

\n

Hi everyone!<\/strong><\/p>\n

I\u2019m part of an interdisciplinary team (Sociology + Engineering) at Universidad Alberto Hurtado (Chile)<\/strong>. We are developing Tuk\u00fcyen<\/strong>, a non-profit app designed to foster self-regulation and resilience in university students.<\/p>\n

Our project is backed by the Science, Technology, and Society (STS) Research Center<\/strong>. We are moving away from “Black Box” commercial AIs because we want to fight Surveillance Capitalism<\/strong> and the “Somatic Gap” (the physiological deregulation caused by addictive UI\/UX).<\/p>\n

The Goal:<\/strong> Build a Retrieval-Augmented Generation (RAG)<\/strong> system using a corpus of ~500 high-quality academic papers in Sociology and Psychology (specifically focusing on somatic regulation, identity transition, and critical tech studies).<\/p>\n

The Technical Challenge:<\/strong> We need to move from a manually curated set of 50 papers to an automated pipeline of 500+. We\u2019re aiming for a “White Box AI”<\/strong> where every response is traceable to a specific paragraph of a peer-reviewed paper.<\/p>\n

I\u2019m looking for feedback on:<\/strong><\/p>\n

    \n
  1. Sourcing & Scraping:<\/strong> What\u2019s the most efficient way to programmatically access SciELO, Latindex, and Scopus<\/strong> without hitting paywalls or violating terms? Any specific libraries (Python) you\u2019d recommend for academic PDF harvesting?<\/li>\n
  2. PDF-to-Text “Cleaning”:<\/strong> Many older Sociology papers are messy scans. Beyond standard OCR, how do you handle the removal of “noise” (headers, footers, 10-page bibliographies) so they don’t pollute the embeddings?<\/li>\n
  3. Semantic Chunking for Social Science:<\/strong> Academic prose is dense. Does anyone have experience with Recursive Character Text Splitting<\/strong> vs. Semantic Chunking<\/strong> for complex theoretical texts? How do you keep the “sociological context” alive in a 500-character chunk?<\/li>\n
  4. Vector DB & Costs:<\/strong> We\u2019re on a student\/research budget (~$3,500 USD total for the project). We need low latency for real-time “Somatic Interventions.” Pinecone? Milvus? Or just stick to FAISS\/ChromaDB locally?<\/li>\n
  5. Ethical Data Handling:<\/strong> Since we deal with student well-being data (GAD-7\/PHQ-9 scores), we\u2019re implementing Local Differential Privacy<\/strong>. Any advice on keeping the RAG pipeline secure so the LLM doesn’t “leak” user context into the global prompt?<\/li>\n<\/ol>\n

    Background\/Theory:<\/strong> We are heavily influenced by Shoshana Zuboff<\/strong> (Surveillance Capitalism) and Jonathan Haidt<\/strong> (The Anxious Generation). We believe AI should be a tool for autonomy<\/strong>, not a new form of “zombification” or behavioral surplus extraction.<\/p>\n

    Any advice, repo recommendations, or “don’t do this” stories would be gold!<\/strong> Thanks from the South of the world! \ud83c\udde8\ud83c\uddf1<\/p>\n<\/div>\n

    submitted by \/u\/Spare-Customer-506 <\/a>
    [link]<\/a><\/span> [comments]<\/a><\/span><\/p>