I’m looking to scrap the full text for all the proposed bills from the 117th Congress. I want to run the data through NVIVO for content analysis. I tried just downloading all the texts individually from Congress.gov<\/a>, but I am looking to have all 15,224 documents available for analysis so the one-by-one approach is really unrealistic. I haven’t been able to find this data in a pre-existing dataset, but any assistance would be greatly appreciated!<\/p>\n Of note, I have tried utilizing the Congress.gov<\/a> API but I can’t figure out how to get all proposed texts. I then tried to run a python script in Google Collab, but I kept getting a “gaierror” error that I couldn’t resolve. I’ve also tried ProPublica and govtrack.us<\/a> but I couldn’t find a bulk data download option — only a bulk data query for view. I would still have to individually download each bill.<\/p>\n Reference Python Script:<\/p>\n #I removed my API key for privacy purposes, but I assure you it was in the script when I ran it<\/p>\n \u200b<\/p>\n import requests<\/p>\n import json<\/p>\n def get_bill_data(congress_number):<\/p>\n base_url = “https:\/\/api.govinfo.gov<\/a>“<\/p>\n endpoint = “https:\/\/api.congress.gov\/v3\/bill\/117\/hr\/1\/text?api_key=DEMO_KEY”.format(congress_number)<\/a><\/p>\n api_key = “[SQUATTINGFOX_API_KEY]” <\/p>\n url = base_url + endpoint<\/p>\n headers = {<\/p>\n “X-API-KEY”: api_key,<\/p>\n “Content-Type”: “application\/json”<\/p>\n }<\/p>\n response = requests.get(url, headers=headers)<\/p>\n if response.status_code == 200:<\/p>\n data = response.json()<\/p>\n return data<\/p>\n else:<\/p>\n print(“Error retrieving bill data. Status Code:”, response.status_code)<\/p>\n return None<\/p>\n def save_bill_data(data, output_file):<\/p>\n with open(output_file, ‘w’) as file:<\/p>\n json.dump(data, file)<\/p>\n congress_number = “117”<\/p>\n output_file = “bills_data.json”<\/p>\n bill_data = get_bill_data(congress_number)<\/p>\n if bill_data:<\/p>\n save_bill_data(bill_data, output_file)<\/p>\n print(“Bill data saved to”, output_file)<\/p>\n<\/div>\n submitted by \/u\/squattingfox <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
