Category: Datatards

Here you can observe the biggest nerds in the world in their natural habitat, longing for data sets. Not that it isn’t interesting, i’m interested. Maybe they know where the chix are. But what do they need it for? World domination?

Maternal Mortality Rates In The United States, 2021

Hey Everyone!

I’m having issues attempting to decode the information provided by the CDC. I downloaded the Mortality Multiple Cause File for 2021, and the .txt file is – not only over 2GB, but also incomprehensible. I followed the accompanying .pdf file and was even more confused by its “List of File Data Elements and Tape Locations”, and how I’m supposed to use the file to comprehend a list of codes upon codes upon codes? Especially, when the .txt file has no structure, and when I try to follow a top down approach, codes don’t seem to match.

I wanted to ask if there is a common approach to this, or if I am missing something?

Additional Info:

I am using R for statistical analysis. I wanted the raw data for this reason. I attempted to convert the .txt file to a .csv file format using Python, and it helped by structuring the data a little, but I still don’t know what I am looking at in terms of what it all means.

This is how the rows look now: 11 7101 F1080 422210 4D1 2021U7CN C851129 039 13 0511I509 21I518 31I513 41C851 61M481 05 C851 I509 I513 I518 M481 100 01 184005949020

I would appreciate any, and all help. Thank you all very much in advance.

submitted by /u/Meece156
[link] [comments]

Building A Behavior Classifier For Flies: How To Approach The Problem?

Hey everyone,

I have a dataset that contains the positions of 9 flies for each frame. I want to build a behavior classifier based on this data, but I’m not sure how to approach the problem.

Sample: https://drive.google.com/file/d/1W960Z92f1im80o1l6FveWXBQI5883iRx/view?usp=sharing

My goal is to create an input that takes 9 rows at once, where each row represents the position of one fly, and then learn from it by finding distances between each body part of the flies with each other to determine whether they are touching, grooming, or avoiding.

Additionally, I would like to consider past frames while predicting current frame outputs. Does anyone have suggestions on how to approach this problem? Are there any similar models or approaches that already exist for this?

I’m open to using various machine learning models such as decision trees, support vector machines, or even deep learning models.

If you have any insights or resources that could help me get started, please let me know! Thanks in advance.

submitted by /u/SahilSingh2402
[link] [comments]

Data On Human Beings With Birth And Death Dates?

Hello data people!

I’m interested in any datasets of specific humans (identifying details not needed) but with their specific date of birth AND date of death, i.e. to be able to perform inference procedures on exact ages at death (but with specific dates).

Thank you in advance! I searched a bit but didn’t find exactly those specifics.

submitted by /u/ghabibi
[link] [comments]

117th U.S. Congress Bill Full Text Data Scrap?

I’m looking to scrap the full text for all the proposed bills from the 117th Congress. I want to run the data through NVIVO for content analysis. I tried just downloading all the texts individually from Congress.gov, but I am looking to have all 15,224 documents available for analysis so the one-by-one approach is really unrealistic. I haven’t been able to find this data in a pre-existing dataset, but any assistance would be greatly appreciated!

Of note, I have tried utilizing the Congress.gov API but I can’t figure out how to get all proposed texts. I then tried to run a python script in Google Collab, but I kept getting a “gaierror” error that I couldn’t resolve. I’ve also tried ProPublica and govtrack.us but I couldn’t find a bulk data download option — only a bulk data query for view. I would still have to individually download each bill.

Reference Python Script:

#I removed my API key for privacy purposes, but I assure you it was in the script when I ran it

import requests

import json

def get_bill_data(congress_number):

base_url = “https://api.govinfo.gov

endpoint = “https://api.congress.gov/v3/bill/117/hr/1/text?api_key=DEMO_KEY”.format(congress_number)

api_key = “[SQUATTINGFOX_API_KEY]”

url = base_url + endpoint

headers = {

“X-API-KEY”: api_key,

“Content-Type”: “application/json”

}

response = requests.get(url, headers=headers)

if response.status_code == 200:

data = response.json()

return data

else:

print(“Error retrieving bill data. Status Code:”, response.status_code)

return None

def save_bill_data(data, output_file):

with open(output_file, ‘w’) as file:

json.dump(data, file)

congress_number = “117”

output_file = “bills_data.json”

bill_data = get_bill_data(congress_number)

if bill_data:

save_bill_data(bill_data, output_file)

print(“Bill data saved to”, output_file)

submitted by /u/squattingfox
[link] [comments]

Anyone Here Work For One Of The Big Brokers?

Hey all, I am hoping to learn more about how the big data industry works. Like buying datasets, where you go to find them, how much they cost etc

I’d appreciate any advice or even just a direction to head in. I’ve spoken to Snowflake and Datarade already but they don’t have much insight on what kind of data is actually being purchased or why (apparently anyway)

submitted by /u/Crumbedsausage
[link] [comments]

Looking For: Dataset (Gender) Bias Detection

Hello everyone!

I am planning to test an AI language model for bias – using SHAP and a lexical analyzer – and thus need a dataset I can feed into it. My preferred bias would be gender bias, e.g., a set of statements that are classified as either biased or not biased. However, if such a dataset does not exist I am open for other suggestions or ways how to create such a dataset manually/with AI support.

I am really grateful for any hints/help!

Cheers

submitted by /u/obeseelk
[link] [comments]

Any Way To Search For Similar Datasets?

I don’t know if anything exists like this but I have a data set that shows percent change year over year since 2016 in a specific industry. I took the USA GDP percent change over the same time period (just for curiosity) and there was somewhat of a correlation between the two datasets. My question is, are there any tools that search public data sets for similar percentage changes? I understand there is a high percentage of the correlation being “coincidence”, but does anything like this exist?

submitted by /u/lil_cheeks
[link] [comments]

Found Some More Simple Csv & Excel Datasets For Business Use Cases

I have been looking for simple public datasets in csv & excel format which does not require me to be a data analyst to draw some charts and understand my data. Usually many of the public ones are super complicated plus usually about public topics on health, economy etc. Found some interesting ones on Kaggle already, but just discovered Maven openly available datasets. Here it is https://mavenanalytics.io/data-playground (I have not affiliation to them btw)
Also I find interesting data on Statista, but all is behind an expensive paywall. If you know of any similar but free or cheaper alternative, pls share. thank you 🙂

submitted by /u/andidia82
[link] [comments]

Mental Health Misinformation Dataset

I’ve been searching for datasets regarding misinformation in mental health but I can’t find any. I understand that for ethical reasons they might not be public but even ones that I can fill out a form/ask an author for access to aren’t showing in my search results. I tried asking Chat GPT and Bard which both listed three datasets but when I simply search for those names nothing about them appears and LLMs don’t provide links to their “knowledge”.

Would be very happy if I got any leads.

submitted by /u/homebutnothome
[link] [comments]

Requesting An Images Dataset With Annotated Human Actions To Train Visual Description Model For Accessibility App

Hi everyone, I need help finding a dataset of images annotated with human actions [such as sitting+in-chair, working+on-laptop, etc.]. I found a model capable of generating such tags on Huggingface here, however I was unable to locate its source dataset.

Just for context, I am trying to create a fine-tuned ViT model, that incorporates as broad a set of visual tags as possible. My plan is to optimize this model for edge devices [using Quantization aware training + TFLite model conversion] and open-source the weights. Eventually, I am hoping this can be used for a broad range of visual search/tagging/QnA tasks. Currently, I am training the model on top 2500 Danbooru tags + MIT SUN indoor location tags.

An online demo of the model can be found here. If anyone has any suggestions regarding what other dataset/tags to add, or would like to help with the training efforts, please drop a line. I would really appreciate it.

[Disclosures: I am not affiliated in any way with any of the HuggingFace /Arxiv/Mit.edu links I posted here. The link to the online-demo is maintained by me, but there are no ads or anything else that procures me financial gain on it.]

submitted by /u/DisintegratingBo
[link] [comments]

Where Can I Find Data About Scientific Papers?

Hi,
I am searching for a database of scientific papers, the bigger the better. Like the web of science papers, but I d like to have all the information of the paper; like the abstract text and the introduction (all the paper would be perfect). Information about the authors, academic affiliation, and sector where it has been published

submitted by /u/riegel_d
[link] [comments]