I am currently doing a massive analysis of Reddit's entire publicly available comment dataset. The dataset is ~1.7 billion JSON objects complete with the comment, score, author, subreddit, position in comment tree and other fields that are available through Reddit's API.

I'm currently doing NLP analysis and also putting the entire dataset into a large searchable database using Sphinxsearch (also testing ElasticSearch).

This dataset is over 1 terabyte uncompressed, so this would be best for larger research projects. If you're interested in a sample month of comments, that can be arranged as well. I am trying to find a place to host this large dataset -- I'm reaching out to Amazon since they have open data initiatives.

EDIT: I'm putting up a Digital Ocean box with 2 TB of bandwidth and will throw an entire months worth of comments up (~ 5 gigs compressed) It's now a torrent. This will give you guys an opportunity to examine the data. The file is structured with JSON blocks delimited by new lines (\n).

____________________________________________________

One month of comments is now available here:

Download Link: Torrent

Direct Magnet File: magnet:?xt=urn:btih:32916ad30ce4c90ee4c47a95bd0075e44ac15dd2&dn=RC%5F2015-01.bz2&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969

Tracker: udp://tracker.openbittorrent.com:80

Total Comments: 53,851,542

Compression Type: bzip2 (5,452,413,560 bytes compressed | 31,648,374,104 bytes uncompressed)

md5: a3fc3d9db18786e4486381a7f37d08e2 RC_2015-01.bz2

____________________________________________________

Example JSON Block:

{"gilded":0,"author_flair_text":"Male","author_flair_css_class":"male","retrieved_on":1425124228,"ups":3,"subreddit_id":"t5_2s30g","edited":false,"controversiality":0,"parent_id":"t1_cnapn0k","subreddit":"AskMen","body":"I can't agree with passing the blame, but I'm glad to hear it's at least helping you with the anxiety. I went the other direction and started taking responsibility for everything. I had to realize that people make mistakes including myself and it's gonna be alright. I don't have to be shackled to my mistakes and I don't have to be afraid of making them. ","created_utc":"1420070668","downs":0,"score":3,"author":"TheDukeofEtown","archived":false,"distinguished":null,"id":"cnasd6x","score_hidden":false,"name":"t1_cnasd6x","link_id":"t3_2qyhmp"}

UPDATE (Saturday 2015-07-03 13:26 ET)

I'm getting a huge response from this and won't be able to immediately reply to everyone. I am pinging some people who are helping. There are two major issues at this point. Getting the data from my local system to wherever and figuring out bandwidth (since this is a very large dataset). Please keep checking for new updates. I am working to make this data publicly available ASAP. If you're a larger organization or university and have the ability to help seed this initially (will probably require 100 TB of bandwidth to get it rolling), please let me know. If you can agree to do this, I'll give your organization priority over the data first.

UPDATE 2 (15:18)

I've purchased a seedbox. I'll be updating the link above to the sample file. Once I can get the full dataset to the seedbox, I'll post the torrent and magnet link to that as well. I want to thank /u/hak8or for all his help during this process. It's been a while since I've created torrents and he has been a huge help with explaining how it all works. Thanks man!

UPDATE 3 (21:09)

I'm creating the complete torrent. There was an issue with my seedbox not allowing public trackers for uploads, so I had to create a private tracker. I should have a link up shortly to the massive torrent. I would really appreciate it if people at least seed at 1:1 ratio -- and if you can do more, that's even better! The size looks to be around ~160 GB -- a bit less than I thought.

UPDATE 4 (00:49 July 4)

I'm retiring for the evening. I'm currently seeding the entire archive to two seedboxes plus two other people. I'll post the link tomorrow evening once the seedboxes are at 100%. This will help prevent choking the upload from my home connection if too many people jump on at once. The seedboxes upload at around 35MB a second in the best case scenario. We should be good tomorrow evening when I post it. Happy July 4'th to my American friends!

UPDATE 5 (14:44)

Send more beer! The seedboxes are around 75% and should be finishing up within the next 8 hours. My next update before I retire for the night will be a magnet link to the main archive. Thanks!

UPDATE 6 (20:17)

This is the update you've been waiting for!

The entire archive:

magnet:?xt=urn:btih:7690f71ea949b868080401c749e878f98de34d3d&dn=reddit%5Fdata&tr=http%3A%2F%2Ftracker.pushshift.io%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80

Please seed!

UPDATE 7 (July 11 14:19)

User /u/fhoffa has done a lot of great work making this data available within Google's BigQuery. Please check out this link for more information: /r/bigquery/comments/3cej2b/17_billion_reddit_comments_loaded_on_bigquery/

Awesome work!

Dataset: https://huggingface.co/datasets/datasocial/tiktok-5.6B-videos

If you want to explore the data without downloading billions of rows, you can query my ClickHouse database directly. It includes:

Creators table - 4.5 billion rows
Videos table - 5.6 billion rows
Sounds table - 633 million rows

Comment below and I’ll DM you the database credentials.

Im self-hosting my database so please don't run heavy queries and crash my server.

You have probably seen most of these, but I thought I'd share anyway:

Spreadsheets and Datasets:

• https://www.worldometers.info/coronavirus/
• John Hopkins University Github confirmed case numbers.
• Google Sheets From DXY.cn (Contains some patient information [age,gender,etc] )
• Kaggle Dataset
• Strain Data repo
• https://covid2019.app/ (Google Sheets, thanks /u/supertyler)
• ECDC (Daily Spreadsheets, Thanks /u/n3ongrau)

Other Good sources:

• BNO Seems to have latest number w/ sources. (scrape)
• What we can find out on a Bioinformatics Level
• DXY.cn Chinese online community for Medical Professionals *translate page.
• John Hopkins University Live Map
• Mutations (thanks /u/Mynewestaccount34578)
• Protein Data Bank File
• Early Transmission Dynamics Provides statistics on the early cases, median age, gender etc.

[IMPORTANT UPDATE: From February 12th the definition of confirmed cases has changed in Hubei, and now includes those who have been clinically diagnosed. Previously China's confirmed cases only included those tested for SARS-CoV-2. Many datasets will show a spike on that date.]

There have been a bunch of great comments with links to further resources below!
[Last Edit: 15/03/2020]

Hi all,

I needed a country / state / city list for ShopClass, our open source classifieds app. Sounds easy but every option I found had a catch. Either the licence needed attribution or was share-alike, or it was a paid API with keys and rate limits, or the data was old and half the cities had no coordinates.

Why no other share-alike database considered?
See, I am happy to give attribution, buty anyone using my CMS for there website they need to give it too, which I think is a limitation. So I created this no-attribution dataset.

So I built our own dataset from Wikidata and released it as CC0. Public domain, no attribution needed, use it in anything including commercial stuff.

What's in it:

• 255 countries and territories, 4,314 administrative divisions (states, provinces, regions) and 1,671,395 cities and towns

• every single place has latitude and longitude, if wikidata has no position for it we don't ship it

• Wikidata QID as the id, so when you re-import a later release a renamed place stays the same row

• population, timezone, GeoNames id where known

• countries have capital, currency, calling code, ISO alpha-2/3 and numeric, flag emoji etc

How to get it:

• plain JSON, one file per country, with sha256 checksums for every file

• or call it from the browser or your server, no API key, no account, CORS is open: https://api.placedb.org/v1/countries.json

• there are prefix files for autocomplete too, so a city search box can work without a backend

• the pipeline that builds it is GPLv3 and the output is deterministic, same wikidata dump gives byte-identical files

Also, it's not perfect and here is where:

• about 883k settlements are dropped because wikidata has no coordinates for them, mostly villages in China, India, Russia, Uganda and Myanmar

• names that exist only in Arabic script (and some other scripts) are not transliterated yet, so those places are missing a usable name

• Vietnam is undercounted

All of it is in the LIMITATIONS doc in the repo, with numbers.

Site: https://placedb.org

Repo and the full limits doc: https://github.com/mindstellar/location-data

If you spot a wrong place, a missing country division or something silly, please tell me. Happy to take PRs too.

Hey everyone,

I wanted to share a dataset I published for anyone working on low-resource NLP, tokenization, or language modeling for Haitian Creole (Kreyòl ayisyen): the Haitian Creole Word Frequency dataset (haitian-creole-word-freq), now live on both Hugging Face and Kaggle.

Overview

This is a word frequency list for Haitian Creole built from the Carnegie Mellon University (CMU) Haitian newswire corpus. It contains 17,947 unique lowercase words with their occurrence counts, sorted in descending order by count.

Links

• Hugging Face: https://huggingface.co/datasets/Speedk4011/haitian-creole-word-freq
• Kaggle: https://www.kaggle.com/datasets/spdk4011/haitian-creole-word-freq

Potential Use Cases

• Stopword Candidates: Extracting function words from the top of the frequency list (e.g., te, yo, nan, yon, li, pou, ki, ...).
• Tokenizer Customization: Fine-tuning or building BPE/WordPiece vocabularies for low-resource LLMs.
• Spellcheck and Auto-correct: Prioritizing candidate suggestions by word popularity.
• Vocabulary Membership Tests and Orthographic Checks: Testing if a token is spelled like a Haitian Creole word or checking lexical presence.
• Lexicography and Language Learning: Extracting core vocabulary lists based on news text.
• Language Identification and N-gram Models: Statistical language modeling for text classification pipelines.
• Testing Fixtures: Generating reproducible data inputs for unit testing Haitian Creole NLP pipelines.

Unified Epstein Estate Archive (House Oversight, DOJ, Logs, & Multimedia)

TL;DR: I am aggregating all public releases regarding the Epstein estate into a single repository for OSINT analysis. While I finish processing the data (OCR and Whisper transcription), I have opened a Google Drive for public access to the raw files.

Project Goals:

This archive aims to be a unified resource for research, expanding on previous dumps by combining the recent November 2025 House Oversight releases with the DOJ’s "First Phase" declassification.

I am currently running a pipeline to make these files fully searchable:

• OCR: Extracting high-fidelity text from the raw PDFs.
• Transcription: Using OpenAI Whisper to generate transcripts for all audio and video evidence.

Current Status (Migration to Google Drive):

Due to technical issues with Dropbox subfolder permissions, I am currently migrating the entire archive (150GB+) to Google Drive.

• Please be patient: The drive is being updated via a Colab script cloning my Dropbox. Each refresh will populate new folders and documents.
• Legacy Dropbox: I have provided individual links to the Dropbox subfolders below as a backup while the Drive syncs.

Future Access:

Once processing is complete, the structured dataset will be hosted on Hugging Face, and I will release a Gradio app to make searching the index user-friendly.

Please Watch or Star the GitHub repository for updates on the final dataset and search app.

Access & Links

Content Warning: This repository contains graphic and highly sensitive material regarding sexual abuse, exploitation, and violence, as well as unverified allegations. Discretion is strongly advised.

• Google Drive Archive (Primary Source - Currently Syncing)
• GitHub Repository (Documentation & Updates)
• Original Repo for 20k Emails (Contains Nov dataset & Gradio app)

Dropbox Subfolders (Backup/Individual Links):

• Subfolder 1
• Subfolder 2
• Subfolder 3
• Subfolder 4
• Subfolder 5
• Subfolder 6
• Subfolder 7

Note: If prompted for a password on protected folders, use my GitHub username: theelderemo

Edit: It's been well over 16 hours, and data is still uploading/processing. Be patient. The google drive is where all the raw files can be found, as that's the first priority. Dropbox is shitty, so i'm migrating from it

Edit: All files have been uploaded. Currently manually going through them, to remove duplicates.

Update to this: In the google drive there are currently two csv files in the top folder. One is the raw dataset. The other is a dataset that has been deduplicated. Right now, I am running a script that tries to repair the OCR noise and mistakes. That will also be uploaded as a unique dataset.

I’ve spent the last few weeks working on a pipeline to solve a problem that has frustrated me (and likely other researchers) for a while: working with arXiv source files at scale.

If you have ever tried to analyze the LaTeX source code of arXiv papers, you have probably run into two major roadblocks:

1. The Egress Tax: arXiv’s official bulk S3 bucket is configured as "requester-pays." If you try to download the complete 5 TB corpus to any machine outside of the AWS us-east-1 region, you get hit with standard egress fees. At $0.09 per GB, a single full download can cost over $450 in bandwidth alone.
2. Unpacking Pain: The raw S3 data is packaged as hundreds of nested .tar archives containing gzipped payloads of individual papers. Extracting these, parsing the inner LaTeX code, and matching the files with their JSON metadata snapshots is quite CPU-intensive and requires a lot of boilerplate ingestion code.

To make this easier, I built a pipeline that runs inside AWS us-east-1 (where transfer is free), pulls the raw source files, unpacks them, matches them with the official metadata, and bundles them into ready-to-query Parquet partitions.

• HuggingFace Dataset Link: https://huggingface.co/datasets/scholarweave/arxiv

What is inside:

Each row represents a single paper and contains both the official metadata and the parsed source files:

• Core Metadata: id, title, authors, abstract, doi, categories, license, versions, etc.
• latex (Large String): The parsed, compiled LaTeX source code from the paper. I wrote a parser to bundle the primary .tex, .bib, and .sty files into a single, readable Markdown-style tree structure.

Maintenance & Syncing:

• Monthly Updates: I plan to sync the pipeline once a month to capture new uploads.
• Resilient Syncing: I maintain an XML manifest file in the HuggingFace repository (arxiv_parquet_manifest.xml) that maps each Parquet partition to its size, MD5 checksum, and the raw S3 .tar source files used to generate it. This should make incremental syncing or troubleshooting much easier.

If you are working on NLP, training LLMs on scientific text, analyzing citation networks, or doing sociolinguistic research, hopefully this saves you some time and cloud budget.

Hi everyone!

I'm a Master's student in Statistics at the University of Padua (Italy), currently developing my thesis on the application of Survival Analysis and Machine Learning to mortgage credit risk.

I'm looking for a European residential mortgage dataset with longitudinal loan-level observations, ideally containing:

- Loan characteristics (LTV, interest rate, loan amount, maturity, etc.).

- Repayment or performance histories.

- Default events and their timing.

- Early repayment/prepayment events, if available.

- Enough information to reconstruct loan durations and right-censoring.

My goal is to compare traditional survival models (such as Cox proportional hazards) with Survival Machine Learning methods, potentially including competing risks for default and prepayment.

I'm aware of the Freddie Mac Single-Family Loan-Level Dataset, but I would particularly like to explore European mortgage markets.

So far, I've investigated European DataWarehouse, De Nederlandsche Bank, and several academic studies using proprietary banking datasets. I've also contacted some institutions to enquire about academic access.

Does anyone know of other European mortgage datasets, research repositories, or academic collaborations that might be worth exploring?

I'm also interested in datasets that aren't publicly downloadable but might be accessible to university researchers under specific agreements.

Any suggestions, relevant papers, or contacts would be greatly appreciated.

Thanks in advance!

Created a dataset of fund returns on Snowflake. It's quite extensive showing both trailing and calendar year returns for every mutual fund and ETF since inception.

You can fetch a mutual funds trailing annualized performance every trading day going back 15+ years.

Each trailing returns row shows: 1D, 1W, 1M, 2M, 3M, 1Y, 2Y, 3Y, 4Y, 5Y, 7Y, 10Y, 12Y, 15Y, and earliest available inception. I usually never see websites (even Morningstar) provide this breakdown which I think is useful when exploring different economic cycles.

Calendar year returns go back as far as possible as well.

Data is updated daily.

​

I have a ~59 GB Parquet dataset stored in Google Drive, and I want to use it for ML training on Kaggle GPUs.

My folder structure is:

My Drive/

└── DATASET/

└── data/

└── processed/

└── era5_land_preprocessed/

├── train/ → 60 Parquet files → 44.34 GB

├── validation/ → 12 Parquet files → 8.83 GB

└── test/ → 8 Parquet files → 5.82 GB

Total: 80 Parquet files, ~58.99 GB

I don't want to download the dataset to my local computer. I was planning to use Google Colab + Kaggle API to transfer the files directly from Google Drive to a Kaggle Dataset.

My questions are:

1. Is uploading a ~59 GB dataset to Kaggle this way reliable?

2. Is Google Drive → Colab → Kaggle API the best approach?

3. Should I create one Kaggle dataset containing "train/", "validation/", and "test/", or split them into multiple Kaggle datasets?

4. Are there any problems with uploading large Parquet files to Kaggle?

5. Is there a better/faster method for transferring this amount of data to Kaggle?

The dataset will be used for climate/weather ML forecasting, and I need to train models using Kaggle's GPUs.

Any advice from people who have handled 50+ GB datasets on Kaggle would be appreciated.

Hey all, I work at CompanyData. A while back we published free business datasets for a few major cities on Datahub: https://datahub.io/@companydatadotcom (i can also share a link from our own website, just let me know in comments)

Each dataset covers companies in one city, sourced from official government registries. Current fields per company: registered address, registration number, SIC industry code, revenue, employee count, and ownership information through headquarter linkages, so you can see which companies belong to the same group. The Berlin one turned out to be the most popular by far, which we didn't expect.

No commercial angle here, downloads are free. We publish these because registry-sourced company data should be easier to get your hands on than it is.

What I'd really like feedback on is the fields themselves. Which ones actually matter for your work, and what's missing? Founding date, legal form, ownership links, NACE instead of SIC, officer counts, website domains? We decide what goes into the next batch based on what people here say, so be critical. Also open to suggestions for the next city or country.

I’m conducting research on project management and need a dataset that includes Microsoft Teams or Slack messages (messages from other platforms are also acceptable) exchanged within a project managed through Jira.

Ideally, the dataset should contain both the Jira project data and the associated communication history, so the messages can be linked to a specific project created in Jira.
How can I collect or obtain such a dataset for my research?

For anyone practicing country-level joins or exploring water access, these two CSVs use the same reporting year and country IDs:

• At least basic drinking water: 217 country/area rows, with 202 reported values
• Safely managed drinking water: 217 country/area rows, with 148 reported values

Original data: WHO/UNICEF Joint Monitoring Programme (JMP), distributed through the World Bank’s World Development Indicators. At least basic source · Safely managed source

Disclosure: I work on Global Data Tracker, which hosts the CSV exports below. They’re free to download without an account.

Download at least basic, 2024 CSV Download safely managed, 2024 CSV

Each file includes country_id, country, value, unit, year, source, publication and provider-note fields. Blank values mean no published observation for that selection; they are not zeros. The files retain all 217 countries/areas, including those with missing values. World and regional aggregate rows are excluded.

A useful exercise: join on country_id, count missing values first, then compare the two percentages for the 148 countries/areas with both observations. “At least basic” already includes safely managed service, so don’t add the columns. Safely managed also requires an improved source on premises, available when needed and free from specified contamination. JMP definitions

These links pin the saved September 19, 2026 release; the observations are for 2024. Checked October 10: the values and missing entries in both files match the current World Bank API. National estimates can hide local differences and may be revised. Release/method notes · World Bank data-use terms and attribution

I have a large number of infographic images that i have built for a vendor but due to the price conflict deal failed, now i want to sell this dataset. If someone wants this data for their company or research can purchase. I can share the samples u want. This dataset was fully verified by the client and highly dense infographic images. if someone wants it we can discuss price, quantity and licensing.

FDA publishes pet food recalls only through its Data Dashboard and Enforcement Report search. There's no bulk download for pet food, and openFDA's recall data barely covers it. So I built one.

• recalls.csv: one row per recall, with dates (initiated, classified, terminated), class, firm, reason category and species
• products.csv: one row per product, with description, UPCs, lot codes, recall number and quantity
• excluded.csv: every veterinary row I filtered out (drugs, livestock feed…) with the reason, so you can check the filter

One finding: the median time from a company starting a recall to FDA classifying it dropped from 139 days (2012–2019) to 30 days (2020 on), and it dropped in every class. Recalls started in 2020–2023 have all had years to be classified, and their median is 28 days, so the drop isn't just recent recalls still waiting.

Page + docs: https://perbowl.com/recalls/dataset/

Also on Kaggle (with a starter notebook), GitHub and Zenodo (DOI 10.5281/zenodo.23255873). It's updated monthly.

Corrections welcome.

Hey everyone,

Ihave been working on a project to bridge the gap between 3D rendering and aerospace computer vision. I built a fully headless Python-Blender rendering architecture to procedurally generate physically accurate synthetic data for asteroids.

The pipeline automates the extraction of:

• Multi-pass EXR renders (RGB, Z-Depth, Camera-Space Surface Normals and many more optional passes)
• Photometric lightcurve CSVs (tracking total flux, projected area, mean depth, and sun/observer coordinates per rotational step)
• Procedural material setups for both uniform and variable regolith albedos.

I have open-sourced a sample dataset of 600 meshes and their corresponding lightcurves on Kaggle for anyone who wants to train photometric inversion or pose estimation models.

Basically I wanted to research on the effects of albedo variation comapred to uniform asteroid. So what I did was took real asteroid meshes from DAMIT (coverted it to obj files), created a fully procedural and realistic shader applied it on the meshes and rendered a full revolution of asteroid in simulated space conditions. To acheive this, in result, I build a full headless python pipeline that does the complete job, with all the optimization I could do in the world, and even with my old GTX 970, the render time was insanely good. The pipleline automatically creates the lightcurve csv files (with multiple phase angles ) and with all the physics data as well such as normals and depth so I can have the option to train PINN model as well. However I have my exams so had to stop here.

That being said,

If you need massive scale or want to generate your own data locally, I have also packaged the full 3,000+ mesh dataset (6000 light curves) and the actual Python/Blender codebase (the Pipeline Toolkit).

• Free Sample Dataset: https://www.kaggle.com/datasets/hassaanfazal/astrovision-3d-asteroid-and-photometry-dataset
• Full Dataset & Codebase: https://stellarmeshlabs.gumroad.com/

Note: If you are a student or independent researcher who really needs this data but cannot afford the Gumroad tier, hit me up via DM. I will be happy to arrange a free, expanded subset of the data to support your work.

Disclaimer (per Rule 1): I am the creator of this pipeline and the Gumroad links go to my own store, StellarMesh Labs.

Over the past few months, we’ve been building a structured POI data marketplace at Agenty.

https://agenty.com/marketplace

We now have datasets covering 3,000+ brands across 12 countries, with 10M+ POI records.

The datasets include things like:

1. Store/location name and address
2. Latitude & longitude
3. Phone numbers and websites
4. Opening hours
5. Brand and location metadata
6. Other useful POI attributes

The goal is to make it easier for AI agents, researchers, data teams, and developers to get clean location data without having to build and maintain scrapers for every brand themselves.

Some of the datasets cover large retail, restaurant, automotive, hospitality and other brand networks.

We’re putting the marketplace together and would love feedback from people who actually work with POI/location data:

What brands or types of POI datasets would be most useful to you?

Disclaimer/Disclosure: I’m the founder of Agenty, so I’m obviously biased here. Sharing this because we’ve spent a lot of time building the datasets and would genuinely like feedback from the community.

I run VeritaHire, which reads job postings straight from US employers' own career sites (Workday, Greenhouse, iCIMS and others) every day, about 2.5 million live postings right now. From that I publish a daily Job Openings Index, free on GitHub:

https://github.com/veritahire/us-job-openings-index

- Daily since Sept 4, 2026 (= 100), published the next morning around 08:25 UTC

- Fixed panel: 18,310 employers, 2,081,433 open postings as of Oct 10

- Columns: daily index, 7-day average, the first-published value (every first print is kept, so revisions are visible), panel size, weekly new-posting and closing rates

- CSV, CC BY 4.0, refreshed every morning by a GitHub Action

Latest: 101.06 (7-day average) through Oct 10, +1.13% over 28 days. Method: https://veritahire.com/job-openings-index/

Since May 2024 every new model got the same prompts: draw a pelican on a bicycle as an SVG, build a landing page in one HTML file, tell five jokes, pick a favourite film, 87 in all. One attempt each, no system prompt, temp 0.7. I kept every answer, and it's now a dataset:

• JSON Lines, one file per question, plus models and questions
• SVG drawings as given, HTML pages with a 1280x800 screenshot each, image and audio answers as files
• release dates, and which of the 378 models are retired (101 aren't served on OpenRouter anymore)
• CC BY 4.0, DOI 10.5281/zenodo.23305275

GitHub: https://github.com/museumofmodels/collection Zenodo: https://doi.org/10.5281/zenodo.23305275 Browse it: https://museumofmodels.com

Nothing is scored, it's just the raw answers. Happy to hear what would make it more useful.

Disclosure: I made the museum and the dataset.