Wikipedia is in nearly every open training corpus in circulation, but most of the value from Wikipedia's knowledge is lost in the parsing step before it gets to those training datasets. Most training corpus creators download a wikitext dump, parse it with a bespoke script, and flatten the output into text. This creates a usable corpus of prose, but it drops highly valuable knowledge, structure, and metadata. Crucial metadata is lost, such as who created the revision, what license governs the article, and when the latest revision was made.
Other highly valuable knowledge gets lost in the flattening: table structures, infoboxes, categories, where an image is placed in a paragraph, and the references and citations that connect knowledge to its source.
The Wikipedia Structured Contents dataset on Hugging Face is the version that keeps all of the above. It is built from the Snapshot API using Structured Contents, and it is free to download and use. Structured Contents parses Wikipedia articles into neat JSON structures, without losing any of the context that makes Wikipedia the largest open source knowledge repository in the world.
What's in the dataset and usage
The dataset covers every article in the English and French language editions of Wikipedia: 7.6 million articles from English Wikipedia and 2.87 million from French, 10,468,881 rows in total at roughly 48 GB. Each row is one article, pre-parsed into a consistent schema, with the non-prose sections neatly structured instead of discarded.
We updated this dataset in May 2026 to include the newest Structured Contents features: Parsed References with their scoring models, Parsed Tables, article images, and lists. Infoboxes, sections, abstracts, and short descriptions were already included. The same update moved the files to Parquet, so the dataset loads straight into the Hugging Face datasets library, pandas, or DuckDB without a conversion step.
The Structured Contents payload is useful across the whole lifecycle of AI models, from pre-training through alignment and fine-tuning. It's equally valuable as a grounding corpus for systems that answer questions against current sources rather than memorized ones.
Dataset Structure
An example of each line of JSON looks as follows (abbreviated data):
{"name": "Josephine Baker","identifier": 255083,"url": "https://en.wikipedia.org/wiki/Josephine_Baker","date_created": "...","date_modified": "...","is_part_of": {"..."},"in_language": {"..."},"main_entity": {"identifier": "Q151972",...},"additional_entities": [...],"version": {...},"description": "American-born French dancer...","abstract": "Freda Josephine Baker, naturalized as ...","image": {"content_url": "https://upload.wikimedia.org/wikipedia/...",...},"infoboxes": [{"name": "Infobox person", "type": "infobox", "has_parts": [ {"name": "Josephine Baker", "type": "section", "has_parts": [ {"name": "Born", "type": "field", "value": "Freda Josephine McDonald June 3, 1906 St. Louis, Missouri, US", "links": [{"url": "https://en.wikipedia.org/wiki/St._Louis", "text": "St. Louis"},...}],"sections": [{"name": "Abstract", "type": "section", "has_parts": [ {"type": "paragraph", "value": "Freda Josephine Baker (née McDonald; June 3, 1906 - April 12, 1975), naturalized as Joséphine Baker...", "links": [{"url": "https://en.wikipedia.org/wiki/Siren_...", "text": "Siren of the Tropics"...}],"license": [...],}Dataset Fields
Data fields included in every line/article are the following, all part of the structured_content model in our API reference:
name- title of the articleidentifier- ID of the articleabstract- lead section, summarizing what the article is about- version - metadata related to the latest specific revision of the article
- version.editor - editor-specific signals that can help contextualize the revision
- version.scores - machine learning scores for the revision from Wikimedia's LiftWing models: revert risk, reference risk, and reference need
url- URL of the articledate_created- timestamp of the article creation event, or the article's first revisiondate_modified- timestamp of the last revision of the article- main_entity - the Wikidata QID the article is related to
- is_part_of - Wikimedia project this article belongs to
- additional_entities - array of Wikidata entities used in this article
- in_language - human language in which the article is written
- image - the main image representing the article's subject
- license - relevant licenses that affect this article and content reuse
description- one-sentence description of the article for quick reference- infoboxes - parsed information from the side panel (aka infobox) on the Wikipedia article
- sections - parsed sections of the full article, including links
A fuller field-by-field reference is in the data dictionary.
Continue working with Structured Contents using our APIs
This dataset is a snapshot, a photograph of Wikipedia as it stood at one point in time. This one was taken in May 2026, and Wikipedia has not stopped moving since. English Wikipedia alone takes hundreds of thousands of edits a month (averaging an edit every 6 seconds!), so the gap between any published snapshot and the live encyclopedia widens every day it sits there.
For pre-training, using a snapshot is usually fine. A corpus with a known cutoff is a reasonable thing to train on, and knowing the cutoff exactly is better than inheriting one you cannot date. For anything that answers questions in front of a user, you need up-to-date information, which is exactly the power of Wikipedia's continuously updated knowledge base.
Use the Hugging Face dataset to evaluate the schema for your use case, develop prototypes using real data, and decide whether it fits your pipeline. Once you're ready to integrate this into production, take a look at our APIs:
- The Snapshot API delivers the same Structured Contents data as a bulk download, across the full set of supported projects and languages rather than two, refreshed on a schedule you control instead of whenever we publish to Hugging Face.
- The On-demand API returns the latest revision of any single article.
- The Realtime API streams every change as it happens, so a corpus you have already built stays current instead of being rebuilt.
With a free account you can request a fresh snapshot from the Snapshot API every month, and make up to 50,000 On-demand requests per month. When you need more egress or fresher data, contact Sales.
Data Licensing and Attribution
All original textual content is licensed under the GNU Free Documentation License (GFDL) and the Creative Commons Attribution-Share-Alike 4.0 License. Some text may be available only under the Creative Commons license; see the Wikimedia Terms of Use for details. Text written by some authors may be released under additional licenses or into the public domain.
Attribution, as required by the Creative Commons license used for this dataset, is core to the sustainability of the Wikimedia projects. It is what drives new editors and donors to Wikipedia. With consistent attribution, this cycle of content creation and reuse ensures encyclopedic content of high-quality, reliability, and verifiability will continue being written on Wikipedia and ultimately remain available for reuse via datasets such as these.
As such, we require all users of this dataset to conform to our expectations for proper attribution. Detailed attribution requirements for use of this dataset are outlined on Hugging Face. Every article in the dataset carries its own license field, and the version object identifies the exact revision the text came from, so a derived dataset can carry its provenance forward instead of reconstructing it later.
Where to get the dataset
The dataset is published at huggingface.co/datasets/wikimedia/structured-wikipedia. You can browse it in the dataset viewer, download it, or load it directly with the Hugging Face datasets library.
Get Started
Play around with the Hugging Face dataset to see whether Structured Contents fits how you work. When you need more languages, more projects, or data that is current rather than dated, sign up for a Wikimedia Enterprise account.
Sign up for a free account to use the Snapshot and On-demand APIs, no credit card required. Get started with our generous free account limits first to see how the APIs seamlessly fit into your project. If you are planning high-volume or high-frequency access, or you want the Realtime API in the mix, contact us so we can work out a plan that fits your needs.
- Wikimedia Enterprise Team