One account. Every library.
HF Data Library · Econ Data Library · more to come
FULL-COVERAGE grouped ingest of the ENTIRE Our World in Data grapher catalog. Source: Our World in Data (CC BY 4.0). license_id = cc-by-4.0. ENUMERATION ----------- OWID has no JSON "list every chart" endpoint, but it publishes a sitemap that lists every public page including every grapher chart:…
Our World in Data. Retrieved from https://ourworldindata.org. Compiled and redistributed by the Elkassabgi Data Library.
Processing: Retrieved from the official source, normalized to a long {series_key, obs_date, value} schema (period-start dates), de-duplicated, and stored as zstd Parquet. Compiled and redistributed by the Elkassabgi Data Library.
FULL-COVERAGE grouped ingest of the ENTIRE Our World in Data grapher catalog. Source: Our World in Data (CC BY 4.0). license_id = cc-by-4.0. ENUMERATION ----------- OWID has no JSON "list every chart" endpoint, but it publishes a sitemap that lists every public page including every grapher chart: https://ourworldindata.org/sitemap.xml We extract every `https://ourworldindata.org/grapher/<slug>` URL from it -> the full set of chart slugs (4,510 as of 2026-06). This is OWID's own authoritative manifest of public charts. PULL ---- For each slug the full long-format data is one CSV: https://ourworldindata.org/grapher/<slug>.csv?csvType=full&useColumnShortNames=true CSV schema is tidy: entity, code, <time>, <one-or-more value columns> * <time> is `year` (annual, the vast majority) OR `date` (YYYY-MM-DD daily). * helper columns we never treat as data: owid_region, owid_* (region tags). * `code` is ISO-alpha3 for countries, an OWID code for aggregates (OWID_WRL, OWID_EU27, ...). Some rows have an empty code (e.g. "World" sub-aggregates); we then fall back to the `entity` label for the key. LICENSE / CARVE-OUT (sources.yaml owid carve_out) ------------------------------------------------- "Host OWID's own work. For upstream third-party series, exclude where the license is not reservable." OWID enforces this at the source: a chart built on non-redistributable third-party data returns HTTP 403 with body {"status":403,"error":"This chart contains non-redistributable data ..."}. We DETECT that, EXCLUDE the chart, and count it under `non_redistributable`. Everything OWID lets us download is CC BY 4.0 and reservable. GROUPED STORAGE (anti-bloat) ---------------------------- ONE Parquet per chart slug -> data/clean_full/owid/<slug>.parquet with columns series_key (string), obs_date (date32), value (float64) where series_key = "<slug>|<value_col>|<code-or-entity>". All entities and all value columns of a chart live inside that ONE file. Max ~4,510 files for the whole source (one per chart) -- NOT one-file-per-series. A per-source JSON summary (data/clean_full/owid/_ingest_summary.json) records coverage. catalog.db is NOT touched here. Usage: python jobs/ingest_owid.py --enumerate # rebuild slug list from sitemap, print count python jobs/ingest_owid.py --dry 20 # download+parse 20 charts, no writes python jobs/ingest_owid.py # FULL run (all slugs), resumable python jobs/ingest_owid.py --workers 6 # tune concurrency (default 6) python jobs/ingest_owid.py --limit 200 # only first N slugs (testing)
{
"@context": "https://schema.org/",
"@type": "Dataset",
"name": "Our World in Data",
"description": "FULL-COVERAGE grouped ingest of the ENTIRE Our World in Data grapher catalog. Source: Our World in Data (CC BY 4.0). license_id = cc-by-4.0. ENUMERATION ----------- OWID has no JSON \"list every chart\" endpoint, but it publishes a sitemap that lists every public page including every grapher chart:…",
"url": "https://econdatalibrary.com/owid.html",
"identifier": "owid",
"isAccessibleForFree": true,
"publisher": {
"@type": "Organization",
"name": "Econ Data Library",
"url": "https://econdatalibrary.com"
},
"creator": {
"@type": "Organization",
"name": "Our World in Data"
},
"license": "https://creativecommons.org/licenses/by/4.0/",
"keywords": [
"climate",
"energy",
"macro",
"owid"
],
"temporalCoverage": "1750-12-31/2024-12-31",
"repeatFrequency": "P1Y",
"creditText": "Source: Our World in Data (CC BY 4.0)",
"citation": "Our World in Data. Retrieved from https://ourworldindata.org. Compiled and redistributed by the Elkassabgi Data Library.",
"sameAs": [
"https://huggingface.co/datasets/econdatalibrary-owid",
"https://zenodo.org/communities/econdatalibrary"
],
"distribution": [
{
"@type": "DataDownload",
"encodingFormat": "text/csv",
"contentUrl": "https://econdatalibrary.com/download.html?source=owid",
"license": "https://creativecommons.org/licenses/by/4.0/"
}
]
}{
"@context": {
"@language": "en",
"@vocab": "https://schema.org/",
"cr": "http://mlcommons.org/croissant/",
"data": {
"@id": "cr:data",
"@type": "@json"
},
"dataType": {
"@id": "cr:dataType",
"@type": "@vocab"
},
"sc": "https://schema.org/",
"conformsTo": "dct:conformsTo",
"dct": "http://purl.org/dc/terms/"
},
"@type": "sc:Dataset",
"conformsTo": "http://mlcommons.org/croissant/1.0",
"name": "owid",
"description": "FULL-COVERAGE grouped ingest of the ENTIRE Our World in Data grapher catalog. Source: Our World in Data (CC BY 4.0). license_id = cc-by-4.0. ENUMERATION ----------- OWID has no JSON \"list every chart\" endpoint, but it publishes a sitemap that lists every public page including every grapher chart:…",
"url": "https://econdatalibrary.com/owid.html",
"creator": {
"@type": "sc:Organization",
"name": "Our World in Data"
},
"publisher": {
"@type": "sc:Organization",
"name": "Econ Data Library",
"url": "https://econdatalibrary.com"
},
"license": "https://creativecommons.org/licenses/by/4.0/",
"keywords": [
"climate",
"energy",
"macro"
],
"temporalCoverage": "1750-12-31/2024-12-31",
"citation": "Our World in Data. Retrieved from https://ourworldindata.org. Compiled and redistributed by the Elkassabgi Data Library.",
"distribution": [
{
"@type": "cr:FileObject",
"@id": "owid-csv",
"name": "owid-csv",
"description": "Per-series CSV, downloadable with a free API key.",
"contentUrl": "https://econdatalibrary.com/download.html?source=owid",
"encodingFormat": "text/csv"
}
]
}