OECD Data Explorer (from 2024-07-01)

Coverage

Licensing & provenance

License

Creative Commons Attribution 4.0 (CC BY 4.0)

License URL

https://creativecommons.org/licenses/by/4.0/

Required attribution

Source: OECD (CC BY 4.0)

Redistribution

Permitted (served here)

Commercial use

Yes

Modification

Permitted

How to cite

Organisation for Economic Co-operation and Development. Retrieved from https://data-explorer.oecd.org. Compiled and redistributed by the Elkassabgi Data Library.

Processing: Retrieved from the official source, normalized to a long {series_key, obs_date, value} schema (period-start dates), de-duplicated, and stored as zstd Parquet. Compiled and redistributed by the Elkassabgi Data Library.

Access & mirrors

Download

Select & download oecd series as CSV →

API

Get a free API key, then GET /v1/series/<id>.csv

Canonical landing

https://econdatalibrary.com/oecd.html

Update cadence

monthly

Update strategy

giant changed units

Storage layout

overwrite-multi-file, ONE parquet per dataflow: data/clean_full/oecd/<AGENCY>__<DATAFLOW_ID>.parquet (currently 1413 files; ~6.0B obs, COVERAGE GIANT, ~57 GB on disk). Schema: series_key (str, dim cols joined by '.'), obs_date (date32), value (float64), obs_status (str), time_raw (str). Each flow streamed to data/raw/oecd/_tmp/<...>.csv then parsed once into a .part and os.replace'd. Run state/logs in data/raw/oecd/: manifest.json, dataflows_all.xml, ingest_results.jsonl, run.log/pid.

Full description

FULL-COVERAGE grouped ingest of the entire OECD SDMX catalog (Data Explorer). Source: OECD (CC BY 4.0). license_id = cc-by-4.0 (from configs/sources.yaml). Strategy (verified against sdmx.oecd.org, NSI Web Service v8.19.8.0): * Catalog = every dataflow from /public/rest/dataflow/all/all/latest (1509 flows). * Most flows are served from the DEFAULT root https://sdmx.oecd.org/public . 27 flows are isExternalReference=true and route to alternate roots (sti-public / dcd-public / archive) -- we resolve each flow's structureURL root. * Data pulled as SDMX-CSV (Accept: application/vnd.sdmx.data+csv) via /rest/data/<agency>,<id>,<ver>/all?dimensionAtObservation=AllDimensions * SDMX-CSV header = DATAFLOW, <dim1..dimN>, TIME_PERIOD, OBS_VALUE, <attrs...>. SERIES KEY = the dimension columns (everything strictly between DATAFLOW and TIME_PERIOD), joined with '.' -> stored as column `series_key`. * OECD caps a response at ~1e6 observations; very large flows return HTTP 500 / never complete. FALLBACK: split the query by REF_AREA using the SDMX path-key (REF_AREA value + dots for the remaining dims, in DSD position order); the set of real reporters comes from /rest/availableconstraint/.../all?mode=available. OUTPUT: ONE Parquet per dataflow -> data/clean_full/oecd/<AGENCY>__<DATAFLOW_ID>.parquet columns: series_key (str), obs_date (date32), value (float64), obs_status (str), time_raw (str) A coarse per-dataflow row also gets written to a sidecar manifest (NOT catalog.db). Memory is bounded: each dataflow is streamed to a temp .csv on disk, parsed in a single pass, and written once; temp files are deleted after. Usage: python jobs/ingest_oecd.py --catalog # (re)download dataflow list, print counts python jobs/ingest_oecd.py --dry 8 # process first 8 flows, no Parquet writes python jobs/ingest_oecd.py --workers 6 # FULL run (default workers=6) python jobs/ingest_oecd.py --only DSD_FOO@DF_BAR # process one flow (debug)

Structured metadata

{
  "@context": "https://schema.org/",
  "@type": "Dataset",
  "name": "OECD Data Explorer (from 2024-07-01)",
  "description": "FULL-COVERAGE grouped ingest of the entire OECD SDMX catalog (Data Explorer). Source: OECD (CC BY 4.0). license_id = cc-by-4.0 (from configs/sources.yaml). Strategy (verified against sdmx.oecd.org, NSI Web Service v8.19.8.0): * Catalog = every dataflow from /public/rest/dataflow/all/all/latest…",
  "url": "https://econdatalibrary.com/oecd.html",
  "identifier": "oecd",
  "isAccessibleForFree": true,
  "publisher": {
    "@type": "Organization",
    "name": "Econ Data Library",
    "url": "https://econdatalibrary.com"
  },
  "creator": {
    "@type": "Organization",
    "name": "OECD Data Explorer (from 2024-07-01)"
  },
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "keywords": [
    "macro",
    "oecd"
  ],
  "temporalCoverage": "1995-01-01/2026-04-01",
  "repeatFrequency": [
    "P1M",
    "P3M"
  ],
  "creditText": "Source: OECD (CC BY 4.0)",
  "citation": "Organisation for Economic Co-operation and Development. Retrieved from https://data-explorer.oecd.org. Compiled and redistributed by the Elkassabgi Data Library.",
  "sameAs": [
    "https://huggingface.co/datasets/econdatalibrary-oecd",
    "https://zenodo.org/communities/econdatalibrary"
  ],
  "distribution": [
    {
      "@type": "DataDownload",
      "encodingFormat": "text/csv",
      "contentUrl": "https://econdatalibrary.com/download.html?source=oecd",
      "license": "https://creativecommons.org/licenses/by/4.0/"
    }
  ]
}

{
  "@context": {
    "@language": "en",
    "@vocab": "https://schema.org/",
    "cr": "http://mlcommons.org/croissant/",
    "data": {
      "@id": "cr:data",
      "@type": "@json"
    },
    "dataType": {
      "@id": "cr:dataType",
      "@type": "@vocab"
    },
    "sc": "https://schema.org/",
    "conformsTo": "dct:conformsTo",
    "dct": "http://purl.org/dc/terms/"
  },
  "@type": "sc:Dataset",
  "conformsTo": "http://mlcommons.org/croissant/1.0",
  "name": "oecd",
  "description": "FULL-COVERAGE grouped ingest of the entire OECD SDMX catalog (Data Explorer). Source: OECD (CC BY 4.0). license_id = cc-by-4.0 (from configs/sources.yaml). Strategy (verified against sdmx.oecd.org, NSI Web Service v8.19.8.0): * Catalog = every dataflow from /public/rest/dataflow/all/all/latest…",
  "url": "https://econdatalibrary.com/oecd.html",
  "creator": {
    "@type": "sc:Organization",
    "name": "OECD Data Explorer (from 2024-07-01)"
  },
  "publisher": {
    "@type": "sc:Organization",
    "name": "Econ Data Library",
    "url": "https://econdatalibrary.com"
  },
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "keywords": [
    "macro"
  ],
  "temporalCoverage": "1995-01-01/2026-04-01",
  "citation": "Organisation for Economic Co-operation and Development. Retrieved from https://data-explorer.oecd.org. Compiled and redistributed by the Elkassabgi Data Library.",
  "distribution": [
    {
      "@type": "cr:FileObject",
      "@id": "oecd-csv",
      "name": "oecd-csv",
      "description": "Per-series CSV, downloadable with a free API key.",
      "contentUrl": "https://econdatalibrary.com/download.html?source=oecd",
      "encodingFormat": "text/csv"
    }
  ]
}