Istat (Italian National Institute of Statistics)

Coverage

Licensing & provenance

License

cc-by-4.0-istat

License URL

https://creativecommons.org/licenses/by/4.0/

Required attribution

Source: Istat, licensed CC BY 4.0.

Redistribution

Permitted (served here)

Commercial use

Yes

Modification

Permitted

Provider homepage

https://www.istat.it/

Provider terms

https://www.istat.it/en/legal-notice/

How to cite

Istat, Italian National Institute of Statistics. Licensed CC BY 4.0. Compiled and redistributed by the Elkassabgi Data Library.

Processing: Retrieved from Istat's SDMX API and stored as zstd Parquet, one file per dataflow. Served at FLOW grain because the source averages 9.2 observations per series; large flows are split on one of their own named dimensions.

Access & mirrors

Download

Select & download istat series as CSV →

API

Get a free API key, then GET /v1/series/<id>.csv

Canonical landing

https://econdatalibrary.com/istat

Update cadence

monthly (target — not yet automated)

Automated refresh

not yet wired — the data here is the verified initial load. See the Source Status board.

Update strategy

sdmx delta

Storage layout

per-flow parquet files in data/clean_full/istat/ (755 files present, many MB-scale, e.g. 101_1015.parquet ~2.9MB; schema series_key:string, obs_date:date32, value:float64, zstd, sorted by (key,date)). SAME dir+filename convention as ingest_sdmx_nso.py so the two jobs dedupe via skip-existing across both. Plus JSON checkpoints _sliced_done.json / _sliced_unrecoverable.json (atomic tmp+os.replace, lines 603-607). Full mode: incomplete = no parquet (is_incomplete, lines 610-614); present flows skipped+marked done (lines 697-700). --only mode INTENTIONALLY ignores skip-existing and the done checkpoint and OVERWRITES the target flow (lines 654-665, 697).

Full description

Complete ISTAT dataflows the normal pull left INCOMPLETE. The normal ISTAT sweep (jobs/ingest_sdmx_nso.py, providers "istat" + "istat_esploradati") writes one Parquet per dataflow into data/clean_full/istat/. A subset of flows never land because a single SDMX data response is too large for the endpoint: the host returns HTTP 500 / times out, the body exceeds what we can parse, or (via DBnomics historically) the flow blows past the 100K-series cap. This job sweeps the ISTAT catalog, finds every flow that has NO parquet yet (or is marked errored), and retrieves it WITHOUT overloading the endpoint: 1. Plain full pull GET /data/IT1,{flow}/ (sdmx-csv) 2. Time slicing startPeriod/endPeriod in decade windows (pre-1960, 1960-69, ... 2030-39), accumulating + de-duping obs. 3. Per-year slicing for any decade that still 500s / is too big. 4. Dimension slicing split the key-path on the most granular dimension (territory/category/...), one request per code, when the time axis alone can't shrink the response. Requires the DSD, fetched from whichever ISTAT host answers. Anything that survives all of the above on BOTH hosts is recorded in _sliced_unrecoverable.json with the exact reason -- never fabricated, never silently dropped. Completed flow ids are checkpointed in _sliced_done.json so the run is fully resumable (a watcher relaunches this after ISTAT, whose esploradati host is chronically flaky, recovers). Output: data/clean_full/istat/{flow_id}.parquet -- SAME dir + filename convention as ingest_sdmx_nso.py, so skip-existing dedupes across both jobs. Schema: {series_key: string, obs_date: date32, value: float64}, zstd. Run: python jobs/ingest_istat_sliced.py # full incomplete sweep python jobs/ingest_istat_sliced.py --only A,B # just these flow ids python jobs/ingest_istat_sliced.py --list # list incomplete, no DL python jobs/ingest_istat_sliced.py --max-size-mb 500

Structured metadata

{
  "@context": "https://schema.org/",
  "@type": "Dataset",
  "name": "Istat (Italian National Institute of Statistics)",
  "description": "Local units and persons employed; Enterprises and persons employed; Persons convicted by final judgement. Istat. 14,267 series, from 1970. Free download.",
  "url": "https://econdatalibrary.com/istat",
  "identifier": "istat",
  "isAccessibleForFree": true,
  "publisher": {
    "@type": "Organization",
    "name": "Econ Data Library",
    "url": "https://econdatalibrary.com"
  },
  "includedInDataCatalog": {
    "@type": "DataCatalog",
    "@id": "https://econdatalibrary.com/#catalog",
    "name": "Econ Data Library",
    "url": "https://econdatalibrary.com/catalog"
  },
  "creator": {
    "@type": "Organization",
    "name": "Istat",
    "url": "https://www.istat.it/"
  },
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "keywords": [
    "Italy",
    "istat"
  ],
  "temporalCoverage": "1970-12-31/..",
  "creditText": "Source: Istat, licensed CC BY 4.0.",
  "citation": "Istat, Italian National Institute of Statistics. Licensed CC BY 4.0. Compiled and redistributed by the Elkassabgi Data Library.",
  "sameAs": [
    "https://huggingface.co/datasets/econdatalibrary-istat",
    "https://zenodo.org/communities/econdatalibrary"
  ],
  "distribution": [
    {
      "@type": "DataDownload",
      "name": "istat — CSV",
      "encodingFormat": "text/csv",
      "contentUrl": "https://econdatalibrary.com/download.html?source=istat",
      "license": "https://creativecommons.org/licenses/by/4.0/"
    }
  ]
}

{
  "@context": {
    "@language": "en",
    "@vocab": "https://schema.org/",
    "cr": "http://mlcommons.org/croissant/",
    "data": {
      "@id": "cr:data",
      "@type": "@json"
    },
    "dataType": {
      "@id": "cr:dataType",
      "@type": "@vocab"
    },
    "sc": "https://schema.org/",
    "conformsTo": "dct:conformsTo",
    "dct": "http://purl.org/dc/terms/"
  },
  "@type": "sc:Dataset",
  "conformsTo": "http://mlcommons.org/croissant/1.0",
  "name": "istat",
  "description": "Local units and persons employed; Enterprises and persons employed; Persons convicted by final judgement. Istat. 14,267 series, from 1970. Free download.",
  "url": "https://econdatalibrary.com/istat",
  "creator": {
    "@type": "sc:Organization",
    "name": "Istat",
    "url": "https://www.istat.it/"
  },
  "publisher": {
    "@type": "sc:Organization",
    "name": "Econ Data Library",
    "url": "https://econdatalibrary.com"
  },
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "keywords": [
    "Italy"
  ],
  "citation": "Istat, Italian National Institute of Statistics. Licensed CC BY 4.0. Compiled and redistributed by the Elkassabgi Data Library.",
  "distribution": [
    {
      "@type": "cr:FileObject",
      "@id": "istat-csv",
      "name": "istat-csv",
      "description": "Per-series CSV, downloadable with a free API key.",
      "contentUrl": "https://econdatalibrary.com/download.html?source=istat",
      "encodingFormat": "text/csv"
    }
  ]
}