Istat

Licensing & provenance

License

License under review

Redistribution

Not permitted (metadata only)

Commercial use

Restricted / no

Modification

Permitted

Access & mirrors

Download

Select & download istat series as CSV →

API

Get a free API key, then GET /v1/series/<id>.csv

Canonical landing

https://econdatalibrary.com/istat.html

Update cadence

monthly

Update strategy

sdmx delta

Storage layout

per-flow parquet files in data/clean_full/istat/ (755 files present, many MB-scale, e.g. 101_1015.parquet ~2.9MB; schema series_key:string, obs_date:date32, value:float64, zstd, sorted by (key,date)). SAME dir+filename convention as ingest_sdmx_nso.py so the two jobs dedupe via skip-existing across both. Plus JSON checkpoints _sliced_done.json / _sliced_unrecoverable.json (atomic tmp+os.replace, lines 603-607). Full mode: incomplete = no parquet (is_incomplete, lines 610-614); present flows skipped+marked done (lines 697-700). --only mode INTENTIONALLY ignores skip-existing and the done checkpoint and OVERWRITES the target flow (lines 654-665, 697).

Full description

Complete ISTAT dataflows the normal pull left INCOMPLETE. The normal ISTAT sweep (jobs/ingest_sdmx_nso.py, providers "istat" + "istat_esploradati") writes one Parquet per dataflow into data/clean_full/istat/. A subset of flows never land because a single SDMX data response is too large for the endpoint: the host returns HTTP 500 / times out, the body exceeds what we can parse, or (via DBnomics historically) the flow blows past the 100K-series cap. This job sweeps the ISTAT catalog, finds every flow that has NO parquet yet (or is marked errored), and retrieves it WITHOUT overloading the endpoint: 1. Plain full pull GET /data/IT1,{flow}/ (sdmx-csv) 2. Time slicing startPeriod/endPeriod in decade windows (pre-1960, 1960-69, ... 2030-39), accumulating + de-duping obs. 3. Per-year slicing for any decade that still 500s / is too big. 4. Dimension slicing split the key-path on the most granular dimension (territory/category/...), one request per code, when the time axis alone can't shrink the response. Requires the DSD, fetched from whichever ISTAT host answers. Anything that survives all of the above on BOTH hosts is recorded in _sliced_unrecoverable.json with the exact reason -- never fabricated, never silently dropped. Completed flow ids are checkpointed in _sliced_done.json so the run is fully resumable (a watcher relaunches this after ISTAT, whose esploradati host is chronically flaky, recovers). Output: data/clean_full/istat/{flow_id}.parquet -- SAME dir + filename convention as ingest_sdmx_nso.py, so skip-existing dedupes across both jobs. Schema: {series_key: string, obs_date: date32, value: float64}, zstd. Run: python jobs/ingest_istat_sliced.py # full incomplete sweep python jobs/ingest_istat_sliced.py --only A,B # just these flow ids python jobs/ingest_istat_sliced.py --list # list incomplete, no DL python jobs/ingest_istat_sliced.py --max-size-mb 500

Structured metadata

{
  "@context": "https://schema.org/",
  "@type": "Dataset",
  "name": "Istat",
  "description": "Complete ISTAT dataflows the normal pull left INCOMPLETE. The normal ISTAT sweep (jobs/ingest_sdmx_nso.py, providers \"istat\" + \"istat_esploradati\") writes one Parquet per dataflow into data/clean_full/istat/. A subset of flows never land because a single SDMX data response is too large for the…",
  "url": "https://econdatalibrary.com/istat.html",
  "identifier": "istat",
  "isAccessibleForFree": false,
  "publisher": {
    "@type": "Organization",
    "name": "Econ Data Library",
    "url": "https://econdatalibrary.com"
  },
  "creator": {
    "@type": "Organization",
    "name": "Istat"
  },
  "keywords": [
    "istat"
  ],
  "sameAs": [
    "https://huggingface.co/datasets/econdatalibrary-istat",
    "https://zenodo.org/communities/econdatalibrary"
  ],
  "usageInfo": "Metadata-only catalog entry. This source's license does not permit redistribution, so the data itself is not served here; use the provider link to obtain the data under its terms."
}

{
  "@context": {
    "@language": "en",
    "@vocab": "https://schema.org/",
    "cr": "http://mlcommons.org/croissant/",
    "data": {
      "@id": "cr:data",
      "@type": "@json"
    },
    "dataType": {
      "@id": "cr:dataType",
      "@type": "@vocab"
    },
    "sc": "https://schema.org/",
    "conformsTo": "dct:conformsTo",
    "dct": "http://purl.org/dc/terms/"
  },
  "@type": "sc:Dataset",
  "conformsTo": "http://mlcommons.org/croissant/1.0",
  "name": "istat",
  "description": "Complete ISTAT dataflows the normal pull left INCOMPLETE. The normal ISTAT sweep (jobs/ingest_sdmx_nso.py, providers \"istat\" + \"istat_esploradati\") writes one Parquet per dataflow into data/clean_full/istat/. A subset of flows never land because a single SDMX data response is too large for the…",
  "url": "https://econdatalibrary.com/istat.html",
  "creator": {
    "@type": "sc:Organization",
    "name": "Istat"
  },
  "publisher": {
    "@type": "sc:Organization",
    "name": "Econ Data Library",
    "url": "https://econdatalibrary.com"
  },
  "isAccessibleForFree": false,
  "usageInfo": "Metadata-only Croissant record; underlying data is not redistributed under this source's license."
}