One account. Every library.
HF Data Library · Econ Data Library · more to come
FULL-COVERAGE grouped ingest of the Australian Bureau of Statistics (ABS) SDMX API. Catalog : https://data.api.abs.gov.au/rest/dataflow/ABS?detail=allstubs (~1223 flows) Data pull: https://data.api.abs.gov.au/rest/data/<FLOW>/all?format=csvfile (compact, codes only) GROUPED storage (mirrors…
Australian Bureau of Statistics. Retrieved from https://www.abs.gov.au. Compiled and redistributed by the Elkassabgi Data Library.
Processing: Retrieved from the official source, normalized to a long {series_key, obs_date, value} schema (period-start dates), de-duplicated, and stored as zstd Parquet. Compiled and redistributed by the Elkassabgi Data Library.
FULL-COVERAGE grouped ingest of the Australian Bureau of Statistics (ABS) SDMX API. Catalog : https://data.api.abs.gov.au/rest/dataflow/ABS?detail=allstubs (~1223 flows) Data pull: https://data.api.abs.gov.au/rest/data/<FLOW>/all?format=csvfile (compact, codes only) GROUPED storage (mirrors jobs/ingest_eurostat.py): ONE Parquet per dataflow under data/clean_full/abs/<FLOW>.parquet with columns (series_key, obs_date, value) -- many series inside each file. The series_key is the dotted SDMX data key: every dimension code column between the DATAFLOW column and TIME_PERIOD, joined with '.'. This matches the existing connectors/abs/connector.py key convention (e.g. "1.10001.10.50.Q"). The ABS compact CSV (format=csvfile) is: DATAFLOW, <dim1>, <dim2>, ..., TIME_PERIOD, OBS_VALUE, <attributes...> Column positions vary per flow, so we locate TIME_PERIOD / OBS_VALUE by name and treat every column strictly between DATAFLOW (col 0) and TIME_PERIOD as a dimension. License = cc-by-4.0 (reservable id from configs/sources.yaml -> sources.abs.license). Resilience: the ABS API is flaky. Each dataflow is streamed and parsed independently; network errors retry with exponential backoff; a 404 / empty body is treated as "no data for this flow" and skipped. Already-written .parquet files are skipped so the job is fully resumable after a crash. A flow whose download exceeds a size guard is split by its FREQ (or first) dimension and written as one Parquet per split value. Usage: python jobs/ingest_abs_full.py --list # just enumerate the catalog python jobs/ingest_abs_full.py --dry 5 # parse 5 flows, print, no writes python jobs/ingest_abs_full.py # full run (resumable) python jobs/ingest_abs_full.py --only CPI,LF # restrict to named flows
{
"@context": "https://schema.org/",
"@type": "Dataset",
"name": "Australian Bureau of Statistics",
"description": "FULL-COVERAGE grouped ingest of the Australian Bureau of Statistics (ABS) SDMX API. Catalog : https://data.api.abs.gov.au/rest/dataflow/ABS?detail=allstubs (~1223 flows) Data pull: https://data.api.abs.gov.au/rest/data/<FLOW>/all?format=csvfile (compact, codes only) GROUPED storage (mirrors…",
"url": "https://econdatalibrary.com/abs.html",
"identifier": "abs",
"isAccessibleForFree": true,
"publisher": {
"@type": "Organization",
"name": "Econ Data Library",
"url": "https://econdatalibrary.com"
},
"creator": {
"@type": "Organization",
"name": "Australian Bureau of Statistics"
},
"license": "https://creativecommons.org/licenses/by/4.0/",
"keywords": [
"labour",
"national-accounts",
"prices",
"trade",
"abs"
],
"temporalCoverage": "1948-09-30/2026-04-30",
"repeatFrequency": [
"P1M",
"P3M"
],
"creditText": "Source: ABS (CC BY 4.0)",
"citation": "Australian Bureau of Statistics. Retrieved from https://www.abs.gov.au. Compiled and redistributed by the Elkassabgi Data Library.",
"sameAs": [
"https://huggingface.co/datasets/econdatalibrary-abs",
"https://zenodo.org/communities/econdatalibrary"
],
"distribution": [
{
"@type": "DataDownload",
"encodingFormat": "text/csv",
"contentUrl": "https://econdatalibrary.com/download.html?source=abs",
"license": "https://creativecommons.org/licenses/by/4.0/"
}
]
}{
"@context": {
"@language": "en",
"@vocab": "https://schema.org/",
"cr": "http://mlcommons.org/croissant/",
"data": {
"@id": "cr:data",
"@type": "@json"
},
"dataType": {
"@id": "cr:dataType",
"@type": "@vocab"
},
"sc": "https://schema.org/",
"conformsTo": "dct:conformsTo",
"dct": "http://purl.org/dc/terms/"
},
"@type": "sc:Dataset",
"conformsTo": "http://mlcommons.org/croissant/1.0",
"name": "abs",
"description": "FULL-COVERAGE grouped ingest of the Australian Bureau of Statistics (ABS) SDMX API. Catalog : https://data.api.abs.gov.au/rest/dataflow/ABS?detail=allstubs (~1223 flows) Data pull: https://data.api.abs.gov.au/rest/data/<FLOW>/all?format=csvfile (compact, codes only) GROUPED storage (mirrors…",
"url": "https://econdatalibrary.com/abs.html",
"creator": {
"@type": "sc:Organization",
"name": "Australian Bureau of Statistics"
},
"publisher": {
"@type": "sc:Organization",
"name": "Econ Data Library",
"url": "https://econdatalibrary.com"
},
"license": "https://creativecommons.org/licenses/by/4.0/",
"keywords": [
"labour",
"national-accounts",
"prices",
"trade"
],
"temporalCoverage": "1948-09-30/2026-04-30",
"citation": "Australian Bureau of Statistics. Retrieved from https://www.abs.gov.au. Compiled and redistributed by the Elkassabgi Data Library.",
"distribution": [
{
"@type": "cr:FileObject",
"@id": "abs-csv",
"name": "abs-csv",
"description": "Per-series CSV, downloadable with a free API key.",
"contentUrl": "https://econdatalibrary.com/download.html?source=abs",
"encodingFormat": "text/csv"
}
]
}