One account. Every library.
HF Data Library · Econ Data Library · more to come
FULL-COVERAGE grouped ingest of FAOSTAT bulk domains. Source: FAO FAOSTAT (CC BY 4.0). license_id = cc-by-4.0. Strategy -------- FAOSTAT publishes a machine manifest listing EVERY domain + its bulk zip URL: https://bulks-faostat.fao.org/production/datasets_E.json We download ALL domain bulk zips…
Food and Agriculture Organization of the United Nations, FAOSTAT. Retrieved from https://www.fao.org/faostat. Compiled and redistributed by the Elkassabgi Data Library.
Processing: Retrieved from the official source, normalized to a long {series_key, obs_date, value} schema (period-start dates), de-duplicated, and stored as zstd Parquet. Compiled and redistributed by the Elkassabgi Data Library.
FULL-COVERAGE grouped ingest of FAOSTAT bulk domains. Source: FAO FAOSTAT (CC BY 4.0). license_id = cc-by-4.0. Strategy -------- FAOSTAT publishes a machine manifest listing EVERY domain + its bulk zip URL: https://bulks-faostat.fao.org/production/datasets_E.json We download ALL domain bulk zips (the "(Normalized).zip" long-format files), stream the single normalized CSV inside each, and write ONE grouped Parquet per domain to data/clean_full/faostat/<DatasetCode>.parquet with columns: series_key, obs_date (date32), value (float64), flag (string) Normalized FAOSTAT schema varies per domain, so the parser is header-driven: * Geography / dimension columns (Area, Item, Element, Indicator, Reporter, Partner, Survey, Currency, Cost Category, Institution, Food Group, ...) are every column that is NOT one of the reserved roles below. * TIME comes from `Year` (single year, or a range like "2014-2016" -> end yr). Some domains add `Months` (calendar month -> that month/year; non-calendar qualifiers like "Annual value", "Meher season" stay in the series key). * VALUE from `Value`; FLAG from `Flag` (data-quality letter). * `Year Code`, `Months Code`, the `(M49)`/`(CPC)`/`(FAO)`/`(ISO3)` code twins, `Note` and `Flag` are excluded from the series key to keep it stable & lean. series_key = pipe-joined dimension values (codes preferred for stability, plus the human label so the key is self-describing), e.g. "QCL|Area=2|Item=221|Element=5312|Almonds, in shell|Area harvested" The grouped Parquet keeps the FULL dimensionality (nothing collapsed/dropped). Memory is bounded: each domain is streamed row-by-row from the zip and flushed to Parquet in 1,000,000-row batches, so even the 52M-row Trade matrix never holds more than one batch in RAM. Usage: python jobs/ingest_faostat.py --list # enumerate catalog, no download python jobs/ingest_faostat.py --dry 3 # download+parse 3 small domains python jobs/ingest_faostat.py # FULL run (all 68 domains) python jobs/ingest_faostat.py --only QCL,TM # specific domains python jobs/ingest_faostat.py --skip TM # all except listed
{
"@context": "https://schema.org/",
"@type": "Dataset",
"name": "FAOSTAT",
"description": "FULL-COVERAGE grouped ingest of FAOSTAT bulk domains. Source: FAO FAOSTAT (CC BY 4.0). license_id = cc-by-4.0. Strategy -------- FAOSTAT publishes a machine manifest listing EVERY domain + its bulk zip URL: https://bulks-faostat.fao.org/production/datasets_E.json We download ALL domain bulk zips…",
"url": "https://econdatalibrary.com/faostat.html",
"identifier": "faostat",
"isAccessibleForFree": true,
"publisher": {
"@type": "Organization",
"name": "Econ Data Library",
"url": "https://econdatalibrary.com"
},
"creator": {
"@type": "Organization",
"name": "FAOSTAT"
},
"license": "https://creativecommons.org/licenses/by/4.0/",
"keywords": [
"agriculture",
"faostat"
],
"temporalCoverage": "1991-12-31/2024-12-31",
"repeatFrequency": "P1Y",
"creditText": "Source: FAO FAOSTAT (CC BY 4.0)",
"citation": "Food and Agriculture Organization of the United Nations, FAOSTAT. Retrieved from https://www.fao.org/faostat. Compiled and redistributed by the Elkassabgi Data Library.",
"sameAs": [
"https://huggingface.co/datasets/econdatalibrary-faostat",
"https://zenodo.org/communities/econdatalibrary"
],
"distribution": [
{
"@type": "DataDownload",
"encodingFormat": "text/csv",
"contentUrl": "https://econdatalibrary.com/download.html?source=faostat",
"license": "https://creativecommons.org/licenses/by/4.0/"
}
]
}{
"@context": {
"@language": "en",
"@vocab": "https://schema.org/",
"cr": "http://mlcommons.org/croissant/",
"data": {
"@id": "cr:data",
"@type": "@json"
},
"dataType": {
"@id": "cr:dataType",
"@type": "@vocab"
},
"sc": "https://schema.org/",
"conformsTo": "dct:conformsTo",
"dct": "http://purl.org/dc/terms/"
},
"@type": "sc:Dataset",
"conformsTo": "http://mlcommons.org/croissant/1.0",
"name": "faostat",
"description": "FULL-COVERAGE grouped ingest of FAOSTAT bulk domains. Source: FAO FAOSTAT (CC BY 4.0). license_id = cc-by-4.0. Strategy -------- FAOSTAT publishes a machine manifest listing EVERY domain + its bulk zip URL: https://bulks-faostat.fao.org/production/datasets_E.json We download ALL domain bulk zips…",
"url": "https://econdatalibrary.com/faostat.html",
"creator": {
"@type": "sc:Organization",
"name": "FAOSTAT"
},
"publisher": {
"@type": "sc:Organization",
"name": "Econ Data Library",
"url": "https://econdatalibrary.com"
},
"license": "https://creativecommons.org/licenses/by/4.0/",
"keywords": [
"agriculture"
],
"temporalCoverage": "1991-12-31/2024-12-31",
"citation": "Food and Agriculture Organization of the United Nations, FAOSTAT. Retrieved from https://www.fao.org/faostat. Compiled and redistributed by the Elkassabgi Data Library.",
"distribution": [
{
"@type": "cr:FileObject",
"@id": "faostat-csv",
"name": "faostat-csv",
"description": "Per-series CSV, downloadable with a free API key.",
"contentUrl": "https://econdatalibrary.com/download.html?source=faostat",
"encodingFormat": "text/csv"
}
]
}