One account. Every library.
HF Data Library · Econ Data Library · more to come
Full-coverage ingest of NOAA NCEI GSOM + GSOY climate summaries. Source: NOAA NCEI bulk archive GSOM (Global Summary of the Month): https://www.ncei.noaa.gov/data/gsom/access/ GSOY (Global Summary of the Year): https://www.ncei.noaa.gov/data/gsoy/access/ License: us-public-domain. Attribution:…
NOAA National Centers for Environmental Information, Global Summary of the Month / Global Summary of the Year (GSOM/GSOY), derived from the Global Historical Climatology Network-Daily dataset. Retrieved from the NCEI bulk archive and redistributed by the Elkassabgi Data Library.
Processing: Retrieved from NCEI's bulk archive (gsom-latest.tar.gz / gsoy-latest.tar.gz), melted from one wide CSV per station to a long {series_key, obs_date, value} schema, and stored as zstd Parquet sharded by GHCN country prefix.
Full-coverage ingest of NOAA NCEI GSOM + GSOY climate summaries. Source: NOAA NCEI bulk archive GSOM (Global Summary of the Month): https://www.ncei.noaa.gov/data/gsom/access/ GSOY (Global Summary of the Year): https://www.ncei.noaa.gov/data/gsoy/access/ License: us-public-domain. Attribution: "Source: NOAA (public domain)". The bulk archive publishes ONE wide CSV per weather station (STATION, DATE, LATITUDE, LONGITUDE, ELEVATION, NAME, then ~55 climate elements each paired with an <ELEM>_ATTRIBUTES column). Counts (enumerated 2026-06): GSOM: 127,905 station CSVs GSOY: 86,018 station CSVs GROUPED storage (anti-bloat -- NEVER one file per station): We melt each wide CSV to LONG form (dataset, station, series_key, obs_date, element, value, attributes) and accumulate rows into SHARDS keyed by the 2-char GHCN country/network prefix of the station id (US, AS, CA, ...). Output: data/clean_full/noaa/gsom__<PREFIX>.parquet (obs, many stations) data/clean_full/noaa/gsom__<PREFIX>__series.parquet (series meta) ~250 prefixes x 2 datasets + sidecars => a few hundred files total. series_key = "<station>:<element>" (one series per station x element). Stages (resumable): --enumerate (re)fetch the station-id manifests + station metadata --download fetch every station CSV (skips files already on disk) --build parse raw CSVs -> grouped long parquet shards + verify --verify re-read shards and print obs/series/coverage report Default (no flag) = --build. Usage: python jobs/ingest_noaa.py --download python jobs/ingest_noaa.py --download --dataset gsoy --workers 6 python jobs/ingest_noaa.py --build python jobs/ingest_noaa.py --verify
{
"@context": "https://schema.org/",
"@type": "Dataset",
"name": "NOAA NCEI / Climate Data Online",
"description": "Monthly; Annual; SANTA ROSA. NOAA NCEI / Climate Data Online. 3,135,873 series, 1763–2026. Free download under public domain.",
"url": "https://econdatalibrary.com/noaa",
"identifier": "noaa",
"isAccessibleForFree": true,
"publisher": {
"@type": "Organization",
"name": "Econ Data Library",
"url": "https://econdatalibrary.com"
},
"includedInDataCatalog": {
"@type": "DataCatalog",
"@id": "https://econdatalibrary.com/#catalog",
"name": "Econ Data Library",
"url": "https://econdatalibrary.com/catalog"
},
"creator": {
"@type": "Organization",
"name": "NOAA NCEI / Climate Data Online"
},
"license": "https://www.usa.gov/government-works",
"keywords": [
"Climate",
"Environment & Climate",
"noaa"
],
"temporalCoverage": "1763-01-01/2026-12-31",
"repeatFrequency": [
"P1Y",
"P1M"
],
"citation": "NOAA National Centers for Environmental Information, Global Summary of the Month / Global Summary of the Year (GSOM/GSOY), derived from the Global Historical Climatology Network-Daily dataset. Retrieved from the NCEI bulk archive and redistributed by the Elkassabgi Data Library.",
"sameAs": [
"https://huggingface.co/datasets/econdatalibrary-noaa",
"https://zenodo.org/communities/econdatalibrary"
],
"distribution": [
{
"@type": "DataDownload",
"name": "noaa — CSV",
"encodingFormat": "text/csv",
"contentUrl": "https://econdatalibrary.com/download.html?source=noaa",
"license": "https://www.usa.gov/government-works"
}
]
}{
"@context": {
"@language": "en",
"@vocab": "https://schema.org/",
"cr": "http://mlcommons.org/croissant/",
"data": {
"@id": "cr:data",
"@type": "@json"
},
"dataType": {
"@id": "cr:dataType",
"@type": "@vocab"
},
"sc": "https://schema.org/",
"conformsTo": "dct:conformsTo",
"dct": "http://purl.org/dc/terms/"
},
"@type": "sc:Dataset",
"conformsTo": "http://mlcommons.org/croissant/1.0",
"name": "noaa",
"description": "Monthly; Annual; SANTA ROSA. NOAA NCEI / Climate Data Online. 3,135,873 series, 1763–2026. Free download under public domain.",
"url": "https://econdatalibrary.com/noaa",
"creator": {
"@type": "sc:Organization",
"name": "NOAA NCEI / Climate Data Online"
},
"publisher": {
"@type": "sc:Organization",
"name": "Econ Data Library",
"url": "https://econdatalibrary.com"
},
"license": "https://www.usa.gov/government-works",
"keywords": [
"Climate",
"Environment & Climate"
],
"temporalCoverage": "1763-01-01/2026-12-31",
"citation": "NOAA National Centers for Environmental Information, Global Summary of the Month / Global Summary of the Year (GSOM/GSOY), derived from the Global Historical Climatology Network-Daily dataset. Retrieved from the NCEI bulk archive and redistributed by the Elkassabgi Data Library.",
"distribution": [
{
"@type": "cr:FileObject",
"@id": "noaa-csv",
"name": "noaa-csv",
"description": "Per-series CSV, downloadable with a free API key.",
"contentUrl": "https://econdatalibrary.com/download.html?source=noaa",
"encodingFormat": "text/csv"
}
]
}