One account. Every library.
HF Data Library · Econ Data Library · more to come
FULL-coverage ingest of Ember's published electricity/energy data (CC BY 4.0). Ember publishes every dataset shown on ember-energy.org/data as flat CSVs on a public Google Cloud Storage bucket (emb-prod-bkt-publicdata). The bucket is openly LISTABLE via the GCS JSON API, so we enumerate the ENTIRE…
Ember (electricity data). Retrieved from https://ember-energy.org. Compiled and redistributed by the Elkassabgi Data Library.
Processing: Retrieved from the official source, normalized to a long {series_key, obs_date, value} schema (period-start dates), de-duplicated, and stored as zstd Parquet. Compiled and redistributed by the Elkassabgi Data Library.
FULL-coverage ingest of Ember's published electricity/energy data (CC BY 4.0). Ember publishes every dataset shown on ember-energy.org/data as flat CSVs on a public Google Cloud Storage bucket (emb-prod-bkt-publicdata). The bucket is openly LISTABLE via the GCS JSON API, so we enumerate the ENTIRE catalog rather than hand-curating a slice. We take every *current* CSV under `public-downloads/` and EXCLUDE the `*-archive/` point-in-time snapshots (those are historical copies of the current files, not new data). GROUPED STORAGE (anti-bloat): ONE Parquet per source CSV -> data/clean_full/ember/<dataset_id>.parquet Each Parquet is a faithful long table carrying a `series_key` column (the comma-joined identifying dimensions), plus obs_date, value, unit, and the descriptive dimension columns. 86 source CSVs -> at most ~86 Parquet files for the whole source. Schema families handled: A. global/europe/india/us "*_full_release_long_format" -> Category/Subcategory/ Variable/Unit/Value with Year or Date (+ optional State for us/india). B. capacity monthly wind/solar (Month,Year,...,Installed Capacity,...). C. customs / solar-export long tables (Area,Date,...,Amount (USD)/(kg)/(items)). D. european wholesale price (Country,ISO3,Date,Price). E. methane / NECP / RES-tracker / ISET / methane_imeo (assorted dim+value+year). F. wide "graphic"/chart helper + Turkiye/price-chart files (columns are geos/fuels) -> melted to long so every cell is one observation. For any file we cannot map to a known shape, we fall back to a generic melter that keeps non-numeric columns as the key and melts numeric columns to long. Usage: python jobs/ingest_ember.py --discover # enumerate catalog only, no download python jobs/ingest_ember.py --dry # download+parse, print, NO parquet python jobs/ingest_ember.py # full run (download + write parquet) python jobs/ingest_ember.py --only yearly_full_release_long_format # one dataset
{
"@context": "https://schema.org/",
"@type": "Dataset",
"name": "Ember (electricity)",
"description": "FULL-coverage ingest of Ember's published electricity/energy data (CC BY 4.0). Ember publishes every dataset shown on ember-energy.org/data as flat CSVs on a public Google Cloud Storage bucket (emb-prod-bkt-publicdata). The bucket is openly LISTABLE via the GCS JSON API, so we enumerate the ENTIRE…",
"url": "https://econdatalibrary.com/ember.html",
"identifier": "ember",
"isAccessibleForFree": true,
"publisher": {
"@type": "Organization",
"name": "Econ Data Library",
"url": "https://econdatalibrary.com"
},
"creator": {
"@type": "Organization",
"name": "Ember (electricity)"
},
"license": "https://creativecommons.org/licenses/by/4.0/",
"keywords": [
"energy",
"ember"
],
"temporalCoverage": "2000-12-31/2026-04-01",
"repeatFrequency": [
"P1Y",
"P1M"
],
"creditText": "Source: Ember (CC BY 4.0)",
"citation": "Ember (electricity data). Retrieved from https://ember-energy.org. Compiled and redistributed by the Elkassabgi Data Library.",
"sameAs": [
"https://huggingface.co/datasets/econdatalibrary-ember",
"https://zenodo.org/communities/econdatalibrary"
],
"distribution": [
{
"@type": "DataDownload",
"encodingFormat": "text/csv",
"contentUrl": "https://econdatalibrary.com/download.html?source=ember",
"license": "https://creativecommons.org/licenses/by/4.0/"
}
]
}{
"@context": {
"@language": "en",
"@vocab": "https://schema.org/",
"cr": "http://mlcommons.org/croissant/",
"data": {
"@id": "cr:data",
"@type": "@json"
},
"dataType": {
"@id": "cr:dataType",
"@type": "@vocab"
},
"sc": "https://schema.org/",
"conformsTo": "dct:conformsTo",
"dct": "http://purl.org/dc/terms/"
},
"@type": "sc:Dataset",
"conformsTo": "http://mlcommons.org/croissant/1.0",
"name": "ember",
"description": "FULL-coverage ingest of Ember's published electricity/energy data (CC BY 4.0). Ember publishes every dataset shown on ember-energy.org/data as flat CSVs on a public Google Cloud Storage bucket (emb-prod-bkt-publicdata). The bucket is openly LISTABLE via the GCS JSON API, so we enumerate the ENTIRE…",
"url": "https://econdatalibrary.com/ember.html",
"creator": {
"@type": "sc:Organization",
"name": "Ember (electricity)"
},
"publisher": {
"@type": "sc:Organization",
"name": "Econ Data Library",
"url": "https://econdatalibrary.com"
},
"license": "https://creativecommons.org/licenses/by/4.0/",
"keywords": [
"energy"
],
"temporalCoverage": "2000-12-31/2026-04-01",
"citation": "Ember (electricity data). Retrieved from https://ember-energy.org. Compiled and redistributed by the Elkassabgi Data Library.",
"distribution": [
{
"@type": "cr:FileObject",
"@id": "ember-csv",
"name": "ember-csv",
"description": "Per-series CSV, downloadable with a free API key.",
"contentUrl": "https://econdatalibrary.com/download.html?source=ember",
"encodingFormat": "text/csv"
}
]
}