8000 GitHub - open-meteo/open-data: Open-Meteo on AWS Open Data
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

open-meteo/open-data

Repository files navigation

Open-Meteo on AWS Open Data

AWS Bucket Name and Region: s3://openmeteo; us-west-2; AWS Registry

Open-Meteo integrates weather models from well-known national weather services, delivering a rapid weather API. Real-time weather forecasts are unified within a time-series database that covers both historical and future weather data. Open-Meteo is designed to analyse long time-series of weather data any place on earth.

This database is made available through the AWS Open Data Sponsorship program.

Weather datasets are sourced from the following national weather services:

  • Forecast: NOAA NCEP, DWD, ECMWF, Environment Canada, MeteoFrance, JMA, BOM, CMA, Met Norway, DMI, KNMI, KMA, ItaliaMeteo
  • Marine Weather: ECMWF, MeteoFrance, Copernicus Marine, DWD, NOAA NCEP
  • Air Quality: CAMS
  • Historical data: Copernicus, ECMWF

This open-data distribution is managed by Open-Meteo and is not directly affiliated with national weather services. Open-Meteo does not guarantee the accuracy, completeness, or uninterrupted provision of the data products, and they are provided without any warranty. For support inquiries, please contact Open-Meteo by creating issues or discussions in this repository.

Weather Models

All available data can be explored using the S3 explorer.

Weather Forecast Models

Weather models can be broadly categories by their coverage:

  • Global models run at lower resolution (11-50 km) but offer 7 to 16 days of forecast
  • Local models use higher resolution (1-7 km) but offer only 2-5 days of weather forecast

Local models are nested into global models and rely on boundary conditions that drive large scale weather patterns. The Open-Meteo API seamlessly combines local and global weather models. Depending on your use-case, you may want to use different weather models. E.g. If you only need 2 days of forecast for North America, use ncep_hrrr_conus, but for more than 2 days, you have to add ncep_gfs013. Further more, you can select only temperature_2m to more fine grained of how much data is being transferred.

Ideally, familiarise yourself with the Weather Forecast API and explore the S3 explorer to select the right weather models.

Model Region Resolution Timeinterval Forecast length Updates # Surface Variables # Pressure Variables Available since
dwd_icon Global 0.1° (~11 km) Hourly 7.5 days Every 6 hours 49 5 (18 levels) 2023-12-15
dwd_icon_eu Europe 0.0625° (~7 km) Hourly 5 days Every 3 hours 42 5 (17 levels) 2023-12-15
dwd_icon_d2 Central Europe 0.02° (~2 km) Hourly 2 days Every 3 hours 44 5 (11 levels) 2023-12-15
dwd_icon_d2_15min " " 15-Minutely " " 8 - 2023-12-15
ncep_gfs013 Global 0.11° (~13 km) Hourly 16 days Every 6 hours 27 - 2023-12-15
ncep_gfs025 Global 0.25° (~25 km) Hourly 16 days Every 6 hours 11 7 (38 levels) 2023-12-15
ncep_nbm_conus U.S. Conus 2.5 km Hourly 11 days Every hour 20 - 2024-10-03
ncep_hrrr_conus U.S. Conus 3 km Hourly 2 days Every hour 23 7 (39 levels) 2023-12-15
ncep_hrrr_conus_15min " " 15-Minutely " " 12 - 2023-12-15
meteofrance_arpege_world025 Global 0.25° (~25 km) Hourly 4 days Every 6 hours 29 6 (23 levels) 2023-12-15
meteofrance_arpege_europe Europe 0.1° (~11 km) Hourly 4 days Every 6 hours 29 6 (23 levels) 2023-12-15
meteofrance_arome_france0025 France 0.025° (~2.5 km) Hourly 51 hours Every 3 hours 29 6 (24 levels) 2023-12-15
meteofrance_arome_france_hd " 0.01° (~1.5 km) " " " 12 - 2023-12-15
ecmwf_ifs04 Global 0.4 (~44 km) 3-Hourly 10 days Every 6 hours 14 7 (9 levels) 2023-12-15
ecmwf_ifs025 Global 0.25 (~25 km) 3-Hourly 10 days Every 6 hours 14 7 (9 levels) 2024-02-03
ukmo_global_deterministic_10km Global 0.09 (~10 km) Hourly 7 days Every 6 hours 19 5 (59 levels) 2022-03-01
ukmo_uk_deterministic_2km UK, Ireland 2 km Hourly 2 days Every hour 24 5 (59 levels) 2022-03-01
cmc_gem_gdps Global 0.15° (~15 km) 3-Hourly 10 days Every 12 hours 24 5 (31 levels) 2023-12-15
cmc_gem_rdps North America, North Pole 10 km Hourly 3.5 days Every 6 hours 24 5 (31 levels) 2023-12-15
cmc_gem_hrdps Canada, Northern US 2.5 km Hourly 2 days Every 6 hours 24 5 (28 levels) 2023-12-15
jma_gsm Global 0.5° (~55 km) 6-Hourly 11 days Every 6 hours 8 6 (11 levels) 2023-12-15
jma_msm Japan, Korea 0.05° (~5 km) Hourly 4 days Every 3 hours 11 - 2023-12-15
metno_nordic_pp Norway, Denmark, Sweden, Finland 1 km Hourly 2.5 days Every hour 9 - 2023-12-15
cma_grapes_global Global 0.125° (~13 km) 3-Hourly 10 days Every 6 hours 48 8 2024-01-01
bom_access_global Global 0.175°/0.117° (~15 km) Hourly 10 days Every 6 hours 33 - 2024-01-01
dmi_harmonie_arome_europe Central & Northern Europe 2 km Hourly 60 hours Every 3 hours 39 - 2024-07-01
knmi_harmonie_arome_europe Central & Northern Europe 5.5 km Hourly 60 hours Every hour 22 5 (5 levels) 2024-07-01
knmi_harmonie_arome_netherlands Netherlands, Belgium 2 km Hourly 60 hours Every hour 28 - 2024-07-01
kma_gdps Global 0.13° (~12 km) 3-Hourly 12 days Every 6 hours 28 - 2024-07-01
kma_ldps South And North Korea 1.5 km Hourly 2 days Every 6 hours 28 - 2024-07-01
italia_meteo_arpae_icon_2i Southern Europe 2 km Hourly 60 hours Every 12 hours 28 - 2025-04-13

Marine Wave Models

The following ocean wave models are integrated into the Marine Wave API.

Model Region Resolution Timeinterval Forecast length Updates # Surface Variables # Pressure Variables Available since
ecmwf_wam025 Global 0.25° (~25 km) 3-Hourly 10 days Every 6 hours 4 - 2024-03-01
meteofrance_currents Global 0.08° (~8 km) Hourly 10 days Every 24 hours 1 - 2022-01-01
meteofrance_wave Global 0.08° (~8 km) 3-Hourly 10 days Every 12 hours 9 - 2021-10-01
meteofrance_sea_surface_temperature Global 0.08° (~8 km) 6-Hourly 10 days Every 24 hours 1 - 2022-01-01
ncep_gfswave025 Global 0.25° (~25 km) Hourly 16 days Every 6 hours 9 - 2024-06-20
dwd_gwam Global 0.25° (~25 km) Hourly 7.5 days Every 12 hours 11 - 2023-12-15
dwd_ewam Europe 0.05° (~5 km) Hourly 4 days Every 12 hours 11 - 2023-12-15
copernicus_era5_ocean Global 0.5° (~50 km) Hourly 5 days delay Every 24 hours 5 - 2023-12-15

Air Quality Models

The following models are used in the Air Quality API

Model Region Resolution Timeinterval Forecast length Updates # Surface Variables # Pressure Variables Available since
cams_global Global 0.4° (~44 km) Hourly 4.5 days Every 12 hours 10 - 2023-12-15
cams_europe Europe 0.1° (~11 km) Hourly 4 days Every 12 hours 14 - 2023-12-15

Historical Weather Data

The following models are used in the Historical Weather API.

Model Region Resolution Timeinterval Delay to realtime Updates # Surface Variables # Pressure Variables Available since
copernicus_era5 Global 0.25° (~25 km) Hourly 5 days Every 24 hours 23 - 1940-01-01
copernicus_era5_land Global 0.1° (~11 km) Hourly 5 days Every 24 hours 11 - 1950-01-01
ecmwf_ifs Global 9 km Hourly 2 days Every 24 hours 24 - 2017-01-01

Digital Elevation Models

Based on the GLO-90 digital elevation model (DEM) from Copernicus, the weather API uses terrain information to optimise and downscale weather data. Although the Open-Meteo weather API works without elevation information, forecasts in mountainous terrain is less accurate.

Model Region Resolution Timeinterval Forecast length Updates # Surface Variables # Pressure Variables Available since
copernicus_dem90 Global 90 m - - - - - 2023-12-15

Other Models

Climate, flood, satellite and ensemble models are not published on AWS due to their immense size.

Data Organization

Data is structured by weather models, variable and time. This allows to retrieve only a small subset of required data. In many cases only a limited number of weather variables like temperature is required significantly reducing the required data size.

Bucket contents:

  • data/<model>/<weather-variable>/<time>.om
  • data_spatial/<model>/YYYY/MM/DD/HHMMZ/<variable>.om
  • README.md

Types:

  • data: Data is optimized for time-series access. Meaning you can access long time-series for a individual location quickly.
  • data_spatial: Data is stored per time-step and is ideal to read large areas and generate interactive maps, but only for a single time-step. Internally each file is chunked by [32,32] and enables cloud native access to a small portion of data of each file.

URL components:

  • model: All data is grouped by weather model. E.g. ncep_gfs013 or dwd_icon_eu
  • variable. Each model contains multiple weather variables. E.g. temperature_2m or relative_humidity_2m. Some weather variables like wind_speed_10m are calculated by the API, but wind_u_component_10m and wind_v_component_10m are stored.
  • time: For each variable, data is split by time. This can be an entire year for historical data, or chunks of 1-2 weeks of data. An entire year is specified like year_2010.om while chunks of varying size use arbitrary indices like chunk_927382.om
  • .om file extension: All data is stored in compressed multi-dimensional arrays. For an optimal compression, a custom file format is used. See below.

Updates to Real-time Weather Forecasts

Real-time weather models refresh every 1, 3, 6, or 12 hours. Once the first data is published by the national weather services, Open-Meteo starts downloading and processing data. Data for spatial access data_spatial is generated and uploaded immediately after each time-step has been processed. Upon completion of a new weather model run, the time-series optimized database is generate and uploaded to AWS.

Time-series data is structured into chunks covering 3 to 14 days per file, resulting in existing files being overwritten with the most recent data. The length of each time-chunk is manually picked for each model for a good balance between file size, compression ratio and read performance.

Spatial data overwrites updated timestamps with each run. Only the most recent data remains for each time-step. It is therefore possible to also generate maps of the past.

Typically, historical weather data doesn't undergo updates. However, in the case of ERA5, daily updates are applied with a 5-7 day delay. Older historical data spanning the past 80 years remains unaltered, of course.

Download and Interact With Data

Open-Meteo provides a free API for quick data retrieval without the need to download from the AWS bucket.

However, there are two primary scenarios where downloading data locally is advantageous:

  1. Research with Historical Weather Data: Conducting intensive analyses on millions of events with varying locations and time steps is facilitated by having data available locally or on a dedicated high-performance VM instance. With Open-Meteo on AWS Open Data, you can download temperature data for the past 80 years using the Copernicus ERA5-Land dataset. Basic steps include:
  • Installing the Open-Meteo Docker image docker pull ghcr.io/open-meteo/open-meteo
  • Download archived ERA5 data for temperature from AWS docker run open-meteo sync copernicus_era5_land temperature_2m --past-days 730 (roughly 8 GB)
  • Launch your local API endpoint docker run -p 8080:8080 open-meteo serve
  • Get data for individual coordinates curl "http://127.0.0.1:8080/v1/archive?latitude=47.1&longitude=8.4&hourly=temperature_2m&start_date=20220101&end_date=20231031"

Note: Docker commands are exemplary. Follow the Tutorial for downloading historical weather data to get it working quickly.

  1. Running Your Own Weather API: If you require an extensive amount of weather data through an API daily and wish to run your own weather API, you can obtain current weather data from Open-Meteo on AWS Open Data. The Open-Meteo Docker container can listen for newly published data and keep your local database up-to-date. Similar to using past weather data:
  • Install the Open-Meteo Docker image
  • Start the data synchronization for a given weather model docker run open-meteo sync ncep_gfs013 temperature_2m,relative_humidity_2m,wind_u_component_10m,wind_v_component_10m --past_days 3 --repeat-interval 5
  • Launch the API instance and get the latest forecast from your new API endpoint

To help you in setting up your own weather API you can follow this tutorial to setup your own weather API.

File Format

Data is stored in a custom file format called OM-Files, designed for chunked data access and efficient compression. The structure is similar to NetCDF or HDF5 but with significantly lower overhead.

OM-Files are cloud-native and can be accessed directly from S3. Thanks to small chunk sizes, users can retrieve only the data they need—without downloading the entire file. Chunks typically range from 1–4 KiB. For example, a temperature dataset like ERA5-Land might be 9 GiB, but reading data for a single location requires just 16 KiB—including all metadata, indices, and data.

OM-Files can also be written sequentially and streamed directly to S3. Paired with a fast chunking and compression pipeline, this allows for data generation at GB/s speeds. This high performance is essential for working with large-scale meteorological datasets. Open-Meteo alone processes over 2 TiB of weather data every day.

The underlyin 577C g OM-File library is implemented in C — source code available here. OM-Files can also be read from Python, Rust, Swift, and TypeScript WASM.

License

CC-BY-4.0

0