Comment by pkhodiyar
1 year ago
curious to know how you guys store and process the data and make meaningful inferences out of the data collected. Redshift? BigQuery? Datazip?
1 year ago
curious to know how you guys store and process the data and make meaningful inferences out of the data collected. Redshift? BigQuery? Datazip?
Internally yes we use a data warehouse to store the raw observations. We go through a similar process to traditional NWP to produce forecasts except with a few AI steps in between. Unfortunately, the standard currently is to store these as large multi-dimensional data files like grib2/netCDF/zarr.
In case you’re curious, here’s where NOAA stores all their GFS related forecasts: https://registry.opendata.aws/noaa-gfs-bdp-pds/