IOBS
iOBS (Improved Observation Usage in Numerical Weather Prediction)
This page contains public information about the project iOBS - Improved Observation Usage in Numerical Weather Prediction.
Phase: Execution
Project Partners
- CSC - IT Center for Science Ltd.
- Finnish Meteorological Institute (FMI), FI
- Norwegian Meteorological Institute MET Norway, NO
- Nordic e-Infrastructure Collaboration (NeIC)
- Swedish Meteorological and Hydrological Institute (SMHI), SE
Background and Goals
This 10M NOK (50% in-kind), two-year cooperation project is planned to spend an effort equal to approximately 100 person months on improving observation usage in Numerical Weather Prediction (NWP). The iOBS initiative will accommodate an increasing amount and diversity of observation data, and provide a system of harmonised data pooling and merging. Observations from the “Internet of things” (IoT), such as intelligent cars, phones, buildings and personal weather stations (PWS), including commodity weather sensors, provide detailed information on local to hyper-local meteorological phenomena. The targeted breakthrough and measurable benefit of this project is the effective assimilation of diverse observations in regional high-resolution NWP models for the delivery of reliable and accurate weather forecasts and warnings for the benefit of operations, business and society. The basis will be the current operational NWP model, AROME-MetCoOp and/or the very recent addition of a nowcasting suite; ultra-local resolution observation network (both spatial and temporal) may be even more suitable for very high-resolution NWP models on the sub-kilometric scale, and the project results will be a valuable contribution to its on-going development.
The iOBS project contributes to improved weather forecast quality by the improved use of existing and emerging observation types in operational NWP combined with Glenna2 assisted future generation e-infrastructure: machine learning, analytics and IaaS support.
Expected Benefit
The project will improve, develop and implement timely quality control (QC) algorithms for a massive amount of private observations of surface pressure using an existing data source covering the Nordic countries. Since the Scalable Acquisition and Pre-Processing system (SAPP) is modular the QC could be added to the overall observation handling. Other emerging observation types will also be explored within the limits of the project. If successful, this will to our knowledge be for the first time private pressure observations are assimilated in an operational NWP system. There are some research on assimilation of pressure from mobile phones, e.g. at Danish Meteorological Institute (DMI), and these case studies have identified a potential increase in forecast accuracy by introducing these observations in NWP. Preliminary investigation into both mobile phone pressure data and private in-situ pressure data, shows more promise to the latter due to its stationarity. Mobile phones introduce an artificial pressure tendency when it is moving, especially vertically. Regarding private observations we will maintain the data history and protect personal information and/or sensitive data.
To summarize, the quantifiable benefits in this 2-year project include:
- Improved NWP forecast quality from increased number of observations used in data assimilation
- Improved QC algorithms for pre-processing private observations where machine learning approaches might help to further identify important observation errors and/or instrument malfunctions
- Reduced cost for software maintenance and development for increased efforts in research and development
- Improved conditions for Nordic research collaboration on both novel technologies and handling of different observation types
- Knowledge transfer across scientific disciplines and technological solutions
- Redundancy and flexibility by using both a cloud based research infrastructure (Glenna-2) and a proven operational infrastructure (PPI)
- Raise awareness of benefits of public-private partnerships, e.g. our QC will inform data manufacturers about their data quality (a interest that they have already expressed)
Work packages
The work in the project is divided into work packages:
- Work package 0 Project management
- Work package 1 Pre-processing setup: Establish a continuous in time and redundant reception, storage and pre-processing (SAPP) of in-situ conventional observations directly from GTS on both Glenna2 and PPI
- Work package 2 Machine learning QC algorithms of conventional observations: Investigate machine learning approaches to observations QC on conventional observations
- Work package 3 Private in-situ observations QC: Implement novel QC and pre-processing on a massive amount of private in-situ observations, e.g. Netatmo pressure observations
- Work package 4 Data assimilation studies: Create a test environment to demonstrate the impact on weather forecasts of introducing current and novel in-situ observations in state-of-the-art data assimilation
- Work package 5 E-infrastructure evaluation: Evaluate how future e-infrastructure could make use of cloud and HPC resources. These resources are needed to tackle technical challenges coming from both data ingestion from non-conventional (private) data sources and from integration of machine learning components to standard IT systems.
People
Steering group
Meetings, 3-4 per year: iOBS steering and reference group minutes
Please see the NeIC official website.
Reference group
Please see the NeIC official website.
Project personnel
Please see the NeIC official website.
Management
Please see the NeIC official website.
Meetings, Weekly: Management minutes (internal).
Outreach
- iOBS events
- Interviews etc:
Public documents
Results:
Approved deliverables
- Deliverables 1.1 and 3.1: A data management document on existing observations types available for data assimilation in Numerical Weather Prediction (NWP) File:D1-1-D3-1_iOBS_Observation_data_inventory.pdf
- Deliverable 3.4: Quality controlled bias corrected data set with error characterization for use in deliverable 4.2
- Deliverable 1.2 Prototype system on Glenna2 and PPI for transfer, storage and pre-processing relevant observations and results
- Deliverable 1.3 Monitoring systems to provide logged feedback on data quality to the data providers (NMS), possibly using SAPP logs
- Deliverable 4.1: A model system that handles conventional observations pre-processed using SAPP.
- Deliverable 0.1: Mid-term report to the NeIC Board
- Deliverable 3.2: QC algorithms for the prototype private in-situ datasets. Open repositories
- Deliverable 3.3: Monitoring systems for feedback on data quality to the private data providers
- Deliverable 4.2: Quality controlled, bias corrected data sets with error characterization implemented in HARMONIE data assimilation
- Deliverable 4.3: Advanced diagnostic tools to infer the potential of emerging observation types File:D4.3_Advanced_diagnostic_tools_to_infer_the_potential_of_emerging_observation_types.pdf
- Deliverable 4.4: Report: Study of forecast quality improvement (magnitude and duration), relative to using a reference set of observations, from assimilation of novel observations (Paper to be published in peer reviewed journal, report to be published through Zenodo if journal allows)
- Deliverable 5.1: Report on e-infrastructure evaluation, including any documentation. The report is published on Zenodo
- Deliverable 2.1: Machine learning algorithm approaches to observations quality control. Open repositories
- Deliverable 2.2: Algorithm descriptions for the ML-based spatial QC analysis and the automatic adaptive threshold method The report on D2.1 and D2.2 is published on Zenodo
- Deliverable 0.2: Final report File:Final_report_iOBS_20210811.pdf
Deliverables to be approved
All deliverables approved.
Governance documents:
- Collaboration Agreement (including Project Directive and suggested Terms of Reference for Steering Group)
- Project Directive (and other Appendices to Collaboration Agreement)
- Agreed Terms of Reference for the iOBS Steering Group