ANNEX 4: Verification setup

Annex 4

Verification setup

Introduction

Sochi-2014 Olympics and Paralympics will be held on February 8-23 and March 7-16, 2014 correspondingly. Formal forecast evaluation period for the FDP component of FROST-2014 project is tentatively set up from 20 January to 20 March 2014 (it is to be approved by the SSC). Preliminary testing results will be obtained during the pre-trial (winter 2011-12) and final trial (winter 2012-13) periods. Along with formal evaluation in the framework of the demonstration project component various kinds of diagnostic verification will be implemented for better understanding of weak and strong points of the involved forecast systems.

Nowcast and forecast data types to be verified

Grid-point fields

Due to complexity of the region the available objective analysis is neither detailed nor reliable enough to be used as an etalon for forecasts validation. Thus, the grid-point fields will be verified mostly in the points with nearby observations. The precipitation fields will be also verified against radar composites with horizontal resolution of 2 km. Potentiality of the wind field verification on the basis of radar radial wind observations is also considered.

Individual pointwise time series (nowcasts, original and post-prosessed model) at locations associated with observation sites

In order to assess deterministic high-resolution forecast performance of participating technologies varioustime series of near-surface parameters and vertical profiles are to be studied at AMSs, temperature/humidity and wind profilers, MRR(s) (MRR – Micro Rain Radar) locations(Annex 1).

At profiler locations, model fields (temperature, humidity, horizontal wind) are to be verified up to 3 km above the land surface with the resolution of several tens of meters (to be detailed).

MRR vertical resolution is adjustable. Typical values are 30-100 m. The maximum number of vertical steps is 31. The exact vertical structure is to be specified.

As for SNOW-V10, it is of interest to quantify the added value of forecast refinement between:

- Global model;
- Regional model without and with its own data assimilation;
- High-resolution model with and without data assimilation;
- Post-processed model output (Kalman filter, MOS, 1D-model etc.);
- Nowcasting (based on latest observations and blended with NWP).

Verified parameters

The nowcast and EPS output verification is proposed to concentrate on sensible weather and high-impact parameters: precipitation conditions, T2m, near-surface wind, visibility, cloud base, location of the zero isotherm. For deterministic forecasts an expanded list of parameters and levels will be used (Table 4.1).

Table 4.1

	Nowcasting	Deterministic Forecast: FDP	Deterministic Forecast: RDP	Ensemble Forecast: FDP	Ensemble Forecast: RDP
Temperature	T2m	+ See Comment	+ See Comment	T2m	T2m
Relative Humidity	+	+ See Comment	+ See Comment	Ensemble mean & control run	Ensemble mean & control run
Mean Sea Level Pressure	-	+	+	Ensemble mean & control run	Ensemble mean & control run
Liquid Precip. Rate	+	+	+	+	+
Solid Precip. Rate	+	+	+	+	+
Freezing/Frozen Precip (if possible to single out)	+	+	+	+	+
Accumulated snow	+	+	+	+	+
Wind gusts	+	+	+	+	+
Wind Speed & Direction	+	+ See Comment	+ See Comment	+	+
Visibility	+	+	+	+	+
Cloud base height	+	+	+	+	+
Cloudiness	+	+	+	+	+
Height of the zero isotherm	-	+	+	+	+
Snow Surface Temperature	?	+	+	-	-
Liquid water content, Drop Size, Rain Rate, Reflectivity (where provided by models) – on the basis of MRR data	-	-	multi-level ?	-	-

Comment: Temperature, Relative Humidity and Wind forecast data are to be multi-level at grid-points associated with locations of temperature/humidity and wind profilers.

Temporal resolution

Various regional AMSs transmit their reports using different time steps. For example, the land-based AMSs used 10 min steps during the 2010-2011 winter. The AMSs on mobile communication towers reported more frequently. The exact frequency regime is station-dependent and it is to be specified.

10-minutes temporal resolution is planned for the radar scans.

The proposed temporal resolution for forecast and nowcast grid-point fields and time series at locations associated with observation sites is presented in «Forecast data exchange requirements for FROST-2014 project» (The multiple of native model time step can be used for pointwise time series as well).

How the forecast fields are to be verified?

For deterministic forecasts of such parameters as Temperature, Horizontal winds, Humidity, Surface pressure, Cloud base height, Cloudiness, Height of the zero isotherm, the basic scores for continuous variables will be used. For Precipitation rate, snowfall, accumulated snow, Wind gusts and Visibility additional scores will be specified.

A unified analysis of radar and rain gauges data (radar QPE) with horizontal resolution about 2 km will be developed and used for verification of gridded precipitation products from the participating systems. Radar based verification techniques applied to precipitation fields are to be specified.

A set of critical thresholds will be used to transform continuous variables into dichotomous events. These thresholds will be adjusted for the SOCHI region, considering the sport needs of the Olympic Games.Preliminary list of thresholds is presented in the Annex 5. In general SNOW-V10 thresholds are considered to be applicable for FROST-2014.

Common verification measures for dichotomous variables (POD, FAR, TS, ETS, KSS, HSS) will be applied along with some novelty measures, especially for the most extreme events (SEDS, EDI). For EPS output, relevant probabilistic measures (ROC, Brier Score, Reliability Diagram and Frequency Histograms, BSS, RPS, CRPSS) are to be applied. Object-oriented verification, fuzzy verification, and conditional verification will be applied to the project nowcasts and forecasts.

Real-time forecast verification will be implemented to at least a subset of forecast variables, and presented on the FROST-2014 web-site http://frost2014.meteoinfo.ru, whereas various kinds of diagnostic verification statistics can be produced in a delayed mode.