Simulation of observation and

1.Introduction

In order to decide how to develop meteorological observing systems the benefits from new observations needs to be accurately assessed; such information helps determine which systems and instruments to invest in and which investment will provide best value. The use of Observing System Simulation Experiments (OSSEs) is a well-established technique providing objective and quantitative evaluation of future observing systems and instruments (Masutani et al. 2010b). OSSEs have been proposed and conducted for many years but the amount of work involved in a complete OSSE which includes calibration, (hereafter referred to as a Full OSSE) has been the great obstacle in conducting a reliable OSSE. Ideally, every proposed observing system should be tested by OSSEs before being selected to be built. OSSEs can be important in influencing the configuration of an observing system or the design of a satellite instrument. Subsequently, while the instruments are being built OSSEs can help prepare both the science and software of NWP data assimilation systems for the new observation types.

Simulation experiments use a model-generated proxy for the real atmosphere, commonly called the ‘Nature Run’. This defines the evolution of the atmosphere over the entire experimental period. Simulated observations are generated from the Nature Run and used in data assimilation experiments where their impact on analysis and forecast accuracy can be assessed. In such idealized experiments, the Nature Run also provides the ‘truth’ which all results are verified against. The Nature Run has to be produced using a state of the art NWP model at the highest resolution feasible. In addition, the simulation of observations from a Nature Run requires a large amount of computing resources. Simulations and assimilations may have to be repeated using more than one configuration to achieve a recommendation with confidence. OSSEs require expert knowledge in modeling, data assimilation and observing system technology. Consequently, efficient and effective collaboration is essential for sharing the workload and producing reliable results. Using a common Nature Run is the foundation for these collaborations.

2. Collaboration on Observing System Simulation Experiments

An internationally collaborative effort called Joint OSSE (http://www.emc.ncep.noaa.gov/research/JointOSSEs) was formed in order to perform Full OSSEs. The first Nature Run for the Joint OSSE was produced by ECMWF. The starting point of an OSSE is the Nature Run, which serves as ‘truth’ for the simulations. The team behind the Joint OSSE decided to use a long forecast from the ECMWF model, forced with daily sea-surface temperature and sea ice, as the Nature Run. It is unavoidable that the Nature Run gradually diverges from the real atmosphere during the first few weeks of the forecast. This is not a concern within the context of an OSSE, provided that the climatological statistics of the simulation match those of the real atmosphere.

The first Nature Run for Joint OSSE is a 13-month forecast using cycle 30r1 of ECMWF’s Integrated Forecasting System (IFS) with T511 horizontal resolution (40 km) and 91 levels in the vertical, and the output is saved every 3 hours (hereafter referred as T511NR). The initial condition is the operational analysis on 12 UTC on 1 May 2005 and the Nature Run ends at 00 UTC on 1 June 2006 (Andersson and Masutani 2010b). The T511NR was evaluated and very realistic hurricanes and mid-latitude cyclone statistics were reported (Masutani et al. 2007, Reale et al. 2007, Andersson and Masutani 2010). The cloud distribution is much more realistic than in the previous Nature Run (Masutani et al. 2010a). Statistics of the mid-latitude jet were also studied and found to be realistic. Further evaluation was conducted at NASA/GLA, NOAA/ESRL, Simpson Weather Associates (SWA), NOAA/NCEP, ECMWF and KNMI and the T511NR was found to be satisfactory for conducting OSSEs.

Access to the complete T511NR is available from the NASA/GSFC/NCCS portal system. Currently the data is available from https://portal.nccs.nasa.gov/osse. This data must not be used for commercial purposes and re-distribution rights are not given. ECMWF and Joint OSSE must be given credit in any publication in which this data is used. The list of users is maintained by Michiko Masutani of NCEP and those who are interested in access to the Joint OSSE Nature Run must contact her at michiko.masutani@noaa.gov. Access to the data from this site requires an account, which is available to the research community. User IDs and passwords are available from Ellen Salmon (Ellen.M.Salmon@NASA.gov) of NASA/NCCS.

3. Simulation of Control data for OSSEs

Conventional data are simulated based on 2005-2006 NCEP CDAS usage and simulated for the entire period of the Nature Run. TC vitals are simulated for entire Nature Run period using NCEP software developed at NCEP (Marchok 2002, Lou 2008). Solar Backscatter UltraViolet (SBUV-2) ozone data were also simulated at NCEP for the first four months.

Radiance data have been simulated from the T511NR for an entire year at NOAA/NCEP. The data distribution is based on real observations during the Nature Run period. Radiance data were simulated using the Community Radiative Transfer Model (CRTM) version 1.2.2. Land surface types and vegetation types provided by the Nature Run were used for the simulation. The first simulation was conducted at the foot print used by the NCEP operational system for the four months from May 2005 to August 2005 (SimRad.v3). Then radiance data for the entire Nature Run period were simulated based on NCEP reanalysis usage (SimRad.v4).

Flexible simulation strategies (Fig.1) were used at NCEP-NESDIS when radiance was simulated. It is recognized that radiation experts are not familiar with handling large atmospheric data in grib or Binary Universal Form for the Representation of meteorological data (BUFR) format. Simulating orbits and sampling the data often require a separate project. Contents of each data set are:

Template:

Location of foot prints

Mask for channel

Information about satellite

DBL91:

Nature Run data at foot print

91 level 3-D data (12 Variables)

2-D data (71 Variables)

Climatological data (including land surface type and vegetation)

All other information needed to simulate Radiance from the template file.

It is an option whether to save DBL91 and exchange it among various projects, or to treat DBL91 as a temporary file as it tends to have a large size. The DBL91 is also used for development of the RTM. The various sampling strategies can be provided in the template.

A part of the simulated observations are posted at http://portal.nccs.nasa.gov/josse. This data must not be used for commercial purposes and redistribution rights are not given. Originating institutes must be given credit in any publications in which this data is used. If you are interested in using the data or need assistance please contact the originating institute. As of January 2011, NCEP-NESDIS has posted four months of SimRad.v3 in bufr format and ascii format. The supporting data set is also posted for this period. The support data set includes a bufr template file, DBL91 data, ascii formatted radiance data for easy verification. Four months of SBUV-2 data are also posted. Conventional data is posted for the entire T511NR period.

Some simulated observations are available for verification and are posted at NCEP. As of January 2011 simulated radiance version 4 and simulated TC_vitals are currently available to experts for evaluation. Simulation of observations will be extended to the observation type used for the 2009-2010 period.

4. Calibration of OSSE

In the initial iterations through this cycle, comparisons are made between experiments using simulated and real observations, both using the same data assimilation system and forecast model. Inconsistencies in results between the simulated world and real world may necessitate a calibration of one or more components of the OSSE system. Calibration may involve, for example, improving the errors that are applied to the simulated observations or enhancing certain Nature Run fields to achieve greater realism.

Calibration was conducted at NCEP-JCSDA and NOAA/ESRL using the NCEP GDAS system. At NCEP-JCSDA, an OSSE system was developed using the operational data assimilation system implemented at NCEP in December 2009 (FY10) and SimRad.v3. Calibration was also conducted at NOAA/ESRL using simulated radiance data from NASA/GMAO and the 2007 NCEP GDAS system (Xsi et al 2010) . For both calibrations, conventional data and SBUV-2 data from NCEP and GOES data from NESDIS were used. The comparisons between radiance data simulated by GMAO and SimRad.v3 were conducted at ESRL. SimRad.v3 gave a slightly better skill score compared to the GMAO radiance. Considering the fact that SimRad.v3 did not include observational error and the GMAO radiance does not include radiance over land and ice, the agreement was reasonable.

Although some agreements on the data impact in real and simulated experiments were achieved, there are significant differences which need to be considered when the OSSE results are assessed. Assignment of observational error and error in the data assimilation system are being conducted. After the initial calibration is performed, the OSSE system is ready for actual experiments with new observing systems. Calibration and assessments are iterative processes and repeated when new systems are introduced and provide the limits or the reliability of the OSSE results.

5. Progress in Joint OSSE

The Joint Center for Satellite Data Assimilation (JCSDA) has recognized that it is very important that future observing systems be tested by OSSEs. Now several OSSEs using the T511NR are being conducted. The simulation of DWL for the Global Wind Observing Sounder (GWOS) concept is being conducted by Simpson Weather Associates (SWA) and assimilations are being conducted at JCSDA and NCEP (Riishojgaard et al 2011, Ma 2011, Masutani 2011). Simulation of ADM-Aeolus using LIPAS developed at KNMI is being conducted at NASA/GMAO and OSSEs are being conducted at GMAO. Limited assimilation and verifications are being conducted at NCEP to set up the GSI codes for ADM-Aeolus. OSSEs for an Unmanned Aircraft System (UAS) are in progress at NOAA/ESRL (Xsi et al 2011). OSSEs to evaluate PCW are being conducted at Environment of Canada. The University of Utah, Florida State University, Mississippi State University and more institutes starting OSSEs using the Joint OSSE Nature Run.

One of the first candidate observing systems that is being assessed by this system is a DWL based on the Global Wind Observing Sounder (GWOS) concept developed by NASA in response to the NRC Decadal Survey. A space-borne hybrid (coherent and direct detection) lidar system providing dual perspective (both vector components) wind observations was evaluated. The simulation of DWL was conducted by Simpson Weather Associates (SWA, Wood et al 2009, Emmitt et al 2011). A set of six-week assimilation and forecast experiments have been conducted. The hypothetical impact of the simulated observations on NCEP forecast skill showed a robust positive impact. The impact seen in the standard 500hPa anomaly correlation may not be significant, particularly in the summer hemisphere. However, an improvement in the location of tropical cyclones was noted (Masutani 2011).

DWL assimulation code in the NCEP data assimilation system analysis (Gridpoint Statistical Interpolation, GSI) was tested and an upgrade was posted to be reviewed for implementation. The new version allows DWL with various error characteristics. Improvement from a variational quality control was noted and switched on.

6. The Challenge

It is a challenging task to evaluate the realism of impacts from OSSEs as the results are affected by uncertainties in an OSSE, the differences between the Nature Run and real atmosphere, the process of simulating data, and the estimation of observational errors. The choice of evaluation metrics also affects the conclusions drawn.

Consistency in results is important. However, it is also important to be able to evaluate the source of the errors and uncertainties. As more information is gathered we can perform more credible OSSEs. If the results are inconsistent, the cause of the inconsistency needs to be investigated carefully. NCEP’s OSSEs have demonstrated that carefully conducted OSSEs are able to provide useful recommendations to influence the design of future observing systems (Masutani et al. 2010b). OSSEs are able to provide guidance on where more observations are required and where the model needs to be improved.

Evaluation metric should be carefully selected (Yang 2011). Yang (2011) showed increase of number of good and reduction of number of poor forecast show improvement of forecast much better than traditional skill score.

OSSEs will be conducted by various scientists with different interests. Some are investigating the potential applications of particular prospective instruments. Others may want to aid in the design of the global observing system.

At the current time, the American space agencies are getting involved in prioritizing and funding the most urgent impact simulations from their perspective. Now that the Joint OSSE framework has been built and validated, there is a wide range of possibilities for its application.

Contributors

Michiko Masutani[1,2,#], Jack S. Woollen[1,+],

Zaizhong Ma[2,$], Lars Peter Riishojgaard, [2,$,5],

Tong Zhu[3,@], Haibing Sun[3,%], Nikki Prive[4,@], Yuanfu Xie[4],

Dave Emmitt[6], Sid Wood[6], Steve Greco[6],

Thomas J. Kleespies[3], Fuzhong Weng[3], Yong Han[3]

Ellen Salmon[5] , Bill McHale[5],

Guan Ping Lou[1+], FangLin Yang[1,+], Ronald Vogel[3,+],

Diane Stokes[1], Kate. L. Howard[1,+],

[1]NOAA/National Centers for Environmental Prediction (NCEP)

[2]Joint Center for Satellite and Data Assimilation (JCSDA)

[3]NOAA/ NESDIS/STAR,

[4]NOAA/Earth System Research Laboratory (ESRL)

[5] NASA/GSFC

[6]Simpson Weather Associates

# Wyle Information Systems, McLean, VA,

+IM Sciance Group(IMSG), MD

$Goddard Earth Science and Technology Center, University of Maryland, Baltimore, MD,

@Cooperative Institute for Research in the Atmosphere (CIRA)/CSU, CO

%Perot System Government Services, Virginia

Acknowledgements

The Nature Runs for Joint OSSEs were produced by Dr. Erik Andersson of ECMWF. We appreciate GMAO providing simulated satellite data for calibration at ESRL. GMAO also provided a sample simulation for ADM-Aeolus to compare with DWL data from SWA.

References (Online location to be updated)

Andersson, E. and M. Masutani 2010: Collaboration on Observing System Simulation Experiments (Joint OSSE), ECMWF News Letter No. 123, Spring 2010, 14-16.

[ Available on line at http://www.ecmwf.int/publications/newsletters/pdf/123.pdf ]

Emmitt, G. D. , S. A. Wood , M. J. Kavaya, B. Bentry, and W. Baker 2011: Technology and data utility challenges for a Doppler Wind Lidar on the International Space Station,15th Symposium on Integrated Observing and Assimilation Systems for the Atmosphere, Oceans, and Land Surface (IOAS-AOLS), Seattle, WA, Amer. Meteor. Soc., 2.2,

[Available online at http://ams.confex.com/ams/91Annual/webprogram/Paper182281.html ]

Lou, G. P., G. Dimego, T. P. Marchok, and B. Zhou, (NOAA/NWS/NCEP/EMC) 2008: Operational Global Cyclone Tracking and Verification System.

http://www.emc.ncep.noaa.gov/seminars/presentations/2008/Lou_EMC_seminar_NCEP_global_cyclone_tracking_and_verification.ppt (accessed February 2011)

Marchok, T. P., 2002: How the NCEP tropical cyclone tracker works. Extended Abstract, 25th Conference on Hurricanes and Tropical Meteorolog, San Diego, CA, Amer. Meteor. Soc., P1.13.
[Avaliable online at http://ams.confex.com/ams/pdfpapers/37628.pdf ]

Ma, Z, L. P. Riishøjgaard, M. Masutani 2011: Initial results of Wind lidar Observing Systems Simulation Experiments, Presentation, Working Group on Space –Based Lidar Winds,Miami, FL, Univ. Space Research Association,

[Avalilable online at http://space.hsv.usra.edu/LWG/Feb11/Papers.feb11/Ma.feb11.ppt ]

Masutani, M., and Coauthors, 2007: Progress in Observing Systems Simulation Experiments (a new nature run and International collaboration) Extended Abstracts, 18th Conference on Numerical Weather Prediction,; AMS conference, Parkcity UT, Amer.Meteor. Soc. 12B.5.

[Available online at http://ams.confex.com/ams/pdfpapers/124080.pdf ]

Masutani, M., T. W. Schlatter, R. M. Errico, A. Stoffelen, E. Andersson, W. Lahoz, J. S.. Woollen, G. D. Emmitt,L.-P. Riishøjgaard, S. J. Lord, 2010a: Observing System Simulation Experiments. Data Assimilation: Making sense of observations, Lahoz, W., B. Khattatov, R. Menard, Eds., Springer, 647-679.

Masutani, M, J.S. Woollen, S. J. Lord, G. D.Emmitt, T. J. Kleespies, S. A. Wood, S. Greco, H. Sun, J. Terry, V. Kapoor, R.Treadon, K. A. Campana 2010b: Observing system simulation experiments at the National Centers for Environmental Prediction, J. Geophys. Res., 115, D07101, doi:10.1029/2009JD012528.

Masutani, M. 2011: Progress in Joint OSSE, Simulation of observation for calibration

Preparation for DWL OSSE, Presentation, Working Group on Space –Based Lidar Winds,Miami, FL, Univ. Space Research Association,

[Avalilable online at http://space.hsv.usra.edu/LWG/Feb11/Papers.feb11/Masutani.feb11.ppt ]

Reale, O., J. Terry, M. Masutani, E. Andersson, L.P. Riishojgaard & J.C. Jusem, 2007: Preliminary evaluation of the European Centre for Medium-Range Weather Forecasts’ (ECMWF) Nature Run over the tropical Atlantic and African monsoon region. Geophys. Res. Lett., 34, L22810, doi:10.1029/2007GL031640.

Riishøjgaard, L. P., Z. Ma, M. Masutani, J. Woollen, G. D.Emmitt, S. Wood , S. Greco 2011: Observing System Simulation Experiments for a US Wind Lidar space mission, Presentation, ,15th Symposium on Integrated Observing and Assimilation Systems for the Atmosphere, Oceans, and Land Surface (IOAS-AOLS), Seattle, WA, Amer. Meteor. Soc., 2.5,

[Available online at http://ams.confex.com/ams/91Annual/webprogram/Paper181598.html ]