UC7: Climate Assessment Based on Past Data

Preamble

This usecase is based on the activities of the Atmospheric Circulation Reconstruction over Earth (ACRE) initiative. In order to support the INSPIRE TWG, the ACRE scenario described here will be complemented by a number of user stories of how to use the past-data / climatology for decision support.

ACRE

Overview

The Atmospheric Circulation Reconstruction over Earth (ACRE) initiative (http://www.met-acre.org/) aims to help understand climate variability and climate change from synoptic to decadal timescales throughout the 20th century. ACRE combines data captured from ships logs, other historical records and existing digital weather observation archives with advanced data-assimilation methods to drive reanalysis of historical weather conditions. It provides a unique long-term weather database which can be 'downscaled' to finer resolution to serve a diverse community of users including climate researchers, educators, students and the general public.

ACRE, and the family of projects it is associated with, aim to determine whether the models of past and future climate correlate – or are likely to correlate – with the kind of events that happened in the past.

The initiative works closely with the international surface weather observations data community, linking with the international RECLAIM (RECovery of Logbooks And International Marine data) (http://icoads.noaa.gov/reclaim/), ISPD (global maps of historical terrestrial atmospheric pressure station distribution can be found at: ftp://ftp.ncdc.noaa.gov/pub/data/ispd/add-station/) through the National Climatic Data Centre (NCDC) in the US and ICOADS (http://icoads.noaa.gov/) projects, plus the Climatological Database for the World’s Oceans; 1750 to 1850 (CLIWOC) (http://www.ucm.es/info/cliwoc/) and UK Colonial Registers and Royal Navy Logbooks (CoRRaL) (http://www.corral.org.uk/) project participants, and various international academics and archives, to expand the recovery, imaging and digitisation of historical instrumental weather observations in terrestrial registers and marine logbooks. All of this material is then available to ACRE’s US partners for their historical global reanalyses - these have begun with the 20th Century Reanalysis Project (http://www.esrl.noaa.gov/psd/data/20thC_Rean/).

Recovery, imaging and digitisation of historical global surface weather observations

With endorsement from organizations such as the World Meteorological Organisation (WMO), the Group on Earth Observations (GEO), the Global Climate Observing System (GCOS), wide international support and the aid of various working groups of GCOS and the World Climate Research Program (WCRP), ACRE provides an umbrella that links together some 35+ projects, institutions, organisations, data rescue and climate applications activities around the globe. In 2010, ACRE and its activities were ratified by the WMO Commission for Climatology, extolled in a letter of recognition from GCOS, and endorsed by the JCOMM Expert Team on Marine Climatology and, most recently, WCRP.

Ship logbook and assorted terrestrial collections

The English East India Company (EEIC) (1780s-1830s) - 900 logbooks from ships of the EEIC held in the British Library contain historical instrumental pressure, air temperature and sea surface temperature data. These have now all been imaged by the British Library; they are being digitised by CDMP and the digitised data from about half of the logs have been completed and made available for preliminary checking (see the latest report on these data at: http://philip.brohan.org.transfer.s3.amazonaws.com/EIC_summary.pdf).

CoRRal: UK Colonial registers and Royal Naval Logbooks (http://www.corral.org.uk/)–; A UK Joint Information Systems Committee (JISC) Digitisation Programme: Enriching Digital Resources project provided the funds to image and digitise logbooks of ships on voyages of discovery (1700-1850: ADM-55 at The National Archives in the UK), and attached to the UK Hydrographic Survey (1830-1850: held at The National Archives) plus UK colonial meteorological registers, journals and gazettes (held at The National Archives and the Met Office Archives). ACRE digitised large portions of the records of six Bahamian Lighthouses (1870s-1905) and from St Helena (1892-1930) in the South Atlantic and Malden Island (1888-1926) in the Pacific. Digital images of the latter island observations, together with those from other developing countries in the Caribbean, Africa and the Indo-Pacific region have been provided to the British Atmospheric Data Centre (BADC) for display and access on their WWW site (http://badc.nerc.ac.uk/browse/badc/corral/images/metobs). This will ultimately be extended to provide an interim repository of all the images of historical weather observations made by ACRE.

An extended period during and following World War 1 (1914-1923) - 7,000-8,000 Royal Navy logbooks held in the UK National Archives containing instrumental pressure, air temperature and sea surface temperature data. Additional Royal Navy logbooks are being digitized to provide an extended duration of coverage much later into the 20th century (see example below).

Ship remarks books

British hydrographic and survey vessel remarks books (1759-1909) – 6,000 remarks books with weather observations useful to ACRE are held at the UK Hydrographic Office (UKHO). At a minimum, these books contain instrumental pressure, air temperature and sea surface temperature data. An inventory was completed at the end of August 2008, but no funds are available to image and digitise them.

Expeditions, travels, circumnavigations and ships of exploration

A small, on-going initiative under ACRE continues to digitise various marine and terrestrial weather data from printed/published books, journals, ship logbooks and reports that had been scanned and made generally available on the WWW – such as via Google Book Search. Though secondary sources, they often provided valuable insights into the type and extent of instrumental weather observations made, and sometimes are the only surviving source of data if the original material cannot be found or has been destroyed. The latter material was especially valuable in providing essential weather observations for Antarctic regions for the 20th Century Reanalysis Project: 1891-2008 and Brohan et al. (2010).

Inventory of historical Chinese and South China Sea instrumental weather observations

In 2008, ACRE was able to provide funds (from the MOHC's Integrated Climate Program and an AHRC Knowledge Catalyst Scheme grant) for members of the Chinese Maritime Customs Project at Bristol University (http://www.bristol.ac.uk/history/customs/) to undertake an inventory of old daily to sub-daily historical meteorological observations made at Chinese inland stations, and by ships travelling along its rivers, coastline and in the South China Sea. This has provided the basis for the imaging, digitisation and archiving of these data, and ultimately their use in the 20th Century Reanalysis Project and future reanalyses.

ACRE-facilitated reanalyses

ACRE has been working to improve the quality and quantity of historical global surface terrestrial and marine instrumental weather observations going into the ISPD (ftp://ftp.ncdc.noaa.gov/pub/data/ispd/) and ICOADS (http://icoads.noaa.gov/reclaim/) repositories respectively. These vastly improved data bases have provided the historical daily to sub-daily instrumental data required by ACRE’s US partners 20th Century Reanalysis Project to produce a reanalysis dataset spanning the entire twentieth century (1871-2008), assimilating only surface observations of synoptic pressure, monthly sea surface temperature (SST) and sea-ice distributions. The SST and sea-ice data are courtesy of the UK MOHC from their HadISST dataset. The pressure observations dataset, the ISPD, has been assembled through international co-operation under the auspices of not only ACRE, but via links with the working groups of GCOS and WCRP.

The first ACRE-facilitated reanalysis product, global four-times-daily atmospheric and surface analysis fields spanning 1871 to 2008 from the 20th Century Reanalysis Project Version 2 dataset are available at (http://www.esrl.noaa.gov/psd/data/gridded/data.20thC_ReanV2.html) in netCDF format, courtesy of the NOAA ESRL, Physical Sciences Division (PSD), and CIRES/CDC. Reanalyses fields are now also available for the 1871-2008 period via ESRL/PSD/CDC partners at the US National Center for Atmospheric Research (http://dss.ucar.edu/datasets/ds131.1) in GRIB format, and will become available via our partners at NOAA’s National Climatic Data Center (NCDC) through the NOAA Operational Model Archive and Distribution System (NOMADS) (http://nomads.ncdc.noaa.gov) and the NCDC Data Archive in the near future. Additional fields will also become available at ESRL/PSD/CDC.

Beta composite plotting tools for four-times-times daily and monthly mean fields from 1871-2008 are now available courtesy of NOAA/ESRL/PSD and CU/CIRES/CDC at (http://www.esrl.noaa.gov/psd/data/composites/day_20thc/).

Pre-generated synoptic maps showing the ensemble mean analysis and analysis uncertainty of Sea Level Pressure and 500 hPa height over the Northern and Southern Hemispheres and for Sea Level Pressure and 850 hPa zonal wind in the tropics are available at (http://www.esrl.noaa.gov/psd/data/20thC_Rean/hem_images.html).

The Project’s home page can be found at (http://www.esrl.noaa.gov/psd/data/20thC_Rean/)

Notes from discussions with ACRE team

Traceability and provenance of weather observations

The data-capture from ship logbooks and other sources has created millions of JPGs - digital photographic records of the physical source themselves.

The ACRE project aims to provide transparency to its population of users (especially in the light of the Climategate scandal - see BBC News link: http://news.bbc.co.uk/1/hi/8388485.stm). To facilitate this, it is essential to capture the provenance of the observations that constitute each reanalysis - traceability right back to the orginal ships log itself! Furthermore, the ACRE project want to expose the vast collection of digitized records to users who, say, may be researching family history and want to understand the weather conditions on board the ship on which their great grandfather emigrated, or follow the second survey voyage of the HMS Beagle (http://en.wikipedia.org/wiki/HMS_Beagle) with the young Charles Darwin aboard.

Each digitized document will contain a collection of observations; e.g. for each of the phenomena monitored by the ship's crew.

Quality control of source observations

Each reanalysis provides a dynamically consistent model of the earth's atmosphere. The complex algorithms employed in teh numerical model can be used to assess whether an observation that is being assimilated is close enough to the expected result. In this way, the assimilation process can provide quality indicators for the source observations, enabling identification of broken or miscalibrated equipment. Identifying 'black-listed' observations enables the quality of the whole observation dataset to be improved.

What - no observations? Probablistic modelling

The polar regions have very few observations compared to the mid-latitudes. The sparcity of observations leads to a great deal of uncertainty in those regions about the actual state of the atmosphere. To remedy this, the reanalysis is run with 56 ensemble members. The spread of results allows the uncertainty to be assessed, and data to be excluded from the reanalyses result-sets where the uncertainty is deemed too great.

Statistics of the reanalyses

• Approximately 2-degree grid [16200 grid points]
• 140 years of reanalyses at 6-hourly intervals [204400 time points]
• 20 vertical levels
• 56 ensemble members
• Approximately 50 different parameters

That's a dataset with around 185 trillion numbers!

The dataset is spread across approximately 10 million files - stored in netCDF and GRIB

Interesting phenomena & events

Mid-latitude wind storms are a key point of interest (derived from pressure & rainfall fields etc.). The team aim to compile a database of storm events that will enable risk modellers to assert the statistical chance of a storm of a certain ferocity occuring in a given region.

The key point here is how one traverses from scientific GRIDDED datasets to derived features such as individual wind storms. The identification of historical weather event will require sensible weather objects to be defined for those phenomena of interest.

User queries

The kind of simple queries that an intelligent user may wish to execute include:

• time-series of one or more variables at a specific location
• assess the weather status for a given region that tallies with a specific historical event

Given that we are interested in locations, a gazetteer of geographic identifiers will greatly enhance the usability.

The query model should support extraction of the raw reanalysis data for input into subsequent analyses performed by the user where dynamic consistency of the input data is important - or allow the user to sub-sample of interpolate. Whichever mode is used, the user needs to be in control of the options.

Candidate data objects

Observation records

• Collections of observed phenomena (historical instrumental weather observations) - grouped at a specific location & time
  • frequency of observations: daily or sub-daily

• International Comprehensive Ocean-Atmosphere Data Set (ICOADS) (http://icoads.noaa.gov/reclaim/)
• International Surface Pressure Databank (ISPD) (ftp://ftp.ncdc.noaa.gov/pub/data/ispd/)
• Climatological Database for the World's Oceans (1750-1850) (CLIWOC) (http://www.ucm.es/info/cliwoc)
• 900 logbooks from ships of the EEIC held in the British Library
  • historical instrumental pressure, air temperature and sea surface temperature
• UK Colonial Registers and Royal Navy Logbooks (CoRRaL) (http://www.corral.org.uk)
  • logbooks of ships on voyages of discovery (1700-1855)
  • UK colonial meterological registers, journals and gazettes
  • Records of six Bahamian Lighthouses (1870s-1905) & St Helena (1892-1930) and South Atlantic and Malden Island (1888-1926) in the Pacific
• 7000-8000 Royal Navy logbooks help in the UK National Archives
  • instrumental pressure, air temperature and sea surface temperature

Input fields to reanalysis:

• surface observations of synoptic pressure
• monthly sea surface temperature (SST)
• sea ice distributions*

SST and sea-ice data courtesy of UK Met Office Hadley Centre - HadISST dataset

Reanalysis fields

• Regular gridded fields with 56x ensemble members, 20x vertical levels and 2-degree grid spacing
• Monthly mean fields
• ensemble mean analysis & analysis uncertainty of Sea Level Pressure and 500hPa over northern & southern hemispheres, & sea level pressure & 850hPa zonal wind in tropics

outputs are available in both netCDF and GRIB

Note that future versions of the reanalysis will include chemical models.

Derived datasets

• historical global tropical cyclone activity
• global surface air temperature
• historical ENSO events (El Niño-Southern Oscillation)
• major drought episodes around the world
• large scale atmospheric indices and storminess for Northern hemisphere
• rainfall variability plus tracks and intensities of tropical and extra-tropical cyclones
• etc.

Sample datasets

Selection of sample datasets that may be used to drive an interoperability experiment / plugfest.

Each sample dataset needs to have:

• textual description - including parameters, geographic region etc.
• metadata record (if available)
• format specification - including details of any local 'profiles' of standard formats
• controlled vocabulary links

Mapping sample datasets to ISO19156 Observations and Measures

Attempt to map the content of each sample dataset onto an instance of the O&M conceptual model

Where possible, express this in UML

Mapping sample datasets to existing standard forms

Attempt to map the content of each sample dataset onto one or more of the standard forms:

• BUFR
• GRIB2
• CF-netCDF
• NcML-GML
• CSML3
• WXXM

Identify where mapping activity becomes dysfunctional - this will define the limits of interoperability across these formats

Attribution

Many thanks to my colleagues Rob Allan and Philip Brohan from the Met Office Hadley Centre for providing me with much of the information about ACRE

Reference material

The best up-to-date overview of the ACRE project is a feature article for the Bulletin of the American Meterological Society (BAMS). A copy is available here

This article by Gil Compo (NOAA) describes the historical reanalyses facilited by ACRE: http://ascr-discovery.science.doe.gov/feature/weather1.shtml

Several papers recently published on ACRE's data reconstruction work are listed below:

• P. Brohan, C. Ward, G. Willetts, C. Wilkinson, R. Allan, and D. Wheeler, Arctic marine climate of the early nineteenth century, Clim. Past, 6, 315-324, 2010 (http://www.clim-past.net/6/315/2010/cp-6-315-2010.html)
• Philip Brohan, Rob Allan, J. Eric Freeman, Anne M. Waple, Dennis Wheeler, Clive Wilkinson and Scott Woodruff, Marine observations of old weather, BAMS, 90, 2, 219-230 (http://ams.allenpress.com/perlserv/?request=get-abstract&doi=10.1175%2F2008BAMS2522.1)
• Gilbert P. Compo, Jeffrey S. Whitaker, and Prashant D. Sardeshmukh, Feasibility of a 100-Year Reanalysis Using Only Surface Pressure Data, BAMS, 87: 175-190, (http://journals.ametsoc.org/doi/abs/10.1175/BAMS-87-2-175)

INSPIRE TWG derived sub-usecases for exploitation of past-data / climatologies in decision support

-TBD-

-- JeremyTandy - 13 Jul 2010

• ACRE_BAMS.pdf: BAMS Inbox contribution entitled: The International Atmospheric Circulation Reconstructions over the Earth (ACRE) initiative