Point Processing Tool

Point Processing Tool cindyhg Thu, 04/25/2019 - 09:53

PB2NC

PB2NC cindyhg Thu, 04/25/2019 - 09:54

PB2NC Functionality

The PB2NC tool is used to stratify the contents of an input PrepBufr point observation file and reformat it into NetCDF format for use by the Point-Stat tool. The PB2NC tool must be run on the input PrepBufr point observation file prior to running Point-Stat.

Fortran-Blocking

Please note that in earlier versions of the PB2NC tool, users were typically required to run their PrepBufr files through the cwordsh utility to perform Fortran-blocking prior to running them through PB2NC. This blocking step is now done internally and is set when configuring MET. The --disable-block4 MET compilation option indicates that 4-byte blocking should be performed instead of the default 8-byte blocking. If you experience runtime errors from the PB2NC tool try adding or removing the --disable-block4 flag, recompiling, and rerunning.

PB2NC Usage

View the usage statement for PB2NC by simply typing the following:

pb2nc

At a minimum, the input prepbufr_file, the output netcdf_file, and the configuration config_file must be passed in on the command line. You may use the -pbfile command line argument to run PB2NC using multiple input PrepBufr files, typically adjacent in time.

Configure

Configure cindyhg Thu, 04/25/2019 - 09:56

The behavior of PB2NC is controlled by the contents of the configuration file passed to it on the command line. The default PB2NC configuration may be found in the met-8.0/share/met/config/PB2NCConfig_default file. Additional information about configuration files can be found in $MET_BASE/config/README file. The configuration used by the test script may be found in the met-8.0/scripts/config/PB2NCConfig_G212 file. Prior to modifying the configuration file, users are advised to make a copy of the default:

cp $MET_BASE/config/PB2NCConfig_default $MET_TUTORIAL_DATA/config/PB2NCConfig_tutorial

Open up the $MET_TUTORIAL_DATA/config/PB2NCConfig_tutorial file for editing with your preferred text editor.

The configurable items for PB2NC are used to filter out the PrepBufr observations that should be retained or derived. You may find a complete description of the configurable items in the MET Users Guide or in the $MET_BASE/config/README file. Please take some time to review them.

For this tutorial, edit the $MET_TUTORIAL_DATA/config/PB2NCConfig_tutorial file as follows:

  • Set message_type = [ "ADPUPA", "ADPSFC" ]; to retain only those 2 message types.
  • In obs_window, set beg = -1800; and end = 1800; so that only observations within 30 minutes (1800 seconds) of the reference time will be retained.
  • In mask, set grid = "G212"; to retain only those observations residing within the bounds of NCEP Grid 212 (on which the forecast data resides).
  • Set obs_bufr_var = [ "QOB", "TOB", "ZOB", "UOB", "VOB", "D_WIND", "D_RH" ]; to retain observations for specific humidity, temperature, the u-component of wind, and the v-component of wind and to derive observation values for wind speed and relative humidity. This configuration replaces obs_grib_code. A list of valid GRIB codes, their units, and the corresponding abbreviations may be found here.

Save and close this file.

Run

Run cindyhg Thu, 04/25/2019 - 09:59

Next, run PB2NC on the command line using the following command:

pb2nc \
$MET_TUTORIAL_DATA/input/sample_obs/prepbufr/ndas.t00z.prepbufr.tm12.20070401.nr \
$MET_TUTORIAL_DATA/output/pb2nc/tutorial_pb.nc \
$MET_TUTORIAL_DATA/config/PB2NCConfig_tutorial \
-v 2

PB2NC is now filtering the observations from the input PrepBufr file using the configuration settings we specified and writing the output to the NetCDF file name we chose. This should take a few minutes to run. As it runs, you should see several status messages printed to the screen to indicate progress. You may use the -v command line option to turn off or change the amount of log information printed to the screen.

Runtime Errors

Listed below are two common error messages from the BUFRLIB:

  1. +++++++++++++++++++++++WARNING+++++++++++++++++++++++++
    BUFRLIB: OPENBF - INPUT BUFR FILE IN UNIT 11 IS EMPTY
    +++++++++++++++++++++++WARNING+++++++++++++++++++++++++

    For this error message, ensure that BUFRLIB was built with the *same* family of compilers used to build MET.

  2. **************BUFR ARCHIVE LIBRARY ABORT*****************
    BUFRLIB: OPENBF - FIRST 4 BYTES READ FROM RECORD IN INPUT FILE
    CONNECTED TO UNIT 11 NOT 'BUFR', DOES NOT CONTAIN BUFR DATA
    **************BUFR ARCHIVE LIBRARY ABORT*****************
  3. For this error message, refer to the PB2NC Tool: General page and follow the instructions regarding the --disable-block4 configure option.

Output

Output cindyhg Thu, 04/25/2019 - 10:02

When PB2NC is finished, you'll see a summary of how many PrepBufr messages were processed, retained, and rejected for various reasons. In that output, we see that 69833 PrepBufr messages were processed of which only 8415 were retained. And from those 8415 messages, 62443 observation values were extracted.

You may view the output NetCDF file that PB2NC wrote using the ncdump utility (if available on your machine). Run the following command to view the header of the NetCDF output file:

ncdump -h $MET_TUTORIAL_DATA/output/pb2nc/tutorial_pb.nc

In the NetCDF header, we see that the file contains 22 variables. The obs_ variables contain information about the observations while the hdr_ arrays contain information about the observing location, type, and time. See the comments in the ncdump output for more details.

When processing Global GDAS PrepBufr observations, users should note that the quality marker is set to a value of 9 for surface observations. This is done to prevent them from being used in the data assimilation process. However, these observations will not be retained by the default PB2NC configuration. Users may configure PB2NC with quality_mark_thesh = 9; to retain GDAS surface observations.

ASCII2NC

ASCII2NC cindyhg Thu, 04/25/2019 - 10:05

ASCII2NC Functionality

The ASCII2NC tool is used to reformat ASCII point observations into the intermediate NetCDF format that Point-Stat and Ensemble-Stat accept. In METv8.0, ASCII2NC supports five types of input formats: SURFRAD and Integrated Surface Irradiance Study (ISIS), little_r, wwsis, AERONET, and an 11-column MET point format. Only the MET point format will be discussed in this tutorial for which no configuration file is required. However, the ASCII2NC configuration file found in met-8.0/share/met/config/Ascii2NcConfig_default may be used to define how SURFRAD observations should be summarized through time and a mapping of little_r input strings to output message types.

Point Observation Format

The input ASCII MET point observation format contains one observation per line. Each input observation line should consist of the following 11 columns of data:

  1. Message_Type
  2. Station_ID
  3. Valid_Time in YYYYMMDD_HHMMSS format
  4. Lat in degrees North
  5. Lon in degrees East
  6. Elevation in meters above sea level
  7. Grib_Code as the integer GRIB code value or variable name corresponding to this observation type
  8. Level as the pressure level in hPa or accumulation interval in hours
  9. Height in meters above sea level or above ground level
  10. QC_String corresponding to the quality control value
  11. Observation_Value in the units prescribed for the grib code

In order to use ASCII2NC, it is the user's responsibility to get their ASCII point observations into this format.

ASCII2NC Usage

View the usage statement for ASCII2NC by simply typing the following:

ascii2nc

At a minimum, the input ascii_file and the output netcdf_file must be passed in on the command line. ASCII2NC inspects the input data to determine its format or the user may explicitly specify it using the -format command line option.

Run

Run cindyhg Thu, 04/25/2019 - 10:07

Since ASCII2NC simply performs a reformatting step for data in the MET point observation format, no configuration file is needed.

Run ASCII2NC on the command line twice using the following commands:

ascii2nc \
$MET_TUTORIAL_DATA/input/sample_obs/ascii/sample_ascii_obs.txt \
$MET_TUTORIAL_DATA/output/ascii2nc/tutorial_ascii.nc \
-v 2
ascii2nc \
$MET_TUTORIAL_DATA/input/sample_obs/ascii/precip24_2010010112.ascii \
$MET_TUTORIAL_DATA/output/ascii2nc/precip24_2010010112.nc \
-v 2

ASCII2NC should perform these refomatting steps very quickly.

Output

Output cindyhg Thu, 04/25/2019 - 10:07

When ASCII2NC is finished, you may view the output NetCDF files it wrote using the ncdump utility (if available on your machine). Run the following commands to view the headers of the NetCDF output files:

ncdump -h $MET_TUTORIAL_DATA/output/ascii2nc/tutorial_ascii.nc
ncdump -h $MET_TUTORIAL_DATA/output/ascii2nc/precip24_2010010112.nc

The NetCDF header should look nearly identical to the NetCDF output of PB2NC.

After running MADIS2NC to reformat MADIS data, we'll use the NetCDF output of PB2NC to perform Grid-to-Point verification with the Point-Stat tool.

MADIS2NC

MADIS2NC cindyhg Thu, 04/25/2019 - 10:08

MADIS2NC Functionality

The MADIS2NC tool is used to reformat Meteorological Assimilation Data Ingest System (MADIS) point observations into the NetCDF format expected by the Point-Stat tool. More information about MADIS data and formatting is available at NOAA's MADIS page. Since the MADIS2NC tool simply performs a reformatting step, no configuration file is needed.

MADIS2NC Usage

View the usage statement for MADIS2NC by simply typing the following:

madis2nc

At a minimum, the input madis_file and the output out_file, and the type string (-type str) must be passed in on the command line.

Run

Run cindyhg Thu, 04/25/2019 - 10:08

Since MADIS2NC simply performs a reformatting step, no configuration file is needed.

Run MADIS2NC on the command line using the following command:

madis2nc \
$MET_TUTORIAL_DATA/input/sample_obs/madis/metar/metar_2012040912_F000.nc \
$MET_TUTORIAL_DATA/output/madis2nc/metar_2012040912_F000.nc \
-type metar -rec_beg 2000 -rec_end 6000 -v 2

MADIS2NC is now reformatting the metar observations, processing only record numbers 2000 through 6000. MADIS2NC should perform this refomatting step fairly quickly.

Output

Output cindyhg Thu, 04/25/2019 - 10:09

When MADIS2NC is finished, you may view the output NetCDF file it wrote using the ncdump utility (if available on your machine). Run the following command to view the header of the NetCDF output file:

ncdump -h $MET_TUTORIAL_DATA/output/madis2nc/metar_2012040912_F000.nc

Next, we'll use the NetCDF output of PB2NC to perform Grid-to-Point verification with the Point-Stat tool.

LIDAR2NC

LIDAR2NC cindyhg Thu, 04/25/2019 - 10:10

LIDAR2NC Functionality

The LIDAR2NC tool is used to read Calipso and CATS satellite lidar datasets.

LIDAR2NC Usage

View the usage statement for LIDAR2NC by simply typing the following:

lidar2nc

At a minimum, the input lidar_filename and the output nc_filename must be passed in on the command line.

Run

Run cindyhg Thu, 04/25/2019 - 10:16

Since LIDAR2NC simply performs a reformatting step, no configuration file is needed.

Run LIDAR2NC on the command line using the following command:

lidar2nc \
$MET_TUTORIAL_DATA/input/lidar_data/CAL_LID_L2_05kmCLay-Prov-V3-40.2016-12-01T01-24-58ZN.hdf \
-out $MET_TUTORIAL_DATA/output/lidar2nc/CAL_LID_L2_05kmCLay-Prov-V3-40.2016-12-01T01-24-58ZN.nc 

 

LIDAR2NC is now reformatting the Calipso lidar file into NetCDF.

Output

Output cindyhg Thu, 04/25/2019 - 10:17

When LIDAR2NC is finished, you may view the output NetCDF file it wrote using the ncdump utility (if available on your machine). Run the following command to view the header of the NetCDF output file:

ncdump -h $MET_TUTORIAL_DATA/output/lidar2nc/CAL_LID_L2_05kmCLay-Prov-V3-40.2016-12-01T01-24-58ZN.nc