MET Tool: ASCII2NC

MET Tool: ASCII2NC cindyhg Tue, 06/25/2019 - 09:32

ASCII2NC Tool: General

ASCII2NC Functionality

The ASCII2NC tool reformats ASCII point observations into the intermediate NetCDF format that Point-Stat and Ensemble-Stat read. ASCII2NC simply reformats the data and does much less filtering of the observations than PB2NC does. ASCII2NC supports a simple 11-column format, described below, the Little-R format often used in data assimilation, surface radiation (SURFRAD) data, Western Wind and Solar Integration Studay (WWSIS) data, and AErosol RObotic NEtwork (Aeronet) data. Future version of MET may be enhanced to support additional commonly used ASCII point observation formats based on community input.

MET Point Observation Format

The MET point observation format consists of one observation value per line. Each input observation line should consist of the following 11 columns of data:

  1. Message_Type
  2. Station_ID
  3. Valid_Time in YYYYMMDD_HHMMSS format
  4. Lat in degrees North
  5. Lon in degrees East
  6. Elevation in meters above sea level
  7. Variable_Name for this observation (or GRIB_Code for backward compatibility)
  8. Level as the pressure level in hPa or accumulation interval in hours
  9. Height in meters above sea level or above ground level
  10. QC_String quality control string
  11. Observation_Value

It is the user's responsibility to get their ASCII point observations into this format.

ASCII2NC Usage

View the usage statement for ASCII2NC by simply typing the following:

ascii2nc
Usage: ascii2nc  
  ascii_file1 [...] One or more input ASCII path/filename
  netcdf_file Output NetCDF path/filename
  [-format ASCII_format] Set to met_point, little_r, surfrad, wwsis, or aeronet
  [-config file] Configuration file to specify how observation data should be summarized
  [-mask_grid string] Named grid or a gridded data file for filtering point observations spatially
  [-mask_poly file] Polyline masking file for filtering point observations spatially
  [-mask_sid file|list] Specific station ID's to be used in an ASCII file or comma-separted list
  [-log file] Outputs log messages to the specified file
  [-v level] Level of logging
  [-compress level] NetCDF compression level

At a minimum, the input ascii_file and the output netcdf_file must be passed on the command line. ASCII2NC interrogates the data to determine it's format, but the user may explicitly set it using the -format command line option. The -mask_grid, -mask_poly, and -mask_sid options can be used to filter observations spatially.

ASCII2NC Tool: Run

ASCII2NC Tool: Run cindyhg Tue, 06/25/2019 - 09:33

ASCII2NC Tool: Run

Start by making an output directory for ASCII2NC and changing directories:

mkdir -p ${METPLUS_TUTORIAL_DIR}/output/met_output/ascii2nc
cd ${METPLUS_TUTORIAL_DIR}/output/met_output/ascii2nc

Since ASCII2NC performs a simple reformatting step, typically no configuration file is needed. However, when processing high-frequency (1 or 3-minute) SURFRAD data, a configuration file may be used to define a time window and summary metric for each station. For example, you might compute the average observation value +/- 15 minutes at the top of each hour for each station. In this example, we will not use a configuration file.

The sample ASCII observations in the MET tarball are still identified by GRIB code rather than the newer variable name option. Dump that file and notice that the GRIB codes in the seventh column could be replaced by corresponding variable names. For example, 52 corresponds to RH:

cat ${METPLUS_DATA}/met_test/data/sample_obs/ascii/sample_ascii_obs.txt

Run ASCII2NC on the command line using the following command:

ascii2nc \
${METPLUS_DATA}/met_test/data/sample_obs/ascii/sample_ascii_obs.txt \
tutorial_ascii.nc \
-v 2

ASCII2NC should perform this reformatting step very quickly since the sample file only contains data for 5 stations.

ASCII2NC Tool: Output

ASCII2NC Tool: Output cindyhg Tue, 06/25/2019 - 09:37

ASCII2NC Tool: Output

When ASCII2NC is finished, you may view the output NetCDF file it wrote using the ncdump utility. Run the following command to view the header of the NetCDF output file:

ncdump -h tutorial_ascii.nc

The NetCDF header should look nearly identical to the output of the NetCDF output of PB2NC. You can see the list of stations for which we have data by inspecting the hdr_sid_table variable:

ncdump -v hdr_sid_table tutorial_ascii.nc

Feel free to inspect the contents of the other variables as well.

This ASCII data only contains observations at a few locations. Use the plot_point_obs to plot the locations, increasing the level of verbosity to 3 to see more detail:

plot_point_obs \
tutorial_ascii.nc \
tutorial_ascii.ps \
-data_file ${METPLUS_DATA}/met_test/data/sample_fcst/2007033000/nam.t00z.awip1236.tm00.20070330.grb \
-v 3
gv tutorial_ascii.ps &

Next, we'll use the NetCDF output of PB2NC and ASCII2NC to perform Grid-to-Point verification using the Point-Stat tool.