MET Tool: Gen-Vx-Mask

MET Tool: Gen-Vx-Mask
IMPORTANT NOTE: If you are returning to the tutorial, you must source the tutorial setup script before running the following instructions. If you are unsure if you have done this step, please navigate to the Verify Environment is Set Correctly page.

Gen-Vx-Mask Functionality

The Gen-Vx-Mask tool may be run to speed up the execution time of the other MET tools. Gen-Vx-Mask defines a bitmap masking region for your domain. It takes as input a gridded data file defining your domain and a second argument to define the area of interest (varies by masking type). It writes out a NetCDF file containing a bitmap for that masking region. You can run Gen-Vx-Mask iteratively, passing its output back in as input, to define more complex masking regions.

You can then use the output of Gen-Vx-Mask to define masking regions in the MET statistics tools. While those tools can read ASCII lat/lon polyline files directly, they are able to process the output of Gen-Vx-Mask much more quickly than the original polyline. The idea is to define your masking region once for your domain with Gen-Vx-Mask and apply the output many times in the MET statistics tools.

Gen-Vx-Mask Usage

View the usage statement for Gen-Vx-Mask by simply typing the following:

gen_vx_mask
Usage: gen_vx_mask  
  input_file Gridded data file defining the domain
  mask_file Defines the masking region and varies by -type
  out_file Output NetCDF mask file to be written
  -type string Masking type: poly, poly_xy, box, circle, track, grid, data, solar_alt, solar_azi, lat, lon, shape
  [-input_field string] Define field from input_file for grid point initialization values, rather than 0.
  [-mask_field string] Define field from mask_file for data masking.
  [-complement, -union, -intersection, -symdiff] Set logic for combining input_field initialization values with the current mask values.
  [-thresh string] Define threshold for circle, track, data, solar_alt, solar_azi, lat, and lon masking types.
  [-height n, -width n] Define dimensions for box masking.
  [-shapeno n] Define the index of the shape for shapefile masking.
  [-value n] Output mask value to be written, rather than 1.
  [-name str] Specifies the name to be used for the mask.
  [-log file] Outputs log messages to the specified file
  [-v level] Level of logging
  [-compress level] NetCDF compression level

At a minimum, the input data_file, the input mask_poly polyline file, the output netcdf_file, and the type must be passed on the command line.

admin Mon, 06/24/2019 - 16:06

Run Poly Type

Run Poly Type

Start by making an output directory for Gen-Vx-Mask and changing directories:

mkdir -p ${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask
cd ${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask

Since Gen-Vx-Mask performs a simple masking step, no configuration file is needed.

We'll run the Gen-Vx-Mask tool to apply a polyline for the CONUS (Contiguous United States) to our model domain. Run Gen-Vx-Mask on the command line using the following command:

gen_vx_mask \
${METPLUS_DATA}/met_test/data/sample_obs/ST2ml/ST2ml2005080712.Grb_G212 \
${MET_BUILD_BASE}/share/met/poly/CONUS.poly \
CONUS_mask.nc \
-type poly -v 2

Re-run using verbosity level 3 and look closely at the log messages. How many grid points were included in this mask?

Gen-Vx-Mask should run very quickly since the grid is coarse (185x129 points) and there are 243 lat/lon points in the CONUS polyline. The more you increase the grid resolution and number of polyline points, the longer it will take to run. View the NetCDF bitmap file generated by executing the following command:

ncview CONUS_mask.nc &

Notice that the bitmap has a value of 1 inside the CONUS polyline and 0 everywhere else. We'll use the CONUS mask we just defined in the next step.

You could try running plot_data_plane to create a PostScript image of this masking region. Can you remember how?

Notice that there are several ways that gen_vx_mask can be run to define regions of interest, some of which will be demonstrated over the next few pages.

admin Mon, 06/24/2019 - 16:16

Run Lat/Lon and Grid Types

Run Lat/Lon and Grid Types
After running each of the Gen-Vx-Mask commands on this, and following, pages, users are encouraged to inspect the header of the NetCDF output file by running ncdump -h and view the masking regions by running ncview or plot_data_plane.

Using a pre-defined NCEP grid from the NCEP ON388 Grid Identification Table, we'll create a latitude band, using the "lat" masking type, for the tropics region. Run Gen-Vx-Mask on the command line using the following command:

gen_vx_mask \
G004 \
G004 \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/G004_Tropics.nc \
-type lat -thresh 'ge-30 && le30'

Most of the grids defined in ON388 Table B can be referenced in MET as "GNNN" where NNN is the 3-digit grid number. Run "ncdump -h" on the output file and notice that the mask variable is named "lat_mask":

ncdump -h ${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/G004_Tropics.nc
float lat_mask(lat, lon) ;
        lat_mask:long_name = "lat_mask masking region" ;
        lat_mask:_FillValue = -9999.f ;
        lat_mask:mask_type = "lat>=-30&&<=30" ;

Use the "-name" command line option, as shown below, to override this default.

To compute the intersection or union of two masks, use the output from the first run as input to the second. Run Gen-Vx-Mask on the command line using the following command, which uses the "lon" masking type:

gen_vx_mask \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/G004_Tropics.nc \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/G004_Tropics.nc \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/G004_EastPac.nc \
-type lon -thresh 'le-70 && ge-130' -intersection -name EastPac

Compare this intersection output to the union of the two masks, computed below:

gen_vx_mask \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/G004_Tropics.nc \ ${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/G004_Tropics.nc \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/G004_EastPac.nc \
-type lon -thresh 'le-70 && ge-130' -union

Now we'll use the "grid" masking type to select a subgrid from a larger grid. Run Gen-Vx-Mask on the command line using the following command:

gen_vx_mask \
G004 \
${METPLUS_DATA}/met_test/data/sample_obs/ST2ml/ST2ml2005080712.Grb_G212 \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/G004_SUBGRID.nc \
-type grid

On the next page, we'll demonstrate using the "data" and "solar_alt" masking types.

johnhg Fri, 12/10/2021 - 15:04

Run Data and Solar Types

Run Data and Solar Types

On this page, we provide examples for land/sea mask and also a solar altitude to show where it is daytime on a global grid.

Run Gen-Vx-Mask on the command line using the following command:

gen_vx_mask \
${METPLUS_DATA}/model_applications/medium_range/grid_to_obs/gfs/pgbf00.gfs.2017060100 \
${METPLUS_DATA}/model_applications/medium_range/grid_to_obs/gfs/pgbf00.gfs.2017060100 \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/GFS_LAND.nc \
-type data -mask_field 'name="LAND"; level="L0";' -thresh eq1 -name LAND

corresponding water mask could be created using two different methods. One way is to simply rerun the land mask command above using the -complement option:

gen_vx_mask \
${METPLUS_DATA}/model_applications/medium_range/grid_to_obs/gfs/pgbf00.gfs.2017060100 \
${METPLUS_DATA}/model_applications/medium_range/grid_to_obs/gfs/pgbf00.gfs.2017060100 \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/GFS_LAND_COMP.nc \
-type data -mask_field 'name="LAND"; level="L0";' -thresh eq1 -name LAND_COMP -complement

Another way is to specify a different threshold (eq1 instead of eq0) rather than taking the complement:

gen_vx_mask \
${METPLUS_DATA}/model_applications/medium_range/grid_to_obs/gfs/pgbf00.gfs.2017060100 \
${METPLUS_DATA}/model_applications/medium_range/grid_to_obs/gfs/pgbf00.gfs.2017060100 \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/GFS_WATER.nc \
-type data -mask_field 'name="LAND"; level="L0";' -thresh eq0 -name WATER

Now we'll run Gen-Vx-Mask to show where it’s daytime on a global grid:

gen_vx_mask \
G004 \
20170601_183000 \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/SOLAR_DAY.nc \
-type solar_alt -thresh ge0 -name DAY

Next, combine the LAND output from the previous run with the solar altitude mask for daytime:

gen_vx_mask \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/GFS_LAND.nc \
20170601_183000 \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/DAYLIGHT_LAND.nc \
-type solar_alt -thresh ge0 -name DAYLIGHT_LAND -intersection

This creates a mask for grid points on land experiencing daylight at 18:30 UTC on 20170601.

On the next page, we'll demonstrate using the "track" and "circle" masking types.

 

jpresto Fri, 12/10/2021 - 15:13

Run Track and Circle Types

Run Track and Circle Types

On this page, we provide examples for using the "track" masking type using BEST track hurricane data and "circle" masking type.

Start by extracting the lat/lon locations for Hurricane Dorian:

echo "DORIAN" > ${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/dorian.poly
cat ${METPLUS_DATA}/model_applications/medium_range/dorian_data/track_data/bal052019.dat | awk '{printf "%d %d\n", $7, $8}' | awk '{printf "%.1f %.1f\n", $1/(10), $2/(-10)}' | uniq >> ${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/dorian.poly

Inspect the dorian.poly file that begins with "DORIAN" and has the lat/lon storm locations on subsequent lines.

cat ${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/dorian.poly

Compute a mask for all grid points within 200 km of the Hurricane Dorian track. Run Gen-Vx-Mask using the following command:

gen_vx_mask \
G003 \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/dorian.poly \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/Dorian_Track.nc \
-type track -name Dorain_Track -thresh le200

Extract the first track point for Dorian and use the circle masking option to select all grid points within 500 km:

head -2 ${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/dorian.poly > ${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/dorian_first.poly

Run Gen-Vx-Mask on the command line using the following command:

gen_vx_mask \
G003 \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/dorian_first.poly \
${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask/Dorian_Origin.nc \
-type circle -name Dorain_Track -thresh le500

The "circle" mask type draws a circle around each of the lat/lon points in the input poly file. The "circle" option may be useful when verifying within a certain radius of known radar locations. If you rerun without the "-thresh" command line option, Gen-Vx-Mask still runs but prints a warning message:

WARNING: apply_circle_mask() -> since "-thresh" was not used to specify a threshold in kilometers for circle masking, the minimum distance to the points will be written.

Can you figure out what this output file now contains? Run ncview or plot_data_plane to visualize it.

The "-thresh" option can also be omitted from the "track", "data", "solar_alt", and "solar_azi" masking types. If no threshold is specified, a warning message is printed, and the raw, un-thresholded values are written to the output NetCDF file.

Gen-Vx-Mask also supports the "box", "solar_azi", and "shape" masking types, not covered in these exercises. Interested users can download Natural Earth shapefiles and run Gen-Vx-Mask using the "-type shape" option.

Next, we'll take a look at using the "shape" masking type with Gen-Vx-Mask.

jpresto Fri, 12/10/2021 - 15:14

Run Shape Type

Run Shape Type

We will demonstrate the Gen-Vx-Mask "shape" masking type using freely available shapefiles from Natural Earth.  While multiple resolutions are provided, we'll use the coarsest version for this example since it's the smallest in size.

Download the Natural Earth administrative shapefiles for countries boundaries.

cd ${METPLUS_TUTORIAL_DIR}/output/met_output/gen_vx_mask
mkdir ne_shapefiles; cd ne_shapefiles
wget https://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/cultural/ne_110m_admin_0_countries.zip
unzip ne_110m_admin_0_countries.zip

Run gen_vx_mask to define the mask for the USA (index 4).

gen_vx_mask G004 ne_110m_admin_0_countries.shp G004_USA_mask.nc -name USA -type shape -shapeno 4 -v 3

Re-run to compute the union with Canada (index 3).

gen_vx_mask G004_USA_mask.nc ne_110m_admin_0_countries.shp G004_USA_Canda_mask.nc -name USA_Canada -type shape -shapeno 3 -union -v 3

Re-run to add Mexico (index 27).

gen_vx_mask G004_USA_Canda_mask.nc ne_110m_admin_0_countries.shp G004_North_America_mask.nc -name North_America -type shape -shapeno 27 -union -v 3

The result is good but not perfect. There are a few missing grid points along the boundary. But this demonstrates how the tool works. Consider re-running all three commands again, but this time use the "-value" command line option to define the mask value to be written. Just make "-value" match the "-shapeno" option (.e.g. -value 4 for USA, -value 3 for Canada, and -value 27, for Mexico). What impact does that have on the result?

Next, we'll take a look at the functionality that Grid-Stat offers.

jpresto Fri, 12/17/2021 - 12:59