TC-Pairs Tool: General

TC-Pairs Usage

View the usage statement for TC-Pairs by simply typing the following:

At a minimum, the input ATCF format -adeck (or -edeck) source, the input ATCF format -bdeck source, and the configuration -config file must be passed in on the command line.

The -adeck, -edeck, and -edeck options can be set to either a specific file name or a top-level directory which will be searched for files ending in ".dat".

TC-Pairs Tool: Configure

Start by making an output directory for TC-Pairs and changing directories:

Like the Point-Stat tool, the TC-Pairs tool is controlled by the contents of the configuration file passed to it on the command line. The default TC-Pairs configuration file may be found in the data/config/TCPairsConfig_default file.

The configurable items for TC-Pairs are used to specify which data are ingested and how the input data are filtered. The configurable items include specifications for the following:

• All model names to include in matched pairs
• Storms to include in verification: parsed by storm_id, basin, storm_name, cyclone
• Date/Time/Space restrictions: initialization time, valid time, lead time, masking
• User defined consensus forecasts
• Thresholds over watch/warning status of storm

The first step is to set-up the configuration file. Keep in mind, TC-Pairs should be run with the largest desired sample in mind to minimize re-running this tool! First, make a copy of the default:

Next, open up the TCPairsConfig_tutorial_run file for editing and modify it as follows:

Set:

To identify the models to we want to verify, as well as the basin. The three models identified are all U.S. operational models: HWRF, GFS (AVNO), and GFDL. Any field left blank will assume no restictions.

Set:

To compute a user defined consensus named "CONS". The membership of the consensus are the three listed models, none are required, but 2 must be present to calculate the consensus.

Set:

To only keep pairs that are the intersection of the model forecast and observation.

Next, save the TCPairsConfig_tutorial_run file and exit the text editor.

TC-Pairs Tool: Run

Next, run TC-Pairs to compare all three ATCF forecast models specified in configuration file to the ATCF format best track analysis. Run the following command line:

TC-Pairs is now grabbing the model data we requested in the configuration file, generating the consensus, and performing the additional filtering criteria. It should take only a few minutes to run. You should see several status messages printed to the screen to indicate progress.

If you are running many models over many storms/seasons, it is best to run TC-Pairs using a script to call TC-Pairs for each storm. This avoids potential memory issues when parsing very large datasets.

TC-Pairs Tool: Output

The output of TC-Pairs is an ASCII file containing matched pairs for each of the models requested. In this example, the output is written to the tc_pairs.tcst file as we requested on the command line. This output file is in TCST format, which is similar to the STAT output from the Point-Stat and Grid-Stat tools. For more header information on the TCST format, see section 19.2.3 of the MET User's Guide.

Remember to configure your text editor to NOT use dynamic word wrapping. The files will be much easier to read that way:

• In the kwrite editor, select Settings->Configure Editor, de-select Dynamic Word Wrap and click OK.
• In the vi editor, type the command :set nowrap. To set this as the default behavior, run the following command:
  

  echo "set nowrap" >> ~/.exrc

Open tc_pairs.tcst

• Notice this is a simple ASCII file with rows for each matched pair for a single valid time..
• Any field that has A in the column name (such as AMAX_WIND, ALAT) indicate the model forecast.
• Any field that has B in the column name (such as BMAX_WIND, BLAT) indicate the observation field (Best Track).
• The TK_ERR, X_ERR, Y_ERR, ALTK_ERR, CRTK_ERR columns are calculated track error, X component position error, Y component postion error, along track error, and cross track error, respectively.
• Columns 34-63 are the 34-, 50-, and 64-kt wind radii for each quadrant.

TC-Stat Tool: Configure

Start by making an output directory for TC-Stat and changing directories:

The behavior of TC-Stat is controlled by the contents of the configuration file or the job command passed to it on the command line. The default TC-Stat configuration may be found in the data/config/TCStatConfig_default file. Make a copy of the default configuration file and make following modifications:

Open up the TCStatConfig_tutorial file for editing with your preferred text editor.

Set:

To only parse over two of the three model names in the tc_pairs.tcst file. The event_equal=TRUE flag will keep pairs that are in both HWRF and GFDL pairs.

Many of the filter options are left blank, indicating TC-Stat should parse over all available fields. You will notice many more available filter options beyond what was available with the TCPairsConfig.

Now, scroll all the way to the bottom of the TCStatConfig, and you will find the jobs[] section. Edit this section as follows:

TC-Stat: Run on TC-Pairs output

Run the TC-Stat using the following command:

Open the output file tc_stat.tcst. We can see that this filter job simply event equalized the two models specified in amodel.

Let's try to filter further, this time using the command line rather than the configuration file:

Here, we ran a filter job at the command line: only keeping tracks over water (not encountering land) and with categories Hurricane (HU), Tropical Storm (TS), Tropical Depression (TD), Subtropical Storm (SS), and Subtropical Depression (SD).

Open the output file tc_stat2.tcst: notice fewer lines have been kept. Look at the "LEVEL" column ... note all the non-tropical and subtropical level classifications have been filtered out of the sample.

Also, find the columns ADLAND and BDLAND. All these values are now positive, meaning the tracks over land (negative values) have been filtered.

With the filtering jobs mastered, lets give the second type of job - summary jobs - a try!

TC-Stat: Run on TC-Pairs output

Now, we will run a summary job using TC-Stat on the command line using the following command:

Open up the file tc_stat_summary.tcst. Notice this output is much different from the filter jobs.

The track data is event equalized for the HWRF and GFDL models, and summary statistics are produced for the TK_ERR column for each model by lead time.

TC-Stat: Plotting with R

In this section, you will use the R statistics software package to produce a plot of a few results. R was introduced in practical session 1.

The TC-Stat tool can be called from the Rscript to do additional filter jobs on the TCST output from TC-Pairs. This can be done on the command line by calling a filter job (following tc-stat), or a configuraton file can be used to select filtering criteria. A default configuration file can be found in Rscripts/include/plot_tcmpr_config_default.R.

The configuration file also includes various plot types to generate: MEAN, MEDIAN, SCATTER, REFPERF, BOXPLOT, and RANK. All of the plot commands can be called on the command line as well as with the configuration file. Plots are configurable (title, colors, axis labels, etc) either by modifying the configuration file or setting options on the command line. To run from the command line:

This plots the track error for two models: HWRF and CONS. CONS is the user defined consensus you generated in TC-Pairs.

Next, open up the output *png files:

The script produces the three plot types called in the configuration file:

1. Boxplot showing distribution of errors for a homogeneous sample of the two models (TK_ERR_boxplot.png).
2. Mean Errors with 95% CI for the same sample (TK_ERR_mean.png).
3. Rank plot indicating performance of HWRF model relative to CONS (TK_ERR_rank.png).

Series-Analysis Tool: Configure

Start by making an output directory for Series-Analysis and changing directories:

The behavior of Series-Analysis is controlled by the contents of the configuration file passed to it on the command line. The default Series-Analysis configuration file may be found in the data/config/SeriesAnalysisConfig_default file. Prior to modifying the configuration file, users are advised to make a copy of the default:

The configurable items for Series-Analysis are used to specify how the verification is to be performed. The configurable items include specifications for the following:

• The forecast fields to be verified at the specified vertical level or accumulation interval.
• The threshold values to be applied.
• The area over which to limit the computation of statistics - as predefined grids or configurable lat/lon polylines.
• The confidence interval methods to be used.
• The smoothing methods to be applied.
• The types of statistics to be computed.

You may find a complete description of the configurable items in section 13.2.3 of the MET User's Guide. Please take some time to review them.

For this tutorial, we'll run Series-Analysis to verify a time series of 3-hour accumulated precipitation. We'll use GRIB1 for the forecast files and NetCDF for the observation files. Since the forecast and observations are different file formats, we'll specify the name and level information for them slightly differently.

Open up the SeriesAnalysisConfig_tutorial file for editing with your preferred text editor and edit it as follows:

• Set the fcst dictionary to
  
  fcst = {
     field = [
        {
          name  = "APCP";
          level = [ "A3" ];
        }
     ];
  }

  To request the GRIB abbreviation for precipitation (APCP) accumualted over 3 hours (A3).

• Delete obs = fcst; and insert
  
  obs = {
     field = [
        {
          name  = "APCP_03";
          level = [ "(*,*)" ];
        }
     ];
  }

  To request the NetCDF variable named APCP_03 where it's two dimensions are the gridded dimesnions (*,*).

• Set cat_thresh = [ >0.0, >=5.0 ];
  To define the categorical thresholds of interest. By defining this at the top level of config file context, these thresholds will be applied to both the fcst and obs settings.
• In the mask dictionary, set grid = "G212";
  To limit the computation of statistics to only those grid points falling inside the NCEP Grid 212 domain.
• Set block_size = 10000;
  To process 10,000 grid points in each pass through the data. Setting block_size larger should make the tool run faster but use more memory.
• In the output_stats dictionary, set
  
     fho    = [ "F_RATE", "O_RATE" ];
     ctc    = [ "FY_OY", "FN_ON" ];
     cts    = [ "CSI", "GSS" ];
     mctc   = [];
     mcts   = [];
     cnt    = [ "TOTAL", "RMSE" ];
     sl1l2  = [];
     pct    = [];
     pstd   = [];
     pjc    = [];
     prc    = [];

  For each line type, you can select statistics to be computed at each grid point over the series. These are the column names from those line types. Here, we select the forecast rate (FHO: F_RATE), observation rate (FHO: O_RATE), number of forecast yes and observation yes (CTC: FY_OY), number of forecast no and observation no (CTC: FN_ON), critical success index (CTS: CSI), and the Gilbert Skill Score (CTS: GSS) for each threshold, along with the root mean squared error (CNT: RMSE).

Save and close this file.

Series-Analysis Tool: Run

First, we need to prepare our observations by putting 1-hourly StageII precipitation forecasts into 3-hourly buckets. Run the following PCP-Combine commands to prepare the observations:

Note that the previous set of PCP-Combine commands could easily be run by looping through times in a shell script or through a METplus use case! The MET tools are often run using a scripting language rather than typing individual commands by hand. You'll learn more about automation using the METplus python scripts.

Next, we'll run Series-Analysis using the following command:

The statistics we requested in the configuration file will be computed separately for each grid location and accumulated over a time series of eight three-hour accumulations over a 24-hour period. Each grid point will have up to 8 matched pair values.

Note how long this command line is. Imagine how long it would be for a series of 100 files! Instead of listing all of the input files on the command line, you can list them in an ASCII file and pass that to Series-Analysis using the -fcst and -obs options.

Series-Analysis Tool: Output

The output of Series-Analysis is one NetCDF file containing the requested output statistics for each grid location on the same grid as the input files.

You may view the output NetCDF file that Series-Analysis wrote using the ncdump utility. Run the following command to view the header of the NetCDF output file:

In the NetCDF header, we see that the file contains many arrays of data. For each threshold (>0.0 and >=5.0), there are values for the requested statistics: F_RATE, O_RATE, FY_OY, FN_ON, CSI, and GSS. The file also contains the requested RMSE and TOTAL number of matched pairs for each grid location over the 24-hour period.

Next, run the ncview utility to display the contents of the NetCDF output file:

Click through the different variables to see how the performance varies over the domain. Looking at the series_cnt_RMSEvariable, are the errors larger in the south eastern or north western regions of the United States?

Why does the extent of missing data increase for CSI for the higher threshold? Compare series_cts_CSI_gt0.0 to series_cts_CSI_ge5.0. (Hint: Find the definition of CSI in Appendix C of the MET User's Guide and look closely at the denominator.)

Try running Plot-Data-Plane to visualize the observation rate variable for non-zero precipitation (i.e. series_fho_O_RATE_gt0.0). Since the valid range of values for this data is 0 to 1, use that to set the -plot_range option.

Setting block_size to 10000 still required 3 passes through our 185x129 grid (= 23865 grid points). What happens when you increase block_size to 24000 and re-run? Does it run slower or faster?

METplus Use Case: Feature Relative (Series analysis by lead, by forecast hour grouping)

Setup

In this exercise, you will perform a series analysis based on the lead time (forecast hour) of your sample data and organize your results by forecast hour groupings. This use case utilizes the MET tc_pairs, tc_stat, and series analysis tools, and the METplus wrappers: TcPairs, ExtractTiles, TcStat, and SeriesByLead. Please refer to the MET User's Guide for a description of the MET tools and section 3.5, 3.13, and 3.16 of the METplus 2.0.4 Users Guide for more details about the wrappers. Please note that the METplus User's Guide is a work-in-progress and may have missing content.

Open the mycustom.conf file in an editor of your choice:

Replace the following:

with this:

to keep this feature relative output separate from the other examples.

Using your custom configuration file that you created in the METplus: CUSTOM Configuration File Settings section and the Feature Relative use case configuration files that are distributed with METplus, you should be able to run the use case using the sample input data set without any other changes.

Review Use Case Configuration File: series_by_lead_by_fhr_grouping.conf

View the file and study the configuration variables that are defined.

Note that variables in series_by_lead_fhr_grouping.conf reference other variable you defined in your mycustom.conf configuration file.
For example: 
SERIES_LEAD_OUT_DIR = {OUTPUT_BASE}/series_analysis_lead

This references OUTPUT_BASE which you set in your custom config file. METplus config variables can reference other config variables, even if they are defined in a config file that is read afterwards.

Run METplus

Run the following command:

You will see output streaming to your screen. This may take up to 3 minutes to complete. When it is complete, your prompt returns.

Review the Output Files

You should have the following output directories that result from running the wrappers TcPairs, ExtractTiles, and TcStat, respectively:

and the output directory of interest, from the SeriesByLead wrapper:

which has a directory corresponding to the date(s) of data in the time window:

and under that directory are subdirectories named by the storm:

and under each of those directories lies the files of interest:

Day1
Day2
Day3
Day4
series_animate

The directories Dayn contain the following files: 

The ANLY_FILES_F000_to_F018 is a text file that contains a list of the data that is included in the series analysis.

The .nc files are the series analysis output generated from the MET series_analysis tool.

The .png and .ps files are graphics files that can be viewed using okular:

Review the Log File

A log file is generated in your logging directory: ${HOME}/metplus_tutorial/output/series_by_lead_fhr_grouping. The filename contains the timestamp corresponding to the current day. To view the log file:

Review the Final Configuration File

The final configuration file is $HOME/metplus_tutorial/output/grid_to_grid/metplus_final.conf. This contains all of the configuration variables used in the run.

METviewer: Time Series of Categorical Statistics Plot

The following list shows the setting name and the setting value for each of the controls on the web application. Please go through them in order, starting at the top, setting and checking the controls as suggested. The settings that you need to change are marked in blue. The settings are grouped according to the areas on the web app. If you need more information about the controls, please click the little i in a circle to the right of METviewer 2.9 at the very top of the page. This will take you to the METviewer documentation.

• Database: Click on the box named Select databases and from the Verification group, select mv_met_tutorial
  To make a plot with sample tutorial data.
• Tab: Series - Create a time-series (as we will do here) or threshold-series line plot. The different tabs correspond to the types of plots that METviewer can generate, such as box plots, bar plots, histograms, and ensemble plots.
• Plot Data: Stat - Plot traditional statistics as opposed to MODE object attributes.
• Y1 Axis variables - These controls specify the statistics and curves for the Y1 axis of the plot.

  Y1 Dependent (Forecast) Variables - pairs of forecast variables and statistics to be plotted

  • Click dropdown list and select: APCP_03 - to select 3-hourly Accumulated Precipitation
  • Click Select attribute stat dropdown list and select: ACC and CSI - to plot Accuracy and Critical Success Index

  Y1 Series Variables - specifies the lines on the plot for each dependent variable

  • Click the dropdown list and select: MODEL
  • Click the Select value dropdown list and select: QPF - output for only one model is present
  • Next, click the + Series Variable button, click the dropdown list, and select: FCST_THRESH
  • Click the Select value dropdown list and select: >0.0, >2.54, and >6.35 - to plot statistics for 3 categorical thresholds
• Y2 Axis Tab - these controls specify the statistics and curves for the Y2 axis of the plot; it is not used in this demonstration
   
• Fixed Values - field/value pairs that are held constant for all data on the plot
  • Click the +Fixed Value button - to add a fixed value selection option
  • Click the dropdown list and select VX_MASK
  • Click the Select value dropdown list and select: FULL - to plot data for FULL model domain (the only one present in this dataset)
• Independent Variable - the x-axis field and values over which the data are displayed on the plot
  • Click the dropdown list and select FCST_LEAD
  • Click the Select value dropdown list and click the Check all option - to select all items
• Statistics - 
  The Summary radio button tells MET to plot the mean or median of the statistics pulled directly from the database.
  The Aggregation statistics radio button tells METviewer to aggregate contingency table counts or partial sums and compute aggregate summary statistics and confidence intervals.
  Use the default Summary radio button selection.
   
• Series Formatting - settings to control colors, line types, plotting symbols, and confidence intervals
   
  • There should be settings for 6 lines: 3 lines for ACC followed by 3 lines for CSI. In the Line Color column:
  • Set the 1st and 4th lines to the same RED color.
  • Set the 2nd and 5th lines to the same GREEN color.
  • Set the 3rd and 6th lines to the same BLUE color.
  • In the Series Line Type column:
  • Set the 4th, 5th, and 6th to Dashed.
• Plot Formatting - settings that control the appearance of the plot, along with several miscellaneous features
   
  • On the Titles & Labels tab, set Plot Title to ACC / CSI vs. Lead Time
  • On the Titles & Labels tab, set X label to Forecast Lead
  • On the Titles & Labels tab, set Y1 label to ACC / CSI
  • On the Common tab, select Display Number of Stats
• Click the Generate Plot button at the top of the page - to submit your plot request to the METviewer plotting engine.

METviewer: Time Series of Categorical Statistics Plot Output

If you successfully followed the instructions on the previous page, you should see a plot appear in the in the plot tab of the METviewer window. Your plot should look like this. And here is the corresponding plot xml.

Studying the plot shows that the Critical Success Index (CSI) decreases for higher thresholds, as if often the case. This suggests that the forecast models have more skill at lower precipitation thresholds. This contrasts with the story told by the accuracy statistic, which increases as the threshold increases. This can be explained by the simplicity of the accuracy statistic, and the effect that always predicting no event can have for extreme events.

The following section discusses the output that METviewer generates for each plot. Click through the tabs above the METviewer plot to examine each:

• XML Tab: When the Generate Plot button is clicked, the web application serializes the information in the web page controls into an XML document and sends it across the internet to METviewer. This XML document can be used to generate a plot directly from METviewer using the command line interface, also called the batch engine. The batch engine can create many plots from a single XML document, using only small changes. Thus, a good way to use the METviewer web application is to "design" plots and then take the XML for the plot and extend it to create many different plots.
• Log Tab: The METviewer plot engine generates status output as it runs, and this information can be useful for finding problems.
• R script Tab: METviewer uses the R statistics package to generate plots by constructing an R script and running it in R. The R script that is constructed can be saved and modified if the users would like to make changes to the plot that are not supported in METviewer.
• R data Tab: Along with the R script generated by METviewer for the plot, the plot data is stored in a file that is read when the plot is generated in R. This file can be useful for problem solving, since it shows all of the data used in the plot and can be loaded directly into R for analysis.
• SQL Tab: All of the MET verification statistics output is stored in a database, which METviewer searches when it generates a plot. The SQL generated by METviewer to gather the data for a plot can be viewed, in case the user wants to explore or debug using the SQL directly.

METviewer: Object Based Attribute Area Box Plot

For the next plot, you'll try out the XML upload feature. Each plot created by METviewer corresponds to an XML file that defines the plot. In this example, you'll load the XML from a previous plot and regenerate it.

• Right click on this XML file, select the Save Link As option, and save the file to your machine.
   
• In the METviewer window, click on the Load XML button in the top-right corner.
   
• In the Upload plot XML file dialog box, click the Browse button, and navigate the location of the XML file.
   
• Select the XML file, click the Open button, and then the OK button.
   
• METviewer will now populate all of the selections with those specified in that XML file.
   
• Lastly, click the Generate Plot button at the top of the page to create the image. Your plot should look like this.
   

This plot shows the area of MODE objects defined for 1-hourly accumulated precipitation thresholded at 0.1 inch. The data has been filtered down to the 00Z initialization and the areas are plotted by lead time. Each lead time has 3 boxplots: core1 in red, core2 in blue, and the observations in gray. The numbers in black across the top show the number of objects plotted for each lead time.

Note that we have only plotted the observations once since they should be the same for both models. Look in the Series Formatting section at the bottom of the METviewer webapp in the Hide column to see that we're hiding the 4th series titled core2_hwt APCP_01 AREA_OSA. Also note that in the Legend Text column, we've specified our own strings rather than using the METviewer defaults.

The XML Upload feature of METviewer is very powerful and has saved users a lot of time by uploading previous work rather than starting from scratch each time.

Feel free to experiment with METviewer and make additional plots.

End of Practical Session 5

Congratulations! You have completed Session 5!