Output

Output

Point-Stat Tool: Output

The output of Point-Stat is one or more ASCII files containing statistics summarizing the verification performed. Since we wrote output to the current directory, it should now contain 6 ASCII files that begin with the point_stat_ prefix, one each for the FHO, CTC, CNT, ECLV, and MPR types, and a sixth for the STAT file. The STAT file contains all of the output statistics while the other ASCII files contain the exact same data organized by line type.

Since the lines of data in these ASCII files are so long, we strongly recommend configuring your text editor to NOT use dynamic word wrapping. The files will be much easier to read that way:

  • In the kwrite editor, select Settings->Configure Editor, de-select Dynamic Word Wrap and click OK.
  • In the vi editor, type the command :set nowrap. To set this as the default behavior, run the following command:
    echo "set nowrap" >> ~/.exrc
Open up the point_stat_run1_360000L_20070331_120000V_ctc.txt CTC file using the text editor of your choice and note the following:
vi point_stat_run1_360000L_20070331_120000V_ctc.txt
  • This is a simple ASCII file consisting of several rows of data.
  • Each row contains data for a single verification task.
  • The FCST_LEAD, FCST_VALID_BEG, and FCST_VALID_END columns indicate the timing information of the forecast field.
  • The OBS_LEAD, OBS_VALID_BEG, and OBS_VALID_END columns indicate the timing information of the observation field.
  • The FCST_VAR, FCST_UNITS, FCST_LEV, OBS_VAR, OBS_UNITS, and OBS_LEV columns indicate the two parts of the forecast and observation fields set in the configure file.
  • The OBTYPE column indicates the PrepBufr message type used for this verification task.
  • The VX_MASK column indicates the masking region over which the statistics were accumulated.
  • The INTERP_MTHD and INTERP_PNTS columns indicate the method used to interpolate the forecast data to the observation location.
  • The FCST_THRESH and OBS_THRESH columns indicate the thresholds applied to FCST_VAR and OBS_VAR.
  • The COV_THRESH column is not applicable here and will always have NA when using point_stat.
  • The ALPHA column indicates the alpha used for confidence intervals.
  • The LINE_TYPE column indicates that these are CTC contingency table count lines.
  • The TOTAL column indicates the total number of matched pairs.
  • The remaining columns contain the counts for the contingency table computed by applying the threshold to the forecast/observation matched pairs. The FY_OY (forecast: yes, observation: yes), FY_ON (forecast: yes, observation: no), FN_OY (forecast: no, observation: yes), and FN_ON (forecast: no, observation: no) columns indicate those counts.
Next, answer the following questions about this contingency table output:
  1. What do you notice about the structure of the contingency table counts with respect to the two thresholds used? Does this make sense?
  2. Does the model appear to resolve relatively cold surface temperatures?
  3. Based on these observations, are temperatures >273 relatively rare or common in the P850-500 range? How can this affect the ability to get a good score using contingency table statistics? What about temperatures <=273 at the surface?
Close that file, open up the point_stat_run1_360000L_20070331_120000V_cnt.txt CNT file, and note the following:
vi point_stat_run1_360000L_20070331_120000V_cnt.txt
  • The columns prior to LINE_TYPE contain the same data as the previous file we viewed.
  • The LINE_TYPE column indicates that these are CNT continuous lines.
  • The remaining columns contain continuous statistics derived from the raw forecast/observation pairs. See the CNT OUTPUT FORMAT section in the Point-Stat section of the MET User's Guide for a thorough description of the output.
  • Again, confidence intervals are given for each of these statistics as described above.
Next, answer the following questions about these continuous statistics:
  1. What conclusions can you draw about the model's performance at each level using continuous statistics? Justify your answer. Did you use a single metric in your evaluation? Why or why not?
  2. Comparing the first line with an alpha value of 0.05 to the second line with an alpha value of 0.10, how does the level of confidence change the upper and lower bounds of the confidence intervals (CIs)?
  3. Similarly, comparing the first line with few numbers of matched pairs in the TOTAL column to the third line with more, how does the sample size affect how you interpret your results?
Close that file, open up the point_stat_run1_360000L_20070331_120000V_fho.txt FHO file, and note the following:
vi point_stat_run1_360000L_20070331_120000V_fho.txt
  • The columns prior to LINE_TYPE contain the same data as the previous file we viewed.
  • The LINE_TYPE column indicates that these are FHO forecast-hit-observation rate lines.
  • The remaining columns are similar to the contingency table output and contain the total number of matched pairs, the forecast rate, the hit rate, and observation rate.
  • The forecast, hit, and observation rates should back up your answer to the third question about the contingency table output.
Close that file, open up the point_stat_run1_360000L_20070331_120000V_mpr.txt MPR file, and note the following:
vi point_stat_run1_360000L_20070331_120000V_mpr.txt
  • The columns prior to LINE_TYPE contain the same data as the previous file we viewed.
  • The LINE_TYPE column indicates that these are MPR matched pair lines.
  • The remaining columns are similar to the contingency table output and contain the total number of matched pairs, the matched pair index, the latitude, longitude, and elevation of the observation, the forecasted value, the observed value, and the climatological value (if applicable).
  • There is a lot of data here and it is recommended that the MPR line_type is used only to verify the tool is working properly.
cindyhg Tue, 06/25/2019 - 09:42