Session 8: METplus Analysis Tools

Session 8: METplus Analysis Tools jopatz Wed, 02/23/2022 - 16:39

(content)

METviewer

METviewer cindyhg Mon, 06/24/2019 - 14:37

METviewer: General

METviewer Overview

METviewer is a database and display system for storing and plotting data from the MET .stat and MODE _obj.txt files. It is used heavily within the DTC and by NOAA-GSD and NOAA-EMC. While its distribution is limited, it is available to the community through a Docker container. For more information, please see:

Here, we will use the publicly available version of METviewer running at NCAR to demonstrate the METviewer webapp.

METviewer reads MET verification statistics from a database and creates plots using the R statistical package. The tool includes a web application that can be accessed from a web browser to create a single plot. The specification for each plot is built using a series of controls and then serialized into XML. For each plot, METviewer generates a SQL query, an R script to create the plot, a flat file containing the data that will be plotted and the plot itself.

METviewer Web Application

The following example can be run using the METviewer instance located at the link below. It is recommended to move the new tab into a new browser window, so that you can view both the example instructions and the METviewer web application side-by-side.

The following pages have instructions for generating two different types of plots:

  • A threshold series plot of Accuracy (ACC) and Critical Success Index (CSI)
  • A forecast lead time box plot series of MODE object area distributions

Please be sure to follow these instructions in order.

Time Series of Categorical Statistics Plot

Time Series of Categorical Statistics Plot cindyhg Mon, 06/24/2019 - 14:37

METviewer: Time Series of Categorical Statistics Plot

The following list shows the setting name and the setting value for each of the controls on the web application. Please go through them in order, starting at the top, setting and checking the controls as suggested. The settings that you need to change are marked in blue. The settings are grouped according to the areas on the web app. If you need more information about the controls, please click the little i in a circle to the right of METviewer X.Y at the very top of the page. This will take you to the METviewer documentation.

  • Database: Click on the box named Select databases and from the Verification group, select mv_met_tutorial to make a plot with sample tutorial data.
  • Tab: Series - Create a time-series (as we will do here) or threshold-series line plot. The different tabs correspond to the types of plots that METviewer can generate, such as box plots, bar plots, histograms, and ensemble plots.
  • Plot Data: Stat - Plot traditional statistics as opposed to MODE object attributes.
  • Y1 Axis variables - These controls specify the statistics and curves for the Y1 axis of the plot.
    • Y1 Dependent (Forecast) Variables - pairs of forecast variables and statistics to be plotted
      • Click the dropdown list and select: APCP_03 to select 3-hourly Accumulated Precipitation
      • Click Select attribute stat dropdown list and select: ACC and CSI to plot Accuracy and Critical Success Index
    • Y1 Series Variables - specifies the lines on the plot for each dependent variable
      • Click the dropdown list and select: MODEL
      • Click the Select value dropdown list and select: QPF to output the one model present
      • Next, click the + Series Variable button, click the dropdown list, and select: FCST_THRESH
      • Click the Select value dropdown list and select: >0.0, >2.54, and >6.35 to plot statistics for 3 categorical thresholds
  • Y2 Axis Tab - these controls specify the statistics and curves for the Y2 axis of the plot; it is not used in this demonstration
  • Fixed Values - field/value pairs that are held constant for all data on the plot
    • Click the + Fixed Value button - to add a fixed value selection option
    • Click the dropdown list and select VX_MASK
    • Click the Select value dropdown list and select: FULL to plot data for FULL model domain (the only one present in this dataset)
  • Independent Variable - the x-axis field and values over which the data are displayed on the plot
    • Click the dropdown list and select FCST_LEAD
    • Click the Select value dropdown list and click the Check all option to select all items
  • Statistics
    • The Summary radio button tells MET to plot the mean or median of the statistics pulled directly from the database.
    • The Aggregation statistics radio button tells METviewer to aggregate contingency table counts or partial sums and compute aggregate summary statistics and confidence intervals.
    • Use the default Summary radio button selection
  • Series Formatting - settings to control colors, line types, plotting symbols, and confidence intervals
    • There should be settings for 6 lines: 3 lines for ACC followed by 3 lines for CSI
    • In the Line Color column:
      • Set the 1st and 4th lines to the same RED color
      • Set the 2nd and 5th lines to the same GREEN color
      • Set the 3rd and 6th lines to the same BLUE color
    • In the Line Type column:
      • Set the 4th, 5th, and 6th lines to dashed
  • Plot Formatting - settings that control the appearance of the plot, along with several miscellaneous features
    • On the Titles & Labels tab
      • set Plot Title to ACC / CSI vs. Lead Time
      • set X label to Forecast Lead
      • set Y1 label to ACC / CSI
    • On the Common tab, select Display Number of Stats
  • Click the Generate Plot button at the top of the page to submit your plot request to the METviewer plotting engine.

Time Series of Categorical Statistics Plot Output

Time Series of Categorical Statistics Plot Output cindyhg Mon, 06/24/2019 - 14:39

METviewer: Time Series of Categorical Statistics Plot Output

If you successfully followed the instructions on the previous page, you should see a plot appear in the in the plot tab of the METviewer window. Your plot should look like this PNG IMAGE. And here is the corresponding plot XML File.

Studying the plot shows that the Critical Success Index (CSI) decreases for higher thresholds, as if often the case. This suggests that the forecast models have more skill at lower precipitation thresholds. This contrasts with the story told by the accuracy statistic, which increases as the threshold increases. This can be explained by the simplicity of the accuracy statistic, and the effect that always predicting no event can have for extreme events.

The following section discusses the output that METviewer generates for each plot. Click through the tabs above the METviewer plot to examine each:

  • XML Tab: When the Generate Plot button is clicked, the web application serializes the information in the web page controls into an XML document and sends it across the internet to METviewer. This XML document can be used to generate a plot directly from METviewer using the command line interface, also called the batch engine. The batch engine can create many plots from a single XML document, using only small changes. Thus, a good way to use the METviewer web application is to "design" plots and then take the XML for the plot and extend it to create many different plots.
  • Log Tab: The METviewer plot engine generates status output as it runs, and this information can be useful for finding problems.
  • R script Tab: METviewer uses the R statistics package to generate plots by constructing an R script and running it in R. The R script that is constructed can be saved and modified if the users would like to make changes to the plot that are not supported in METviewer.
  • R data Tab: Along with the R script generated by METviewer for the plot, the plot data is stored in a file that is read when the plot is generated in R. This file can be useful for problem solving, since it shows all of the data used in the plot and can be loaded directly into R for analysis.
  • SQL Tab: All of the MET verification statistics output is stored in a database, which METviewer searches when it generates a plot. The SQL generated by METviewer to gather the data for a plot can be viewed, in case the user wants to explore or debug using the SQL directly.

Object Based Attribute Area Box Plot

Object Based Attribute Area Box Plot cindyhg Mon, 06/24/2019 - 14:40

METviewer: Object Based Attribute Area Box Plot

For the next plot, you'll try out the XML upload feature. Each plot created by METviewer corresponds to an XML file that defines the plot. In this example, you'll load the XML from a previous plot and regenerate it.

  • Right click on this XML file, select the Save Link As option, and save the file to your machine.
     
  • In the METviewer window, click on the Load XML button in the top-right corner.
     
  • In the Upload plot XML file dialog box, click the Browse button, and navigate the location of the XML file.
     
  • Select the XML file, click the Open button, and then the OK button.
     
  • METviewer will now populate all of the selections with those specified in that XML file.
     
  • Lastly, click the Generate Plot button at the top of the page to create the image. Your plot should look like this PNG Image.
     

This plot shows the area of MODE objects defined for 1-hourly accumulated precipitation thresholded at 0.1 inch. The data has been filtered down to the 00Z initialization and the areas are plotted by lead time. Each lead time has 3 boxplots: core1 in red, core2 in blue, and the observations in gray. The numbers in black across the top show the number of objects plotted for each lead time.

Note that we have only plotted the observations once since they should be the same for both models. Look in the Series Formatting section at the bottom of the METviewer webapp in the Hide column to see that we're hiding the 4th series titled core2_hwt APCP_01 AREA_OSA. Also note that in the Legend Text column, we've specified our own strings rather than using the METviewer defaults.

The XML Upload feature of METviewer is very powerful and has saved users a lot of time by uploading previous work rather than starting from scratch each time.

Feel free to experiment with METviewer and make additional plots.

End of Practical Session 5

End of Practical Session 5 cindyhg Mon, 06/24/2019 - 14:41

End of Practical Session 5

Congratulations! You have completed Session 5!

METplotpy

METplotpy
jpresto Wed, 04/06/2022 - 11:53

Histogram

Histogram

Histogram Overview:

The histogram source code is located in the https://github.com/dtcenter/METplotpy repository, under the METplotpy/metplotpy/plots/histogram directory.  Custom configuration files and sample data are located under the METplotpy/test/histogram directory.  

There are three specialized histogram plots available:

  • Rank Histogram
  • Relative Frequency Histogram
  • Probability Histogram

Detailed information about these plots is available from Read the Docs:

https://metplotpy.readthedocs.io/en/latest/Users_Guide/histogram.html

These instructions are relevant for generating the rank histogram from the command line.  For METplotpy versions 1.0 and 1.1, development was performed with Python 3.6 and related third-party packages (i.e. Python packages outside the Python standard library).  Future releases of METplotpy will be based on Python version 3.8.6.  

Each histogram plot requires two configuration files, a default and custom configuration file in YAML (https://yaml.org/).  One default configuration file is used for all three histogram plots: histogram_defaults.yaml.  This default configuration file is automatically loaded by the source code.  The custom configuration file is useful for overriding the default settings in the histogram_defaults.yaml configuration file.  If the user chooses to use all the settings in the default configuration file, an empty custom configuration file can be provided (however, the user will have the sample data and output plot located in the specified directories set in the default configuration file- some users may not have permission to do this) .   The custom configuration files and sample data are located in the METplotpy/test/histogram directory of the downloaded/cloned source code.

Set up pre-requisites:

The Python requirements for METplotpy are found in the User's Guide, under the Installation section

METcalcpy is a requirement for METplotpy.  The following description is one of numerous methods to set up the working environment to utilize METcalcpy from the METplotpy source code.   These instructions are suitable for users that are not working within a conda environment:

Clone the METcalcpy repository to a directory where you have read/write permissions (replace /path/to/code with the full path to where you are saving the METcalcpy source code):
mkdir /path/to/code
cd /path/to/code
git clone https://github.com/dtcenter/METcalcpy

 

Set the PYTHONPATH to indicate where the METcalcpy modules can be found:

*Replace /path/to/code with the full path to where you saved the METcalcpy source code

csh:

setenv PYTHONPATH /path/to/code/METcalcpy:/path/to/code/METcalcpy/metcalcpy

bash:

export PYTHONPATH=/path/to/code/METcalcpy:/path/to/code/METcalcpy/metcalcpy

 

The following instructions are for users who have permission to create, modify, and use conda environments (skip if you already set the PYTHONPATH above).  Select either Method 1 or Method 2:
  • Method 1: Follow instructions in Section 1.1.2 of the Installation section of the METplotpy User's Guide
  • Method 2: Using using PyPI (Python Package Index) enter the following:
pip install metcalcpy==1.1

 

Create the Rank Histogram plot:

Overview of steps:

  • create a directory where you wish to save the METplotpy source code
  • create a WORKING_DIR where you have read/write permissions
  • set the WORKING_DIR, METPLOTPY_BASE, and PYTHONPATH environments
  • copy the histogram data to the WORKING_DIR
  • copy the custom configuration for the rank histogram to the WORKING_DIR
  • modify the location of the input data and output plot in the custom configuration file
  • run the Python script to generate the rank histogram plot

 

Replace /path/to with the full path to the METplotpy source code and working_dir 
mkdir /path/to
mkdir /path/to/working_dir

 

csh:

setenv PYTHONPATH /path/to/METcalcpy:/path/to/METcalcpy/metcalcpy:/path/to/METplotpy:/path/to/METplotpy/metplotpy:/path/to/METplotpy/metplotpy/plots
setenv METPLOTPY_BASE/path/to/METplotpy
setenv WORKING_DIR /path/to/working_dir/histogram

bash:

export PYTHONPATH=/path/to/METcalcpy:/path/to/METcalcpy/metcalcpy:/path/to/METplotpy:/path/to/METplotpy/metplotpy:/path/to/METplotpy/metplotpy/plots

 

export METPLOTPY_BASE=/path/to/METplotpy
export WORKING_DIR=/path/to/working_dir

 

Copy data and custom config file to WORKING_DIR:
cp $METPLOTPY_BASE/test/histogram/rank_hist.data $WORKING_DIR
cp $METPLOTPY_BASE/test/histogram/rank_hist.yaml $WORKING_DIR
cd $WORKING_DIR

Modify custom configuration file:

Open the rank_hist.yaml file in a text editor of your choice.  Modify the stat_input and plot_filename to point explicitly to the location of the input data you copied and the name and location of the output plot:
stat_input: /path/to/working_dir/histogram/rank_hist.data
plot_filename: /path/to/working_dir/histogram/rank_hist.png

Save and close the rank_hist.yaml file. 

 

Generate the rank histogram plot:
python $METPLOTPY_BASE/metplotpy/plots/histogram/rank_hist.py $WORKING_DIR/histograms/rank_hist.yaml

 

In your WORKING_DIR, you will now see a rank_hist.png file.  You can view the png file using an appropriate viewer, such as 'display' if on a Linux/Unix host:

cd $WORKING_DIR
display rank_hist.png

 

Your histogram will look like the following:

 

 

 

 

 

Create the Relative Frequency Histogram and Probability Histogram

 

Repeat the steps for the rank histogram, except copy the following to the WORKING_DIR:

Relative frequency histogram:

cp $METPLOTPY_BASE/test/histogram/rel_hist.data $WORKING_DIR
cp $METPLOTPY_BASE/test/histogram/rel_hist.yaml $WORKING_DIR

 

Probability histogram:

cp $METPLOTPY_BASE/test/histogram/prob_hist.data $WORKING_DIR
cp $METPLOTPY_BASE/test/histogram/prob_hist.yaml $WORKING_DIR

 

Make the same modifications in the rel_hist.yaml and prob_hist.yaml files to the stat_input and plot_filename settings.

Relative frequency histogram:

stat_input: /path/to/working_dir/histogram/rel_hist.data
plot_filename: /path/to/working_dir/histogram/rel_hist.png

 

Probability histogram:

stat_input: /path/to/working_dir/histogram/prob_hist.data
plot_filename: /path/to/working_dir/histogram/prob_hist.png

 

Generate the Relative Frequency and Probability Histogram plots using these Python scripts:

Relative frequency histogram:

python $METPLOTPY_BASE/metplotpy/plots/histogram/rel_hist.py $WORKING_DIR/histogram/rel_hist.yaml

Probability histogram:

python $METPLOTPY_BASE/metplotpy/plots/histogram/prob_hist.py $WORKING_DIR/histograms/prob_hist.yaml

You will see the rel_hist.png and prob_hist.png files in your WORKING_DIR. Your plots will look like the following:

Relative Frequency Histogram:

 

Probability Histogram:

 

 

 

 

 

 

jpresto Wed, 04/06/2022 - 11:54

Wind Rose

Wind Rose

Wind Rose Overview

 

The wind rose diagram source code is located in the https://github.com/dtcenter/METplotpy repository, under the METplotpy/metplotpy/plots/wind_rose directory.  Custom configuration files and sample data are located under the METplotpy/test/wind_rose directory.  

Detailed information about the wind rose diagram is available from Read the Docs:

https://metplotpy.readthedocs.io/en/latest/Users_Guide/wind_rose.html

These instructions are relevant for generating the wind rose diagram from the command line.  For METplotpy versions 1.0 and 1.1, development was performed with Python 3.6 and related third-party packages (i.e. Python packages outside the Python standard library).  Future releases of METplotpy will be based on Python version 3.8.6.  

The wind rose diagram requires two configuration files, a default and custom configuration file in YAML (https://yaml.org/).   The default configuration file is automatically loaded by the source code.  The custom configuration file is useful for overriding the default settings in the wind_rose_defaults.yaml configuration file.  If the user chooses to use all the settings in the default configuration file, an empty custom configuration file can be provided (however, the user will have the sample data and output plot located in the directories specified in the default configuration file.  Some users may not have permission to do this) .  The custom configuration files and sample data are located in the METplotpy/test/wind_rose directory of the downloaded/cloned source code.  The default configuration file, wind_rose_defaults.yml is located in the METplotpy/metplotpy/plots/config directory. 

 

Set up pre-requisites:

The Python requirements for METplotpy are found in the User's Guide, under the Installation section

METcalcpy is a requirement for METplotpy.  The following description is one of numerous methods to set up the working environment to utilize METcalcpy from the METplotpy source code.   These instructions are suitable for users that are not working within a conda environment:

Clone the METcalcpy repository to a directory where you have read/write permissions (replace /path/to/code with the full path to where you are saving the METcalcpy source code):
mkdir /path/to/code
cd /path/to/code
git clone https://github.com/dtcenter/METcalcpy

 

Set the PYTHONPATH to indicate where the METcalcpy modules can be found:

*Replace /path/to/code with the full path to where you saved the METcalcpy source code

csh:

setenv PYTHONPATH /path/to/code/METcalcpy:/path/to/code/METcalcpy/metcalcpy

bash:

export PYTHONPATH=/path/to/code/METcalcpy:/path/to/code/METcalcpy/metcalcpy

 

The following instructions are for users who have permission to create, modify, and use conda environments (skip if you already set the PYTHONPATH above).  Select either Method 1 or Method 2:
  • Method 1: Follow instructions in Section 1.1.2 of the Installation section of the METplotpy User's Guide
  • Method 2: Using using PyPI (Python Package Index) enter the following:
pip install metcalcpy==1.1

 

Create the Wind Rose diagram:

Overview of steps:

  • create a directory where you wish to save the METplotpy source code
  • create a WORKING_DIR where you have read/write permissions
  • set the WORKING_DIR, METPLOTPY_BASE, and PYTHONPATH environments
  • copy the wind rose sample data to the WORKING_DIR
  • copy the wind rose custom configuration file to the WORKING_DIR
  • modify the location of the input data and output plot in the custom configuration file
  • run the Python script to generate the wind rose diagram 

 

Replace /path/to with the full path to the METplotpy source code and working_dir 

mkdir /path/to
mkdir /path/to/working_dir

 

csh:

setenv PYTHONPATH /path/to/METcalcpy:/path/to/METcalcpy/metcalcpy:/path/to/METplotpy:/path/to/METplotpy/metplotpy:/path/to/METplotpy/metplotpy/plots
setenv METPLOTPY_BASE/path/to/METplotpy
setenv WORKING_DIR /path/to/working_dir/wind_rose

bash:

export PYTHONPATH=/path/to/METcalcpy:/path/to/METcalcpy/metcalcpy:/path/to/METplotpy:/path/to/METplotpy/metplotpy:/path/to/METplotpy/metplotpy/plots
export METPLOTPY_BASE=/path/to/METplotpy
export WORKING_DIR=/path/to/working_dir

 

Copy data and custom config file to WORKING_DIR:
cp $METPLOTPY_BASE/test/wind_rose/point_stats_mpr.txt $WORKING_DIR
cp $METPLOTPY_BASE/test/wind_rose/wind_rose_custom.yaml $WORKING_DIR
cd $WORKING_DIR

 

Modify the custom configuration file:

Open the wind_rose_custom.yaml file in a text editor of your choice.  Modify the stat_input and plot_filename to point explicitly to the location of the input data you copied and the name and location of the output plot:
stat_input: /path/to/working_dir/wind_rose/point_stats_mpr.txt
plot_filename: /path/to/working_dir/wind_rose/wind_rose_custom.png

Save and close the wind_rose_custom.yaml file. 

Generate the wind rose diagram:
python $METPLOTPY_BASE/metplotpy/plots/wind_rose/wind_rose.py $WORKING_DIR/wind_rose/wind_rose_custom.yaml

 

In your WORKING_DIR, you will now see a wind_rose_custom.png file.  You can view the png file using an appropriate viewer, such as 'display' if on a Linux/Unix host:
cd $WORKING_DIR
display wind_rose_custom.png

 

Your wind rose diagram will look like the following:

jpresto Wed, 04/06/2022 - 11:55