METplus Setup

METplus Setup

METplus Overview

METplus is a set of Python modules that have been developed with the flexibility to run the MET applications for various use cases or scenarios. The goal is to simplify the running of MET for scientists. Currently, the primary means of achieving this is through the use of METplus configuration files, aka "conf files." It is designed to provide a framework in which additional use cases can be added. The conf file implementation utilizes a Python package called produtil that was developed by NOAA/NCEP/EMC for the HWRF system.

Please be sure to follow the instructions in order.

METplus Useful Links

The following links are just for reference, and not required for this practical session. METplus releases are available on GitHub along with sample data and instructions.

METplus User's Guide
METplus Releases on GitHub

The source code for the METplus components are publicly available in the following GitHub repositories:

New features are developed and bugs are tracked using GitHub issues in each repository.

admin Mon, 06/24/2019 - 15:58

METplus: Initial setup

METplus: Initial setup

Prerequisites: Software

The Requirements section in the Software Installation chapter of the METplus User's Guide lists the software and Python packages that are required to run the METplus wrappers. Note that there is a core set of requirements needed to run the METplus wrappers and additional requirements needed to utilize some of the more advanced features.

 

Prerequisites: Environment

This tutorial assumes MET 11.0.1 and METplus 5.0.1 have been installed on the machine used to run the exercises and that the sample input data is also available. If this is not the case, please navigate to the Software Installation chapter of the METplus User's Guide.

The following instructions are required so the commands in this tutorial can be copied and run without modification. The steps involve creating a working directory to store files used/generated by the tutorial and configuring a simple script that can be run to set up the shell environment. Once configured correctly, the script can be run upon returning to the tutorial content to easily resume progress.

Click on the link that corresponds to the environment you are setting up.
If you are running the tutorial instructions on your own computer, select the bash or csh instructions depending on which shell you prefer. We recommend using bash if you do not have a preference.

Pre-Configured Environments

Setting up the Tutorial Environment on Hera (NOAA)

Setting up the Tutorial Environment on Jet (NOAA)

Setting up the Tutorial Environment on Cheyenne (NCAR)

Setting up the Tutorial Environment on Seneca (NCAR)

User Configured Environments

Setting up the Tutorial Environment (bash)

Setting up the Tutorial Environment (csh)

 

admin Mon, 06/24/2019 - 15:59

Setting up the Tutorial Environment on Cheyenne (NCAR)

Setting up the Tutorial Environment on Cheyenne (NCAR)

Setting up the Tutorial Environment on Cheyenne (NCAR)

The following instructions should be run if configuring the shell environment to run the METplus Tutorial on Cheyenne (NCAR). If you are running on your own computer or a NOAA machine that has been set up to run the tutorial, please go back and click the appropriate link for those instructions.

Create a Working Directory

Create a directory called 'METplus-5.0.0_Tutorial' to hold all of the files you will create during the tutorial. This can be any directory that you have write permission.

In the following instructions, change "/path/to" to the directory you chose: EDIT AFTER COPYING and BEFORE HITTING RETURN!
Change directory to the location where you want to put your tutorial files.
cd /path/to
Create a directory called METplus-5.0.0_Tutorial.
This directory will contain all of your tutorial work, including configuration files, output data, and any other notes you'd like to keep.
mkdir METplus-5.0.0_Tutorial
Change directory into the tutorial directory.
cd METplus-5.0.0_Tutorial

Create Directories for Configuration Files and Output Data

mkdir user_config
mkdir output

Obtain the Tutorial Setup Script

Copy the tutorial setup shell script to configure your runtime environment for use with METplus. You will also copy over a METplus .conf file that you will use to run:
cp /glade/p/ral/jntp/MET/METplus/Tutorial_Files/METplus-5.0.0_Tutorial_Files/tutorial-5.0.0.conf ./tutorial.conf
Run the command that corresponds your default shell.
DO NOT RUN BOTH SETS OF COMMANDS.
If you are unsure which shell you are using, run the command "echo $SHELL" to check.
If you are using bash:
cp /glade/p/ral/jntp/MET/METplus/Tutorial_Files/METplus-5.0.0_Tutorial_Files/METplus-5.0.0_TutorialSetup.cheyenne.bash ./METplus-5.0.0_TutorialSetup.sh
If you are using csh:
cp /glade/p/ral/jntp/MET/METplus/Tutorial_Files/METplus-5.0.0_Tutorial_Files/METplus-5.0.0_TutorialSetup.cheyenne.csh ./METplus-5.0.0_TutorialSetup.sh
Next navigate to the Verify Environment is Set Correctly page.
mccabe Thu, 07/29/2021 - 09:50

Setting up the Tutorial Environment on Jet (NOAA)

Setting up the Tutorial Environment on Jet (NOAA)

Setting up the Tutorial Environment on Jet (NOAA)

The following instructions should be run if configuring the shell environment to run the METplus Tutorial on Jet (NOAA). If you are running on your own computer or an NCAR machine that has been set up to run the tutorial, please go back and click the appropriate link for those instructions.

Create a Working Directory

Create a directory called 'METplus-5.0.0_Tutorial' to hold all of the files you will create during the tutorial. This can be any directory that you have write permission.

In the following instructions, change "/path/to" to the directory you chose: EDIT AFTER COPYING and BEFORE HITTING RETURN!
Change directory to the location where you want to put your tutorial files.
cd /path/to
Create a directory called METplus-5.0.0_Tutorial. This directory will contain all of your tutorial work, including configuration files, output data, and any other notes you'd like to keep.
mkdir METplus-5.0.0_Tutorial
Change directory into the tutorial directory.
cd METplus-5.0.0_Tutorial

Create Directories for Configuration Files and Output Data

mkdir user_config
mkdir output

Obtain the Tutorial Setup Script

Copy the tutorial setup shell script to configure your runtime environment for use with METplus. You will also copy over a METplus .conf file that you will use to run:
cp /lfs1/HFIP/dtc-hurr/METplus/METplus-5.0.0_Tutorial_Files/METplus-5.0.0_TutorialSetup.jet.`basename $SHELL` ./METplus-5.0.0_TutorialSetup.sh
cp /lfs1/HFIP/dtc-hurr/METplus/METplus-5.0.0_Tutorial_Files/tutorial-5.0.0.conf ./tutorial.conf

 

Next navigate to the Verify Environment is Set Correctly page.
mccabe Wed, 11/17/2021 - 13:07

Setting up the Tutorial Environment on Seneca (NCAR)

Setting up the Tutorial Environment on Seneca (NCAR)

Setting up the Tutorial Environment on Seneca (NCAR)

The following instructions should be run if configuring the shell environment to run the METplus Tutorial on Seneca (NCAR). If you are running on your own computer or a NOAA machine that has been set up to run the tutorial, please go back and click the appropriate link for those instructions.

Create a Working Directory

Create a directory called 'METplus-5.0.0_Tutorial' to hold all of the files you will create during the tutorial. This can be any directory that you have write permission.

In the following instructions, change "/path/to" to the directory you chose: EDIT AFTER COPYING and BEFORE HITTING RETURN!
Change directory to the location where you want to put your tutorial files.
cd /path/to
Create a directory called METplus-5.0.0_Tutorial.
This directory will contain all of your tutorial work, including configuration files, output data, and any other notes you'd like to keep.
mkdir METplus-5.0.0_Tutorial
Change directory into the tutorial directory.
cd METplus-5.0.0_Tutorial

Create Directories for Configuration Files and Output Data

mkdir user_config
mkdir output

Obtain the Tutorial Setup Script

Copy the tutorial setup shell script to configure your runtime environment for use with METplus. You will also copy over a METplus .conf file that you will use to run:
cp /d1/projects/METplus/METplus-5.0.0_Tutorial_Files/tutorial-5.0.0.conf ./tutorial.conf
cp /d1/projects/METplus/METplus-5.0.0_Tutorial_Files/METplus-5.0.0_TutorialSetup.seneca.`basename $SHELL` ./METplus-5.0.0_TutorialSetup.sh
 

(OPTIONAL) Set up conda Environment for METplus Analysis and s2s Tutorial Sections

Due to the need for external package dependencies for the METplus Analysis and s2s sections, the following instructions are also necessary. If you do not plan on executing the commands of those sections, you can skip these steps.

First, ensure that you are running the bash environment:

bash

Now, verify that you have a .condarc file in your /home/user directory. If the file does not exist, create one. In that file, you will need the following entries, in the exact same format (dashes and colons):

channels:
- defaults
- conda-forge
- anaconda

Verify that you are using the most recent version of anaconda3 in your .bashrc file. Verify that your conda_setup in your .bashrc file is using /usr/local/anaconda3/bin/conda. You should see a code block similar to following:

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/usr/local/anaconda3/bin/conda' 'shell.bash' 'hook'2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/usr/local/anaconda3/etc/profile.d/conda.sh" ]; then
. "/usr/local/anaconda3/etc/profile.d/conda.sh"
else
export PATH="/usr/local/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<

If you do have to make any changes to your .bashrc file, it would be best at this point to sign out, then sign back into your Seneca connection from the terminal. Errors have been found to happen even after sourcing your .bashrc file for the updated content.

From the command line, enter the following:

/usr/local/anaconda3/bin/conda create -n metplus_analysis_tutorial_env python=3.8
mamba -y

If this was successful, you will see the following at the end of the output streamed to the screen:

Downloading and Extracting Packages
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate metplus_analysis_tutorial_env
#
# To deactivate an active environment, use
#
# $ conda deactivate

This creates a basic Python 3.8 conda environment named ‘metplus_analysis_tutorial_env’ that contains mamba (to accelerate installing the remaining Python packages).

Activate this environment by running the following from the command line:

conda activate metplus_analysis_tutorial_env

Install the following packages to the conda environment by running the following from the command line (using mamba to speed up the installation):

mamba install imageio=2.19.3 imutils=0.5.4 matplotlib=3.5.2 metpy=1.3.1 netcdf4=1.6.2 numpy=1.24.2 pandas=1.5.1 pint=0.19.2 plotly=5.9.0 pytest=7.2.1 python-kaleido=0.2.1 pyyaml=6.0 scikit-image=0.19.3 scipy=1.8.1 lxml=4.9.1 eofs=1.4.0 scikit-learn=1.2.0 -y

You will see many lines of information streaming to your screen. You will see following output to your screen if everything was successful:

Preparing transaction: done
Verifying transaction: done
Executing transaction: /
Installed package of scikit-learn can be accelerated using scikit-learn-intelex.
More details are available here: https://intel.github.io/scikit-learn-intelex

For example:
$ conda install scikit-learn-intelex
$ python -m sklearnex my_application.py
done

Now you are ready to run the METplus Analysis tutorials.

Next navigate to the Verify Environment is Set Correctly page.
mccabe Wed, 06/30/2021 - 12:24

Setting up the Tutorial Environment on Hera (NOAA)

Setting up the Tutorial Environment on Hera (NOAA)

Setting up the Tutorial Environment on Hera (NOAA)

The following instructions should be run if configuring the shell environment to run the METplus Tutorial on Hera (NOAA). If you are running on your own computer or an NCAR machine that has been set up to run the tutorial, please go back and click the appropriate link for those instructions.

Create a Working Directory

Create a directory called 'METplus-5.0.0_Tutorial' to hold all of the files you will create during the tutorial. This can be any directory that you have write permission.

In the following instructions, change "/path/to" to the directory you chose: EDIT AFTER COPYING and BEFORE HITTING RETURN!
Change directory to the location where you want to put your tutorial files.
cd /path/to
Create a directory called METplus-5.0.0_Tutorial. This directory will contain all of your tutorial work, including configuration files, output data, and any other notes you'd like to keep.
mkdir METplus-5.0.0_Tutorial
Change directory into the tutorial directory.
cd METplus-5.0.0_Tutorial

Create Directories for Configuration Files and Output Data

mkdir user_config
mkdir output

Obtain the Tutorial Setup Script

Copy the tutorial setup shell script to configure your runtime environment for use with METplus. You will also copy over a METplus .conf file that you will use to run:
cp /scratch1/BMC/dtc/METplus/METplus-5.0.0_Tutorial_Files/METplus-5.0.0_TutorialSetup.hera.`basename $SHELL` ./METplus-5.0.0_TutorialSetup.sh
cp /scratch1/BMC/dtc/METplus/METplus-5.0.0_Tutorial_Files/tutorial-5.0.0.conf ./tutorial.conf

 

Next navigate to the Verify Environment is Set Correctly page.
mccabe Wed, 06/30/2021 - 12:22

Setting up the Tutorial Environment (bash)

Setting up the Tutorial Environment (bash)

Setting up the Tutorial Environment (bash)

The following instructions should be run if configuring the bash environment to run the METplus Tutorial on your own computer. If you are running on an NCAR or NOAA machine that has been set up to run the tutorial, please go back and click the appropriate link for those instructions.

Create a Working Directory

Create a directory called 'METplus-5.0.0_Tutorial' to hold all of the files you will create during the tutorial. This can be any directory that you have write permission.

In the following instructions, change "/path/to" to the directory you chose: EDIT AFTER COPYING and BEFORE HITTING RETURN!
Change directory to the location where you want to put your tutorial files.
cd /path/to
Create a directory called METplus-5.0.0_Tutorial.
This directory will contain all of your tutorial work, including configuration files, output data, and any other notes you'd like to keep.
mkdir METplus-5.0.0_Tutorial
Change directory into the tutorial directory.
cd METplus-5.0.0_Tutorial

Create Directories for Configuration Files and Output Data

mkdir user_config
mkdir output

Obtain the Tutorial Setup Script

Copy the tutorial setup shell script to configure your runtime environment for use with METplus. You will also copy over a METplus .conf file that you will use to run:
wget https://dtcenter.org/sites/default/files/community-code/metplus/tutorial-data/METplus-5.0.0_TutorialSetup.bash.sh.txt -O ./METplus-5.0.0_TutorialSetup.sh
wget https://dtcenter.org/sites/default/files/community-code/metplus/tutorial-data/tutorial-5.0.0.conf -O ./tutorial.conf

Configure the Tutorial Setup Script

Open the tutorial setup script with the editor of your choice and modify the values for METPLUS_BUILD_BASE, MET_BUILD_BASE, and METPLUS_DATA.
vi METplus-5.0.0_TutorialSetup.sh
METPLUS_BUILD_BASE is the full path to the METplus installation (/path/to/METplus-X.Y)
MET_BUILD_BASE is the full path to the MET installation (/path/to/met-X.Y)
METPLUS_DATA is the location of the sample test data directory

 

Next navigate to the Verify Environment is Set Correctly page.
mccabe Wed, 06/30/2021 - 12:06

Setting up the Tutorial Environment (csh)

Setting up the Tutorial Environment (csh)

Setting up the Tutorial Environment (csh)

The following instructions should be run if configuring a csh environment to run the METplus Tutorial on your own computer. If you are running on an NCAR or NOAA machine that has been set up to run the tutorial, please go back and click the appropriate link for those instructions.

Create a Working Directory

Create a directory called 'METplus-5.0.0_Tutorial' to hold all of the files you will create during the tutorial. This can be any directory that you have write permission.

In the following instructions, change "/path/to" to the directory you chose: EDIT AFTER COPYING and BEFORE HITTING RETURN!
Change directory to the location where you want to put your tutorial files.
cd /path/to
Create a directory called METplus-5.0.0_Tutorial.
This directory will contain all of your tutorial work, including configuration files, output data, and any other notes you'd like to keep.
mkdir METplus-5.0.0_Tutorial
Change directory into the tutorial directory.
cd METplus-5.0.0_Tutorial

Create Directories for Configuration Files and Output Data

mkdir user_config
mkdir output

Obtain the Tutorial Setup Script

Copy the tutorial setup shell script to configure your runtime environment for use with METplus. You will also copy over a METplus .conf file that you will use to run:
wget https://dtcenter.org/sites/default/files/community-code/metplus/tutorial-data/METplus-5.0.0_TutorialSetup.csh.sh.txt -O ./METplus-5.0.0_TutorialSetup.sh
wget https://dtcenter.org/sites/default/files/community-code/metplus/tutorial-data/tutorial-5.0.0.conf -O ./tutorial.conf

Configure the Tutorial Setup Script

Open the tutorial setup script with the editor of your choice and modify the values for METPLUS_BUILD_BASE, MET_BUILD_BASE, and METPLUS_DATA.
vi METplus-5.0.0_TutorialSetup.sh
METPLUS_BUILD_BASE is the full path to the METplus installation (/path/to/METplus-X.Y)
MET_BUILD_BASE is the full path to the MET installation (/path/to/met-X.Y)
METPLUS_DATA is the location of the sample test data directory

 

Next navigate to the Verify Environment is Set Correctly page.
mccabe Wed, 06/30/2021 - 12:21

Verify Environment is Set Correctly

Verify Environment is Set Correctly

Verify Environment is Set Correctly

Run the Tutorial Setup Script

Navigate to your METplus tutorial directory and source the environment file to apply the settings to the current shell. Each time you log in, you will have to source this file again.
In the following instructions, change "/path/to" to the path to your tutorial directory. EDIT AFTER COPYING and BEFORE HITTING RETURN!
cd /path/to/METplus-5.0.0_Tutorial
source METplus-5.0.0_TutorialSetup.sh

The tutorial setup script sets the paths for METPLUS_TUTORIAL_DIR, METPLUS_BUILD_BASE, MET_BUILD_BASE, and METPLUS_DATA. It also appends the $PATH environment variable to include the directory where the METplus scripts are located. If necessary, it may also load modules needed for the METplus software to run correctly.

Check Path

Make sure that all of the environment variables are set to the appropriate values and that the path is set up to locate the METplus components.

Run the 'which run_metplus.py' command.
If you did everything correctly, the full path displayed should be the script in the shared location, ${METPLUS_BUILD_BASE}:
which run_metplus.py
Run the 'point_stat' command without any arguments.
This should display the usage information for the application.
point_stat

You should see the usage statement for Point-Stat. The version number listed should correspond to the version listed in MET_BUILD_BASE. If it does not, you will need to either reload the met module, or add ${MET_BUILD_BASE}/bin to your PATH.

Check that the environment variables required to run the tutorial instructions are set correctly.

$METPLUS_TUTORIAL_DIR

The directory you created to store all of your tutorial files

echo ${METPLUS_TUTORIAL_DIR}

Example value:

/home/metplus_user/METplus_Tutorial
ls ${METPLUS_TUTORIAL_DIR} -1

Example contents:

METplus-5.0.0_TutorialSetup.sh
output/
tutorial.conf
user_config/

$MET_BUILD_BASE

The directory where MET is installed

echo ${MET_BUILD_BASE}

Example value:

/home/metplus_user/met-11.0.1
ls ${MET_BUILD_BASE} -1

Example contents

bin
share
Note: there may be more files/directories than shown here

Check contents of MET bin directory

ls ${MET_BUILD_BASE}/bin -1

Example contents:

ascii2nc
ensemble_stat
gen_ens_prod
gen_vx_mask
gis_dump_dbf
gis_dump_shp
gis_dump_shx
grid_diag
grid_stat
gsid2mpr
gsidens2orank
ioda2nc
lidar2nc
madis2nc
mode
mode_analysis
modis_regrid
mtd
pb2nc
pcp_combine
plot_data_plane
plot_mode_field
plot_point_obs
point2grid
point_stat
regrid_data_plane
rmw_analysis
series_analysis
shift_data_plane
stat_analysis
tc_dland
tc_gen
tc_pairs
tc_rmw
tc_stat
wavelet_stat
wwmca_plot
wwmca_regrid

$METPLUS_BUILD_BASE

The directory where METplus is installed

echo ${METPLUS_BUILD_BASE}

Example value:

/home/metplus_user/METplus-5.0.1
ls ${METPLUS_BUILD_BASE} -1

Example contents:

build_components
docs
environment.yml
internal
manage_externals
metplus
parm
produtil
pyproject.toml
README.md
requirements.txt
setup.py
ush

$METPLUS_DATA

The directory containing sample input data to use for the tutorial

echo ${METPLUS_DATA}

Example value:

/d1/metplus_user/METplus_Data
ls ${METPLUS_DATA} -1

Example contents:

met_reformat
met_test
model_applications
mccabe Wed, 06/30/2021 - 13:59