Python Embedding
Python Embedding cindyhg Thu, 04/25/2019 - 13:13Python Embedding Functionality
As of version 8.0, MET includes the ability to embed Python to a limited degree. Users may use scripts and any associated Python packages in order to prepare 2D gridded data fields for reading by the MET tools. The degree of embedding is expected increase in the future. Plans are in place to extend Python with MET in upcoming releases, allowing users to invoke MET tools directly from their Python script.
Currently, only the use of Python 2.7 is supported, but support will be extended to Python 3 in the future.
The compilation of Python embedding is disabled by default. If you did not configure MET with --enable-python, you may proceed to the next section, or go back and reconfigure and reinstall MET. If compiling support for Python embedding, be sure set $MET_PYTHON_CC and $MET_PYTHON_LD before running configure as described in the "Set Environment Variables" section on the Compilation page of this tutorial. For more detailed information on the Python embedding functionality, see Appendix F in the METv8.0 User's Guide.
Python Embedding Usage
Currently, MET offers two ways to interact with Python scripts: by using NumPy arrays or Xarray objects. The user's scripts can use any Python libraries that are desired in order to implement the Python script, so long as the data has been loaded into either a NumPy array or an Xarray object by the end of the script. This offers advantages when using data file formats that MET does not directly support. If there is Python code to read the data format, the user can use those tools to read the data, and then copy the data into a NumPy array or an Xarray object. MET can then ingest the data via the Python script. Note that whether a NumPy array or an Xarray object is used, the data should be stored as double precision floating point numbers. Using different data types, such as integers or single precision floating point numbers, will lead to unexpected results in MET.
The interface to be used (NumPy or Xarray) is specified on the command line. Python embedding can be used with any MET tool that reads 2-D data. This functionality is used when replacing the path to a gridded data file with PYTHON_NUMPY or PYTHON_XARRAY. Additionally, the "name" entry specifies the Python command that should be executed to read the data. The "level" entry is not required for Python. For example:
Run
Run cindyhg Thu, 04/25/2019 - 13:14In this first example, we'll use Python to pass data to the Plot-Data-Plane utility. The Plot-Data-Plane utility provides a way to visualize the gridded data fields that MET can read.
We'll plot ASCII CMORPH convective rain data from a NetCDF file using the following command:
PYTHON_NUMPY $MET_TUTORIAL_DATA/output/python/crain.ps\
'name="${MET_TUTORIAL_DATA}/input/sample_python/plot_data_plane/read_AWS_CMORPH.py ${MET_TUTORIAL_DATA}/input/sample_python/plot_data_plane/AWS_CMORPH_crain-2017043006.txt";'
Note how "PYTHON_NUMPY" is used in place of the input_filename, replacing the path to a gridded data file, in the call to plot_data_plane.
Now try running the Grid-Stat utility using the Python embedding functionality.
PYTHON_NUMPY PYTHON_NUMPY \
$MET_TUTORIAL_DATA/input/sample_python/grid_stat/GridStatConfig_python_tutorial \
-outdir $MET_TUTORIAL_DATA/output/python -v 1
Note how "PYTHON_NUMPY" is used in place of both the fcst_filename and the obs_filename, replacing the path to a gridded data file, in the call to grid_stat.
Take a look inside the $MET_TUTORIAL_DATA/input/sample_python/grid_stat/GridStatConfig_python_tutorial file. You can see the "name" entry under "field" in the fcst dictionary, where it specifies the Python command to be executed to read the data. In this case, it is
{
name = "${MET_BASE}/python/read_ascii_numpy.py ${MET_TUTORIAL_DATA}/input/sample_python/grid_stat/fcst.txt FCST"
}
];
The same is true for the "name" entry under "field" in the obs dictionary, where it specifies the Python command to be executed to read the data. In this case, it is
{
name = "${MET_BASE}/python/read_ascii_numpy.py ${MET_TUTORIAL_DATA}/input/sample_python/grid_stat/obs.txt OBS"
}
];
Output
Output cindyhg Thu, 04/25/2019 - 13:15Display the output PostScript file, which was generated on the previous page using Plot-Data-Plane, by running the following command:
Close this file.
On the previous page, Grid-Stat was run. It produced an output NetCDF file containing difference fields. Now dump the header using the ncdump utility (if available on your machine):
Users can view the NetCDF header to see how the variable names are defined.
Use the ncview utility (if avaliable on your machine) to view the NetCDF output of Grid-Stat:
Click through the variable names in the ncview window to see plots of the forecast, observation, and difference fields for each masking region. Note that in this example the FCST_FCST_Surface_FULL field represents the text "FCST" as opposed to actual forecast values, the OBS_OBS_Surface_FULL field represents the text "OBS" as opposed to actual observation values, and the DIFF_FCST_Surface_OBS_Surface_FULL field represents where the differences lie between the text "FCST" and "OBS".
To see other examples of using Python to pass data (ASCII, NetCDF, and Binary) to MET's plot_data_plane utility please see the "Python Scripts" section on our Sample MET Analysis Scripts page.