METplus Practical Session Guide (Version 4.0) | Session 2: Grid-to-Obs > METplus Use Case: ASCII2NC with Python Embedding

IMPORTANT NOTE: If you are returning to the tutorial, you must source the tutorial setup script before running the following instructions. If you are unsure if you have done this step, please navigate to the Verify Environment is Set Correctly page.

METplus Use Case: ASCII2NC with Python Embedding

This use case utilizes the MET ASCII2NC tool to demonstrate Python Embedding. Python embedding is a novel capability within METplus that allows a user to place a Python script into a METplus workflow. For example, if a user has a data format that is unsupported by the MET tools then a user could write their own Python file reader, and hand off the data to the MET tools within a workflow.

The data utilized in this use case are hypothetical accumulated precipitation data in ASCII format.

Optional: Refer to the MET Users Guide for a description of the MET tools used in this use case.
Optional: Refer to the MET Users Guide Appendix F: Python Embedding for details on how Python embedding works for MET tools.
Optional: Refer to the METplus Config Glossary section of the METplus Users Guide for a reference to METplus variables used in this use case.

 

  1. View the template Python Embedding scripts available with MET

MET includes example scripts for reading point data and gridded data (all in ASCII format). These can be used by users as a jumping-off point for developing their own Python embedding scripts, and are also used by this use case and other METplus use cases that demonstrate Python Embedding

ls -1 ${MET_BUILD_BASE}/share/met/python
read_ascii_mpr.py
read_ascii_numpy_grid.py
read_ascii_numpy.py
read_ascii_point.py
read_ascii_xarray.py

In this use case, read_ascii_point.py is used to read the sample accumulated precipitation data and serve them to ASCII2NC to format into the MET 11-column netCDF format.

  1. Inspect a sample of the accumulated precipitation point data being read with Python.
head -3 ${METPLUS_DATA}/met_test/data/sample_obs/ascii/sample_ascii_obs.txt

These data are already in the 11-column format that MET requires, so using Python Embedding is fairly straightforward. If your data do not follow this format, some pre-processing of the data will be required to align your data to the required format. To learn more about what the 11 columns are and what MET expects each column to represent, please reference this tablefrom the MET users guide.

  1. Inspect the Python Embedding script being used in this use case
less ${MET_BUILD_BASE}/share/met/python/read_ascii_point.py

Since the sample data are point data, the Python module Pandas is utilized to read the ASCII data, assign column names that match the MET 11-column format, and then pass the data to the MET ASCII2NC tool.

  1. Run the ASCII2NC Python Embedding use case:
run_metplus.py \
${METPLUS_BUILD_BASE}/parm/use_cases/met_tool_wrapper/ASCII2NC/ASCII2NC_python_embedding.conf \
${METPLUS_TUTORIAL_DIR}/tutorial.conf \
config.OUTPUT_BASE=${METPLUS_TUTORIAL_DIR}/output/ASCII2NC_python_embedding

This command utilizes the METplus wrappers to execute a Python script to read in the ASCII data and pass them off to ASCII2NC to write to a netCDF file. While the Python script called here is simply for demonstration purposes, your Python script could serve to perform various data manipulation and formatting, or reading of proprietary or unsupported file formats to end up with a netCDF file compatible with the MET tools.

  1. Review the Output Files
ls ${METPLUS_TUTORIAL_DIR}/output/ASCII2NC_python_embedding/met_tool_wrapper/ASCII2NC

You should see the output netCDF file (ascii2nc_python.nc), which conforms to the 11-column format required by the MET tools.

  1. Let's Write Some Python!

Copy the METplus use configuration file to your current working directory:

cp ${METPLUS_BUILD_BASE}/parm/use_cases/met_tool_wrapper/ASCII2NC/ASCII2NC_python_embedding.conf ${METPLUS_TUTORIAL_DIR}

Copy the Python Embedding script to your current working directory:

cp ${MET_BUILD_BASE}/share/met/python/read_ascii_point.py ${METPLUS_TUTORIAL_DIR}

Open the Python Embedding script (${METPLUS_TUTORIAL_DIR}/read_ascii_point.py) in a text editor of your choice, and add the following on line 34 of the file (make sure the indentation matches the previous line):

print("HELLO! CREATING PANDAS DATAFRAME OF POINT OBS.")

Save the Python Embedding script.

  1. Modify the METplus Use Case Configuration File to Use Your Python Script

Open the METplus use case file (${METPLUS_TUTORIAL_DIR}/ASCII2NC_python_embedding.conf) in a text editor of your choice and modify line 108 to use your Python script:

ASCII2NC_INPUT_TEMPLATE = "{ENV[METPLUS_TUTORIAL_DIR]}/read_ascii_point.py {INPUT_BASE}/met_test/data/sample_obs/ascii/sample_ascii_obs.txt"

Save the METplus use case configuration file.

  1. Re-run the Use Case, and See If Your Python Code Is Used
run_metplus.py \
${METPLUS_TUTORIAL_DIR}/ASCII2NC_python_embedding.conf \
${METPLUS_TUTORIAL_DIR}/tutorial.conf \
config.OUTPUT_BASE=${METPLUS_TUTORIAL_DIR}/output/ASCII2NC_python_embedding
  1. Look for Your Python Print Statement In the METplus Log File
less ${METPLUS_TUTORIAL_DIR}/output/ASCII2NC_python_embedding/logs/metplus.log.YYYYMMSSHHMMSS

(NOTE: Replace YYYYMMDDHHMMSS with the time and date appended to the log file from the run in step 8).

You should see the print statement written to the log file, showing that your modifications were used by METplus and ASCII2NC!