METplus Use Case: ASCII2NC with Python Embedding

METplus Use Case: ASCII2NC with Python Embedding

This use case utilizes the MET ASCII2NC tool to demonstrate Python Embedding. Python embedding is a novel capability within METplus that allows a user to place a Python script into a METplus workflow. For example, if a user has a data format that is unsupported by the MET tools then a user could write their own Python file reader, and hand off the data to the MET tools within a workflow.

The data utilized in this use case are hypothetical accumulated precipitation data in ASCII format.

Optional: Refer to the MET Users Guide for a description of the MET tools used in this use case.
Optional: Refer to the MET Users Guide Appendix F: Python Embedding for details on how Python embedding works for MET tools.
Optional: Refer to the METplus Config Glossary section of the METplus Users Guide for a reference to METplus variables used in this use case.

1. View the template Python Embedding scripts available with MET

MET includes example scripts for reading point data and gridded data (all in ASCII format). These can be used by users as a jumping-off point for developing their own Python embedding scripts, and are also used by this use case and other METplus use cases that demonstrate Python Embedding

In this use case, read_ascii_point.py is used to read the sample accumulated precipitation data and serve them to ASCII2NC to format into the MET 11-column netCDF format.

2. Inspect a sample of the accumulated precipitation point data being read with Python.

These data are already in the 11-column format that MET requires, so using Python Embedding is fairly straightforward. If your data do not follow this format, some pre-processing of the data will be required to align your data to the required format. To learn more about what the 11 columns are and what MET expects each column to represent, please reference this table from the MET users guide.

3. Inspect the Python Embedding script being used in this use case

Since the sample data are point data, the Python module Pandas is utilized to read the ASCII data, assign column names that match the MET 11-column format, and then pass the data to the MET ASCII2NC tool.

4. Run the ASCII2NC Python Embedding use case:

This command utilizes the METplus wrappers to execute a Python script to read in the ASCII data and pass them off to ASCII2NC to write to a netCDF file. While the Python script called here is simply for demonstration purposes, your Python script could serve to perform various data manipulation and formatting, or reading of proprietary or unsupported file formats to end up with a netCDF file compatible with the MET tools.

5. Review the Output Files

You should see the output netCDF file (ascii2nc_python.nc), which conforms to the 11-column format required by the MET tools.

6. Let's Write Some Python!

Copy the METplus use configuration file to your current working directory:

Copy the Python Embedding script to your current working directory and rename it to my_ascii_point.py:

Open the Python Embedding script in a text editor of your choice, and add the following on line 7 of the file (make sure the indentation matches the previous line):

Add on line 7:

Save the Python Embedding script.

7. Modify the METplus Use Case Configuration File to Use Your Python Script

Open the METplus use case file in a text editor of your choice and modify line 43 to use your Python script:

Change line 43 to:

The Python script contains an import of met_point_obs. This is found automatically by the read_ascii_point.py script because it is in the same share/met/python directory. Since you are calling my_ascii_point.py from your tutorial directory, you will have to set the PYTHONPATH to include the share/met/python directory so the import succeeds.

Add the following to the very end of ${METPLUS_TUTORIAL_DIR}/ASCII2NC_python_embedding.conf:

Save the METplus use case configuration file.

8. Re-run the Use Case, and See If Your Python Code Is Used

9. Look for Your Python Print Statement In the METplus Log File

The 2nd last log message printed to the screen will contain the full path to the log file that was created for this run. Copy this path and run less to view the log file. It will look something like this:

(NOTE: Replace YYYYMMDDHHMMSS with the time and date appended to the log file from the run in step 8). You should see the print statement written to the log file, showing that your modifications were used by METplus and ASCII2NC!