Software Containers

Software Containers

Each component of the end-to-end NWP container system has its own software container. There are three options for establishing each image from which the software container will be instantiated: 

1. Pull the image from Docker Hub
2. Build the image from scratch
3. Pull the Docker Hub image and convert it to a Singularity image

Instructions are provided for all three options. Please follow the appropriate section that fits your needs.

DTC NWP Flow
The gray boxes represent the Software Containers you will create and run during this tutorial.

jwolff Tue, 03/19/2019 - 11:42

WPS and WRF NWP Components

WPS and WRF NWP Components

The wps_wrf software container consists of two components:

  • WRF Preprocessing System (WPS)
  • Advanced Weather Research and Forecasting (WRF-ARW) model

There are three options for establishing the image from which the software container will be instantiated.  Please follow the appropriate section below that fits your needs.

Option 1: Pull the dtcenter/wps_wrf image from Docker Hub

If you do not want to build the image from scratch, simply use the prebuilt image by pulling it from Docker Hub.

cd ${PROJ_DIR}
docker pull dtcenter/wps_wrf:${PROJ_VERSION}

To see what images you have available on your system, type:

docker images

Option 2: Build the dtcenter/wps_wrf image from scratch

Building the WPS/WRF image from scratch may require more than the default amount of memory in Docker Desktop. If you experience build failures, try increasing Docker Desktop > Preferences > Resources > Memory to at least 10.00 GB.

To access the preprocessing (WPS) and model (WRF) code from the Git repository and build the image from scratch, first go to your project space where you cloned the repository:

cd ${PROJ_DIR}/container-dtc-nwp/components

and then build an image called dtcenter/wps_wrf:

cd wps_wrf ; docker build -t dtcenter/wps_wrf:${PROJ_VERSION} . ; cd ..

This command goes into the wps_wrf directory and reads the Dockerfile directives. Please review the contents of the wps_wrf/Dockerfile for additional information.

To see what images you have available on your system, type:

docker images

Option 3: Pull the dtcenter/wps_wrf Docker Hub image and convert it to a Singularity image

If you are using Singularity rather than Docker, the commands are similar:

cd ${PROJ_DIR}
singularity pull docker://dtcenter/wps_wrf:${PROJ_VERSION}_for_singularity
NOTE: We provide a Singularity-specific tag for the wps_wrf container because it removes some permissions features that are not necessary for Singularity, and can potentially cause problems for multi-node applications. For more information, reference the page "Running WRF on multiple nodes with Singularity". 

Unlike Docker, Singularity does not keep track of available images in a global environment; images are stored in image files with the .sif extension. Use the `ls` command to see the image file you just downloaded

ls -al *.sif
-rwxr-xr-x 1 ec2-user ec2-user 1022369792 Feb  8 20:26 wps_wrf_${PROJ_VERSION}_for_singularity.sif
jwolff Tue, 03/19/2019 - 11:43

GSI data assimilation

GSI data assimilation

The GSI container will be used to perform data assimilation. There are three options for establishing the image from which the software container will be instantiated.  Please follow the appropriate section below that fits your needs.

Option 1: Pull the dtcenter/gsi image from Docker Hub

If you do not want to build the image from scratch, simply use the prebuilt image by pulling it from Docker Hub.

cd ${PROJ_DIR}
docker pull dtcenter/gsi:${PROJ_VERSION}

To see what images you have available on your system, type:

docker images

Option 2: Build the dtcenter/gsi image from scratch

To access the GSI container from the Git repository and build the image from scratch, first go to your project space where you cloned the repository:

cd ${PROJ_DIR}/container-dtc-nwp/components

and then build an image called dtcenter/gsi:

cd gsi ; docker build -t dtcenter/gsi:${PROJ_VERSION} . ; cd ..

This command goes into the gsi directory and reads the Dockerfile directives. Please review the contents of the gsi/Dockerfile for additional information.

To see what images you have available on your system, type:

docker images

OPTION 3: Pull the dtcenter/gsi Docker Hub image and convert it to a Singularity image

If you are using Singularity rather than Docker, the commands are similar:

cd ${PROJ_DIR}
singularity pull docker://dtcenter/gsi:${PROJ_VERSION}

Unlike Docker, Singularity does not keep track of available images in a global environment; images are stored in image files with the .sif extension. Use the `ls` command to see the image file you just downloaded

ls -al *.sif
-rwxr-xr-x 1 ec2-user ec2-user 758505472 Sep 29 20:55 gsi_${PROJ_VERSION}.sif
-rwxr-xr-x 1 ec2-user ec2-user 879349760 Sep 29 20:49 wps_wrf_${PROJ_VERSION}.sif
jwolff Tue, 03/19/2019 - 11:43

Unified Post Processor (UPP)

Unified Post Processor (UPP)

The UPP container will be used to post process the model output. There are three options for establishing the image from which the software container will be instantiated.  Please follow the appropriate section below that fits your needs.

Option 1: Pull the dtcenter/upp image from Docker Hub

If you do not want to build the image from scratch, simply use the prebuilt image by pulling it from Docker Hub.

cd ${PROJ_DIR}
docker pull dtcenter/upp:${PROJ_VERSION}

To see what images you have available on your system, type:

docker images

Option 2: Build the dtcenter/upp image from scratch

To access the UPP container from the Git repository and build the image from scratch, first go to your project space where you cloned the repository:

cd ${PROJ_DIR}/container-dtc-nwp/components

and then build an image called dtcenter/upp:

cd upp ; docker build -t dtcenter/upp:${PROJ_VERSION} . ; cd ..

This command goes into the upp directory and reads the Dockerfile directives. Please review the contents of the upp/Dockerfile for additional information.

To see what images you have available on your system, type:

docker images

Option 3: Pull the dtcenter/upp Docker Hub image and convert it to a Singularity image

If you are using Singularity rather than Docker, the commands are similar:

cd ${PROJ_DIR}
singularity pull docker://dtcenter/upp:${PROJ_VERSION}

Unlike Docker, Singularity does not keep track of available images in a global environment; images are stored in image files with the .sif extension. Use the `ls` command to see the image file you just downloaded

ls -al *.sif
-rwxr-xr-x 1 ec2-user ec2-user  758505472 Sep 29 20:55 gsi_${PROJ_VERSION}.sif
-rwxr-xr-x 1 ec2-user ec2-user 1224183808 Sep 29 21:04 upp_${PROJ_VERSION}.sif
-rwxr-xr-x 1 ec2-user ec2-user  879349760 Sep 29 20:49 wps_wrf_${PROJ_VERSION}.sif
jwolff Tue, 03/19/2019 - 11:44

Python Graphics

Python Graphics

The Python container will be used to plot model forecast fields. There are three options for establishing the image from which the software container will be instantiated.  Please follow the appropriate section below that fits your needs.

Option 1: Pull the dtcenter/python image from Docker Hub

If you do not want to build the image from scratch, simply use the prebuilt image by pulling it from Docker Hub.

cd ${PROJ_DIR}
docker pull dtcenter/python:${PROJ_VERSION}

To see what images you have available on your system, type:

docker images

Option 2: Build the dtcenter/python image from scratch

To access the Python scripts from the Git repository and build the image from scratch, first go to your project space where you cloned the repository:

cd ${PROJ_DIR}/container-dtc-nwp/components

and then build an image called dtcenter/python:

cd python ; docker build -t dtcenter/python:${PROJ_VERSION} . ; cd ..

This command goes into the python directory and reads the Dockerfile directives. Please review the contents of the python/Dockerfile for additional information.

To see what images you have available on your system, type:

docker images

Option 3: Pull the dtcenter/python Docker Hub image and convert it to a Singularity image

If you are using Singularity rather than Docker, the commands are similar:

cd ${PROJ_DIR}
singularity pull docker://dtcenter/python:${PROJ_VERSION}

Unlike Docker, Singularity does not keep track of available images in a global environment; images are stored in image files with the .sif extension. Use the `ls` command to see the image file you just downloaded

ls -al *.sif
-rwxr-xr-x 1 ec2-user ec2-user  758505472 Sep 29 20:55 gsi_${PROJ_VERSION}.sif
-rwxr-xr-x 1 ec2-user ec2-user  646881280 Sep 29 21:28 python_${PROJ_VERSION}.sif
-rwxr-xr-x 1 ec2-user ec2-user 1224183808 Sep 29 21:04 upp_${PROJ_VERSION}.sif
-rwxr-xr-x 1 ec2-user ec2-user  879349760 Sep 29 20:49 wps_wrf_${PROJ_VERSION}.sif
harrold Thu, 09/24/2020 - 07:58

 

 

MET Verification

MET Verification

Verification of the model fields will be performed using the Model Evaluation Tools (MET) container. There are three options for establishing the image from which the software container will be instantiated.  Please follow the appropriate section below that fits your needs.

Option 1: Pull the dtcenter/nwp-container-met image from Docker Hub

If you do not want to build the image from scratch, simply use the prebuilt image by pulling it from Docker Hub.

cd ${PROJ_DIR}
docker pull dtcenter/nwp-container-met:${PROJ_VERSION}

To see what images you have available on your system, type:

docker images

Option 2: Build the dtcenter/nwp-container-met image from scratch

To access the Model Evaluation Tools (MET) code from the Git repository and build the verification software from scratch, first go to your project space where you cloned the repository:

cd ${PROJ_DIR}/container-dtc-nwp/components

and then build an image called dtcenter/nwp-container-met:

cd met/MET ; docker build -t dtcenter/nwp-container-met:${PROJ_VERSION} . ; cd ../..

This command goes into the met/MET directory and reads the Dockerfile directives. Please review the contents of the met/MET/Dockerfile for additional information.

To see what images you have available on your system, type:

docker images

Option 3: Pull the dtcenter/met Docker Hub image and convert it to a Singularity image

If you are using Singularity rather than Docker, the commands are similar:

cd ${PROJ_DIR}
singularity pull docker://dtcenter/nwp-container-met:${PROJ_VERSION}

Unlike Docker, Singularity does not keep track of available images in a global environment; images are stored in image files with the .sif extension. Use the `ls` command to see the image file you just downloaded

ls -al *.sif
-rwxr-xr-x 1 ec2-user ec2-user  758505472 Sep 29 20:55 gsi_${PROJ_VERSION}.sif
-rwxr-xr-x 1 ec2-user ec2-user 1045712896 Sep 29 21:34 nwp-container-met_${PROJ_VERSION}.sif
-rwxr-xr-x 1 ec2-user ec2-user  646881280 Sep 29 21:28 python_${PROJ_VERSION}.sif
-rwxr-xr-x 1 ec2-user ec2-user 1224183808 Sep 29 21:04 upp_${PROJ_VERSION}.sif
-rwxr-xr-x 1 ec2-user ec2-user  879349760 Sep 29 20:49 wps_wrf_${PROJ_VERSION}.sif
jwolff Tue, 03/19/2019 - 11:45

METviewer Database and Display

METviewer Database and Display

Once verification scores for each valid time have been computed in the MET container, the METviewer database can be loaded and the user interface can be launched in your web-browser to compute summary statistics and display them. There are two options for establishing the image from which the software container will be instantiated.  Please follow the appropriate section below that fits your needs.

Option 1: Pull the dtcenter/metviewer image from Docker Hub

If you do not want to build the image from scratch, simply use the prebuilt image by obtaining the tar file and loading it.

cd ${PROJ_DIR}
docker pull dtcenter/nwp-container-metviewer:${PROJ_VERSION}

If you followed all the instructions up to this point in the tutorial, the output should look similar to this:

docker images
 
REPOSITORY TAG IMAGE ID CREATED SIZE
dtcenter/wps_wrf 3.5.1 9b2f58336cf9 3 hours ago 3.22GB
dtcenter/nwp-container-met 3.5.1 9e3d0e924a30 3 hours ago 5.82GB
dtcenter/upp 3.5.1 82276dfccbc3 4 hours ago 3.36GB
dtcenter/nwp-container-metviewer 3.5.1 f2095355127f 4 hours ago 2.8GB
dtcenter/python 3.5.1 7ea61e625dff 4 hours ago 1.28GB
dtcenter/gsi   3.5.1 c6bebdbfb674 5 hours ago 2.88GB

Option 2: Build the dtcenter/metviewer image from scratch

To access the METviewer database and display system from the Git repository and build the verification software from scratch, first go to your project space where you cloned the repository:

cd ${PROJ_DIR}/container-dtc-nwp/components

and then build an image called dtcenter/metviewer:

cd metviewer/METviewer ; docker build -t dtcenter/nwp-container-metviewer:${PROJ_VERSION} . ; cd ../..

This command goes into the metviewer/METviewer directory and reads the Dockerfile directives. Please review the contents of the metviewer/METviewer/Dockerfile for additional information.

To see what images you have available on your system, type:

docker images

Option 3: Pull the dtcenter/metviewer Docker Hub image and convert it to a Singularity image

If you are using Singularity rather than Docker, the commands are similar:

cd ${PROJ_DIR}
singularity pull docker://dtcenter/nwp-container-metviewer-for-singularity:${PROJ_VERSION}

Unlike Docker, Singularity does not keep track of available images in a global environment; images are stored in image files with the .sif extension. Use the `ls` command to see the image file you just downloaded

ls -al *.sif

 

-rwxr-xr-x 1 ec2-user ec2-user  758505472 Sep 29 20:55 gsi_${PROJ_VERSION}.sif
-rwxr-xr-x 1 ec2-user ec2-user 1045712896 Sep 29 21:34 nwp-container-met_${PROJ_VERSION}.sif
-rwxr-xr-x 1 ec2-user ec2-user 1045712896 Sep 29 21:34 nwp-container-metviewer-for-singularity_${PROJ_VERSION}.sif
-rwxr-xr-x 1 ec2-user ec2-user  646881280 Sep 29 21:28 python_${PROJ_VERSION}.sif
-rwxr-xr-x 1 ec2-user ec2-user 1224183808 Sep 29 21:04 upp_${PROJ_VERSION}.sif
-rwxr-xr-x 1 ec2-user ec2-user  879349760 Sep 29 20:49 wps_wrf_${PROJ_VERSION}.sif

 

jwolff Tue, 03/19/2019 - 11:45