Removal of Content Advisory - April 2024

Advisory to Hurricane WRF (HWRF) users: As of the beginning of April 2024, all support assets for Hurricane WRF (HWRF) will be removed from the DTC website. Users should download all reference materials of interest prior to April 2024.

Hurricane WRF (HWRF) | FAQs

Please see example under the heading "Update your branch with trunk updates" on this page

Rocoto is a workflow manager that must be run repeatedly to monitor the status of the jobs in the workflow. Each job has a set of dependencies, or requirements, that must be met before the next job may be submitted. Each time Rocoto is run, it assess the current status of the dependencies of each job and performs any actions necessary based on the dependencies that have been met. To run HWRF using the Rocoto workflow manager, do the following:

  1. Choose configure options for HWRF. These options are contained in conf files of the parm/ directory.
  2. Run run_hwrf.py without the -f option to create a workflow (.xml) file specific to your configuration. Make sure that all the required options are also submitted on the command line. Any configuration options not changed in teh parm directory may be submitted as a command line argument, i.e.
    run_hwrf.py 2012 18L HISTORY dir.CDNOSCRUB='/pan2/projects/dtc-hurr/experiment' config.run_gsi=no

    will create a workflow that will run all cases of Hurricane Sandy in 2012 for which there are TCVitals records. The output of the will be redirected to '/pan2/projects/dtc-hurr/experiment' and there will be no data assimilation for the experiment.

  3. After a few minutes have passed, resubmit the same run_hwrf.py command with the exception of an added "-f" option, which will instruct the script to use the current XML and database file already created instead of overwriting the files. This would like
    run_hwrf.py -f 2012 18L HISTORY dir.CDNOSCRUB='/pan2/projects/dtc-hurr/experiment' config.run_gsi=no
  4. You may either submit this command by hand until the workflow is complete, or use the UNIX cron utility. Edit the cron to run the previous command every 3-5 minutes.
  5. For more information on using Rocoto for HWRF, please refer to the training slides by Sam Trahan (here) and Christina Holt (here) .

Let us know that you have developments ready to go into the trunk by bringing it up during the Monday HWRF Committee Meetings, or by emailing the HWRF Developers Committee. Once the changes have been approved by the HWRF Committee, then we will discuss the proper testing necessary for merging the code back into the trunk.

The following list provides a good starting place of the necessary guidelines, however there may be additional requirements that should be maintained depending on the problem. Staying in touch with the HWRF Developers Committee about your ongoing developments is still of paramount importance.

  1. Do not call any external programs except for ones we compile in the sorc/ directory. Use the produtil package Python standard library, or other Python code instead.
  2. All code must be readable and commented.
  3. All code is in Python, for now must be compatible with Red Had Python 2.6.6 and 2.7. Required Python versions may change in the future.
  4. No duplication of functionality without a very good reason. If something already exists in a package, use it. If it cannot be used, improve it so it can be used.
  5. Everything is connected to the central configuration mechanism (config.py, parm/*.conf).
  6. All code follows the multi-layer architecture (https://wiki.ucar.edu/display/DTCHWRF/Technical+Overview). In short: Platform-dependent logic is in the ush/produtil or rocoto/ and scripts. Low-level logic goes in ush/hwrf, high-level logic goes in scripts/*.py and everything is as configurable as possible. Configuration is done in the hwrf_expt.py and parm/*.conf files.
  7. Nothing should be hard-coded to specific domain sizes, physics schemes, resolutions, etc. below the Experiment Layer.
  8. Do not use any Python standard library functionality if the equivalent produtil functionality is available.
  9. All functionality must be connected to the Tasks/Products/Datastore family of classes.

No, it is not required that you have an access to HPSS to run HWRF, but it is required that the required input data be accessible by the compute node and that the configure files know where to find it. There are two HWRF tasks that rely on HPSS -- gathering input data and archiving HWRF runs. To point to data on disk, you can edit parm/hwrf_input.conf variable, e.g. input_catalog=[jet_hist_PROD2014], or edit the [comm_hist] section to point to a user-specified location. You must make sure that the naming convention section is also consistent with the data you are using.

When running HWRF, you will also need to disable the archiving to hpss. This is done by setting archive=none or archive=disk:some/path in the parm/system.conf file.

To obtain an HPSS access, submit a help ticket to Jet Help.

To build HWRF on Jet, there is a new (as of May 2015) build process that is optional, but strongly recommended. Once a copy of HWRF is checked out from the SVN repository (located on disk at ${location}), use the following commands to make use of the module files that are included in the HWRF system:

  • module use /${location}/modulefiles/$site # $site=jet, wcoss, etc.
  • module load HWRF/build

This will purge all loaded modules in the user's environment and load the ones necessary for HWRF to compile. Once the HWRF build module is loaded, then continue with the build as outlined here in the "Compile" section.

By default, the configuration of the operational HWRF is domain-dependent. To disable per-basin reconfiguration altogether, change this line of hwrf_basic.conf:

  • basin_overrides=yes ; read parm/hwrf_(basin).conf

Replace "yes" with "no".

Most failures are the result of one of three issues:

  1. User does not have HPSS (see related question above).
  2. User does not have access to linux group rstprod for retrieval of restricted files from HPSS.
  3. User does not have access to NOAA's Zeus super computer.

HPSS access can be acquired by submitting a help ticket to Jet Help. Access to restricted data can be granted by filling out the NCO form here. Unless you are explicitly a part of one of the Centers listed, choose you Sponsoring Center as the Environmental Modeling Center.

The variable nio_tasks_per_group has been changed from a scalar to an array in a recent commit. It should be specified, for example, as

&namelist_quilt
poll_servers = T
nio_tasks_per_group = 4,4,4
nio_groups = 4,

That means the 4*4+4*4+4*4=48 processors will be used for I/O, so you need to adjust your total number of processors accordingly.

Please refer to the documentation included in the doc/multistorm.md file that is available as a part of the standard check out of HWRF. You may also find the information at the EMC HWRF Scripts Documentation webpage.

Use the WRF namelist option mp_physics=15. The configuration setting for this option is moad_namelist.physics.mp_physics=15. Please make sure that you are using revision 8758 of the branches/HWRF of the WRF repository at a minimum.

The following options are configurable within parm/hwrf.conf. They follow certain rules, as follows. The example below assumes an HWRF timestep of 30s.

&domains
  time_step = 30,	      ;; Dynamic time steps (seconds).
  parent_time_step_ratio = 1, 3, 3,     ;; Ratio of dynamic timesteps to parent. 

&physics
  nrads = 30, 90, 270,        ;; Number of dynamic timesteps between shortwave radiation calls.
  nradl = 30, 90, 270,        ;; Number of dynamic timesteps between shortwave radiation calls.
  nphs = 2, 6, 6,             ;; Number of dynamic timesteps between physics calls.
  ncnvc = 2, 6, 6,            ;; Number of dynamic timesteps between convection calls.
  ntrack = 9, 9, 18, 	      ;; Number of physics calls between tracker timesteps.

  Parent Outer Nest Inner Nest
HWRF Timestep (s): 30 10 3 1/3
Phys & Conv Timestep (s): 60 60 20
Radiation Timestep (s): 900 900 900

These choices of WRF timesteps must also align with the coupler timesteps (set to dt_c=540 in hwrf.conf by default) such that:

 

	ntrack * nphs * timestep                 e.g. with D02:       9 * 6 * 30
       -------------------------- = dt_c                             ------------ = 540
         parent_timestep_ratio					          3

There are several considerations to make when changing the number of processors for the multistorm (MS) configuration forecast.

  • The processor counts on the hardward you are using.
  • The number of storms you are configuring.
  • How quickly you need to run.
  • How to make these changes in the HWRF scripts and configuration files.

Examples here are intended to reflect the best practices. More information can be found in the documentation here: HWRF Scripts

Hardware interactions:

It is important to consider how many processors are your machine, or partition on Jet. This will allow for more effecient communicatino between patches in the forecast. You probably want to choose the x-direction processors to reflect the number of processors per node on the architecture you choose to run on. If there are 16 processors per node on your favorite machine, then choose numbers that are multiples of 16 in the x-direction.

Number of storms:

You may want to think about scaling as if you were running multiple copies of the single-storm (SS) HWRF. For each additional storm, add enough extra processors to compute an extra copy of HWRF. (Conf example given below.) If SS HWRF runs with 480 processors, then consider tripling that number (1440) to run the 3-storm case for MS.

Speed of forecast:

The above guideline of using the SS HWRF configuration as a guideline assumes that you will want to try to run the MS coniguration in a similar amount of time as the SS HWRF. If that is not necessary, follow guidelines that scale similarly, but keep your total compute time within the 8 hour maximum job time. Generally attempt to use the same number of processors for each set of nests and increase the total number of processors for the MOAD to be the storm number multiple. For example, if you think that you can afford half the processors of a standard SS HWRF run for a 3 storm case, use 240 processors per set of nests and a total of 720 processors.

HWRF configuration and job submission:

Changes will be required in two places: in the configuration (an additional parm/*.conf file) and in the job submission scripts (rocoto/sites/*.ent).

First, the conf file: 
 The important variables
  • max-dom: Integer automatically set by HWRF scripts depending on number of storms in configuration. Indicates the number of columns to be read for any arrays.
  • comm_start: Array of labels of the first processor of the group.
  • nest_pes_x: Array of number of processors to use in the x-direction for each domain.
  • nest_pes_y: Array of number of processors to use in the y-direction for each domain.
  • nio_tasks_per_group: Array of number of I/O tasks to use for each group.
  • nio_groups: Integer indicating the number of I/O groups to use.

For each of the arrays above, the first position corresponds to the parent domain (MOAD, or D01), and the subsequent positions refer to pairs of nests associated with up to five storms.

When running HWRF, some processors can be used exclusively for I/O, while others are used exclusively for computations. The sum of the I/O and compute processors should equal the number of cores requested from the batch system.

 Then, the Rocoto files: 

You will need to change the Rocoto scripts to reflect these choices, and ask for the exact number of cores. Edit HWRF/rocoto/sites/*.ent, choosing your preferred machine, to make sure there is a corresponding set of ATM_FCST and ATM_TASKS variables for each option of storm number, up to 5.

Configuration example:

The following example is for a 3 storm case running on s/vjet (16ppn) following the rule of thumb that SS HWRF runs with 16x28 processors, so uses 3x that number. The MOAD runs first and uses all processors, then set of nests split those processors evenly. The comm_start indicates that the first set of nests run on the first 448 compute procs, the second set of nests run on procs 448-895, and so on.

number of storms = 3
number of cores = (48x28) compute procs + (16 + (16 + 8) * 3) I/O procs  = 1432

Configuration: 
  comm_start = 0, 0, 0, 448, 448, 896, 896, 1344, 1344, 1792, 1792,
  nest_pes_x = 48, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16,
  nest_pes_y = 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28,

  nio_tasks_per_group = 4, 4, 2, 4, 2, 4, 2, 4, 2, 4, 2,
  nio_groups = 4

Processor layout for s/v jet (16ppn) totalling 1432 (88*16 + 2*12):
  88:ppn=16+2:ppn=12