Ensemble DA tasks ('enshx') Failing on Hera for HWRF-B

Dear HWRF Developer's Forum,

I am running experiments with the Basin-scale HWRF (HWRF-B) on Hera. Unfortunately, the ensemble DA is failing. Specifically, the "enshx_DA_0[01-40]" tasks are all failing when running the hwrf_gsi executable. The failure happens within 30 seconds of the start time, and might be associated with a reference to "inconsistent ndat,npe" (see below). The most confusing aspect of this issue is that my colleague, Sarah Ditchek, can run the enshx tasks for the same forecast cycle successfully. I have tried rerunning the enshx tasks after copying her executables (no luck) and copying her ProdGSI source code (no luck). I do not appear to have any notable differences in the configuration (parm) beyond unique paths from Sarah. Also, I was able to run these tasks successfully a few months ago. That leads me to believe there is some issue in my environment. I confirmed that Sarah and I are running with the same modules, but I can't guarantee that the environments are identical. I am stumped... any ideas?

The rocoto log file: /scratch2/AOML/aoml-hafs1/Ghassan.Alaka/pytmp/HB20_dorian_TEST/2019082418/00L/hwrf_init_hx001_storm2.log

The stdout file: /scratch2/AOML/aoml-hafs1/Ghassan.Alaka/pytmp/HB20_dorian_TEST/2019082418/05L/ensda/001/gsihx/stdout

The error:
 ****STOP2****  ABORTING EXECUTION w/code=         330
  observer_set: inconsistent ndat,npe           92         160  /=           92
         200
 ****STOP2****  ABORTING EXECUTION w/code=         330
 

Thank you very much,

Gus Alaka

ghassan.alaka@noaa.gov