openmpi跨節點報錯tcp_peer_send_blocking: send() to socket 9 failed: Broken pipe (32)

客戶反應作業無法跨節點，運作測試指令如下

mpirun -np 8 -hostfile hostfilt.txt sleep 5
           

運作後報錯如下：

[test02:01719] [[24772,0],1] tcp_peer_send_blocking: send() to socket 9 failed: Broken pipe (32)
--------------------------------------------------------------------------
ORTE was unable to reliably start one or more daemons.
This usually is caused by:

* not finding the required libraries and/or binaries on
  one or more nodes. Please check your PATH and LD_LIBRARY_PATH
  settings, or configure OMPI with --enable-orterun-prefix-by-default

* lack of authority to execute on one or more specified nodes.
  Please verify your allocation and authorities.

* the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base).
  Please check with your sys admin to determine the correct location to use.

*  compilation of the orted with dynamic libraries when static are required
  (e.g., on Cray). Please check your configure cmd line and consider using
  one of the contrib/platform definitions for your system type.

* an inability to create a connection back to mpirun due to a
  lack of common network interfaces and/or no route found between
  them. Please check network connectivity (including firewalls
  and network routing requirements).
-------------------------------------------------------------------------- 
           

網上查到的解決方法：

I had this same problem on Cygwin with OpenMPI 1.10.4.

Try adding "-report-uri -" to your mpirun command to see what IP address it's trying to use for connection:

mpirun -report-uri - -np 2 a.exe

It should print out a line that looks something like this:

568328192.0;tcp://192.168.10.103,169.254.247.11,0.0.0.0,0.0.0.0,0.0.0.0:55600

If the first IP address after the "tcp://" is not a current valid address for your machine, that's the problem and things are likely to break (even if the correct IP appears later in the list). Apparently ORTE is not smart enough to order the interfaces based on what is actually enabled and online.

If the wrong IP corresponds to an old/disabled interface, uninstall it (if possible)

Reference: https://stackoverflow.com/questions/34032655/cygwin-error-tcp-peer-send-blocking-send-to-socket