I have a 5 node cluster where two of the nodes are refusing to join. I suspected this could be hardware but looking at this in detail I am not convinced.

All 5 nodes have been working fine, but two nodes have developed issues, been restarted and then refused to join the cluster.
LNDCS produces a fatal 12038 SBD has reported a fatal error, when loaded on these two nodes. The others are fine. I have tried shutting down the whole cluster and this has not fixed the problem.

If I run NSSMU from these servers, the two problem servers can see all the partitions but it shows as 0 partitioned space 0 unpartitioned space shareable for clustering no, on two partitions, one of which of course holds the cluster info. This is consistent on the two problem servers. The other three can read the partition table okay.

These servers are dual attached via two switches to the storage. HP Securepath is installed and configured and everything has been working fine. We have not patched anything other than SAV in the last few months, so there are no software changes that could account for this.

We have added two cluster pools recently, but these can be seen from all servers.

Any ideas?