!UPDATE!

Breakthrough Test

While doing some research, some technical documentation was reviewed that
indicated drives would deactivate when utilities scan or perform management
on the drives. For example, LOAD CONFIG, LIST DEVICES, and SCAN FOR NEW
DEVICES are utilities/commands that can scan or perform management on the
drives.

I used this information and accelerated the regular server processes on a
lab server to perform the same daily processes in less time and stress the
server. My first iteration triggered an event in approximately 16 hours (in
normal time we were seeing spans of 90 days between events) and through
changes to the timing of the processes (executing a script called
TRIGGER.NCF every minute), I was able to trigger an event in less than two
(2) hours.

The TRIGGER.NCF script is as shown here:

LIST DEVICES
?Y # Waiting for 10 seconds...
SCAN FOR NEW DEVICES
?Y # Waiting for 10 seconds...
LOAD CONFIG /merna0vbl
?Y # Waiting for 10 seconds...
LIST DEVICES
?Y # Waiting for 10 seconds...
SCAN FOR NEW DEVICES
?Y # Waiting for 10 seconds...
LOAD CONFIG /merna0vbl

This test was a significant step forward in identifying the root cause of
the problem. This information and script were provided to HP for them to
accelerate their testing with an easily repeatable process to trigger the
event.

Cause

HP has determined the cause of this event which we have named the All Drive
Partitions Unavailable problem. Through analysis and troubleshooting, HP
has isolated the condition to the following events.

1.An scan command from the driver to the firmware gets stuck
2.The array controller driver tries to abort the command and times out trying.
3.The driver fails the controller as a result of the time-out
4.The array is taken off-line when the drive fails the controller
5.The NSS pools deactivate and volumes dismount as a result of the array
being taken off-line.

Fix

Currently, there is no fix for the All Drive Partitions Unavailable problem.

HP engineers are working with their developers to make a change to either
the array controller driver or the array controller firmware to address and
resolve the problem.

Workaround

Since the All Drive Partitions Unavailable condition looks to be directly
related to any utilities (i.e. CONFIG, LIST DEVICES, and/or SCAN FOR NEW
DEVICES) that scan or perform management on a drive or drives. Until a
solution is provided, I recommended (for my environment) that the number of
times per day a CONFIG report is executed be reduced to minimize this
activity that contributes to the problem.

The current recommendation is to reduce the number of CONFIG reports from
seven (7) per day (six (6) text format, one (1) XML format) to two (2) per
day (one (1) text format, one (1) XML format).

This is not a solution for the problem, nor does it prevent it from
occurring. It simply reduces the likelihood of it occurring and so far has
proven to be effective.

Additional Information

Additional testing was performed to investigate whether event could be
triggered on other hardware platforms or with Novell NetWare 6.5.

The All Drive Partitions Unavailable problem could not be triggered on a HP
ProLiant DL580 G2 with Novell NetWare 6 SP5, nor a Dell PowerEdge 2650 with
Novell NetWare 6 SP5.

The All Drive Partitions Unavailable problem was triggered on a HP ProLiant
ML530 G2 with Novell NetWare 6.5 SP3 and a HP ProLiant ML530 G2 with Novell
NetWare 6.5 SP3 and Post SP3 updated SERVER.EXE. It did though take
significantly longer to trigger the event on NetWare 6.5 than NetWare 6.0.

BRiAN HUNT