Having installed and supported Groupwise for a long time -- my first Netware install was 2.15 -- this is an embarrassing problem to admit, but I've put myself squarely into Murphy's cross-hairs, and am hoping someone else can help me recover from my extensive temporal stupidity (a/k/a "I'll fix that next week"). Here's my problem.

I have a GW2012 post office that has experienced PO corruption when the POA was shut down gracefully on three prior occasions, producing an "8101 Memory function failure" or "8101 Error reading time zone record" message, crashing the POA at startup, and requiring a ConsoleOne PO "Rebuild database" to correct. The last time this happened, though, all of the user mailboxes were cleared: no contact, email, calendar, etc., although the message store appears to still have all of the message data intact. I've tried to rebuild the user databases, but have gotten only a few messages and calendar items, all from the last 30 days, a very small portion of the message store: dozens recovered, out of tens of thousands total. Checking the /ofuser folder, all of the USERnnn.db files appear to have been wiped clean for all mailboxes, but the /ofmsg and /offiles contents appear normal size.

The server is GW12.0.2, running on SLES11 SP2 and OES11 SP1. Physical hardware is a six core CPU, 16GB of RAM, and 300GB+ free on the raid'ed NSS volume that holds the ~90GB Groupwise store. Workstations are Win 7 Pro 64-bit, Win 8 Pro, or Win XP, and we see the same info via WebAccess as the direct clients, so this is not a client issue. The PO issues did not exist until a few weeks ago, when we finally got around to retiring our last Netware server and upgrading this box to OES11 SP1, which also required the upgrade to GW 12.0.2 from 12.0.0, and we consolidated two post offices at the same time. Just after the upgrade, the PO crashes started on the server, which behaved itself for a few weeks... until this last crash, which has been the worst of the bunch.

Normally, this would be bad, but that's what backups are for, right? Well... we screwed that up, as this past week we reformatted and reinstalled the Netware box as a new Linux server... the same Netware box that was the target of the Groupwise backups, so backups aren't available. Since we used to backup GW using dbcopy and rsync to the Netware server, we didn't include it in our normal file backups... yet... another fatal flaw in the plan. While the NSS volume on the new server has Salvage enabled, there are no salvageable copies of the USERnnn.db files. As part of the upgrade, we transported some of the affected users from a different PO in the same domain, but have nothing left there that would appear to be useful for recovery.

All I have left is thousands of messages in 90GB of message store which I can't seem to bring back, trying both "Recreate user database" and "Structural rebuild" on the mailboxes from both the C1 and standalone 12.0.2 GWCheck. The GWCheck log message counts and file sizes give every impression that the messages are still present, but I'm failing to recreate the user databases. I recognize that some elements (i.e. rules, contacts?) are probably toast, but being able to recover even the calendars and email messages would be a dramatic improvement.

Am I missing an option here that might bring this data back?

Michael Zore
Sterling Technologies Group, Ltd.