Hi

today, we realized that we can not remove or add any replicas in our ring.

iManager shows Master-Replica and all R/W-Replicas in state "on". ndsrepair shows no errors when listing servers and replica-ring.

ndstrace shows skulk (-698) errors for every server with a replica.
below, i have copied part of the nds-trace.

i know that skulk-errors are supposed to go away, but in our case they aren't.

Any ideas how to proceed?

Thanks
Andrej

Using Sync Point Type 2, for .KABE. to .kabestorage.KABE.KABE.
Sync - [0000805d] <.KABE.> [1996/12/23 9:13:42, 1, 1].
Skulk Partition - change cache entry count 115 for .KABE.
2014/12/18 18:04:28 Start partition sync.KABE., server .arneggoes.KABE.KABE. state:[0], type:[0].
Sync - Start outbound sync with (#=1, state=0, type=1 partition .KABE.) .arneggoes.KABE.KABE..
Error _StartUpdateReplica to .arneggoes.KABE.KABE., failed, replica in skulk (-698)
Send Partition Updates completed in Seconds 0, in MilliSeconds 433 - Total objects 1 Total Changes 7, 1 Packet(s) Sent
Signaling time vector merge on server <.kabestorage.KABE.KABE.> for .KABE..
Sync - objects: 1, total changes: 7, sent to server <.kabestorage.KABE.KABE.> for .KABE..
Sync - Process: Send updates to <.kabestorage.KABE.KABE.> for .KABE. succeeded.
Scheduling ObitProc for .KABE.
Sync - Partition .KABE. All processed = YES
Skulk Partition - change cache entry count 115 for .KABE.
2014/12/18 18:04:29 Start partition sync.KABE., server .kabevsc.KABE.KABE. state:[0], type:[0].
Sync - Start outbound sync with (#=3, state=0, type=1 partition .KABE.) .kabevsc.KABE.KABE..
Sync - using version 9 on server <.kabevsc.KABE.KABE.>.
Sending to ----> .kabevsc.KABE.KABE.
Sync - sending updates to server <.kabevsc.KABE.KABE.>.
Send Partition Updates started usingDispatcher=0
ComputeLowestCompareTime 0x54930910 (2014/12/18 18:04:16, 7, 2)
Using Sync Point Type 2, for .KABE. to .kabevsc.KABE.KABE.
Sync - [0000805d] <.KABE.> [1996/12/23 9:13:42, 1, 1].
Send Partition Updates completed in Seconds 0, in MilliSeconds 433 - Total objects 1 Total Changes 7, 1 Packet(s) Sent
Signaling time vector merge on server <.kabevsc.KABE.KABE.> for .KABE..
Sync - objects: 1, total changes: 7, sent to server <.kabevsc.KABE.KABE.> for .KABE..
Sync - Process: Send updates to <.kabevsc.KABE.KABE.> for .KABE. succeeded.
Scheduling ObitProc for .KABE.
Sync - Partition .KABE. All processed = YES
Skulk Partition - change cache entry count 115 for .KABE.
2014/12/18 18:04:29 Start partition sync.KABE., server .kabemail2.KABE.KABE. state:[0], type:[0].
Sync - Start outbound sync with (#=7, state=0, type=1 partition .KABE.) .kabemail2.KABE.KABE..
Sync - using version 9 on server <.kabemail2.KABE.KABE.>.
Sending to ----> .kabemail2.KABE.KABE.
Sync - sending updates to server <.kabemail2.KABE.KABE.>.
Send Partition Updates started usingDispatcher=0
ComputeLowestCompareTime 0x54930913 (2014/12/18 18:04:19, 8, 4)
Using Sync Point Type 2, for .KABE. to .kabemail2.KABE.KABE.
Sync - [0000805d] <.KABE.> [1996/12/23 9:13:42, 1, 1].
Skulk Partition - change cache entry count 115 for .KABE.
2014/12/18 18:04:29 Start partition sync.KABE., server .kabestorage2.KABE.KABE. state:[0], type:[0].
Sync - Start outbound sync with (#=4, state=0, type=1 partition .KABE.) .kabestorage2.KABE.KABE..
Sync - using version 9 on server <.kabestorage2.KABE.KABE.>.
Sending to ----> .kabestorage2.KABE.KABE.
Sync - sending updates to server <.kabestorage2.KABE.KABE.>.
Send Partition Updates started usingDispatcher=0
ComputeLowestCompareTime 0x54930916 (2014/12/18 18:04:22, 7, 2)
Using Sync Point Type 2, for .KABE. to .kabestorage2.KABE.KABE.
Sync - [0000805d] <.KABE.> [1996/12/23 9:13:42, 1, 1].
Send Partition Updates completed in Seconds 0, in MilliSeconds 407 - Total objects 1 Total Changes 7, 1 Packet(s) Sent
Signaling time vector merge on server <.kabemail2.KABE.KABE.> for .KABE..
Sync - objects: 1, total changes: 7, sent to server <.kabemail2.KABE.KABE.> for .KABE..
Sync - Process: Send updates to <.kabemail2.KABE.KABE.> for .KABE. succeeded.