I'm using BM 3.8 with SP4 installed and have a client who is trying to use
the VPN client that came with SP4 on a Compaq N1000C notebook. They do not
have the Netware client installed, only the VPN client. Their network is
using a DSL connection and a D-Link DI-604 router with a built in firewall.
I've added filter rules to the router to allow all UDP & TCP traffic on all
ports from the VPN server's public address to the the router's public
address. We issued a certifice to the edirectory user using iManager and
imported it into the VPN client by copying it to the certificates folder.

When the user tries to connect to the VPN server the connection is never
completed. I can see that IKE is receiving the request but fails to
complete the connnection. After a couple of minutes the VPN client pops up
ths message:

"An error was reported by the IKE application. Either [IP] address is an
invalid vpn server address or the IKE is not loaded on the VPN server. For
details please look at IKE.log"

Here's the IKE log:

2-1-2006 6:30:53 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:30:53 pm
I-COOKIE=5C77B3216A4259AC,R-COOKIE=0000000000000000,MsgID=0,1stPL=SA-PAYLOAD,state=-871929040
2-1-2006 6:30:53 pm Start IKE-SA CDACD4E0 - Responder,src=[VPN
IP],dst=[Client IP],TotSA=6
2-1-2006 6:30:53 pm AUTH ALG IS 3
2-1-2006 6:30:53 pm IKE SA NEGOTIATION: Peer lifetime = 28800 My
lifetime=28800
2-1-2006 6:30:53 pm ****DH private exponent size is 1016****
2-1-2006 6:30:53 pm Local server's interfaces : [VPN Private IP]
2-1-2006 6:30:53 pm Local server's interfaces : [VPN IP]
2-1-2006 6:30:53 pm Recieved Supported Vendor id Novell Border Manager VPN
4.0 client - Protected Net from [Client IP]
2-1-2006 6:30:53 pm Recieved Supported Vendor id
draft-ietf-ipsec-nat-t-ike-03 from [Client IP]
2-1-2006 6:30:53 pm ***Send Main Mode message to [Client IP]
2-1-2006 6:30:53 pm
I-COOKIE=5C77B3216A4259AC,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=SA-PAYLOAD,state=-871929040
2-1-2006 6:30:54 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:30:54 pm
I-COOKIE=5C77B3216A4259AC,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=KEY-PAYLOAD,state=-871928988
2-1-2006 6:30:54 pm There is NAT in between server and client
2-1-2006 6:30:54 pm info: sending certificate request payload is disabled
2-1-2006 6:30:54 pm ***Send Main Mode message to [Client IP]
2-1-2006 6:30:54 pm
I-COOKIE=5C77B3216A4259AC,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=KEY-PAYLOAD,state=-871928988
2-1-2006 6:30:54 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:30:54 pm
I-COOKIE=5C77B3216A4259AC,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=ID-PAYLOAD,state=-871928976
2-1-2006 6:30:54 pm Recieved MM ID payload type 3 protocol 0 portnum 0
length 72
2-1-2006 6:30:54 pm Recieved notify message type 24578 from [Client IP]
2-1-2006 6:30:54 pm Adding user :original address is [Client IP]
2-1-2006 6:30:54 pm Adding user :Nat inbetween Nat address is [Client IP]
2-1-2006 6:30:54 pm Adding user :Nat address is [Client IP]
2-1-2006 6:30:54 pm
Client 192.168.3.6 is added successfully
2-1-2006 6:30:54 pm *Sending MM id payload Type 9 - subject name :9
subject alternative name :2,3
2-1-2006 6:30:54 pm *protocol 0 portnum 0 length 59
2-1-2006 6:30:54 pm ***Send Main Mode message to [Client IP]
2-1-2006 6:30:54 pm
I-COOKIE=5C77B3216A4259AC,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=ID-PAYLOAD,state=-871928976
2-1-2006 6:30:58 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:30:58 pm
I-COOKIE=5C77B3216A4259AC,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=ID-PAYLOAD,state=-871928976
2-1-2006 6:30:58 pm Invalid payload length - ID-PAYLOAD payload
2-1-2006 6:30:58 pm Processed ID-PAYLOAD unsuccessful - Received the
message in the wrong state. Lost our reply, dst=[Client IP].
2-1-2006 6:30:58 pm Failed to create IKE-SA - Received the message in the
wrong state. Lost our reply , dst = [Client IP]
2-1-2006 6:31:05 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:31:05 pm
I-COOKIE=5C77B3216A4259AC,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=ID-PAYLOAD,state=-871928976
2-1-2006 6:31:05 pm Invalid payload length - ID-PAYLOAD payload
2-1-2006 6:31:05 pm Processed ID-PAYLOAD unsuccessful - Received the
message in the wrong state. Lost our reply, dst=[Client IP].
2-1-2006 6:31:05 pm Failed to create IKE-SA - Received the message in the
wrong state. Lost our reply , dst = [Client IP]
2-1-2006 6:31:09 pm Final IKE (phase 1) SA lifetime is 28800 secs
2-1-2006 6:31:09 pm IKE-SA is created. rekey time = 21600
encr=5,hash=2,auth=3,lifesec=28800
2-1-2006 6:31:09 pm dst=[Client IP],time=1610980
2-1-2006 6:31:15 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:31:15 pm
I-COOKIE=5C77B3216A4259AC,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=ID-PAYLOAD,state=-871928976
2-1-2006 6:31:15 pm Invalid payload length - ID-PAYLOAD payload
2-1-2006 6:31:15 pm Processed ID-PAYLOAD unsuccessful - Received the
message in the wrong state. Lost our reply, dst=[Client IP].
2-1-2006 6:31:15 pm Failed to create IKE-SA - Received the message in the
wrong state. Lost our reply , dst = [Client IP]
2-1-2006 6:31:32 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:31:32 pm
I-COOKIE=472D73E5DBFDA1EB,R-COOKIE=0000000000000000,MsgID=0,1stPL=SA-PAYLOAD,state=-871929040
2-1-2006 6:31:32 pm Start IKE-SA CDACDBA0 - Responder,src=[VPN
IP],dst=[Client IP],TotSA=7
2-1-2006 6:31:32 pm AUTH ALG IS 3
2-1-2006 6:31:32 pm IKE SA NEGOTIATION: Peer lifetime = 28800 My
lifetime=28800
2-1-2006 6:31:32 pm ****DH private exponent size is 1016****
2-1-2006 6:31:32 pm Local server's interfaces : [VPN Private IP]
2-1-2006 6:31:32 pm Local server's interfaces : [VPN IP]
2-1-2006 6:31:32 pm Recieved Supported Vendor id Novell Border Manager VPN
4.0 client - Protected Net from [Client IP]
2-1-2006 6:31:32 pm Recieved Supported Vendor id
draft-ietf-ipsec-nat-t-ike-03 from [Client IP]
2-1-2006 6:31:32 pm ***Send Main Mode message to [Client IP]
2-1-2006 6:31:32 pm
I-COOKIE=472D73E5DBFDA1EB,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=SA-PAYLOAD,state=-871929040
2-1-2006 6:31:32 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:31:32 pm
I-COOKIE=472D73E5DBFDA1EB,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=KEY-PAYLOAD,state=-871928988
2-1-2006 6:31:32 pm There is NAT in between server and client
2-1-2006 6:31:32 pm info: sending certificate request payload is disabled
2-1-2006 6:31:32 pm ***Send Main Mode message to [Client IP]
2-1-2006 6:31:32 pm
I-COOKIE=472D73E5DBFDA1EB,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=KEY-PAYLOAD,state=-871928988
2-1-2006 6:31:33 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:31:33 pm
I-COOKIE=472D73E5DBFDA1EB,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=ID-PAYLOAD,state=-871928976
2-1-2006 6:31:33 pm Recieved MM ID payload type 3 protocol 0 portnum 0
length 72
2-1-2006 6:31:33 pm Recieved notify message type 24578 from [Client IP]
2-1-2006 6:31:33 pm Adding user :original address is [Client IP]
2-1-2006 6:31:33 pm Adding user :Nat inbetween Nat address is [Client IP]
2-1-2006 6:31:33 pm Adding user :Nat address is [Client IP]
2-1-2006 6:31:33 pm
Client 192.168.3.7 is added successfully
2-1-2006 6:31:33 pm *Sending MM id payload Type 9 - subject name :9
subject alternative name :2,3
2-1-2006 6:31:33 pm *protocol 0 portnum 0 length 59
2-1-2006 6:31:33 pm ***Send Main Mode message to [Client IP]
2-1-2006 6:31:33 pm
I-COOKIE=472D73E5DBFDA1EB,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=ID-PAYLOAD,state=-871928976
2-1-2006 6:31:37 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:31:37 pm
I-COOKIE=472D73E5DBFDA1EB,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=ID-PAYLOAD,state=-871928976
2-1-2006 6:31:37 pm Invalid payload length - ID-PAYLOAD payload
2-1-2006 6:31:37 pm Processed ID-PAYLOAD unsuccessful - Received the
message in the wrong state. Lost our reply, dst=[Client IP].
2-1-2006 6:31:37 pm Failed to create IKE-SA - Received the message in the
wrong state. Lost our reply , dst = [Client IP]
2-1-2006 6:31:44 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:31:44 pm
I-COOKIE=472D73E5DBFDA1EB,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=ID-PAYLOAD,state=-871928976
2-1-2006 6:31:44 pm Invalid payload length - ID-PAYLOAD payload
2-1-2006 6:31:44 pm Processed ID-PAYLOAD unsuccessful - Received the
message in the wrong state. Lost our reply, dst=[Client IP].
2-1-2006 6:31:44 pm Failed to create IKE-SA - Received the message in the
wrong state. Lost our reply , dst = [Client IP]
2-1-2006 6:31:47 pm Final IKE (phase 1) SA lifetime is 28800 secs
2-1-2006 6:31:47 pm IKE-SA is created. rekey time = 21600
encr=5,hash=2,auth=3,lifesec=28800
2-1-2006 6:31:47 pm dst=[Client IP],time=1611664
2-1-2006 6:31:54 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:31:54 pm
I-COOKIE=472D73E5DBFDA1EB,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=ID-PAYLOAD,state=-871928976
2-1-2006 6:31:54 pm Invalid payload length - ID-PAYLOAD payload
2-1-2006 6:31:54 pm Processed ID-PAYLOAD unsuccessful - Received the
message in the wrong state. Lost our reply, dst=[Client IP].
2-1-2006 6:31:54 pm Failed to create IKE-SA - Received the message in the
wrong state. Lost our reply , dst = [Client IP]
2-1-2006 6:32:12 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:32:12 pm
I-COOKIE=0ED68683337B269D,R-COOKIE=0000000000000000,MsgID=0,1stPL=SA-PAYLOAD,state=-871929040
2-1-2006 6:32:12 pm Start IKE-SA CDACE260 - Responder,src=[VPN
IP],dst=[Client IP],TotSA=8
2-1-2006 6:32:12 pm AUTH ALG IS 3
2-1-2006 6:32:12 pm IKE SA NEGOTIATION: Peer lifetime = 28800 My
lifetime=28800
2-1-2006 6:32:12 pm ****DH private exponent size is 1016****
2-1-2006 6:32:12 pm Local server's interfaces : [VPN Private IP]
2-1-2006 6:32:12 pm Local server's interfaces : [VPN IP]
2-1-2006 6:32:12 pm Recieved Supported Vendor id Novell Border Manager VPN
4.0 client - Protected Net from [Client IP]
2-1-2006 6:32:12 pm Recieved Supported Vendor id
draft-ietf-ipsec-nat-t-ike-03 from [Client IP]
2-1-2006 6:32:12 pm ***Send Main Mode message to [Client IP]
2-1-2006 6:32:12 pm
I-COOKIE=0ED68683337B269D,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=SA-PAYLOAD,state=-871929040
2-1-2006 6:32:13 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:32:13 pm
I-COOKIE=0ED68683337B269D,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=KEY-PAYLOAD,state=-871928988
2-1-2006 6:32:13 pm There is NAT in between server and client
2-1-2006 6:32:13 pm info: sending certificate request payload is disabled
2-1-2006 6:32:13 pm ***Send Main Mode message to [Client IP]
2-1-2006 6:32:13 pm
I-COOKIE=0ED68683337B269D,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=KEY-PAYLOAD,state=-871928988
2-1-2006 6:32:13 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:32:13 pm
I-COOKIE=0ED68683337B269D,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=ID-PAYLOAD,state=-871928976
2-1-2006 6:32:13 pm Recieved MM ID payload type 3 protocol 0 portnum 0
length 72
2-1-2006 6:32:13 pm Recieved notify message type 24578 from [Client IP]
2-1-2006 6:32:13 pm Adding user :original address is [Client IP]
2-1-2006 6:32:13 pm Adding user :Nat inbetween Nat address is [Client IP]
2-1-2006 6:32:13 pm Adding user :Nat address is [Client IP]
2-1-2006 6:32:13 pm
Client 192.168.3.8 is added successfully
2-1-2006 6:32:13 pm *Sending MM id payload Type 9 - subject name :9
subject alternative name :2,3
2-1-2006 6:32:13 pm *protocol 0 portnum 0 length 59
2-1-2006 6:32:13 pm ***Send Main Mode message to [Client IP]
2-1-2006 6:32:13 pm
I-COOKIE=0ED68683337B269D,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=ID-PAYLOAD,state=-871928976
2-1-2006 6:32:17 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:32:17 pm
I-COOKIE=0ED68683337B269D,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=ID-PAYLOAD,state=-871928976
2-1-2006 6:32:17 pm Invalid payload length - ID-PAYLOAD payload
2-1-2006 6:32:17 pm Processed ID-PAYLOAD unsuccessful - Received the
message in the wrong state. Lost our reply, dst=[Client IP].
2-1-2006 6:32:17 pm Failed to create IKE-SA - Received the message in the
wrong state. Lost our reply , dst = [Client IP]
2-1-2006 6:32:24 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:32:24 pm
I-COOKIE=0ED68683337B269D,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=ID-PAYLOAD,state=-871928976
2-1-2006 6:32:24 pm Invalid payload length - ID-PAYLOAD payload
2-1-2006 6:32:24 pm Processed ID-PAYLOAD unsuccessful - Received the
message in the wrong state. Lost our reply, dst=[Client IP].
2-1-2006 6:32:24 pm Failed to create IKE-SA - Received the message in the
wrong state. Lost our reply , dst = [Client IP]
2-1-2006 6:32:28 pm Final IKE (phase 1) SA lifetime is 28800 secs
2-1-2006 6:32:28 pm IKE-SA is created. rekey time = 21600
encr=5,hash=2,auth=3,lifesec=28800
2-1-2006 6:32:28 pm dst=[Client IP],time=1612420
2-1-2006 6:32:35 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:32:35 pm
I-COOKIE=0ED68683337B269D,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=ID-PAYLOAD,state=-871928976
2-1-2006 6:32:35 pm Invalid payload length - ID-PAYLOAD payload
2-1-2006 6:32:35 pm Processed ID-PAYLOAD unsuccessful - Received the
message in the wrong state. Lost our reply, dst=[Client IP].
2-1-2006 6:32:35 pm Failed to create IKE-SA - Received the message in the
wrong state. Lost our reply , dst = [Client IP]
2-1-2006 6:32:53 pm ***Receive Main Mode message from [Client IP]
2-1-2006 6:32:53 pm
I-COOKIE=818645D710930781,R-COOKIE=0000000000000000,MsgID=0,1stPL=SA-PAYLOAD,state=-871929040
2-1-2006 6:32:53 pm Start IKE-SA CDACE920 - Responder,src=[VPN
IP],dst=[Client IP],TotSA=9
2-1-2006 6:32:53 pm AUTH ALG IS 3
2-1-2006 6:32:53 pm IKE SA NEGOTIATION: Peer lifetime = 28800 My
lifetime=28800
2-1-2006 6:32:53 pm ****DH private exponent size is 1016****
2-1-2006 6:32:53 pm Local server's interfaces : [VPN Private IP]
2-1-2006 6:32:53 pm Local server's interfaces : [VPN IP]
2-1-2006 6:32:53 pm Recieved Supported Vendor id Novell Border Manager VPN
4.0 client - Protected Net from [Client IP]
2-1-2006 6:32:53 pm Recieved Supported Vendor id
draft-ietf-ipsec-nat-t-ike-03 from [Client IP]
2-1-2006 6:32:53 pm ***Send Main Mode message to [Client IP]
2-1-2006 6:32:53 pm
I-COOKIE=818645D710930781,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=SA-PAYLOAD,state=-871929040
2-1-2006 6:32:58 pm Retransmit timer expired :Peer lost our reply
retransmit the old packet to [Client IP]
2-1-2006 6:32:58 pm ***Send Main Mode message to [Client IP]
2-1-2006 6:32:58 pm
I-COOKIE=818645D710930781,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=SA-PAYLOAD,state=-871929040
2-1-2006 6:33:05 pm Retransmit timer expired :Peer lost our reply
retransmit the old packet to [Client IP]
2-1-2006 6:33:05 pm ***Send Main Mode message to [Client IP]
2-1-2006 6:33:05 pm
I-COOKIE=818645D710930781,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=SA-PAYLOAD,state=-871929040
2-1-2006 6:33:15 pm Retransmit timer expired :Peer lost our reply
retransmit the old packet to [Client IP]
2-1-2006 6:33:15 pm ***Send Main Mode message to [Client IP]
2-1-2006 6:33:15 pm
I-COOKIE=818645D710930781,R-COOKIE=7D08C1C4AA6F260B,MsgID=0,1stPL=SA-PAYLOAD,state=-871929040
2-1-2006 6:33:29 pm Retransmit timer expired :Peer lost our reply
retransmit the old packet to [Client IP]
2-1-2006 6:33:29 pm IKE-SA is deleted- packet retransmit exceeded the
limit, dst=[Client IP]
2-1-2006 6:33:32 pm The client [Client IP] removed from vpninf