Netware 6.5 SP5+ replica server running in an MPLS WAN environment. Both of our replica servers keep running low/out of logical address space and we can't track down what's causing it. The replica server I'll provide files on holds the master replica of all of our other Novell servers. The replica servers also act as our SLPDA servers and LDAP servers. We know when one of the replica servers starts running low because our Helpdesk starts getting calls that people across the WAN links can't login to their local Novell servers (even though the remote Novell servers hold their own container's r/w replica), or they start logging into the replica servers intead of their own servers, or suddenly NAL icons start disappearing from their desktops (?). Also, LDAP starts failing within our Corporate LDAP applications. We have MLA licensing, located in each site's container where their ids/servers are located.

I have included the latest segstats.txt for review - one when the server is initially booted up, and the other when it's about to require rebooting.

It looks like the DS.NLM and DSLOADER.NLM files keep growing. The logical address space gets down to about 250 Meg, when the server becomes completely unusable and we have to reboot it.

I changed the cachesize to 16384 in the bti.cfg file and restarted the server, but for some reason it's still grabbing about 89 meg at startup. Not sure why, as we've made that change to all other servers and the nwmkde.nlm only took what we assigned in this file.

Let me know if you see anything obvious. I did see that the 'hostname' file's server name entry was wrong and I have fixed that since running the included config.txt report.

When I was about to send this post, I took a quick look at the stats again via Remote Manager's Health page. For some reason, various ids (random) are attempting to login to the replica servers instead of their own local servers. Not a huge amount of ids, but I don't understand why this is happening, since we push their preferred server and such out via DHCP, and we have secondary SLP scopes setup. Many remote Centers have also stated that if a PC is turned off and then turned back on, it will attempt to login to the replica servers instead of their local Novell server. Just throwing that out there - not sure there is any direct correlation with our logical address space issues on the replica servers. This logical address space problem doesn't happen on any of the other Novell servers on the network - about 55 of them - just the 2 replica servers.

Okay - another update. The 'Fragmented Kernal Space' keeps growing and growing. It started at about 235 Meg and after 1/2 hour is now at 943 Meg and growing. I also see that the ds.nlm started at 88.7 Meg and is at 370 Meg and growing. Looks like the NSS.NLM is steadily growing too. It started at 75.9 Meg and is currently at 136 Meg and growing. The other 'red' nlms are: connmgr, dsloader and ncpip. Looks like I'll have to reboot it soon again.

While writing this, the other replica server did the same thing. Dropped to like 245 Meg of available logical space. I got a call saying one of our apps that uses LDAP wasn't working again. Center users were starting to complain about an inability to login again. I just had to reboot it. It exhibits basically the same problems. Maybe the problem is that the rep servers' nds databases are too large and I need to limit how much they can grow ??

I posted the config.txt file separately, as the post was too large when I included it.

Thanks!

April



First segstats.txt


*** Memory Pool Configuration for : PA_REPLICA2
Time and date : 02:48:14 PM 06/11/2007
Server version : NetWare 6.5 Support Pack 5
Server uptime : 0d 00h 13m 17s
SEG.NLM uptime : 0d 00h 00m 10s
SEG.NLM version : v2.00.17

Original Memory : 4,227,403,776 bytes (3.94 GB)
ESM Memory : 469,757,952 bytes (448.0 MB)
0xFFFFFFFF --------------------------------------------------------------
| Kernel Reserved Space |
| |
| Size : 180,355,071 bytes (172.0 MB) |
| |
0xF5400000 --------------------------------------------------------------
| User Address Space (L!=P) |
| |
| User Pool Size : 884,998,144 bytes (844.0 MB) |
| High Water Mark : 566,394,880 bytes (540.2 MB) |
| PM Pages In Use : 11,264,000 bytes (10.7 MB) |
| |
0xC0800000 --------------------------------------------------------------
| Virtual Memory Address Space (L!=P) |
| |
| VM Address Space : 2,155,872,256 bytes (2.01 GB) |
| Available : 1,184,231,424 bytes (1.10 GB) |
| Total VM Pages : 1,167,650,816 bytes (1.09 GB) |
| Free Clean VM : 1,163,554,816 bytes (1.08 GB) |
| Free Cache VM : 4,096,000 bytes (3.9 MB) |
| Total LP Pages : 0 bytes (0 KB) |
| Free Clean LP : 0 bytes (0 KB) |
| Free Cache LP : 0 bytes (0 KB) |
| Free Dirty : 0 bytes (0 KB) |
| NLM Memory In Use : 450,535,424 bytes (429.7 MB) |
| NLM/VM Memory : 458,407,936 bytes (437.2 MB) |
| Largest Segment : 42,766,336 bytes (40.8 MB) |
| Lowest Kernel Page: 2,147,483,648 bytes (2.00 GB) |
| : [0x80000000] |
| High Water Mark : 496,443,392 bytes (473.4 MB) |
| Alloc Failures : 0 |
| |
0x40000000 --------------------------------------------------------------
| File System Address Space (L==P or L!=P) |
| |
| FS Address Space : 1,067,458,560 bytes (1018.0 MB) |
| Available : 299,073,536 bytes (285.2 MB) |
| Largest Segment : 290,324,480 bytes (276.9 MB) |
| |
| NSS Memory (85%) : 1,684,680,704 bytes (1.57 GB) |
| NSS (avail cache) : 1,681,698,816 bytes (1.57 GB) |
| |
0x005FE000 --------------------------------------------------------------
| DOS / SERVER.NLM |
| |
| Size : 6,283,264 bytes (6.0 MB) |
| |
0x00000000 --------------------------------------------------------------


Total NLMs loaded on the server: 256



Top 20 Memory Consuming NLMs
NLM Name Version Date Total NLM Memory
================================================== ===============================
1. NWMKDE.NLM 8.50 Oct 29, 2003 93,641,874 bytes (89.3 MB)
2. DS.NLM 10552.79 Aug 9, 2005 86,726,414 bytes (82.7 MB)
3. NSS.NLM 3.24.01 Mar 10, 2006 79,557,408 bytes (75.9 MB)
4. SERVER.NLM 5.70.05 Apr 11, 2006 78,810,458 bytes (75.2 MB)
5. APACHE2.NLM 2.00.54 Oct 31, 2005 16,611,813 bytes (15.8 MB)
6. FATFS.NLM 1.24 Dec 7, 2005 5,714,391 bytes (5.4 MB)
7. PORTAL.NLM 4.01 Dec 8, 2005 3,246,793 bytes (3.1 MB)
8. CIOS.NLM 1.50 Nov 29, 2005 3,130,496 bytes (3.0 MB)
9. NWPA.NLM 3.20.08 Nov 16, 2005 2,982,806 bytes (2.8 MB)
10. CONNMGR.NLM 5.60.01 Oct 26, 2005 2,700,307 bytes (2.6 MB)
11. JVM.NLM 1.42.06 Dec 1, 2005 2,681,823 bytes (2.6 MB)
12. LIB0.NLM 5.90.12 Nov 9, 2005 2,454,330 bytes (2.3 MB)
13. LIBC.NLM 9.00.03 Dec 15, 2005 2,441,854 bytes (2.3 MB)
14. NLDAP.NLM 10554.24 Aug 8, 2005 2,350,092 bytes (2.2 MB)
15. CDBE.NLM 6.01 Sep 2, 2005 2,303,632 bytes (2.2 MB)
16. NDSIMON.NLM 20210.67 Jul 19, 2005 2,209,353 bytes (2.1 MB)
17. NWENC103.NLM 8.50 Oct 29, 2003 2,126,589 bytes (2.0 MB)
18. APRLIB.NLM 0.09.06 Oct 31, 2005 2,053,136 bytes (2.0 MB)
19. NETLIB.NLM 6.50.22 Feb 12, 2003 1,520,060 bytes (1.4 MB)
20. TCP.NLM 6.63.01 Oct 5, 2005 1,486,043 bytes (1.4 MB)






Top 20 NLM - Memory Trends
NLM Name Original Memory Current Change
================================================== ===============================
1. NWMKDE.NLM 93,641,874 bytes 93,641,874 bytes 0 KB
2. DS.NLM 93,017,870 bytes 86,726,414 bytes -6.0 MB
3. NSS.NLM 79,553,312 bytes 79,557,408 bytes 4 KB
4. SERVER.NLM 78,728,538 bytes 78,810,458 bytes 80 KB
5. APACHE2.NLM 16,611,813 bytes 16,611,813 bytes 0 KB
6. FATFS.NLM 5,714,391 bytes 5,714,391 bytes 0 KB
7. PORTAL.NLM 3,246,793 bytes 3,246,793 bytes 0 KB
8. CIOS.NLM 3,130,496 bytes 3,130,496 bytes 0 KB
9. NWPA.NLM 2,982,806 bytes 2,982,806 bytes 0 KB
10. CONNMGR.NLM 2,696,211 bytes 2,700,307 bytes 4 KB
11. JVM.NLM 2,681,823 bytes 2,681,823 bytes 0 KB
12. LIB0.NLM 2,454,330 bytes 2,454,330 bytes 0 KB
13. LIBC.NLM 2,441,854 bytes 2,441,854 bytes 0 KB
14. NLDAP.NLM 2,350,092 bytes 2,350,092 bytes 0 KB
15. CDBE.NLM 2,303,632 bytes 2,303,632 bytes 0 KB
16. NDSIMON.NLM 2,209,353 bytes 2,209,353 bytes 0 KB
17. NWENC103.NLM 2,126,589 bytes 2,126,589 bytes 0 KB
18. APRLIB.NLM 2,053,136 bytes 2,053,136 bytes 0 KB
19. NETLIB.NLM 1,520,060 bytes 1,520,060 bytes 0 KB
20. TCP.NLM 1,486,043 bytes 1,486,043 bytes 0 KB




Logical Memory Summary Information
================================================== ===============================

File System Cache Information
FS Cache Free : 291,540,992 bytes (278.0 MB)
FS Cache Fragmented : 7,532,544 bytes (7.2 MB)
FS Cache Largest Segment : 290,324,480 bytes (276.9 MB)

Logical System Cache Information
LS Cache Free : 88,977,408 bytes (84.9 MB)
LS Cache Fragmented : 410,267,648 bytes (391.3 MB)
LS OS Reserved Data : 333,455,360 bytes (318.0 MB)
LS Cache Largest Segment : 42,766,336 bytes (40.8 MB)
LS Cache Largest Position : A5DB0000

Summary Statistics
Total Address Space : 4,294,967,296 bytes (4.00 GB)
Total Free : 1,454,260,224 bytes (1.35 GB)
Total Fragmented : 417,800,192 bytes (398.4 MB)
Highest Physical Address : DFFF2000
User Space : 884,998,144 bytes (844.0 MB)
User Space (High Water Mark) : 566,394,880 bytes (540.2 MB)
NLM Memory (High Water Mark) : 496,443,392 bytes (473.4 MB)
Kernel Address Space In Use : 2,270,134,272 bytes (2.11 GB)
Available Kernel Address Space : 541,679,616 bytes (516.6 MB)


Memory Summary Screen (.ms)
================================================== ===============================

KNOWN MEMORY Bytes Pages Bytes Pages
Server: 3757575216 917376 Video: 8192 2
Dos: 71632 17 Other: 131072 32

FS CACHE KERNEL NLM MEMORY
Original: 3753439232 916367 Code: 35889152 8762
Current: 299073536 73016 Data: 22417408 5473
Dirty: 0 0 Sh Code: 40960 10
Largest seg: 290324480 70880 Sh Data: 20480 5
Non-Movable: 12288 3 Help: 139264 34
Other: 1810993152 442137 Message: 1073152 262
Avail NSS: 1681698816 410571 Alloc L!=P: 393146368 95983
Movable: 8192 2 Alloc L==P: 5681152 1387
Total: 458407936 111916
VM SYSTEM
Free clean VM: 1163554816 284071
Free clean LP: 0 0
Free cache VM: 4096000 1000
Free cache LP: 0 0
Free dirty: 0 0
In use: 11264000 2750
Total: 1184231424 289119





Memory Configuration (set parameters)
================================================== ==============================

Auto Tune Server Memory = ON
File Cache Maximum Size = 1073741825
File Service Memory Optimization = 1
Logical Space Compression = 1
Garbage Collection Interval (ON) = 299.9 seconds
VM Garbage Collector Period (ON) = 300.0 seconds
server -u<number> = 884998144



NSS Configuration File:
C:\NWSERVER\NSSSTART.CFG
------------------------------
File does not exist,
or is zero byte in size.



DS Configuration File:
SYS:\_NETWARE\_NDSDB.INI
------------------------------
File does not exist,
or is zero byte in size.




TSAFS Memory Information/Configuration
================================================== ==============================

Cache Memory Threshold : 10%
Read Buffer Size : 65536 bytes
Max Data Sets for Read Ahead : 2
Read Threads Per Job : 4




NSS Memory Information/Configuration
================================================== ==============================

Current NSS Memory Settings
-------------------------------------------
Cache Balance Percentage : 85%
Cache Memory Allocated : 1.57 GB
Available Cache from NSS : 1.57 GB


Current NSS Caching Percentages
-------------------------------------------
Buffer cache hit percentage : 92%
Name Tree cache hit percentage : 97%
File cache hit percentage : 99%

NSS Flush Status: Not Flushed




Server High/Low Water Mark Values
================================================== ==============================

NLM Memory High Water Mark = 496,443,392 bytes
File System High Water Mark = 439,632 bytes

User Space Information:
User Space High Water Mark = 566,394,880 bytes
Committed Pages High Water Mark = 0 pages
Mapped VM Pages High Water Mark = 5,047 pages
Reserved Pages High Water Mark = 330,045 pages
Swapped Pages High Water Mark = 1,000 pages

Available Low Water Mark = 1,455,755,264
ESM Memory High Water Mark = 104 pages



SECOND SEGSTATS.TXT


*** Memory Pool Configuration for : PA_REPLICA2
Time and date : 05:04:20 PM 06/11/2007
Server version : NetWare 6.5 Support Pack 5
Server uptime : 0d 02h 29m 24s
SEG.NLM uptime : 0d 02h 16m 16s
SEG.NLM version : v2.00.17

Original Memory : 4,227,403,776 bytes (3.94 GB)
ESM Memory : 469,757,952 bytes (448.0 MB)
0xFFFFFFFF --------------------------------------------------------------
| Kernel Reserved Space |
| |
| Size : 180,355,071 bytes (172.0 MB) |
| |
0xF5400000 --------------------------------------------------------------
| User Address Space (L!=P) |
| |
| User Pool Size : 884,998,144 bytes (844.0 MB) |
| High Water Mark : 566,394,880 bytes (540.2 MB) |
| PM Pages In Use : 0 bytes (0 KB) |
| |
0xC0800000 --------------------------------------------------------------
| Virtual Memory Address Space (L!=P) |
| |
| VM Address Space : 2,155,872,256 bytes (2.01 GB) |
| Available : 1,766,899,712 bytes (1.65 GB) |
| Total VM Pages : 1,765,773,312 bytes (1.64 GB) |
| Free Clean VM : 1,742,848,000 bytes (1.62 GB) |
| Free Cache VM : 22,925,312 bytes (21.9 MB) |
| Total LP Pages : 0 bytes (0 KB) |
| Free Clean LP : 0 bytes (0 KB) |
| Free Cache LP : 0 bytes (0 KB) |
| Free Dirty : 0 bytes (0 KB) |
| NLM Memory In Use : 835,129,344 bytes (796.4 MB) |
| NLM/VM Memory : 832,876,544 bytes (794.3 MB) |
| Largest Segment : 15,101,952 bytes (14.4 MB) |
| Lowest Kernel Page: 1,073,741,824 bytes (1024.0 MB) |
| : [0x40000000] |
| High Water Mark : 905,990,144 bytes (864.0 MB) |
| Alloc Failures : 0 |
| |
0x40000000 --------------------------------------------------------------
| File System Address Space (L==P or L!=P) |
| |
| FS Address Space : 1,067,458,560 bytes (1018.0 MB) |
| Available : 153,456,640 bytes (146.3 MB) |
| Largest Segment : 146,382,848 bytes (139.6 MB) |
| |
| NSS Memory (85%) : 864,825,344 bytes (824.8 MB) |
| NSS (avail cache) : 861,564,928 bytes (821.7 MB) |
| |
0x005FE000 --------------------------------------------------------------
| DOS / SERVER.NLM |
| |
| Size : 6,283,264 bytes (6.0 MB) |
| |
0x00000000 --------------------------------------------------------------


Total NLMs loaded on the server: 256



Top 20 Memory Consuming NLMs
NLM Name Version Date Total NLM Memory
================================================== ===============================
1. DS.NLM 10552.79 Aug 9, 2005 398,059,278 bytes (379.6 MB)
2. NSS.NLM 3.24.01 Mar 10, 2006 143,565,600 bytes (136.9 MB)
3. NWMKDE.NLM 8.50 Oct 29, 2003 93,731,986 bytes (89.4 MB)
4. SERVER.NLM 5.70.05 Apr 11, 2006 80,403,802 bytes (76.7 MB)
5. APACHE2.NLM 2.00.54 Oct 31, 2005 16,611,813 bytes (15.8 MB)
6. CONNMGR.NLM 5.60.01 Oct 26, 2005 6,542,355 bytes (6.2 MB)
7. FATFS.NLM 1.24 Dec 7, 2005 5,714,391 bytes (5.4 MB)
8. PORTAL.NLM 4.01 Dec 8, 2005 3,246,793 bytes (3.1 MB)
9. CIOS.NLM 1.50 Nov 29, 2005 3,130,496 bytes (3.0 MB)
10. NWPA.NLM 3.20.08 Nov 16, 2005 2,974,614 bytes (2.8 MB)
11. DSLOADER.NLM 10552.66 May 9, 2005 2,705,828 bytes (2.6 MB)
12. JVM.NLM 1.42.06 Dec 1, 2005 2,669,535 bytes (2.5 MB)
13. LIB0.NLM 5.90.12 Nov 9, 2005 2,454,330 bytes (2.3 MB)
14. LIBC.NLM 9.00.03 Dec 15, 2005 2,441,854 bytes (2.3 MB)
15. CDBE.NLM 6.01 Sep 2, 2005 2,303,632 bytes (2.2 MB)
16. NLDAP.NLM 10554.24 Aug 8, 2005 2,300,940 bytes (2.2 MB)
17. NCPIP.NLM 6.02 Nov 21, 2005 2,292,473 bytes (2.2 MB)
18. NDSIMON.NLM 20210.67 Jul 19, 2005 2,209,353 bytes (2.1 MB)
19. NWENC103.NLM 8.50 Oct 29, 2003 2,126,589 bytes (2.0 MB)
20. APRLIB.NLM 0.09.06 Oct 31, 2005 2,053,136 bytes (2.0 MB)






Top 20 NLM - Memory Trends
NLM Name Original Memory Current Change
================================================== ===============================
1. DS.NLM 93,017,870 bytes 398,059,278 bytes 290.9 MB
2. NSS.NLM 79,553,312 bytes 143,565,600 bytes 61.0 MB
3. NWMKDE.NLM 93,641,874 bytes 93,731,986 bytes 88 KB
4. SERVER.NLM 78,728,538 bytes 80,403,802 bytes 1.6 MB
5. APACHE2.NLM 16,611,813 bytes 16,611,813 bytes 0 KB
6. CONNMGR.NLM 2,696,211 bytes 6,542,355 bytes 3.7 MB
7. FATFS.NLM 5,714,391 bytes 5,714,391 bytes 0 KB
8. PORTAL.NLM 3,246,793 bytes 3,246,793 bytes 0 KB
9. CIOS.NLM 3,130,496 bytes 3,130,496 bytes 0 KB
10. NWPA.NLM 2,982,806 bytes 2,974,614 bytes -8 KB
11. DSLOADER.NLM 1,292,708 bytes 2,705,828 bytes 1.3 MB
12. JVM.NLM 2,681,823 bytes 2,669,535 bytes -12 KB
13. LIB0.NLM 2,454,330 bytes 2,454,330 bytes 0 KB
14. LIBC.NLM 2,441,854 bytes 2,441,854 bytes 0 KB
15. CDBE.NLM 2,303,632 bytes 2,303,632 bytes 0 KB
16. NLDAP.NLM 2,350,092 bytes 2,300,940 bytes -48 KB
17. NCPIP.NLM 1,141,497 bytes 2,292,473 bytes 1.1 MB
18. NDSIMON.NLM 2,209,353 bytes 2,209,353 bytes 0 KB
19. NWENC103.NLM 2,126,589 bytes 2,126,589 bytes 0 KB
20. APRLIB.NLM 2,053,136 bytes 2,053,136 bytes 0 KB




Logical Memory Summary Information
================================================== ===============================

File System Cache Information
FS Cache Free : 147,599,360 bytes (140.8 MB)
FS Cache Fragmented : 5,857,280 bytes (5.6 MB)
FS Cache Largest Segment : 146,382,848 bytes (139.6 MB)

Logical System Cache Information
LS Cache Free : 146,452,480 bytes (139.7 MB)
LS Cache Fragmented : 968,232,960 bytes (923.4 MB)
LS OS Reserved Data : 333,455,360 bytes (318.0 MB)
LS Cache Largest Segment : 15,101,952 bytes (14.4 MB)
LS Cache Largest Position : 77FD0000

Summary Statistics
Total Address Space : 4,294,967,296 bytes (4.00 GB)
Total Free : 294,051,840 bytes (280.4 MB)
Total Fragmented : 974,090,240 bytes (929.0 MB)
Highest Physical Address : DFFF2000
User Space : 884,998,144 bytes (844.0 MB)
User Space (High Water Mark) : 566,394,880 bytes (540.2 MB)
NLM Memory (High Water Mark) : 905,990,144 bytes (864.0 MB)
Kernel Address Space In Use : 1,833,082,880 bytes (1.71 GB)
Available Kernel Address Space : 422,440,960 bytes (402.9 MB)


Memory Summary Screen (.ms)
================================================== ===============================

KNOWN MEMORY Bytes Pages Bytes Pages
Server: 3757575216 917376 Video: 8192 2
Dos: 71632 17 Other: 131072 32

FS CACHE KERNEL NLM MEMORY
Original: 3753439232 916367 Code: 35889152 8762
Current: 153456640 37465 Data: 22417408 5473
Dirty: 0 0 Sh Code: 40960 10
Largest seg: 146382848 35738 Sh Data: 20480 5
Non-Movable: 12288 3 Help: 139264 34
Other: 994967552 242912 Message: 1073152 262
Avail NSS: 861564928 210343 Alloc L!=P: 767623168 187408
Movable: 8192 2 Alloc L==P: 5672960 1385
Total: 832876544 203339
VM SYSTEM
Free clean VM: 1742848000 425500
Free clean LP: 0 0
Free cache VM: 22925312 5597
Free cache LP: 0 0
Free dirty: 0 0
In use: 0 0
Total: 1766899712 431372





Memory Configuration (set parameters)
================================================== ==============================

Auto Tune Server Memory = ON
File Cache Maximum Size = 1073741825
File Service Memory Optimization = 1
Logical Space Compression = 1
Garbage Collection Interval (ON) = 299.9 seconds
VM Garbage Collector Period (ON) = 300.0 seconds
server -u<number> = 884998144



NSS Configuration File:
C:\NWSERVER\NSSSTART.CFG
------------------------------
File does not exist,
or is zero byte in size.



DS Configuration File:
SYS:\_NETWARE\_NDSDB.INI
------------------------------
File does not exist,
or is zero byte in size.




TSAFS Memory Information/Configuration
================================================== ==============================

Cache Memory Threshold : 10%
Read Buffer Size : 65536 bytes
Max Data Sets for Read Ahead : 2
Read Threads Per Job : 4




NSS Memory Information/Configuration
================================================== ==============================

Current NSS Memory Settings
-------------------------------------------
Cache Balance Percentage : 85%
Cache Memory Allocated : 824.8 MB
Available Cache from NSS : 821.7 MB


Current NSS Caching Percentages
-------------------------------------------
Buffer cache hit percentage : 96%
Name Tree cache hit percentage : 97%
File cache hit percentage : 97%

NSS Flush Status: Flushed




Server High/Low Water Mark Values
================================================== ==============================

NLM Memory High Water Mark = 905,990,144 bytes
File System High Water Mark = 439,632 bytes

User Space Information:
User Space High Water Mark = 566,394,880 bytes
Committed Pages High Water Mark = 0 pages
Mapped VM Pages High Water Mark = 5,047 pages
Reserved Pages High Water Mark = 330,045 pages
Swapped Pages High Water Mark = 5,743 pages

Available Low Water Mark = 1,455,755,264
ESM Memory High Water Mark = 104 pages