We just upgraded recently from nw65sp5. We only had minimal issue with
memory fragmentation.

Server has been up for 11 days.

Now, our largest segment size in File System Address space is only 8MB
which is leading to the following messages:

2-09-2009 9:05:06 am: SERVER-5.70-0 [nmID=2000A]
Short term memory allocator is out of memory.
1 attempts to get more memory failed.
request size in bytes 33554447 from Module ADS.NLM


2-09-2009 9:05:07 am: SERVER-5.70-0 [nmID=6001D]
Cache memory allocator out of available memory.

Our segstats.txt file is indicated below.

We have the following statement as the first item in startup.ncf:

set file cache maximum size=2147483648

and as the last item in autoexec.ncf

set file cache maximum size=2147483648

We changed our NSS cache balance from 60% to 85%. The system autotunes
FCMS down from 2GB to 1GB.

We upgraded our Netshield from version 4.63.01 to 4.63.08 (4.63h).

We dropped our bti.cfg setting for cache size from 131072 to 65536

CacheSize=65536

NWMKDE.NLM do change in size from its current 87MB to over 200MB when
the end of week full backups, versus daily incremental backups, are
done. It then reduces in size after a period of time.

The largest cache segment started out at about 195MB, then drops after a
little why, then stayed static. It then dropped to about 40MB and
stayed for several days. Over the weekend, from Sunday night to Monday,
morning, it dropped to 8MB.

NLMs that increase, then decrease in size.

NWMKDE, ADS.

What can be done to stabilize the memory usage or identify what may be
causing fragmentation that ultimately leads to memory allocation errors.

Cliff


>>>>>>>>>> segstats.txt

*** Memory Pool Configuration for : AFI_NW411
Time and date : 09:34:30 AM 02/09/2009
Server version : NetWare 6.5 Support Pack 8
Server uptime : 11d 03h 25m 54s
SEG.NLM uptime : 11d 03h 21m 49s
SEG.NLM version : v2.00.17

Original Memory : 4,294,336,512 bytes (4.00 GB)
ESM Memory : 805,302,272 bytes (768.0 MB)
0xFFFFFFFF -------------------------------------------------------------
| Kernel Reserved Space
| |
| Size : 180,355,071 bytes (172.0 MB) |
| |
0xF5400000 -------------------------------------------------------------
| User Address Space (L!=P)
|
| User Pool Size : 884,998,144 bytes (844.0 MB)
| High Water Mark : 297,218,048 bytes (283.4 MB)
| PM Pages In Use : 0 bytes (0 KB)
|
0xC0800000 -------------------------------------------------------------
| Virtual Memory Address Space (L!=P)
|
| VM Address Space : 2,155,872,256 bytes (2.01 GB)
| Available : 1,530,667,008 bytes (1.43 GB)
| Total VM Pages : 1,530,527,744 bytes (1.43 GB)
| Free Clean VM : 1,527,963,648 bytes (1.42 GB)
| Free Cache VM : 2,564,096 bytes (2.4 MB)
| Total LP Pages : 0 bytes (0 KB)
| Free Clean LP : 0 bytes (0 KB)
| Free Cache LP : 0 bytes (0 KB)
| Free Dirty : 0 bytes (0 KB)
| NLM Memory In Use : 947,785,728 bytes (903.9 MB)
| NLM/VM Memory : 930,754,560 bytes (887.6 MB)
| Largest Segment : 45,084,672 bytes (43.0 MB)
| Lowest Kernel Page: 0 bytes (0 KB)
| : [0x00000000]
| High Water Mark : 1,261,654,016 bytes (1.18 GB)
| Alloc Failures : 1
|
0x40000000 -------------------------------------------------------------
| File System Address Space (L==P or L!=P)
|
| FS Address Space : 1,067,343,872 bytes (1017.9 MB)
| Available : 122,183,680 bytes (116.5 MB)
| Largest Segment : 8,691,712 bytes (8.3 MB)
|
| NSS Memory (85%) : 688,496,640 bytes (656.6 MB)
| NSS (avail cache) : 680,005,632 bytes (648.5 MB)
|
0x0061A000 -------------------------------------------------------------
| DOS / SERVER.NLM
|
| Size : 6,397,952 bytes (6.1 MB)
|
0x00000000 -------------------------------------------------------------

Total NLMs loaded on the server: 318

Top 20 Memory Consuming NLMs
NLM Name Version Date Total NLM Memory
================================================== ===============================
1. NSS.NLM 3.27 Nov 13, 2008 205,536,336 bytes 196.0 MB)
2. ADS.NLM 8.10 Jan 25, 2007 192,400,300 bytes(183.5 MB)
3. NETSHLD.NLM 4.63.08 May 14, 2008 159,289,140 bytes(151.9 MB)
4. SERVER.NLM 5.70.08 Oct 3, 2008 110,878,304 bytes(105.7 MB)
5. NWMKDE.NLM 8.50 Oct 29, 2003 87,235,730 bytes (83.2 MB)
6. DS.NLM 10554.44 Aug 12, 2008 48,097,422 bytes (45.9 MB)
7. OWCIMOMD.NLM 3.02 Nov 27, 2007 9,318,616 bytes (8.9 MB)
8. APACHE2.NLM 2.00.63 Apr 25, 2008 7,702,469 bytes (7.3 MB)
9. CIOS.NLM 1.60 Feb 12, 2008 5,744,479 bytes (5.5 MB)
10. FATFS.NLM 1.24 Aug 27, 2007 5,716,053 bytes (5.5 MB)
11. CONNMGR.NLM 5.60.01 Sep 7, 2006 4,563,987 bytes (4.4 MB)
12. PKI.NLM 3.32 Aug 25, 2008 4,069,957 bytes (3.9 MB)
13. WS2_32.NLM 6.24.01 Feb 14, 2008 3,553,964 bytes (3.4 MB)
14. MCSCAN32.NLM 53.00 Jul 8, 2008 3,407,489 bytes (3.2 MB)
15. PORTAL.NLM 4.03 Sep 22, 2008 3,375,904 bytes (3.2 MB)
16. NWPA.NLM 3.21.02 Oct 29, 2008 3,319,518 bytes (3.2 MB)
17. LIBC.NLM 9.00.05 Oct 3, 2008 3,134,134 bytes (3.0 MB)
18. JVM.NLM 1.43 Oct 16, 2008 2,701,919 bytes (2.6 MB)
19. LIB0.NLM 5.90.15 Mar 10, 2008 2,462,042 bytes (2.3 MB)
20. CDBE.NLM 6.01 Sep 21, 2006 2,418,912 bytes (2.3 MB)

Top 20 NLM - Memory Trends
NLM Name Original Memory Current Change
================================================== ===============================
1. NSS.NLM 78,883,920 bytes 205,536,336 bytes 120.8 MB
2. ADS.NLM 68,520,876 bytes 192,400,300 bytes 118.1 MB
3. NETSHLD.NLM 45,666,100 bytes 159,289,140 bytes 108.4 MB
4. SERVER.NLM 101,854,816 bytes 110,878,304 bytes 8.6 MB
5. NWMKDE.NLM 72,490,130 bytes 87,235,730 bytes 14.1 MB
6. DS.NLM 48,576,654 bytes 48,097,422 bytes -468 KB
7. OWCIMOMD.NLM 9,326,808 bytes 9,318,616 bytes -8 KB
8. APACHE2.NLM 7,702,469 bytes 7,702,469 bytes 0 KB
9. CIOS.NLM 5,805,919 bytes 5,744,479 bytes -60 KB
10. FATFS.NLM 5,695,573 bytes 5,716,053 bytes 20 KB
11. CONNMGR.NLM 635,923 bytes 4,563,987 bytes 3.7 MB
12. PKI.NLM 4,442,693 bytes 4,069,957 bytes -364 KB
13. WS2_32.NLM 3,521,196 bytes 3,553,964 bytes 32 KB
14. MCSCAN32.NLM 3,407,489 bytes 3,407,489 bytes 0 KB
15. PORTAL.NLM 3,384,096 bytes 3,375,904 bytes -8 KB
16. NWPA.NLM 3,569,374 bytes 3,319,518 bytes -244 KB
17. LIBC.NLM 2,032,310 bytes 3,134,134 bytes 1.1 MB
18. JVM.NLM 2,706,015 bytes 2,701,919 bytes -4 KB
19. LIB0.NLM 2,466,138 bytes 2,462,042 bytes -4 KB
20. CDBE.NLM 2,423,008 bytes 2,418,912 bytes -4 KB

Logical Memory Summary Information
================================================== ===============================
File System Cache Information
FS Cache Free : 58,351,616 bytes (55.6 MB)
FS Cache Fragmented : 63,832,064 bytes (60.9 MB)
FS Cache Largest Segment : 8,691,712 bytes (8.3 MB)

Logical System Cache Information
LS Cache Free : 720,236,544 bytes (686.9 MB)
LS Cache Fragmented : 488,214,528 bytes (465.6 MB)
LS OS Reserved Data : 333,455,360 bytes (318.0 MB)
LS Cache Largest Segment : 45,084,672 bytes (43.0 MB)
LS Cache Largest Position : 33D7E000

Summary Statistics
Total Address Space : 4,294,967,296 bytes (4.00 GB)
Total Free : 778,588,160 bytes (742.5 MB)
Total Fragmented : 552,046,592 bytes (526.5 MB)
Highest Physical Address : CFFC7000
User Space : 884,998,144 bytes (844.0 MB)
User Space (High Water Mark) : 297,218,048 bytes (283.4 MB)
NLM Memory (High Water Mark) : 1,261,654,016 bytes (1.18 GB)
Kernel Address Space In Use : 1,831,743,488 bytes (1.71 GB)
Available Kernel Address Space : 845,824,000 bytes (806.6 MB)

Memory Summary Screen (.ms)
================================================== ===============================
KNOWN MEMORY Bytes Pages Bytes Pages
Server: 3488811776 851760 Video: 8192 2
Dos: 223488 54 Other: 131072 32

FS CACHE KERNEL NLM MEMORY
Original: 3484594176 850731 Code: 53559296 13076
Current: 122183680 29830 Data: 28917760 7060
Dirty: 0 0 Sh Code: 49152 12
Largest seg: 8691712 2122 Sh Data: 20480 5
Non-Movable: 81920 20 Help: 221184 54
Other: 502616064 122709 Message: 1388544 339
Avail NSS: 680148992 166052 Alloc L!=P: 837935104 204574
Movable: 8192 2 Alloc L==P: 8663040 2115
Total: 930754560 227235
VM SYSTEM
Free clean VM: 1527963648 373038
Free clean LP: 0 0
Free cache VM: 2564096 626
Free cache LP: 0 0
Free dirty: 0 0
In use: 0 0
Total: 1530667008 373698

Memory Configuration (set parameters)
================================================== ==============================

Auto Tune Server Memory = ON
File Cache Maximum Size = 1073741825
File Service Memory Optimization = 1
Logical Space Compression = 1
Garbage Collection Interval (ON) = 299.9 seconds
VM Garbage Collector Period (ON) = 300.0 seconds
server -u<number> = 884998144

NSS Configuration File:
C:\NWSERVER\NSSSTART.CFG
------------------------------
File does not exist,
or is zero byte in size.

DS Configuration File:
SYS:\_NETWARE\_NDSDB.INI
------------------------------
cache=39769334
cacheadjustinterval=15
cachecleanupinterval=15
blockcachepercent=50
preallocatecache=true

TSAFS Memory Information/Configuration
================================================== ==============================
Cache Memory Threshold : 1%
Read Buffer Size : 65536 bytes
Max Data Sets for Read Ahead : 2
Read Threads Per Job : 4

NSS Memory Information/Configuration
================================================== ==============================
Current NSS Memory Settings
-------------------------------------------
Cache Balance Percentage : 85%
Cache Memory Allocated : 656.6 MB
Available Cache from NSS : 648.6 MB

Current NSS Caching Percentages
-------------------------------------------
Buffer cache hit percentage : 89%
Name Tree cache hit percentage : 89%
File cache hit percentage : 98%

NSS Flush Status: Formula not met
NSS Size + Free FS Cache < FSAS Size
NSS Size = 688496640
Free FS Cache = 122183680
FSAS Size = 1067343872

Server High/Low Water Mark Values
================================================== ==============================
NLM Memory High Water Mark = 1,261,654,016 bytes
File System High Water Mark = 445,211 bytes

User Space Information:
User Space High Water Mark = 297,218,048 bytes
Committed Pages High Water Mark = 91 pages
Mapped VM Pages High Water Mark = 660 pages
Reserved Pages High Water Mark = 4,976 pages
Swapped Pages High Water Mark = 626 pages

Available Low Water Mark = 778,723,328
ESM Memory High Water Mark = 158 pages


>>>>>>>>>>>> bti.cfg


[InstallInfo]
InstallDir=F:\PVSW\
SystemDir=F:\SYSTEM\
InstallIniName=PTKSETUP.INI
VersionLevel=8.50
PatchLevel=0
BuildLevel=189

[NetWare Patches]
Btrieve=7 3 112
DOCS=7 3 112


[Database Names]
DBNamesDirectory=SYS:/SYSTEM


[MicroKernel]
MaxFiles=200
MaxCursors=150
BalancedTrees=NO
ForceFileVersion=0700
SystemData=YES
MaxDatabases=10
Logging=NO
;CacheSize=16384
;CacheSize=393216
;CacheSize=262144 ; changed from
262144 to 131072 9/24/06
;CacheSize=131072 ; changed from
131072 to 65536 1-27-09
CacheSize=65536
CompressedBufferSize=5
ExtendedBufferSize=16
MergeSortBufferSize=0
MaxRecSize=16
CachePartitioning=NO
TransDurability=YES
TransLogBufferSize=1024
TransLogFileSize=1024
SysTransBundleLimit=1000
SysTransTimeLimit=10000
MaxClients=150
BackgroundThreads=50
WaitLockTimeout=30
TransLogDirectory=SYS:/SYSTEM/MKDE/LOG
WorkDirectory=
Trace=NO
TraceFile=SYS:/SYSTEM\MKDE.TRA
TraceDataBufferLength=32
TraceKeyBufferLength=32
TraceOpsList=ALL
LoadRouter=NO
RouterCommBufferSize=16
RuntimeServerSupportMode=0000

;MaxCacheUsage=60 ; changed from
60 to 15 9/24/06
MaxCacheUsage=15
TransLogging=YES
Validate Request=NO
Max Pending IO=1024
File Growth Factor=15
Page Server Allow Client Cache=YES
Prompt For Client Credentials=NO
Allow Client-Stored Credentials=YES
Use FileIO Mutex=NO

[Btrieve Interface]
Embedded Spaces=NO


[Btrieve Communications Manager]
MaxWorkerThreads=50
MaxRecSize=16384
MaxClients=150
MaxReceivePacketSize=1500
Use SAP=AUTODETECT
Supported Protocols=SPX,TCPIP
Listen IP Address=
TCP/IP Multihomed=YES
Enable AutoReconnect=NO
AutoReconnect Timeout=180