I have hade (I think) the same problem and same setup.
AMD Optron, 2 GB Ram. NW6.5 Sp5. ArcServer 11.1 sp1.
My serve also abend or just went blank.
I have read this thread and done some experiments of my own.
I increased the CacheSize=419339 (20% of total memory) in the BTI.CFG file.
After this the server seems stable.

Hoppe it can help you.