Hi,

We´ve Operator Console version 5.5.0.95397.
NOC is installed at a Windows 2012 R2, 64bit, with 16GB, CPU E5-2660 0 @ 2.20GHz 2.20 GHz (2 processors).

We´re experiencing performance problems when accessing to Dashboard, and very often we´ve to reboot NOC server to be able to access normally.

Looking into the performance values, we´ve realized mosformula.exe process is reaching 60% of CPU used and 10,7 GB of physical memory usage.

Taking traces from the server at fsgc.log we´ve seen there´re several GC Full lasting more than 10 seconds.

8630.222: [GC 2805128K->1649305K(9320960K), 0.1236321 secs]
8642.405: [GC 2815129K->1672177K(9320960K), 0.0896312 secs]
8659.722: [GC 2838001K->1656895K(9320960K), 0.0586107 secs]
8742.577: [GC 2822719K->1719361K(9320960K), 0.0928835 secs]
8748.110: [GC 2885185K->1663569K(9320960K), 0.0613850 secs]
8752.750: [GC 2829393K->1956750K(9320960K), 0.2911541 secs]
8757.544: [GC 3122574K->2217929K(9261568K), 0.4694375 secs]
8762.825: [GC 3383753K->2487556K(9291264K), 0.5285144 secs]
8767.966: [GC 3653380K->2750924K(9257472K), 0.4765678 secs]
8772.989: [GC 4051404K->3009343K(9280512K), 0.4754824 secs]
8777.794: [GC 4309823K->3262983K(9271808K), 0.4917974 secs]
8787.357: [GC 4724743K->3512705K(9323008K), 0.4311396 secs]
8792.921: [GC 4974465K->3788537K(9333248K), 0.3096845 secs]
8802.898: [GC 5419257K->4489799K(9389056K), 0.6268906 secs]
8811.663: [GC 6120519K->5595406K(9029120K), 1.9566129 secs]
8829.974: [GC 6759470K->6062346K(9320960K), 1.2565860 secs]
8838.768: [GC 7179880K->6813280K(9320960K), 1.2740531 secs]
8840.042: [Full GC 6813280K->2669313K(9320960K), 10.2774719 secs]
8862.157: [GC 3835137K->3529935K(9320960K), 1.2981796 secs]
8895.494: [GC 4695759K->4138069K(9320960K), 1.6524014 secs]
9041.827: [GC 5303893K->4087223K(9320960K), 0.7954276 secs]
9048.131: [GC 5253047K->4109166K(9320960K), 0.2513479 secs]
9053.197: [GC 5274990K->4322545K(9320960K), 0.3763990 secs]
9058.301: [GC 5488369K->4585053K(9320960K), 0.4438994 secs]
9063.614: [GC 5750877K->4854808K(8422400K), 0.6582260 secs]
9068.788: [GC 6020632K->5095296K(9355264K), 0.6951164 secs]
9073.750: [GC 6306688K->5331447K(8434688K), 0.6292327 secs]
9078.653: [GC 6542839K->5564156K(9331200K), 0.6564848 secs]
9087.864: [GC 6842108K->5772862K(9287168K), 0.5832189 secs]
9092.859: [GC 7050814K->6009454K(9355776K), 0.3392437 secs]
9101.470: [GC 7455342K->5879484K(9369600K), 0.2772083 secs]
9110.789: [GC 7325360K->6792969K(9171968K), 1.0702350 secs]
9117.535: [GC 7972956K->7627842K(9122816K), 1.7847435 secs]
9119.320: [Full GC 7627842K->3438887K(9122816K), 15.9126911 secs]
9151.730: [GC 4691796K->3944187K(9320960K), 0.8838764 secs]
9160.272: [GC 5110011K->4791328K(9320960K), 1.2575966 secs]
9173.585: [GC 5957152K->5554172K(9320960K), 1.8707636 secs]
9203.218: [GC 6719996K->5972964K(9320960K), 1.9759851 secs]
9216.419: [GC 7138788K->5945541K(9320960K), 0.5990875 secs]
9217.019: [Full GC 5945541K->1608645K(9320960K), 2.8022803 secs]
9231.361: [GC 2774469K->1639474K(9320960K), 0.1169940 secs]
9243.694: [GC 2805298K->1671493K(9320960K), 0.1175107 secs]
9256.869: [GC 2837317K->1650797K(9320960K), 0.0662953 secs]
9311.053: [GC 2816621K->1649171K(9304576K), 0.0622443 secs]
9346.820: [GC 2814995K->1677136K(8193536K), 0.0634376 secs]
9351.634: [GC 2842960K->1883544K(9292800K), 0.3532391 secs]
9356.931: [GC 3150232K->2164849K(9294848K), 0.3473941 secs]
9362.247: [GC 3431537K->2452887K(9210880K), 0.3839065 secs]
9367.127: [GC 3618711K->2710007K(9320960K), 0.5444038 secs]
9371.569: [GC 3875831K->2938936K(9320960K), 0.8576148 secs]
9376.514: [GC 4104760K->3176546K(8387072K), 0.6632585 secs]
9384.888: [GC 4342370K->3352026K(9353216K), 0.4246764 secs]
9389.818: [GC 4558810K->3586835K(8430080K), 0.2969842 secs]
9396.037: [GC 4793619K->3359066K(9288704K), 0.0557430 secs]
9403.014: [GC 4594522K->3991595K(9277440K), 0.6191770 secs]
9409.706: [GC 5227051K->4797696K(9251328K), 1.2907230 secs]
9422.853: [GC 5963520K->5356972K(9320960K), 1.5347086 secs]
9434.415: [GC 6522796K->6045414K(9320960K), 1.0562505 secs]
9442.158: [GC 7211238K->6884896K(9320960K), 1.6541811 secs]
9443.813: [Full GC 6884896K->3164392K(9320960K), 12.8183958 secs]
9472.721: [GC 4330216K->3909440K(9320960K), 1.0659397 secs]
9573.171: [GC 5095717K->4020372K(9320960K), 1.0479182 secs]

At formula.trc I can see:

"
2017-03-15 13:13:07,071 WARN Server.Memory - Total memory threshold violated: 25% of VM memory total remaining (7.146.835.560 of 9.544.663.040 used); setting condition to: MINOR
2017-03-15 13:13:37,118 INFO Server.Memory - Resetting memory threshold condition to OK: 31% of VM memory total remaining (6.562.121.120 of 9.544.663.040 used)
2017-03-15 13:13:46,869 INFO Server.SCM - Build complete for element org=NetIQ%28r%29+AppManager+Control+Center/root=Organizations; build took 197.049 seconds
2017-03-15 13:15:00,025 INFO Script - Starting Control Center SCM Job in 30 seconds
2017-03-15 13:15:08,572 WARN Server.Memory - Total memory threshold violated: 30% of VM memory total remaining (6.718.784.936 of 9.544.663.040 used); setting condition to: MINOR
2017-03-15 13:15:23,791 WARN Server.Memory - Total memory threshold violated: 29% of VM memory total remaining (6.746.558.712 of 9.544.663.040 used); setting condition to: MINOR
2017-03-15 13:15:30,026 INFO Server.SCM - User admins has invoked view generation for: org=NetIQ%28r%29+AppManager+Control+Center/root=Organizations
2017-03-15 13:15:30,026 INFO Server.SCM - Building views (manual invocation) for: org=NetIQ%28r%29+AppManager+Control+Center/root=Organizations (manual invocation)
2017-03-15 13:15:54,245 WARN Server.Memory - Total memory threshold violated: 14% of VM memory total remaining (8.222.625.216 of 9.544.663.040 used); setting condition to: MAJOR
2017-03-15 13:16:23,995 INFO com.mosol.Formula.Templates.NQCCDB.CCCache - Building cache of Control Center Data from NetIQ%28r%29+AppManager+Control+Center=NetIQ%28r%2 9+AppManager+Control+Center/root=Elements
2017-03-15 13:16:26,057 INFO Server.Memory - Resetting memory threshold condition to OK: 52% of VM memory total remaining (4.536.536.176 of 9.544.663.040 used)
2017-03-15 13:16:51,479 INFO Script.Builder post definition Script (@templates/ControlCenter/PostProcess.scm.fs) - Begin post-processing
2017-03-15 13:18:53,496 WARN Server.Memory - Total memory threshold violated: 25% of VM memory total remaining (7.180.770.672 of 9.544.663.040 used); setting condition to: MINOR
2017-03-15 13:19:12,543 WARN Server.Memory - Total memory threshold violated: 17% of VM memory total remaining (7.905.276.608 of 9.544.663.040 used); setting condition to: MAJOR
2017-03-15 13:19:35,153 WARN Server.Memory - Total memory threshold violated: 14% of VM memory total remaining (8.194.286.248 of 9.544.663.040 used); setting condition to: MAJOR
2017-03-15 13:19:58,387 WARN Server.Memory - Total memory threshold violated: 14% of VM memory total remaining (8.231.495.280 of 9.544.663.040 used); setting condition to: MAJOR
2017-03-15 13:20:30,216 WARN Server.Memory - Total memory threshold violated: 13% of VM memory total remaining (8.310.044.384 of 9.544.663.040 used); setting condition to: MAJOR
2017-03-15 13:20:51,794 WARN Server.Memory - Total memory threshold violated: 13% of VM memory total remaining (8.326.799.392 of 9.544.663.040 used); setting condition to: MAJOR
2017-03-15 13:21:44,529 WARN Server.Memory - Total memory threshold violated: 12% of VM memory total remaining (8.351.804.360 of 9.544.663.040 used); setting condition to: MAJOR
2017-03-15 13:22:21,607 WARN Server.Memory - Total memory threshold violated: 13% of VM memory total remaining (8.350.487.736 of 9.544.663.040 used); setting condition to: MAJOR
2017-03-15 13:22:40,373 WARN Server.Memory - Total memory threshold violated: 13% of VM memory total remaining (8.347.055.696 of 9.544.663.040 used); setting condition to: MAJOR
2017-03-15 13:23:30,295 WARN Server.Memory - Total memory threshold violated: 13% of VM memory total remaining (8.351.335.128 of 9.544.663.040 used); setting condition to: MAJOR
"

At daemon.trc I can see:

"
2017-03-14 13:17:43,841 INFO CORBA.Daemon.Formula.out - Occurred 185 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:17:43,841 INFO CORBA.Daemon.Formula.out - Occurred 930 times - No specialized update boundary for g()
2017-03-14 13:17:43,841 INFO CORBA.Daemon.Formula.out - Objective Instance counter: 5600
2017-03-14 13:17:45,482 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:17:45,810 INFO CORBA.Daemon.Formula.out - Occurred 1 times - Objective Instance counter: 5600
2017-03-14 13:17:45,810 INFO CORBA.Daemon.Formula.out - Occurred 9 times - No specialized update boundary for g()
2017-03-14 13:17:45,810 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:17:50,482 INFO CORBA.Daemon.Formula.out - Occurred 3 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:17:50,482 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:17:50,794 INFO CORBA.Daemon.Formula.out - Occurred 9 times - No specialized update boundary for g()
2017-03-14 13:17:50,794 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:17:55,482 INFO CORBA.Daemon.Formula.out - Occurred 4 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:17:55,482 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:17:55,826 INFO CORBA.Daemon.Formula.out - Occurred 9 times - No specialized update boundary for g()
2017-03-14 13:17:55,826 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:18:00,419 INFO CORBA.Daemon.Formula.out - Occurred 4 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:18:00,419 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:18:05,857 INFO CORBA.Daemon.Formula.out - Occurred 9 times - No specialized update boundary for g()
2017-03-14 13:18:05,857 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:18:20,967 INFO CORBA.Daemon.Formula.out - Occurred 3 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:18:20,967 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:18:20,998 INFO CORBA.Daemon.Formula.out - Occurred 9 times - No specialized update boundary for g()
2017-03-14 13:18:20,998 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:18:21,342 INFO CORBA.Daemon.Formula.out - Occurred 11 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:18:21,342 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:18:26,029 INFO CORBA.Daemon.Formula.out - Occurred 9 times - No specialized update boundary for g()
2017-03-14 13:18:26,029 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:19:09,436 INFO CORBA.Daemon.Formula.out - Occurred 10 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:19:09,436 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:19:16,155 INFO CORBA.Daemon.Formula.out - Occurred 9 times - No specialized update boundary for g()
2017-03-14 13:19:16,155 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:19:30,640 INFO CORBA.Daemon.Formula.out - Occurred 14 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:19:30,640 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:19:31,186 INFO CORBA.Daemon.Formula.out - Occurred 9 times - No specialized update boundary for g()
2017-03-14 13:19:31,186 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:20:09,453 INFO CORBA.Daemon.Formula.out - Occurred 26 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:20:09,453 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:20:11,297 INFO CORBA.Daemon.Formula.out - Occurred 9 times - No specialized update boundary for g()
2017-03-14 13:20:11,297 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:21:09,377 INFO CORBA.Daemon.Formula.out - Occurred 43 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:21:09,377 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:21:11,705 INFO CORBA.Daemon.Formula.out - Occurred 9 times - No specialized update boundary for g()
2017-03-14 13:21:11,705 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:21:20,861 INFO CORBA.Daemon.Formula.out - Occurred 3 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:21:20,861 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:21:32,081 INFO CORBA.Daemon.Formula.out - Occurred 18 times - No specialized update boundary for g()
2017-03-14 13:21:32,081 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:21:35,440 INFO CORBA.Daemon.Formula.out - Occurred 4 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:21:35,440 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:21:38,315 INFO CORBA.Daemon.Formula.out - Occurred 9 times - No specialized update boundary for g()
2017-03-14 13:21:38,315 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:21:41,253 INFO CORBA.Daemon.Formula.out - Occurred 4 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:21:41,253 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:21:48,316 INFO CORBA.Daemon.Formula.out - Occurred 18 times - No specialized update boundary for g()
2017-03-14 13:21:48,316 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:21:50,441 INFO CORBA.Daemon.Formula.out - Occurred 3 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:21:50,441 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:22:08,348 INFO CORBA.Daemon.Formula.out - Occurred 18 times - No specialized update boundary for g()
2017-03-14 13:22:08,348 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:22:09,426 INFO CORBA.Daemon.Formula.out - Occurred 3 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:22:09,426 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:22:28,379 INFO CORBA.Daemon.Formula.out - Occurred 36 times - No specialized update boundary for g()
2017-03-14 13:22:28,379 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:22:30,489 INFO CORBA.Daemon.Formula.out - Occurred 3 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:22:30,489 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:22:33,380 INFO CORBA.Daemon.Formula.out - Occurred 9 times - No specialized update boundary for g()
2017-03-14 13:22:33,380 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:22:40,411 INFO CORBA.Daemon.Formula.out - Occurred 3 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:22:40,411 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
2017-03-14 13:22:48,411 INFO CORBA.Daemon.Formula.out - Occurred 18 times - No specialized update boundary for g()
2017-03-14 13:22:48,411 INFO CORBA.Daemon.Formula.out - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:22:50,505 INFO CORBA.Daemon.Formula.out - Occurred 2 times - mailAlarmSummary(): element condition is equal or less severe than prior. Skipping email...
2017-03-14 13:22:50,505 INFO CORBA.Daemon.Formula.out - No specialized update boundary for g()
"

I read this forum talking about JVM memory management
https://www.netiq.com/communities/co...er-tuning-jvm/

I´ve check the configuration of Java Runtime, on Operation Center Configuration Manager, and
NOC Server Java Runtime Options are:
-server -Xmx10g -XX:MaxPermSize=128m "-Xloggc:\Appsys\NetIQOperationsCenter\NOC\logs\fsgc.log"

What I see is that NOC is consuming the maximum amount of memory configured to be used by JVM.
How can I diagnostic and solve the issue? I´m afraid that to increase physical memory wouldn´t be enough, as there´s something causing the memory leak.

I´ve followed https://www.netiq.com/documentation/...a/blx27bc.html to diangose the problem,
How could I upload some logs to the forum?
Thank you.