As you say, you must upgrade the Forge to 11.3 Appliance 4, which includes VMware ESXi 6.5, as this ESXi 5.5 does not support Windows Server 2016 virtual machines.
https://www.netiq.com/documentation/...e-options.html
Assuming your Forge has a maintenance contract with Micro Focus then you would be entitled to a VMware ESXi 6.5 license to perform the upgrade. Forge appliances also include licenses for the management VM, i.e. Windows Server 2012 R2 and SQL Server 2014. If your management VM is already running these versions then you will not need new keys for these, if not then please also request these from Micro Focus Customer Care.