VNC VPN L2TP/IPSec Will Not Connect Grey Screen

[paulsmeds]

VNC L2TP/IPSec VPN (UltraVNC version 1.2.1.2)

I have been using UltraVNC for years and have relied on it. No problems at all other than a little problem here and there. We had it implemented with a PPTP VPN connection first and then the UltraVNC connection. Everybody happy. Then iOS10 came out and no more support for PPTP VPN. We “upgraded” to L2TP/IPSec VPN without issue except for UltraVNC. Some machines it would work, others grey screen then works, other never work.

I scoured the Internet and notice many others having similar problems for years going way back. Then I read a post back 2006 regarding the MTU setting. Low and behold I changed the MTU range on the UltraVNC Server and suddenly things started working!

We are using a CISCO ASA5512X firewall that handles the L2TP/IPSec connection. The MTU for the interfaces is set to 1500. Our UltraVNC Server is on a Windows 2012R2 server standard installation no mirror drivers and no add-ins. Note on iOS10 devices using the standard built in L2TP/IPSec VPN with RealVNC but on Windows clients using UltraVNC. What we found by changing the MTU on the server many times was the following:

VPN Connected - then VNC Connection Attempt:

MTU=1500 no connections will work - grey screen after log in. Eventually disconnects

MTU=1390 or higher causes remote VPN client not to connect (grey) 1390 to 1500 will not work.

MTU=1360 remote VPN desktop client connects perfectly and fast but causes iOS10 to reconnect three times then connects solid all automatically.

MTU=1380 seems to make both VPN clients desktop and iOS10 to work. There is a long pause on the iOS10 devices but it does connect.

You can change the Windows Server’s NTU settings by first typing at a command prompt:

netsh interface ipv4 show interfaces

You will get a list of Idx connections and the MTU associated with the connections. Look for the Idx number of the Local Area Connection in use. Write it down – say 12. You can then change the MTU settings with:

netsh interface ipv4 set subinterface "12" mtu=1380 store=persistent

Assuming that your Idx is 12. If it is something else change it to that value. I noticed that on the client workstation if I do an: netsh interface ipv4 show interfaces AFTER I have the CISCO VPN connection up it shows an MTU setting on that connection at 1400. Please note this NOT done on the Client it is only done on the UltraVNC Server machine.

Okay UltraVNC folks, with this information I think you have enough to fix this problem – I hope! How, I don’t know, I will leave it up to you guys the experts. But I hope you can fix it and post a new release. I’m able to replicate these exact results on many systems and many servers. All have the same results. I’m convinced this is an MTU issue - hopefully this helps somebody.