[BUG] Thread Deadlock When Multiple Viewers Connect at Once

[rdonnerhack]

Our software team has a product that typically utilizes concurrent connections to an UltraVNC 1.0.9.6.1 Server. However, we have come across intermittent deadlocks which requires killing and then restarting the process. I was able to reproduce the issue by doing the following:
- Write a script to have 5 viewers attempt to connect to the Server at the same time.
- If they all connect, stop the viewers and try again
- At some point, the Viewers will hang and UltraVNC will become unresponsive. This becomes apparent by right-clicking the VNC Server icon, which will not bring up the context menu.

I attached the Visual Studio 2010 Debugger and found the following threads stuck waiting for m_clientsLock (i.e., omni_mutex_lock l(m_clientsLock)):
- vncServer::AddClient
- vncServer::Authenticated
- vncServer::AreThereMultipleViewers

As a test, I removed the lock from AreThereMultipleViewers and simple returned m_authClients.size() > 1. This seems to prevent the deadlock; however, I obviously don't just want to start removing locks just because it may work. If anyone has any advice or if this is a known defect, please let me know. Any suggestions would be greatly appreciated.

Thanks,

Randal

[rdonnerhack]

We figured out the cause and it appears to be a defect on the VNC Server. The problem is that different threads are attempting to lock the same mutex. Since the second thread may not always realize that the first thread has released the mutex, it waits indefinitely. Once the first thread goes to the lock, it also waits since the first thread is waiting. We fixed it by removing the lock from AreThereMultipleViewers(). It's not the best solution; it'd probably be better to create a wait() and signal() call between the two threads, but for our needs, it works fine.

[B]

Thanks for sharing your findings; maybe the developer(s) (Rudi) can include something like that next time.

[nerdx86]

We have also encountered a similar problem, but it has been infrequent enough that we haven't traced it down yet.. We used to get a server hang (couldn't even restart the service) on version 1.0.9.5 when our full screen applications would open/close/restart. To fix it, we backed back to version 1.0.8.0 and it seemed to go away..

Also, I have twice or three times had a similar hang in 1.0.8.0 (I wasn't at the machine which is in Utah and I'm in Michigan) immediately following a Control - C operation on an edit control through the VNC connection.. Server hung and machine needed to be restarted.. There is a chance that a simultaneous connection was being accepted at the same time I hit Control-C..

For the last week, we have been testing version 1.0.9.6.1 at a couple sites and have had 1 or 2 Server hangs. I'm going to try to stress test this a bit locally..

My questions:
1. Has anyone came up with a good auto-restart function for 1.0.9.x Server?
2. Are there others with "new" server Hang's in the new versions that may want to try to put our heads together to help the track the problem?
3. Besides the debug log, are there any additional logs/reports/etc that I should be evaluating when this occurs?

We do not currently have a system setup to build UltraVNC.. Thanks in advance for anybody's 2 cents (or flames for that matter).