Windows 10.1607 vnchooks.dll constantly crashing

[Tki2001]

I have constant crashes on W10 1607 anniversary edition, whenever I enable "System Hook Dll" on a Teclast X98 Pro.
If I try to connect to server 1.2.1.1 with whatever version of viewer (several versions tried), I can only see the screen for a couple of seconds, then the server crashes and the service relaunches another vnc server instance. The crashed server leaves the yellow icon on tray, that dissapears when mouse cursor enters it. No logs are leaved in system logs.
With "System Hook Dll" disabled, everything works fine.
Replacing vnchooks.dll from server 1.2.1.1 with vnchooks.dll from server 1.2.0.5 works perfect. I can enable "System Hook Dll" and install mirror driver and everything works perfect. I have not tried to replace vnchooks.dll with another version other than 1.2.0.5. But the changes made since that version prevent it from working.
The teclast screen resolution is 2048x1536. I think that it may be the reason, as I have seen somewhere that no vertical monitors have been tested with ultravnc. The fact is that the previous vnchooks.dll version I tried, worked. Tried rotating the screen, but same result.

The 1.2.1.1 server version on a W7 machine does work well. It does not crash at all on a 16:10 monitor.

If you need some additional info I will be happy to provide it.

Any clue?

[Rudi De Vos]

The hookdll capture system message to give hints about changed possition. On win8 >= the hookdll isn't realy needed as the desktop
duplication already give the exact positions of the changed rects.

But, the dll should not let vnc crash.

The hookdll doesn't have anything resolution related, it just capture window messages and extract screen position from it.
Then the dll inject the possible changed rectangles in vnc, vnc then give priority to thoose rects.

Let see if i can repeat it ...

[Tki2001]

I made some further testing and found this results.

It seems that the crash situation is a combination of vnchooks and mirror driver.

NO System HookDll - NO Mirror Driver ---> OK (High CPU, one core goes to 100%)
NO System HookDll - YES Mirror Driver ---> OK (Low CPU)
YES System HookDll - YES Mirror Driver ---> OK (Low CPU)
YES System HookDll - NO Mirror Driver ---> Crash

Whatever vnchook.dll version I use, I get the same results. It seems that when I changed the version, I activated the mirror driver too, so I got no more crashes.

On Win7 it seems that I cannot reproduce the crash. On Win10 it is a sure crash.

Hope this helps.

Edit: I found that the wallpaper on remote is removed by altering the theme settings. If ultravnc crashes, you get a desktop without wallpaper as it is not restored. You have to manually restore it, which is a bit annoying.

[Rudi De Vos]

On win8>= selecting hookdll also activate desktop duplication engine ( ddengine).
On win7, the same exe sees that ddengine doesn't exist and disable that code part.
I guess the crash is from this engine.

This can be easy tested, start winvnc and make sure the vnchook.dll doesn't exist in that folder.
This way only the ddengine is active.
Normal, it should also show on the taskbar w8hook == ddengine
Or
If you use the winvnc version ( vista, xp) from the bin zip ( just make sure that you use the same arch 32/64).
That's build without the ddengine, activating the hookdll is hook only.

Testing
With (en_windows_10_multiple_editions_version_1607_updated_jul_2016_x64_dvd_9058187.iso), that should be the latest.
Can not crash the system.

My resolution was set 1280x1024 and changed while viewer connected to 1920x1080. Vnc viewer reset to new resolution and ddengine is still on. That's the max i go with this video card, overwriting monitor resolution .


If time, can you also try with a lower resolution.

[Tki2001]

Tried some more tests.

I have moved out of the directory all versions of vnchooks.dll I had, so the server will not load any.
Disabled "Mirror Driver" in server Properties.
Left enabled "System HookDll" in server Properties.
Connect and crash!.

If I have moved out vnchooks.dll, then why is it crashing? What is the server loading?

Tried lower resolutions as low as 1024x768. As long as Property "System HookDll" is checked and "Mirror Driver" is not checked, the server crashes. Even without vnchooks.dll.

Restored vnchooks.dll onto directory, enabled "Mirror Driver" and "System HookDll". Connect OK. Unlocker tells me that vnchooks.dll is locked by process WinVnc.exe, so it is loading.

May be an Intel HD graphics driver issue? I'm totally clueless at this point.

Edit: Tried in view only mode too, just to see if it is something keyboard related, as the tablet does not have any keyboard. Crash again...

[Rudi De Vos]

Then the crash is from the ddengine.
On win8>=, selecting hookdll also activate the ddegine.
A crash if vnchook.dll is not installed need to be from ddegine part.

Is the video driver wdm1.2 or higher?

As workaround you can to use the winvnc.exe 1211 for XP, this doesn't have the ddegine compiled in.

[Tki2001]

dxdiag tells me that the intel hd driver is 10.18.15.4248, WDDM2.0 compliant. So that requisite should be covered.

I will try the XP version tomorrow. Will post results.

Should it give a clue about the crash, I can tell that when the viewer connects to the server, sometimes there's a brief time (never more than 2-3 seconds) before crashing. Sometimes it does not even draw the whole screen and it crashes half way. Sometimes it draws the screen and let me click a mouse button, then crashes...

[Rudi De Vos]

If you have time i can create a debug version that should create a dump on crash.
If i import the dump, i pinpoint the src location.

But, I don't have time before the weekend to compile the special version, still have a daytime job...

[Tki2001]

Hello.

I tried the XP x64 version as you told me, and collected these results.

NO System HookDll - NO Mirror Driver ---> OK (High CPU, one core goes to 100%)
NO System HookDll - YES Mirror Driver ---> OK (Low CPU)
YES System HookDll - YES Mirror Driver ---> OK (Low CPU)
YES System HookDll - NO Mirror Driver ---> OK (High CPU, one core goes to 100%)

So, everything works as expected.
The crash seems to be ddengine related.

I have observed another issue too. If I stop the service while connected, sometimes I get a BSOD. Both versions gives the same BSOD, the XP one and the ddengine enabled one.
I got a BSOD too once while I rebooted the tablet from remote, but I think it is caused by the same "stopping service" issue I observed, as services are shut down when rebooting.

I'll be glad to test a debug version if it can reveal some valuable data.

[Rudi De Vos]

Test builds.

In case of a crash , this build generate a dump..

3 builds
XP32
win8>=32
win8>=64

http://www.uvnc.eu/download/forum1211/w ... _test1.zip

Some changes that were added
-win10 and potrait mode should work
-disable systemhook while running crash
-some w8hook mem protection, in case of an error switch to polling mode.

[Tki2001]

Sorry for the delay. I'm in the middle of a moving, and I do not have access to internet regularly, nor to the tablet right now.

Thank you for the test builds.

I will post results of the test builds in 3-4 days. Right now the tablet is 300km away from where I am

[Tki2001]

Hello again, and sorry for the looooong delay. As I said I was in the middle of a moving, and could not test anything.

I have made some more testing with the test builds you left me, and these are some of the results.

I first had to find the crashtest libraries as they were not found on the test builds. I found them on another thread and have been using on the tests, both x86 and x64 libraries with the respectives x86 and x64 binaries.
I could not test the Win8 x86 build as I could not even start the service.
I could not test the XP build as I'm using ms-logon authentication, and even when the binary starts up, it complaints about authentication in a missing dll, and authentication fails. Did not go any further with this build.
I tested then the Win8 x64 build, with the x64 crashtest libraries, and could not make it to crash, so, so far, I cannot give you any useful info. The only thing I observed is that the binary can switch from Mirror mode to non Mirror mode on the fly, but it cannot write the settings back to the ultravnc.ini file (permissions denial, I suppose), so I had to manually edit the file to start the service with my test config.

I only observed once, that restarting the service after replacing the ultravnc.ini file, had frozen completely the tablet. No BSODs, but frozen, so no log either. I do not mind it, as I was restarting the service in a non standard way. I could not make the service to crash if I restarted it several times in a normal way (without touching the ini file externally).

Another thing I observed, is that the cursor does not scale with the screen. I select the screen to scale by half, but the cursor on viewer stays at original size, so it is too much laaarge...

I've seen that you have released another step up version, so after testing this one a little more, I will try 1.2.1.2, and see what happens.

Thank you very much for your attention.

[RedAero]

Just a note: this still happens. The VNC Server crashes on connect if Desktop Duplication is enabled (which it is by default).