Possible memory leak on Windows 10

[LRN]

32-bit Windows 10 1.2.0.6 32-bit server, w8hook and schook are installed.

When a connection is established, winvnc.exe (the child one) memory usage (private bytes) starts to increase rapidly, reaches past 1GB (the laptop on which i'm testing this only has ~3GB RAM, so it could be that 1GB is not the maximum, just the point where the OS starts to run out of memory) then drops down to ~40MB, then increases again, and so on.

When viewer is minimized, the memory use does not change at all.
With no or little activity (no mouse movement, just the system info chart from a maximized Process Hacker window) the memory usage increase is ~30MB per second.
When a lot of activity (mouse movement, frequent window switches) the memory usage increase is more than 200MB per second.

When w8hook is removed, memory usage remains small (the downside is, obviously, that without w8hook VNC eats more CPU and also it has to run a keyboard helper; not sure what else).

This happens regardless of what kind of client is used (happens with both 64-bit Windows 7 1.2.0.6 64-bit client and SSVNC running on Debian, although the behaviour of minimized client is different, as SSVNC can't be minimized).

On 64-bit Windows 8.1 64-bit 1.2.0.6 server (w8hook and schook installed) memory use also changes rapidly from ~40MB to ~300MB depending on the amount of activity, but does not exceed the ~300MB threshold.

Looks like a Windows 10-specific memory leak.

[LRN]

Only happens on 32-bit Windows 10. 64-bit OS (using 64-bit server) is not affected.

I could try 32-bit server on 64-bit Windows 10, if you think that'll help.

[Rudi De Vos]

The desktop duplication engine use a lot of memory (w8hook) but this is allocated by the MS functions.
There is no difference in src between 32 or 64 bit.

Possible you are testing with 2 different pc's and is the 32bit a lot slower.
If we duplicate to fast, and the system can't follow the engine internal queue the updates, that's a lot of memory
as each framebuffer is 10-20MB.
It could be that we just ask more updates that the system can handle... i need to verify this on a slower pc.

[LRN]

Quite possible. The laptop on which i've seen this bug is quite old (Thinkpad R500). I don't remember seeing it on Windows 8.1 that was there previously, but i wasn't looking too closely back then.

[LRN]

1.2.0.8 pre-release does not fix this (in case anyone is wondering). That said, since w8hook is now built into winvnc.exe, the remove-w8-hook workaround is no longer available.

[Rudi De Vos]

I'm testing with win32 and mem usage is 100MB for a 1920X1200 display.

But as the api seems to behave bad on some video drivers, some extra option is needed to disable the new api
and default to the previous system.