A scheme for enhancing transfer of video-media playback

[Inventus]

Dear UltraVNC developers,

I would like to suggest a method by which you could possibly enhance the transfer-rate/update-speed and lower the CPU load, when viewing video-media (defined as "any kind of video being decoded through a windows codec", ie. typically shown in Windows Media Player or alternatives).

I realise that few people will probably have a need for watching disk-based video through a remote connection, but seing the posts about web-cam support and imagining a possible need for watching similar streamed video material, it might be worth considering. Also, it might possibly provide a path to improving the overall performance of (the video-transmission in) UltraVNC?

Note: This scheme is possibly platform-dependant (specifically Windows), although I don't know if video codecs (as seen in Windows) are also used in other OS'es, and if they are sufficiently "compatible" between platforms to allow this scheme to be truly platform independant , but to some extent it should be possible though (seeing as the video-media itself seems to be quite platform-independant?)

The idea is a simple (well, in theory anyway) two-step process (as dictated by the client+server nature of VNC):

1. Make the server ('s video-hook driver or similar) somehow hook into the relevant parts of the windows video-codec subsystem, so as to allow it to "intercept" anny video data-streams before they are decoded and sent to the application requesting the decode (possibly replacing the original stream with a simple "dummy stream", requiring little CPU power to render by the server-side codec and/or viewer-application). The server can then transmit this stream, unmodified (and assumably already near-optimally compressed, typically using MPEG4, DivX, or similar modern codec) to the client (along with some relevant information about the codec used, the current window-position and -size etc.). The area covered by the video (the viewers client-area/video-viewport) is otherwise left untouched (unencoded and untransmitted) by the server.

2. Make the client support the standard Windows (and/or other) video codecs, allowing it to pass the recieved video-stream directly and unmodified to the codec for decoding, and then paint the result into the viewport at the transmitted coordinates, thus, if all went well, covering the area left otherwise untransmitted by the server with the, locally decoded, video image.

Note that aside from making use of the near-optimal compression of modern video-media (streamed as well as disk-based), the scheme also takes advantage of most codecs being resolution-independant, thus allowing the same high frame-rates to be achieved, regardless of the size/resolution of the transmitted video-window(s).

One drawback of the scheme is that the codec needed for a given type of video-stream (or file) must be present on both machines, notably on the client (assuming that the server-side viewer application would display an error if the codec wasn't present there, unless the dummy-stream can also be made to specify some fixed, always-present, codec, such as RLE). However, a client using this scheme might send a list of installed codecs to the server when connecting, allowing the server to decide if and when this method can be employed, and otherwise fall back on whatever scheme is in use for other, video or non-video, content. This would also allow the (advanced) user (and/or server-admin) to "blacklist" certain codecs from the list sent to the server, in case of buggy codecs or other issues.

Since it is allowed for multiple processes to decode video through the standard codecs (as it is possible to open any number of mediaplayers at once, showing the same or different video streams, using the same or different codecs), it should not even be a problem to show multiple video-streams being played on the server at once.

It might even be possible to extend this scheme to be used for transmitting other parts of the server-side display, in a similar manner as the Thight scheme use jpeg (mpeg?) compression, only with the user (or possibly some auto-detection scheme?) selecting which codec(s) to use (perhaps even allowing the use of different codecs or codec-settings for different "content-types"). This is assuming that certain parts of the remote desktop and/or its contents can be encoded (and decoded) more efficiently by the server (and client), than the existing schemes (notably Tight for slow connections and Hextile for for fast ones)?

Finally, as I think it has been suggested in other feature-requests, the same (or a similar) scheme could be used for a number of other "standardised data-streams", such as OpenGL and Direct3D, possibly even some rich-text formats (HTML? RTF? PDF?) etc. Where the common denominator is that the involved data-stream, before "rendering", is MUCH more efficient to transmit (not to mention less CPU intensive for the server), than the resulting stream of pixels (even when this is compressed, further adding to the server-side CPU load). If done properly, such schemes could revolutionise how CAD designers and 3D artist can do their work (such as the movie, vehicle and aircraft industries to mention a few), since it would then be possible to work with any OpenGL based 3D application (such as AutoCAD, 3D Studio, Softimage and similar) on even the slightest laptop or other "thin" client, letting the powerful server (actually a graphical workstation, but the machine running the VNC server) take care of all the heavy work, letting the VNC client PC handle just the 3D rendering (OpenGL or D3D), which even small laptops handle ok, due the recent developments in consumer 3D acceleration for laptops. Effectivly this would allow such people to work from home, use less noisy and bulky workstations in the office environment (relegating all the CPU power to the backoffice) and allow work-in-progress to be presented and interacted with remotely without fuss.

I hope this suggestion is of interest, even if it might not be something you will consider for imidiate implementation. If not, then you should at least be able to find something of interrest in my two (rather long, I'm afraid) post scripts.


Regards,

Adam.


P.S. While I'm at it, a few items from my feature-request short-list:

- Option to make client send keyboard events as long as it has focus, specifically when mouse is not over its window (which is not the default behaviour of most Windows applications, and thus it looses any possible reason for existence in Win32 based clients, since the other open windows typically won't accept the keyboard input anyway, as long as the UltraVNC client has the focus, regardless of where the mouse is!)

- (Possibly option to) show the "transfer-indication lights" in the window title-bar, when the toolbar is hidden.

- Consider smoothing (doing a running average on) the (abovementioned) "lights" over a second or two, using the floating-point result as the intensity of the "light", so as to make them appear less "strobing" and giving a much better indication of actual bandwidth usage (as indicated by the smoothly varying intesity, rather than a difficult to gauge and constantly fluctuating "strobe frequency"). It is perhaps not of great importance, but should on the other hand be a matter of a few, simple, lines of code.

- Server-side, possibly also client-side, option to limit the CPU load from UltraVNC. If possibly a better way of doing this than is (AFAIK) provided by Windows (since I get the impression that it tends to not having the desired effect, of letting UltraVNC stay responsive to client-input and always provide at least a few frames/second, while making sure to release as much of the remainding CPU power to other applications).

- Better pointer-shape handling would be VERY nice. It might just be me, but I keep being, "distracted" is perhaps the best word for it, by my pointer "forgetting" to take on the expected shape (typically when I resize windows, as I then use the shape-change as a cue for when my mouse is over the thin frame I want to "grab"). I can't imagine it too difficult to have the server (reliably, that is) send a notification each time the server OS changes the pointer-shape, which must be "centralised" operation for most OS'es. Notice that I'm not even suggesting that you should look into supporting "custom pointer-shapes" of any kind, just the most common, OS-specified, pointers (or even the sub-set of pointer-shapes that are present, and reasonably similar in usage, across the most used OS'es). As it is, it is more often than not the case that the pointer is left with whatever shape it had previously, whenever a change of pointer-shape should have occured, even when using the most simple and conformant windows applications? (In case this is a platform specific problem: I'm running Windows 2000 Pro. on both the server and client PC).

- I don't know if this has been implemented by some third-party (and if so, has it been implemented properly?) but: How about providing a remote-installer for the server (possibly all components?) So as to allow those of us running our servers without permanently attached local input/output devices to install, and notably re-install/update, the UltraVNC server, without the risk of having to go to the server location and connect a monitor and a keyboard, possibly even a mouse, just to get the server up and running again, in case the service for some reason (such as an installation issue, failed installation, if you want to do a clean install, or simply due a crash, bug or other non-recoverable issue).

- In case the above is infeasible (say, if there is no way to reliably upload, install and run a service on a PC not already running some kind of remote-managment, -shell and/or -filetransfer service) You could, alternativly, implement some kind of server-side "watchdog-scheme", ensuring that a (previously set aside, be it manually or automatically) "last-known-good" version (and configuration) of the UltraVNC service is always started (or at least that one or more "attempts" is made at starting it). This would make UltraVNC a much more reliable tool for server maintenance.

- Option to have the client send a WOL request to the server before connecting (with a user-configurable extended timeout, to allow the server time to boot and start the VNC service, before the client aborts the attempt). In case you consider this functionality "bloat" in light of some (percieved or actual) "mission-statement" for UltraVNC, consider instead an option to run a batch script (waiting for it to finish) before connecting, allowing users to implement such (and other) functionality by this means.


P.P.S. I see that a number of people have requested support for audio (mostly from server to client, although some want it both ways). Since the, quite reasonable, argument has been put forward, that such functionality (two-way realtime audio transmission between two networked PC's) is readily available, even as freeware, I would like to suggest a possible "workaround" (or "common middleground" if you like):

Instead of going through the trouble of implementing such audio support directly in UltraVNC, why not simply "leech" on those existing applications and protocols out there, notably the freeware / opensource ones. In its simplest form, you could simply make the client and server start the appropriate (possibly user selected/configured) third-part (VoIP or similar) application, which would then take care of the audio side of things, completely (or mostly) seperate from UltraVNC (ie. using whatever ports and protocols the given application would normally use, and communicating directly with each other, or possibly tunnelling through VNC).

As far as I can see, this would provide the audio-whishers (of which I am one) with what we need (even giving us the choice of application to handle our various audio needs, be they voice, music or mixed), while adding little to UltraVNC for the purists to scream "Bloat!" at (after all, who can complain about a simple feature for starting and stopping a choosen external application on each side, when a connection is made. Such a feature could have many different uses, audio support merely being one of them. Also it is so small an additon, code-wise or functionality/interface wise that it hardly does the word "bloat" justice.)