Invalid Plugin/No Connection/Works w/o Plug in

curtis_rapp · Post by **curtis_rapp** » 2006-02-09 15:08

First my project scope: I am in charge of installing encrypted VNC on my domain's servers, and probably ultimately desktops. We have been using various flavors of VNC for years now, using Ultra at the moment. We have not been using MSLogon and I plan on sticking with that. I want to have an encrypted tunnel b/t viewer and server, that said, UltraVNC + the DSM Plugin looks like what I want.

In the past we have used fastpush to install our flavors and do our password updates. Now I want to be able to push out the encryption key in addition. I came across a product called VNCScan that is a gui that purports to do the same thing as fastpush (which is cmd line). Well after trying it I am stumped.

I am able to push my plug-in and key to the client machines (it shows up along with the vncviewer.exe in c:\program files\orl\vnc). When I connect w/o the dsm plugin, both machines are able to view the other's desktop (the connection works). However, when I enable the plugin and configure it, I receive the dreaded "Invalid Protocol" error message. I am then told that one of four possibilities can cause this:
1. You've forgotten to select a DSMPlugin and the server uses a DSMPlugin.
2. The selected DSMPlugin is not compatible with the one running on the Server
3. The seelcted DSMPlugin is not correctly configured (also possibly on the server)
4. The password you've possbily entered is incorrect.

My thoughts on these are:

case 1. I selected "use plugin" on the viewer, I've gone to the client machine and configured its viewer for the same dsm plugin. This can't be the problem. How do I configure it on the server-end?

case 2: The DSM Plugin is the same on both machines, it certainly cannot be incompatible. Can it?

case 3: This is so cryptic it doesn't help me. I've ensured that all rc4.key files on both viewer and server are written to c:\program files\orl\vnc which is where the server/viewer .exe's reside. Other than that, I don't know what there is to configure.

case 4: I didn't enter a password at the dsm plugin configure screen. Where else would a pasword come into this equation?

When I was checking the logs that are created by the testviewer.cmd and testserver.cmd files that come with the plugin
After checking the log, I see that with the plugin it IS connecting, but somehow it is deciding to discconnect. Below are the log files:
----------------------NO PLUGIN--------------------------------
Started and Winsock (v 2) initialised
bufsize expanded to 4352
Registered connection with app
Saving connection info
Saving to C:\DOCUME~1\carapp\LOCALS~1\Temp\options.vnc
Viewer is C:\Program Files\Orl\VNC\vncviewer.exe
Connected to inind00001 port 5900
RFB server supports protocol version 3.6
Connected to RFB server, using protocol version 3.4
VNC authentication succeeded
Clipboard changed
Don't send initial clipboard!
Desktop name "inind00001 ( 138.154.148.66 ) "
Geometry 1024 x 768 depth 32
Screen work area is 1280 x 994
Cache: Cache buffer bitmap creation
Requesting 8-bit truecolour
Screen work area is 1280 x 994
Update-processing thread started
Screen work area is 1280 x 994
Screen work area is 1280 x 994
bufsize expanded to 16640
Requesting new pixel format
Receiving cursor shape update, cursor 32x32
rdr::Exception (1): rdr::SystemException: read: Unknown error (10004)
Update-processing thread finishing
Deregistered connection from app
Exiting
VNC viewer closing down

--------------------WITH PLUG-IN-----------------------
Started and Winsock (v 2) initialised
bufsize expanded to 4352
Registered connection with app
Saving connection info
Saving to C:\DOCUME~1\carapp\LOCALS~1\Temp\options.vnc
Viewer is C:\Program Files\Orl\VNC\vncviewer.exe
Connected to inind00001 port 5900
Deregistered connection from app
Exiting
VNC viewer closing down

Can anyone make sense of what I've got wrong?
If there is pertinent information I have left out, please let me know.

Additionally, I have seen suggestions to restart VNC services, I've done that on all machines, to no avail.

bevtech · Post by **bevtech** » 2006-02-11 02:32

Ok...

You have the correct rc4.key in the same directory along with the DSM.noreg
.dsm file in the same directory controlled via the admin properties..

be sure to check the use dsm and make sure that the same dsm is listed restart the service

curtis_rapp · Post by **curtis_rapp** » 2006-02-13 15:39

This didn't do anything for me. However, something is wrong b/c I had to run a new install to get to the properties screen and that should not be necessary. Has anyone out there in cyberspace have experience using fastpush to implement a VNC change? I inherited this assignment and my lone previous experience with fastpush was password changes. Now I am charged with upgrading all servers to Ultra and pushing out the encryption plug-in. Any words of wisdom?

redge · Post by **redge** » 2006-03-02 00:43

what is your OS at viewer and server ?
MSRC4 plugin build ? (we know you use same build at server and viewer)

curtis_rapp · Post by **curtis_rapp** » 2006-03-02 20:23

I've got this problem licked. For me it was a matter of setting up the server properly. I had to get a better understanding of the plugin and the key as well. When you follow directions to a "T" it makes it easier.

snteran · Post by **snteran** » 2006-06-12 23:24

"When you follow directions to a "T" it makes it easier." What directions are you talking about. I am having the same issue and would like to resolve the issue. I would like to push out each install with the plugin's and rc4.key already in the correct folder and the setting checked to accept encrypted login. Not sure if this is able to happen or if I just have to push and then copy files to the correct folder, however on my test box I was able to push OK but can not get the plugin to work. Any help/Advise greatly apreciated.

snteran · Post by **snteran** » 2006-06-12 23:40

OK, I realized that I created this install from the default fastpush install and it sets the program files directory to orl\vnc and the config for the plugin is set to program files\ultravnc. I made the change in the config section to point to the correct location and now I can use the dsm plugin.