can't reverse connect with encryption

[microshnik]

I'm spam-new to UltraVNC, just installed the latest version (1.8.2.4) on two 64-bit machines (Win11 server and Win10 viewer), and went through what I thought was all necessary setup of both to use SecureVNCPlugin64.dsm, but attempting to initiate an outgoing connection from the server results in a hard stop with error message "plugin has wrong arch (x86)". When I click OK to dismiss this, it is followed by an alert with warning message "The Plugin cannot be loaded. Please check its name integrity." When I click OK to dismiss the alert, I get an Information window with message "SecureVNC Plugin authentication failure (no plugin interface available)".

I went back and downloaded the encryptionplugins.zip and put the 64-bit version from there in place (even though it seems to be a solid decade old!) and the outgoing connection still fails in the same way; I also tried downgrading to 1.6.4.0 with the same result. When I initiate a connection from the viewer, it connects without issue unless "Use 56-Blowfish cipher for passphrase" is enabled, in which case the server crashes. An outgoing connection with just the VNC password and no encryption also works fine.

"Reverse VNC" is the only use I have for this software but I really need it to be encrypted, so please help me get this working. I also don't understand why the password or passphrase is still required with exactly one public/private key pair properly placed, so an explanation would be helpful; it was requested by someone else previously (https://forum.uvnc.com/viewtopic.php?p=112399#p112399) but never provided. Thank you in advance for both!

[RudiDeVos]

UltraVNC SecureVNC Plugin Testing
For simplicity, I run server and viewer on the same PC to avoid firewall issues on both PCs.

Step 1 — Test Normal (Direct) Connection
Server: Settings → Security, enable [✓] Use encryption and select SecureVNCPlugin64.dsm (the only plugin in the x64 folder, which is the correct one).
Restart the server — plugin activation requires a restart. (This should be investigated to see if it can be done without a restart.)
Don't configure the plugin yet — just enable encryption with the VNC password.
Viewer: Tab Security, enable [✓] Use encryption and select SecureVNCPlugin64.dsm (same plugin).
Connect.
Result: Status window shows AES-256... — connection works.
Step 2 — Test Reverse Connection
Viewer: Go to tab Listen and press Start Listening.
Server (still running in systray): Select Add New Connection, enter localhost, and connect.
Result: Status window shows AES-256... — reverse connection works.
Step 3 — Create Keys and Configure Plugin
Server: Tab Security → click Configure Encryption.
Set options:
[✓] AES 128
[✓] 256-bit
Set a passphrase (this allows setting a long password).
Click Generate Client Authentication Key — this generates two keys:
20260704_Server_ClientAuth.pubkey ← key for the server
20260704_Viewer_ClientAuth.pkey ← key for the viewer
The caption shows the ini file path: path\ultravnc.ini. Opening this file shows the configuration and long password (scrambled):


UseDSMPlugin=1
DSMPlugin=SecureVNCPlugin64.dsm
DSMPluginConfig=SecureVNC;0;0x00104001;blablabla
Step 4 — Retest with Keys
Repeat Step 2 (reverse connection).
You now need to enter the long password.
Result: Works — connection succeeds with the long password.
The status window shows the selected encryption options.
Both direct and indirect connections work with the configured plugin.

Note on Blowfish-only Mode
[ ] Use 56-bit Blowfish only — configuring the server via the viewer's configure button does not work. This must be done on the server side, as the server needs to know the selections and save them in its own ini file.