After more 1 000 000 (one million) views on forum for 1.5.0.x development versions... and 1.6.1.0, 1.6.3.0-dev versions A new stable version, UltraVNC 1.6.4.0 and UltraVNC SC 1.6.4.0 have been released: https://forum.uvnc.com/viewtopic.php?t=38095 Feedback is welcome
You may also want to update your machine's video drivers as well. I've seen instances where the video looks really choppy on VNC, and upgrading the drivers on the viewer resolved this.
Any logs on the ISA server in terms of traffic being directed to the VISTA machine ? You could enable TELNET on the Vista machine, and see if you can gain a connection to this on the outside using "telnet x.x.x.x 5500" where "x.x.x.x" is the IP Address.