You may also want to update your machine's video drivers as well. I've seen instances where the video looks really choppy on VNC, and upgrading the drivers on the viewer resolved this.
Any logs on the ISA server in terms of traffic being directed to the VISTA machine ? You could enable TELNET on the Vista machine, and see if you can gain a connection to this on the outside using "telnet x.x.x.x 5500" where "x.x.x.x" is the IP Address.