[SOLVED] Repeater Crashing

[Prisma]

(Sorry, don't know where else to post this, so I took the newest tree)

We're using repeater 1.1.0.4 over aprox. 2 years and have still problems with random crashes. After disconnecting the last session the repeater sometimes runs with 100% processor time of one core. Sometimes not. But that's not the central problem. He keeps working anyway. The main problem are sporadic crashes. I think the crash happens on connect from client or server, not disconnect.

The Eventlog is attached. Is there a possibility to enable a debug logging to get closer to that problem? Do we use the newest repeater or is a newer version (possibly a beta?) available?

Code: Select all

<?xml version="1.0" encoding="utf-8"?>
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2013-04-29T11:32:17.000Z" />
    <EventRecordID>91384</EventRecordID>
    <Channel>Application</Channel>
    <Computer>xx.yy.zz</Computer>
    <Security />
  </System>
  <EventData>
    <Data>repeater.exe</Data>
    <Data>1.1.0.4</Data>
    <Data>4dc59522</Data>
    <Data>ntdll.dll</Data>
    <Data>6.0.6002.18541</Data>
    <Data>4ec3e3d5</Data>
    <Data>c0000374</Data>
    <Data>000b06b7</Data>
    <Data>df0</Data>
    <Data>01ce449933bf807b</Data>
  </EventData>
</Event>

[Rudi De Vos]

yep, you best post it with the latest version.

1.1.0.4, was that the repeater with the web gui ?

The embedded webserver i used had plenty of error, if found most of them but i'm not sure all of them are gone.
( I didn't wrote that part, so it's a hell to debug)
The problem seems to be that some data connecting to that port and not being browser data behave bad, if that's the case,
try to use another port that's not used by port scanners.

[Prisma]

The webservers port isn't opened to the internet. It's rather unlikely that we have portscanners within our private network. Only the pure repeater ports are opened. Here is the setup:

Hostname: xx
IP Address #1: 192.168.1.5
IP Address #2: 192.168.1.119
Listen Port Viewer: 5901
Listen Port Server: 5500
Web Server : 8088
Use comment as extra viewer check: 0

Is there possibly a bug related to the secondary IPv4 above? But the connects to the 2 repeater ports are always from the internet. And they are always forwarded only to 192.168.1.5.

[Prisma]

Please Rudi, we have now crashes multiple times a day.

Would it be possible to get an EXE writing some generic debug information? We have to get closer to that problem.
Are the repeater sources also available for free? Is it possible to build repeater with VS 2008/2010/1012 Express?

[Rudi De Vos]

I'm sorry, but it is hard to find bugs just reading the source and static analysers doesn't reveal a lot.

We actual should be able to run the the repeater in a life environment insite the VS debugger.
On a crash you can see what error happen and debug the var content..

Yes, sources are online and can be downloaded but i never tested it with an express version.
Does it have a good debugger gui ?
You could install a time limited try version of VS.

I can create a version that generate dumps on crash and import then later in my debug environment.

Adding debug info never gonna tell the content of a var or if some functions are used after deletion, you
need to debug or create dump files.

[Prisma]

Does it have a good debugger gui ?

I guess it's the normal VS debugger. Only some resources and some tools are only availably within the full version.

I can create a version that generate dumps on crash and import then later in my debug environment.

It's up to you to decide about the fastest way. We can try it with a version generating dumps on crash. Possibly that's enough. Possibly not. We are the latest repeater sources located? I found only some old sources from 2007... Could you please post a svn link?

[Rudi De Vos]

It's on the website
http://www.uvnc.com/downloads.html
click repeater
click download repeater

Latest update is 2013

The exe is distributed togther with the source.
If you don't have that source, are you sure you are using the latest version ?

[Rudi De Vos]

First try it with this exe.
If it still crash try to debug it in the express version.
If you are unable to debug it i will create a special version, this take a little time.

[Prisma]

OK, Rudi, thx. Will be done

[Prisma]

Short and first notice, running distributer.exe in Debugger. Crash happens in function:

Code: Select all

DWORD WINAPI do_repeater_wait(LPVOID lpParam)

in line:

Code: Select all

remote=inout->remote;

[Prisma]

But I had to stop debugging and restart the service because we wanted continue working...
Next time I try to find some invalid variable or pointers or whatever.

Or is this info enough?

[Rudi De Vos]

the it looks like "inout" is null or was already deleted and is invalid.

Yes, this help to narrow the search.

[Rudi De Vos]

inout can't be null or invalid as it is defined as var in the beginning of the functions
server_listen() and mode12listener()
As long as the function are in a loop waiting for new connection inout exist.
while(notstopped)
{
}

notstopped must be false to stop looping, but only insite the loop do_repeater_wait
is called.... require more info to check it.


How many simultanious connections do you have waiting ?
connected ?

[Rudi De Vos]

Did some debugging and found that
after
accept( socket_server_listen and
accept(socket
I needed to add a Sleep(1000) , else the viewer sometimes stay at connecting while it should be Negotiate protocol version.
-----------
after accept( socket_server_listen...)
Sleep(1000);<<<<<<<<<<<<<<<<< added
---------------------------

Test done with repeater in a virtualpc.

Possible this is not related to the crash but it cause the connection sometimes to fail.

[Prisma]

Hi Rudi,

although this is possibly not related to our problem, did you update the sources? In Germany we had a holiday on Thursday. So I'll be able to continue debugging not before Monday. But it would be nice to continue with updated sources.

[Prisma]

How many simultanious connections do you have waiting ?
connected ?

to answer question: not more than 4-8 simultaneously. Normal is only a single connection or up to 3. Our helpdesk is only started on demand. So, not all of our 500 customers are hanging on the repeater all the time. In case of the crash I'd say it should have been a single connection with a viewer waiting before the server was started. the crash seemed to happen on arrival of the server. Not sure...

[Rudi De Vos]

download updated with extra timout on accept()

[Prisma]

Regarding our second 100%-processor-usage-problem. For your information, while this loop no connection to the web front end was possible:



Here's the Loop

[Rudi De Vos]

the web interface was token from codeproject... only it was so buggy.

corrected and reuploaded.
FD_isset=true and recv()=0 -remote closed connect , so we also need to close the socket.

[Prisma]

New compiled and running in debugger. For your information:
Since you applied the timeout patch, no crash happened again. Before the patch the crashes happened very frequently. Seems to be better or solved.

[Prisma]

No more crashes or 100% processor usage since the applied patches. May be the crashes have been caused by a DOS attacks stopping randomly after the applied patch. But what are the odds !?!

Good work Rudi, thank you.

[naveedm1]

Hi Every one,
I am running the latest repeater on win2008 Server x64
Listen Port Viewer: 5900
Listen Port Server: 80
Web Server : 81
Only one winvnc server connected for test. After 6 hours or so it crashes.
I did not connect the server with Viewer. Just wanted to isolate the issue.

I also noticed if I run any version of repeater in win98 Compatibility mode it never crashes. But after 10 or 15 hours loses all the servers in the list even though servers are live and trying to connect.
I tried to connect servers to repeater via service or via command line or using GUI same results.
Any one suggest what should I do? I looked at the code and also noticed many places malloc is not correctly defined , I think code was originated from Linux world.

[Prisma]

Hmm Rudi had applied some changes and our crashes were gone. But we still have problems with 100% CPU utilization. For this reason, we run distributer.exe in VC2010 again to find the error.

Any further investigation of you, we would really appreciate. The repeater is an enterprise function. Therefore, we are very keen that it runs 100% stable.

[Prisma]

Regarding the 100% processor usage. Debugged following:

* Thread "TreadstartWeb" definitely causes the usage. If you lock the thread the usage goes down.
* In Loop "while(notwebstopped)" sessions = -1001 therefore "detrap()" is called.
* detrap() makes nothing meaningful (printf("dtrap - need breakpoint");)
* Webserver closes connections (in our case 8088) not correct (see picture)

[Rudi De Vos]

The problem is the webinterface....The repeater part himself is stable.

I just wonder if it would not be usefull to add the option to disable/enable the web interface
from the repeater tray. Even if it's just to check if it run stable without.

[Prisma]

Yeah, I know. You can lock the webserver thread, but the repeater keeps working.

Disabling the web interface part isn't useful. We often need the web interface to check whether a connection comes in, and to determine why possibly not and so on....
AND this is the keyfeature of the new repeater. How to disable a keyfeature?

Please don't kill me, but has WebIO really a good reputation? What about switching to Mongoose? The footprint of mongoose is very very small.
I know you guys are discussing about a UDP solution for v2, possibly a repeater isn't needed any more in future. But that's the future. Until now nobody knows what exactly will happen.

[Prisma]

Continuing processor usage problem in thread this thread: https://forum.ultravnc.net/viewtopic.php?t=30385