Panic Mode On (78) Server Problems?


log in

Advanced search

Message boards : Number crunching : Panic Mode On (78) Server Problems?

Previous · 1 . . . 15 · 16 · 17 · 18 · 19 · 20 · 21 . . . 22 · Next
Author Message
Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5831
Credit: 59,453,219
RAC: 47,880
Australia
Message 1306162 - Posted: 14 Nov 2012, 18:48:42 UTC - in response to Message 1306147.
Last modified: 14 Nov 2012, 18:56:06 UTC

After almost an hour of button clicking i was finally able to report all the tasks on my second machine.
Niether is able to get any new work- every single request resulsts in a Scheduler Timeout. I expect to run out of work completely in the next 40 min on one system, the other later today.



Please, please, please can someone let the satff know that limiting the number of tasks hasn't helped in the slightest. When it does start to help- it will only be becasue everyone is out of work.
Until the Scheduler is fixed they need to stop all AP production & distribution. They need to fix the Scheduler problem.


EDIT- this problem only started 3 (or was it 4?) weeks ago ofter the weekly outage. Whatever changes they did then to cause the problem, please undo them.
____________
Grant
Darwin NT.

rob smithProject donor
Volunteer tester
Send message
Joined: 7 Mar 03
Posts: 8431
Credit: 57,518,842
RAC: 73,924
United Kingdom
Message 1306167 - Posted: 14 Nov 2012, 18:53:34 UTC

Task limits may be helping a little.
BUT, stopping AP production did nothing over the weekend, I was still suffering server side time-outs and very slow delivery with no AP production, not only that the number of ghosts in my possession went up, from less than 5 to about 50.
____________
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

WezH
Volunteer tester
Send message
Joined: 19 Aug 99
Posts: 96
Credit: 3,906,885
RAC: 14,521
Finland
Message 1306187 - Posted: 14 Nov 2012, 19:49:59 UTC - in response to Message 1306167.

Task limits may be helping a little.
BUT, stopping AP production did nothing over the weekend, I was still suffering server side time-outs and very slow delivery with no AP production, not only that the number of ghosts in my possession went up, from less than 5 to about 50.


Well, Rob, last AP unit was produced in Sunday, and late Monday/early Tuesday Cricket dropped down and hosts were getting their tasks.

This is free speech forum, so do Your own assumptions.
____________

tbretProject donor
Volunteer tester
Avatar
Send message
Joined: 28 May 99
Posts: 2807
Credit: 211,564,412
RAC: 170,040
United States
Message 1306198 - Posted: 14 Nov 2012, 20:21:20 UTC - in response to Message 1306167.

Task limits may be helping a little.
BUT, stopping AP production did nothing over the weekend, I was still suffering server side time-outs and very slow delivery with no AP production, not only that the number of ghosts in my possession went up, from less than 5 to about 50.


I'm a little concerned to see you report this.

My ability to get to the scheduler went from nothing to everything.

Are you using a proxy and is it still working?

I run a lot of computers and they all seem to have-had a much different experience than you had.

Cherokee150
Send message
Joined: 11 Nov 99
Posts: 103
Credit: 24,795,902
RAC: 26,905
United States
Message 1306212 - Posted: 14 Nov 2012, 21:05:53 UTC - in response to Message 1306107.

Regarding my earlier post, Message 1306107, I will also add that I did notice the massive increase in tasks sent me in the days leading up to all our current problems. At the time, I thought it might be due to the shorties, but now I realize the schedulers were sending me all those tasks regardless of the estimated run time.

Perhaps I'm making an incorrect assumption here, but it would seem to me that, if the schedulers were sending thousands of hosts nearly three times the correct number of units, it would cause some of the problems we have been getting.

There have always been problems with getting uploads and downloads, and there have also been many times APs were sent out. The real disaster seems to be something newly introduced, along with the newly imposed limits of 100/CPU-100/GPU.

Please, let's get some discussion going on this observation of mine. Can anyone else find a similar history in your logs? It might not be readily apparent to the really big crunchers because you usually have problems keeping your caches full. What about others, though?

Thanks! :)

Profile Brother Frank
Send message
Joined: 10 Dec 11
Posts: 26
Credit: 15,142,410
RAC: 0
United States
Message 1306261 - Posted: 14 Nov 2012, 23:32:18 UTC - in response to Message 1306198.

Cherokee150,

I know something went very seriously wrong after an update or maintenance period three or four weeks ago. Things went very well for a few hours and then I stopped getting work. Seti@Home has never been right since. My notebooks without dedicated graphics cards are getting work in and then back out fine. One has a RAC of about 700; another has a RAC of about 1250 and then a single core machine has a RAC of about 175. (That's an older Pentium machine about ten years old). My desktops with Nvidia GTX 550Ti Cards are the ones having a lot of trouble. My Core Duo with one Nvidia GTX 550 Ti is about to run out of work and is getting lots of time outs and some transient https failures. It has a RAC of about 7,400, but normally has been around 8,000 My i7 2600k machine with two Nvidia GTX 550's normally has a RAC of 22,000 to 24,000, but is down to about 18,000 now. Again, lots of time outs and no work available messages, transient https failures, and time outs as well as backoffs. I have had to play with the NNT button a lot with the desktops to keep them in work, but that tends to foul up the getting work routine. I also have a newer notebook with an i7-2670qm processor and an Nvidia 525m graphics processor. It tends to keep its queue full, but sometimes I've had to play with the NNT and update buttons. I think I get upset with it all and kinda forget for a few minutes that it runs pretty much ok if I leave it unattended, but just check in twice a day. It has a RAC of just about 5,000. I don't know for sure about the number of ghosts, but I believe both the desktops have a lot of them because I see a message every so often about recovering lost work or jobs. With the new notebook running at about 5,000 RAC I should have an overall RAC of about 37,000 or close, but I am running several thousand behind that even though we run 24/7. This is another indication to me that something might be pretty wrong in getting work credit or sending work back. I don't know what is wrong. I am not a technical or database expert, but like I said near the start of this message something very wrong happened with a maintenance cycle around 4 or 5 weeks ago and the problem has remained in both sending work out and getting new work. Brother Frank.

Black, white, and tabby kitties here are worried about me. Henri, my big tabby, lies on my chest comforting me from PSA (Post Seti Anxiety).

zoom314Project donor
Avatar
Send message
Joined: 30 Nov 03
Posts: 46393
Credit: 36,742,544
RAC: 4,856
United States
Message 1306270 - Posted: 15 Nov 2012, 0:18:21 UTC

I can only get some work from S@H, I've got 40 that won't report, I could for a bit with a proxy get some work, but that went 403 after a day or so, which I assume is an html 403 which means forbidden, so most of the time I've been getting Milkyway work instead. Matt please hurry back, S@H has gone to hell in a handbag...
____________
My Facebook, War Commander, 2015

chromespringerProject donor
Avatar
Send message
Joined: 3 Dec 05
Posts: 269
Credit: 21,803,273
RAC: 46,912
United States
Message 1306293 - Posted: 15 Nov 2012, 1:51:12 UTC

at the time of this posting, with 0 tasks to process and 0 tasks to report in BOINC Manager, i have 134 ghost tasks that have a deadline of 11/17 and 178 ghost tasks that have a deadline of 11/18. these tasks are allocated to mach xxxx033 at a rate of 20 per scheduler request.
i have only successfully contacted project 3 times today due to scheduler time outs, for a total of 60 cpu/gpu tasks downloaded and completed .. at this rate i will have approx. 14 error-ed time outs on 11/17 and 118 error-ed time outs on 11/18 (if my math is correct) :)
the last successful download i received was at 2:41 pm mst today. all uploads are painfully completed utilizing NNT.
____________

Profile [seti.international] Dirk SadowskiProject donor
Volunteer tester
Avatar
Send message
Joined: 6 Apr 07
Posts: 7091
Credit: 60,531,852
RAC: 18,729
Germany
Message 1306308 - Posted: 15 Nov 2012, 3:49:29 UTC

Ops .. - since ~ 4 hours no U/L & D/L activity shown on the Cricket graph ..


My BOINC:
U/L - OK
D/L - a few tries needed for complete
Report & Request - successful, a few not, successful, a few not ..


* Best regards! :-) * Sutaru Tsureku, team seti.international founder. * Optimize your PC for higher RAC. * SETI@home needs your help. *
____________
BR

SETI@home Needs your Help ... $10 & U get a Star!

Team seti.international

Das Deutsche Cafe. The German Cafe.

Lionel
Send message
Joined: 25 Mar 00
Posts: 545
Credit: 230,939,685
RAC: 235,456
Australia
Message 1306319 - Posted: 15 Nov 2012, 4:36:13 UTC

.
.
Scheduler request failed: Timeout was reached
.
.
.
.
Scheduler request failed: HTTP Internal Server Error
.
.
.
.
Scheduler request failed: Failure when receiving data from peer
.
.

____________

Lee Gresham
Avatar
Send message
Joined: 12 Aug 03
Posts: 131
Credit: 101,284,697
RAC: 39,230
United States
Message 1306334 - Posted: 15 Nov 2012, 5:31:42 UTC

Can someone point me to a forum entry detailing how to limit number of files being reported. I've looked thru the boinc & seti XMLs and didn't see a likely entry. I know I've done this before but as they say "the memory is the second thing to go" I forget what's first!

Thanks..............
____________
Delta-V

tbretProject donor
Volunteer tester
Avatar
Send message
Joined: 28 May 99
Posts: 2807
Credit: 211,564,412
RAC: 170,040
United States
Message 1306336 - Posted: 15 Nov 2012, 5:35:14 UTC - in response to Message 1306334.

Can someone point me to a forum entry detailing how to limit number of files being reported. I've looked thru the boinc & seti XMLs and didn't see a likely entry. I know I've done this before but as they say "the memory is the second thing to go" I forget what's first!

Thanks..............


<cc_config>
<options>
<max_tasks_reported>N</max_tasks_reported>
</options>
</cc_config>

N=whatever number you set.


You have to add the line.

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5831
Credit: 59,453,219
RAC: 47,880
Australia
Message 1306338 - Posted: 15 Nov 2012, 5:42:28 UTC - in response to Message 1306334.
Last modified: 15 Nov 2012, 5:43:54 UTC

I got back from work. One system completely out of work, the other running out of CPU work (GPU ran out ages ago).
Every single Scheduler request while i was at work timed out.

I set No New Tasks & managed to report them all, although one machine took almost 5 min before it got a response. I then allowed new work & one machine then requested work & got some, the other requested work and got a Scheduler Timeout.

The graphs show AP work still going out. I'm quite sure if the AP work stopped going out, i'd then be able to contact the Scheduler again.


As someone pointed out some time ago- Synergy has a lot of processes, and i think it's got more than it can hamdle.
____________
Grant
Darwin NT.

rob smithProject donor
Volunteer tester
Send message
Joined: 7 Mar 03
Posts: 8431
Credit: 57,518,842
RAC: 73,924
United Kingdom
Message 1306347 - Posted: 15 Nov 2012, 6:31:36 UTC

Grant, in another thread you suggested using a proxy - have you considered that this problem may be a routing problem between your empty cruncher and the lab and that the cruncher concerned is attempting to use a different one to those whose connection is OK - I have to crunchers sharing the same internet connection, but when I do a tractroute they show they actually, consistently, use different paths through the internet to almost every server I look at....
____________
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5831
Credit: 59,453,219
RAC: 47,880
Australia
Message 1306352 - Posted: 15 Nov 2012, 6:43:39 UTC - in response to Message 1306347.

Grant, in another thread you suggested using a proxy - have you considered that this problem may be a routing problem between your empty cruncher and the lab and that the cruncher concerned is attempting to use a different one to those whose connection is OK

I used the proxy for both systems.
Proxy- both get a response, no proxy- neither get a response when trying to report & request work. No proxy- both sometimes get work if not reporting tasks.

So we'll go with the proxy, till it gets canned & i have to find another one.
____________
Grant
Darwin NT.

Lee Gresham
Avatar
Send message
Joined: 12 Aug 03
Posts: 131
Credit: 101,284,697
RAC: 39,230
United States
Message 1306357 - Posted: 15 Nov 2012, 6:58:58 UTC - in response to Message 1306336.

<cc_config>
<options>
<max_tasks_reported>N</max_tasks_reported>
</options>
</cc_config>

N=whatever number you set.


You have to add the line.
[/quote]



Thanks Again
____________
Delta-V

Cosmic_Ocean
Avatar
Send message
Joined: 23 Dec 00
Posts: 2268
Credit: 8,720,277
RAC: 4,168
United States
Message 1306360 - Posted: 15 Nov 2012, 7:05:45 UTC

My requests in the past 24 hours have been about 50/50 time-out and success. Most of the time-outs end up with a "resent lost task" on the next successful contact. This has been how it has been working all day and evening, until I managed to be issued an AP with a six minute deadline. Well there goes my consecutive valid streak of ~1100. "didn't resend lost task..(expired)" *sigh*
____________

Linux laptop uptime: 1484d 22h 42m
Ended due to UPS failure, found 14 hours after the fact

Profile Khangollo
Avatar
Send message
Joined: 1 Aug 00
Posts: 245
Credit: 36,410,524
RAC: 0
Slovenia
Message 1306370 - Posted: 15 Nov 2012, 7:57:07 UTC
Last modified: 15 Nov 2012, 7:59:32 UTC

All my tasks got auto-abandoned on the server side over night, once again!!!
http://setiathome.berkeley.edu/results.php?hostid=5323998&offset=0&show_names=0&state=6&appid=
I've had it with this project now.
____________

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5831
Credit: 59,453,219
RAC: 47,880
Australia
Message 1306371 - Posted: 15 Nov 2012, 8:00:09 UTC - in response to Message 1306370.

All my tasks got auto-aborted over night, once again!!!
http://setiathome.berkeley.edu/results.php?hostid=5323998&offset=0&show_names=0&state=6&appid=
I've had it with this project now.

Even with all the wierdness going on, i've yet to have that happen- not even once.
You haven't got any software that mucks about with your system clock at all?
____________
Grant
Darwin NT.

Profile Khangollo
Avatar
Send message
Joined: 1 Aug 00
Posts: 245
Credit: 36,410,524
RAC: 0
Slovenia
Message 1306372 - Posted: 15 Nov 2012, 8:02:43 UTC - in response to Message 1306371.

All my tasks got auto-aborted over night, once again!!!
http://setiathome.berkeley.edu/results.php?hostid=5323998&offset=0&show_names=0&state=6&appid=
I've had it with this project now.

Even with all the wierdness going on, i've yet to have that happen- not even once.
You haven't got any software that mucks about with your system clock at all?

No, I don't. Like I said, it happened on the server side! My host didn't even succeed in making a scheduler request. This is the second time it happened. I'm not the only one this happened to; there were other users reporting the same thing.
____________

Previous · 1 . . . 15 · 16 · 17 · 18 · 19 · 20 · 21 . . . 22 · Next

Message boards : Number crunching : Panic Mode On (78) Server Problems?

Copyright © 2014 University of California