Panic Mode On (78) Server Problems?

Message boards : Number crunching : Panic Mode On (78) Server Problems?

To post messages, you must log in.

Previous · 1 . . . 15 · 16 · 17 · 18 · 19 · 20 · 21 . . . 22 · Next

AuthorMessage
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 7486
Credit: 91,091,755
RAC: 46,398
Australia
Message 1306162 - Posted: 14 Nov 2012, 18:48:42 UTC - in response to Message 1306147.  
Last modified: 14 Nov 2012, 18:56:06 UTC

After almost an hour of button clicking i was finally able to report all the tasks on my second machine.
Niether is able to get any new work- every single request resulsts in a Scheduler Timeout. I expect to run out of work completely in the next 40 min on one system, the other later today.



Please, please, please can someone let the satff know that limiting the number of tasks hasn't helped in the slightest. When it does start to help- it will only be becasue everyone is out of work.
Until the Scheduler is fixed they need to stop all AP production & distribution. They need to fix the Scheduler problem.


EDIT- this problem only started 3 (or was it 4?) weeks ago ofter the weekly outage. Whatever changes they did then to cause the problem, please undo them.


Grant
Darwin NT

ID: 1306162 · Report as offensive
rob smithProject Donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 13335
Credit: 154,725,982
RAC: 117,353
United Kingdom
Message 1306167 - Posted: 14 Nov 2012, 18:53:34 UTC

Task limits may be helping a little.
BUT, stopping AP production did nothing over the weekend, I was still suffering server side time-outs and very slow delivery with no AP production, not only that the number of ghosts in my possession went up, from less than 5 to about 50.


Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

ID: 1306167 · Report as offensive
WezH
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 571
Credit: 64,164,243
RAC: 12,016
Finland
Message 1306187 - Posted: 14 Nov 2012, 19:49:59 UTC - in response to Message 1306167.  

Task limits may be helping a little.
BUT, stopping AP production did nothing over the weekend, I was still suffering server side time-outs and very slow delivery with no AP production, not only that the number of ghosts in my possession went up, from less than 5 to about 50.


Well, Rob, last AP unit was produced in Sunday, and late Monday/early Tuesday Cricket dropped down and hosts were getting their tasks.

This is free speech forum, so do Your own assumptions.
"Please keep Your signature under four lines so Internet traffic doesn't go up too much"

- In 1992 when I had my first e-mail address -

ID: 1306187 · Report as offensive
tbret
Volunteer tester
Avatar

Send message
Joined: 28 May 99
Posts: 3373
Credit: 248,478,648
RAC: 19,869
United States
Message 1306198 - Posted: 14 Nov 2012, 20:21:20 UTC - in response to Message 1306167.  

Task limits may be helping a little.
BUT, stopping AP production did nothing over the weekend, I was still suffering server side time-outs and very slow delivery with no AP production, not only that the number of ghosts in my possession went up, from less than 5 to about 50.


I'm a little concerned to see you report this.

My ability to get to the scheduler went from nothing to everything.

Are you using a proxy and is it still working?

I run a lot of computers and they all seem to have-had a much different experience than you had.

ID: 1306198 · Report as offensive
Cherokee150

Send message
Joined: 11 Nov 99
Posts: 139
Credit: 34,312,188
RAC: 19,930
United States
Message 1306212 - Posted: 14 Nov 2012, 21:05:53 UTC - in response to Message 1306107.  

Regarding my earlier post, Message 1306107, I will also add that I did notice the massive increase in tasks sent me in the days leading up to all our current problems. At the time, I thought it might be due to the shorties, but now I realize the schedulers were sending me all those tasks regardless of the estimated run time.

Perhaps I'm making an incorrect assumption here, but it would seem to me that, if the schedulers were sending thousands of hosts nearly three times the correct number of units, it would cause some of the problems we have been getting.

There have always been problems with getting uploads and downloads, and there have also been many times APs were sent out. The real disaster seems to be something newly introduced, along with the newly imposed limits of 100/CPU-100/GPU.

Please, let's get some discussion going on this observation of mine. Can anyone else find a similar history in your logs? It might not be readily apparent to the really big crunchers because you usually have problems keeping your caches full. What about others, though?

Thanks! :)

ID: 1306212 · Report as offensive
Profile Brother Frank

Send message
Joined: 10 Dec 11
Posts: 26
Credit: 15,142,410
RAC: 0
United States
Message 1306261 - Posted: 14 Nov 2012, 23:32:18 UTC - in response to Message 1306198.  

Cherokee150,

I know something went very seriously wrong after an update or maintenance period three or four weeks ago. Things went very well for a few hours and then I stopped getting work. Seti@Home has never been right since. My notebooks without dedicated graphics cards are getting work in and then back out fine. One has a RAC of about 700; another has a RAC of about 1250 and then a single core machine has a RAC of about 175. (That's an older Pentium machine about ten years old). My desktops with Nvidia GTX 550Ti Cards are the ones having a lot of trouble. My Core Duo with one Nvidia GTX 550 Ti is about to run out of work and is getting lots of time outs and some transient https failures. It has a RAC of about 7,400, but normally has been around 8,000 My i7 2600k machine with two Nvidia GTX 550's normally has a RAC of 22,000 to 24,000, but is down to about 18,000 now. Again, lots of time outs and no work available messages, transient https failures, and time outs as well as backoffs. I have had to play with the NNT button a lot with the desktops to keep them in work, but that tends to foul up the getting work routine. I also have a newer notebook with an i7-2670qm processor and an Nvidia 525m graphics processor. It tends to keep its queue full, but sometimes I've had to play with the NNT and update buttons. I think I get upset with it all and kinda forget for a few minutes that it runs pretty much ok if I leave it unattended, but just check in twice a day. It has a RAC of just about 5,000. I don't know for sure about the number of ghosts, but I believe both the desktops have a lot of them because I see a message every so often about recovering lost work or jobs. With the new notebook running at about 5,000 RAC I should have an overall RAC of about 37,000 or close, but I am running several thousand behind that even though we run 24/7. This is another indication to me that something might be pretty wrong in getting work credit or sending work back. I don't know what is wrong. I am not a technical or database expert, but like I said near the start of this message something very wrong happened with a maintenance cycle around 4 or 5 weeks ago and the problem has remained in both sending work out and getting new work. Brother Frank.

Black, white, and tabby kitties here are worried about me. Henri, my big tabby, lies on my chest comforting me from PSA (Post Seti Anxiety).

ID: 1306261 · Report as offensive
zoom314
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 56760
Credit: 40,746,426
RAC: 5,110
United States
Message 1306270 - Posted: 15 Nov 2012, 0:18:21 UTC

I can only get some work from S@H, I've got 40 that won't report, I could for a bit with a proxy get some work, but that went 403 after a day or so, which I assume is an html 403 which means forbidden, so most of the time I've been getting Milkyway work instead. Matt please hurry back, S@H has gone to hell in a handbag...


Pluto is still a planet

Beep! Beep!

ID: 1306270 · Report as offensive
chromespringer
Avatar

Send message
Joined: 3 Dec 05
Posts: 296
Credit: 44,809,643
RAC: 20,128
United States
Message 1306293 - Posted: 15 Nov 2012, 1:51:12 UTC

at the time of this posting, with 0 tasks to process and 0 tasks to report in BOINC Manager, i have 134 ghost tasks that have a deadline of 11/17 and 178 ghost tasks that have a deadline of 11/18. these tasks are allocated to mach xxxx033 at a rate of 20 per scheduler request.
i have only successfully contacted project 3 times today due to scheduler time outs, for a total of 60 cpu/gpu tasks downloaded and completed .. at this rate i will have approx. 14 error-ed time outs on 11/17 and 118 error-ed time outs on 11/18 (if my math is correct) :)
the last successful download i received was at 2:41 pm mst today. all uploads are painfully completed utilizing NNT.


ID: 1306293 · Report as offensive
Profile Dirk Sadowski
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7066
Credit: 101,439,037
RAC: 80,734
Germany
Message 1306308 - Posted: 15 Nov 2012, 3:49:29 UTC

Ops .. - since ~ 4 hours no U/L & D/L activity shown on the Cricket graph ..


My BOINC:
U/L - OK
D/L - a few tries needed for complete
Report & Request - successful, a few not, successful, a few not ..


* Best regards! :-) * Sutaru Tsureku, team seti.international founder. * Optimize your PC for higher RAC. * SETI@home needs your help. *


ID: 1306308 · Report as offensive
Lionel

Send message
Joined: 25 Mar 00
Posts: 665
Credit: 351,027,481
RAC: 139,661
Australia
Message 1306319 - Posted: 15 Nov 2012, 4:36:13 UTC

.
.
Scheduler request failed: Timeout was reached
.
.
.
.
Scheduler request failed: HTTP Internal Server Error
.
.
.
.
Scheduler request failed: Failure when receiving data from peer
.
.


ID: 1306319 · Report as offensive
Lee Gresham
Avatar

Send message
Joined: 12 Aug 03
Posts: 159
Credit: 130,116,228
RAC: 0
United States
Message 1306334 - Posted: 15 Nov 2012, 5:31:42 UTC

Can someone point me to a forum entry detailing how to limit number of files being reported. I've looked thru the boinc & seti XMLs and didn't see a likely entry. I know I've done this before but as they say "the memory is the second thing to go" I forget what's first!

Thanks..............


Delta-V

ID: 1306334 · Report as offensive
tbret
Volunteer tester
Avatar

Send message
Joined: 28 May 99
Posts: 3373
Credit: 248,478,648
RAC: 19,869
United States
Message 1306336 - Posted: 15 Nov 2012, 5:35:14 UTC - in response to Message 1306334.  

Can someone point me to a forum entry detailing how to limit number of files being reported. I've looked thru the boinc & seti XMLs and didn't see a likely entry. I know I've done this before but as they say "the memory is the second thing to go" I forget what's first!

Thanks..............


<cc_config>
<options>
<max_tasks_reported>N</max_tasks_reported>
</options>
</cc_config>

N=whatever number you set.


You have to add the line.

ID: 1306336 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 7486
Credit: 91,091,755
RAC: 46,398
Australia
Message 1306338 - Posted: 15 Nov 2012, 5:42:28 UTC - in response to Message 1306334.  
Last modified: 15 Nov 2012, 5:43:54 UTC

I got back from work. One system completely out of work, the other running out of CPU work (GPU ran out ages ago).
Every single Scheduler request while i was at work timed out.

I set No New Tasks & managed to report them all, although one machine took almost 5 min before it got a response. I then allowed new work & one machine then requested work & got some, the other requested work and got a Scheduler Timeout.

The graphs show AP work still going out. I'm quite sure if the AP work stopped going out, i'd then be able to contact the Scheduler again.


As someone pointed out some time ago- Synergy has a lot of processes, and i think it's got more than it can hamdle.


Grant
Darwin NT

ID: 1306338 · Report as offensive
rob smithProject Donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 13335
Credit: 154,725,982
RAC: 117,353
United Kingdom
Message 1306347 - Posted: 15 Nov 2012, 6:31:36 UTC

Grant, in another thread you suggested using a proxy - have you considered that this problem may be a routing problem between your empty cruncher and the lab and that the cruncher concerned is attempting to use a different one to those whose connection is OK - I have to crunchers sharing the same internet connection, but when I do a tractroute they show they actually, consistently, use different paths through the internet to almost every server I look at....


Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

ID: 1306347 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 7486
Credit: 91,091,755
RAC: 46,398
Australia
Message 1306352 - Posted: 15 Nov 2012, 6:43:39 UTC - in response to Message 1306347.  

Grant, in another thread you suggested using a proxy - have you considered that this problem may be a routing problem between your empty cruncher and the lab and that the cruncher concerned is attempting to use a different one to those whose connection is OK

I used the proxy for both systems.
Proxy- both get a response, no proxy- neither get a response when trying to report & request work. No proxy- both sometimes get work if not reporting tasks.

So we'll go with the proxy, till it gets canned & i have to find another one.
Grant
Darwin NT

ID: 1306352 · Report as offensive
Lee Gresham
Avatar

Send message
Joined: 12 Aug 03
Posts: 159
Credit: 130,116,228
RAC: 0
United States
Message 1306357 - Posted: 15 Nov 2012, 6:58:58 UTC - in response to Message 1306336.  

<cc_config>
<options>
<max_tasks_reported>N</max_tasks_reported>
</options>
</cc_config>

N=whatever number you set.


You have to add the line.
[/quote]



Thanks Again


Delta-V

ID: 1306357 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 2871
Credit: 10,621,745
RAC: 322
United States
Message 1306360 - Posted: 15 Nov 2012, 7:05:45 UTC

My requests in the past 24 hours have been about 50/50 time-out and success. Most of the time-outs end up with a "resent lost task" on the next successful contact. This has been how it has been working all day and evening, until I managed to be issued an AP with a six minute deadline. Well there goes my consecutive valid streak of ~1100. "didn't resend lost task..(expired)" *sigh*


Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)

ID: 1306360 · Report as offensive
Profile Khangollo
Avatar

Send message
Joined: 1 Aug 00
Posts: 245
Credit: 36,410,524
RAC: 0
Slovenia
Message 1306370 - Posted: 15 Nov 2012, 7:57:07 UTC
Last modified: 15 Nov 2012, 7:59:32 UTC

All my tasks got auto-abandoned on the server side over night, once again!!!
http://setiathome.berkeley.edu/results.php?hostid=5323998&offset=0&show_names=0&state=6&appid=
I've had it with this project now.


ID: 1306370 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 7486
Credit: 91,091,755
RAC: 46,398
Australia
Message 1306371 - Posted: 15 Nov 2012, 8:00:09 UTC - in response to Message 1306370.  

All my tasks got auto-aborted over night, once again!!!
http://setiathome.berkeley.edu/results.php?hostid=5323998&offset=0&show_names=0&state=6&appid=
I've had it with this project now.

Even with all the wierdness going on, i've yet to have that happen- not even once.
You haven't got any software that mucks about with your system clock at all?
Grant
Darwin NT

ID: 1306371 · Report as offensive
Profile Khangollo
Avatar

Send message
Joined: 1 Aug 00
Posts: 245
Credit: 36,410,524
RAC: 0
Slovenia
Message 1306372 - Posted: 15 Nov 2012, 8:02:43 UTC - in response to Message 1306371.  

All my tasks got auto-aborted over night, once again!!!
http://setiathome.berkeley.edu/results.php?hostid=5323998&offset=0&show_names=0&state=6&appid=
I've had it with this project now.

Even with all the wierdness going on, i've yet to have that happen- not even once.
You haven't got any software that mucks about with your system clock at all?

No, I don't. Like I said, it happened on the server side! My host didn't even succeed in making a scheduler request. This is the second time it happened. I'm not the only one this happened to; there were other users reporting the same thing.

ID: 1306372 · Report as offensive
Previous · 1 . . . 15 · 16 · 17 · 18 · 19 · 20 · 21 . . . 22 · Next

Message boards : Number crunching : Panic Mode On (78) Server Problems?


 
©2016 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.