Panic Mode On (80) Server Problems?

Message boards : Number crunching : Panic Mode On (80) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · 13 . . . 24 · Next

AuthorMessage
Profile ivan
Volunteer tester
Avatar

Send message
Joined: 5 Mar 01
Posts: 783
Credit: 348,560,338
RAC: 223
United Kingdom
Message 1325050 - Posted: 5 Jan 2013, 20:36:39 UTC - in response to Message 1325042.  

Just now downloads are back to their normal - about as fast as a snail with arthritis moving across coarse glass paper...

Scheduler response times are still measured in seconds though, here at home at least. I wish they'd start to increase the limits a bit though, my faster machines can't last too long a break at 100 GPU WUs/machine.
ID: 1325050 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13947
Credit: 208,696,464
RAC: 304
Australia
Message 1325058 - Posted: 5 Jan 2013, 20:54:57 UTC - in response to Message 1325050.  

I wish they'd start to increase the limits a bit though, my faster machines can't last too long a break at 100 GPU WUs/machine.

Or 100 per CPU.

Grant
Darwin NT
ID: 1325058 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9959
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1325069 - Posted: 5 Jan 2013, 21:56:42 UTC

Just now downloads are back to their normal - about as fast as a snail with arthritis moving across coarse glass paper...


Hmm that fast!! I was getting speeds of 350K now I'm lucky if it 3K!!
ID: 1325069 · Report as offensive
fscheel

Send message
Joined: 13 Apr 12
Posts: 73
Credit: 11,135,641
RAC: 0
United States
Message 1325255 - Posted: 6 Jan 2013, 13:29:58 UTC

I notice this morning that my pc 6738288 has marked around 100 tasks as abandoned. What caused this and is there anything I need to do? It hasn't abandoned anymore in the past 6-7 hours and appears to be working. Boinc tasks is reporting about 200 tasks on the pc but the web site reports about 120.

thanks...Frank
ID: 1325255 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51540
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1325263 - Posted: 6 Jan 2013, 13:38:26 UTC - in response to Message 1325255.  

I notice this morning that my pc 6738288 has marked around 100 tasks as abandoned. What caused this and is there anything I need to do? It hasn't abandoned anymore in the past 6-7 hours and appears to be working. Boinc tasks is reporting about 200 tasks on the pc but the web site reports about 120.

thanks...Frank

Random task abandonment has been reported before. Nothing you can or need to do on your end. Just keep crunching, and if the servers pick up the problem, they might resend the missing tasks. If not, no worries, they'll get crunched by somebody sooner or later.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1325263 · Report as offensive
fscheel

Send message
Joined: 13 Apr 12
Posts: 73
Credit: 11,135,641
RAC: 0
United States
Message 1325270 - Posted: 6 Jan 2013, 13:45:32 UTC - in response to Message 1325263.  

I notice this morning that my pc 6738288 has marked around 100 tasks as abandoned. What caused this and is there anything I need to do? It hasn't abandoned anymore in the past 6-7 hours and appears to be working. Boinc tasks is reporting about 200 tasks on the pc but the web site reports about 120.

thanks...Frank

Random task abandonment has been reported before. Nothing you can or need to do on your end. Just keep crunching, and if the servers pick up the problem, they might resend the missing tasks. If not, no worries, they'll get crunched by somebody sooner or later.



Thanks
ID: 1325270 · Report as offensive
MikeN

Send message
Joined: 24 Jan 11
Posts: 319
Credit: 64,719,409
RAC: 85
United Kingdom
Message 1325314 - Posted: 6 Jan 2013, 15:07:28 UTC - in response to Message 1325270.  

I notice this morning that my pc 6738288 has marked around 100 tasks as abandoned. What caused this and is there anything I need to do? It hasn't abandoned anymore in the past 6-7 hours and appears to be working. Boinc tasks is reporting about 200 tasks on the pc but the web site reports about 120.

thanks...Frank

Random task abandonment has been reported before. Nothing you can or need to do on your end. Just keep crunching, and if the servers pick up the problem, they might resend the missing tasks. If not, no worries, they'll get crunched by somebody sooner or later.



Thanks

I think you need to abandon all the tasks on your PC by resetting the project. If BOINC has marked them as abandoned you will not get credit for them, nor will any results you return count. I would also reboot the PC as there may be some sort of problem with it.

This happened to me three times in two days over Christmas, always with the same PC. Everytime it got 100 CPU tasks, a couple of hours later they would all be marked abandoned. In the end I suspended CPU processing until I could get back to work to reboot he system and check there was nothing wrong with the PC. It has been fine since.
ID: 1325314 · Report as offensive
Profile Cliff Harding
Volunteer tester
Avatar

Send message
Joined: 18 Aug 99
Posts: 1432
Credit: 110,967,840
RAC: 67
United States
Message 1325326 - Posted: 6 Jan 2013, 16:16:57 UTC

Looks like things are grinding to a halt again. Can't get any GPU d/ls because I have 2 AP tasks that have been stuck for several hours and retries don't help much as it quickly drops to under 1.0 kb then goes into retry status.


I don't buy computers, I build them!!
ID: 1325326 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13947
Credit: 208,696,464
RAC: 304
Australia
Message 1325476 - Posted: 7 Jan 2013, 10:21:23 UTC - in response to Message 1325326.  


Whatever was causing the MB validators to get bogged down appears to have cleared- the backlog is now declining instead of growing. Also the AP work appears to have pretty much cleared, so hopefully the download traffic will settle down shortly.
And whatever they did to the splitters after that last outage appears to have worked- i can't remember a time when the splitters were able to pump out 60 WU/s time after time, but now the Ready to send buffer is back down to normal levels, that's what they're doing. 50/s is the worst so far, for a quite a while there they were struggling to do 20/s.
Grant
Darwin NT
ID: 1325476 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22798
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1325623 - Posted: 7 Jan 2013, 22:24:45 UTC

The current round of sick download performance was happening before the APs started to move, so you can forget them as the culprit - the root cause is a very poorly though through and implemented download scheduling algorithm that can do nothing apart from cause problems.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1325623 · Report as offensive
.clair.

Send message
Joined: 4 Nov 04
Posts: 1300
Credit: 55,390,408
RAC: 69
United Kingdom
Message 1325661 - Posted: 8 Jan 2013, 1:58:31 UTC - in response to Message 1325623.  

- the root cause is a very poorly though through and implemented download scheduling algorithm that can do nothing apart from cause problems.

Iz that inside info ?
If they have found another gremlin with its neck stuck in a bottle,
the hIT men can have a smashing time with it,
unless the Trainman says otherwize :(
ID: 1325661 · Report as offensive
Profile Cliff Harding
Volunteer tester
Avatar

Send message
Joined: 18 Aug 99
Posts: 1432
Credit: 110,967,840
RAC: 67
United States
Message 1325670 - Posted: 8 Jan 2013, 3:40:15 UTC - in response to Message 1325661.  

- the root cause is a very poorly though through and implemented download scheduling algorithm that can do nothing apart from cause problems.

Iz that inside info ?
If they have found another gremlin with its neck stuck in a bottle,
the hIT men can have a smashing time with it,
unless the Trainman says otherwize :(


I don't know what's broken, but I have stopped the scheduler from sending me any more AP tasks until this get cleared up. I have 10 on one machine, 7 that has been sitting here since yesterday and the scheduler won't send any more because of the tasks are stuck. There are 23 that are stuck on the other one that are holding up 99 other tasks from d/l'ing. Since everything here is being held up by the APs, I'm seriously considering aborting them just to get some work. The retry button on both machines are in severe pain because I am abusing them so much.


I don't buy computers, I build them!!
ID: 1325670 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13947
Credit: 208,696,464
RAC: 304
Australia
Message 1325673 - Posted: 8 Jan 2013, 4:30:25 UTC - in response to Message 1325623.  

The current round of sick download performance was happening before the APs started to move, so you can forget them as the culprit - the root cause is a very poorly though through and implemented download scheduling algorithm that can do nothing apart from cause problems.

?
My hosts file is set to use the one server that behaves OK. Even that has been struggling since the network traffic maxed out. Before the network traffic maxed out downloads were coming down quickly.
With the severe WU limits in place, and the reduced availability of AP work, it was possible that the network traffic would back off. For whatever reason, that hasn't happened.
Grant
Darwin NT
ID: 1325673 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22798
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1325715 - Posted: 8 Jan 2013, 6:32:41 UTC

You were the lucky one. Before the servers maxed out I was lucky to get any work, and when I did an MB was taking about 30 minutes to download, with numerous re-tries, now the servers are maxed out MBs are either taking 1 minute or, in the majority of cases 30 minutes with many re-tries.
In "normal" times an MB takes about 10 seconds, and an AP a couple of minutes, with only the odd re-try - even when the servers are maxed those times only double.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1325715 · Report as offensive
Profile James Sotherden
Avatar

Send message
Joined: 16 May 99
Posts: 10436
Credit: 110,373,059
RAC: 54
United States
Message 1325819 - Posted: 8 Jan 2013, 16:20:24 UTC
Last modified: 8 Jan 2013, 16:20:45 UTC

Some weeks here at Seti the work units just fly from the downloader, AP included .Then we get weeks where they crawl at snail speed if your lucky.
I see the GPU group has another drive for two new servers. Im just wondering if that will help. After all if your trying to push 10" diameter of water thru a 1" diameter pipe, how effective will that be?

Is it hardware or software? or a combination of both.
[/quote]

Old James
ID: 1325819 · Report as offensive
tbret
Volunteer tester
Avatar

Send message
Joined: 28 May 99
Posts: 3380
Credit: 296,162,071
RAC: 40
United States
Message 1325823 - Posted: 8 Jan 2013, 16:36:16 UTC - in response to Message 1325819.  
Last modified: 8 Jan 2013, 16:36:45 UTC



Is it hardware or software? or a combination of both.



My personal belief is that nobody knows.

My fear is that nobody cares.

My annoyance is that nobody seems to be trying to find-out (just for fun).

My response is to "abandon all hope."
ID: 1325823 · Report as offensive
Profile shizaru
Volunteer tester
Avatar

Send message
Joined: 14 Jun 04
Posts: 1130
Credit: 1,967,904
RAC: 0
Greece
Message 1325937 - Posted: 9 Jan 2013, 3:28:14 UTC - in response to Message 1325823.  


Is it hardware or software? or a combination of both.

My personal belief is that nobody knows.
My fear is that nobody cares.
My annoyance is that nobody seems to be trying to find-out (just for fun).
My response is to "abandon all hope."


Every now and then it sure does feel like it. Then eventually lucidity gets the better of me and I realize that (as far as my little cruncher is concerned) Seti enjoys 100% uptime. Well, at least for the past 2 years it has...

Now obviously to make a statement like that I must be bending the rules a little bit. And by that I mean I'm not counting a couple weeks of server replacements or a multi day power outage or any other scenario the Lab has no control over (though I can't remember any others in the 2yr timeframe mentioned).

Now I know this is no consolation to anyone with a PC quick enough to burn through their cache in a matter of hours. But I'd be lying if I said my little netbook hasn't been crunching pretty much non-stop for a fraction over 24 months now.

All that means is, a lot of people must be doing something right. And it's not just the people in the lab that care. What about Richard who's always looking out for us? What about Brad who just posted a wall-of-text of accomplishments for the lab? That's just a couple of the people going above and beyond so things run smoothly for us. But not a definitive list of people who care. What about the Raistmers and the Jasons? The Joes, Claggys and Mikes? I'm sure that's just the tip of the iceberg. In fact I think the reality is that everybody here cares for the project.
ID: 1325937 · Report as offensive
tbret
Volunteer tester
Avatar

Send message
Joined: 28 May 99
Posts: 3380
Credit: 296,162,071
RAC: 40
United States
Message 1326112 - Posted: 9 Jan 2013, 17:36:10 UTC - in response to Message 1325937.  


Is it hardware or software? or a combination of both.

My personal belief is that nobody knows.
My fear is that nobody cares.
My annoyance is that nobody seems to be trying to find-out (just for fun).
My response is to "abandon all hope."


Every now and then it sure does feel like it. Then eventually lucidity gets the better of me and I realize that (as far as my little cruncher is concerned) Seti enjoys 100% uptime. Well, at least for the past 2 years it has...

Now obviously to make a statement like that I must be bending the rules a little bit. And by that I mean I'm not counting a couple weeks of server replacements or a multi day power outage or any other scenario the Lab has no control over (though I can't remember any others in the 2yr timeframe mentioned).

Now I know this is no consolation to anyone with a PC quick enough to burn through their cache in a matter of hours. But I'd be lying if I said my little netbook hasn't been crunching pretty much non-stop for a fraction over 24 months now.

All that means is, a lot of people must be doing something right. And it's not just the people in the lab that care. What about Richard who's always looking out for us? What about Brad who just posted a wall-of-text of accomplishments for the lab? That's just a couple of the people going above and beyond so things run smoothly for us. But not a definitive list of people who care. What about the Raistmers and the Jasons? The Joes, Claggys and Mikes? I'm sure that's just the tip of the iceberg. In fact I think the reality is that everybody here cares for the project.


So that my comment isn't misunderstood as being snarky: I did say my "fear" is that nobody cares, and by "nobody" I meant anyone who could do anything about server issues. I didn't even go so far as to say, my "observation" is that nobody cares.

It is plain that a great number of people care a lot about this project.

My slowest cruncher has only run out of work once in many months. Your point is well taken.

ID: 1326112 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9959
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1326191 - Posted: 9 Jan 2013, 21:38:27 UTC

I did say my "fear" is that nobody cares, and by "nobody" I meant anyone who could do anything about server issues.


I do not believe that Matt Jeff and Eric "don't care". Sorry that is going too far.
I don't believe they have the time and resources to do what they want but to say the guys in the lab "don't care" is to me and insult.
ID: 1326191 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1646
Credit: 12,921,799
RAC: 89
New Zealand
Message 1326222 - Posted: 9 Jan 2013, 22:19:58 UTC - in response to Message 1326191.  

but to say the guys in the lab "don't care" is to me and insult.

+1
ID: 1326222 · Report as offensive
Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · 13 . . . 24 · Next

Message boards : Number crunching : Panic Mode On (80) Server Problems?


 
©2025 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.