Panic Mode On (60) Server problems?

Message boards : Number crunching : Panic Mode On (60) Server problems?
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · 4 . . . 9 · Next

AuthorMessage
Profile arkayn
Volunteer tester
Avatar

Send message
Joined: 14 May 99
Posts: 4438
Credit: 55,006,323
RAC: 0
United States
Message 1167949 - Posted: 5 Nov 2011, 3:50:20 UTC

Red Alert, servers are down at this moment.....

ID: 1167949 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1167951 - Posted: 5 Nov 2011, 3:54:52 UTC - in response to Message 1167949.  

Red Alert, servers are down at this moment.....

Oh, you mean more down than they have been for the last hour or two?
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1167951 · Report as offensive
Profile arkayn
Volunteer tester
Avatar

Send message
Joined: 14 May 99
Posts: 4438
Credit: 55,006,323
RAC: 0
United States
Message 1167952 - Posted: 5 Nov 2011, 4:08:51 UTC - in response to Message 1167951.  

Red Alert, servers are down at this moment.....

Oh, you mean more down than they have been for the last hour or two?


More like the last 6 hours or so.

That is when the upload server started misbehaving.

ID: 1167952 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1167953 - Posted: 5 Nov 2011, 4:19:32 UTC - in response to Message 1167952.  

Red Alert, servers are down at this moment.....

Oh, you mean more down than they have been for the last hour or two?


More like the last 6 hours or so.

That is when the upload server started misbehaving.

And kinda looks like it's gonna be rough ridin' until somebody can get in to da lab to set things straight.

Eric was trying to remote boot thingys when he got home, but I don't think it quite worked out yet.

Sometimes the servers doin' 'most alright...
And sometimes I tink dey ain't.
Once in a while dey get a few bits out,
but most da time they cain't.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1167953 · Report as offensive
Profile arkayn
Volunteer tester
Avatar

Send message
Joined: 14 May 99
Posts: 4438
Credit: 55,006,323
RAC: 0
United States
Message 1167954 - Posted: 5 Nov 2011, 4:24:30 UTC - in response to Message 1167953.  

Red Alert, servers are down at this moment.....

Oh, you mean more down than they have been for the last hour or two?


More like the last 6 hours or so.

That is when the upload server started misbehaving.

And kinda looks like it's gonna be rough ridin' until somebody can get in to da lab to set things straight.

Eric was trying to remote boot thingys when he got home, but I don't think it quite worked out yet.

Sometimes the servers doin' 'most alright...
And sometimes I tink dey ain't.
Once in a while dey get a few bits out,
but most da time they cain't.


I just allowed Milkyway for Nvidia for a little while to see how fast the GTX560 can crunch a standard unit. Most likely way slower than my HD5830 in the same machine.


ID: 1167954 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1167957 - Posted: 5 Nov 2011, 4:31:48 UTC - in response to Message 1167954.  

Red Alert, servers are down at this moment.....

Oh, you mean more down than they have been for the last hour or two?


More like the last 6 hours or so.

That is when the upload server started misbehaving.

And kinda looks like it's gonna be rough ridin' until somebody can get in to da lab to set things straight.

Eric was trying to remote boot thingys when he got home, but I don't think it quite worked out yet.

Sometimes the servers doin' 'most alright...
And sometimes I tink dey ain't.
Once in a while dey get a few bits out,
but most da time they cain't.


I just allowed Milkyway for Nvidia for a little while to see how fast the GTX560 can crunch a standard unit. Most likely way slower than my HD5830 in the same machine.

Well, y'all enjoy......
The kitties are gonna proceed to crunch what they got, and hope things can get fixed before the kibble bowls run dry.

Meow meow meow!
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1167957 · Report as offensive
Terror Australis
Volunteer tester

Send message
Joined: 14 Feb 04
Posts: 1817
Credit: 262,693,308
RAC: 44
Australia
Message 1167960 - Posted: 5 Nov 2011, 4:36:35 UTC

Sometimes the servers doin' 'most alright...
And sometimes I tink dey ain't.
Once in a while dey get a few bits out,
but most da time they cain't.

Hey Mark.
As the author of this original work, may I please have your permission to print it out and stick it on the wall of the server room at work ?
:-)

T.A.
ID: 1167960 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1167963 - Posted: 5 Nov 2011, 4:42:34 UTC - in response to Message 1167960.  
Last modified: 5 Nov 2011, 4:46:48 UTC

Sometimes the servers doin' 'most alright...
And sometimes I tink dey ain't.
Once in a while dey get a few bits out,
but most da time they cain't.

Hey Mark.
As the author of this original work, may I please have your permission to print it out and stick it on the wall of the server room at work ?
:-)

T.A.

LOL....certainly. I would be most honored.

EDIT...
Somebody should print up a really pretty copy, frame it nicely, and send it to da boyz in da lab for the Seti server closet.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1167963 · Report as offensive
Profile arkayn
Volunteer tester
Avatar

Send message
Joined: 14 May 99
Posts: 4438
Credit: 55,006,323
RAC: 0
United States
Message 1167964 - Posted: 5 Nov 2011, 4:46:06 UTC - in response to Message 1167957.  



I just allowed Milkyway for Nvidia for a little while to see how fast the GTX560 can crunch a standard unit. Most likely way slower than my HD5830 in the same machine.

Well, y'all enjoy......
The kitties are gonna proceed to crunch what they got, and hope things can get fixed before the kibble bowls run dry.

Meow meow meow!


My CUDA kibble bowl is dry, I ran through those 140 units almost as fast as I could download them.

The other machine still has a good supply though.

ID: 1167964 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1167966 - Posted: 5 Nov 2011, 4:48:47 UTC - in response to Message 1167964.  



I just allowed Milkyway for Nvidia for a little while to see how fast the GTX560 can crunch a standard unit. Most likely way slower than my HD5830 in the same machine.

Well, y'all enjoy......
The kitties are gonna proceed to crunch what they got, and hope things can get fixed before the kibble bowls run dry.

Meow meow meow!


My CUDA kibble bowl is dry, I ran through those 140 units almost as fast as I could download them.

The other machine still has a good supply though.

Kitties still got a bit o' kibble stretched across the 8 rigs in the crunching den...
But dem Cuda beasties be mean ol' crunchers, and go through the kibble fast!
All out, no in, makes for hungry kitties.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1167966 · Report as offensive
Terror Australis
Volunteer tester

Send message
Joined: 14 Feb 04
Posts: 1817
Credit: 262,693,308
RAC: 44
Australia
Message 1167974 - Posted: 5 Nov 2011, 5:05:06 UTC

Uploads have been down for 6 to 7 hours now but the green on the Cricket Graphs is still maxxed out.

Shows how many downloads must have been backed up. (Unfortunately none of them are mine.) :(

T.A.
ID: 1167974 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1167982 - Posted: 5 Nov 2011, 5:07:59 UTC - in response to Message 1167974.  

Uploads have been down for 6 to 7 hours now but the green on the Cricket Graphs is still maxxed out.

Shows how many downloads must have been backed up. (Unfortunately none of them are mine.) :(

T.A.

There is still a bit of work being issued on the few scheduler requests that make it in and out of the black hole.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1167982 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13731
Credit: 208,696,464
RAC: 304
Australia
Message 1168071 - Posted: 5 Nov 2011, 6:05:45 UTC - in response to Message 1167982.  

There is still a bit of work being issued on the few scheduler requests that make it in and out of the black hole.

6.12.33 is showing it's limits again. The machine with 6.10.58 continues to occasionally report & get new work, but 6.12.33 backs off so far with each failed attempt that it hasn't reported or received work for hours.
Grant
Darwin NT
ID: 1168071 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1168074 - Posted: 5 Nov 2011, 6:08:30 UTC - in response to Message 1168071.  

There is still a bit of work being issued on the few scheduler requests that make it in and out of the black hole.

6.12.33 is showing it's limits again. The machine with 6.10.58 continues to occasionally report & get new work, but 6.12.33 backs off so far with each failed attempt that it hasn't reported or received work for hours.

And some question why I have not moved beyond........
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1168074 · Report as offensive
W-K 666 Project Donor
Volunteer tester

Send message
Joined: 18 May 99
Posts: 19048
Credit: 40,757,560
RAC: 67
United Kingdom
Message 1168127 - Posted: 5 Nov 2011, 7:29:46 UTC

I can't report or request now.

And in reply to the last two post, I am sure the extra long back-offs in 6.12.nn are actually causing more problems than they are curing. Very noticable if you abuse the buttons.

Would you believe that in the distant past, I actually gave written warnings to people who played with switches and twiddled with variable controls for no good reason.
ID: 1168127 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22190
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1168130 - Posted: 5 Nov 2011, 7:36:12 UTC

Looking at the crickets there have been a series of drop-outs for the last few hours, every couple of hours the throughput has dropped by about 10%.
Coupled with a very poor upload performance its obvious that the servers are less than happy. We're going to have to wait a few hours until anyone in Berkeley is awake, or maybe a couple of days until someone gets in to work on Monday.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1168130 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1168132 - Posted: 5 Nov 2011, 7:48:08 UTC - in response to Message 1168127.  

I can't report or request now.

And in reply to the last two post, I am sure the extra long back-offs in 6.12.nn are actually causing more problems than they are curing. Very noticable if you abuse the buttons.

Would you believe that in the distant past, I actually gave written warnings to people who played with switches and twiddled with variable controls for no good reason.

You have to realize that the small percentage of 'us'...who might play with the buttons or abuse them, are such a small percentage of the total Seti user base that nothing we could possibly do could affect server performance.
Not ONE iota of difference.

Even myself, with 8 rigs running, in the top producers on the project, could not possibly nudge the buttons enough to be seen on the Cricket graphs.

My rigs are active enough that when things go awry, I have a pretty good handle on what's going on by watching what they can or cannot do.

The backoffs in latter day Boincs may be good for the project, because most of the hosts which are using them will just 'go away' for a while....sometimes a long while. But they are no good at all for a power cruncher striving to do the most work he can for the project.

Double edged sword.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1168132 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22190
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1168148 - Posted: 5 Nov 2011, 9:02:44 UTC

Work outside the project indicates that long back-off are actually counter productive in terms of overall throughput on saturated data links. This is particularly so when you have a long first back-off. If you must use back-off then a short, random initial back-off is far more effective in load spreading.

Far more effective is load throttling, where you reduce the data rate to each of your concurrent clients by a small faction.
So, if you have 100 concurrent clients who would normally each have a 1% share of the available bandwidth and you are suffering signs of congestion (increases packet drop for example) you reduce the bandwidth share of each client by 1%, that is from 1% of total to 0.99% of total. This has virtually no effect on the data rate to the user, but does reduce the instantaneous data rate enough to reduce the number of destructive collisions, so reducing packet loss, and the client actual observes an INCREASE in effective data rate at their end of the bit of wire, but you are only using 99% of the available bandwidth.

Obviously if you have a situation where the feed server is off-line (for maintenance, or it has crashed) then you have to implement a message that says so, and a realistic "wait for" delay.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1168148 · Report as offensive
Profile hiamps
Volunteer tester
Avatar

Send message
Joined: 23 May 99
Posts: 4292
Credit: 72,971,319
RAC: 0
United States
Message 1168182 - Posted: 5 Nov 2011, 12:27:35 UTC - in response to Message 1168127.  

I can't report or request now.

And in reply to the last two post, I am sure the extra long back-offs in 6.12.nn are actually causing more problems than they are curing. Very noticable if you abuse the buttons.

Would you believe that in the distant past, I actually gave written warnings to people who played with switches and twiddled with variable controls for no good reason.


Thats one of the reasons I left.....
Official Abuser of Boinc Buttons...
And no good credit hound!
ID: 1168182 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13731
Credit: 208,696,464
RAC: 304
Australia
Message 1168378 - Posted: 5 Nov 2011, 21:19:19 UTC - in response to Message 1168182.  
Last modified: 5 Nov 2011, 21:19:56 UTC

Uploads piled up overnight, so i just had a look at the network traffic graphs and that is one weird looking graph. I don't know how the Scheduler is going at the moment- but the upload server is certainly having kittens.


EDIT- i'm pretty sure we had similar issues a few months ago.
Grant
Darwin NT
ID: 1168378 · Report as offensive
1 · 2 · 3 · 4 . . . 9 · Next

Message boards : Number crunching : Panic Mode On (60) Server problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.