Panic Mode On (60) Server problems?


log in

Advanced search

Message boards : Number crunching : Panic Mode On (60) Server problems?

1 · 2 · 3 · 4 . . . 11 · Next
Author Message
Profile arkaynProject donor
Volunteer tester
Avatar
Send message
Joined: 14 May 99
Posts: 3623
Credit: 48,549,071
RAC: 29,278
United States
Message 1167949 - Posted: 5 Nov 2011, 3:50:20 UTC

Red Alert, servers are down at this moment.....
____________

msattlerProject donor
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38923
Credit: 578,842,429
RAC: 514,019
United States
Message 1167951 - Posted: 5 Nov 2011, 3:54:52 UTC - in response to Message 1167949.

Red Alert, servers are down at this moment.....

Oh, you mean more down than they have been for the last hour or two?
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Profile arkaynProject donor
Volunteer tester
Avatar
Send message
Joined: 14 May 99
Posts: 3623
Credit: 48,549,071
RAC: 29,278
United States
Message 1167952 - Posted: 5 Nov 2011, 4:08:51 UTC - in response to Message 1167951.

Red Alert, servers are down at this moment.....

Oh, you mean more down than they have been for the last hour or two?


More like the last 6 hours or so.

That is when the upload server started misbehaving.
____________

msattlerProject donor
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38923
Credit: 578,842,429
RAC: 514,019
United States
Message 1167953 - Posted: 5 Nov 2011, 4:19:32 UTC - in response to Message 1167952.

Red Alert, servers are down at this moment.....

Oh, you mean more down than they have been for the last hour or two?


More like the last 6 hours or so.

That is when the upload server started misbehaving.

And kinda looks like it's gonna be rough ridin' until somebody can get in to da lab to set things straight.

Eric was trying to remote boot thingys when he got home, but I don't think it quite worked out yet.

Sometimes the servers doin' 'most alright...
And sometimes I tink dey ain't.
Once in a while dey get a few bits out,
but most da time they cain't.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Profile arkaynProject donor
Volunteer tester
Avatar
Send message
Joined: 14 May 99
Posts: 3623
Credit: 48,549,071
RAC: 29,278
United States
Message 1167954 - Posted: 5 Nov 2011, 4:24:30 UTC - in response to Message 1167953.

Red Alert, servers are down at this moment.....

Oh, you mean more down than they have been for the last hour or two?


More like the last 6 hours or so.

That is when the upload server started misbehaving.

And kinda looks like it's gonna be rough ridin' until somebody can get in to da lab to set things straight.

Eric was trying to remote boot thingys when he got home, but I don't think it quite worked out yet.

Sometimes the servers doin' 'most alright...
And sometimes I tink dey ain't.
Once in a while dey get a few bits out,
but most da time they cain't.


I just allowed Milkyway for Nvidia for a little while to see how fast the GTX560 can crunch a standard unit. Most likely way slower than my HD5830 in the same machine.

____________

msattlerProject donor
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38923
Credit: 578,842,429
RAC: 514,019
United States
Message 1167957 - Posted: 5 Nov 2011, 4:31:48 UTC - in response to Message 1167954.

Red Alert, servers are down at this moment.....

Oh, you mean more down than they have been for the last hour or two?


More like the last 6 hours or so.

That is when the upload server started misbehaving.

And kinda looks like it's gonna be rough ridin' until somebody can get in to da lab to set things straight.

Eric was trying to remote boot thingys when he got home, but I don't think it quite worked out yet.

Sometimes the servers doin' 'most alright...
And sometimes I tink dey ain't.
Once in a while dey get a few bits out,
but most da time they cain't.


I just allowed Milkyway for Nvidia for a little while to see how fast the GTX560 can crunch a standard unit. Most likely way slower than my HD5830 in the same machine.

Well, y'all enjoy......
The kitties are gonna proceed to crunch what they got, and hope things can get fixed before the kibble bowls run dry.

Meow meow meow!
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Terror Australis
Volunteer tester
Send message
Joined: 14 Feb 04
Posts: 1711
Credit: 204,388,895
RAC: 24,887
Australia
Message 1167960 - Posted: 5 Nov 2011, 4:36:35 UTC

Sometimes the servers doin' 'most alright...
And sometimes I tink dey ain't.
Once in a while dey get a few bits out,
but most da time they cain't.

Hey Mark.
As the author of this original work, may I please have your permission to print it out and stick it on the wall of the server room at work ?
:-)

T.A.

msattlerProject donor
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38923
Credit: 578,842,429
RAC: 514,019
United States
Message 1167963 - Posted: 5 Nov 2011, 4:42:34 UTC - in response to Message 1167960.
Last modified: 5 Nov 2011, 4:46:48 UTC

Sometimes the servers doin' 'most alright...
And sometimes I tink dey ain't.
Once in a while dey get a few bits out,
but most da time they cain't.

Hey Mark.
As the author of this original work, may I please have your permission to print it out and stick it on the wall of the server room at work ?
:-)

T.A.

LOL....certainly. I would be most honored.

EDIT...
Somebody should print up a really pretty copy, frame it nicely, and send it to da boyz in da lab for the Seti server closet.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Profile arkaynProject donor
Volunteer tester
Avatar
Send message
Joined: 14 May 99
Posts: 3623
Credit: 48,549,071
RAC: 29,278
United States
Message 1167964 - Posted: 5 Nov 2011, 4:46:06 UTC - in response to Message 1167957.



I just allowed Milkyway for Nvidia for a little while to see how fast the GTX560 can crunch a standard unit. Most likely way slower than my HD5830 in the same machine.

Well, y'all enjoy......
The kitties are gonna proceed to crunch what they got, and hope things can get fixed before the kibble bowls run dry.

Meow meow meow!


My CUDA kibble bowl is dry, I ran through those 140 units almost as fast as I could download them.

The other machine still has a good supply though.
____________

msattlerProject donor
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38923
Credit: 578,842,429
RAC: 514,019
United States
Message 1167966 - Posted: 5 Nov 2011, 4:48:47 UTC - in response to Message 1167964.



I just allowed Milkyway for Nvidia for a little while to see how fast the GTX560 can crunch a standard unit. Most likely way slower than my HD5830 in the same machine.

Well, y'all enjoy......
The kitties are gonna proceed to crunch what they got, and hope things can get fixed before the kibble bowls run dry.

Meow meow meow!


My CUDA kibble bowl is dry, I ran through those 140 units almost as fast as I could download them.

The other machine still has a good supply though.

Kitties still got a bit o' kibble stretched across the 8 rigs in the crunching den...
But dem Cuda beasties be mean ol' crunchers, and go through the kibble fast!
All out, no in, makes for hungry kitties.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Terror Australis
Volunteer tester
Send message
Joined: 14 Feb 04
Posts: 1711
Credit: 204,388,895
RAC: 24,887
Australia
Message 1167974 - Posted: 5 Nov 2011, 5:05:06 UTC

Uploads have been down for 6 to 7 hours now but the green on the Cricket Graphs is still maxxed out.

Shows how many downloads must have been backed up. (Unfortunately none of them are mine.) :(

T.A.

msattlerProject donor
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38923
Credit: 578,842,429
RAC: 514,019
United States
Message 1167982 - Posted: 5 Nov 2011, 5:07:59 UTC - in response to Message 1167974.

Uploads have been down for 6 to 7 hours now but the green on the Cricket Graphs is still maxxed out.

Shows how many downloads must have been backed up. (Unfortunately none of them are mine.) :(

T.A.

There is still a bit of work being issued on the few scheduler requests that make it in and out of the black hole.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5792
Credit: 58,045,038
RAC: 48,281
Australia
Message 1168071 - Posted: 5 Nov 2011, 6:05:45 UTC - in response to Message 1167982.

There is still a bit of work being issued on the few scheduler requests that make it in and out of the black hole.

6.12.33 is showing it's limits again. The machine with 6.10.58 continues to occasionally report & get new work, but 6.12.33 backs off so far with each failed attempt that it hasn't reported or received work for hours.
____________
Grant
Darwin NT.

msattlerProject donor
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38923
Credit: 578,842,429
RAC: 514,019
United States
Message 1168074 - Posted: 5 Nov 2011, 6:08:30 UTC - in response to Message 1168071.

There is still a bit of work being issued on the few scheduler requests that make it in and out of the black hole.

6.12.33 is showing it's limits again. The machine with 6.10.58 continues to occasionally report & get new work, but 6.12.33 backs off so far with each failed attempt that it hasn't reported or received work for hours.

And some question why I have not moved beyond........
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

WinterKnight
Volunteer tester
Send message
Joined: 18 May 99
Posts: 8630
Credit: 23,730,717
RAC: 19,156
United Kingdom
Message 1168127 - Posted: 5 Nov 2011, 7:29:46 UTC

I can't report or request now.

And in reply to the last two post, I am sure the extra long back-offs in 6.12.nn are actually causing more problems than they are curing. Very noticable if you abuse the buttons.

Would you believe that in the distant past, I actually gave written warnings to people who played with switches and twiddled with variable controls for no good reason.

rob smithProject donor
Volunteer tester
Send message
Joined: 7 Mar 03
Posts: 8310
Credit: 55,277,804
RAC: 75,401
United Kingdom
Message 1168130 - Posted: 5 Nov 2011, 7:36:12 UTC

Looking at the crickets there have been a series of drop-outs for the last few hours, every couple of hours the throughput has dropped by about 10%.
Coupled with a very poor upload performance its obvious that the servers are less than happy. We're going to have to wait a few hours until anyone in Berkeley is awake, or maybe a couple of days until someone gets in to work on Monday.
____________
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

msattlerProject donor
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38923
Credit: 578,842,429
RAC: 514,019
United States
Message 1168132 - Posted: 5 Nov 2011, 7:48:08 UTC - in response to Message 1168127.

I can't report or request now.

And in reply to the last two post, I am sure the extra long back-offs in 6.12.nn are actually causing more problems than they are curing. Very noticable if you abuse the buttons.

Would you believe that in the distant past, I actually gave written warnings to people who played with switches and twiddled with variable controls for no good reason.

You have to realize that the small percentage of 'us'...who might play with the buttons or abuse them, are such a small percentage of the total Seti user base that nothing we could possibly do could affect server performance.
Not ONE iota of difference.

Even myself, with 8 rigs running, in the top producers on the project, could not possibly nudge the buttons enough to be seen on the Cricket graphs.

My rigs are active enough that when things go awry, I have a pretty good handle on what's going on by watching what they can or cannot do.

The backoffs in latter day Boincs may be good for the project, because most of the hosts which are using them will just 'go away' for a while....sometimes a long while. But they are no good at all for a power cruncher striving to do the most work he can for the project.

Double edged sword.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

rob smithProject donor
Volunteer tester
Send message
Joined: 7 Mar 03
Posts: 8310
Credit: 55,277,804
RAC: 75,401
United Kingdom
Message 1168148 - Posted: 5 Nov 2011, 9:02:44 UTC

Work outside the project indicates that long back-off are actually counter productive in terms of overall throughput on saturated data links. This is particularly so when you have a long first back-off. If you must use back-off then a short, random initial back-off is far more effective in load spreading.

Far more effective is load throttling, where you reduce the data rate to each of your concurrent clients by a small faction.
So, if you have 100 concurrent clients who would normally each have a 1% share of the available bandwidth and you are suffering signs of congestion (increases packet drop for example) you reduce the bandwidth share of each client by 1%, that is from 1% of total to 0.99% of total. This has virtually no effect on the data rate to the user, but does reduce the instantaneous data rate enough to reduce the number of destructive collisions, so reducing packet loss, and the client actual observes an INCREASE in effective data rate at their end of the bit of wire, but you are only using 99% of the available bandwidth.

Obviously if you have a situation where the feed server is off-line (for maintenance, or it has crashed) then you have to implement a message that says so, and a realistic "wait for" delay.
____________
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

Profile hiamps
Volunteer tester
Avatar
Send message
Joined: 23 May 99
Posts: 4292
Credit: 72,971,319
RAC: 0
United States
Message 1168182 - Posted: 5 Nov 2011, 12:27:35 UTC - in response to Message 1168127.

I can't report or request now.

And in reply to the last two post, I am sure the extra long back-offs in 6.12.nn are actually causing more problems than they are curing. Very noticable if you abuse the buttons.

Would you believe that in the distant past, I actually gave written warnings to people who played with switches and twiddled with variable controls for no good reason.


Thats one of the reasons I left.....
____________
Official Abuser of Boinc Buttons...
And no good credit hound!

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5792
Credit: 58,045,038
RAC: 48,281
Australia
Message 1168378 - Posted: 5 Nov 2011, 21:19:19 UTC - in response to Message 1168182.
Last modified: 5 Nov 2011, 21:19:56 UTC

Uploads piled up overnight, so i just had a look at the network traffic graphs and that is one weird looking graph. I don't know how the Scheduler is going at the moment- but the upload server is certainly having kittens.


EDIT- i'm pretty sure we had similar issues a few months ago.
____________
Grant
Darwin NT.

1 · 2 · 3 · 4 . . . 11 · Next

Message boards : Number crunching : Panic Mode On (60) Server problems?

Copyright © 2014 University of California