Panic Mode On (72) Server problems?


log in

Advanced search

Message boards : Number crunching : Panic Mode On (72) Server problems?

Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · Next
Author Message
msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38322
Credit: 560,278,901
RAC: 653,159
United States
Message 1211140 - Posted: 28 Mar 2012, 17:48:59 UTC - in response to Message 1211139.

Mods...
You have locked the wrong thread..
This one should be locked and the new one (73) opened for use whilst arkayn is on vacation. He just opened the new one a little early, as he would not be here to do it when this one filled up.

I imagine he left instructions to allow this one to fill up to 200 posts as usual, and then bring forward the replacement. We should be able to do that quickly enough, between us.

LOL...maybe he went on vacation and left the thread on Modomatic.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Profile soft^spirit
Avatar
Send message
Joined: 18 May 99
Posts: 6374
Credit: 28,628,617
RAC: 649
United States
Message 1211163 - Posted: 28 Mar 2012, 18:52:10 UTC

we were given instructions and a stick. Carry on! omg o noes!!
____________

Janice

Profile Michel448a
Volunteer tester
Avatar
Send message
Joined: 27 Oct 00
Posts: 1201
Credit: 2,891,635
RAC: 0
Canada
Message 1211182 - Posted: 28 Mar 2012, 19:34:45 UTC - in response to Message 1211163.

we were given instructions and a stick. Carry on! omg o noes!!


38 more replys to go ^^
____________

tbret
Volunteer tester
Avatar
Send message
Joined: 28 May 99
Posts: 2620
Credit: 191,235,512
RAC: 526,368
United States
Message 1211189 - Posted: 28 Mar 2012, 19:48:02 UTC - in response to Message 1210955.

My 3 are still bouncing off the limits here.

Downloads are also good here, it's uploads that are the issue


I've got that issue, too.

They haven't been disabling the uploads during the outages, but for whatever reason they seem to have taken uploads offline yesterday.

I'm guessing we are just playing "catch-up" so there are a lot of outstanding uploads and we're fighting each-other for the upload server's attention.

Hopefully it will clear itself up over the next couple of days. If not, it will need to be brought to someone's attention.

Profile Zapped Sparky
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 30 Aug 08
Posts: 6667
Credit: 1,200,844
RAC: 46
United Kingdom
Message 1211233 - Posted: 28 Mar 2012, 21:22:30 UTC

I'm sure there's plenty of astropulses going out but my estimates have gone from 280hrs with the first V6, the second was 264hrs and with the third that just finished downloading 241hrs. Phew, it's gonna be a while for things to get back to normal (cache wise) for me :)
____________
In an alternate universe, it was a ZX81 that asked for clothes, boots and motorcycle.

Client error 418: I'm a teapot

Tropical Goldfish Fish 13: You're not crazy if you crunch for Seti :)

Cosmic_Ocean
Avatar
Send message
Joined: 23 Dec 00
Posts: 2237
Credit: 8,450,484
RAC: 4,096
United States
Message 1211309 - Posted: 28 Mar 2012, 23:07:33 UTC - in response to Message 1211233.

I'm sure there's plenty of astropulses going out but my estimates have gone from 280hrs with the first V6, the second was 264hrs and with the third that just finished downloading 241hrs. Phew, it's gonna be a while for things to get back to normal (cache wise) for me :)

Agreed. Mine started at 209 hours and after doing six of them, they are now down to 192. 10-day cache is 3 running + 3 waiting to run. The ETAs should start dropping a bit faster after 10 are completed though. I know they did on this new machine back in January for v505. First dozen or so APs were all 200+ and then the next one I was issued after the 10th one validated was down in the 40 range and took about 20 or so to come down the rest of the way to ~13.
____________

Linux laptop uptime: 1484d 22h 42m
Ended due to UPS failure, found 14 hours after the fact

Profile arkayn
Volunteer tester
Avatar
Send message
Joined: 14 May 99
Posts: 3594
Credit: 47,339,819
RAC: 386
United States
Message 1211331 - Posted: 28 Mar 2012, 23:51:07 UTC

Unfortunately, my vacation is not going too well yet.

Found a flat tire on the car and it was determined that a new tire was needed instead of fixing. That took most of the day and $118.

I plan on leaving in the morning now.
____________

Josef W. Segur
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4203
Credit: 1,030,528
RAC: 265
United States
Message 1211348 - Posted: 29 Mar 2012, 0:47:50 UTC - in response to Message 1211309.

I'm sure there's plenty of astropulses going out but my estimates have gone from 280hrs with the first V6, the second was 264hrs and with the third that just finished downloading 241hrs. Phew, it's gonna be a while for things to get back to normal (cache wise) for me :)

Agreed. Mine started at 209 hours and after doing six of them, they are now down to 192. 10-day cache is 3 running + 3 waiting to run. The ETAs should start dropping a bit faster after 10 are completed though. I know they did on this new machine back in January for v505. First dozen or so APs were all 200+ and then the next one I was issued after the 10th one validated was down in the 40 range and took about 20 or so to come down the rest of the way to ~13.

The difference now is that it'll take longer to get the completed count up to 10 since only validated tasks which ran full length and had less than 10% blanking go into the average. Based on our observations of v505 work showing about 1 in 4 having higher blanking plus the early exits, at 14 validated there would be about a 50/50 chance of having 10 in the average. But the protections ought to mean after 10 completed the estimates will be fairly close.

For someone running only AP, DCF will have gone down while getting there, but in a way it's fortunate the initial estimates are so high. Whenever the runtime is less than 1/10 the estimate, DCF only goes down a little.
Joe

B-Man
Volunteer tester
Send message
Joined: 11 Feb 01
Posts: 253
Credit: 147,366
RAC: 0
United States
Message 1211349 - Posted: 29 Mar 2012, 0:51:21 UTC - in response to Message 1211331.

Unfortunately, my vacation is not going too well yet.

Found a flat tire on the car and it was determined that a new tire was needed instead of fixing. That took most of the day and $118.

I plan on leaving in the morning now.

Better luck in the morning. Have fun on your vacation.
____________

Cosmic_Ocean
Avatar
Send message
Joined: 23 Dec 00
Posts: 2237
Credit: 8,450,484
RAC: 4,096
United States
Message 1211392 - Posted: 29 Mar 2012, 3:37:52 UTC - in response to Message 1211348.

The difference now is that it'll take longer to get the completed count up to 10 since only validated tasks which ran full length and had less than 10% blanking go into the average. Based on our observations of v505 work showing about 1 in 4 having higher blanking plus the early exits, at 14 validated there would be about a 50/50 chance of having 10 in the average. But the protections ought to mean after 10 completed the estimates will be fairly close.

For someone running only AP, DCF will have gone down while getting there, but in a way it's fortunate the initial estimates are so high. Whenever the runtime is less than 1/10 the estimate, DCF only goes down a little.

And so far what I've seen in my handy spreadsheet is that of the 11 APs that I have crunched and returned with r557, one was a B3_P1 with 100% blanking, and only three of the other 10 had less than 10% blanked.

It does of course depend on the data itself, but a very early and large margin-of-error estimate is about 1 in 3 have <10% based on my data.
____________

Linux laptop uptime: 1484d 22h 42m
Ended due to UPS failure, found 14 hours after the fact

Profile Slavac
Volunteer tester
Avatar
Send message
Joined: 27 Apr 11
Posts: 1932
Credit: 17,952,639
RAC: 0
United States
Message 1211397 - Posted: 29 Mar 2012, 3:43:19 UTC - in response to Message 1211392.

Getting quite a lot of shorties though I don't mind, I enjoy crunching them.
____________


Executive Director GPU Users Group Inc. -
brad@gpuug.org

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38322
Credit: 560,278,901
RAC: 653,159
United States
Message 1211433 - Posted: 29 Mar 2012, 6:15:25 UTC - in response to Message 1211397.

Getting quite a lot of shorties though I don't mind, I enjoy crunching them.

The kitties luv shorties too. Just as long as the servers can keep up and keep the limited kibble bowls full.
The Frozen 920 has about 1560 GPU tasks in cache...right now about 1150 of them are shorties. It would take about 9-1/2 hours for it to crunch all of the shorties if the servers went down.
Hence my wish for an increase or lifting of the current limits.

____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5695
Credit: 56,392,621
RAC: 48,762
Australia
Message 1211442 - Posted: 29 Mar 2012, 7:08:03 UTC - in response to Message 1211433.


Uploads were a bit iffy for a while there, but they seem to be going through now. Eventually.
____________
Grant
Darwin NT.

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5695
Credit: 56,392,621
RAC: 48,762
Australia
Message 1211459 - Posted: 29 Mar 2012, 8:29:11 UTC - in response to Message 1211442.


Uploads were a bit iffy for a while there, but they seem to be going through now. Eventually.

Make that uploads are still very iffy.
Looking at the network traffic, inbound traffic is looking a bit jagged. Things aren't good in upload land.
____________
Grant
Darwin NT.

Profile Brother Frank
Send message
Joined: 10 Dec 11
Posts: 26
Credit: 15,142,410
RAC: 0
United States
Message 1211564 - Posted: 29 Mar 2012, 15:47:59 UTC - in response to Message 1210940.

I've been seeing this too along with many long backoff's on my two best crunchers. Then I retry for both downloads and uploads and it will all get through. I thought things were returning to near normal on Tuesday evening but then it go much worse again on Wednesday. This maintenance period has been a nightmare for my uploads. Even with my slowest machine I have had to manually update and retry a great deal. I joined just a few months ago, but this has not happened before for me except for that really bad period around the 2nd, 3rd, and 4th weeks in Feb.

Brother Frank

Profile cliff
Avatar
Send message
Joined: 16 Dec 07
Posts: 322
Credit: 2,509,590
RAC: 0
United Kingdom
Message 1211572 - Posted: 29 Mar 2012, 16:00:18 UTC - in response to Message 1211459.
Last modified: 29 Mar 2012, 16:00:45 UTC

Things aint to good in d/l land either, every single WU download has had to be retried a min of 3 times..

Not a great problem during my waking hours, but on waking to find both rigs in project backoffs etc is getting a tad wearing:-)

I now find my waking hours are somewhat altered, up at between 03:00 and 04:00hrs, back to kip at 06:00, awake again at 09:30hrs.. Kip again about 23:59hrs.. and so on..

I request a public warning be posted:- SETI@HOME can be ADDICTIVE :-)

Regards,
____________
Cliff,
Been there, Done that, Still no damm T shirt!

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38322
Credit: 560,278,901
RAC: 653,159
United States
Message 1211580 - Posted: 29 Mar 2012, 16:27:34 UTC - in response to Message 1211572.

Things aint to good in d/l land either, every single WU download has had to be retried a min of 3 times..

Not a great problem during my waking hours, but on waking to find both rigs in project backoffs etc is getting a tad wearing:-)

I now find my waking hours are somewhat altered, up at between 03:00 and 04:00hrs, back to kip at 06:00, awake again at 09:30hrs.. Kip again about 23:59hrs.. and so on..

I request a public warning be posted:- SETI@HOME can be ADDICTIVE :-)

Regards,

LOL...you are talking to the KING of Seti induced bad sleep habits.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Josef W. Segur
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4203
Credit: 1,030,528
RAC: 265
United States
Message 1211632 - Posted: 29 Mar 2012, 18:53:52 UTC - in response to Message 1211392.

The difference now is that it'll take longer to get the completed count up to 10 since only validated tasks which ran full length and had less than 10% blanking go into the average. Based on our observations of v505 work showing about 1 in 4 having higher blanking plus the early exits, at 14 validated there would be about a 50/50 chance of having 10 in the average. But the protections ought to mean after 10 completed the estimates will be fairly close.
...

And so far what I've seen in my handy spreadsheet is that of the 11 APs that I have crunched and returned with r557, one was a B3_P1 with 100% blanking, and only three of the other 10 had less than 10% blanked.

It does of course depend on the data itself, but a very early and large margin-of-error estimate is about 1 in 3 have <10% based on my data.

Only one of your first 5 validated tasks went into the average, I hope that's just a Murphy's Law effect. As of last Monday, over 110000 AP v6 tasks from hosts running the stock Windows build had gone into the averages, but that discussion with Dr. korpela was on another subject and he didn't indicate how many total tasks were involved.

That one which did go into your average produced an APR very close to that you had for many AP v505 tasks, a good sign.
Joe

Profile S@NL - XP_Freak
Send message
Joined: 10 Jul 99
Posts: 99
Credit: 4,674,955
RAC: 2,121
Netherlands
Message 1211646 - Posted: 29 Mar 2012, 19:21:16 UTC - in response to Message 1211572.

Things aint to good in d/l land either, every single WU download has had to be retried a min of 3 times..


At the moment I can't download a single WU.
____________

Goodbye Seti Classic

Sten-Arne
Volunteer tester
Send message
Joined: 1 Nov 08
Posts: 3334
Credit: 19,048,788
RAC: 20,203
Sweden
Message 1211649 - Posted: 29 Mar 2012, 19:30:55 UTC

Same issue as many many times before. Direct connection to the download servers, and the download is totally borked. Finding a good proxy, preferrably on the U.S west coast, and downloading goes like a rocket.

WTF?
____________

Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · Next

Message boards : Number crunching : Panic Mode On (72) Server problems?

Copyright © 2014 University of California