Panic Mode On (74) Server problems?
Panic Mode On (74) Server problems? |
|
| log in |
Message boards : Number crunching : Panic Mode On (74) Server problems?
Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · Next
| Author | Message |
|---|---|
This WU looks like it'll be fun. _0 and _1 both failed on download. _2 managed to get it, supposedly. _3 and _4 (me) failed on download. _5 is unsent at the moment. Yeah, me too, lots of these: 1735 SETI@home 2012-05-18 21:17:31 [error] File 23fe11ab.28540.3508.8.10.14 has wrong size: expected 375361, got 0 1736 SETI@home 2012-05-18 21:17:31 [error] Checksum or signature error for 23fe11ab.28540.3508.8.10.14 ____________ /The grumpy old Swede.  "I'm so old, that 98% of all trees in the forest, are younger than I am" | |
ID: 1232940 ·  | |
This WU looks like it'll be fun. _0 and _1 both failed on download. _2 managed to get it, supposedly. _3 and _4 (me) failed on download. _5 is unsent at the moment. Could that be due to the RAID resynchs Matt mentioned earlier? Or maybe just saturation on the download pipe? ____________ Donald Infernal Optimist / Submariner, retired | |
| ID: 1233000 · | |
|
Josef W. Segur  Volunteer developer Volunteer tester Send message Joined: 30 Oct 99 Posts: 3932 Credit: 934,327 RAC: 15  | |
This WU looks like it'll be fun. _0 and _1 both failed on download. _2 managed to get it, supposedly. _3 and _4 (me) failed on download. _5 is unsent at the moment. See Khangollo's message 1232644, posted before the power failure, showing that kind of error. Whatever the problem is, it's not an aftereffect of the power failure. Joe | |
| ID: 1233051 · | |
This WU looks like it'll be fun. _0 and _1 both failed on download. _2 managed to get it, supposedly. _3 and _4 (me) failed on download. _5 is unsent at the moment. I suspect some of the drive volumes were unreachable by the download servers shortly before everything shut down. Are all the servers on the same UPS ? | |
| ID: 1233071 · | |
|
Finally done uploading for me. | |
| ID: 1233080 · | |
I suspect some of the drive volumes were unreachable by the download servers shortly before everything shut down. Are all the servers on the same UPS ? I'm pretty sure they've got several UPSs. Just had a look at my tasks, and i've got quite a few download errors as well. If you look at the WUs, that occured for everyone else trying to download those WUs, although there several which were just downloaded today with no problems. So your theory sounds good. The Scheduler & download servers were still up, but the work file storage was down (or at least in accessable) at the time. Might be an idea for someone that can to notifiy the staff- many of those WUs have been cancelled (Too many errors (may have bug) Work Unit cancelled) when in actual fact they are OK. ____________ Grant Darwin NT. | |
| ID: 1233082 · | |
|
Josef W. Segur Volunteer developer Volunteer tester Send message Joined: 30 Oct 99 Posts: 3932 Credit: 934,327 RAC: 15 | |
IIRC they only have 120 volt 20 amp AC circuits in the closet so are restricted to using 2200 VA UPS units. I don't know how many servers are on each UPS, but one UPS certainly won't handle all the listed servers. The post I referenced was made long enough before the power outage I don't think they're related, though of course it was in the recovery period from the scheduled Tuesday outage so is similar to the current conditions that way. Joe | |
| ID: 1233087 · | |
|
They have several UPS systems in the closet. I'll ask Eric if they need upgrades/more. | |
| ID: 1233093 · | |
I suspect some of the drive volumes were unreachable by the download servers shortly before everything shut down. Are all the servers on the same UPS ? I hope they are on the case. Someone seems to have cancelled some WU's before they reached the Max # of errors. e.g this one http://setiathome.berkeley.edu/workunit.php?wuid=992536710 which still has 2 tasks in progress, but has been cancelled. I thought there were more than one UPS for all those boxes, I suspect the drive arrays may have gone down before the servers due to the higher power requirement of all those spinning motors. | |
| ID: 1233100 · | |
|
Richard Haselgrove Volunteer tester Send message Joined: 4 Jul 99 Posts: 7461 Credit: 39,727,302 RAC: 34,698  | |
Owing to timezones, I don't usually see the Tuesday maintenance outage recovery. When I went to bed on Tuesday, the outage was over: message boards were up: old work had been reported, and new work had been allocated: and I had a large download queue. When I peeked in on Wednesday morning, Cricket was already flatlined. All my queued downloads had completed, except some were showing as download error. That suggests that the zero file size error is more likely to be a splitter problem... ...except I suppose we could concoct an explanation where the download servers, switches and routers were all on a UPS which kept the downloads alive for the first few minutes of the outage, but the workunit storage array was either unprotected, or on a UPS which ran out of juice before the communications link died. That might create the symptoms observed. | |
| ID: 1233106 · | |
|
Josef W. Segur Volunteer developer Volunteer tester Send message Joined: 30 Oct 99 Posts: 3932 Credit: 934,327 RAC: 15 | |
|
Another issue, maybe... Joe | |
| ID: 1233115 · | |
Another issue, maybe... Clicked on all three and they all work (here). | |
| ID: 1233143 · | |
Another issue, maybe... All 3 links work fine for me as well. ____________  | |
| ID: 1233154 · | |
|
Power went down between Tuesday 10:35PM PDT and a few hours before. | |
| ID: 1233157 · | |
No main GPU work available though. ? GPU work is all i'm getting at the moment. Not expecting any CPU work till the GPU cache has been topped up sufficiently. So far there's only been a couple of shorties in sight, so hopefully it won't take too long. ____________ Grant Darwin NT. | |
| ID: 1233171 · | |
|
Just got word from Eric. Good news is that there doesn't appear to have been any lost hardware. We were worried about lost hardware in light of Bane's death along with Dan's old workstation during the last outage. | |
| ID: 1233174 · | |
Just got word from Eric. Good news is that there doesn't appear to have been any lost hardware. We were worried about lost hardware in light of Bane's death along with Dan's old workstation during the last outage. Excellent news.... ____________ ****** "Ask not, what your kitty can do for you. Ask what you can do for your kitty." As it is kitten, so shall it be done.  | |
| ID: 1233176 · | |
No main GPU work available though. I had to complain about it before I got any apparently. ____________   | |
| ID: 1233177 · | |
This WU looks like it'll be fun. _0 and _1 both failed on download. _2 managed to get it, supposedly. _3 and _4 (me) failed on download. _5 is unsent at the moment. Since these download errors were already being discussed nearly four hours before the power went out, they can't very well be a result of it. If the Cricket graph is a reliable indicator of the time (I don't see why it wouldn't be), the outage started at just about precisely (suspiciously precisely) 2000 PDT, or 0300 UTC. I have 39 such download errors, most if not all from several hours before 0300 UTC, so once again, they are clearly not a result of the power outage. What I'm wondering about is that some of my cancelled WUs still show as in progress for other users. ____________ David Sitting on my butt while others boldly go, Waiting for a message from a small furry creature from Alpha Centauri.  | |
| ID: 1233217 · | |
Another issue, maybe... I'm logged on the the first url already, normally have to go through the certificate error to do so, looking at my account on the 2nd and 3rd url gives me the certificate error, Claggy | |
| ID: 1233306 · | |
Message boards : Number crunching : Panic Mode On (74) Server problems?
| Copyright © 2013 University of California |
