Panic Mode On (100) Server Problems?

Message boards : Number crunching : Panic Mode On (100) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 11 · 12 · 13 · 14 · 15 · 16 · 17 . . . 32 · Next

AuthorMessage
Ulrich Metzner
Volunteer tester
Avatar

Send message
Joined: 3 Jul 02
Posts: 1256
Credit: 13,565,513
RAC: 13
Germany
Message 1727035 - Posted: 19 Sep 2015, 18:18:44 UTC

Have a look over here: http://ucbsystems.org/
Aloha, Uli

ID: 1727035 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1727037 - Posted: 19 Sep 2015, 18:21:13 UTC

There was a fire in a another co-lo server (not one of SETI's) at the co-lo facility. It triggered the halon fire-suppression system, and the whole facility had to be evacuated and powered down. It's slowly being brought back to life, bit by bit.
ID: 1727037 · Report as offensive
Swibby Bear

Send message
Joined: 1 Aug 01
Posts: 246
Credit: 7,945,093
RAC: 0
United States
Message 1727040 - Posted: 19 Sep 2015, 18:29:57 UTC

Thank you for the info update. I thought it was bad, because all of the university internet connection was down. I was guessing some contractor cut a line, but wow - a fire! Hope all is well after the smoke clears.
ID: 1727040 · Report as offensive
Profile Gary Charpentier Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 25 Dec 00
Posts: 30651
Credit: 53,134,872
RAC: 32
United States
Message 1727044 - Posted: 19 Sep 2015, 18:33:47 UTC

ID: 1727044 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1727050 - Posted: 19 Sep 2015, 18:52:45 UTC

Or, as it's now been posted on the SETI front page,

There was a fire in the data center last night.

The entire data center (where our servers are) had to be shut down. That crisis is now over and we're carefully bringing our servers back up. See details here.
ID: 1727050 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1853
Credit: 268,616,081
RAC: 1,349
United States
Message 1727064 - Posted: 19 Sep 2015, 19:21:12 UTC

Good argument here for bumping up the per CPU/GPU cache above 100 WUs, to lessen the impact of these things.
Been out of work here for hours, and outages are going to happen from time to time.
ID: 1727064 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1727066 - Posted: 19 Sep 2015, 19:26:09 UTC - in response to Message 1727064.  

Good argument here for bumping up the per CPU/GPU cache above 100 WUs, to lessen the impact of these things.
Been out of work here for hours, and outages are going to happen from time to time.

Better argument for choosing an alternate project or several. My machines have been fully occupied all day.
ID: 1727066 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 1727082 - Posted: 19 Sep 2015, 20:23:51 UTC

SSP shows only 5 servers up but that is probably fixable and not cause for a panic yet.
ID: 1727082 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 1727087 - Posted: 19 Sep 2015, 20:33:29 UTC - in response to Message 1727085.  

SSP shows only 5 servers up but that is probably fixable and not cause for a panic yet.

Bah, we've been in full panic for almost 24 hours now :-)

A real panic looks like this, PANIC
ID: 1727087 · Report as offensive
Profile TimeLord04
Volunteer tester
Avatar

Send message
Joined: 9 Mar 06
Posts: 21140
Credit: 33,933,039
RAC: 23
United States
Message 1727088 - Posted: 19 Sep 2015, 20:34:29 UTC

Well, one computer, (Exeter, GTX-760), is out of work and CAN'T Upload completed work. The other computer, (Prometheus, GTX-750 TI SC), is almost out of work, and also CAN'T Upload completed work. :-(


TL
TimeLord04
Have TARDIS, will travel...
Come along K-9!
Join Calm Chaos
ID: 1727088 · Report as offensive
WezH
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 576
Credit: 67,033,957
RAC: 95
Finland
Message 1727092 - Posted: 19 Sep 2015, 20:47:27 UTC

Well, it seems that uploads are working somehow now... Couple WU's uploaded, but I think that servers are in heavy stress right now...
ID: 1727092 · Report as offensive
Profile TimeLord04
Volunteer tester
Avatar

Send message
Joined: 9 Mar 06
Posts: 21140
Credit: 33,933,039
RAC: 23
United States
Message 1727095 - Posted: 19 Sep 2015, 20:53:59 UTC - in response to Message 1727092.  

Well, it seems that uploads are working somehow now... Couple WU's uploaded, but I think that servers are in heavy stress right now...

I have 46 Units on Exeter stuck in perpetual Upload, going NOWHERE... :-(

About the same on Prometheus... :-(

Nothings moving!!!... :-(


TL
TimeLord04
Have TARDIS, will travel...
Come along K-9!
Join Calm Chaos
ID: 1727095 · Report as offensive
Profile Mark Wyzenbeek
Avatar

Send message
Joined: 28 Jun 99
Posts: 134
Credit: 6,203,079
RAC: 0
United States
Message 1727096 - Posted: 19 Sep 2015, 20:56:12 UTC

It took some button clicking, but I got my uploads uploaded.
The Universe is not only stranger than you imagine, it's stranger than you can imagine.

SETI@home classic workunits 1,405 CPU time 57,318 hours
ID: 1727096 · Report as offensive
WezH
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 576
Credit: 67,033,957
RAC: 95
Finland
Message 1727097 - Posted: 19 Sep 2015, 20:56:30 UTC - in response to Message 1727095.  
Last modified: 19 Sep 2015, 20:59:51 UTC

Well, it seems that uploads are working somehow now... Couple WU's uploaded, but I think that servers are in heavy stress right now...

I have 46 Units on Exeter stuck in perpetual Upload, going NOWHERE... :-(

About the same on Prometheus... :-(

Nothings moving!!!... :-(


They are moving, but very slowly. My old laptop uploaded and reported tasks, and got new ones and it's crunching again.

So Time and Patience :)

There is almost 180.000 computers trying to send results and asking for more.
ID: 1727097 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1727107 - Posted: 19 Sep 2015, 21:18:23 UTC - in response to Message 1727096.  
Last modified: 19 Sep 2015, 21:35:19 UTC

It took some button clicking, but I got my uploads uploaded.

A lot of clicking.
At the present rate it'll probably 20-30min before I get enough uploads through to get new work, when there's some in the feeder at the time I request it.
Best upload I've had so far was about 5kB/s. 56k dialup speed. Most are barely 3kB/s.

So actually downloading work could take, forever...


EDIT- just got some work on one systems, and no issues with downloads at all.

EDIT- and to exacerbate the upload issues, and people's lack of work, 95% of my first couple of downloads have been shorties...
Grant
Darwin NT
ID: 1727107 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1727117 - Posted: 19 Sep 2015, 21:52:44 UTC - in response to Message 1727107.  

I notice on the Server Status page a lot of channels have ended in error; it may be worth re-running them in case it was due to the co-lo fire/heat/shutdown issues.

And unfortunately the splitters are still struggling. Given the demand & their low output it won't be long before the ready-to-send buffer is empty again.
Although as long as there are upload issues that will help reduce the load if there aren't people there to keep hitting retry.
Grant
Darwin NT
ID: 1727117 · Report as offensive
Dave(The Admiral)Nelson

Send message
Joined: 4 Jun 99
Posts: 415
Credit: 22,293,483
RAC: 1
United States
Message 1727120 - Posted: 19 Sep 2015, 21:58:45 UTC - in response to Message 1715033.  

Take heart; if you're half as intelligent as the average person on these forums that makes you twice as intelligent as I. Everything is relative.

"Too know that one is ignorant is the beginning of wisdom".
Dave Nelson
ID: 1727120 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1853
Credit: 268,616,081
RAC: 1,349
United States
Message 1727184 - Posted: 20 Sep 2015, 2:08:15 UTC - in response to Message 1727066.  

Good argument here for bumping up the per CPU/GPU cache above 100 WUs, to lessen the impact of these things.
Been out of work here for hours, and outages are going to happen from time to time.

Better argument for choosing an alternate project or several. My machines have been fully occupied all day.

Well, if there was alternate project crunching SETI data, or anything else I was interested in, I would most assuredly be signed up :)
ID: 1727184 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1727197 - Posted: 20 Sep 2015, 3:15:33 UTC

Glad that the colo fire was minimal, nobody was hurt, and the Seti servers seem to have been spared any damage from the incident.

I am sure it was ET with a high energy plasma beam targeting the Seti servers because we are getting too close to finding them.
Glad that they hit the wrong server.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1727197 · Report as offensive
Profile Gary Charpentier Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 25 Dec 00
Posts: 30651
Credit: 53,134,872
RAC: 32
United States
Message 1727220 - Posted: 20 Sep 2015, 4:15:30 UTC - in response to Message 1727184.  

Good argument here for bumping up the per CPU/GPU cache above 100 WUs, to lessen the impact of these things.
Been out of work here for hours, and outages are going to happen from time to time.

Better argument for choosing an alternate project or several. My machines have been fully occupied all day.

Well, if there was alternate project crunching SETI data, or anything else I was interested in, I would most assuredly be signed up :)

Well, Einstein @ Home is crunching the same data as Seti from Areicbo, just looking for pulsars.
ID: 1727220 · Report as offensive
Previous · 1 . . . 11 · 12 · 13 · 14 · 15 · 16 · 17 . . . 32 · Next

Message boards : Number crunching : Panic Mode On (100) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.