Panic Mode On (93) Server Problems?

Message boards : Number crunching : Panic Mode On (93) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 . . . 24 · Next

AuthorMessage
WezH
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 576
Credit: 67,033,957
RAC: 95
Finland
Message 1612849 - Posted: 12 Dec 2014, 16:58:14 UTC - in response to Message 1612837.  

Hrmpf...
AP's please. I thought the database was fixed. Now it seems fixed, as in "the cat is fixed"
Grmpf.

:-)


They did fix the database problem, but maybe they know now the cause of problem?
"Please keep Your signature under four lines so Internet traffic doesn't go up too much"

- In 1992 when I had my first e-mail address -
ID: 1612849 · Report as offensive
Profile Uli
Volunteer tester
Avatar

Send message
Joined: 6 Feb 00
Posts: 10923
Credit: 5,996,015
RAC: 1
Germany
Message 1612883 - Posted: 12 Dec 2014, 18:05:59 UTC

You guys are too much. We had a massive storm pass thru. So good thing someone shut things down as a preventive measure or would you like to deal with other server failures.
There where also power outages in the Bay Area.

Give the guys a break. No matter what they do, it is never the right thing.

End rant!
Pluto will always be a planet to me.

Seti Ambassador
Not to late to order an Anni Shirt
ID: 1612883 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1612892 - Posted: 12 Dec 2014, 18:28:12 UTC - in response to Message 1612883.  

I´m not rant, just curious...

May i ask: Why the massive storm only afects the AP work? If all the servers are located in the same place/rack?
ID: 1612892 · Report as offensive
WezH
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 576
Credit: 67,033,957
RAC: 95
Finland
Message 1612897 - Posted: 12 Dec 2014, 18:35:52 UTC - in response to Message 1612883.  

You guys are too much. We had a massive storm pass thru. So good thing someone shut things down as a preventive measure or would you like to deal with other server failures.
There where also power outages in the Bay Area.

Give the guys a break. No matter what they do, it is never the right thing.

End rant!


[TROLL]
Yes. Storm. I know it.

Just wondering how only APv7 -services were shut down? Not the servers, like synergy. It is still running 'scheduling server' and 'scheduler process'. But not 'ap assimilator X' -service. Or server georgem.

Strange storm. Shutting down services, not servers.
[/TROLL]

I have never, ever, blamed S@H staff about these breakdowns (I blame funding). They do work as hard as possible they can do.

I respect them. Working long hours.

Respect.
"Please keep Your signature under four lines so Internet traffic doesn't go up too much"

- In 1992 when I had my first e-mail address -
ID: 1612897 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1612924 - Posted: 12 Dec 2014, 19:29:26 UTC - in response to Message 1612897.  

You guys are too much. We had a massive storm pass thru. So good thing someone shut things down as a preventive measure or would you like to deal with other server failures.
There where also power outages in the Bay Area.

Give the guys a break. No matter what they do, it is never the right thing.

End rant!


[TROLL]
Yes. Storm. I know it.

Just wondering how only APv7 -services were shut down? Not the servers, like synergy. It is still running 'scheduling server' and 'scheduler process'. But not 'ap assimilator X' -service. Or server georgem.

Strange storm. Shutting down services, not servers.
[/TROLL]

I have never, ever, blamed S@H staff about these breakdowns (I blame funding). They do work as hard as possible they can do.

I respect them. Working long hours.

Respect.

Did AP ever come back up when everything else was shut down? If it wasn't I would guess someone just forgot to start it back up.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1612924 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1612937 - Posted: 12 Dec 2014, 20:07:36 UTC - in response to Message 1612924.  


Did AP ever come back up when everything else was shut down? If it wasn't I would guess someone just forgot to start it back up.



Yes, if you mean after the normal project Tuesday turndown. AP briefly came back up and the site was extremely slow for a couple of hours as usual. When I came back to look at the router outbound loading and server stats, I noticed the AP generation was zero and then noticed the AP servers and processes were disabled.

Keith
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1612937 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1612942 - Posted: 12 Dec 2014, 20:19:25 UTC - in response to Message 1612937.  


Did AP ever come back up when everything else was shut down? If it wasn't I would guess someone just forgot to start it back up.



Yes, if you mean after the normal project Tuesday turndown. AP briefly came back up and the site was extremely slow for a couple of hours as usual. When I came back to look at the router outbound loading and server stats, I noticed the AP generation was zero and then noticed the AP servers and processes were disabled.

Keith

No I'm referring to yesterday when everything except the fora were off.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1612942 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13731
Credit: 208,696,464
RAC: 304
Australia
Message 1612951 - Posted: 12 Dec 2014, 20:34:45 UTC - in response to Message 1612942.  

I notice everything AP is still down, and MB still appears to have 14 splitters running on 13 files.
Grant
Darwin NT
ID: 1612951 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1612952 - Posted: 12 Dec 2014, 20:34:58 UTC - in response to Message 1612942.  
Last modified: 12 Dec 2014, 20:35:31 UTC

No I'm referring to yesterday when everything except the fora were off.

No, AP did not come back after that.

Timeline is.. around 1815utc (1015 local), basically everything was shut off for about 5 hours. After that, all of the MB processes came back up and running, but now nearly 24 hours later, ALL AP processes are still disabled.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1612952 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 1612956 - Posted: 12 Dec 2014, 20:48:13 UTC - in response to Message 1612952.  
Last modified: 12 Dec 2014, 20:48:37 UTC

ALL AP processes are still disabled.

That is a perfectly good reason to panic.
ID: 1612956 · Report as offensive
Profile Julie
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 28 Oct 09
Posts: 34053
Credit: 18,883,157
RAC: 18
Belgium
Message 1612961 - Posted: 12 Dec 2014, 20:59:35 UTC - in response to Message 1612956.  

ALL AP processes are still disabled.

That is a perfectly good reason to panic.


+1
rOZZ
Music
Pictures
ID: 1612961 · Report as offensive
David S
Volunteer tester
Avatar

Send message
Joined: 4 Oct 99
Posts: 18352
Credit: 27,761,924
RAC: 12
United States
Message 1612972 - Posted: 12 Dec 2014, 21:09:38 UTC

I suppose it's possible they shut down the AP database just in case the storm knocks it out, because they aren't sure if it would come back if it suddenly cut off. But I doubt it.
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.

ID: 1612972 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 1613098 - Posted: 13 Dec 2014, 1:10:08 UTC

I would be interested to see how long the current MB work supply lasts for. I hope we don't run out before APs are brought back online, as of 1 o'clock UTC there are 85 channels to do with 14 in progress and 111 done & 333 548 ready to send with the creation rate at 38.1414/sec
ID: 1613098 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 1613103 - Posted: 13 Dec 2014, 1:52:34 UTC - in response to Message 1613098.  

IMO, rather than doing nothing if we run out, we should reprocess very old data that was run wo the benefit of autocorrelation of other newer widgets.
ID: 1613103 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1853
Credit: 268,616,081
RAC: 1,349
United States
Message 1613108 - Posted: 13 Dec 2014, 2:03:39 UTC - in response to Message 1612924.  

Did AP ever come back up when everything else was shut down? If it wasn't I would guess someone just forgot to start it back up.

Yep, it was all up and running for quite a while. Then Marvin died, and my recollection is that since that's the ultimate end point for data, when it crashes the upstream processes shut down to avoid data loss. Afaik, this has no relationship to the earlier issues with Bruno that had everything down for so long, except that perhaps trying to deal with the massive backlog after Bruno was restored may have caused Marvin's crash. Would only make sense ... but what do I know:)
ID: 1613108 · Report as offensive
Profile cliff
Avatar

Send message
Joined: 16 Dec 07
Posts: 625
Credit: 3,590,440
RAC: 0
United Kingdom
Message 1613132 - Posted: 13 Dec 2014, 3:42:34 UTC - in response to Message 1613098.  

Hi Speedy,
Well not too long:-(

Stats page shows loads of GPU tasks, but none are actually forthcoming when S@H updates, I just get told CPU tasks are available, but I haven't selected CPU in settings..

Something electronic needs a boot in the rear end:-)

Regards,
Cliff,
Been there, Done that, Still no damm T shirt!
ID: 1613132 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 1613140 - Posted: 13 Dec 2014, 4:14:57 UTC - in response to Message 1613132.  

Hi Speedy,
Well not too long:-(

Stats page shows loads of GPU tasks, but none are actually forthcoming when S@H updates, I just get told CPU tasks are available, but I haven't selected CPU in settings..

Something electronic needs a boot in the rear end:-)

Regards,

I just received 32 GPU tasks 13/12/2014 5:11:45 p.m. | SETI@home | Scheduler request completed: got 32 new tasks
someone must of given it a boot as you said
ID: 1613140 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13731
Credit: 208,696,464
RAC: 304
Australia
Message 1613142 - Posted: 13 Dec 2014, 4:25:59 UTC - in response to Message 1613140.  

I just received 32 GPU tasks 13/12/2014 5:11:45 p.m. | SETI@home | Scheduler request completed: got 32 new tasks
someone must of given it a boot as you said

There are still a lot of VLAR tasks about, so it can take several requests for GPU work before you're likely to get some.
Grant
Darwin NT
ID: 1613142 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1613150 - Posted: 13 Dec 2014, 4:51:58 UTC - in response to Message 1613142.  

Looks like more problems for with the server.

Sounds like it's really giving them some problems.

Fingers crossed and positive thoughts for them
ID: 1613150 · Report as offensive
Profile Mark Wyzenbeek
Avatar

Send message
Joined: 28 Jun 99
Posts: 134
Credit: 6,203,079
RAC: 0
United States
Message 1613152 - Posted: 13 Dec 2014, 4:59:16 UTC

Eric posted in Technical News why the astropulse servers are down. Message
The Universe is not only stranger than you imagine, it's stranger than you can imagine.

SETI@home classic workunits 1,405 CPU time 57,318 hours
ID: 1613152 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 . . . 24 · Next

Message boards : Number crunching : Panic Mode On (93) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.