Panic Mode On (91) Server Problems?

Message boards : Number crunching : Panic Mode On (91) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 15 · 16 · 17 · 18 · 19 · 20 · 21 · Next

AuthorMessage
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1598353 - Posted: 8 Nov 2014, 1:32:52 UTC - in response to Message 1598292.  

Lots of processes on the Server Status Page are presently not running/disabled (Validators, assimilators & AP splitters).
And those that say they are running, aren't. There are supposedly 6 PFB splitters running, but output has dropped to barely 10/s; that many splitters should be producing 30/s. Ready-to-send buffer is less than half full, so there should be no work to send in a hour or 3.
Hope they can sort it out soon, or it will be a weekend without any work.

Basically, every process that has to do with AP shows as disabled.

The AP science database is showing as running again. The rest of the AP processes still show disabled. Still, this is a good sign.

Maybe it will take overnight for marvin to get all synced up again?
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1598353 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 1598360 - Posted: 8 Nov 2014, 1:51:43 UTC

So I surmise that APs are not being sent out into the wild yet.
ID: 1598360 · Report as offensive
David S
Volunteer tester
Avatar

Send message
Joined: 4 Oct 99
Posts: 18352
Credit: 27,761,924
RAC: 12
United States
Message 1598366 - Posted: 8 Nov 2014, 2:17:13 UTC

Okay, it's after 6pm in Berkeley. I'm assuming Beta won't be back for the weekend. I let my tablet have some Main work and will do the same with my phone in a few minutes.
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.

ID: 1598366 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1598439 - Posted: 8 Nov 2014, 8:29:03 UTC

Something is realy wrong with the servers, AP split is stuck for days and now the MB spliting seems to work very wierd too (15 channels in progress!).

Does anyone realy knows what is happening? I see the msg about bruno but that was days ago.
ID: 1598439 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1598508 - Posted: 8 Nov 2014, 10:52:47 UTC - in response to Message 1598439.  

Something is realy wrong with the servers, AP split is stuck for days and now the MB spliting seems to work very wierd too (15 channels in progress!).

Does anyone realy knows what is happening? I see the msg about bruno but that was days ago.

Are you still seeing to many AP channels in progress? It has been showing 7 for me since they took all of the AP services down the other day. Currently all of the AP services are still disabled so there are really 0 active.
However I do see 14 in progress MB. Which is interesting considering there are only 8 MB splitters. With 7 of then running.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1598508 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1598513 - Posted: 8 Nov 2014, 11:21:29 UTC - in response to Message 1598508.  

Something is realy wrong with the servers, AP split is stuck for days and now the MB spliting seems to work very wierd too (15 channels in progress!).

Does anyone realy knows what is happening? I see the msg about bruno but that was days ago.

Are you still seeing to many AP channels in progress? It has been showing 7 for me since they took all of the AP services down the other day. Currently all of the AP services are still disabled so there are really 0 active.
However I do see 14 in progress MB. Which is interesting considering there are only 8 MB splitters. With 7 of then running.

I'm pretty certain that 'in progress' can mean 'started, but not completed'. In the same way, the BOINC clients running on our own computers can switch from project to project, leaving unfinished tasks 'waiting to run', and return to them later.

If you stop your local BOINC client, and start it again later, it won't necessarily restart exactly the same set of projects and tasks that were active before you shut it down. I think the splitters work the same way: if they are shut down for weekly maintenance, and restarted afterwards, they might start on a different batch from the ones which were active before the outrage. The cricket graph suggests that the server daemons were shut down for about an hour yesterday morning (Pacific time): I suspect the new tapes were added then, and the splitter tape selection algorithm saw the new batch of tapes and picked them when the daemons started back up.
ID: 1598513 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1598514 - Posted: 8 Nov 2014, 11:32:06 UTC - in response to Message 1598513.  

Something is realy wrong with the servers, AP split is stuck for days and now the MB spliting seems to work very wierd too (15 channels in progress!).

Does anyone realy knows what is happening? I see the msg about bruno but that was days ago.

Are you still seeing to many AP channels in progress? It has been showing 7 for me since they took all of the AP services down the other day. Currently all of the AP services are still disabled so there are really 0 active.
However I do see 14 in progress MB. Which is interesting considering there are only 8 MB splitters. With 7 of then running.

I'm pretty certain that 'in progress' can mean 'started, but not completed'. In the same way, the BOINC clients running on our own computers can switch from project to project, leaving unfinished tasks 'waiting to run', and return to them later.

If you stop your local BOINC client, and start it again later, it won't necessarily restart exactly the same set of projects and tasks that were active before you shut it down. I think the splitters work the same way: if they are shut down for weekly maintenance, and restarted afterwards, they might start on a different batch from the ones which were active before the outrage. The cricket graph suggests that the server daemons were shut down for about an hour yesterday morning (Pacific time): I suspect the new tapes were added then, and the splitter tape selection algorithm saw the new batch of tapes and picked them when the daemons started back up.

That does make sense, but I was thinking that during maintenance it typically showed 0 in progress. That may be due to specific things being offline at the time I suppose.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1598514 · Report as offensive
Profile cov_route
Avatar

Send message
Joined: 13 Sep 12
Posts: 342
Credit: 10,270,618
RAC: 0
Canada
Message 1598553 - Posted: 8 Nov 2014, 15:21:14 UTC - in response to Message 1598414.  

I'm going 0.25 on my work buffer (BOINC 6.10.58), in preparation for when AP splitting starts again. I don't want a big cache of MB's when and if AP is available again.

I wrote a Python script to do that automatically. I'm giving up on it though, things are just so chaotic it's pointless. I set SAH it to accept only AP and added Einstein, 3,333 credits for a wu is ok by me.
ID: 1598553 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1598568 - Posted: 8 Nov 2014, 16:14:27 UTC - in response to Message 1598524.  

Well, at least we're being told the reason for the problems (AP drought for example), as well as the approximate date for when it should be fixed...

I'm starting to think part of the Project is to elicit speculation from people who can only observe ;-)

Well, I've observed file 28mr11ab has already made it through one Tuesday session and remained stuck. So, expecting another Tuesday session to unstick it, and the others, may be optimistic. I've also observed that my 3 hosts have many more APv7 listings than APv6. We can only observe the APv6 numbers and speculate on the APv7 numbers. Looking at my 3 main hosts I see;
AstroPulse v6 (53) · AstroPulse v7 (990)
AstroPulse v6 (67) · AstroPulse v7 (1281)
AstroPulse v6 (177) · AstroPulse v7 (3696)
All we can observe are the APv6 numbers, Workunits waiting for assimilation: 41,963
We can speculate on the APv7 numbers.

I'm not really thrilled about using the same technology that has failed to find ET after 15 years and covering the entire observable sky at least once, maybe twice. I thought the newer technology might produce different results. Right now it appears I'm doing the same thing over again and expecting different results. Someone once said something about that scenario...
ID: 1598568 · Report as offensive
Profile Michael W.F. Miles
Avatar

Send message
Joined: 24 Mar 07
Posts: 268
Credit: 34,410,870
RAC: 0
Canada
Message 1598575 - Posted: 8 Nov 2014, 16:36:08 UTC - in response to Message 1598568.  

Someone once said something about that scenario...


Yes, it's called insanity
ID: 1598575 · Report as offensive
Profile Martin

Send message
Joined: 7 Aug 13
Posts: 3
Credit: 1,604,771
RAC: 2
United Kingdom
Message 1598576 - Posted: 8 Nov 2014, 16:36:55 UTC - in response to Message 1598136.  

Doesn't seem likely, given that the servers run on UTC, but as Ford said to Arthur on Fintlewoodlwix, in an infinite universe, anything is possible.

Answer is 42

What do you get if you multiply six by nine?

Rather more than you get if you multiply six by seven? ;)


6 times 9 = 42 (base 13)
ID: 1598576 · Report as offensive
JohnDK Crowdfunding Project Donor*Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 28 May 00
Posts: 1222
Credit: 451,243,443
RAC: 1,127
Denmark
Message 1598596 - Posted: 8 Nov 2014, 18:16:58 UTC - in response to Message 1598157.  

07/11/2014 19:45:07 SETI@home Message from server: Project is temporarily shut down for maintenance

Are we getting AP work for the weekend?

Guess not.
ID: 1598596 · Report as offensive
Profile cov_route
Avatar

Send message
Joined: 13 Sep 12
Posts: 342
Credit: 10,270,618
RAC: 0
Canada
Message 1598600 - Posted: 8 Nov 2014, 18:27:18 UTC - in response to Message 1598589.  

The project always intended to review the results it received, and Nitpicker was designed to take a second sift through the results already returned. For various reasons it is only running in Beta mode in the background. The algorithms used today are quite different to those used in Seti Classic, therefore it is reasonable to re-run certain results to get even more information.

There is also the time dimension. That means looking in the same direction at multiple points in time. Signals can be time varying. Wouldn't it be nice to catch the next WOW signal? As far as we know that happened only once.

SETI might have to keep looking for the next 100 years to hit on something like that. I hope they do. Keep looking for that long, I mean.
ID: 1598600 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13727
Credit: 208,696,464
RAC: 304
Australia
Message 1598686 - Posted: 8 Nov 2014, 21:55:36 UTC - in response to Message 1598600.  
Last modified: 8 Nov 2014, 21:56:23 UTC

Whatever the issue with the PFB splitters was, it seems to have sorted itself out. They're now keeping ahead of demand & able to rebuild the ready-to-send buffer & shut down when it is full & restart when it drops, as per normal.

However the AP validators, assimilators & splitters still all remain offline.
Grant
Darwin NT
ID: 1598686 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 1598691 - Posted: 8 Nov 2014, 22:01:36 UTC - in response to Message 1598686.  


However the AP validators, assimilators & splitters still all remain offline.

Also MB results waiting DB purging is over 1.8 million results it has been hovering between 1.7 and 1.8 for a few days now. I think this is to do with some services still not running on the up load server
ID: 1598691 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1598718 - Posted: 8 Nov 2014, 22:39:37 UTC - in response to Message 1598691.  


However the AP validators, assimilators & splitters still all remain offline.

Also MB results waiting DB purging is over 1.8 million results it has been hovering between 1.7 and 1.8 for a few days now. I think this is to do with some services still not running on the up load server

MB results waiting DB purging has averaged 1.55 million for the last year.

sah_purge
ID: 1598718 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1598797 - Posted: 9 Nov 2014, 0:25:31 UTC

ID: 1598797 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1598801 - Posted: 9 Nov 2014, 0:30:36 UTC

Eric's post clears up a lot of questions.

Basically, expect it to be at least another 7 days before AP gets split.


Speaking of splitters, when I checked a few minutes ago, MB was breezing along at nearly 40/sec and the RTS buffer was over 300k. Looks like AP is going to have a bit of catching up to do once that can be split again, but it will not take long for it to quickly go back into waiting for MB to catch up.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1598801 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 1598896 - Posted: 9 Nov 2014, 5:29:24 UTC - in response to Message 1598718.  


However the AP validators, assimilators & splitters still all remain offline.

Also MB results waiting DB purging is over 1.8 million results it has been hovering between 1.7 and 1.8 for a few days now. I think this is to do with some services still not running on the up load server

MB results waiting DB purging has averaged 1.55 million for the last year.

sah_purge

Thanks Richard, I was unaware it had been a round 1.5 million for a year
ID: 1598896 · Report as offensive
Profile Jeff Buck Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 11 Feb 00
Posts: 1441
Credit: 148,764,870
RAC: 0
United States
Message 1598900 - Posted: 9 Nov 2014, 5:41:14 UTC - in response to Message 1598896.  


However the AP validators, assimilators & splitters still all remain offline.

Also MB results waiting DB purging is over 1.8 million results it has been hovering between 1.7 and 1.8 for a few days now. I think this is to do with some services still not running on the up load server

MB results waiting DB purging has averaged 1.55 million for the last year.

sah_purge

Thanks Richard, I was unaware it had been a round 1.5 million for a year

Since results are purged about 24 hours after validation, I suspect that multiplying the "Results received in last hour" number by 24 will usually give a rough approximation of the "Results waiting for db purging" figure. Of course, server outages would screw that up a bit.
ID: 1598900 · Report as offensive
Previous · 1 . . . 15 · 16 · 17 · 18 · 19 · 20 · 21 · Next

Message boards : Number crunching : Panic Mode On (91) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.