Panic Mode On (113) Server Problems?

Message boards : Number crunching : Panic Mode On (113) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 34 · 35 · 36 · 37

AuthorMessage
Profile Pierre A Renaud
Avatar

Send message
Joined: 3 Apr 99
Posts: 998
Credit: 9,101,544
RAC: 65
Canada
Message 1965634 - Posted: 16 Nov 2018, 18:49:18 UTC - in response to Message 1965563.  
Last modified: 16 Nov 2018, 19:07:28 UTC

The wildfire situation in California is indeed not looking good when such measures have to be taken =:/

Worsening air quality forces cancellation of classes
By Public Affairs, UC Berkeley| November 15, 2018
https://news.berkeley.edu/2018/11/15/worsening-air-quality-forces-cancellation-of-classes/

The forecast for tomorrow from the Bay Area Air Quality Management District now indicates that conditions are expected to deteriorate. In that context, and based on the guidance of experts on campus, we must assume that the 200 AQI threshold will be exceeded tomorrow. As a result, we have decided to cancel all classes for the remainder of the day today, and on Friday.

Berkeley has cancelled classes today due to smoke issues. So unless the team can fix things remotely, we might be looking at a problem for a while. Hoping the Seti group stays safe and that the firefighters can get this fire out soon.

edit : just took a look at the status and all seems ok at the moment.

edit: ok, I see it now. it isn't cleaning up. my list of valids is way too long. more than the 24 hour standard. just as was said in previous post, it will go boom at some point, but for now is still running.
A preoccupying situation for which I have started a thread
Apr 3, 1999 - May 3, 2020
ID: 1965634 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1849
Credit: 268,616,081
RAC: 1,349
United States
Message 1965635 - Posted: 16 Nov 2018, 18:50:58 UTC

Well, if nothing else someone threw up a load of tapes sufficient to get through the weekend, by the looks of it ...
ID: 1965635 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13159
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1965640 - Posted: 16 Nov 2018, 19:19:23 UTC
Last modified: 16 Nov 2018, 19:21:25 UTC

I think the conclusion is that "no one" threw any new tapes onto the splitters. The Arecibo tapes are auto-mounted automatically by a script. Probably the Green Bank tapes too but not sure since I didn't see the post saying that GBT had a permanent lab connection.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1965640 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13396
Credit: 208,696,464
RAC: 304
Australia
Message 1965696 - Posted: 16 Nov 2018, 23:06:36 UTC - in response to Message 1965635.  

Well, if nothing else someone threw up a load of tapes sufficient to get through the weekend, by the looks of it ...

As long as the system doesn't choke on the Validation/Assimilation backlogs.
Grant
Darwin NT
ID: 1965696 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1965704 - Posted: 17 Nov 2018, 0:06:36 UTC - in response to Message 1965640.  

I think the conclusion is that "no one" threw any new tapes onto the splitters. The Arecibo tapes are auto-mounted automatically by a script. Probably the Green Bank tapes too but not sure since I didn't see the post saying that GBT had a permanent lab connection.


. . I remember some discussion about trialling one and deciding it was too expensive so it was dropped. I haven't seen anything since to say that it had been reinstated. There were many volunteers offering to share bandwidth and relay the data to Berkeley but obviously none were taken up. There was also mention of trying to feed it via an internet connection but it was waaay too slow and also worked out very expensive. So I am guessing he was right and someone (thank you) actually took part in the mounting of the additional tapes.

. . But all is for nought if the validators remain stalled and a logjam brings the splitters to a halt as before. For that matter the no14aa tape seems stuck as well.

Stephen

? ?
ID: 1965704 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13159
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1965709 - Posted: 17 Nov 2018, 0:30:25 UTC - in response to Message 1965704.  

Well Richard's post said the Arecibo files are automatically loaded remotely by a script. So I guess they could afford the permanent connection to AO but not the permanent connection to GBT.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1965709 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1965730 - Posted: 17 Nov 2018, 1:59:29 UTC - in response to Message 1965704.  
Last modified: 17 Nov 2018, 2:21:56 UTC

. . But all is for nought if the validators remain stalled and a logjam brings the splitters to a halt as before. For that matter the 14no18aa tape seems stuck as well.


. . Correction ...

Stephen

:(
ID: 1965730 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1965731 - Posted: 17 Nov 2018, 2:05:25 UTC - in response to Message 1965709.  
Last modified: 17 Nov 2018, 2:16:17 UTC

Well Richard's post said the Arecibo files are automatically loaded remotely by a script. So I guess they could afford the permanent connection to AO but not the permanent connection to GBT.


. . Yep, he did say they have a link to AO. That was not mentioned in the other discussion that I recall but they did say they were investigating getting a piggyback over some other government network. Maybe they were able to do that for AO, so maybe they managed it for GBT as well but that is purely speculation. But since they only seem to be getting 1 or 2 AO tapes per day they may have a cheap low bandwidth link that can support that amount of data. GBT data is far more prolific and would require some serious bandwidth. Even if the roughly 100 tapes that are being mounted in each of the series we have seen lately represent a weeks effort that is about a dozen tapes per day and could be fairly costly bandwidth. Maybe some kind soul will find the time and post about the actual details of their setup.

Stephen

? ?
ID: 1965731 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1965815 - Posted: 17 Nov 2018, 14:07:21 UTC
Last modified: 17 Nov 2018, 14:29:34 UTC

Sat 17 Nov 2018 09:02:40 AM EST |  | Project communication failed: attempting access to reference site
Sat 17 Nov 2018 09:02:40 AM EST | SETI@home | Scheduler request failed: Timeout was reached
Sat 17 Nov 2018 09:02:42 AM EST |  | Internet access OK - project servers may be temporarily down.


Get this answer when the host ask for new WU.

Something to turn ON the Panic Switch?

<edit> Back to work normally now.
ID: 1965815 · Report as offensive
James W

Send message
Joined: 26 May 12
Posts: 51
Credit: 4,956,027
RAC: 13
United States
Message 1966013 - Posted: 18 Nov 2018, 10:01:25 UTC - in response to Message 1965563.  


edit: ok, I see it now. it isn't cleaning up. my list of valids is way too long. more than the 24 hour standard. just as was said in previous post, it will go boom at some point, but for now is still running.

I've got a number of completed WUs dated 11/14 as well in my valid list. Agree with Grant that need to keep are fingers crossed that servers will hold out for the weekend!
ID: 1966013 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1966135 - Posted: 19 Nov 2018, 8:33:13 UTC - in response to Message 1965731.  
Last modified: 19 Nov 2018, 8:40:08 UTC

Well Richard's post said the Arecibo files are automatically loaded remotely by a script. So I guess they could afford the permanent connection to AO but not the permanent connection to GBT.

. . Yep, he did say they have a link to AO. That was not mentioned in the other discussion that I recall but they did say they were investigating getting a piggyback over some other government network. Maybe they were able to do that for AO, so maybe they managed it for GBT as well but that is purely speculation. But since they only seem to be getting 1 or 2 AO tapes per day they may have a cheap low bandwidth link that can support that amount of data. GBT data is far more prolific and would require some serious bandwidth. Even if the roughly 100 tapes that are being mounted in each of the series we have seen lately represent a weeks effort that is about a dozen tapes per day and could be fairly costly bandwidth. Maybe some kind soul will find the time and post about the actual details of their setup.
Stephen


. . Hi Keith, have you seen your fund raiser e-mail yet? Interesting timing. So the link to Arecibo is not a permanent data link but a virtual link over the internet as was proposed in the previously mentioned discussion.

. . The most exciting part of the letter is the information that data from Parkes is in house and about ready for the splitters. With all the GBT tapes mounted at the moment it will probably still be a while before we begin to see them but sooner now rather than later.

. . So many questions soon to be answered. Will they use the same BLC naming format as GBT? Will they use the same data format and will they process as the same rate or be slower? Will the target rich skies produce a higher rate of overflows (late not early) and will the nature of the area have a bigger noise problem?

. . They are still asking for more crunching power so maybe it is time for yet another Ryzen rig with a few GTX2080ti cards this time :). Sadly I have doubts that the servers can currently handle the load that a significant leap in processing power would produce ... :(

Stephen

:)
ID: 1966135 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 21175
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1966136 - Posted: 19 Nov 2018, 9:37:27 UTC

Given that the "telescope end" of the feed chain is based on that used for generating the data from Green Bank it is quite possible that the data format is the same, with just a different prefix code for the work unit name.
It will certainly be an interesting step, something from a "long way South", and a second "horizon to horizon" telescope. Actually about the same latitude as the GBT, just South rather than North of the equator.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1966136 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13159
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1966168 - Posted: 19 Nov 2018, 16:37:10 UTC - in response to Message 1966136.  

@Stephen, actually I think I was one of the first to receive the donation email letter and posted about it.https://setiathome.berkeley.edu/forum_thread.php?id=83560
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1966168 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1637
Credit: 12,921,799
RAC: 89
New Zealand
Message 1966312 - Posted: 20 Nov 2018, 5:53:16 UTC

Things seem to be holding well with over 1.3 million results waiting in the administrator queue. It will be interesting to see if it has any knock-on effect when it is cleared apart from more results unable to be deleted. Or maybe is this the new normal with the current throughput of work?
ID: 1966312 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13396
Credit: 208,696,464
RAC: 304
Australia
Message 1966331 - Posted: 21 Nov 2018, 2:54:51 UTC
Last modified: 21 Nov 2018, 2:56:11 UTC

The forums are back, now to see if there's any life with the servers.

Reporting is possible, "Projects has no new tasks" is the response to work requests.
Grant
Darwin NT
ID: 1966331 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13159
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1966332 - Posted: 21 Nov 2018, 3:04:46 UTC - in response to Message 1966331.  

But they barely made any dent in the validation, deletion, purging backlog.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1966332 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1966333 - Posted: 21 Nov 2018, 3:05:23 UTC - in response to Message 1966331.  

The forums are back, now to see if there's any life with the servers.

Reporting is possible, "Projects has no new tasks" is the response to work requests.


. . Same here, another 14 hour outrage ... :(

. . Machines out of work long ago ... :(

Stephen

:(
ID: 1966333 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13396
Credit: 208,696,464
RAC: 304
Australia
Message 1966335 - Posted: 21 Nov 2018, 3:11:57 UTC - in response to Message 1966332.  
Last modified: 21 Nov 2018, 3:24:45 UTC

But they barely made any dent in the validation, deletion, purging backlog.

Whatever they were doing, it wasn't that particular housekeeping.

Server Status is updating again, plenty of work available, the Scheduler just isn't giving it away at the moment.
"Project has no tasks available"


Edit-
That's everything reported, now we just need some new work.

Servers must be finally coming back to life- Database master-queries-per second has peaked, and the forums & web page are now in their go slow mode.
Grant
Darwin NT
ID: 1966335 · Report as offensive
Profile Pierre A Renaud
Avatar

Send message
Joined: 3 Apr 99
Posts: 998
Credit: 9,101,544
RAC: 65
Canada
Message 1966385 - Posted: 21 Nov 2018, 7:04:02 UTC

New thread started here
Apr 3, 1999 - May 3, 2020
ID: 1966385 · Report as offensive
Previous · 1 . . . 34 · 35 · 36 · 37

Message boards : Number crunching : Panic Mode On (113) Server Problems?


 
©2022 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.