Panic Mode On (114) Server Problems?

Message boards : Number crunching : Panic Mode On (114) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 13 · 14 · 15 · 16 · 17 · 18 · 19 . . . 45 · Next

AuthorMessage
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1971992 - Posted: 25 Dec 2018, 22:58:23 UTC - in response to Message 1971991.  

RTS over 800K. We have seen this before where the splitter cutoff mechanism doesn't trip correctly and the splitters continue to split but the scheduler doesn't recognize requests for work and sends out no tasks available messages. Looks like the scheduler picked up the pieces again and tasks are going out to refill caches.
That's the resends sending the RTS up Keith, not the splitters. ;-)

Cheers.

Really? We get 300K resends for every hour??
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1971992 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1972005 - Posted: 25 Dec 2018, 23:54:53 UTC - in response to Message 1971976.  

Thanks for the confirmation.
Hopefully just a chron job that will clear in a bit.

Meow.


. . I don't think I have asked this before, but what is a Chron job?

Stephen

? ?
ID: 1972005 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1972006 - Posted: 25 Dec 2018, 23:56:05 UTC - in response to Message 1971980.  

Well, the kitties will send the word out.
If Eric or Jeff are about and wish to attend to it, fine.
If it doesn't get fixed before tomorrow's outage, that is just fine too.

Meow.


. .Exactly ..

Stephen

. .
ID: 1972006 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1972007 - Posted: 26 Dec 2018, 0:04:18 UTC - in response to Message 1972005.  

Simply put, a shell script that runs on a timed interval.

If you look in your /etc folder there are cron.hourly ... cron.daily .. etc
If you put shell scripts in those folders, they will be started by the system.
ID: 1972007 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1972008 - Posted: 26 Dec 2018, 0:04:55 UTC - in response to Message 1972005.  

Thanks for the confirmation.
Hopefully just a chron job that will clear in a bit.

Meow.


. . I don't think I have asked this before, but what is a Chron job?

Stephen

? ?


cron job is just a script that is set to go off at a certain time.
ID: 1972008 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1972010 - Posted: 26 Dec 2018, 0:27:17 UTC - in response to Message 1972007.  

. . Hi Brent,

. . Thanks for the explanation ...

Stephen

:)
ID: 1972010 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1972022 - Posted: 26 Dec 2018, 2:25:19 UTC

Just got a bunch of gpu tasks after I went from 2 gpus to 5 gpus on one box :)

Tom
A proud member of the OFA (Old Farts Association).
ID: 1972022 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1972023 - Posted: 26 Dec 2018, 2:37:11 UTC - in response to Message 1972022.  

I assume you are speaking of Host #8565979? Looks like you are only crunching on the gpus? Only see 500 tasks in progress and it would be 600 if you used the cpu.

But I also see your other Host #8610031 with 579 tasks in progress but only two gpus listed. Have you got almost 300 'ghosts" on that machine? Or is the new five gpu host and the website hasn't caught up yet with the changes?
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1972023 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1972035 - Posted: 26 Dec 2018, 6:39:15 UTC - in response to Message 1972023.  

I assume you are speaking of Host #8565979? Looks like you are only crunching on the gpus? Only see 500 tasks in progress and it would be 600 if you used the cpu.

But I also see your other Host #8610031 with 579 tasks in progress but only two gpus listed. Have you got almost 300 'ghosts" on that machine? Or is the new five gpu host and the website hasn't caught up yet with the changes?


https://setiathome.berkeley.edu/show_host_detail.php?hostid=8565979

Let me check. Its showing 5 gpus now so must have been a slow update.

I seem to be getting gpu tasks but no seti cpu tasks right now on it. I run a mixed bunch of projects on that box so sometimes the scheduler is convinced I don't need Seti CPU tasks.

https://setiathome.berkeley.edu/show_host_detail.php?hostid=8610031

This one is pure Seti. And it only has two GPUs. Seems to be full up. So far.

But here comes Wed morning. :)

Tom
A proud member of the OFA (Old Farts Association).
ID: 1972035 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1972036 - Posted: 26 Dec 2018, 6:44:22 UTC - in response to Message 1972035.  

Host 8610031 was the one I was talking about. Either you have already started to bunker or it has 300 ghosts since its cache should only be 300. The task in progress shows 579.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1972036 · Report as offensive
Profile Freewill Project Donor
Avatar

Send message
Joined: 19 May 99
Posts: 766
Credit: 354,398,348
RAC: 11,693
United States
Message 1972048 - Posted: 26 Dec 2018, 11:38:27 UTC

Looks like the servers are slowing down just ahead of the shutdown. Cannot get any new tasks this morning on the east coast.

12/26/2018 6:34:49 AM | SETI@home | Reporting 4 completed tasks
12/26/2018 6:34:49 AM | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
12/26/2018 6:35:11 AM | SETI@home | Scheduler request failed: Couldn't connect to server
12/26/2018 6:35:12 AM | | Project communication failed: attempting access to reference site
12/26/2018 6:35:13 AM | | Internet access OK - project servers may be temporarily down.

Roger
ID: 1972048 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1855
Credit: 268,616,081
RAC: 1,349
United States
Message 1972050 - Posted: 26 Dec 2018, 11:58:45 UTC - in response to Message 1972048.  

Looks like the servers are slowing down just ahead of the shutdown. Cannot get any new tasks this morning on the east coast.

12/26/2018 6:34:49 AM | SETI@home | Reporting 4 completed tasks
12/26/2018 6:34:49 AM | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
12/26/2018 6:35:11 AM | SETI@home | Scheduler request failed: Couldn't connect to server
12/26/2018 6:35:12 AM | | Project communication failed: attempting access to reference site
12/26/2018 6:35:13 AM | | Internet access OK - project servers may be temporarily down.

Roger

Yep
ID: 1972050 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1972051 - Posted: 26 Dec 2018, 12:21:33 UTC
Last modified: 26 Dec 2018, 12:24:19 UTC

Wed 26 Dec 2018 07:19:25 AM EST | SETI@home | Scheduler request failed: HTTP internal server error

<Panic ON>
ID: 1972051 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1972052 - Posted: 26 Dec 2018, 12:35:59 UTC

I released my 2900ish bunkered tasks. I hope the project goes down soon :)
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1972052 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1972053 - Posted: 26 Dec 2018, 12:43:13 UTC

The data is flowing again, refill your caches while you can before the outage.
ID: 1972053 · Report as offensive
Profile Freewill Project Donor
Avatar

Send message
Joined: 19 May 99
Posts: 766
Credit: 354,398,348
RAC: 11,693
United States
Message 1972054 - Posted: 26 Dec 2018, 12:46:36 UTC - in response to Message 1972053.  

The data is flowing again, refill your caches while you can before the outage.


Thanks for the heads-up. What time does the outage usually start?
ID: 1972054 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1972055 - Posted: 26 Dec 2018, 12:48:09 UTC - in response to Message 1972053.  

The data is flowing again, refill your caches while you can before the outage.


Figures. Right as I get on the plane haha. If it goes down at 8am ET like it has been recently, then it won’t be much consequence. I can remote in and I have a layover in Chicago at 10am ET, but it will probably be down by then.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1972055 · Report as offensive
Sirius B Project Donor
Volunteer tester
Avatar

Send message
Joined: 26 Dec 00
Posts: 24882
Credit: 3,081,182
RAC: 7
Ireland
Message 1972064 - Posted: 26 Dec 2018, 15:44:47 UTC

Beta down.
ID: 1972064 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1972068 - Posted: 26 Dec 2018, 23:11:13 UTC
Last modified: 26 Dec 2018, 23:40:29 UTC

. . Interesting outage. It was the first 'false start' I have seen. It went down at 3:30pm UTC but was back up at 4:25pm, then went down again at 4:55pm. I know it was up because I received new work during that interval. Now it is back up at 11:00pm.

. . Hopefully it went smoothly despite that hiccough.

Stephen

:)

[edit]

. . Ruh Roh! It seems the upload/download servers are crumbling under the post outage onslaught ...
ID: 1972068 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1972073 - Posted: 26 Dec 2018, 23:45:22 UTC

Going to have to be patient. Servers are timing out on connection requests.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1972073 · Report as offensive
Previous · 1 . . . 13 · 14 · 15 · 16 · 17 · 18 · 19 . . . 45 · Next

Message boards : Number crunching : Panic Mode On (114) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.