Panic Mode On (113) Server Problems?

Message boards : Number crunching : Panic Mode On (113) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 37 · Next

AuthorMessage
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1953464 - Posted: 2 Sep 2018, 21:47:35 UTC

Just as I start to think I'm panicing all alone as the RTS starts to fall, they load another Aricebo file (31au18ab). Many thanks to the Seti person (people) taking time out of their holiday weekend to load more data.
ID: 1953464 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1953470 - Posted: 2 Sep 2018, 22:18:16 UTC - in response to Message 1953434.  

The RTS continues to slowly clim (228k at the moment), but the results waiting for db purge is at 4.8 million and also climbing.
I'm betting we get a long outrage on Tuesday to clean all this up.


. . If we are chewing on mainly/only Blc11 WUs this is going to be a very dry outrage. With about half of them still being noise bombs, our caches won't last long ...

Stephen

:(
ID: 1953470 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1953496 - Posted: 3 Sep 2018, 1:34:22 UTC - in response to Message 1953470.  



. . If we are chewing on mainly/only Blc11 WUs this is going to be a very dry outrage. With about half of them still being noise bombs, our caches won't last long ...

Stephen

:(


Hopefully we finish the blc11 WUs and start getting some from the remnants of the previous data files still left before the outage.
ID: 1953496 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1953499 - Posted: 3 Sep 2018, 1:58:03 UTC - in response to Message 1953496.  

I'm already seeing tasks from the 31Aug file.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1953499 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 1953511 - Posted: 3 Sep 2018, 3:04:47 UTC - in response to Message 1953499.  
Last modified: 3 Sep 2018, 4:02:37 UTC

I'm already seeing tasks from the 31Aug file.

These Vlar's run for a fraction over 12 minutes on my 970 using sog app
ID: 1953511 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1953515 - Posted: 3 Sep 2018, 4:31:43 UTC - in response to Message 1953511.  

Hmmm,
Even My 750Ti runs them in under 9 minutes using the CUDA App, https://setiathome.berkeley.edu/result.php?resultid=6945809274
A little under 4 minutes for a 1060 in MacOS, https://setiathome.berkeley.edu/result.php?resultid=6946013261
A 1080Ti on a Mac will do it in less than 2 minutes, https://setiathome.berkeley.edu/result.php?resultid=6946353300
Arecibo VLARs have been tamed...
ID: 1953515 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1953525 - Posted: 3 Sep 2018, 7:03:46 UTC

For those of you wondering where the BOINC forums went, they're now on https://boinc.berkeley.edu/dev/forum_index.php, while a lot of other links from the BOINC main page are broken. Someone is moving things around and hoping no one notices.
ID: 1953525 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13727
Credit: 208,696,464
RAC: 304
Australia
Message 1953527 - Posted: 3 Sep 2018, 7:28:06 UTC - in response to Message 1953525.  

For those of you wondering where the BOINC forums went, they're now on https://boinc.berkeley.edu/dev/forum_index.php, while a lot of other links from the BOINC main page are broken. Someone is moving things around and hoping no one notices.

We still haven't recovered from the last clean-up here. Colum widths for the forum topic pages and topics/threads are still all mucked up.
Grant
Darwin NT
ID: 1953527 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13727
Credit: 208,696,464
RAC: 304
Australia
Message 1953531 - Posted: 3 Sep 2018, 8:04:20 UTC

Getting "Project has no tasks available" messages, and the graphs are showing a spike in the Ready-to-send buffer, but no increase in splitter output.
Hopefully it's just the random recurring glitch again.
Grant
Darwin NT
ID: 1953531 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13727
Credit: 208,696,464
RAC: 304
Australia
Message 1953534 - Posted: 3 Sep 2018, 8:32:07 UTC

Hmm, just picked up a WU with a longer than usual estimated run time, and a 3 month deadline.
If it's not a noise bomb, it will be interesting to see how long it does take to crunch, and what the payout is.
Grant
Darwin NT
ID: 1953534 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1953540 - Posted: 3 Sep 2018, 8:47:41 UTC - in response to Message 1953525.  

For those of you wondering where the BOINC forums went, they're now on https://boinc.berkeley.edu/dev/forum_index.php
I was notified that it's about to be moved to https://boinc.berkeley.edu/forum_index.php (not there yet) to coincide with where the forums are on most all BOINC projects.
ID: 1953540 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1953541 - Posted: 3 Sep 2018, 8:53:15 UTC - in response to Message 1953540.  

At one point this morning I got redirected to http://boinc.berkeley.edu/dev/dev/forum_index.php (not making that a link because there's nothing there). It's certainly going walkabout.
ID: 1953541 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22184
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1953549 - Posted: 3 Sep 2018, 10:35:06 UTC

Still nothing there - Jord, you can ignore my PM as it predates me reading this conversation....
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1953549 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1953552 - Posted: 3 Sep 2018, 11:02:05 UTC - in response to Message 1953549.  

Still nothing there - Jord, you can ignore my PM as it predates me reading this conversation....


. . More to the point ... "project has no tasks available" on all machines, rapidly running out of work ...

Stephen :(

:(
ID: 1953552 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1953553 - Posted: 3 Sep 2018, 11:14:56 UTC - in response to Message 1953552.  

Still nothing there - Jord, you can ignore my PM as it predates me reading this conversation....


. . More to the point ... "project has no tasks available" on all machines, rapidly running out of work ...

Stephen :(

:(

Same here. 'no tasks available' even though the SPP shows work in RTS.
Meowsigh.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1953553 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1953554 - Posted: 3 Sep 2018, 11:24:09 UTC - in response to Message 1953553.  
Last modified: 3 Sep 2018, 12:18:26 UTC

Same here. 'no tasks available' even though the SPP shows work in RTS.
Meowsigh.

Allready running empty. Rescheduled the few remaning CPU WU to GPU but that will last for few minutes only since most of them are BLC11 noise bombs.

Following murphy's law, IIRC today's is a holiday in the US, labor day, somebody could confirm or not that, please.

<edit> As expected after 15 min al WU gone. Finishing the last 8 CPU WU and after that, the host will run dry.

With the new apps and the noise bombs the 100 cache limit is simply too small.
ID: 1953554 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1953558 - Posted: 3 Sep 2018, 12:02:59 UTC - in response to Message 1953554.  
Last modified: 3 Sep 2018, 12:18:22 UTC

Same here. 'no tasks available' even though the SPP shows work in RTS.
Meowsigh.

Allready running empty. Rescheduled the few remaning CPU WU to GPU but that will last for few minutes only since most of them are BLC11 noise bombs.

Following murphy's law, IIRC today's is a holiday in the US, labor day, somebody could confirm or not that, please.

<edit> As expected after 15 min al WU gone. Finishing the last 8 CPU WU and after that, the host will run dry.

With the new apps and the boise bombs the 100 cache limit is simply too small.


. . There may be some hope, this machine got a fair amount of work for the GPU but nothing for the CPU, so I repaid some WUs to the CPU for those I borrowed earlier. I will check the other machines soon.

. . Not much hope I'm afraid, the other machines are still getting nothing and one is dry, running the last CPU task as I type.

. . Someone needs to play kick the servers.

Stephen

:)
ID: 1953558 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1953560 - Posted: 3 Sep 2018, 12:18:29 UTC

Yes, it's Labor Day here in the USA.
I sent word to Eric and Jeff, but it's only 5:20AM in Berkeley and a holiday.
So I have no idea if either of them are available or willing to poke the servers until the outage tomorrow.

Meow.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1953560 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1953561 - Posted: 3 Sep 2018, 12:58:04 UTC
Last modified: 3 Sep 2018, 13:49:34 UTC

At 20 to 30 bombs every 5 minutes it doesn't take long to run out of work. The machine is down to just over a hundred tasks. What's that, about 20 minutes before it's empty?

Mon Sep  3 08:28:52 2018 | SETI@home | [sched_op] Starting scheduler request
Mon Sep  3 08:28:57 2018 | SETI@home | Reporting 20 completed tasks
Mon Sep  3 08:28:59 2018 | SETI@home | Scheduler request completed: got 0 new tasks
Mon Sep  3 08:28:59 2018 | SETI@home | Project has no tasks available
Mon Sep  3 08:34:09 2018 | SETI@home | Sending scheduler request: To report completed tasks.
Mon Sep  3 08:34:09 2018 | SETI@home | Reporting 19 completed tasks
Mon Sep  3 08:34:11 2018 | SETI@home | Scheduler request completed: got 0 new tasks
Mon Sep  3 08:34:11 2018 | SETI@home | Project has no tasks available
Mon Sep  3 08:39:20 2018 | SETI@home | Sending scheduler request: To report completed tasks.
Mon Sep  3 08:39:20 2018 | SETI@home | Reporting 10 completed tasks
Mon Sep  3 08:39:25 2018 | SETI@home | Scheduler request completed: got 0 new tasks
Mon Sep  3 08:39:25 2018 | SETI@home | Project has no tasks available
Mon Sep  3 08:44:36 2018 | SETI@home | Sending scheduler request: To report completed tasks.
Mon Sep  3 08:44:36 2018 | SETI@home | Reporting 31 completed tasks
Mon Sep  3 08:44:42 2018 | SETI@home | Scheduler request completed: got 0 new tasks
Mon Sep  3 08:44:42 2018 | SETI@home | Project has no tasks available
Mon Sep  3 08:49:52 2018 | SETI@home | Sending scheduler request: To report completed tasks.
Mon Sep  3 08:49:52 2018 | SETI@home | Reporting 28 completed tasks
Mon Sep  3 08:49:57 2018 | SETI@home | Scheduler request completed: got 0 new tasks
Mon Sep  3 08:49:57 2018 | SETI@home | Project has no tasks available
Mon Sep  3 08:55:08 2018 | SETI@home | Sending scheduler request: To report completed tasks.
Mon Sep  3 08:55:08 2018 | SETI@home | Reporting 22 completed tasks
Mon Sep  3 08:55:14 2018 | SETI@home | Scheduler request completed: got 0 new tasks
Mon Sep  3 08:55:14 2018 | SETI@home | Project has no tasks available

Now at 78 tasks, this will be over quickly...

Look at that. It was down to 12 tasks and I was about to call Bombs Away...but it got a reprieve,

Mon Sep  3 09:26:18 2018 | SETI@home | Sending scheduler request: To report completed tasks.
Mon Sep  3 09:26:18 2018 | SETI@home | Reporting 16 completed tasks
Mon Sep  3 09:26:20 2018 | SETI@home | Scheduler request completed: got 34 new tasks

34 won't last long...

That's it, Bombs Away.
4 more devices sitting idle;

Mon Sep 3 09:47:09 2018 | SETI@home | Sending scheduler request: To report completed tasks.
Mon Sep 3 09:47:09 2018 | SETI@home | Reporting 6 completed tasks
Mon Sep 3 09:47:09 2018 | SETI@home | Requesting new tasks for NVIDIA GPU
Mon Sep 3 09:47:09 2018 | SETI@home | [sched_op] NVIDIA GPU work request: 725760.00 seconds; 4.00 devices
Mon Sep 3 09:47:12 2018 | SETI@home | Scheduler request completed: got 0 new tasks
Mon Sep 3 09:47:12 2018 | SETI@home | Project has no tasks available
ID: 1953561 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1953571 - Posted: 3 Sep 2018, 14:53:54 UTC

I'm going to guess someone at Seti got things "fixed" or as fixed as they can be with this situation.
The server looks like it is SLOWLY giving out WUs, and the splitters are starting to run again.
But recovery will be slow and it is still only handing out crap data bombs.
ID: 1953571 · Report as offensive
Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 37 · Next

Message boards : Number crunching : Panic Mode On (113) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.