Panic Mode On (114) Server Problems?

Message boards : Number crunching : Panic Mode On (114) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 19 · 20 · 21 · 22 · 23 · 24 · 25 . . . 45 · Next

AuthorMessage
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1973303 - Posted: 3 Jan 2019, 23:24:50 UTC

Notice: Undefined variable: nresults in /disks/carolyn/b/home/boincadm/projects/sah/html/inc/host.inc on line 207

Wow. I was hoping it would be all fixed by the time I got home. Thanks to the seti people who have to deal with this issue.
ID: 1973303 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1973305 - Posted: 3 Jan 2019, 23:46:19 UTC

I think the roadblock just cleared, two of my computers just got big hits for downloads.
Fingers crossed ...
ID: 1973305 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1973306 - Posted: 3 Jan 2019, 23:54:22 UTC - in response to Message 1973305.  

I think the roadblock just cleared, two of my computers just got big hits for downloads.
Fingers crossed ...


. . Always! ... lately ...

Stephen

:(
ID: 1973306 · Report as offensive
Profile Bill G Special Project $75 donor
Avatar

Send message
Joined: 1 Jun 01
Posts: 1282
Credit: 187,688,550
RAC: 182
United States
Message 1973308 - Posted: 4 Jan 2019, 0:06:56 UTC - in response to Message 1973306.  

Tasks may be downloading but there are still parts of the site that are not working, esp. those that deal with tasks.

SETI@home classic workunits 4,019
SETI@home classic CPU time 34,348 hours
ID: 1973308 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1853
Credit: 268,616,081
RAC: 1,349
United States
Message 1973312 - Posted: 4 Jan 2019, 0:44:46 UTC

Unable to handle request
This feature is turned off temporarily

While trying to get status of tasks ...
ID: 1973312 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1973313 - Posted: 4 Jan 2019, 0:49:13 UTC - in response to Message 1973308.  

Yes, you can't view your Tasks. So you have no idea how many you have unless you have a third party tool like BoincTasks.

You could always physically count the number you have in your cache, possible now with the small amount of work that has been delivered. Would only take ten fingers and ten toes.

You also can't see if a task is validated, inconclusive, errored or pending. Viewing tasks is one of the purposes of having the replica database to relief the I/O stress on the main database by anyone wanting to view their current task cache.

I don't know whether turning off task viewing is allowing the replica lag to finally start reducing from its peak. I still believe the replica being brought back online is the root cause of all the projects troubles lately in delivering work.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1973313 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1853
Credit: 268,616,081
RAC: 1,349
United States
Message 1973330 - Posted: 4 Jan 2019, 2:19:01 UTC - in response to Message 1973313.  
Last modified: 4 Jan 2019, 2:22:38 UTC

Yes, you can't view your Tasks. So you have no idea how many you have unless you have a third party tool like BoincTasks. ...
Yeah, I've been running Fred's BoincTasks for years (in spite of its instability on at least my Win10, it's the only decent game in town for centrally monitoring multiple crunchers) so no big deal.
Of course BT can't deal with credit granted, validation state or the status of wingmen so some web functions remain useful.
Just thought it was curious to see a result like that (perhaps indicative of new development, to shed load on the DB when needed?)
ID: 1973330 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13751
Credit: 208,696,464
RAC: 304
Australia
Message 1973341 - Posted: 4 Jan 2019, 3:49:35 UTC
Last modified: 4 Jan 2019, 3:50:13 UTC

24 hours later and the recovery has only just started. And even then, it's at an extremely slow pace.
And even after that bit of recovery, we're still behind where we usually are after the weekly outage. Almost 4.5 million in progress, another 600k to go till things are back to normal.

Given the difficulties still occurring in getting work, that could be a long way off yet.
Things haven't been good for a few weeks now, but it looks like they are starting to come to a head.
Grant
Darwin NT
ID: 1973341 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13751
Credit: 208,696,464
RAC: 304
Australia
Message 1973347 - Posted: 4 Jan 2019, 5:13:41 UTC

And to add to that, splitter output is less than demand- now that work is finally going out again (In-progress is about halfway to it's usual level), the splitters can't keep up. Ready-to-send is rapidly declining.
Hopefully the splitters will get a move along before we run out.
Grant
Darwin NT
ID: 1973347 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1973357 - Posted: 4 Jan 2019, 6:57:10 UTC

Went into the other room for some reason and saw that the HDD activity light on Altair was lit up solid. RDP was very slow, and everything was very slow to react/respond, though it eventually did. Not sure what was up with it, but I figured with 53 days since last restart.. might as well, since BOINC had crashed approximately 48 hours ago and even telling the service to stop and start again didn't fix it.

System restart fixed it. I also--for some reason--had to fix the date and mash the 'update now' button for internet time. I don't know why or when it happened, but the clock became +78640 seconds from what it should have been, which was too far out of the range for w32time to adjust which required manual adjustment to get it close, and then 'update now' will do it properly.

Anyway, did that, and restarted, and looks like BOINC came back up just fine.

2019-01-04 01:48:39 SETI@home [sched_op_debug] Starting scheduler request
2019-01-04 01:48:39 SETI@home Sending scheduler request: To fetch work. Requesting 3454 seconds of work, reporting 6 completed tasks
2019-01-04 01:48:40 Project communication failed: attempting access to reference site
2019-01-04 01:48:41 Internet access OK - project servers may be temporarily down.

2019-01-04 01:48:44 SETI@home Scheduler request succeeded: got 1 new tasks
2019-01-04 01:48:44 SETI@home [sched_ops_debug] Server version 709
2019-01-04 01:48:44 SETI@home Project requested delay of 303.000000 seconds
2019-01-04 01:48:44 [sched_op_debug] handle_scheduler_reply(): got ack for result 26dc18ac.28654.1550684.6.33.97.vlar_0
2019-01-04 01:48:44 [sched_op_debug] handle_scheduler_reply(): got ack for result blc16_2bit_guppi_58405_85309_GJ687_0026.28254.818.22.45.101.vlar_0
2019-01-04 01:48:44 [sched_op_debug] handle_scheduler_reply(): got ack for result 29dc18aa.13524.19290.16.43.51_1
2019-01-04 01:48:44 [sched_op_debug] handle_scheduler_reply(): got ack for result blc16_2bit_guppi_58405_85972_GJ687_0028.24700.818.21.44.119.vlar_0
2019-01-04 01:48:44 [sched_op_debug] handle_scheduler_reply(): got ack for result blc16_2bit_guppi_58405_86306_HIP85612_0029.19316.409.21.44.81.vlar_1
2019-01-04 01:48:44 [sched_op_debug] handle_scheduler_reply(): got ack for result blc16_2bit_guppi_58405_85972_GJ687_0028.1839.818.21.44.134.vlar_2
2019-01-04 01:48:44 SETI@home [sched_op_debug] Deferring communication for 5 min 3 sec
2019-01-04 01:48:44 SETI@home [sched_op_debug] Reason: requested by project
2019-01-04 01:48:46 SETI@home Started download of blc16_2bit_guppi_58406_25930_HIP20901_0100.24062.818.21.44.133.vlar
2019-01-04 01:48:49 SETI@home Finished download of blc16_2bit_guppi_58406_25930_HIP20901_0100.24062.818.21.44.133.vlar


Still weird that it says comms failed...but completes the comms anyway, uploads WUs, and downloads new ones without issues. It is only during scheduler request that it immediately says that it failed.

I'm aware of the scheduler being hammered after maintenance..or an unscheduled outage, but my logs show this behavior for several months now. what's up with that?
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1973357 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13751
Credit: 208,696,464
RAC: 304
Australia
Message 1973363 - Posted: 4 Jan 2019, 9:42:35 UTC
Last modified: 4 Jan 2019, 10:13:17 UTC

Looks like the usual daily glitch has begun. Forums slow & "Project has no tasks available" Scheduler responses.

Edit- and it's been almost an hour since most of the Server stats were updated.
Grant
Darwin NT
ID: 1973363 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1853
Credit: 268,616,081
RAC: 1,349
United States
Message 1973367 - Posted: 4 Jan 2019, 10:21:11 UTC - in response to Message 1973363.  
Last modified: 4 Jan 2019, 11:11:30 UTC

All GBT splitters now shut down.
ID: 1973367 · Report as offensive
Chris Oliver Project Donor
Avatar

Send message
Joined: 4 Jul 99
Posts: 72
Credit: 134,288,250
RAC: 15
United Kingdom
Message 1973370 - Posted: 4 Jan 2019, 10:40:32 UTC

I have not seen one single new WU in nearly 36 hours so I'm boosting my rac on Primegrid.
ID: 1973370 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1973378 - Posted: 4 Jan 2019, 13:25:13 UTC - in response to Message 1973370.  

I’ve been getting work regularly since last night. But no stats update on the website.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1973378 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1973390 - Posted: 4 Jan 2019, 15:35:48 UTC

the results out in the field has been slowly climbing and is at 4.8 now, which I hope means that most people now have have some seti data.
ID: 1973390 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22227
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1973422 - Posted: 4 Jan 2019, 18:33:23 UTC

It looks as if the stats are beginning to come back to life.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1973422 · Report as offensive
Profile Kissagogo27 Special Project $75 donor
Avatar

Send message
Joined: 6 Nov 99
Posts: 716
Credit: 8,032,827
RAC: 62
France
Message 1973444 - Posted: 4 Jan 2019, 20:26:11 UTC

some stuck in tape processing ... not FIFO ? but LIFO ?

not seen any BLC 15 14 12 6 5 4 Wu in my cache ...
ID: 1973444 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1973464 - Posted: 4 Jan 2019, 21:13:00 UTC - in response to Message 1973444.  

some stuck in tape processing ... not FIFO ? but LIFO ?

not seen any BLC 15 14 12 6 5 4 Wu in my cache ...


. . That's because the Blc16 tapes being split are older than the tapes of the other varieties. When they reach the same age tapes the others will begin to split again. 1st the Blc15's because they have the next oldest tapes, then the 14/12s etc.

Stephen

:)
ID: 1973464 · Report as offensive
Profile Kissagogo27 Special Project $75 donor
Avatar

Send message
Joined: 6 Nov 99
Posts: 716
Credit: 8,032,827
RAC: 62
France
Message 1973602 - Posted: 5 Jan 2019, 12:23:28 UTC

okay, thks for your reply :)
ID: 1973602 · Report as offensive
Profile Kissagogo27 Special Project $75 donor
Avatar

Send message
Joined: 6 Nov 99
Posts: 716
Credit: 8,032,827
RAC: 62
France
Message 1973626 - Posted: 5 Jan 2019, 15:21:57 UTC

it seems that the "no task available " horror show is back ...

for CPU here


05-Jan-2019 15:40:44 [SETI@home] Sending scheduler request: To fetch work.
05-Jan-2019 15:40:44 [SETI@home] Requesting new tasks for CPU
05-Jan-2019 15:40:57 [SETI@home] Scheduler request completed: got 0 new tasks
05-Jan-2019 15:40:57 [SETI@home] Project has no tasks available
05-Jan-2019 15:53:03 [SETI@home] Sending scheduler request: To fetch work.
05-Jan-2019 15:53:03 [SETI@home] Requesting new tasks for CPU
05-Jan-2019 15:53:12 [SETI@home] Scheduler request completed: got 0 new tasks
05-Jan-2019 15:53:12 [SETI@home] Project has no tasks available
05-Jan-2019 15:57:24 [SETI@home] Computation for task blc16_2bit_guppi_58406_26617_HIP20917_0102.21911.409.21.44.210.vlar_0 finished
05-Jan-2019 15:57:24 [SETI@home] Starting task blc16_2bit_guppi_58406_26617_HIP20917_0102.21911.409.21.44.228.vlar_0
05-Jan-2019 15:57:26 [SETI@home] Started upload of blc16_2bit_guppi_58406_26617_HIP20917_0102.21911.409.21.44.210.vlar_0_r509214313_0
05-Jan-2019 15:57:30 [SETI@home] Finished upload of blc16_2bit_guppi_58406_26617_HIP20917_0102.21911.409.21.44.210.vlar_0_r509214313_0
05-Jan-2019 16:02:50 [SETI@home] update requested by user
05-Jan-2019 16:02:53 [SETI@home] Sending scheduler request: Requested by user.
05-Jan-2019 16:02:53 [SETI@home] Reporting 1 completed tasks
05-Jan-2019 16:02:53 [SETI@home] Requesting new tasks for CPU
05-Jan-2019 16:02:58 [SETI@home] Scheduler request completed: got 0 new tasks
05-Jan-2019 16:02:58 [SETI@home] Project has no tasks available


and then for GPU got some Wu with slow downloading rates and CPU too

forum is a little bit laggy too ;)
ID: 1973626 · Report as offensive
Previous · 1 . . . 19 · 20 · 21 · 22 · 23 · 24 · 25 . . . 45 · Next

Message boards : Number crunching : Panic Mode On (114) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.