The Server Issues / Outages Thread - Panic Mode On! (118)

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (118)
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 11 · 12 · 13 · 14 · 15 · 16 · 17 . . . 94 · Next

AuthorMessage
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 2025043 - Posted: 26 Dec 2019, 6:33:12 UTC

Results-received-in-last-hour 171,859. Well, that's certainly going to give the servers a pounding.
Grant
Darwin NT
ID: 2025043 · Report as offensive
Profile Chris904395093209d Project Donor
Volunteer tester

Send message
Joined: 1 Jan 01
Posts: 112
Credit: 29,923,129
RAC: 6
United States
Message 2025072 - Posted: 26 Dec 2019, 14:53:11 UTC

I'm getting the 'Project has not tasks available' message this morning on my 2 windows machines. At least I was able to get restocked last night.
~Chris

ID: 2025072 · Report as offensive
Profile B. Ahmet KIRAN

Send message
Joined: 19 Oct 14
Posts: 77
Credit: 36,140,903
RAC: 140
Turkey
Message 2025073 - Posted: 26 Dec 2019, 14:54:57 UTC

Since last 45 minutes, I have been updating 5 computers, all the time the system responding "no tasks available" while the "work done" and "average work done" seems to be frozen... Where do these results go??? Am I the only one who is being targeted??? I see no posts in this issue (now nearly one hour passed and still no updates on work done and average work done)
SOMEONE PLEASE RESPOND...
ID: 2025073 · Report as offensive
Oddbjornik Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 15 May 99
Posts: 220
Credit: 349,610,548
RAC: 1,728
Norway
Message 2025075 - Posted: 26 Dec 2019, 15:14:40 UTC - in response to Message 2025073.  

Since last 45 minutes, I have been updating 5 computers, all the time the system responding "no tasks available" while the "work done" and "average work done" seems to be frozen... Where do these results go??? Am I the only one who is being targeted??? I see no posts in this issue (now nearly one hour passed and still no updates on work done and average work done)
SOMEONE PLEASE RESPOND...
I'm guessing the servers are busy with the daily statistics dump, and that the system will be back in 3-2-1...
ID: 2025075 · Report as offensive
Profile B. Ahmet KIRAN

Send message
Joined: 19 Oct 14
Posts: 77
Credit: 36,140,903
RAC: 140
Turkey
Message 2025077 - Posted: 26 Dec 2019, 15:41:49 UTC - in response to Message 2025075.  

Thanks for the update...
Is there a time schedule for this "daily statistics dump"? If so I will try to avoid those times for my uploads, and prevent my stomach acid burst...
Thanks again...
ID: 2025077 · Report as offensive
Oddbjornik Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 15 May 99
Posts: 220
Credit: 349,610,548
RAC: 1,728
Norway
Message 2025078 - Posted: 26 Dec 2019, 15:51:46 UTC - in response to Message 2025077.  

Thanks for the update...
Is there a time schedule for this "daily statistics dump"? If so I will try to avoid those times for my uploads, and prevent my stomach acid burst...
Thanks again...
I don't think the schedule is very reliable, but the files from the dump go here, and the process will have started at the first file timestamp and ended at the last one.
The time of day will normally be the same from day to day, but then it will occasionally jump to some other time.
And there are two runs a day, with approximately 12 hours between.
ID: 2025078 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 2025079 - Posted: 26 Dec 2019, 15:58:23 UTC - in response to Message 2024760.  
Last modified: 26 Dec 2019, 16:00:46 UTC

Another problem shared from BETA, some of these Host detail numbers have been frozen all day. Namely the APR numbers.
SETI@home v8 8.11 x86_64-apple-darwin (cuda42_mac)
Number of tasks completed 135
Max tasks per day 156
Number of tasks today 2873
Consecutive valid tasks 54
Average processing rate 1,740.18 GFLOPS

SETI@home v8 8.19 x86_64-apple-darwin (opencl_nvidia_mac_old)
Number of tasks completed 139
Max tasks per day 52
Number of tasks today 2000
Consecutive valid tasks 2
Average processing rate 1,768.32 GFLOPS

By this time, the cuda42 numbers should be quite a bit higher, but, they haven't changed.
https://setiathome.berkeley.edu/host_app_versions.php?hostid=6796479
Just an update with the current 709 Server. Now the numbers are being updated normally, every time you report work and it's validated the APR numbers change. Using the 715 Server the APR numbers would stay frozen for weeks.

SETI@home v8 8.11 x86_64-apple-darwin (cuda42_mac)
Number of tasks completed 4895
Max tasks per day 5665
Number of tasks today 1854
Consecutive valid tasks 80
Average processing rate 1,786.14 GFLOPS

SETI@home v8 8.19 x86_64-apple-darwin (opencl_nvidia_mac_old)
Number of tasks completed 371
Max tasks per day 255
Number of tasks today 0
Consecutive valid tasks 21
Average processing rate 1,463.42 GFLOPS
ID: 2025079 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 2025082 - Posted: 26 Dec 2019, 16:05:17 UTC

Everything seems quite stabilized now. there was a little hiccup about an hour ago but i just had to kick the Update button on a few machines. no biggie.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 2025082 · Report as offensive
Profile B. Ahmet KIRAN

Send message
Joined: 19 Oct 14
Posts: 77
Credit: 36,140,903
RAC: 140
Turkey
Message 2025092 - Posted: 26 Dec 2019, 17:56:32 UTC - in response to Message 2025078.  
Last modified: 26 Dec 2019, 17:57:00 UTC

Thanks a zillion for the invaluable info... I'll be watching this address I did not know existed...
Thanks for the incredible response... :)
ID: 2025092 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 2025150 - Posted: 27 Dec 2019, 6:21:25 UTC
Last modified: 27 Dec 2019, 6:47:56 UTC

Trying to view my Error list results in minutes of spinning wheels and then this pops up;
Error tasks for computer 6813106
Database Error
Database Error
Warning: Invalid argument supplied for foreach() in /disks/carolyn/b/home/boincadm/projects/sah/html/inc/result.inc on line 757 Database Error
Warning: Invalid argument supplied for foreach() in /disks/carolyn/b/home/boincadm/projects/sah/html/inc/result.inc on line 766
No tasks to display
Should we be concerned?

This is rising again, Replica seconds behind master: 2,728
ID: 2025150 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 2025154 - Posted: 27 Dec 2019, 7:29:23 UTC

I didn't get any errors when I looked up my stats, but my list is probably alot shorter. The errors do sound scary.

The replica is falling behind, but the assimilation number is being worked on... it is now down to 1.4 million, when it was over 2 million yesterday. I think the system can't do it all well at the same time, so it is focused on assimilation at the moment, so replica lag and increase in db purging numbers happen. Hopefully the system will turn to focus lag and db purging in a bit.
ID: 2025154 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 2025162 - Posted: 27 Dec 2019, 9:29:44 UTC

I just noticed something interesting when looking at my tasks page...

I have 200 in progress instead of 100. CPU-only machine.

Not complaining.. just interesting that the task limit got doubled since last I knew.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 2025162 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34748
Credit: 261,360,520
RAC: 489
Australia
Message 2025165 - Posted: 27 Dec 2019, 9:59:25 UTC - in response to Message 2025162.  

I just noticed something interesting when looking at my tasks page...

I have 200 in progress instead of 100. CPU-only machine.

Not complaining.. just interesting that the task limit got doubled since last I knew.
The whole story thus far, https://setiathome.berkeley.edu/forum_thread.php?id=84932&sort_style=6&start=0. ;-)

Cheers.
ID: 2025165 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 2025172 - Posted: 27 Dec 2019, 10:51:16 UTC
Last modified: 27 Dec 2019, 11:00:21 UTC

Scheduler starting to supply less tasks than reported, and got a couple of "Project has no tasks available" responses as well (Edit- make that 4 in a row, so far).
Must be heading in to the weekend.
Grant
Darwin NT
ID: 2025172 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 2025174 - Posted: 27 Dec 2019, 11:02:07 UTC - in response to Message 2025172.  

Scheduler starting to supply less tasks than reported, and got a couple of "Project has no tasks available" responses as well (Edit- make that 4 in a row, so far).
Must be heading in to the weekend.

The servers know it's friday and we have a holiday ahead. LOL

Yes everything seems slow now.
ID: 2025174 · Report as offensive
Profile Siran d'Vel'nahr
Volunteer tester
Avatar

Send message
Joined: 23 May 99
Posts: 7379
Credit: 44,181,323
RAC: 238
United States
Message 2025175 - Posted: 27 Dec 2019, 11:14:54 UTC

Greetings,

And the server says?!:
Results ready to send	0	0	571,913	16m

And yet my Event log says:
Fri 27 Dec 2019 05:07:31 AM CST | SETI@home | Reporting 3 completed tasks
Fri 27 Dec 2019 05:07:31 AM CST | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
Fri 27 Dec 2019 05:07:31 AM CST | SETI@home | [sched_op] CPU work request: 70476.83 seconds; 0.00 devices
Fri 27 Dec 2019 05:07:31 AM CST | SETI@home | [sched_op] NVIDIA GPU work request: 70092.50 seconds; 1.00 devices
Fri 27 Dec 2019 05:07:32 AM CST | SETI@home | Scheduler request completed: got 0 new tasks
Fri 27 Dec 2019 05:07:32 AM CST | SETI@home | [sched_op] Server version 709
Fri 27 Dec 2019 05:07:32 AM CST | SETI@home | Project has no tasks available

It is most definitely Friday, going into a weekend and another holiday week. :|

Have a great day and a Happy New Year! :)

Siran
CAPT Siran d'Vel'nahr - L L & P _\\//
Winders 11 OS? "What a piece of junk!" - L. Skywalker
"Logic is the cement of our civilization with which we ascend from chaos using reason as our guide." - T'Plana-hath
ID: 2025175 · Report as offensive
Profile Mr. Kevvy Crowdfunding Project Donor*Special Project $250 donor
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 15 May 99
Posts: 3776
Credit: 1,114,826,392
RAC: 3,319
Canada
Message 2025184 - Posted: 27 Dec 2019, 12:49:14 UTC
Last modified: 27 Dec 2019, 17:52:53 UTC

Looks like this is a common issue... also getting no tasks available yet with 763K in the RTS queue and the replica was only 46 seconds out from the master when I checked. As well the scheduler is very slow to respond and also throwing HTTP errors unless I enable NNT and/or set max_tasks_reported to 125 or less in cc_config.

So other than the scheduler version showing as 709, it's just what was happening before the restoration. Is someone messing with the codebase again? Sigh.

Edit: To confirm there are still issues, I tried to view Invalids only both for my entire account and host 8873220 and got:

Database Error
Database Error
Warning: Invalid argument supplied for foreach() in /disks/carolyn/b/home/boincadm/projects/sah/html/inc/result.inc on line 757 Database Error
Warning: Invalid argument supplied for foreach() in /disks/carolyn/b/home/boincadm/projects/sah/html/inc/result.inc on line 766


This was at approx. 09h50 PST, 17h50 UTC.
ID: 2025184 · Report as offensive
Profile Gary Charpentier Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 25 Dec 00
Posts: 30651
Credit: 53,134,872
RAC: 32
United States
Message 2025189 - Posted: 27 Dec 2019, 14:30:03 UTC - in response to Message 2025184.  

Is someone messing with the codebase again? Sigh.
Well it is getting close to end of year, were funds left that needed to be spent?</cynic>
ID: 2025189 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 2025223 - Posted: 27 Dec 2019, 19:15:33 UTC - in response to Message 2025184.  
Last modified: 27 Dec 2019, 19:16:18 UTC


Edit: To confirm there are still issues, I tried to view Invalids only both for my entire account and host 8873220 and got:

Database Error
Database Error
Warning: Invalid argument supplied for foreach() in /disks/carolyn/b/home/boincadm/projects/sah/html/inc/result.inc on line 757 Database Error
Warning: Invalid argument supplied for foreach() in /disks/carolyn/b/home/boincadm/projects/sah/html/inc/result.inc on line 766



Somebody noticed because while it took its own sweet time, I just got a display of the 3 invalids I currently have listed.

Tom
A proud member of the OFA (Old Farts Association).
ID: 2025223 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2025266 - Posted: 28 Dec 2019, 1:57:26 UTC

Back to the issue that one can't view any tasks on the host because the page never populates. Replica is caught up too.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2025266 · Report as offensive
Previous · 1 . . . 11 · 12 · 13 · 14 · 15 · 16 · 17 . . . 94 · Next

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (118)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.