Panic Mode On (115) Server Problems?

Message boards : Number crunching : Panic Mode On (115) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 19 · 20 · 21 · 22 · 23 · 24 · 25 . . . 30 · Next

AuthorMessage
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1985866 - Posted: 18 Mar 2019, 20:34:13 UTC - in response to Message 1985863.  

mine should be pretty close to yours. I'm not having too much trouble loading tasks. seems about normal to me.

viewing individual host stats, all categories of tasks load withing 5 seconds.

the only time i saw a little more lag, was when viewing all host stats aggregated together (nearly 80k tasks). it still loads each category, but maybe 10-15s lag sometimes.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1985866 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1985868 - Posted: 18 Mar 2019, 20:42:36 UTC
Last modified: 18 Mar 2019, 20:45:45 UTC

In my case, try to see the pendings for example is simply impossible, you click, wait, wait & wait and nothing happening.
I just try now. Wait more than 15 secs and nothing...
But without warning sometimes it's works. Never understand why.
My host has a dedicated 160 DL /10 UL MB connection. So i imagine connection speed is out of the scope here.

<edit>Just because i spoke about, just tried again and after 10 secs the task list appears.
I imagine has something to do with the servers load.
ID: 1985868 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1985873 - Posted: 18 Mar 2019, 20:58:29 UTC

Yes, might be when I hit the database. My timing might be bad. Just now accessed my 73K tasks and it took 43 seconds to populate. I do that every day for my AP task collection for the day post. Getting an individual hosts tasks takes much shorter. But my comment about now and last week was for the same database access of total tasks on the account. The number of tasks hasn't changed but is taking longer now. I am chalking it up to a more fragmented database as a guess.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1985873 · Report as offensive
Sirius B Project Donor
Volunteer tester
Avatar

Send message
Joined: 26 Dec 00
Posts: 24991
Credit: 3,081,182
RAC: 7
Ireland
Message 1985999 - Posted: 19 Mar 2019, 19:06:36 UTC

Another fast outage.
ID: 1985999 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11451
Credit: 29,581,041
RAC: 66
United States
Message 1986001 - Posted: 19 Mar 2019, 19:10:06 UTC

It's back, but connecting to upload and report is an issue.
ID: 1986001 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1986002 - Posted: 19 Mar 2019, 19:13:03 UTC - in response to Message 1986001.  

yup. uploads haven't been going through the whole outage.

large cache of uploads stacking up, will likely prevent any downloads if they can't be flushed out and reported.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1986002 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1986014 - Posted: 19 Mar 2019, 20:10:21 UTC

Master database queries/second 4,259

4000+ is just too much for the system to handle. A bunch will have to get put off in backoff timing. It might need some tweaking of how this is done and for how long to deal with after outage recovery. I try and keep my system out of communications with the server for as long as I can on Tuesdays just to stay out of the way of those who need WUs. Saw on another thread that there are others who do that too... might be worth a thread to get more people to use the "Daily Schedules" feature to limit communications on Tuesday if they aren't in the likely to run out of WUs during a maintenance shutdown group.
ID: 1986014 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1986016 - Posted: 19 Mar 2019, 20:13:39 UTC

Yes we got into trouble the first part of the outage with the upload server offline along with the rest of the project. Lots of stalled uploads.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1986016 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11451
Credit: 29,581,041
RAC: 66
United States
Message 1986025 - Posted: 19 Mar 2019, 21:32:23 UTC

2 hrs after coming back uploads are still borked.
ID: 1986025 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1986026 - Posted: 19 Mar 2019, 21:56:12 UTC - in response to Message 1986001.  

It's back, but connecting to upload and report is an issue.


. . My rigs were OK before the outage with uploads but all during the outage and now they are having big problems uploading. Constantly hanging and going into system backoffs.

Stephen

:(
ID: 1986026 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5126
Credit: 276,046,078
RAC: 462
Message 1986032 - Posted: 19 Mar 2019, 23:21:25 UTC - in response to Message 1986014.  
Last modified: 19 Mar 2019, 23:27:00 UTC

...might be worth a thread to get more people to use the "Daily Schedules" feature to limit communications on Tuesday if they aren't in the likely to run out of WUs during a maintenance shutdown group.


Ok, I will byte :)

How do you use the local manager settings for the network over-ride, to shut down communications? I am getting the impression that the "time" needs to be based off of UTC rather than the local time equivalent?

Since my local UTC is -5 or -6 and they go down about 8am and come back up about 3pm (or earlier)[on short days] by my local time, what should my "day/time" setting be?

Tom
A proud member of the OFA (Old Farts Association).
ID: 1986032 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1986042 - Posted: 20 Mar 2019, 1:02:53 UTC - in response to Message 1986032.  

The time you suspend communications should also factor in how long your machine can go before running out of WUs to work on. I will admit that I turn communications back on at 23:45 UTC so that I can upload my finished WUs to keep my daily graph pretty. I also need to adjust my time as we had daylight savings and I didn't change it. You are right though that the setting takes your local time to act on and this needs to be thought of when the outage usually hits you as it is on California time that the events happen on a Tuesday.
ID: 1986042 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1861
Credit: 268,616,081
RAC: 1,349
United States
Message 1986043 - Posted: 20 Mar 2019, 1:34:41 UTC
Last modified: 20 Mar 2019, 1:38:21 UTC

Definitely still an issue with uploads
[edit] Who'd have thought posting a message would help :) [/edit]
ID: 1986043 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1986045 - Posted: 20 Mar 2019, 1:43:49 UTC - in response to Message 1986032.  
Last modified: 20 Mar 2019, 1:49:48 UTC

...might be worth a thread to get more people to use the "Daily Schedules" feature to limit communications on Tuesday if they aren't in the likely to run out of WUs during a maintenance shutdown group.


Ok, I will byte :)

How do you use the local manager settings for the network over-ride, to shut down communications? I am getting the impression that the "time" needs to be based off of UTC rather than the local time equivalent?

Since my local UTC is -5 or -6 and they go down about 8am and come back up about 3pm (or earlier)[on short days] by my local time, what should my "day/time" setting be?

Tom

You use your local time zone in the configuration (use the 24hr format)

I am at UTC -5 too and this are my settings: (in the global_prefs_override.xml file)

<day_prefs>
      <day_of_week>2</day_of_week>
      <net_start_hour>22.00</net_start_hour>
      <net_end_hour>8.00</net_end_hour>
</day_prefs>


It stops the network around the outage start time 8 AM and return after 14 hrs at 22:00 hrs (10 PM).

But i run with the spoofed builds so my large cache handle the long delay to restore the network access.

In your case, to return at 3PM as you posted, your must use: <net_start_hour>15.00</net_start_hour>
ID: 1986045 · Report as offensive
W-K 666 Project Donor
Volunteer tester

Send message
Joined: 18 May 99
Posts: 19988
Credit: 40,757,560
RAC: 67
United Kingdom
Message 1986046 - Posted: 20 Mar 2019, 2:06:09 UTC

Just looked on my account at the tasks, and in the valid or validation pending columns the time of the last task that reported in was 18 Mar 2019, 20:54:42 UTC.

But I cannot see any of the tasks that have been completed and reported in since then in the in progress pages, Looks like they are in some black hole.
ID: 1986046 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1986047 - Posted: 20 Mar 2019, 2:10:32 UTC - in response to Message 1986046.  

Just looked on my account at the tasks, and in the valid or validation pending columns the time of the last task that reported in was 18 Mar 2019, 20:54:42 UTC.

But I cannot see any of the tasks that have been completed and reported in since then in the in progress pages, Looks like they are in some black hole.


My tasks have been reported and uploaded with no problem (small amount), but they are not "processed" yet. I can see my result and the wingman result are there and credit has been given, but the WU is still in my "In Progress" column.

The system has some catching up to do.
ID: 1986047 · Report as offensive
W-K 666 Project Donor
Volunteer tester

Send message
Joined: 18 May 99
Posts: 19988
Credit: 40,757,560
RAC: 67
United Kingdom
Message 1986048 - Posted: 20 Mar 2019, 2:17:57 UTC - in response to Message 1986047.  
Last modified: 20 Mar 2019, 2:21:28 UTC

Just looked on my account at the tasks, and in the valid or validation pending columns the time of the last task that reported in was 18 Mar 2019, 20:54:42 UTC.

But I cannot see any of the tasks that have been completed and reported in since then in the in progress pages, Looks like they are in some black hole.


My tasks have been reported and uploaded with no problem (small amount), but they are not "processed" yet. I can see my result and the wingman result are there and credit has been given, but the WU is still in my "In Progress" column.

The system has some catching up to do.

I'll have a look through the "in progress" tasks to see if I can find some that are stuck in the processing stage, but it also looks like the processing stopped, from my evidence, about 18 hours before the outage.

edit] I can confirm that I have seen work units in the in progress pages that are validated.
But have yet to be transferred to the valid pages.
ID: 1986048 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11451
Credit: 29,581,041
RAC: 66
United States
Message 1986049 - Posted: 20 Mar 2019, 2:24:35 UTC - in response to Message 1986048.  

It wasn't broken but it got fixed.
ID: 1986049 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1986053 - Posted: 20 Mar 2019, 3:02:07 UTC - in response to Message 1986043.  

Who'd have thought posting a message would help :)
Post a message about my stuck uploads please.
If it worked for you, you have special powers!
ID: 1986053 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 38677
Credit: 261,360,520
RAC: 489
Australia
Message 1986055 - Posted: 20 Mar 2019, 3:04:52 UTC

I thought after preparing the brewroom (for another long session tomorrow) and having a feed at the top pub (along with a few beers of course) that the upload problem would've gone away,

but I was sadly wrong. :-)

Cheers.
ID: 1986055 · Report as offensive
Previous · 1 . . . 19 · 20 · 21 · 22 · 23 · 24 · 25 . . . 30 · Next

Message boards : Number crunching : Panic Mode On (115) Server Problems?


 
©2026 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.