Panic Mode On (116) Server Problems?

Message boards : Number crunching : Panic Mode On (116) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 30 · 31 · 32 · 33 · 34 · 35 · 36 . . . 47 · Next

AuthorMessage
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1997338 - Posted: 8 Jun 2019, 2:38:41 UTC - in response to Message 1997310.  

Keith I'm sorry you don't think I'm convinced. I find it interesting that all talk of change to 4 bit work has been discussed on beta however since it has gone to main there has been no mention of it on these boards. Simple reason I am not going to PM Matt, Jeff & Eric is because I believe they only respond to a few people.


. . That is the simplest thing to understand. In Beta it was new and under test and the discussion there was about the manner in which it processed and whether there were any issues. When it moved to main that was all very much settled. But there were still some messages about the new format, I know that for a fact because I posted some of them, but the topic faded quickly because for the rank and file it was a non-event. There were no obvious changes apart from the larger files, which was pretty much the sole topic in main, allied with the fact they did NOT taken any longer to process which was the only major concern with the larger file sizes. I think there were one or two who were worried about the extra download time because they were on slow internet connections but it was all over quite quickly. I am sure if you want to go back over the closed versions of this thread dating back about 2 years or so you can still find those messages.

Stephen

:(
ID: 1997338 · Report as offensive
Profile Pierre A Renaud
Avatar

Send message
Joined: 3 Apr 99
Posts: 998
Credit: 9,101,544
RAC: 65
Canada
Message 1997376 - Posted: 8 Jun 2019, 8:06:08 UTC - in response to Message 1997337.  

OK, Wiggo . . . . you had me going Huh? with that reference. So I had to look it up to see what it might be referring to.

Landed on this video. https://www.youtube.com/watch?v=3sHE_AzN3lU
And this. https://www.telegraph.co.uk/news/worldnews/australiaandthepacific/newzealand/4315307/Swirling-cloud-captured-above-New-Zealand-The-Land-of-the-Long-White-Cloud.html
Totally cool. I get it now.
Really nice finds, Keith.

Now returning to the standard Panic Mode ^^
Apr 3, 1999 - May 3, 2020
ID: 1997376 · Report as offensive
Profile Pierre A Renaud
Avatar

Send message
Joined: 3 Apr 99
Posts: 998
Credit: 9,101,544
RAC: 65
Canada
Message 1997381 - Posted: 8 Jun 2019, 8:51:55 UTC

Scheduled Maintenance: Tuesday 2019-06-11

Start: 2019-06-11 04:00:00 PDT
End: 2019-06-11 07:00:00 PDT
Severity: Minor Performance Issue
Status: Planning

IST network will be replacing the hardware for one of the campus core routers.

The other core router will continue to carry traffic during the maintenance and no service interruption is expected.

-----

Start: 2019-06-11 05:30:00 PDT
End: 2019-06-11 06:30:00 PDT
Severity: Major Performance Issue
Status: Planning

One of the two DHCP servers providing network address assignments to the UC Berkeley campus will be physically moved to a new rack and network location.

During the time the DHCP server is offline network assignments may be delayed for some customers (less than a minute) due to the the way redundancy works with the DHCP protocol.

Apr 3, 1999 - May 3, 2020
ID: 1997381 · Report as offensive
Kiska
Volunteer tester

Send message
Joined: 31 Mar 12
Posts: 302
Credit: 3,067,762
RAC: 0
Australia
Message 1997391 - Posted: 8 Jun 2019, 12:27:27 UTC - in response to Message 1997288.  
Last modified: 8 Jun 2019, 12:30:04 UTC

Yes, we went from the 2 to 4 bit shortly after Eric posted that in the Beta thread back in 2016. No follow up post by him in Beta since it went live on Main.

Yes you are absolutely correct the post I am referring to was posted in 2016. Since there has been no update I still believe we are processing 2 bit work. Until I see 4 bit in the task name or hear from Jeff, Matt, Eric or Mark or see an official post on main I will have to agree to disagree that we are processing 4 bit work


I pulled this from "blc25_2bit_guppi_58405_83639_HIP85509_0021.21321.0.21.44.254.vlar"

  <recorder_cfg>
    <name>seti_gbt_4bit</name>
    <bits_per_sample>4</bits_per_sample>
    <sample_rate>3125000</sample_rate>
    <beams>2</beams>
    <version>0.100000001</version>
  </recorder_cfg>


  <splitter_cfg>
    <version>0.400000006</version>
    <data_type>encoded</data_type>
    <fft_len>256</fft_len>
    <ifft_len>1</ifft_len>
    <filter>polyphase</filter>
    <window>hanning</window>
    <samples_per_wu>1048576</samples_per_wu>
    <highpass>0</highpass>
    <blanker_filter>randomize</blanker_filter>
    <pfb_ntaps>16</pfb_ntaps>
    <pfb_width_factor>1.04999995</pfb_width_factor>
    <wu_bits_per_sample>4</wu_bits_per_sample>
  </splitter_cfg>


EDIT: Ah whoops, we already resolved this issue. :D
ID: 1997391 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 1997515 - Posted: 9 Jun 2019, 5:01:32 UTC - in response to Message 1997391.  

EDIT: Ah whoops, we already resolved this issue. :D

Yes we have resolved this discussion. Thank you for posting the header information from the work unit
ID: 1997515 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1997545 - Posted: 9 Jun 2019, 12:19:07 UTC

Is anyone else still experiencing those "random" stuck downloads?

Tom
A proud member of the OFA (Old Farts Association).
ID: 1997545 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1997546 - Posted: 9 Jun 2019, 12:46:16 UTC - in response to Message 1997545.  

Is anyone else still experiencing those "random" stuck downloads?

. . Yes occasionally ...

Stephen

:(
ID: 1997546 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13760
Credit: 208,696,464
RAC: 304
Australia
Message 1997606 - Posted: 9 Jun 2019, 22:11:19 UTC - in response to Message 1997545.  

Is anyone else still experiencing those "random" stuck downloads?

Not since editing my hosts file.
Grant
Darwin NT
ID: 1997606 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1997788 - Posted: 11 Jun 2019, 12:30:24 UTC

. . I am finding downloads very, very slow. I am guessing it is due to preparatory work for the DNS alterations.

Stephen

? ?
ID: 1997788 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1997808 - Posted: 11 Jun 2019, 18:38:02 UTC

and we are back...
ID: 1997808 · Report as offensive
Sirius B Project Donor
Volunteer tester
Avatar

Send message
Joined: 26 Dec 00
Posts: 24882
Credit: 3,081,182
RAC: 7
Ireland
Message 1997816 - Posted: 11 Jun 2019, 19:26:05 UTC
Last modified: 11 Jun 2019, 19:32:13 UTC

This is weird. Is the task d/b stuck in limbo? Normally have enough tasks to see out the outage but found that I was crunching the last 3 beta & 4 main. Just downloaded the max on beta & got 78 & 18 on main, no trouble downloading. Yet, all tasks page still only showing 4 in progress instead of 100.
Edit: Just checked on Beta, stating 10 in progress (can understand why - 7 were waiting to report in after the outage).
ID: 1997816 · Report as offensive
Profile Mr. Kevvy Crowdfunding Project Donor*Special Project $250 donor
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 15 May 99
Posts: 3776
Credit: 1,114,826,392
RAC: 3,319
Canada
Message 1997817 - Posted: 11 Jun 2019, 19:32:47 UTC - in response to Message 1997816.  

Excuse the obvious, but: Did you click it to "Show Active Tasks" rather than "Show All Tasks"?
ID: 1997817 · Report as offensive
Sirius B Project Donor
Volunteer tester
Avatar

Send message
Joined: 26 Dec 00
Posts: 24882
Credit: 3,081,182
RAC: 7
Ireland
Message 1997818 - Posted: 11 Jun 2019, 19:36:01 UTC - in response to Message 1997817.  
Last modified: 11 Jun 2019, 19:40:04 UTC

All tasks.
Edit:
Status all 35 - in progress 4 - pending 13 - valid 18 (should now be 131 all)
ID: 1997818 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1997819 - Posted: 11 Jun 2019, 19:49:17 UTC - in response to Message 1997816.  

Check the server stats page.

Replica is 2273 seconds behind the master. That means your stats are 37 minutes out of date. Wait until that number drops to 0
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1997819 · Report as offensive
Sirius B Project Donor
Volunteer tester
Avatar

Send message
Joined: 26 Dec 00
Posts: 24882
Credit: 3,081,182
RAC: 7
Ireland
Message 1997820 - Posted: 11 Jun 2019, 20:01:22 UTC - in response to Message 1997819.  
Last modified: 11 Jun 2019, 20:07:01 UTC

Plausible I suppose. However, in the past, can recall when replica been behind yet the status all updated pretty quickly to show total in progress.
11/06/2019 20:19:38 | SETI@home | Scheduler request completed: got 18 new tasks
That 37 min window is running late. :-)Edit: Make that 47 min behind.
All present & correct. :-)
ID: 1997820 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1997822 - Posted: 11 Jun 2019, 20:07:44 UTC

Not a graceful recovery after an outage. The server is getting pounded by over 2000 db queries/second, and the replica is now 2923 seconds behind.

The creation rate seems a bit low to keep the ready to send queue full, so we might hit no WUs to send for a short time, but the system will recover. I suggest if you have enough WUs at the moment to keep your systems busy, that it might be nice to just suspend network activity and give the seti system a chance to recover and fill up the needier clients first.
ID: 1997822 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1855
Credit: 268,616,081
RAC: 1,349
United States
Message 1997828 - Posted: 11 Jun 2019, 20:45:08 UTC - in response to Message 1997816.  
Last modified: 11 Jun 2019, 20:47:55 UTC

This is weird. Is the task d/b stuck in limbo? Normally have enough tasks to see out the outage but found that I was crunching the last 3 beta & 4 main. Just downloaded the max on beta & got 78 & 18 on main, no trouble downloading. Yet, all tasks page still only showing 4 in progress instead of 100.
Edit: Just checked on Beta, stating 10 in progress (can understand why - 7 were waiting to report in after the outage).

If you look at the SSP, you'll see a place where it shows how far behind the master database the replica database is. Normally not, but after an outage, it has to catch up. Until it does, all data like you're referring to will be off, as those stats are pulled off the replica to reduce load on the master. Normal ops.

[edit]
Sorry, already answered.
So, in the interests of making this message useful:
Nice recovery after the outage. All caches are now full.
[/edit]
ID: 1997828 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13760
Credit: 208,696,464
RAC: 304
Australia
Message 1997995 - Posted: 13 Jun 2019, 4:37:08 UTC

I've been wondering if the splitter output has been tweaked recently?
With the initial return of Arecibo work, I probably had a 60-70% of my cache being Arecibo. But I've noticed over the last few days that Arecibo WUs are now probably 15% or less of my cache.
Grant
Darwin NT
ID: 1997995 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1998000 - Posted: 13 Jun 2019, 6:34:39 UTC - in response to Message 1997995.  

I've been wondering if the splitter output has been tweaked recently?
With the initial return of Arecibo work, I probably had a 60-70% of my cache being Arecibo. But I've noticed over the last few days that Arecibo WUs are now probably 15% or less of my cache.


. . Yes, I have noticed that the GBT splitters seem to be getting more priority now and GBT work is dominating even though there is still Arecibo work being split, which is quite the opposite of how it seemed to be prior to this outage. Perhaps it is a by product of the changes they made to the server locations in terms of routing, maybe now the GBT splitters have more access to the RTS queue.

Stephen

? ?
ID: 1998000 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 1998004 - Posted: 13 Jun 2019, 9:04:18 UTC - in response to Message 1998000.  

I've been wondering if the splitter output has been tweaked recently?
With the initial return of Arecibo work, I probably had a 60-70% of my cache being Arecibo. But I've noticed over the last few days that Arecibo WUs are now probably 15% or less of my cache.


. . Yes, I have noticed that the GBT splitters seem to be getting more priority now and GBT work is dominating even though there is still Arecibo work being split, which is quite the opposite of how it seemed to be prior to this outage. Perhaps it is a by product of the changes they made to the server locations in terms of routing, maybe now the GBT splitters have more access to the RTS queue.

Stephen

? ?

Maybe they are also restricting multibeam work because I noticed if reasonable number were noisy. When I have returned my 73 units sometime tomorrow I will be at least a week break from here. I will certainly be back
ID: 1998004 · Report as offensive
Previous · 1 . . . 30 · 31 · 32 · 33 · 34 · 35 · 36 . . . 47 · Next

Message boards : Number crunching : Panic Mode On (116) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.