Panic Mode On (100) Server Problems?

Message boards : Number crunching : Panic Mode On (100) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 4 · 5 · 6 · 7 · 8 · 9 · 10 . . . 32 · Next

AuthorMessage
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1719040 - Posted: 27 Aug 2015, 11:47:35 UTC - in response to Message 1719039.  

For those who don't bother to read Matt's post in Tech News on 21 August:

Message 1716662 - Posted: 21 Aug 2015, 22:47:12 UTC

Those panicking about a coming storm due to lack of data... The well is pretty dry but Jeff and I just uncovered a stash of tapes from 2011 that require some re-analysis, so that's why you'll see a bunch showing up in splitter queue over the weekend (hopefully before the the results-to-send queue drops to zero).

In the meantime, we are still recording data at AO (not fast enough to keep our crunchers supplied), but.... this situation has really pushed us to devote more resources to finally finishing the GBT splitter, which will avail to us another reserve supply of data in case we hit another dry spell.

The network switch on Tuesday seems to have gone fairly well. We are now sending all our bits over the campus net just like the very old days .

- Matt


So, the tapes being split just now have already been split once, but are being culled for more usable data. More effort is being put into the splitter for the GBT data (coo, what a surprise - its "tapes" are not the same format as those from AO...) Having two data sources should really improve the situation somewhat.

Next challenge. We must burn through data faster than two sources can provide!
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1719040 · Report as offensive
Filipe

Send message
Joined: 12 Aug 00
Posts: 218
Credit: 21,281,677
RAC: 20
Portugal
Message 1719109 - Posted: 27 Aug 2015, 15:14:55 UTC

Next challenge. We must burn through data faster than two sources can provide!



+1
ID: 1719109 · Report as offensive
WezH
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 576
Credit: 67,033,957
RAC: 95
Finland
Message 1719114 - Posted: 27 Aug 2015, 15:34:47 UTC - in response to Message 1719027.  

I agree with Grant, splitters are just too slow at moment. Results received in last hour is above 100.000 all time, so Current result creation rate should be at least 28/sec to keep it even.

I just hope that they have time and manpower to sort it out...

If you read Matt's technical posts, all three of time, manpower, and Arecibo recording time are in short supply.

What time and manpower are available are being concentrated on working up the alternative tools, so we can search the data from other telescopes and hence other parts of the sky. Keeping every volunteer supplied with every WU they ask for isn't the highest of their priorities, and to be honest I don't think it should be.


Yes, I did read Matt's technical post, and I know they are concentrating to get GBT tasks on-line.

I just should write that I hoped and wished they have more manpower and time (and money) to solve issues like current MB splitters are too slow at this moment.
ID: 1719114 · Report as offensive
WezH
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 576
Credit: 67,033,957
RAC: 95
Finland
Message 1719124 - Posted: 27 Aug 2015, 16:02:42 UTC - in response to Message 1718977.  

I see the splitters are still not up to the task.


Now they are, somehow.

RTS is about 84K, and it has been raising since last 6 hours. I think that AP splitter running is helping MB splitters again.
ID: 1719124 · Report as offensive
Profile Oz
Avatar

Send message
Joined: 6 Jun 99
Posts: 233
Credit: 200,655,462
RAC: 212
United States
Message 1719159 - Posted: 27 Aug 2015, 17:29:51 UTC - in response to Message 1719124.  

I agree that that seems to be true. What I have noticed is that high MB production and having the replica running don't seem to coexist, one can either have one or the other but not both. At least it would appear that way to me, of late.
Member of the 20 Year Club



ID: 1719159 · Report as offensive
Profile Oz
Avatar

Send message
Joined: 6 Jun 99
Posts: 233
Credit: 200,655,462
RAC: 212
United States
Message 1719161 - Posted: 27 Aug 2015, 17:37:28 UTC - in response to Message 1718446.  

I got one! I got an AP! I haven't seen one in a long time.

Edit: I seem to have passed 2 Million credits recently. Yay!



Congratulations on your milestone! Keep on crunching.
Member of the 20 Year Club



ID: 1719161 · Report as offensive
Tom*

Send message
Joined: 12 Aug 11
Posts: 127
Credit: 20,769,223
RAC: 9
United States
Message 1719172 - Posted: 27 Aug 2015, 17:55:12 UTC

Splitters are as simple as 1,2,3 - When shorties are mainly being split
RTS goes way down, We are now back to mainly VLARs being split and RTS goes up.

1,2,3 = 1. Shorties go down and 2. VLARs go up 3. Normal AR's go sideways:-)


I was getting mainly shorties ie 30 minute AVX now I
have mostly VLAR's taking 90 minutes on AVX CPU

Shorties are really short on GPU's which contributes to lower RTS

while VLAR's only done on CPU's .

This is mainly for newbies and Grant:-)
ID: 1719172 · Report as offensive
Andrew Scharbarth
Volunteer tester

Send message
Joined: 29 May 07
Posts: 40
Credit: 5,984,436
RAC: 0
United States
Message 1719217 - Posted: 27 Aug 2015, 19:17:35 UTC

Since none of the processed data has been reviewed and just shoved into a database, how about we start chopping that up into work units for the crunchers to start comparing?
ID: 1719217 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 1719236 - Posted: 27 Aug 2015, 20:00:10 UTC

No Aps being split and replica DB offline = small panic
ID: 1719236 · Report as offensive
wiesel111

Send message
Joined: 5 Jan 08
Posts: 9
Credit: 1,227,675
RAC: 1
Germany
Message 1719255 - Posted: 27 Aug 2015, 20:28:16 UTC - in response to Message 1719236.  
Last modified: 27 Aug 2015, 20:30:36 UTC

replica DB is online again - replica seconds behind master 74,211
...that is about 20 hours.
ID: 1719255 · Report as offensive
WezH
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 576
Credit: 67,033,957
RAC: 95
Finland
Message 1719507 - Posted: 28 Aug 2015, 7:44:57 UTC - in response to Message 1719255.  

Well it took about 8 hours replica to catch up.
ID: 1719507 · Report as offensive
Profile JaundicedEye
Avatar

Send message
Joined: 14 Mar 12
Posts: 5375
Credit: 30,870,693
RAC: 1
United States
Message 1719636 - Posted: 28 Aug 2015, 15:19:09 UTC

Anyone else notice slowed download speeds since the switch to the campus system? Maybe it's just the WOW event activity but it looks to be around 50% slower recently.

"Sour Grapes make a bitter Whine." <(0)>
ID: 1719636 · Report as offensive
Profile Bill G Special Project $75 donor
Avatar

Send message
Joined: 1 Jun 01
Posts: 1282
Credit: 187,688,550
RAC: 182
United States
Message 1719686 - Posted: 28 Aug 2015, 17:22:12 UTC - in response to Message 1719636.  

Anyone else notice slowed download speeds since the switch to the campus system? Maybe it's just the WOW event activity but it looks to be around 50% slower recently.

Not really. I picked up a 132 WU download yesterday and they flew off the Transfers tab.

SETI@home classic workunits 4,019
SETI@home classic CPU time 34,348 hours
ID: 1719686 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1719698 - Posted: 28 Aug 2015, 18:03:22 UTC - in response to Message 1719636.  

Anyone else notice slowed download speeds since the switch to the campus system? Maybe it's just the WOW event activity but it looks to be around 50% slower recently.

I haven't noticed anything, but most of my machines list their Average download rate from 1300 KB/sec to 1800 KB/sec.
I would have to lookup how BOINC calculates that averages to see if I could reset it. Then see if the average is much less.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1719698 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13732
Credit: 208,696,464
RAC: 304
Australia
Message 1719787 - Posted: 28 Aug 2015, 22:36:19 UTC - in response to Message 1719698.  

And the splitters continue to struggle...
At least there have been bursts of AP work to help take the load off of MB.
Grant
Darwin NT
ID: 1719787 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1719824 - Posted: 29 Aug 2015, 0:01:22 UTC - in response to Message 1719636.  

Anyone else notice slowed download speeds since the switch to the campus system? Maybe it's just the WOW event activity but it looks to be around 50% slower recently.

I was finally able to watch an AP download happen. It used to start off about 400 KB/sec and often end up well over 1000 KB/sec on the last update interval of the UI.

Now it starts off about 200 KB/sec, but ends up about 900 KB/sec by the last update interval. Old HE link generally took 8-10 seconds for downloading one AP, and... the campus connection looks to take... 8-10 seconds as well.

So I don't think there's much of a change on that regard, but since it is the campus connection, if the campus itself is going through a high-congestion time, then perhaps our traffic might slow down a little bit. If they're letting us use their connection without paying for it, I would completely agree with having a low-priority when it comes to QoS kind of stuff.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1719824 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1719830 - Posted: 29 Aug 2015, 0:14:47 UTC - in response to Message 1719824.  

Latest AP versus older APs earlier in the year on my computer (with 200Mbit connection, but somewhere between me and Berk things slow down)

29/08/2015 01:04:55 | SETI@home | Started download of ap_26se11ae_B2_P1_00241_20150828_15663.wu
29/08/2015 01:05:24 | SETI@home | Finished download of ap_26se11ae_B2_P1_00241_20150828_15663.wu
29/08/2015 01:05:24 | SETI@home | [file_xfer] Throughput 297520 bytes/sec

17-Jan-2015 20:45:48 [SETI@home] Started download of ap_25oc11ad_B4_P1_00015_20150117_09113.wu
17-Jan-2015 20:46:12 [SETI@home] Finished download of ap_25oc11ad_B4_P1_00015_20150117_09113.wu
17-Jan-2015 20:46:12 [SETI@home] [file_xfer] Throughput 361300 bytes/sec

The following were two APs that downloaded together with 11 MBs simultaneously.
24-Jun-2015 00:33:37 [SETI@home] Started download of ap_01fe15aa_B0_P0_00221_20150623_13173.wu
24-Jun-2015 00:33:37 [SETI@home] Started download of ap_02fe15ab_B0_P0_00205_20150623_13249.wu
24-Jun-2015 00:34:10 [SETI@home] Finished download of ap_01fe15aa_B0_P0_00221_20150623_13173.wu
24-Jun-2015 00:34:10 [SETI@home] [file_xfer] Throughput 255558 bytes/sec
24-Jun-2015 00:34:05 [SETI@home] Finished download of ap_02fe15ab_B0_P0_00205_20150623_13249.wu
24-Jun-2015 00:34:05 [SETI@home] [file_xfer] Throughput 303754 bytes/sec

MBs normally download at speeds between low 90Ks to low 120Ks.
ID: 1719830 · Report as offensive
Rasputin42
Volunteer tester

Send message
Joined: 25 Jul 08
Posts: 412
Credit: 5,834,661
RAC: 0
United States
Message 1719834 - Posted: 29 Aug 2015, 0:33:35 UTC - in response to Message 1719830.  

Did you check the boinc settings where you can limit the download speed?
Maybe that got changed somehow?
ID: 1719834 · Report as offensive
Profile JaundicedEye
Avatar

Send message
Joined: 14 Mar 12
Posts: 5375
Credit: 30,870,693
RAC: 1
United States
Message 1719863 - Posted: 29 Aug 2015, 2:05:40 UTC

It has looked normal for the last several hours. Probably just a temporary traffic issue between me and the server.

"Sour Grapes make a bitter Whine." <(0)>
ID: 1719863 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1720028 - Posted: 29 Aug 2015, 16:41:03 UTC - in response to Message 1719834.  
Last modified: 29 Aug 2015, 16:41:37 UTC

Did you check the boinc settings where you can limit the download speed?
Maybe that got changed somehow?

Both download and upload are set at a max of 1024 kilobytes/second in the web preferences, not doing overriding through local preferences.(*)
But as shown throughout the year, with 300K max for APs and 120K max for MBs, I never get the speeds.

global_prefs.xml
<max_bytes_sec_down>1024000.0</max_bytes_sec_down>
<max_bytes_sec_up>1024000.0</max_bytes_sec_up>

global_prefs_override.xml
<max_bytes_sec_up>0.000000</max_bytes_sec_up>
<max_bytes_sec_down>0.000000</max_bytes_sec_down>

(*) Now wondering though, when I have set a value in web preferences, but no value in local preferences, do the local preferences then still override the web preferences (set to zero = no limitation), or do they follow the web preferences and only override when there's a value > 1 set? Hmmmm, pondering.

In any case, My true connection
ID: 1720028 · Report as offensive
Previous · 1 . . . 4 · 5 · 6 · 7 · 8 · 9 · 10 . . . 32 · Next

Message boards : Number crunching : Panic Mode On (100) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.