Panic Mode On (116) Server Problems?

Message boards : Number crunching : Panic Mode On (116) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 38 · 39 · 40 · 41 · 42 · 43 · 44 . . . 47 · Next

AuthorMessage
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1999873 - Posted: 27 Jun 2019, 1:49:53 UTC - in response to Message 1999868.  

Well I've suspended GPU processing on my 2 rigs until I can get a bit of a buffer downloaded, but it's not fun downloading all this work when most of it is just garbage. :-(
Well all that crap that I downloaded has exceeded my peak internet allowance so I'm on a slow boat now until the end of the month during the day. :-(

Cheers.


. . Well if the blc41/42 tasks prove to be long and slow it may balance out a bit. Here's hoping :)

Stephen

:)
ID: 1999873 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1999874 - Posted: 27 Jun 2019, 1:52:17 UTC - in response to Message 1999873.  

Results returned per hour just hit 217K.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1999874 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1999878 - Posted: 27 Jun 2019, 1:57:00 UTC - in response to Message 1999874.  
Last modified: 27 Jun 2019, 1:57:55 UTC

Results returned per hour just hit 217K.


. . That would be all the unreported noise bombs finally getting uploaded (particularly with Windows hosts). It should drop now with the blc41/42 tasks.

Stephen

:(
ID: 1999878 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1999881 - Posted: 27 Jun 2019, 2:23:20 UTC - in response to Message 1999878.  

Results returned per hour just hit 217K.


. . That would be all the unreported noise bombs finally getting uploaded (particularly with Windows hosts). It should drop now with the blc41/42 tasks.

Stephen

:(

That would be me finally getting all my 27,700 BLC24 Linux tasks finally uploaded.

The BLC42 tasks aren't going to be any help. Processing them in 30 minutes on the cpu and 45 seconds on the gpus. AR=0.55
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1999881 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13727
Credit: 208,696,464
RAC: 304
Australia
Message 1999883 - Posted: 27 Jun 2019, 3:35:16 UTC - in response to Message 1999870.  
Last modified: 27 Jun 2019, 3:36:28 UTC

And now the splitters have fallen on their face with the downloads finally working. RTS basically nil.

And it appears the splitters are continuing to have issues. After hitting 50/s for a while, they then fell over & were just hitting 20/s, when they hadn't stopped completely.
Ready_to_send was rapidly heading for 0 again till their latest burst over 40/s.
WU awaiting deletion are accumulating.

And things were going so well even with the huge server load, till everything came crashing down once again.



And 18dc09aa still sits there, taunting us.
Grant
Darwin NT
ID: 1999883 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13727
Credit: 208,696,464
RAC: 304
Australia
Message 1999884 - Posted: 27 Jun 2019, 4:17:32 UTC - in response to Message 1999873.  

. . Well if the blc41/42 tasks prove to be long and slow it may balance out a bit. Here's hoping :)

A rather mixed bag. I've had 2 that took 3 times as long to crunch, and another 2 that got done in almost half the usual time.
Grant
Darwin NT
ID: 1999884 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1999885 - Posted: 27 Jun 2019, 4:51:05 UTC
Last modified: 27 Jun 2019, 4:57:34 UTC

Just doing my "first of the day checks" all uploads uploaded and all 6 machines with full caches even the Win ones ;-)

Looking at the ready to report, out of the 18 that have passed through while I watched, only one that took 11 secs, the rest seem to be nearer normal times, hoping it stays that way.

Nope I was wrong, just had 22 "noise bombs" but they were all on the slowest of the 2 Win machines so maybe it has just "caught up"
ID: 1999885 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1999886 - Posted: 27 Jun 2019, 4:57:48 UTC - in response to Message 1999884.  

. . Well if the blc41/42 tasks prove to be long and slow it may balance out a bit. Here's hoping :)
A rather mixed bag. I've had 2 that took 3 times as long to crunch, and another 2 that got done in almost half the usual time.
Some are non-vlar tasks, so I guess they are BLC shorty\ies.
ID: 1999886 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1999895 - Posted: 27 Jun 2019, 6:48:28 UTC - in response to Message 1999881.  

Results returned per hour just hit 217K.

. . That would be all the unreported noise bombs finally getting uploaded (particularly with Windows hosts). It should drop now with the blc41/42 tasks.
Stephen

That would be me finally getting all my 27,700 BLC24 Linux tasks finally uploaded.
The BLC42 tasks aren't going to be any help. Processing them in 30 minutes on the cpu and 45 seconds on the gpus. AR=0.55


. . See I said you were the guilty party :)

. . I have noticed the short run times on the Blc41/42 tasks. That was what I was getting first time around as well except now and then I would see the really slow ones at 3 x normal time. I am guessing those were the usual GBT VLAR types. I haven't gotten around to checking out the stderr details to see what the AR's actually were ( I was afraid I would discover they were noisy overflows). Well at least the rigs are back into productivity. That noise bomb storm really put a crimp in things.

Stephen

:(
ID: 1999895 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1999896 - Posted: 27 Jun 2019, 6:52:15 UTC - in response to Message 1999884.  

. . Well if the blc41/42 tasks prove to be long and slow it may balance out a bit. Here's hoping :)

A rather mixed bag. I've had 2 that took 3 times as long to crunch, and another 2 that got done in almost half the usual time.


. . Apparently Keith checked the results and they are shorties with ARs around 0.55. That would explain the consistency in run times I am seeing, usually noisy overflows are more erratic in times. So the super long ones are the actual VLAR tasks.

Stephen

< shrug >
ID: 1999896 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1999967 - Posted: 27 Jun 2019, 17:26:49 UTC
Last modified: 27 Jun 2019, 17:28:17 UTC

I am getting "aborted runtime limit" on blc41/blc42 tasks on what appear to be gpu tasks. They are running 41 odd minutes and then hitting this error.

They are also hanging in my task list.

Is this me or them?

Tom
A proud member of the OFA (Old Farts Association).
ID: 1999967 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1999970 - Posted: 27 Jun 2019, 17:43:32 UTC

So I went for a walk in my neighborhood a couple weeks ago and there was a computer out for trash, along with a destroyed Ikea desk. I peeked inside the case to see what all was there--I was really only after mobo+cpu+ram, all of which was still there. GPU and drives were gone. That's fine by me. Brought it home, used the new build of Hiren's win10 PE to see what I was working with.

Turns out it's a 4770K with factory watercooling (HP Envy 810 desktop). Neat!

The Sempron 3500 that it replaced was doing these MBs in about 5.5 hours and was single-core. 4770K is doing them in about 1 hour, and I've got it set for 4 at a time. So that's what.. 20-22x faster?

Well it's been a month now and that's enough time for things to settle out. I have a verdict.

Lunar orbit has been reached (click for full-size on imgur)


The old Sempron machine had been on a pretty stable RAC average of ~375. Sitting at ~7150 for the past few days now, so I think that's the new stable figure.

7150/375 = 19.066667. So my rough napkin math of ~5.5x faster * 4 at a time was pretty accurate.

I'm pretty pleased with this machine, especially since someone just....threw it out. The CPU alone is on ebay for $120-175.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1999970 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1999972 - Posted: 27 Jun 2019, 18:15:28 UTC - in response to Message 1999970.  
Last modified: 27 Jun 2019, 18:17:09 UTC

That's on par with my i5-4590 @3.3GHz which is at 7400 right now. https://stats.free-dc.org/charts/hostdaily.php?proj=sah&hostid=8457417
Your could definitely run more than 4 tasks on that machine though.

When you found that machine I mentioned that is a good overclock chip. I real quick look shows 4.7GHz on it :)
ID: 1999972 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1999973 - Posted: 27 Jun 2019, 18:15:38 UTC - in response to Message 1999967.  

I am getting "aborted runtime limit" on blc41/blc42 tasks on what appear to be gpu tasks. They are running 41 odd minutes and then hitting this error.

They are also hanging in my task list.

Is this me or them?

Tom

It would be easier if you at least listed the host you are referring. I would say it is you since I have not had any issues with any of the BLC41/42 tasks.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1999973 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1999974 - Posted: 27 Jun 2019, 18:18:39 UTC - in response to Message 1999973.  

Exceeded runtime limit is usually a result of excessive rescheduling in my experience.
ID: 1999974 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 65738
Credit: 55,293,173
RAC: 49
United States
Message 1999975 - Posted: 27 Jun 2019, 18:37:26 UTC

Now if only We can clear this Shorty Storm...
The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's
ID: 1999975 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1999983 - Posted: 27 Jun 2019, 20:58:35 UTC - in response to Message 1999973.  

I am getting "aborted runtime limit" on blc41/blc42 tasks on what appear to be gpu tasks. They are running 41 odd minutes and then hitting this error.

They are also hanging in my task list.

Is this me or them?

Tom

It would be easier if you at least listed the host you are referring. I would say it is you since I have not had any issues with any of the BLC41/42 tasks.


Later I discovered two gpu tasks in the taskmanager that were unkillable. Had to reboot to get rid of them.

It was this machine: https://setiathome.berkeley.edu/show_host_detail.php?hostid=8684146

Since I am running two new to me gpu's it is entirely likely it was me.

I re-jiggered the bios settings and had to re-install the All-in-One because it wouldn't start after the 2nd app crash.
So will go away and see what happens.

Tom
A proud member of the OFA (Old Farts Association).
ID: 1999983 · Report as offensive
Jeff Cobb Project Donor
Volunteer moderator
Project administrator
Project developer
Project scientist

Send message
Joined: 1 Mar 99
Posts: 122
Credit: 40,367
RAC: 0
United States
Message 1999991 - Posted: 27 Jun 2019, 21:45:19 UTC

Apologies for the upload issue yesterday. As many here properly guessed, this was fallout from the shortie / fast runner / noise bomb file set that was being split. I moved this file set out of the way but it took a few hours to work through the already split data.

We are hoping to replace the upload server (bruno) before too long with a machine that is both faster and will store the results on SSDs.
ID: 1999991 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1999993 - Posted: 27 Jun 2019, 21:47:21 UTC - in response to Message 1999991.  
Last modified: 27 Jun 2019, 21:49:56 UTC

Hi Jeff, good to see you around still. Thanks for the update and news of a possible new upload server.

[Edit] We ran across another noisy file set earlier this year. Any idea of why you are generating noisy files? Bad data recorder channel or something. Target location in the sky coinciding with a satellite constellation or something?
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1999993 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 2000000 - Posted: 27 Jun 2019, 23:06:01 UTC

Thanks for the info Jeff. Always nice to hear about what is going on and what is planned.
ID: 2000000 · Report as offensive
Previous · 1 . . . 38 · 39 · 40 · 41 · 42 · 43 · 44 . . . 47 · Next

Message boards : Number crunching : Panic Mode On (116) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.