Panic Mode On (116) Server Problems?

Message boards : Number crunching : Panic Mode On (116) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 21 · 22 · 23 · 24 · 25 · 26 · 27 . . . 47 · Next

AuthorMessage
Profile B. Ahmet KIRAN

Send message
Joined: 19 Oct 14
Posts: 77
Credit: 36,140,903
RAC: 140
Turkey
Message 1995240 - Posted: 25 May 2019, 18:23:34 UTC - in response to Message 1995225.  

Thanks for the relieving reply... My luck that I happened to be among the 0.001% users...
ID: 1995240 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1995243 - Posted: 25 May 2019, 19:36:24 UTC
Last modified: 25 May 2019, 19:39:12 UTC

Still plowing through some cached shorties here.
But I'm running some old iron due to my good rig blowing itself up due to a coolant leak.
So it's not a very quick rig.
An i7-965 and a pair of GTX 780 Classified gpus.
Was a pretty hot rig back when I built it.

But at least I am still online and crunching some.
I get 'no tasks' for a request or two and then I get enough to top this turtle back up again.

Meow.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1995243 · Report as offensive
Profile Siran d'Vel'nahr
Volunteer tester
Avatar

Send message
Joined: 23 May 99
Posts: 7379
Credit: 44,181,323
RAC: 238
United States
Message 1995248 - Posted: 25 May 2019, 20:03:06 UTC - in response to Message 1995243.  

Still plowing through some cached shorties here.
But I'm running some old iron due to my good rig blowing itself up due to a coolant leak.
So it's not a very quick rig.
An i7-965 and a pair of GTX 780 Classified gpus.
Was a pretty hot rig back when I built it.

But at least I am still online and crunching some.
I get 'no tasks' for a request or two and then I get enough to top this turtle back up again.

Meow.

Hi Mark,

Sorry to hear about you top rig. Bummer dude! :(

I never, ever thought I would see the day when my overall RAC is higher than yours. You ARE going to rebuild you rig, right? :)

Have a great day! :)

Siran
CAPT Siran d'Vel'nahr - L L & P _\\//
Winders 11 OS? "What a piece of junk!" - L. Skywalker
"Logic is the cement of our civilization with which we ascend from chaos using reason as our guide." - T'Plana-hath
ID: 1995248 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1995250 - Posted: 25 May 2019, 20:06:50 UTC - in response to Message 1995248.  
Last modified: 25 May 2019, 20:08:46 UTC

Still plowing through some cached shorties here.
But I'm running some old iron due to my good rig blowing itself up due to a coolant leak.
So it's not a very quick rig.
An i7-965 and a pair of GTX 780 Classified gpus.
Was a pretty hot rig back when I built it.

But at least I am still online and crunching some.
I get 'no tasks' for a request or two and then I get enough to top this turtle back up again.

Meow.

Hi Mark,

Sorry to hear about you top rig. Bummer dude! :(

I never, ever thought I would see the day when my overall RAC is higher than yours. You ARE going to rebuild you rig, right? :)

Have a great day! :)

Siran

I hope to.
I have a new PSU to replace the one that released the inner smoke.
But lord knows what it took out when it made it's exit.
I am hoping at least the CPU survived, but I am not real hopeful after the smoke and spark show.
The PSU might have taken the whole 9 yards out with it when it went out.

Meowsigh.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1995250 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1995251 - Posted: 25 May 2019, 20:38:37 UTC - in response to Message 1995239.  
Last modified: 25 May 2019, 21:04:44 UTC

Looks like they took a bunch of files out of the queue. Probably more of those contributing to the shorties.

Yep, they've pulled all the BLC25 files. Some of them were running normally, but most of them were just noise.
If they were to release a couple of those filet at a time, we could still work through them without pounding the servers in to the ground.

Processing times for the present WUs are about a minute less than for the previous ones on the GPU, 15min less on the CPU (that weren't noise bombs). That'll bump up the load on the servers, but it'll be nothing close to what the noise bombs were doing to them.,
Grant
Darwin NT
ID: 1995251 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1995253 - Posted: 25 May 2019, 21:21:09 UTC

Will be interesting to see if this surge in work results in a longer than usual weekly outage to clean up the database.
Grant
Darwin NT
ID: 1995253 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1995282 - Posted: 25 May 2019, 23:39:43 UTC - in response to Message 1995208.  

Ok, I have to ask. What is a "noise bomb" and how do you tell if you have been bombarded with them or not? ;)
Siran

. . A noise bomb is a WU that has such a high rate of signal 'noise' that it goes pop like a bomb and is gone. They are all those WUs that we are seeing recently that are over in just a few seconds, emptying your cache in no time flat without achieving anything significant.
Stephen

Hi Stephen,
So basically when an app starts working on a WU and it runs into a certain number of unwanted signals, it quits and moves on to the next WU. Correct? So that is why I see WUs done in a few seconds instead of a few minutes. I always wondered about that but was never curious enough to ask until I started seeing "noise bombs" dropping here in the fora. ;) Thanks! :)
Have a great day! :)
Siran


. . I should have elaborated on that as others did. There is a found signal limit of 30, because the expectation is that actual signals would be less frequent . So when a WU produces more than 30 found signals it is considered 'noisy' and abandoned. Often this results in a 'late' overflow error commonly called shorties, but when a signal is particularly noisy it will overflow almost immediately and be gone within seconds. These are what I call noise bombs. And that blc25 series from the date of nn340 was extremely noisy causing a very high level of bombs. Sadly when they pulled the remaining tapes from that date they also pulled the blc25 series from the date nn405/406 but that has been replaced with another series from that same date, this time a blc26 series which so far seems fairly clean.

Stephen

<shrug>
ID: 1995282 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1995283 - Posted: 25 May 2019, 23:44:31 UTC - in response to Message 1995243.  

Still plowing through some cached shorties here.
But I'm running some old iron due to my good rig blowing itself up due to a coolant leak.
So it's not a very quick rig.
Meow.


. . OUCH!! This is why I stay away from liquid cooling solutions. With my luck (or level of clumsiness) I would be the one suffering from such problems.

. . I hope the GPUs were recoverable?

Stephen

? ?
ID: 1995283 · Report as offensive
Profile Siran d'Vel'nahr
Volunteer tester
Avatar

Send message
Joined: 23 May 99
Posts: 7379
Credit: 44,181,323
RAC: 238
United States
Message 1995329 - Posted: 26 May 2019, 10:39:01 UTC - in response to Message 1995283.  

Still plowing through some cached shorties here.
But I'm running some old iron due to my good rig blowing itself up due to a coolant leak.
So it's not a very quick rig.
Meow.


. . OUCH!! This is why I stay away from liquid cooling solutions. With my luck (or level of clumsiness) I would be the one suffering from such problems.

. . I hope the GPUs were recoverable?

Stephen

? ?

Hi Stephen,

Actually, I would be more concerned about the custom water cooling solutions. All those glass or plastic tubes going every which way and all the connectors connecting them. To me those are disasters waiting to happen. ;)

I've been using AiO liquid coolers for years and have yet to see one fail. I currently have 3, 2 being used and one on standby, "just in case". ;) I clean the dust accumulation almost once a month, sometimes a little longer depending how I feel or how lazy I get, and I always try to check the AiO to make sure there are no leaks.

Have a great day! :)

Siran
CAPT Siran d'Vel'nahr - L L & P _\\//
Winders 11 OS? "What a piece of junk!" - L. Skywalker
"Logic is the cement of our civilization with which we ascend from chaos using reason as our guide." - T'Plana-hath
ID: 1995329 · Report as offensive
Profile Siran d'Vel'nahr
Volunteer tester
Avatar

Send message
Joined: 23 May 99
Posts: 7379
Credit: 44,181,323
RAC: 238
United States
Message 1995330 - Posted: 26 May 2019, 10:46:22 UTC - in response to Message 1995250.  

Hi Mark,

The PSU might have taken the whole 9 yards out with it when it went out.

The worst I have ever had done was a capacitor explode in a PSU killing it dead. I still have the aluminum cover that was shot so hard it has scrapes and dents on it. I was sleeping when it happened and it woke me up, that's how loud it was. :D

Have a great day! :)

Siran
CAPT Siran d'Vel'nahr - L L & P _\\//
Winders 11 OS? "What a piece of junk!" - L. Skywalker
"Logic is the cement of our civilization with which we ascend from chaos using reason as our guide." - T'Plana-hath
ID: 1995330 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1995356 - Posted: 26 May 2019, 16:58:28 UTC - in response to Message 1995329.  

Actually, I would be more concerned about the custom water cooling solutions. All those glass or plastic tubes going every which way and all the connectors connecting them. To me those are disasters waiting to happen. ;)

I've been using AiO liquid coolers for years and have yet to see one fail. I currently have 3, 2 being used and one on standby, "just in case". ;) I clean the dust accumulation almost once a month, sometimes a little longer depending how I feel or how lazy I get, and I always try to check the AiO to make sure there are no leaks.

I have similar experience with AIO's. The only thing that can happen to them is either the pump dies or the liquid in the loop evaporates through the tubing and cooling is lost or an air bubble locks up the pump.
As long as you use good quality parts in a custom loop and assemble it correctly, then they are just as reliable, more so because the pump is of much higher quality than the cheap AIO and never fails.
The only leak I ever had in my custom loop is some drips from a 90° rotary joint that was poorly made from a certain brand. I caught that as I was leak testing before I powered up. I substituted a better quality different brand and have had no issues with the rotary joints.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1995356 · Report as offensive
Profile Siran d'Vel'nahr
Volunteer tester
Avatar

Send message
Joined: 23 May 99
Posts: 7379
Credit: 44,181,323
RAC: 238
United States
Message 1995361 - Posted: 26 May 2019, 17:28:36 UTC - in response to Message 1995356.  

Actually, I would be more concerned about the custom water cooling solutions. All those glass or plastic tubes going every which way and all the connectors connecting them. To me those are disasters waiting to happen. ;)

I've been using AiO liquid coolers for years and have yet to see one fail. I currently have 3, 2 being used and one on standby, "just in case". ;) I clean the dust accumulation almost once a month, sometimes a little longer depending how I feel or how lazy I get, and I always try to check the AiO to make sure there are no leaks.

I have similar experience with AIO's. The only thing that can happen to them is either the pump dies or the liquid in the loop evaporates through the tubing and cooling is lost or an air bubble locks up the pump.
As long as you use good quality parts in a custom loop and assemble it correctly, then they are just as reliable, more so because the pump is of much higher quality than the cheap AIO and never fails.
The only leak I ever had in my custom loop is some drips from a 90° rotary joint that was poorly made from a certain brand. I caught that as I was leak testing before I powered up. I substituted a better quality different brand and have had no issues with the rotary joints.

Hi Keith,

I've watched quite a few YouTube videos on custom water coolers and some had to do with failed pumps. As with any mechanical / electrical device, the possibility of failure is there.

From what I get from those videos, the best components come from Germany and are very expensive.

I don't trust myself to do a custom water cooling solution. For one thing, my age. For another, my right arm. Being a rightie, it kinda sucks when my arm starts to shake uncontrollably for some reason. It hinders things like mousing in a game or any mousing. It's hard to keep the cursor on something so it can be clicked. I should be getting a call from the VA in the next couple weeks for my yearly checkup and will talk to the doctor about it then.

Yeah, being on a fixed income doesn't allow for extravagant things like custom cooling. Besides, I gotta get me a new laptop, my old 13 year old just doesn't cut the mustard anymore. :)

Have a great day! :)

Siran
CAPT Siran d'Vel'nahr - L L & P _\\//
Winders 11 OS? "What a piece of junk!" - L. Skywalker
"Logic is the cement of our civilization with which we ascend from chaos using reason as our guide." - T'Plana-hath
ID: 1995361 · Report as offensive
Wild6-NJ
Volunteer tester

Send message
Joined: 4 Aug 99
Posts: 43
Credit: 100,336,791
RAC: 140
Message 1995641 - Posted: 28 May 2019, 18:25:03 UTC - in response to Message 1995251.  

And on this week's return from maintenance, it looks like they put some BLC25 files back in the queue. Let's hope there aren't too many 'noise bombs' in the mix.
ID: 1995641 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1995648 - Posted: 28 May 2019, 18:51:19 UTC - in response to Message 1995641.  

And on this week's return from maintenance, it looks like they put some BLC25 files back in the queue. Let's hope there aren't too many 'noise bombs' in the mix.


very interesting. I know the blc25...108 that I was working on before they pulled them were good WUs, so hopefully we are out of the "shorty storm" or "noise bomb" files.
ID: 1995648 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1995960 - Posted: 30 May 2019, 17:22:51 UTC

No Panic ! They are throwing in blc25 files a few at a time. I'm guessing they are doing this to prevent a shorty storm. I just find this interesting and thought I would mention it. I'm not sure if we will get another batch of files before the weekend, or if they will let us chew through the files that have been there for a while.
ID: 1995960 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22200
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1995970 - Posted: 30 May 2019, 17:53:39 UTC

They do not to cherry pick the data that is sent out, rather just send out what is available.

Does anyone know if there is a direct data link to the lab from Green Bank, or are they still using "SneakerNet"?

The BLCxx part of the file name identifies the data as coming from the Green Bank telescope's receiver channel xx. What is more interesting is the second part of the name which identifies where in the sky the data was collected from.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1995970 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1995981 - Posted: 30 May 2019, 18:17:23 UTC

They had added the blc25s to the list to be split. The data got very noisy and they ended up pulling the rest of the blc25s from the list of files to be split. They are now adding a few of the blc25 files back into the list to be split in small batches. They probably have a TON of Green Bank data, so I do think that they cherry pick which order to put the files on the split pile.

How we get the files I don't know. We have been working on "groundhog day" for so long now I don't know what they will give us next.
ID: 1995981 · Report as offensive
Profile Mr. Kevvy Crowdfunding Project Donor*Special Project $250 donor
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 15 May 99
Posts: 3776
Credit: 1,114,826,392
RAC: 3,319
Canada
Message 1995984 - Posted: 30 May 2019, 18:27:46 UTC - in response to Message 1995970.  
Last modified: 30 May 2019, 18:30:26 UTC

They do not to cherry pick the data that is sent out, rather just send out what is available.


A "shortie storm" drives the return rate up so high that there is a real risk of crashing oscar and/or carolyn. (Edit: also possibly paddym?)
ID: 1995984 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1995987 - Posted: 30 May 2019, 19:06:29 UTC - in response to Message 1995970.  
Last modified: 30 May 2019, 19:10:30 UTC

They do not to cherry pick the data that is sent out, rather just send out what is available.

Does anyone know if there is a direct data link to the lab from Green Bank, or are they still using "SneakerNet"?
From comments that Eric K has made in recent years, I think that the Arecibo data feed is now pretty much fully automated - there is a functional, but throttled, internet link from the island for the raw data. I think it appears on the SSP whenever the data is complete and pre-processing has finished.

Eric also commented a year or two ago about an unexplained slow-down on the - was it described as "Internet 2"? - fibre backbone link from Green Bank to the Bay area. So I'm pretty sure it's a transmission link, not sneakernet, but I think there's still manual batch control, and the removal of the noisy files last week seems to confirm that.

Edit: on the second point, there's a whole News thread on that: Slow network connection to Green Bank
ID: 1995987 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1995989 - Posted: 30 May 2019, 19:14:46 UTC - in response to Message 1995970.  

I thought there was a post that both Observatories transfer their data files over a network connection. The pipe to Green Bank was bigger than Arecibo.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1995989 · Report as offensive
Previous · 1 . . . 21 · 22 · 23 · 24 · 25 · 26 · 27 . . . 47 · Next

Message boards : Number crunching : Panic Mode On (116) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.