Panic Mode On (94) Server Problems?

Message boards : Number crunching : Panic Mode On (94) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 . . . 22 · Next

AuthorMessage
Darth Beaver Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Avatar

Send message
Joined: 20 Aug 99
Posts: 6728
Credit: 21,443,075
RAC: 3
Australia
Message 1624853 - Posted: 7 Jan 2015, 12:55:35 UTC - in response to Message 1624847.  

JohnDK says

Seems we're back to problems getting work. Before Tuesday's outage I had max GPU cache for the first time in about 2 months, now I'm down to half GPUs and it's continues going down.



Roger Roger ..."Houston we seem to be experiencing a slight problem , could you guys take a look please from your end "......"Houston ??..........you there ?"
ID: 1624853 · Report as offensive
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1624869 - Posted: 7 Jan 2015, 13:30:42 UTC

Getting "no tasks available" again.

This truly sucks.

I'm going to be unPC for a moment: Why the Hell can't they fix this? Are they incompetent? They didn't used to be. Do they no longer care? SETI used to have seemingly much worse problems in the not-too-distant past that were fixed.

The colo facility seems to be disintegrating, at least from what I see; this is the worst since the move there, and it's NOT improving.

Is it unfair of me to complain about a seeming lack of effort by staff? If that isn't it, why can't we be given some solid info as to what (they think) the real problem(s) is (are)? I understand they in fact owe us poor shlubs doing the crunching nothing in the final analysis, but common human decency should require them to be more forthcoming, especially with unpaid volunteers who do the work that the project needs done.

/rant (for now)
ID: 1624869 · Report as offensive
Mark Stevenson Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 8 Sep 11
Posts: 1736
Credit: 174,899,165
RAC: 91
United Kingdom
Message 1624875 - Posted: 7 Jan 2015, 14:01:22 UTC - in response to Message 1624869.  

Getting "no tasks available" again.

This truly sucks.

I'm going to be unPC for a moment: Why the Hell can't they fix this? Are they incompetent? They didn't used to be. Do they no longer care? SETI used to have seemingly much worse problems in the not-too-distant past that were fixed.

The colo facility seems to be disintegrating, at least from what I see; this is the worst since the move there, and it's NOT improving.

Is it unfair of me to complain about a seeming lack of effort by staff? If that isn't it, why can't we be given some solid info as to what (they think) the real problem(s) is (are)? I understand they in fact owe us poor shlubs doing the crunching nothing in the final analysis, but common human decency should require them to be more forthcoming, especially with unpaid volunteers who do the work that the project needs done.

/rant (for now)



Id take a chill pill and relax , sure theyl get things going again properly soon as they can
ID: 1624875 · Report as offensive
Profile ReiAyanami
Avatar

Send message
Joined: 6 Dec 05
Posts: 116
Credit: 222,900,202
RAC: 174
Japan
Message 1624878 - Posted: 7 Jan 2015, 14:05:14 UTC

I was thinking this was part of a stress test for the new data base.

I think it is a stress test for us, too :)
ID: 1624878 · Report as offensive
Profile JaundicedEye
Avatar

Send message
Joined: 14 Mar 12
Posts: 5375
Credit: 30,870,693
RAC: 1
United States
Message 1624884 - Posted: 7 Jan 2015, 14:30:40 UTC - in response to Message 1624869.  

Getting "no tasks available" again.

This truly sucks.

I'm going to be unPC for a moment: Why the Hell can't they fix this? Are they incompetent? They didn't used to be. Do they no longer care? SETI used to have seemingly much worse problems in the not-too-distant past that were fixed.

The colo facility seems to be disintegrating, at least from what I see; this is the worst since the move there, and it's NOT improving.

Is it unfair of me to complain about a seeming lack of effort by staff? If that isn't it, why can't we be given some solid info as to what (they think) the real problem(s) is (are)? I understand they in fact owe us poor shlubs doing the crunching nothing in the final analysis, but common human decency should require them to be more forthcoming, especially with unpaid volunteers who do the work that the project needs done.

/rant (for now)


Perhaps if more 'setizens' would 'aquire' a 'green star' the project could afford better equipment and more staff. (Perhaps a 'Public Information Officer' to appease the whining masses with fluff talk..........works for Obama!).

'It doesn't matter what you do, it's what the people THINK you do'.

"Sour Grapes make a bitter Whine." <(0)>
ID: 1624884 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1624885 - Posted: 7 Jan 2015, 14:33:14 UTC - in response to Message 1624869.  

Getting "no tasks available" again.

This truly sucks.

I'm going to be unPC for a moment: Why the Hell can't they fix this? Are they incompetent? They didn't used to be. Do they no longer care? SETI used to have seemingly much worse problems in the not-too-distant past that were fixed.

The colo facility seems to be disintegrating, at least from what I see; this is the worst since the move there, and it's NOT improving.

Is it unfair of me to complain about a seeming lack of effort by staff? If that isn't it, why can't we be given some solid info as to what (they think) the real problem(s) is (are)? I understand they in fact owe us poor shlubs doing the crunching nothing in the final analysis, but common human decency should require them to be more forthcoming, especially with unpaid volunteers who do the work that the project needs done.

/rant (for now)

The "staff" are basically unpaid volunteers as well if you will remember.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1624885 · Report as offensive
Dena Wiltsie
Volunteer tester

Send message
Joined: 19 Apr 01
Posts: 1628
Credit: 24,230,968
RAC: 26
United States
Message 1624886 - Posted: 7 Jan 2015, 14:39:13 UTC

I don't know how they wrote the software, but our product has up to 70 processors running hundreds of task with many talking to each other. The people who wrote the original software ended up with embrace conditions where two task were waiting on resources held by the other task. The required resource wouldn't be released until the resource the task was waiting for was delivered. The only way to resolve this problem was to restructure the tasks to be flow through and not wait on resources or they still needed the resource reserved, the task could process other data while waiting for the resource. I solved several log jams and once the process was understood, others were able to solve some of the others.

Multi task real time software takes a really twisted mind to write because you have to be able to see the whole picture. I suspect the idea of writing many small modules leads people to think the problem is confined to only one module so they don't take the whole picture into account. Some of our modules were over 5000 lines of assembler spaghetti code but they don't embrace.
ID: 1624886 · Report as offensive
JohnDK Crowdfunding Project Donor*Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 28 May 00
Posts: 1222
Credit: 451,243,443
RAC: 1,127
Denmark
Message 1624893 - Posted: 7 Jan 2015, 14:53:05 UTC
Last modified: 7 Jan 2015, 14:53:55 UTC

More green stars would help the project overall, but don't see how that could explain why before Tuesday's outage there was full speed on downloads and after the outage it crawls slooowly.
ID: 1624893 · Report as offensive
Profile JaundicedEye
Avatar

Send message
Joined: 14 Mar 12
Posts: 5375
Credit: 30,870,693
RAC: 1
United States
Message 1624900 - Posted: 7 Jan 2015, 15:02:12 UTC - in response to Message 1624893.  

More green stars would help the project overall, but don't see how that could explain why before Tuesday's outage there was full speed on downloads and after the outage it crawls slooowly.

'(..)'............GREMLINS!

"Sour Grapes make a bitter Whine." <(0)>
ID: 1624900 · Report as offensive
Dena Wiltsie
Volunteer tester

Send message
Joined: 19 Apr 01
Posts: 1628
Credit: 24,230,968
RAC: 26
United States
Message 1624904 - Posted: 7 Jan 2015, 15:09:27 UTC

Green stars and badges go together. I have both but nether shows. I do agree that the amount of funds contributed are very small compared to the number of crunchers.
ID: 1624904 · Report as offensive
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1624908 - Posted: 7 Jan 2015, 15:22:41 UTC - in response to Message 1624885.  

The "staff" are basically unpaid volunteers as well if you will remember.


And recently (unlike before) worth every penny, too.

I call BS on that argument. If I volunteer to help someone do something, I don't leave them hanging in the middle of it. At the very least, I apologize if I have to leave or can't do the job for whatever reason. I don't just clam up and pretend I am still doing it.
ID: 1624908 · Report as offensive
Profile JaundicedEye
Avatar

Send message
Joined: 14 Mar 12
Posts: 5375
Credit: 30,870,693
RAC: 1
United States
Message 1624921 - Posted: 7 Jan 2015, 15:56:06 UTC - in response to Message 1624908.  

The "staff" are basically unpaid volunteers as well if you will remember.


And recently (unlike before) worth every penny, too.

I call BS on that argument. If I volunteer to help someone do something, I don't leave them hanging in the middle of it. At the very least, I apologize if I have to leave or can't do the job for whatever reason. I don't just clam up and pretend I am still doing it.

Yeah, you're right, those who run the project don't care a lick about their life work. They are just after the fat paycheck and constant accolades and awards. Let's ALL just turn off the crunchers and abandon everything..........there really needs to be a sarcasm font.

:{[

"Sour Grapes make a bitter Whine." <(0)>
ID: 1624921 · Report as offensive
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1624932 - Posted: 7 Jan 2015, 16:19:16 UTC - in response to Message 1624921.  

The "staff" are basically unpaid volunteers as well if you will remember.


And recently (unlike before) worth every penny, too.

I call BS on that argument. If I volunteer to help someone do something, I don't leave them hanging in the middle of it. At the very least, I apologize if I have to leave or can't do the job for whatever reason. I don't just clam up and pretend I am still doing it.

Yeah, you're right, those who run the project don't care a lick about their life work. They are just after the fat paycheck and constant accolades and awards. Let's ALL just turn off the crunchers and abandon everything..........there really needs to be a sarcasm font.

:{[


You know, I think you're on to something. Maybe they were out racing their Stingrays around Deadman's Curve, and hit the wall (cf. Jan and Dean).
ID: 1624932 · Report as offensive
JohnDK Crowdfunding Project Donor*Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 28 May 00
Posts: 1222
Credit: 451,243,443
RAC: 1,127
Denmark
Message 1624952 - Posted: 7 Jan 2015, 17:19:31 UTC

No new work the last 4 hours on 3 hosts, craposoid or something.
ID: 1624952 · Report as offensive
Profile JaundicedEye
Avatar

Send message
Joined: 14 Mar 12
Posts: 5375
Credit: 30,870,693
RAC: 1
United States
Message 1624964 - Posted: 7 Jan 2015, 17:33:52 UTC

Space Weather News for Jan. 7, 2015
http://spaceweather.com
GEOMAGNETIC STORM: A surprise geomagnetic storm erupted during the early hours of January 7th, sparking bright auroras around both of Earth's poles. The storm may have been sparked by the arrival of a CME originally expected to miss our planet.

..........Well, there you have it, there is an explanation.

"Sour Grapes make a bitter Whine." <(0)>
ID: 1624964 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1624968 - Posted: 7 Jan 2015, 17:38:27 UTC - in response to Message 1624962.  

Seems the APs hit their peak on Monday. My one machine that had been limited to around 20 GPU APs previously hit 170 APs on Monday, then started falling off. By late last night it was down to 60. Now it's on it's last 20. It will be out in a few hours.
It was doing so well early Monday...
ID: 1624968 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1624970 - Posted: 7 Jan 2015, 17:40:29 UTC - in response to Message 1624968.  

I'm all out of my APs...Can't even get MBs just keep the GPUs running. Had to resort to Beta just to heat the room a little...
ID: 1624970 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 1625057 - Posted: 7 Jan 2015, 19:41:43 UTC - in response to Message 1624885.  


The "staff" are basically unpaid volunteers as well if you will remember.

I have no doubt that Eric, Jeff, and Matt volunteer time beyond what their salaries require, but I'll quote one paragraph from Eric's January 7, 2007 post 499228 about how donations relate:
SETI@home is entirely funded by donations. Most of these donations come from people who also run SETI@home. Universities don't really "back" anybody. Faculty at Univerisities and expected to bring in their own funding. If Dan and I don't bring in money, we don't get paid, but we still keep our jobs (without pay). Matt and Jeff get a pink slip. In exchange for a cut of that funding, the universities provide offices and keep the lights on.

The intervening 7 years has not increased the donations, I am concerned.
                                                                   Joe
ID: 1625057 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1625067 - Posted: 7 Jan 2015, 19:58:54 UTC

This may seem odd, but here goes:

Mark Sattler e-mailed all at the lab and got a reply, I will paste his post from the BIONC boards.

Some news from the kitties that should be relayed to Panic Mode in NC that might explain a few things for some folks.

Matt has explained to me that the AP database is not fully healed, and automated processes are working in the background to complete repairs. It may be some weeks yet before this has been completed.

So, that is the reason that the AP assimilators have been offline, and the AP waiting for assimilation queue has been growing. Since that work has to be stored temporarily until assimilation can commence, that storage space has filled up, and that is the primary reason that the AP splitters are working so slowly...nowhere to go with the work.

They are going to increase the disk space for storing AP work, and that should allow the AP splitters to get back up to speed again, hopefully soon.

ID: 1625067 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1625075 - Posted: 7 Jan 2015, 20:09:33 UTC - in response to Message 1625067.  

And as I replied to the original post:

Since MB work occupies the same fanout directories as AP work, that will affect MB splitting as well. But at least we can crack on with returning MB work for assimilation, and thus recycle the limited space available.
ID: 1625075 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 . . . 22 · Next

Message boards : Number crunching : Panic Mode On (94) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.