Fill 'er up (Jun 23 2010)


log in

Advanced search

Message boards : Technical News : Fill 'er up (Jun 23 2010)

1 · 2 · 3 · Next
Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 1007541 - Posted: 23 Jun 2010, 21:40:02 UTC

Since last I wrote a lot has happened. Looking at the traffic graphs it's like feast or famine - either we are unable to create/send out workunits, or we're sending out as many as we can fit through the pipe. Mostly it's been the usual gremlins.

However regarding the past 24 hours it was a new problem: the result space on the upload server filled up unexpectedly, which would have been fine except this (perhaps) inspired some RAID freakout on the system. We couldn't really sort it out until this morning. From the looks of it we had something like a six drive simultaneous failure. Jeff and I beat on it for a while - we eventually assumed this was just a hardware blip, and the data was more or less intact on the drives, but the RAID metadata got a little screwed up. Long story short we were able to carefully bring down the RAID and recreate the meta devices from scratch with the data intact, and all was well. Phew. For the record we do have a virtually-up-to-date result storage backup at all times in case of catastrophic failure on this system.

In any case, the main culprit was our disks filling up, so as I write this we're keeping the project down until major queues drain and the constituent workunit/result files can be deleted.

On a more happy (perhaps) note, yesterday the core group of us were in the same place at the same time (which is rare) and we had an ad hoc meeting about our current project status/plans, especially in light of many recent server problems, increasingly random schedules, and embarrassingly low funding. We're all kind of tired and beaten up and wanting some results already - so I like to think this paved the way for several large and ultimately positive changes in the future.

Also Jeff has been working on this nagging mysterious problem where some of our raw data files are only getting partially processed (which vastly increases our "burn rate" and leads to unexpected workunit shortages). He found some major clues today, and we brainstormed why this is happening and what the exact effect is. At least there's a smoking gun on that front.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

ClaggyProject donor
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4084
Credit: 32,977,652
RAC: 5,216
United Kingdom
Message 1007545 - Posted: 23 Jun 2010, 21:48:23 UTC - in response to Message 1007541.

Thanks for the update Matt,

Claggy

Profile Taxman59
Send message
Joined: 9 Feb 00
Posts: 13
Credit: 1,913,896
RAC: 5
United States
Message 1007546 - Posted: 23 Jun 2010, 21:48:42 UTC - in response to Message 1007541.

Thanks for the update. You guys do a "Yeoman" job. I am curious, what would it cost for a new server? It seems that if the people who seem to be venting the most are those who haven't given cash. That is not to say they haven't donated time and electrical costs, but the project could use more of the cash. Maybe a "fundraiser" for a new server or two would shake loose a few dollars that may make your lives a bit easier.

Again thanks for the update and the day to day work you do to keep the project going.

____________

Profile Gary CharpentierProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 12475
Credit: 6,791,759
RAC: 6,615
United States
Message 1007547 - Posted: 23 Jun 2010, 21:51:01 UTC - in response to Message 1007541.
Last modified: 23 Jun 2010, 21:57:23 UTC

Thanks for the excellent news item. Most of us are a patient lot and understand you are reduced to rock soup. Don't let the others get you down.
____________

Profile Blurf
Volunteer tester
Send message
Joined: 2 Sep 06
Posts: 7541
Credit: 6,790,498
RAC: 8,011
United States
Message 1007548 - Posted: 23 Jun 2010, 21:52:28 UTC

I know you guys have your hands full-sometimes those sudden meetings are the best to have occurred.
____________


Profile Byron Leigh Hatch @ team Carl SaganProject donor
Volunteer tester
Avatar
Send message
Joined: 5 Jul 99
Posts: 3617
Credit: 11,866,054
RAC: 1,097
Canada
Message 1007553 - Posted: 23 Jun 2010, 22:00:53 UTC - in response to Message 1007541.

Thank you Matt.
David A, Eric, Dan, Jeff, Josh Von Korff, Robert Bankay, Kevin Douglas.
You folks show outstanding dedication to SETI@home.
Best wishes and peace.
from Vancouver, Canada.
Byron.


____________

Profile Chris SProject donor
Volunteer tester
Avatar
Send message
Joined: 19 Nov 00
Posts: 31596
Credit: 12,492,752
RAC: 26,590
United Kingdom
Message 1007568 - Posted: 23 Jun 2010, 22:19:11 UTC

and we had an ad hoc meeting about our current project status/plans, especially in light of many recent server problems, increasingly random schedules, and embarrassingly low funding. We're all kind of tired and beaten up and wanting some results already - so I like to think this paved the way for several large and ultimately positive changes in the future.


We all await to hear further, it sounds positive.

He found some major clues today, and we brainstormed why this is happening and what the exact effect is. At least there's a smoking gun on that front.


Yeeharr!

AS always we appreciate your time to talk to us Matt. Thanks.

____________
Damsel Rescuer, Uli Devotee, Julie Supporter, Kitty sad,
ES99 Admirer, Raccoon Friend, Anniet fan, Hon Triumphvir


KB7RZF
Volunteer tester
Avatar
Send message
Joined: 15 Aug 99
Posts: 9463
Credit: 3,111,828
RAC: 696
United States
Message 1007590 - Posted: 23 Jun 2010, 22:46:10 UTC

Matt, thank you very much for the update, it really shed's a lot of light on the recent problems. Here's hoping you guys get it running smooth again, or as smooth as you can for a bit. LOL Have a great rest of the week!

Profile Codeman05
Send message
Joined: 16 Dec 01
Posts: 33
Credit: 15,457,430
RAC: 0
United States
Message 1007653 - Posted: 24 Jun 2010, 1:52:38 UTC

Thank you for the update Matt and for all the hard work you and the team have put in!

ront
Send message
Joined: 25 Aug 01
Posts: 77
Credit: 386,336
RAC: 0
United States
Message 1007658 - Posted: 24 Jun 2010, 2:29:24 UTC

Thanks for the update. I, also, would like to know the cost of the type of server you need.

Again, thanks for informing us so promptly of the issues involved.

Be Blessed & Be A Blessing,

ront
____________

Profile Todd Hebert
Volunteer tester
Avatar
Send message
Joined: 16 Jun 00
Posts: 647
Credit: 217,127,962
RAC: 0
United States
Message 1007666 - Posted: 24 Jun 2010, 2:54:12 UTC

Thanks Matt! I'm interested in donating to the cause for new hardware.
Please let us know.
Todd
____________

Profile MarkJProject donor
Volunteer tester
Avatar
Send message
Joined: 17 Feb 08
Posts: 937
Credit: 23,640,695
RAC: 82,493
Australia
Message 1007727 - Posted: 24 Jun 2010, 12:03:17 UTC
Last modified: 24 Jun 2010, 12:05:11 UTC

Hi Matt,

Thanks for the update and the great job you guys do to keep things running.

Since Blurf decided to reduce his involvement we don't seem to have anyone to coordinate hardware (or other) funding drives. Is is possible to get the list of required equipment updated so we have an idea of what is currently needed please.

Cheers
____________
BOINC blog

Profile S@NL - BlackBiker
Volunteer tester
Avatar
Send message
Joined: 14 Feb 00
Posts: 10
Credit: 5,912,242
RAC: 0
Netherlands
Message 1007732 - Posted: 24 Jun 2010, 13:26:24 UTC

Hi Guys,

On behalf of the SETI@Netherlands team I would like to thank you for all your hard work you are putting in.
No budget, no manpower and no luck. It's a bitch.
But please, don't get discouraged and keep up the good work.

When you've got the flow running again, we will again do our part in the search :)
____________
Treasurer of team SETI@Netherlands

Profile robin
Volunteer tester
Send message
Joined: 19 Nov 01
Posts: 1
Credit: 5,152,995
RAC: 2,055
United States
Message 1007772 - Posted: 24 Jun 2010, 15:43:09 UTC

Thanks for all your hard work & for keeping us updated. I was concerned when I ran out of work.


____________

Stephen Falken
Volunteer tester
Avatar
Send message
Joined: 23 Apr 06
Posts: 63
Credit: 5,159,519
RAC: 0
United States
Message 1007790 - Posted: 24 Jun 2010, 16:28:52 UTC

Hey folks. We need to do something about this funding issue. SETI has more crunchers than just about any other project. This is a huge resource, and it feels like the operation at Berkeley is slowly deteriorating. The guys there can only do so much to keep things running and eventually, critical hardware will fail that they cannot afford to replace.

Given the broad spectrum of people who contribute to this project, there have to be some profs who can help pursue grants to reinvigorate the project. Any ideas?

Profile perryjay
Volunteer tester
Avatar
Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 15,469,780
RAC: 11,283
United States
Message 1007809 - Posted: 24 Jun 2010, 17:28:15 UTC
Last modified: 24 Jun 2010, 17:29:58 UTC

We're on our way!!! Uploads and downloads are flowing again!!


Sorry, I was so excited to see things moving again I forgot to thank all the guys at SAH for getting us there!
____________


PROUD MEMBER OF Team Starfire World BOINC

Sten-Arne
Volunteer tester
Send message
Joined: 1 Nov 08
Posts: 3406
Credit: 19,877,305
RAC: 20,127
Sweden
Message 1007813 - Posted: 24 Jun 2010, 17:36:05 UTC

Shouldn't the thread title be "Fill 'er up (Jun 23 2010)" ?

It ain't July yet.


____________

Profile Scarecrow
Avatar
Send message
Joined: 15 Jul 00
Posts: 4383
Credit: 459,097
RAC: 14
United States
Message 1007816 - Posted: 24 Jun 2010, 17:45:30 UTC - in response to Message 1007813.

Shouldn't the thread title be "Fill 'er up (Jun 23 2010)" ?

It ain't July yet.


Obviously a severe bug in the poster's date conversion module. Roll 'im back.

zoom314Project donor
Avatar
Send message
Joined: 30 Nov 03
Posts: 46247
Credit: 36,663,150
RAC: 5,225
Message 1007831 - Posted: 24 Jun 2010, 18:41:15 UTC - in response to Message 1007809.
Last modified: 24 Jun 2010, 18:41:40 UTC

We're on our way!!! Uploads and downloads are flowing again!!


Sorry, I was so excited to see things moving again I forgot to thank all the guys at SAH for getting us there!

Only If Your Quota isn't a quota of 904,250,952 tasks... And so I can't download either cpu or gpu tasks...
____________
My Facebook, War Commander, 2015

Profile Chris SProject donor
Volunteer tester
Avatar
Send message
Joined: 19 Nov 00
Posts: 31596
Credit: 12,492,752
RAC: 26,590
United Kingdom
Message 1007837 - Posted: 24 Jun 2010, 19:12:23 UTC

Roll 'im back.


Nah, best roll him over ;-)

____________
Damsel Rescuer, Uli Devotee, Julie Supporter, Kitty sad,
ES99 Admirer, Raccoon Friend, Anniet fan, Hon Triumphvir


1 · 2 · 3 · Next

Message boards : Technical News : Fill 'er up (Jun 23 2010)

Copyright © 2014 University of California