Let me address the "no work" issues as of late. We've been running low on work to send out (or had the schedulers turned off) for several reasons:

1. Each raw data file has to go through a local software based radar analysis - a suite of programs that takes over 3 hours to run per file. This should keep up with the incoming data flow, but some nagging NFS/mounting bugs cause this suite to lock up several times a week. Each time it does the whole systems getting new data on line is clogged until a human can figure out where it was in the process, clean it up, and start the broken file over again (resulting in many hours of lost processing time). For example this morning we found it all jammed last night, cleaned it up around 9am, and finally around 12:30pm new workunits were available again. We're working on adding some band-aid solutions to this particular problem.

2. Server crashes: mork and ptolemy are prone to crashing for no apparent reason. Either of them going down causes the project to halt until we recover. Sometimes it takes days to fully get back to a regular work-flow pace again. We're trying to shuffle services around to get ptolemy out of the picture. Why ptolemy instead of mork? Mork is a much bigger system and therefore much harder to replace - plus when it goes down the download servers are at least still able to work for a while.

3. Some data files error out pretty quickly due to noise or garbage data.

4. The CUDA clients sure burn through work fast.

5. Some CUDA clients were returning garbage. To combat this a fix to the scheduler was put on line this Monday, but was unable to start it without errors. It took Eric, Jeff, and I all day, and most of the next morning, to finally find the obscure problem - which was actually a misleading redirect in the apache config (that was put in many months ago). By the time we fixed it, we were already into the weekly outage.

So lots of battles on this front. In any case we are collecting data at this point (on 2TB drives, which means we'll lose less data waiting for the Arecibo operators to swap out the older 500/750GB drives), and still have a backlog of stuff to process in our archives. The lab is also getting a Gbit link to the world in July so the slow transfers to/from these archives will no longer be a bottleneck. Note this link is for the whole lab and our SETI specific data link will remain at 100MBit. Still, it's an improvement.

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Thanks Matt for the update and all the efforts,

Claggy

Thanks a bunch.

Appreciate the update/information

ron tillman

---

Thank you Matt! This news certainly will be appreciated by all. You are right, in that GPU crunchers can burn through a lot of work.

Steve

---

Warning, addicted to SETI crunching!
Crunching as a member of GPU Users Group.
GPUUG Website

Thanks a bunch.

Appreciate the update/information

The same here too.

---

WOW... Matt.... thanks for the very specific update. Congrat's on getting the Gbit-link to the outside world. That will surely make things a lot better... in some ways.

And... a hearty "thank you" to all of the people who work on SETI.... and keep it going.

---

I DONATE money to SETI@home.... DO YOU?

I'm just slowly BOINC'ing along.

Hey... ET... you have a sister who likes earthlings?

Afraid I was among the "returning garbage" Computers. I finally changed the GPU to "wait until idle 9999 minutes" (NVIDIA GeForce GT 220 (986MB) driver: 19745).

A heads up when it is safe for us to return them to work would be appreciated.
Until then, crunching away on 6.03 only. Thank you for the update Matt!

Janice

Afraid I was among the "returning garbage" Computers. I finally changed the GPU to "wait until idle 9999 minutes" (NVIDIA GeForce GT 220 (986MB) driver: 19745).

A heads up when it is safe for us to return them to work would be appreciated.
Until then, crunching away on 6.03 only. Thank you for the update Matt!

Janice

I don't see any sign of trouble on your GT 220 - host 5385323.

The main 'garbage' offenders were the new GTX 470 and GTX 480. Yours isn't affected - you're fine to go back to normal crunching (as and when work is available, of course).

Thanks Matt for the update.

Byron

---

Again,

Thanks & Kudos to the SETI Staff and volunteers who labor long and hard to keep,what appears to be, a rather cantankerous system up and running.

One cannot help but be impressed with the dedication and determination dispalyed by all of you.

In the mean...............when work becomes available, is there any chance of getting some "Astropulse" tasks?

thank you,


rt

---

Matt, thanks for the info on work issues.

1. Each raw data file has to go through a local software based radar analysis -

I don't suppose you have extensive file locking & error checking code? Some sort of time-out, retry, or checkpoint of input file would be really helpful. Things like signal handling and using chsize(), if you know the output file size ahead of time, can help. Help Wanted forum? There are a lot of nerd programmers out there :cough: who would love to use their 1TB drives, core i7, and Virtualbox with NFS mounts to optimize the software radar blanking code. With internet2 access, testfile size is not an issue.

And, congratulations on the gigabit link next month. You'll be able to use Internet2 to its full potential. :)

Thamks Matt for all your hard work and all the SETI staff.

---

Thanks a lot for maintain us informed...

Ask: may be this post a return to the good old times of "several posts per week" to maintain live the contact with the users?
(Apologizes for my English; is not my mother language)

---

Afraid I was among the "returning garbage" Computers. I finally changed the GPU to "wait until idle 9999 minutes" (NVIDIA GeForce GT 220 (986MB) driver: 19745).

A heads up when it is safe for us to return them to work would be appreciated.
Until then, crunching away on 6.03 only. Thank you for the update Matt!

Janice

I don't see any sign of trouble on your GT 220 - host 5385323.

The main 'garbage' offenders were the new GTX 470 and GTX 480. Yours isn't affected - you're fine to go back to normal crunching (as and when work is available, of course).

Most of the invalid/error have timed out, they were beginning of may when I gave up and went CPU only (no more cuda/6.09).

I will try turning it back on.. hopefully it is fixed. I just figured no units was better for the project than mutilated units.

Oh - seems like I forgot another major reason for little work:

6. "Cannot Find Coincident Blanking Signal" - that's the error reported by half the beams in many recent raw data files. What does that mean? For some reason the data files are being written in a format which makes it impossible for some beams to find the blanking signal information, which they need to process (or else they will be cluttered with radar interference). So they error out, and we're effectively losing half our data at this point, which of course speeds up the burn rate quite a bit. Another thing we're looking into. We should be able to figure it out and reprocess the missing beams in the future.

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Hello! server crashes for no reasons? Maybe you should rename 'em. Zatar and Apollo! Or something. OK I'm just bored.

Thanks Matt, for the update.

Byron

---

Thank you! That is valuable information indeed!

Steve

---

Warning, addicted to SETI crunching!
Crunching as a member of GPU Users Group.
GPUUG Website

I certainly appreciate the update. It seems as though you all are fighting an uphill battle to keep things viable.

Thanks muchly for all the effort.
Jeff

---

Afraid I was among the "returning garbage" Computers. I finally changed the GPU to "wait until idle 9999 minutes" (NVIDIA GeForce GT 220 (986MB) driver: 19745).

A heads up when it is safe for us to return them to work would be appreciated.
Until then, crunching away on 6.03 only. Thank you for the update Matt!

Janice

I don't see any sign of trouble on your GT 220 - host 5385323.

The main 'garbage' offenders were the new GTX 470 and GTX 480. Yours isn't affected - you're fine to go back to normal crunching (as and when work is available, of course).

Most of the invalid/error have timed out, they were beginning of may when I gave up and went CPU only (no more cuda/6.09).

I will try turning it back on.. hopefully it is fixed. I just figured no units was better for the project than mutilated units.

Lovely. I use 6.03 and I know from what I've read that 6.09 is dead, dead, dead, 6.10 is for Fermi.

---

Savoir-Faire is everywhere!
The T1 Trust, T1 Class 4-4-4-4 #5550, America's First HST