I'm just geting back to S@H after a couple years break so maybe this subject has been beaten to death. If so, apologies.

If you are upgrading today you have to be a person who cannot resist a low price or on a very limited budget not to get at least a dual core processor only modestly slower than it replaces. If watch for sales or wait six months you are more likely to buy a quad core. If the average S@H user upgrades every three years (insert your favorite number) the rate of WU processing will increase x4 in 5-6 years.

Is Berkeley ready for this? For ten years I have gotten irregular "no work available" replies.

---

Unvarnished
Haaretz
Jerusalem Post
The origin of the Yahweh Cult

I'm just geting back to S@H after a couple years break so maybe this subject has been beaten to death. If so, apologies.

If you are upgrading today you have to be a person who cannot resist a low price or on a very limited budget not to get at least a dual core processor only modestly slower than it replaces. If watch for sales or wait six months you are more likely to buy a quad core. If the average S@H user upgrades every three years (insert your favorite number) the rate of WU processing will increase x4 in 5-6 years.

Is Berkeley ready for this? For ten years I have gotten irregular "no work available" replies.

Depends a lot on what you mean by "ready."

Berkeley has consistently said they won't always have work.

Since BOINC was released, they've been good about pointing out other projects that can use our help.

In other words, they've got what they've got. They could increase the amount of work (the sensitivity of the search) and they could develop new algorithms to re-farm the existing recordings, but at the end of the day, they're doing what they promised: that's why there isn't always work.

---

I'm just geting back to S@H after a couple years break so maybe this subject has been beaten to death. If so, apologies.

If you are upgrading today you have to be a person who cannot resist a low price or on a very limited budget not to get at least a dual core processor only modestly slower than it replaces. If watch for sales or wait six months you are more likely to buy a quad core. If the average S@H user upgrades every three years (insert your favorite number) the rate of WU processing will increase x4 in 5-6 years.

Is Berkeley ready for this? For ten years I have gotten irregular "no work available" replies.

Depends a lot on what you mean by "ready."

Berkeley has consistently said they won't always have work.

Since BOINC was released, they've been good about pointing out other projects that can use our help.

In other words, they've got what they've got. They could increase the amount of work (the sensitivity of the search) and they could develop new algorithms to re-farm the existing recordings, but at the end of the day, they're doing what they promised: that's why there isn't always work.

I completely agree Ned.

---

The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's

Is Berkeley ready for this? For ten years I have gotten irregular "no work available" replies.

The question is....
Are YOU ready for Berkeley?

The Seti project has had it's ups and downs.
The servers get balky.....lose their handles on things.....
But with a reasonable work cache, it's no problem.

You will never see 'Amazon-level' uptime here........at least not until one of us Seti users wins the lottery and steps up to the plate...LOL.

Don't expect it, don't ask for it.

If you don't want to crunch Seti with it's up and downs, you will have to find other projects.....
But the fact of the matter is, they have all had their bad moments too.

So stick around, my friend. Just get a bit o' cache loaded, and it should carry you though the bad times on the servers.

Crunch on.

Meow.

---

"Freedom is just Chaos, with better lighting." Alan Dean Foster

I'm just geting back to S@H after a couple years break so maybe this subject has been beaten to death. If so, apologies.

If you are upgrading today you have to be a person who cannot resist a low price or on a very limited budget not to get at least a dual core processor only modestly slower than it replaces. If watch for sales or wait six months you are more likely to buy a quad core. If the average S@H user upgrades every three years (insert your favorite number) the rate of WU processing will increase x4 in 5-6 years.

Is Berkeley ready for this? For ten years I have gotten irregular "no work available" replies.

Depends a lot on what you mean by "ready."

Berkeley has consistently said they won't always have work.

Since BOINC was released, they've been good about pointing out other projects that can use our help.

In other words, they've got what they've got. They could increase the amount of work (the sensitivity of the search) and they could develop new algorithms to re-farm the existing recordings, but at the end of the day, they're doing what they promised: that's why there isn't always work.

I presume there is a limitation of raw data based upon the up time at Ariceibo or whatever the spelling. I remember reading about processing actual mag tape to create WUs. I remember increasing the resolution and sending out old WUs for reprocessing.

But in this matter depending upon the rate of computer upgrade us home folks are going to all double in 3-5 years and if everyone is like me quadruple in that time. And that is based upon my "gut feel" that more than four cores is not going to buy much user performance. If I am wrong, which I have been once or twice, in five years 8 cores are going to be common and shortly after that dominant.

So let me speculate a bit.

Is the system still bottlenecked with mag tape from the hinterland of Puerto Rico? Is there no way to go from antenna to internet direct to Berkeley?

Does the creation of WUs still require manual intervention?

Is there nothing else to look for in the raw data?

---

Unvarnished
Haaretz
Jerusalem Post
The origin of the Yahweh Cult

So let me speculate a bit.

Is the system still bottlenecked with mag tape from the hinterland of Puerto Rico? Is there no way to go from antenna to internet direct to Berkeley?

Data is no longer sent on physical tapes, data is recorded on hard drives and shipped to Berkeley. The drives are copied and sent back.

Never, ever, underestimate the amount of bandwidth available via FedEx, UPS or even the USPS.

Of course there is a way to get data from Arecibo to Berkeley over the 'net, it's just that high bandwidth connections are still very expensive in Puerto Rico.

Does the creation of WUs still require manual intervention?

I don't think the process is that labor intensive, if that's what you're asking.

Is there nothing else to look for in the raw data?

The SETI@Home recordings are used for a number of other purposes. A more interesting question is "are there other signs of intelligence that we might find in the data." Multibeam searches for narrowband signals while Astropulse searches for wideband data (and that is applicable to general astronomy). A hydrogen distribution study was done using the SETI@Home data, but not through BOINC.

---

Is Berkeley ready for this? For ten years I have gotten irregular "no work available" replies.

The question is....
Are YOU ready for Berkeley?

I started in May 2000. I'd say it is an even match.

The Seti project has had it's ups and downs.
The servers get balky.....lose their handles on things.....
But with a reasonable work cache, it's no problem.

I do have a problem with the cache now that I have five cores. I re-started in mid November 09 and I have increased my cache each time I have been idle for lack of work. I am up to ten days. Perhaps it was a bad period. I don't criticize as that means I might be taken as a volunteer. I simply observe. In this regard I have a 2-core I need to get my ass in gear to finish assembling and getting online to upgrade my home entertainment system. At the same time I will go from 5 to 7 cores. Yes I am a bit of a nerd but I point out in a couple years it will be hard not to buy a 4 core computer. That is what I am bringing up.

You will never see 'Amazon-level' uptime here........at least not until one of us Seti users wins the lottery and steps up to the plate...LOL.

Don't expect it, don't ask for it.

While I will volunteer to win that lottery I don't quite see the limitation. I assume the processing power at S@H has increased in proportion to mine and that it has always been at several hundred to one. But in my mere two months back I get the same kinds of down times and problems I have been reading about since 2000. And the effective number of active participants appears to have levelled out at about the 2005 number.

If you don't want to crunch Seti with it's up and downs, you will have to find other projects.....

Other than that lottery winner prediction project I don't know what might help.

But the fact of the matter is, they have all had their bad moments too.

So stick around, my friend. Just get a bit o' cache loaded, and it should carry you though the bad times on the servers.

Crunch on.

Meow.

I crossed 200,000 in the last few days. I quit only because I have a bit of a compulsive personality and after 8 years I had to have a break. I am getting back to my 98th percentile status rather faster than I had imagined. I mean two months ago 91st and now 94th. But also only dropping seven percentiles with two years of inactivity.

The message in that is interest in S@H has maxed out and did so long before two years ago.

---

Unvarnished
Haaretz
Jerusalem Post
The origin of the Yahweh Cult

I do have a problem with the cache now that I have five cores. I re-started in mid November 09 and I have increased my cache each time I have been idle for lack of work. I am up to ten days.

Try something more reasonable, such as 4 days.
I have a 4 day cache & have only run out of work 2 or 3 times since Seti moved to the BOINC platform.

---

Grant
Darwin NT

I do have a problem with the cache now that I have five cores. I re-started in mid November 09 and I have increased my cache each time I have been idle for lack of work. I am up to ten days.

Try something more reasonable, such as 4 days.
I have a 4 day cache & have only run out of work 2 or 3 times since Seti moved to the BOINC platform.

To me that is funny and the usual response, Boinc should be able to handle large caches or become obsolete. Computers will get faster and turning a blind eye and telling people that their cache is too big is a copout. How about telling people that you realize there is a problem and it is being worked on? To me running out 2 or 3 times when a cache would have gotten me thru is unaceptable.

---

Official Abuser of Boinc Buttons...
And no good credit hound!

I do have a problem with the cache now that I have five cores. I re-started in mid November 09 and I have increased my cache each time I have been idle for lack of work. I am up to ten days.

Try something more reasonable, such as 4 days.
I have a 4 day cache & have only run out of work 2 or 3 times since Seti moved to the BOINC platform.

To me that is funny and the usual response, Boinc should be able to handle large caches or become obsolete. Computers will get faster and turning a blind eye and telling people that their cache is too big is a copout. How about telling people that you realize there is a problem and it is being worked on? To me running out 2 or 3 times when a cache would have gotten me thru is unaceptable.

The cache size is determined by time, so it follows that a machine that runs twice as fast would cache roughly twice as many work units.

The problem with a big cache (especially if you use "connect every 'x' days" instead of "extra days") has to do with how cache size interacts with deadline pressure.

If there is pressure to meet deadlines, the last thing you need is more deadlines, and BOINC will not fetch work.

With a smaller cache, there are simply fewer opportunities to worry about schedules, and things tend to go more smoothly.

I think as a rule-of-thumb that 4/number of projects is a good cache size.

---

How about telling people that you realize there is a problem and it is being worked on?

The only real problem is people's expectations. How do we work on those?

---

Grant
Darwin NT

How about telling people that you realize there is a problem and it is being worked on?

The only real problem is people's expectations. How do we work on those?

Exactly.

Most of the rules that the client *must* follow are contradictory.

For example, people want to set up resource shares with a 1000:1 ratio. That means the low ratio project gets 1.4 minutes per day.

If that work unit has a deadline two weeks away, 10 hours of (estimated) work, and "connect every 'x'" at 7 days, then it has to be finished in a week, and it needs at least 85 minutes per day -- and a safety margin just in case the estimate is wrong, or the deadline will be missed.

If someone sets their cache to ten days, they expect 10 days of cache no matter what -- and ignore the fact that if BOINC is worried about deadlines, that there is a risk of getting work with a short deadline.

If project "A" is overworked, and "A" is the only one with work units, BOINC should not fill the cache completely full with "A" -- it needs to keep room available to download other projects or it can't possibly honor resource shares.

---

I do have a problem with the cache now that I have five cores. I re-started in mid November 09 and I have increased my cache each time I have been idle for lack of work. I am up to ten days.

Try something more reasonable, such as 4 days.
I have a 4 day cache & have only run out of work 2 or 3 times since Seti moved to the BOINC platform.

To me that is funny and the usual response, Boinc should be able to handle large caches or become obsolete. Computers will get faster and turning a blind eye and telling people that their cache is too big is a copout. How about telling people that you realize there is a problem and it is being worked on? To me running out 2 or 3 times when a cache would have gotten me thru is unaceptable.

If you'd be willing to donate your computer to the dev team I'm sure they'd be happy to work on it. :D
Just messing with you, but really there are some things that are limits no matter what you want to happen. Generally these are limits defined by the programming language and/or the controls they have to work with.
They may decide later to do soemthing along the lines of seperating the GPU processing into another application. So you would run BOINC & BOINC for GPUs.
Seperate functionality is something I'd like to see myself. I don't see the need for one project to hammer the servers requesting GPU work when it'sset to only run CPU tasks. Then have another project that only runs GPU work constantly requesting CPU work when it's not set to run any.

---

SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[

If project "A" is overworked, and "A" is the only one with work units, BOINC should not fill the cache completely full with "A" -- it needs to keep room available to download other projects or it can't possibly honor resource shares.

What I find annoying about how BOINC currently handles it's work unit cache, is that it thrashes back and forth depending on which project it runs first.

For instance, if I have set up shares for 75% project A and 25% project B, but my work queue looks like this:

(First WU)[----------------------75% A------------------][-------25% B-------](Last WU)

Most people would expect the work queue to be pretty stable. However, when BOINC starts processing work units from project A, project A accumulates too much long term debt, just because it is running first, and BOINC starts downloading tons of work for project B without regard to what is already in the queue. So you end up with a work unit queue that looks like this:

[A][------------------------------99% B---------------------------]

which makes people think BOINC is broken. And then it starts processing project B and goes back to the other extreme, downloading only from project A, thrashing back and forth between the two projects.

I've seen it do this, and it drives me nuts.

---

If project "A" is overworked, and "A" is the only one with work units, BOINC should not fill the cache completely full with "A" -- it needs to keep room available to download other projects or it can't possibly honor resource shares.

What I find annoying about how BOINC currently handles it's work unit cache, is that it thrashes back and forth depending on which project it runs first.

For instance, if I have set up shares for 75% project A and 25% project B, but my work queue looks like this:

(First WU)[----------------------75% A------------------][-------25% B-------](Last WU)

Most people would expect the work queue to be pretty stable. However, when BOINC starts processing work units from project A, project A accumulates too much long term debt, just because it is running first, and BOINC starts downloading tons of work for project B without regard to what is already in the queue. So you end up with a work unit queue that looks like this:

[A][------------------------------99% B---------------------------]

which makes people think BOINC is broken. And then it starts processing project B and goes back to the other extreme, downloading only from project A, thrashing back and forth between the two projects.

I've seen it do this, and it drives me nuts.

This is caused by the fix for a much bigger problem.

Before this behaviour was introduced, BOINC would do whatever was needed to get work and keep the cache full. It had a better mix of work, but availability counted for a whole lot more than resource share.

There were too many cases where a project with good uptime would fill up the queue even when that project was overworked.

This will be the most extreme when you carry lots of "extra days" and a "connect every" interval near zero.

... and the problem is thinking. Intuition says that the best way to do this is to make sure you always download minutes worth of work in strict resource ratio, and that we should always have that in the queue.

Go back to my example. Resource share of 1000:1 (which is supposed to work) so we want to download 1.4 minutes of work each day from the low-share project.

Trouble is, that project only has work units that run over an hour, so there is no way to get work in nice chunks in resource share ratio.

There are projects like HashClash (now done) that took about 8 minutes, and CPDN whose work units can take a year.

... and it all has to work. BOINC has to meet deadlines, keep the cache full, and follow resource share. It has to work for someone with "connect every 7 days" (which means they may not be able to connect the other six) and "extra days" at 10 (17 day cache) and it should work for someone who sets 0-0.

... and all of this would be made a lot simpler if we could actually program intelligence (not AI, intelligence) and not rely on algorithms.

---

The trouble with both this discussion, and the equivalent one going on in the BOINC development mailing list, is that they both try to take an early design decision by the BOINC development team, shoehorn a completely different objective into it, and then complain that the glass slipper doesn't turn you into a princess.

"Resource Share". Is that a short term or a long term constraint? Can it be both at the same time?

The discussion at BOINC is mainly about the concept of a 'backup project': break glass for an emergency task if the love of your life has gone offline for a nanosecond. People seem to use a micro-RS for this: "Don't fetch unless..."

But I'm sure the same people want to crunch that task, if and when it is ever downloaded, as quickly as possible - in the hope that their main project will be back up and able to service their every need by the time the interloper is despatched. You cannot achieve both objectives - minimal chance of download, but maximal speed of despatch if the need ever arises - with a single metric.

Those who want a "backup project" facility should proof-read changeset [trac]changeset:20286[/trac] carefully, and check that David incorporated all of the above accurately last night.

The trouble with both this discussion, and the equivalent one going on in the BOINC development mailing list, is that they both try to take an early design decision by the BOINC development team, shoehorn a completely different objective into it, and then complain that the glass slipper doesn't turn you into a princess.

"Resource Share". Is that a short term or a long term constraint? Can it be both at the same time?

If you remember back several versions (maybe back into the 5.x rev.) BOINC did a good job of keeping the resource share -- subject to all of the limitations like meeting deadlines and making progress on long term projects like CPDN.

The reported problem was that it didn't try to keep the cache full, and that was very bad for those on dialups or otherwise very intermittent connections.

So, then they made BOINC keep the cache really full, and if the rule is "keep it full at any cost" then honoring resource share could be impossible.

I don't think resource share is a constraint, I think it's a goal. In a perfect world, all debts are zero because we're doing work in exactly the proportion requested.

We get in trouble with these pesky deadlines.

If you don't have deadlines, projects like CPDN might have to wait decades for a work unit.

It seems to me that the short-term scheduling could be based solely on making deadlines, and resource share could be controlled by limiting work fetch to the "most needy" project, but I'm sure there is a hole in my logic.

[edit] In my opinion, the biggest challenge is the overall complexity caused by the multiple, conflicting requirements like "meet deadlines" and "keep the cache full" and "never let a resource go idle" -- and some of the new ideas like work units that use more than one core just make things a whole lot harder.

If there was a way to simplify things and still meet the requirements, I think it'd be better. I'm also not sure that's possible.

---

It seems to me that the short-term scheduling could be based solely on making deadlines, and resource share could be controlled by limiting work fetch to the "most needy" project, but I'm sure there is a hole in my logic.

I think the hole in that logic is CPDN. If your basic algorithm is:

1) Job scheduling: Of the WUs already downloaded, run the project that has the least leeway with respect to deadlines. I.e., if I run all the SETI WUs I've got before any other project, in deadline order, the closest any of those WUs gets to its deadline is 85 hours. If that's lower than any other project, run the SETI WUs.

2) Work Fetch: If I need more work to fill the cache, fetch work for the project with the most long term debt. If it doesn't have any work, then fetch from the project with the second most long term debt, and so on.

The problem with this very rational approach is that you'll download CPDN WUs but never run them, at least as long as there's anything else in your cache. Eventually you'll have a cache full of CPDN and nothing else, and then finally run one of them. As soon as it completes, you'll download from another project, and not touch the CPDN WUs again until, once again, all you have is CPDN in the cache. So, in essence, what will happen is your cache will be full of CPDNs which won't run, and a tiny portion of the cache will be used for WUs from other projects which will run. This will go on for a year or so until the CPDNs that have been sitting in the cache are themselves under deadline preasure. Then they'll finally run.

It will work, but that's not really the behavior you're expecting.

---

Want to find one of the largest known primes? Try PrimeGrid. Or help cure disease at WCG.

It seems to me that the short-term scheduling could be based solely on making deadlines, and resource share could be controlled by limiting work fetch to the "most needy" project, but I'm sure there is a hole in my logic.

I think the hole in that logic is CPDN. If your basic algorithm is:

1) Job scheduling: Of the WUs already downloaded, run the project that has the least leeway with respect to deadlines. I.e., if I run all the SETI WUs I've got before any other project, in deadline order, the closest any of those WUs gets to its deadline is 85 hours. If that's lower than any other project, run the SETI WUs.

2) Work Fetch: If I need more work to fill the cache, fetch work for the project with the most long term debt. If it doesn't have any work, then fetch from the project with the second most long term debt, and so on.

The problem with this very rational approach is that you'll download CPDN WUs but never run them, at least as long as there's anything else in your cache. Eventually you'll have a cache full of CPDN and nothing else, and then finally run one of them. As soon as it completes, you'll download from another project, and not touch the CPDN WUs again until, once again, all you have is CPDN in the cache. So, in essence, what will happen is your cache will be full of CPDNs which won't run, and a tiny portion of the cache will be used for WUs from other projects which will run. This will go on for a year or so until the CPDNs that have been sitting in the cache are themselves under deadline preasure. Then they'll finally run.

It will work, but that's not really the behavior you're expecting.

That wasn't what I was suggesting, because as you point out it would not work.

The first rule has to be "meet deadlines."

... and you can't wait until you're in deadline trouble because the estimate may be pretty inaccurate -- it is after all an estimate.

So, one idea that might work is to first check to see if any work unit needs to run in "priority" -- if there is a risk of missing deadlines on any one work unit.

From there, I'd think that checking each project to make sure that it is "progressing" toward deadlines, and picking the (started) WU that needs some "catch up" time.

... and then, if no one is "running late" go round-robin by resource share and first-in, first-out.

But, I haven't tried it, haven't simulated it, and I think the short term scheduler basically works.

This discussion has more to do with work-fetch than anything else.

---

The way i see it the manager actually does it's job & does it well.
The problem is people's expectations- they expect shares to be honoured in the short term (24 hours, or even less for some people)- but the larger the cache & the bigger the mix of long & short deadlines then the greater the period of time needed to honour resource shares (at the very least weeks, given the size of some caches & the project mix months even).

If people get all worked up over it struggling with a 10 day cache, the easiest & most effective fix would be to make it 7 days, or even 5 as a maximum.

---

Grant
Darwin NT