The Plan

Message boards : Number crunching : The Plan
Message board moderation

To post messages, you must log in.

Previous · 1 · 2

AuthorMessage
Sergey Broudkov
Avatar

Send message
Joined: 24 May 04
Posts: 221
Credit: 561,897
RAC: 0
Russia
Message 207123 - Posted: 8 Dec 2005, 20:53:49 UTC - in response to Message 207080.  
Last modified: 8 Dec 2005, 20:54:26 UTC

It's not really a matter of credit. It's a matter of making a useful contribution to the project. In my opinion if my computer spends time processing a unit for which a quorum has already been reached the time spent on that unit is wasted. Don't misunderstand me.


You see only one side of the matter. The redundacy is the essential part of the project, so even those who don't provide canonical result or don't make a quorum, they are nevertheless are taking part, and their input is as valuable as other's. Consider it as a sport team. There are attackers who score goals, but there are also defenders, coaches, doctors etc, and they ALL TOGETHER win a cup.

EDIT: typos
Kitty@SETI team (Russia). Our cats also want to know if there is ETI out there
ID: 207123 · Report as offensive
W-K 666 Project Donor
Volunteer tester

Send message
Joined: 18 May 99
Posts: 19062
Credit: 40,757,560
RAC: 67
United Kingdom
Message 207144 - Posted: 8 Dec 2005, 21:06:23 UTC - in response to Message 206915.  
Last modified: 8 Dec 2005, 21:09:24 UTC

As for the three month deadline, it will increase the length of time that information is stored on the drives at berkeley, further burdening an already overloaded file system. Files pertaining to a particular work unit will be held at least three months, possibly much longer if a quorum isn't reached at the end of the three months. This could mean a delay of three months, six months, or longer before credit is granted for some units.

Then there's the factor of the widely varied capabilities of computers. With my computer's current configuration, it's processing work units using setiathome_SSE-naparst-r3.4 at a rate of 2 units every 4.5 hours. There's many newer computers processing work units in one or two hours. If a work unit is assigned to my computer and three faster computers the probability of a quorum being reached before my computer completes the unit is much greater with the enhanced seti app than with the current version. Some may not see how this is possible, but if you consider that the enhanced version will take up to 45 hours to process a work unit on my computer (90 hours if I have to use an unoptimized app) and 10 to 20 hours on newer computers, on many work units a quorum will be reached a full day or more before my computer finishes the work. The time spent processing these units would be wasted. It happens occasionally now, but not very often due to the practice of caching work. With the current app I often see work reported by faster computers a day or more after my computer.

my $0.02

edited to fix typos.


Me thinks you answered you own concerns there about the need for the fourth unit. From observation and from what other people have said the number of units not returned before deadline or are not validated has decreased for the 1 in 4 of six moths ago but it is still significant.

With a three month deadline, if they don't send the fourth unit then, are the majority prepared to wait up to six months for the credits to be granted on 10% or more of their units. My hunch is that because of the longer crunch times and the need for the BOINC manager to calculate the correction factor, there is going to be significant rise in units aborted or not returned. Especially those who use an optimised app, because their friends said it was a good idea, and are used to crunch times measured in 5 hrs or less, and then see it has suddendly risen to a predicted 100's of hours, and don't see, don't realise that it now says Seti@home ver 5.n.x and not 4.18. [editted for typo's]
ID: 207144 · Report as offensive
Profile tekwyzrd
Volunteer tester
Avatar

Send message
Joined: 21 Nov 01
Posts: 767
Credit: 30,009
RAC: 0
United States
Message 207173 - Posted: 8 Dec 2005, 21:39:35 UTC - in response to Message 207118.  
Last modified: 8 Dec 2005, 21:40:24 UTC

I see a strong possibility that the changeover to the enhanced app as planned could lead to a decision to set minimum processor requirements and eliminate many computers from the project.

Why would you see that?


How do I explain this?
As things stand, there's little difference in returning units that complete in 1 hour vs 4 hours. This is due at least in part to work unit caching habits, which will have less of an effect with the enhanced version. When the changeover is made you're going to have differences in run times of days (if optimized apps are available) to a week or more (if no optimized apps are available). As a result, files will reside on the berkeley file system longer. It's likely that some remedy to the buildup will be necessary. A simple solution would be to set minimum processor requirements.

Assuming the use of unoptimized seti enhanced, if a work unit is issued to a mix of machines including a p2 400MHz, a p3 800MHz, and a p4 3.0 GHz, the p4 will likely return results the same day. The p3 will likely return the result a week later. The p2 will likely return results two weeks later. In this case, assume a quorum is reached by these three computers. This means deletion after about two weeks. In cases where a quorum is not reached by the three returned results, and the fourth isn't returned, the files will be in the file system for over three months. As files like this build up, file system performance will be further degraded (in a manner similar to that caused by of the buildup of orphaned units). An easy solution would be to eliminate computers below a certain performance level, thus reducing the difference in processing times, interval between result returns, and the amount of time a set of files reside on the file system.

Nothing travels faster than the speed of light with the possible exception of bad news, which obeys its own special laws.
Douglas Adams (1952 - 2001)
ID: 207173 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 207185 - Posted: 8 Dec 2005, 21:48:54 UTC - in response to Message 207173.  

I see a strong possibility that the changeover to the enhanced app as planned could lead to a decision to set minimum processor requirements and eliminate many computers from the project.

Why would you see that?


How do I explain this?
As things stand, there's little difference in returning units that complete in 1 hour vs 4 hours. This is due at least in part to work unit caching habits, which will have less of an effect with the enhanced version. When the changeover is made you're going to have differences in run times of days (if optimized apps are available) to a week or more (if no optimized apps are available). As a result, files will reside on the berkeley file system longer. It's likely that some remedy to the buildup will be necessary. A simple solution would be to set minimum processor requirements.

Assuming the use of unoptimized seti enhanced, if a work unit is issued to a mix of machines including a p2 400MHz, a p3 800MHz, and a p4 3.0 GHz, the p4 will likely return results the same day. The p3 will likely return the result a week later. The p2 will likely return results two weeks later. In this case, assume a quorum is reached by these three computers. This means deletion after about two weeks. In cases where a quorum is not reached by the three returned results, and the fourth isn't returned, the files will be in the file system for over three months. As files like this build up, file system performance will be further degraded (in a manner similar to that caused by of the buildup of orphaned units). An easy solution would be to eliminate computers below a certain performance level, thus reducing the difference in processing times, interval between result returns, and the amount of time a set of files reside on the file system.


I suspect that the slower pace will mostly solve that problem, but a simple solution is to assign work to machines that work at a similar pace.
ID: 207185 · Report as offensive
Profile tekwyzrd
Volunteer tester
Avatar

Send message
Joined: 21 Nov 01
Posts: 767
Credit: 30,009
RAC: 0
United States
Message 207193 - Posted: 8 Dec 2005, 21:55:19 UTC - in response to Message 207185.  

I suspect that the slower pace will mostly solve that problem, but a simple solution is to assign work to machines that work at a similar pace.


Maybe I should have said easiest solution.
I may be mistaken but I seem to remember seeing the idea of assigning work to similar computers mentioned in another thread some time ago but it was rejected due to an associated need to revise the server software and possible impact on performance.

Nothing travels faster than the speed of light with the possible exception of bad news, which obeys its own special laws.
Douglas Adams (1952 - 2001)
ID: 207193 · Report as offensive
rsisto
Volunteer tester

Send message
Joined: 30 Jul 03
Posts: 135
Credit: 729,936
RAC: 0
Uruguay
Message 207213 - Posted: 8 Dec 2005, 22:11:42 UTC - in response to Message 207193.  

I suspect that the slower pace will mostly solve that problem, but a simple solution is to assign work to machines that work at a similar pace.


Maybe I should have said easiest solution.
I may be mistaken but I seem to remember seeing the idea of assigning work to similar computers mentioned in another thread some time ago but it was rejected due to an associated need to revise the server software and possible impact on performance.


I think this is already implemented, but it is only used when they need to send a fith (or higher) unit. In this cases they send them to host with low Average Turnaround Time.
ID: 207213 · Report as offensive
W-K 666 Project Donor
Volunteer tester

Send message
Joined: 18 May 99
Posts: 19062
Credit: 40,757,560
RAC: 67
United Kingdom
Message 207224 - Posted: 8 Dec 2005, 22:28:43 UTC - in response to Message 207193.  

I suspect that the slower pace will mostly solve that problem, but a simple solution is to assign work to machines that work at a similar pace.


Maybe I should have said easiest solution.
I may be mistaken but I seem to remember seeing the idea of assigning work to similar computers mentioned in another thread some time ago but it was rejected due to an associated need to revise the server software and possible impact on performance.


It's not similar computers, it's similar turnaround times, that needs to be the deciding factor.

i.e. if host 'A' has 0.5 days connect time and crunch time is between nine hours and 12 hrs, then average turnarround would be slightly lower that 0.5 days. While host 'B' has connect time of 10 days and crunch of between one and two hours, the average turnaround would be about 10 days. I'm assuming correct sized caches and connection is always available.
ID: 207224 · Report as offensive
Ingleside
Volunteer developer

Send message
Joined: 4 Feb 03
Posts: 1546
Credit: 15,832,022
RAC: 13
Norway
Message 207450 - Posted: 9 Dec 2005, 1:53:25 UTC - in response to Message 207224.  

It's not similar computers, it's similar turnaround times, that needs to be the deciding factor.

i.e. if host 'A' has 0.5 days connect time and crunch time is between nine hours and 12 hrs, then average turnarround would be slightly lower that 0.5 days. While host 'B' has connect time of 10 days and crunch of between one and two hours, the average turnaround would be about 10 days. I'm assuming correct sized caches and connection is always available.



Since we're talking about Seti_Enhanced, host A will not have between 9h and 12h crunch-times, but more likely between 9h and 135h, while B will have 1h-15h.

Meaning, host A's average turnaround-time on "normal" angle-range should be around 4 days. If host B has decreased cache-size to 4 days they'll both have similar turnaround-times and can be paired together. If it's another "normal" angle-range, no problem, both will report on roughly the same time. But, if it's a "fast" result, host A will report 3 days before host B, while if "slow" result, host A will report 2 days after B.


Now, let's keep host A, but change to a host C, with cache-setting 0.1 days and cpu-times from 6 days to 90 days. If host A has just crunched-through a bunch of "slow" results and host C a bunch of "fast" results, they can be paired-together. If they gets assigned a "fast" result, host A will report 5 days before C. If instead they get assigned a "slow" result, host A will report 80+ days before host C...



Anyway, don't remember if the idea to send copies to hosts with similar average turnaround-times has ever been finalized and it's just like some other things that not all projects is using it, if it's still on the "to do"-list, or if it's been scrapped due to various weaknesses...
ID: 207450 · Report as offensive
W-K 666 Project Donor
Volunteer tester

Send message
Joined: 18 May 99
Posts: 19062
Credit: 40,757,560
RAC: 67
United Kingdom
Message 207724 - Posted: 9 Dec 2005, 7:54:44 UTC

What I didn't make clear was send the four copies of a unit to computers with similar turn round times, but I agree there are too many factors in the equation for any sort of pairing to be practical, even if you factor out the unexpected.
ID: 207724 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 65747
Credit: 55,293,173
RAC: 49
United States
Message 207731 - Posted: 9 Dec 2005, 8:13:26 UTC - in response to Message 206378.  
Last modified: 9 Dec 2005, 8:17:31 UTC

Hi all, I'm about to layout the conversion plan the best I can piece together. Nothing I say here should be interpreted as coming from Berkeley. I'm seated comfortably in my office chair at my home in South Carolina. There the disclaimer is out of the way.

I've been watching boinc develop and grow. I've kept track in my mind what's said by the developers about the plans. Most of what I say can be referenced to statements by developers, but it is in bits and pieces, not one big layed out plan.

My intention is that given what I know that some of the newer users may feel comforted that there is atleast a plan, and that it's more or less on track.

History
Seti classic was wildly successful, so successful that they had more computer power than they needed. In order to keep users happy they would "on occasion" send out the exact same wu upwards of 29 times(source Rom Walton), Matt Lebofsky stated that the average number of times a wu was sent was 6-9. Note: to do good science only two "strongly similar" results are needed to validate an answer. This is inefficient (tony's interpretation). I could see Dr. A saying to himself "Wow this is fantastic, but I hate to see all this wasted computer power, what can I do?". "I know I'll design a program that will manage applications to allow other poor projects to benefit from our success and these extra cpu cycles" Boinc was Born.

The Problems
Now there are two projects crunching similar wus with similar software (the application part). By now Dr. Anderson had made his decision to use boinc, he was getting grant money to do it, it was the future. It was more efficient at acheiving the science since only 4 results (work units) are sent instead of 6-9. Now how to get both these projects merged into one? They have limited hardware and limited money.

The growth
Boinc started small, a couple servers and some volunteers. It grew. they started taking servers from classic to feed boincs growth, and it kept growing. they snag another server. Now both projects are on the ragged edge in need of gear.

The solution
Merge them both, but how to do that??? If we put all those people in one place, and since boinc is more efficient we'll run out of work again, and the existing servers at boinc won't handle it. what to do?? They've chosen to do the following:

1) recommend joining other projects
2) increase the sensitivity of our search (this is what the seti beta team has been working on since June 2005, it doubles the sensitivity of the search and takes 4 to 10 times longer to crunch a result. When released it will reduce the load by at least a factor of 4)
3)add on Astropulse and data from other sources (multibeam receiver) as soon as ready.
4) add the classic equipment to boinc

The plan
1) merge the master science databases (november 15th)
2) send out emails about the closure of classic
3) adopt the beta application as the seti standard (so the load will be reduced on the servers)
4) close classic (december 15th)
5) reconfigure and task the classic gear in with the boinc gear.

Today
Seti planned to merge the master science database of Classic with that of Boinc November 15th (see tech news), software/incompatibility issues postponed it.
Then they released the classic closure emails and blogs. All the new users overloaded the server, while it was operating at a reduced capacity, they decided to do the database merge, it's like killing two birds with one stone. It accomplishes the merge and lets the server operate faster

Since 22 November we've gone from 256K to 311K users(boincstats). This is the problem, but the solution is the next step scheduled to happen before 8 days from now(as I've heard the news,unless it gets pushed back). That's the adoption of the beta client which will cut the load drastically thereby making communications easier. Then when the classic gear is added, it will get even easier and have room for future growth. (new users)

I can't predict the future, maybe something doesn't happen like it should (like the master science database merge), and dates can change. My point here is poeple need to know a plan exists to ease the current problems. Hopefully new users will see this and say "hey they knew there'd be problems and they tried to account for it in the plans".

I hope you stick around, if nothing else the next couple weeks ought to be very interesting.

thanks for your time

tony


Neat, Of course I never received an email at My verizon.net address(I'm on Boinc 5.2.13 now and crunching optimized for sse2&A64), Was this email sent by an SMTP client or PHP? As Verizon DSL requires Server Authentication for all Incoming & Outgoing email and Comcast may not only be experimenting with It, But going in that direction, Heard It on www.dslreports.com of course(about comcast), I've found 21 forums that are verizon.net email compatible so far: Verizon.net email compatible forums list.
The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's
ID: 207731 · Report as offensive
Astro
Volunteer tester
Avatar

Send message
Joined: 16 Apr 02
Posts: 8026
Credit: 600,015
RAC: 0
Message 209843 - Posted: 11 Dec 2005, 1:36:11 UTC
Last modified: 11 Dec 2005, 1:37:53 UTC

I was glad to see Matt mention the new app, but I'd feel better if he'd have inserted a timeframe....oh well

Here is a link to what developer Matt Lebofsky said earlier today
ID: 209843 · Report as offensive
kevin & seth

Send message
Joined: 30 Nov 05
Posts: 128
Credit: 258
RAC: 0
Message 214647 - Posted: 15 Dec 2005, 9:55:26 UTC

mmciastro said in Message 214340 :
fpmmpfmf mfpmffp pfmrfpf pfmp ffmm p ppfmmf pfpmpp ppfmfpmmpmm pmfmfmp ffmpmmpm mpfmppmfpmfpm. mmfpmfpmfmp mpfmfpmfpmfpppfmp 424 mmfpmmpfmfmp mfmfmfp mfpmfp mfmp. mmpfmpfpmf 4443 pmfmfmm fpfmpmfpmpfpm ppmppmpppfm fpmfpmpp pmfpmpfmpfpmp pfmmppfpffm fpmpfmpfpmp. mfpmpfpmfmpppmf mmfpf fffmp mpfmpmppppmffp pfmpfmpfpfpp pmfpmmmfpp pmfpmffmpmmpm mppmpmmfp pmmmf.

mmfpmmp fmpmpfmpmpfpfpm fmfpf mmmfppfpmp fmpp fmpp f pmpmpmpfmfp. mmmpfmmfm mmfmppp fmfm pmpf pfmpfpm pppmpfmp. mmmpfpf ppmpmpfpmpp p ppfpppmpppfm fpfmfpmpfp mpmmfpfpmmfpmmpfm

mmpfmpfpm fmmppf

What plan?
ID: 214647 · Report as offensive
Astro
Volunteer tester
Avatar

Send message
Joined: 16 Apr 02
Posts: 8026
Credit: 600,015
RAC: 0
Message 214714 - Posted: 15 Dec 2005, 12:34:48 UTC

Matt, gave an update, so I guess they're pushing off the release of the "enhanced" application a couple weeks.

See Matts' Post

which states:

Matt Lebofsky
Forum moderator
Project developer
Project scientist

Joined: Mar 1, 1999
Posts: 287
ID: 122079
Credit: 35,864
RAC: 25
Message 214503 - Posted 15 Dec 2005 5:09:00 UTC
Enhanced is still a couple weeks off. These are all regular ol' splitters.

- Matt
____________
-- BOINC/SETI@home network/web/science/development
-- Get the inside scoop: Matt's Unofficial SETI Log

ID: 214714 · Report as offensive
kevin & seth

Send message
Joined: 30 Nov 05
Posts: 128
Credit: 258
RAC: 0
Message 214742 - Posted: 15 Dec 2005, 13:09:05 UTC - in response to Message 214647.  

mmciastro said in Message 214340 :
fpmmpfmf mfpmffp pfmrfpf pfmp ffmm p ppfmmf pfpmpp ppfmfpmmpmm pmfmfmp ffmpmmpm mpfmppmfpmfpm. mmfpmfpmfmp mpfmfpmfpmfpppfmp 424 mmfpmmpfmfmp mfmfmfp mfpmfp mfmp. mmpfmpfpmf 4443 pmfmfmm fpfmpmfpmpfpm ppmppmpppfm fpmfpmpp pmfpmpfmpfpmp pfmmppfpffm fpmpfmpfpmp. mfpmpfpmfmpppmf mmfpf fffmp mpfmpmppppmffp pfmpfmpfpfpp pmfpmmmfpp pmfpmffmpmmpm mppmpmmfp pmmmf.

mmfpmmp fmpmpfmpmpfpfpm fmfpf mmmfppfpmp fmpp fmpp f pmpmpmpfmfp. mmmpfmmfm mmfmppp fmfm pmpf pfmpfpm pppmpfmp. mmmpfpf ppmpmpfpmpp p ppfpppmpppfm fpfmfpmpfp mpmmfpfpmmfpmmpfm

mmpfmpfpm fmmppf

What plan?
Matt, gave an update, so I guess they're pushing off the release of the "enhanced" application a couple weeks.

See Matts' Post

which states:

Matt Lebofsky
Forum moderator
Project developer
Project scientist

Joined: Mar 1, 1999
Posts: 287
ID: 122079
Credit: 35,864
RAC: 25
Message 214503 - Posted 15 Dec 2005 5:09:00 UTC
Enhanced is still a couple weeks off. These are all regular ol' splitters.

- Matt
____________
-- BOINC/SETI@home network/web/science/development
-- Get the inside scoop: Matt's Unofficial SETI Log


So, presently, SETI is still broken? :-(

ID: 214742 · Report as offensive
TPR_Mojo
Volunteer tester

Send message
Joined: 18 Apr 00
Posts: 323
Credit: 7,001,052
RAC: 0
United Kingdom
Message 214802 - Posted: 15 Dec 2005, 15:47:47 UTC - in response to Message 214742.  
Last modified: 15 Dec 2005, 15:50:06 UTC


So, presently, SETI is still broken? :-(


There is no validation queue, upload queue, server issues, database issues, nada (apart from the usual top performance from overworked hardware, software and personnel). The only slight issue at the moment we have is a lower number of units to send out - but the large number of splitters online will sort that.

So HOW precisely is it broken?

But of course you already know all this, troll.

Join your buddy in my ignore list.

ID: 214802 · Report as offensive
Profile Paul D. Buck
Volunteer tester

Send message
Joined: 19 Jul 00
Posts: 3898
Credit: 1,158,042
RAC: 0
United States
Message 214808 - Posted: 15 Dec 2005, 15:56:04 UTC - in response to Message 214802.  


So, presently, SETI is still broken? :-(


There is no validation queue, upload queue, server issues, database issues, nada (apart from the usual top performance from overworked hardware, software and personnel). The only slight issue at the moment we have is a lower number of units to send out - but the large number of splitters online will sort that.

So HOW precisely is it broken?

But of course you already know all this, troll.

Join your buddy in my ignore list.

Here is a coincidence ... we agree again ...

How have you been? I had not seen you much until recently ...
ID: 214808 · Report as offensive
TPR_Mojo
Volunteer tester

Send message
Joined: 18 Apr 00
Posts: 323
Credit: 7,001,052
RAC: 0
United Kingdom
Message 214814 - Posted: 15 Dec 2005, 16:00:02 UTC - in response to Message 214808.  


Here is a coincidence ... we agree again ...

How have you been? I had not seen you much until recently ...


That must be about twice we have agreed then!
Off-topic (sorry Tony) but hello Paul :)

I have been busy busy busy I run an English Pub (bar/restaurant) and have the dubious honour of being chief cook. So this month is hmm... passably frantic for me 8-0

Still here, still Boincing, still unhappy about benchmarks, so nothing has changed :)

ID: 214814 · Report as offensive
Profile Dr. Bob
Avatar

Send message
Joined: 1 Apr 03
Posts: 78
Credit: 623,977
RAC: 0
United States
Message 222548 - Posted: 29 Dec 2005, 1:26:56 UTC - in response to Message 206378.  

[quote]Hi all, I'm about to layout the conversion plan the best I can piece together. Nothing I say here should be interpreted as coming from Berkeley. I'm seated comfortably in my office chair at my home in South Carolina. There the disclaimer is out of the way.

Thanks Tony,

Many of us had no idea of the plans and the ongoing application of them. Maybe I will look to add other BOINC projects to use my client computer time that may be excess now. Since as of this date, there may be reasons of enhanced efficiency that mean I or other client users won't get new data from SETI?

Thanks again...keep on keeping us updated.

Best,

Dr. Bob

Robert L. Hanson, Ed.D.
ID: 222548 · Report as offensive
Previous · 1 · 2

Message boards : Number crunching : The Plan


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.