Possible Workunit Distribution "Solution"


log in

Advanced search

Message boards : Number crunching : Possible Workunit Distribution "Solution"

1 · 2 · Next
Author Message
Profile Christopher Hauber
Avatar
Send message
Joined: 10 Feb 01
Posts: 196
Credit: 71,611
RAC: 0
United States
Message 11934 - Posted: 24 Jul 2004, 14:19:07 UTC

Perhaps in these early days of up and downs with workunit supply, and in the future when there is just plain more power than work, some kind of system should be employed to help better distribute workunits when they are prepared.

Every time UCB queues up a bunch of workunits, they are gone before some people are really able to get much from it because of the caching of workunits on PCs and so-called farms.

When I first started thinking about this a little bit ago, I thought in terms of maybe one unit every 10 minutes, but realized that wasn't very practical for some modem folks. So perhaps creating a maximum download of say 4 or 5 workunits every ten minutes would be a good compromise between dialup and broadband. All that would need to happen is for the server to send a maximum of 4 or 5 to a single computer, and then require BOINC to ask for more later. The 10 minute rule part of that already exists as minimum time between RPCs so the only change would be the amount sent at one time.

I think that might slow down the depletion of workunit supplies enough that more people could benefit from the supplies. It would still allow people to build up their caches, but perhaps add some "fairness" to it. It wouldn't prevent supplies from being exhausted, but it would slow it down a little.

Chris

Profile MikeProject donor
Volunteer tester
Avatar
Send message
Joined: 17 Feb 01
Posts: 24062
Credit: 33,034,167
RAC: 24,581
Germany
Message 11942 - Posted: 24 Jul 2004, 14:51:30 UTC
Last modified: 24 Jul 2004, 14:52:17 UTC

Hi

I dont think its an good idea, because this will increase the server load again.
The WUs load are instead 50 per day / host.
When the storm of download is over it will work nearly normal i think.
The splitter are creating 1000 WUs all 10 minutes as Rom said this should be enough.

greetz Mike

ColdRain
Send message
Joined: 28 Feb 02
Posts: 14
Credit: 1,092,701
RAC: 0
Belgium
Message 11947 - Posted: 24 Jul 2004, 15:10:12 UTC

It's not enough. At all. 1000 WU's every 10 minutes is 144 kWU per day. That's ONLY 1 day of work for about 20000 ppl. The current number of active users on S@H-1 is more then 450000. Imagine what would happen if S@H-2 had to cope with let's say only 10% of them ...

Pascal, K G
Volunteer tester
Avatar
Send message
Joined: 3 Apr 99
Posts: 2343
Credit: 150,491
RAC: 0
United States
Message 11948 - Posted: 24 Jul 2004, 15:10:47 UTC
Last modified: 24 Jul 2004, 15:12:14 UTC

Fair? this is not a game we are volunteers, if there is no work oh well....

Finite Work Supply

Bah!!! I am in it for the money.

Message from the Head ;o) of the
Horse Head Nebula Branch of Seti@Home,

Check is in the mail

Profile RandyC
Avatar
Send message
Joined: 20 Oct 99
Posts: 714
Credit: 1,704,345
RAC: 0
United States
Message 11950 - Posted: 24 Jul 2004, 15:18:41 UTC - in response to Message 11948.

> Fair? this is not a game we are volunteers, if there is no work oh well....
>
Exactly! I've devoted exactly ONE of my systems to BOINC. The others are still running Classic. When/if my BOINC system runs out of SETI WUs, I've got it setup so it works on a non-BOINC/non-classic distributed project (Smallpox research).

When the WUs became available yesterday, my system downloaded 1.5 to 3.0 days of work into my cache. If nothing's available when it's gone...well, that's life... deal with it.

Profile Christopher Hauber
Avatar
Send message
Joined: 10 Feb 01
Posts: 196
Credit: 71,611
RAC: 0
United States
Message 12213 - Posted: 25 Jul 2004, 5:01:08 UTC

Well I'm not sure that it would really increase server load since it wouldn't really be doing anything it doesn't do now. It already prevents RPCs less than 10 minutes apart. It would simply be the addition of an "if 5 sent, stop" type thing. If the client still wants more, it can ask again in 10 minutes. When the splitter goes down temporarily or something and workunits become unavailable, then are suddenly available again, there are a lot of clients that suddenly suck up the available stockpile making unavailable for lots of people trying to fill their cache. Perhaps "fair" wasn't the proper way to word it. There would be more benefit to it than that I believe. Not only would it give more computers a chance to get work (which wouldn't eliminate complaining, but may reduce it), you have more workunits being crunched simultaneously, thus making better use of the resources.

I also believe this would be useful when the time comes that not enough data is recorded to keep up. When that happens, when a new data tape is popped in, there will be a sudden supply of workunits that get sucked up by a few hungry computers. Like I said before, the more computers you have working, the faster you get the work done and make much better use of your resources. And hopefully reduce the amount of ill-will from the users. At least then they feel like they have a better chance of even being able to help.

You all should know how I stand with regards to BOINC and how things have been going so far. I've always been supportive of how things have been going and understanding of things. I've always tried to use sound reasoning and use common sense when posting. You are right Pascal, if there is no work, then oh well. That has never been a complaint of mine. This is for when work becomes available. It's is just an idea for possibly making better allocation of the resources available to the project while possibly alleviating some of the discontent from users -- and with little to no increase in server load. I realize nothing will satisfy some people and people will always complain. I realize that as we volunteer our computers, fairness isn't really a priority. I've dont always EXPECT to be able to obtain work, nor for everyone to obtain work. But I do believe it should be easier for large numbers of computers to get small amounts of work than for smaller numbers computers to get large amounts of work.

If you still don't think it's a good idea, then fine. But since I didn't really have much time to post earlier when it first occured to me and so much disapproval, I thought maybe I should clarify and explain my reasoning on this. It may NOT be a good idea, but as far as I can tell right now, I think it makes sense.

Chris

Profile [B^S] Zain Upton
Volunteer tester
Avatar
Send message
Joined: 16 Feb 01
Posts: 132
Credit: 43,763
RAC: 0
Australia
Message 12217 - Posted: 25 Jul 2004, 5:25:47 UTC - in response to Message 12213.

Chris,

Well I dont know about others, but I think your posts are ALWAYS well thought out, informative, and generally right on the money. I wouldnt be listening to the whiners that will always appear in global projects like this.

My hat is off to you Chris *tip*

---
<p align="center"> </p>Click on the counter to visit our team site including Extended Team Stats with daily totals, Member signatures and more!

Profile Toby
Volunteer tester
Avatar
Send message
Joined: 26 Oct 00
Posts: 1005
Credit: 5,622,795
RAC: 0
United States
Message 12256 - Posted: 25 Jul 2004, 8:06:11 UTC

I'm not sure but I think something like this might already be in place. When I was first able to download work units after the latest outage, I only got 3 at first. Then a little while later I got 3 or 4 more. Eventually my queue was filled up to my 3 day queue depth but it didn't all come in at once. Not sure if that was just a fluke or if it is built into the system. Anyone else notice one way or the other?



-------------------------------------------
- A member of The Knights Who Say NI!
Possibly the best stats site in the universe:
http://boinc-kwsn.no-ip.info

Profile [B^S] Zain Upton
Volunteer tester
Avatar
Send message
Joined: 16 Feb 01
Posts: 132
Credit: 43,763
RAC: 0
Australia
Message 12260 - Posted: 25 Jul 2004, 8:09:36 UTC - in response to Message 12256.

Reports from our team members (SETI Synergy) are highly variable.

Some members did just as you did, got a few work units, then a few more, and have had some to crunch ever since. Other members still have not recieved one work unit... and are getting frustrated even more hearing of these people who are crunching seemingly randomly.

But thats how it goes. You have to trust the dev have a master plan.
<p align="center"> </p>Click on the counter to visit our team site including Extended Team Stats with daily totals, Member signatures and more!

Profile Keck_Komputers
Volunteer tester
Avatar
Send message
Joined: 4 Jul 99
Posts: 1575
Credit: 1,716,533
RAC: 712
United States
Message 12279 - Posted: 25 Jul 2004, 9:10:08 UTC

I don't know how this would effect the back-end. I would expect that when there are multiple projects running this type of idea would be helpfull, because then when the client is doing its backoff from one project it would likely get work from another one. In the current situation with only a few active projects I don't think it would be much help. I would expect this idea to help ease user frustration some since you would be likely to get a workunit in addition to the backoff message.

The way I would work it would be dynamic. If there are few workunits or the server is heavily loaded then give only a few workunits at a time and request a 1 to 3 hour backoff. The exact number of workunits and the length of the backoff would vary based on how heavy the load is and how many workunits are available.

John Keck
BOINCing since 2002/12/08

Scott Brown
Send message
Joined: 5 Sep 00
Posts: 110
Credit: 59,739
RAC: 0
United States
Message 12351 - Posted: 25 Jul 2004, 17:30:42 UTC

Chris' idea is also sound scientifically. It is in the interest of the project to have a wide variety of machines processing units to increase the validity of the results.

Of course, we are overlooking an obvious solution. Instead of only three hosts per work unit, why not increase the number of host to four or five. While we certainly don't want to crunch the same data over and over again as is happening with classic now, verification by 4 or 5 hosts is not absurd and would help alleviate work unit supply/demand.

Scott

Profile Jaaku
Volunteer tester
Avatar
Send message
Joined: 29 Oct 02
Posts: 494
Credit: 346,224
RAC: 0
United Kingdom
Message 12353 - Posted: 25 Jul 2004, 17:34:28 UTC

Sure 5 sounds good, but so does 15 what is the change of getting the same WU and plus you could compare them and delete the stray results. But any way if SETI was real intrested they would go to more dishes and collect there data even if they dont do it them selves but just copy what it looks at!

^ Click Here ^ ^ Click Here ^ ^ Click Here ^

Profile KWSN - MajorKong
Volunteer tester
Avatar
Send message
Joined: 5 Jan 00
Posts: 1426
Credit: 834,224
RAC: 1,865
United States
Message 12386 - Posted: 25 Jul 2004, 18:45:44 UTC - in response to Message 12351.

> Of course, we are overlooking an obvious solution. Instead of only three
> hosts per work unit, why not increase the number of host to four or five.
> While we certainly don't want to crunch the same data over and over again as
> is happening with classic now, verification by 4 or 5 hosts is not absurd and
> would help alleviate work unit supply/demand.
>
> Scott
>
>

That is the EXACT reason that Berkeley has set the number to 3. They don't want BOINC/S@H to be doing busy-work like S@H-Classic is. The BOINC framework is designed to be multi-project, and it is expected that one runs multiple projects to cover 'down time'. If one is ONLY interested in running S@H, then one MUST accept being idle during times of 'no work available'.


------------
KWSN-MajorKong
KWSN Forum Admin (retired)
http://www.kwsnforum.com

BOINC Beta tester

Pascal, K G
Volunteer tester
Avatar
Send message
Joined: 3 Apr 99
Posts: 2343
Credit: 150,491
RAC: 0
United States
Message 12388 - Posted: 25 Jul 2004, 18:53:35 UTC - in response to Message 11950.
Last modified: 25 Jul 2004, 18:55:45 UTC

> > Fair? this is not a game we are volunteers, if there is no work oh
> well....
> >
> Exactly! I've devoted exactly ONE of my systems to BOINC. The others are
> still running Classic. When/if my BOINC system runs out of SETI WUs, I've got
> it setup so it works on a non-BOINC/non-classic distributed project (Smallpox
> research).
>
> When the WUs became available yesterday, my system downloaded 1.5 to 3.0 days
> of work into my cache. If nothing's available when it's gone...well, that's
> life... deal with it.
>
>

I like you Randy, you understand the system...Seti@Home is not here to keep us busy they are here to find ET.....

If I find ET, I hope I do not get stuck with the long distance charges, ;o)
Bah!!! I am in it for the money.

Message from the Head ;o) of the
Horse Head Nebula Branch of Seti@Home,

Check is in the mail

Scott Brown
Send message
Joined: 5 Sep 00
Posts: 110
Credit: 59,739
RAC: 0
United States
Message 12410 - Posted: 26 Jul 2004, 17:10:01 UTC

When I suggested verification by more than 3 hosts per work unit, it was not to advocate "busy-work". But that does beg the question of what is "busy-work". I am afraid that I don't agree that going to 4 hosts (or even 5) per unit is fundamentally different from the arbitrarily set 3 hosts/unit. My point was that by simply raising this bar by a minimal number, some of the work demand could be resolved.

Perhaps 4 hosts might even provide a more efficient manner of crediting for users (not that I am very concerned with receiving credit myself, but it is clearly a major motivator for many). For example, maintain the 3 hosts verification requirement, but send the unit to 4 hosts. The first 3 to return the unit determine the credit received--the fourth hosts merely receives this credit. In cases where one hosts does not return/returns a corrupted unit, the fourth host's results count in determining the credit and such credit is then allocated more quickly. Thus, some portion of work demand is resolved; results are returned more quickly to Berkeley (i.e., without reissuing a unit, etc.); and users concerned with credit are happier since their credits will be posted more quickly.

Scott

Ingleside
Volunteer developer
Send message
Joined: 4 Feb 03
Posts: 1546
Credit: 4,262,061
RAC: 7,287
Norway
Message 12418 - Posted: 26 Jul 2004, 17:54:51 UTC - in response to Message 12410.

AFAIK they was planning on using 2 for verification & distributing 3, but due to all the cry-babies in beta complaining about always being paired off by a debug-free *nix-box or someone with outdated benchmark-numbers, forgetting all beta-stats should be zeroed, they increased the required to verification to 3.

As for upping the distribution to example 5, it's not certain this will help at all since the limiting step currently is slow database-access, and adding 999 results from 333 wu is maybe not slower than adding 1000 results from 200 wu...

But something that can be done is either use a debug-version of seti, or change some of the wu-limits so more science is done... Both will lead to slower crunching, and therefore less wu needs to be distributed.

Profile Rom Walton (BOINC)
Volunteer tester
Avatar
Send message
Joined: 28 Apr 00
Posts: 569
Credit: 109,267
RAC: 219
United States
Message 12426 - Posted: 26 Jul 2004, 18:35:41 UTC

Ingleside is correct, the database currentlly needs a 3Mbit write pipe to the disks and we are not achiving that with the current disk subsystem.

The new disk subsystem is supposed to have a 10Mbit write pipe capability.

Just increasing the number of results means you pay for the writes at the time somebody attemts to return the results as each result/workunit has to be updated to notifiy the transitioner that it needs to be scheduled for validation or more result creation.

----- Rom
BOINC Development Team, U.C. Berkeley

Profile Byron Leigh Hatch @ team Carl SaganProject donor
Volunteer tester
Avatar
Send message
Joined: 5 Jul 99
Posts: 3619
Credit: 11,870,924
RAC: 1,115
Canada
Message 12430 - Posted: 26 Jul 2004, 18:44:46 UTC - in response to Message 12426.
Last modified: 5 Sep 2004, 5:01:19 UTC

Profile DaMaCon
Send message
Joined: 23 May 03
Posts: 5
Credit: 189,415
RAC: 0
United States
Message 12432 - Posted: 26 Jul 2004, 18:46:19 UTC

Neophyte question here (in fact, two)

1) I have received and processed a number of WUs, but under my Project tab I see 0 in Total and Average Credit. I've recently converted from the old seti to boinc, and have followed some of the discussion about the troubles of the new system. Should I expect to see some credit, or do I have to do something else?

2) The rather neat sigs show interesting stats - are there docs that reveal how?

Thanks.
<a> [/url]

Scott Brown
Send message
Joined: 5 Sep 00
Posts: 110
Credit: 59,739
RAC: 0
United States
Message 12433 - Posted: 26 Jul 2004, 18:46:27 UTC - in response to Message 12426.

> Ingleside is correct, the database currentlly needs a 3Mbit write pipe to the
> disks and we are not achiving that with the current disk subsystem.
>
> The new disk subsystem is supposed to have a 10Mbit write pipe capability.
>
> Just increasing the number of results means you pay for the writes at the time
> somebody attemts to return the results as each result/workunit has to be
> updated to notifiy the transitioner that it needs to be scheduled for
> validation or more result creation.
>
> ----- Rom
> BOINC Development Team, U.C. Berkeley
>
>

Rom and Ingleside,

Point well taken. Thanks for the info!

Scott

1 · 2 · Next

Message boards : Number crunching : Possible Workunit Distribution "Solution"

Copyright © 2014 University of California