Possible server storage optimization? (long post)

Message boards : Number crunching : Possible server storage optimization? (long post)
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
NedKelly
Volunteer tester

Send message
Joined: 22 May 99
Posts: 3
Credit: 10,124,064
RAC: 7
United States
Message 804621 - Posted: 3 Sep 2008, 21:02:16 UTC

Some thoughts on modeling the life of a WU and SETI optimizations.

Most everyone with SETI clients know that the servers have evolved to delicately balance the workflow within the limits of available bandwidth, storage capacity, server loads etc. Over time many improvements have been made, but there is also a fair amount of legacy baggage out there. For example, the original WU was sized to the capabilities of servers and clients from an earlier stone age. Modern multi-core hyperthreading CPUs are pounding through the WU’s and the server demands are constantly challenged. Recovery from outages can be lengthy as the various queues sort themselves out.

So let me tease folks with some SETI considerations. First the general life history of a WU and some of the parameters I have considered in a crude Monte Carlo model:

1) Splitters create the WU.
2) The new WU is sent to 3 clients. The delay between 1) and 2) is likely very small.
3) Each client adds the new WU to a list of WU’s in a local buffer. This buffer is the client’s protection against server outages and is manually adjustable. In my modeling I accounted for a client queue but did not try any optimizations.
4) After the client queue delay, the WU is processed. With a fast CPU only running SETI 24/7, this could be a very short time. With slower CPU’s, client sharing with other BOINC projects and less than 24/7 availability, this could be a much longer delay
5) The three results are returned from the three clients. When each result coming back, there are bandwidth and storage demands on the server.
6) When all results are back, the validator can do its thing, extracting the science and marking the WU for deletion.

There are more details than this simple flow to account for clients that never return results etc. but for now I have relegated these special cases to second order effects.

The key interactions I drilled in on are the potential big differences between clients when CPU power and buffer queues are considered. To me, this has opportunity written all over it. Today when I looked there were 3.5 M results out there and 2.6 M results waiting for validation. As I understand it the splitter rate is often throttled to ensure there is enough space for everything, with the penalty being the ready to send queue can be very short (0.0008 M results today). This is a bit of a vicious circle since the SETI servers cannot buffer a big pool of results to send, volunteers increase the queue length and the validation percentage increases since the average return time for clients is now dominated by the queue length.

I tried to model the process to see what might be possible. However, I don’t really have any true data so everything I considered was a WAG. As such, I may be very wrong with what I did so I only want to talk in generalities with what is done today and what might be possible. I am hoping folks with a better understanding of the process and access to better data can jump in and see if there are and significant optimization opportunities within SETI.

Clearly if clients shorten the queue length, results will return faster (more bandwidth, but less results storage waiting for validation/purging). The risk is starvation during outages. This can be an emotional topic as volunteers strive for higher scores. Since there is no results “reward” for minimizing total WU time at a client (rewards are only for CPU cycles per the cobblestone score), getting the volunteers to shorten anything is not likely. However some hard rules on maximum queue lengths might work. Perhaps someone could look at buffer queue sensitivities and postulate an optimum maximum queue length, given the historical availability of the SETI servers.

I used my crude Monte Carlo model to postulate what might happen with a process that allocates WU’s to clients with like capabilities. The thought is that we want to avoid having a fast processor return a result early and the server having to carry the result until a slow CPU is finally done before the validation/purge phase occur. By binning a random set of clients into three pools based on benchmarks only, result storage can be reduced by 20-30%. I know benchmarks are known to the server. I would guess the queue length could be determined as well. So now a fast CPU with a big queue could be pushed into the middle pool to better match total WU client times and minimize the results storage on the server. And for another degree of refinement, the client’s utilization factor could be considered. For example a fast processor that only runs SETI at night should also be pushed into a slower pool with a corresponding server storage improvement.

All that needs to happen is that the server issuing WU’s to clients consider the aggregate capability of the client to best match the first client with the next two who will also process the WU. Benchmark scores would be enough to realize some reduction in server result storage. Additionally considering client utilization (% time in a week available for SETI) and buffer queue length to refine the match would further reduce the result storage demands on the server. There may be other intelligence that could help (client historical averages, more than three speed pools etc.).

From the client side there will be little noticeable effect. Fast clients should see fewer results waiting for credit. Some extra data needs to be associated with WU’s on the server once a WU is sent out to try and best match the next two clients. Perhaps not – it could be simply that splitter 1 be reserved for client pool 1, splitter 2 for pool 2 etc. Now we are into details and I am way out of my depth.

Of course any change that is driven by these optimization considerations needs to be balanced by the potential risks of making the change. And I realize that the SETI support resources are very limited. However with a number of folks out there optimizing clients, I am sure there are some who might get some real data and help out with some analysis of the possibilities. Only when the risk/reward from any analysis shows a meaningful return would any changes be made.

Wow – I got a little carried away. My few paragraphs draft has ballooned out to several pages. So let me get to the bottom line:
A process model and some Monte Carlo sensitivity studies suggest a possible server storage improvement based on “matching” the clients that process a given WU.

It would be nice if some other crunchers could independently support this assertion (ideally with real data).

Thanks


ID: 804621 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 804623 - Posted: 3 Sep 2008, 21:11:45 UTC - in response to Message 804621.  

All that needs to happen is that the server issuing WU’s to clients consider the aggregate capability of the client to best match the first client with the next two who will also process the WU. Benchmark scores would be enough to realize some reduction in server result storage.

Actually, I think there might be one metric that'd handle all of this nicely.

For each host, BOINC knows the average turn-around time.

If you give one result to a machine with an average turn around of 1 day, and the other result in the pair to a machine with a 9 day turn around, you'll have one "unpaired" work unit on the server for 8 days.

If you give both results to be crunched to machines with a four-day turnaround, they'll both come back at about the same time.

That'd work even if one was a multi-core monster with a big cache, and the other was slow antique.

The big question is: which costs more? Carrying more intermediate results, or trying to sort work by turn-around time?
ID: 804623 · Report as offensive
PhonAcq

Send message
Joined: 14 Apr 01
Posts: 1656
Credit: 30,658,217
RAC: 1
United States
Message 804647 - Posted: 3 Sep 2008, 22:45:55 UTC

Is it possible for seti to re-split a 'tape', perhaps selectively. This seems like a 'fast' process. If so, then why not stop saving the split wu's on the server once they are distributed. If one wu does not return, add that wu to a list to resplit at a later date. This way one trades a large amount of static storage for some book keeping and some redundant splitting later. Because the data is not on tape anymore, this actually may be feasible.
ID: 804647 · Report as offensive
Profile RandyC
Avatar

Send message
Joined: 20 Oct 99
Posts: 714
Credit: 1,704,345
RAC: 0
United States
Message 804686 - Posted: 4 Sep 2008, 1:35:38 UTC - in response to Message 804647.  
Last modified: 4 Sep 2008, 1:36:05 UTC

Is it possible for seti to re-split a 'tape', perhaps selectively. This seems like a 'fast' process. If so, then why not stop saving the split wu's on the server once they are distributed. If one wu does not return, add that wu to a list to resplit at a later date. This way one trades a large amount of static storage for some book keeping and some redundant splitting later. Because the data is not on tape anymore, this actually may be feasible.


What? And Matt has nothing better to do than scramble around finding the right drive to remount and re-split?

Besides, I think once the data has been offloaded from the drives to be split, the drive is sent back for more data.

[edit]typos[/edit]
ID: 804686 · Report as offensive
Profile Geek@Play
Volunteer tester
Avatar

Send message
Joined: 31 Jul 01
Posts: 2467
Credit: 86,146,931
RAC: 0
United States
Message 804695 - Posted: 4 Sep 2008, 1:52:14 UTC

I believe Matt addressed this problem just today in this post.

So unless we have a wildly successful fund raising drive it appears we will have to live with these current problems.
Boinc....Boinc....Boinc....Boinc....
ID: 804695 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 804722 - Posted: 4 Sep 2008, 3:10:58 UTC - in response to Message 804686.  

Is it possible for seti to re-split a 'tape', perhaps selectively. This seems like a 'fast' process. If so, then why not stop saving the split wu's on the server once they are distributed. If one wu does not return, add that wu to a list to resplit at a later date. This way one trades a large amount of static storage for some book keeping and some redundant splitting later. Because the data is not on tape anymore, this actually may be feasible.


What? And Matt has nothing better to do than scramble around finding the right drive to remount and re-split?

Besides, I think once the data has been offloaded from the drives to be split, the drive is sent back for more data.

[edit]typos[/edit]

"Tape" is a metaphor.

I know that there would be some small difference between keeping the unsplit "tape" and resplitting it, and keeping the split data, but I suspect that it's about the same amount of disk space either way.

This is one of those places where it is easy to move a problem around, and hard to fix.
ID: 804722 · Report as offensive
Terror Australis
Volunteer tester

Send message
Joined: 14 Feb 04
Posts: 1817
Credit: 262,693,308
RAC: 44
Australia
Message 804757 - Posted: 4 Sep 2008, 5:53:41 UTC

What would also help would be if there was some way not to issue "shorties" to the faster clients. A fast cruncher can knock one of these over in < 20 minutes, < 10 in some cases and thus is continually pounding the server after more work.

This would definitely help with bandwidth and unit availability issues, particularly after an outage. I'm also in favour of a limit on the number of WU's that can be downloaded at any one time.

Brodo
ID: 804757 · Report as offensive
Ingleside
Volunteer developer

Send message
Joined: 4 Feb 03
Posts: 1546
Credit: 15,832,022
RAC: 13
Norway
Message 804788 - Posted: 4 Sep 2008, 9:21:17 UTC - in response to Message 804647.  

Is it possible for seti to re-split a 'tape', perhaps selectively. This seems like a 'fast' process. If so, then why not stop saving the split wu's on the server once they are distributed. If one wu does not return, add that wu to a list to resplit at a later date. This way one trades a large amount of static storage for some book keeping and some redundant splitting later. Because the data is not on tape anymore, this actually may be feasible.

Well, if you're going (crapp keyboard, expect lots o missing caracters) to re-split, you must eiter:
1; Keep te disks at the lab or longer, bt, tese is neede to be sent-back, so more work can be recored at arecibo...
2; Keep te unsplit ino on isk... But, if a few wu's takes lots of space, keeping 50 GB to be re-split will be even worse. Tis wol maybe st move te "full disk" from wu-storage to "tape", so all "tapes" is already split, bt a few wu's must be re-split, meaning no work at all until can delete a "tape" and ready a new...
3; Downloa from their online storage... Just my guess, but you'll maybe need to download wole 50 GB for re-splitting. And, even on a fast connection, downloading 50 GB will take some time...

Also, not sre, bt tere's been some mention tat te radar-blanking is only on one o te cannels or someting, so you'll likely need 100 GB to re-split...


Appart for te problem of actally re-splitting, anoter problem is, at tat time is the wu-file actally downloaded? Just being assigned work isn't enog, it will take some time to actally download te wu. A non-permanent dialup-user can easily use 1 hour to download enoug work. In case tere's a serer-problem (or oerloaded internet-connection) even permanently-connected sers can use over a day to download a wu...

Now, no idea how detailed logging Apace as, bt if partial downloads is conted, yo can't go b #downloaded. Neiter can btes transerred be sed, since BOINC-client will, to work-arond problems wit proxy-servers spitting-ot error-message, delete te last 2k of download in case not finised.

So, at tat point is the wu "safe" to delete?

And, of corse, how often will users, a couple mintes ater wu deleted, report a comptation-errors or someting, so wu needs to be re-split?


Probably more likely to sccees is the sggestion to pair users wit similar turnaround-times, so bot reslts of wu is returned rogl te same time. But, you can't just look on "Average turnarond time", since a slow computer using 7 days on Astropulse-wu and a fast computer sitting wit a 7-days-cace of seti-work will both ave 7-day turnaround, but if assigned seti-wu, te slow computer will likel return witin 6 hours, wile te fast will use 7 days. With Astropulse, the diference will be smaller...

"I make so many mistakes. But then just think of all the mistakes I don't make, although I might."
ID: 804788 · Report as offensive
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 20619
Credit: 7,508,002
RAC: 20
United Kingdom
Message 804822 - Posted: 4 Sep 2008, 11:45:26 UTC - in response to Message 804788.  
Last modified: 4 Sep 2008, 11:46:19 UTC

... Probably more likely to sccees is the sggestion to pair users wit similar turnaround-times, so bot reslts of wu is returned rogl te same time. ...

This has been discussed/proposed elsewhere. I think that is a very good idea.

I also like the idea of the queue analysis for the s@h dataflow. Anyone got a complete diagram and some numbers (bandwidth, CPU processing performance for servers, and for clients) so that we can play the game of chase the queue?

One easy/obvious note is that clients that keep a large local cache delay their turn-around time and so tie up additional resource on the servers proportional to that cache size. That is: A big local cache slows down the servers and the credits.


And credits is a big driver... The present credits system is imperfect but blunders along and is a big motivator. However, there is no reward for being reliable or for being prompt, nor for bandwidth and storage used.

Should we include negative scoring for dumped or delayed results? Can we add a bonus for prompt return and for reliability?

Note that at the moment, there is every incentive to just max out your cache and freely dump WUs if you can't quite clear it in time... Rather wasteful with extra strain on the servers and delayed credits.

There is also an incentive for various 'silliness' to 'adjust' the credits and forge your RAC...

Further thoughts?

(Seriously, the queue modeling for the s@h system should make a very good and valuable Computer Science project. It will also give a feast of graphs for the stats freaks! Anyone remember Roving Mouse?...)

Happy crunchin',
Martin
See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 804822 · Report as offensive
Ingleside
Volunteer developer

Send message
Joined: 4 Feb 03
Posts: 1546
Credit: 15,832,022
RAC: 13
Norway
Message 804862 - Posted: 4 Sep 2008, 13:55:34 UTC - in response to Message 804822.  

One easy/obvious note is that clients that keep a large local cache delay their turn-around time and so tie up additional resource on the servers proportional to that cache size. That is: A big local cache slows down the servers and the credits.


And credits is a big driver... The present credits system is imperfect but blunders along and is a big motivator. However, there is no reward for being reliable or for being prompt, nor for bandwidth and storage used.

Should we include negative scoring for dumped or delayed results? Can we add a bonus for prompt return and for reliability?

Note that at the moment, there is every incentive to just max out your cache and freely dump WUs if you can't quite clear it in time... Rather wasteful with extra strain on the servers and delayed credits.

There is also an incentive for various 'silliness' to 'adjust' the credits and forge your RAC...

Further thoughts?

(Seriously, the queue modeling for the s@h system should make a very good and valuable Computer Science project. It will also give a feast of graphs for the stats freaks! Anyone remember Roving Mouse?...)

Happy crunchin',
Martin

Actually, atleast on Intels, te "best-paying" is VHAR wit 7-day deadline, but if you max-out our cace, you won't get them...

Now, the "easiest" throttle is to just do a one-line edit of the server-conig, and set "max_wu_in_progress" = 2, and users can't download a large cace of work. It's adjusted per cpu, but not sre if the max is 8 cpu's...

The problem with this setting is dialup-users tat genuinely needs a cace o work, so maybe 2 is a little too low. It's not everyone that can run Astropulse, so maybe 8 is more in line... With 8 wu's, "slow" computers using more tan 3 ours/wu will ave a 24-our-cace, wile aster computers tat needs a cace can run Astroplse.

Another metod that can be used is to seriously cut te deadlines. If example VHAR is cut to 3 das and "normal" wu's to 7 days, users can't set a large cace. Astropulse should probably be set at 14 days, and fpops_est doubled so less chance for users getting assigned Astropulse-wu that can't finis b deadline.


Negative scoring on te oter and wouldn't be suc a good idea, since users witout any ault o their own is assigned "gost"-wu and so on... A bonus or fast turnaround is not such a good idea eiter, due to dialup-users that will be penalized.

Now, some more "exotic" features, tat would need programming, would be someting like:
If cace-size > 1.5 days => only send Astroplse-work if already cached-work > 1.5 days.

Tis would give slow computers te opportnity to ave a 1.5 days cace, while fast computers will be assigned Astropulse-work. If they as disabled Astroplse, they'll can't get more tan 1.5 days wort...

Anoter wold be someting like:
If cpu-speed * normal adustments for up-time and so on:
Case of can finish Astropulse-wu in 1 days or less => send only Astropulse-wu.
In case doesn't accept Astropulse-wu, limit to 1 wu in progress per cpu.
If finish Astropulsewu in 1-2 days => send only Astroplse-wu.
Limit to 2 wu/progress if doesn't accept astropulse.
If Astropulse 2-4 das => preverable send Astropulse.
Limit to 4 wu/progress.
If 4-7 das => send whatever, limit to 8 wu/progress.
If > 7 days => send seti@home, except if prefers Astropulse.
No limit of #wu/progress. (afterall, computer is so slow so...)


"I make so many mistakes. But then just think of all the mistakes I don't make, although I might."
ID: 804862 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 804884 - Posted: 4 Sep 2008, 16:16:06 UTC - in response to Message 804862.  


Tis would give slow computers te opportnity to ave a 1.5 days cace, while fast computers will be assigned Astropulse-work. If they as disabled Astroplse, they'll can't get more tan 1.5 days wort...

We need to pitch in and get Ingleside a new keyboard. I hope he isn't writing code on that one.

This is (as so many things are) a cost/benefit argument.

We know it is possible to code so that SETI uses the disk resource more effectively.

The question is: which is more cost-effective? Using CPU time, databases, and the like to assign WUs to "like" machines, or expanding storage so it is not an issue.

ID: 804884 · Report as offensive
Keith White
Avatar

Send message
Joined: 29 May 99
Posts: 392
Credit: 13,035,233
RAC: 22
United States
Message 804886 - Posted: 4 Sep 2008, 16:22:37 UTC

The limiting factor in workunit return rate is not the speed of the cruncher but the length of the client queues.

In a perfect world, when I set my queue to 7 days worth of workunits to process and my system is performing normally, no unexpected downtimes, no periods where I turn Seti off for extended periods of time, no decrease in CPU cycles thrown Seti's way due to user activity, it should take about 7 days to complete a workunit.

Unless of course the server starts to generate bucket fulls of workunits with short deadlines. Due to their short deadline and my queue size, they get to jump to the head of the line for processing. Which means the workunits with the usual 21+ day deadlines get to cool their heels.

In the last 11 days my slow poke of a system only fetched 6 workunits (instead of about 15) which have 21+ day deadlines. The rest of the time it has fetched workunits with 7 day deadlines. Lots of workunits with 7 day deadlines (around 20). All which get processed first.

Now assuming this last batch are the last when I connect tomorrow, the oldest workunit in my queue will now take over 12 days to complete instead of the normal 7. Now on my little slow poke of a crucher, tying up 6 workunits by an extra 5 days is trivial in the big picture but what about the super crunchers with the same 7 day queue size, or worse? The only thing that might save them is the creation cap denying them new workunits allowing them to process those already in their queue.

Now I propose a different "fix". They should cap the creation of these short deadline workunits. Looking at the status page we are running with 9 splitter applications splitting 30+ data files into workunits. Now they duty cycle the splitters once they get around 200K workunits ready to send, just have a similar count of these short deadline workunits and once you hit a limit, swap to another data file. All 30+ can't possibly have the same angle range that create these short deadline workunits.

The only other way I can see them fixing this is to limit our queue size to less than the minimum deadline a workunit can have.
"Life is just nature's way of keeping meat fresh." - The Doctor
ID: 804886 · Report as offensive
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 20619
Credit: 7,508,002
RAC: 20
United Kingdom
Message 804892 - Posted: 4 Sep 2008, 16:46:46 UTC - in response to Message 804884.  
Last modified: 4 Sep 2008, 16:48:02 UTC

Tis would give slow computers te opportnity to ave a 1.5 days cace

We need to pitch in and get Ingleside a new keyboard. I hope he isn't writing code on that one.

This is (as so many things are) a cost/benefit argument.

Looks like sugary-spilt-beverage-onto-the-central-area causing sticky-key syndrome.

Or he's trying to type Olde English!

So is a $6 donation worth reducing our brain-ache decoding the lost characters?!

THis would give slow computers tHe HopportUnity to Have a 1.5 days caHce

Or should that be:

'Tis [noblest that] would give slow computers t' opport'nity to 'ave a 1.5 days cace

Mmmmm... Ok, so that's more of a mix of Shakespear, Yorkshire, and General Vulgar...! :-p


Keep searchin',
Martin
See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 804892 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 804929 - Posted: 4 Sep 2008, 19:08:58 UTC - in response to Message 804892.  


So is a $6 donation worth reducing our brain-ache decoding the lost characters?!

$6? You need to buy better keyboards.

ID: 804929 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 804933 - Posted: 4 Sep 2008, 19:11:35 UTC - in response to Message 804886.  
Last modified: 4 Sep 2008, 19:26:57 UTC


Now I propose a different "fix". They should cap the creation of these short deadline workunits. Looking at the status page we are running with 9 splitter applications splitting 30+ data files into workunits. Now they duty cycle the splitters once they get around 200K workunits ready to send, just have a similar count of these short deadline workunits and once you hit a limit, swap to another data file. All 30+ can't possibly have the same angle range that create these short deadline workunits.

Actually, they could.

Remember that some other organization is controlling the telescope, and they set the motion they want for their study. People have figured out which "active" projects generate different angle-ranges.

... so if one project gets a big block of telescope time, the angle range gets very predictable.
ID: 804933 · Report as offensive
Ingleside
Volunteer developer

Send message
Joined: 4 Feb 03
Posts: 1546
Credit: 15,832,022
RAC: 13
Norway
Message 805012 - Posted: 4 Sep 2008, 21:58:12 UTC - in response to Message 804892.  

Tis would give slow computers te opportnity to ave a 1.5 days cace

We need to pitch in and get Ingleside a new keyboard. I hope he isn't writing code on that one.

This is (as so many things are) a cost/benefit argument.

Looks like sugary-spilt-beverage-onto-the-central-area causing sticky-key syndrome.

Or he's trying to type Olde English!

So is a $6 donation worth reducing our brain-ache decoding the lost characters?!

Thankfully the earlier posts wasn't written on my own keyboard, but in a computer-lab... It's possible could have found a better keyboard, but, after 4 dead computers or "couldn't type full username/password", stopped trying then managed to log-in, since didn't expect much typing...


"I make so many mistakes. But then just think of all the mistakes I don't make, although I might."
ID: 805012 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 805020 - Posted: 4 Sep 2008, 22:18:20 UTC - in response to Message 804933.  

...
All 30+ can't possibly have the same angle range that create these short deadline workunits.

Actually, they could.

Remember that some other organization is controlling the telescope, and they set the motion they want for their study. People have figured out which "active" projects generate different angle-ranges.

... so if one project gets a big block of telescope time, the angle range gets very predictable.

There are of course only 9 mb_splitter processes at one time even if the status page sometimes shows more than that many channels active. The other 20 or so 'tapes' shown on the status page are data which is available to split when one of the active 'tapes' has been completed.

Even if a project which uses high rate scanning only gets 2 or 3 hours a day, if it is the only project using ALFA that day then all the data can be high rate. And the scheduling of Arecibo tends to be fairly consistent over periods of a week or two. I think that's probably because many of the observing projects want to have someone on site in Puerto Rico while the observations are being done, or similar reasons. The result can easily be like the June 2 through 20 period, when only the GALFACTS project had observing time using ALFA. They use the basketweave scanning technique which produces VHAR data and they've only begun what is supposed to be a full survey of the sky visible from Arecibo with about 2000 hours of observing time.

=====================

I don't think there's any possibility of resplitting WUs as needed. The splitter input for a group is 256*1024*1024 data points at the full 2.5 MHz. bandwidth (about 128 MiB since it is 2 bit coded), plus the radar channel which might be half that size if it is just an on/off single bit. After replacing any radar-affected data, it FFTs the data into 2048 subbands, and reverse FFTs to recombine sets of 8 of those. That produces the 256 WUs in a group, a special splitter trying to make only one WU would still have to do all the cleaning and FFTs but only set of 8 reverse FFTs. When producing full groups a splitter seems able to average maybe 3 WUs per second, so the full group takes over 85 seconds. The special splitter might take 70 seconds to make 1 WU.

The Scarecrow graphs for the last 90 days give an indication of how many WUs had to be reissued; the minimum over that period was 2.979 percent (about 7 per group of 256), the average slightly under 6% (about 15 per group of 256). Going back and resplitting even 7 times would simply be too costly even if the special splitter could be made a lot faster than I expect.

However, once a WU has been downloaded by both of the original 2 hosts, it could probably be put into a supplemental slower storage system. That would require both having such a storage system, and modifying the Transitioner such that when it creates a reissue result it first recalls the WU from that storage. There would be other parts of the BOINC server code which would have to be modified, for instance the file deleter would have to be able to work on both the normal and supplementary stores. I suspect this idea isn't practical, but thought I'd make a contribution to the speculation on possible improvements.

The 'tapes' are of course preserved. Some are kept locally for a time, but all are saved at LBNL HPSS. Even after crunching both as setiathome_enhanced and astropulse, the possible data has not all been extracted. And certainly if we find what appears to be an ET signal, other scientists will want to examine the original data.
                                                           Joe
ID: 805020 · Report as offensive
Ingleside
Volunteer developer

Send message
Joined: 4 Feb 03
Posts: 1546
Credit: 15,832,022
RAC: 13
Norway
Message 805055 - Posted: 4 Sep 2008, 23:04:57 UTC - in response to Message 805020.  

However, once a WU has been downloaded by both of the original 2 hosts, it could probably be put into a supplemental slower storage system.

How do you know both has successfully downloaded the wu?

Not sure, but atleast from the look of things, BOINC only accepts one download-directory, but can "mirror" to multiple download-servers, like Einstein@home is doing. It doesn't seem like BOINC has any option to example let SETI@home-wu's reside on download-server-1 while Astropulse-wu's resides on download-server-2. Meaning, if you'll need more disk-space, you'll either need to add more disks to existing server (if possible), or, start with a completely new server...

Now, no idea how linux is on this part, but, download-directory does have 1024 sub-directories, so, it's maybe possible to "link" from another server, and by this method get more disk-space...

Still, having support for multiple download-directories will be an advantage...


Hmm, maybe going under the heading of "slow" storage... According to Distributed file management, the BOINC-client supports being used to store files, and, if needed, being ordered to re-upload these files. So, maybe it's possible to use this, and add some checks along the lines of:
if both computers reports successfully_downloaded_wu => delete from BOINC download-directory.
user_finished_crunching => don't delete wu on client.
If both_users_finished_crunching and wu_validated => mark_wu_for_deletion.
if reported_as_error and not_currently_in_download-dir => order_client_to_re_upload wu-file. Delay sending-out new Task until wu-file successfully uploaded.

Of course, if users routinely nukes their installation and so on, you could risk both computers has "lost" the wu, so it won't work...

So, as a safety, it would be an idea to send the wu-file to more than 2 computers. But, in this scenario you'll hit the download-bandwidth very quickly...

Not sure how it's worked-out with the torrent-support, but, if this is an option and enough users enables this, it could work...

And, no idea how much programming must be done to server-processes to make users store the wu-files be an option...

"I make so many mistakes. But then just think of all the mistakes I don't make, although I might."
ID: 805055 · Report as offensive
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 20619
Credit: 7,508,002
RAC: 20
United Kingdom
Message 805240 - Posted: 5 Sep 2008, 10:26:21 UTC - in response to Message 805055.  

... Now, no idea how linux is on this part, but, download-directory does have 1024 sub-directories, so, it's maybe possible to "link" from another server, and by this method get more disk-space...

You can indeed soft-link across mounts.

Happy crunchin',
Martin


See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 805240 · Report as offensive
MarkJ Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 08
Posts: 1139
Credit: 80,854,192
RAC: 5
Australia
Message 805267 - Posted: 5 Sep 2008, 14:10:05 UTC - in response to Message 805055.  

Hmm, maybe going under the heading of "slow" storage... According to Distributed file management, the BOINC-client supports being used to store files, and, if needed, being ordered to re-upload these files. So, maybe it's possible to use this, and add some checks along the lines of:
if both computers reports successfully_downloaded_wu => delete from BOINC download-directory.
user_finished_crunching => don't delete wu on client.
If both_users_finished_crunching and wu_validated => mark_wu_for_deletion.
if reported_as_error and not_currently_in_download-dir => order_client_to_re_upload wu-file. Delay sending-out new Task until wu-file successfully uploaded.

Of course, if users routinely nukes their installation and so on, you could risk both computers has "lost" the wu, so it won't work...

So, as a safety, it would be an idea to send the wu-file to more than 2 computers. But, in this scenario you'll hit the download-bandwidth very quickly...


Taking this a step further - Seti sends wu to clients to crunch and also sends to "storage" BOINC clients. You would need duplication in case of lost/re-installed machines. Delete the data off Seti's servers once its been sent to both the storage and crunching clients. If a client failed to return the wu then the storage client could be asked to return it so that Seti can reissue it. Once returned and validated wu can be deleted from "storage" client. One drawback would be bandwidth and you'd only want people with a fast connection.

I also think the idea of grouping computers with similar turn around times has merit so that wu's can spend less time out in the wild.
BOINC blog
ID: 805267 · Report as offensive
1 · 2 · Next

Message boards : Number crunching : Possible server storage optimization? (long post)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.