Maxed (Dec 16 2010)

Compressing the data is a waste of time. The radio signals are random noise.
If you want to compress the WUs, then it would require another boinc project for it!

Like concrete vs a sponge - there are no holes to squeeze out because there are no patterns to match. This is what we are doing - looking for patterns so tiny that it needs petaflops of work.

The only sensible option we have is to change the transport encoding in the workunit from:

It's entirely up to the lab staff how they handle it, but either compression or raw CDATA for MB would shave ~30% off that part of the project's download requirements. Whether modifying the splitter code to generate CDATA format, or post-processing the existing format with compression, I'll leave to them. I'll only mention that post-processing would also compress the (non-trivial) XML header, whereas CDATA would not.

FYI, limits have been raised...

- Matt

With the servers running out of work units so fast these days, have there been any thoughts on how to process more work units between each required physical visit to the server room to load tapes or disks of whatever media the raw data is stored on these days?

Is this process already optimized, or would it be possible to perhaps store the data on multiple external hard drives or on it's own separate RAID array?

If it is possible to read and verify the data from the media faster than the machines can initially process said data to create workunits, than it should be possible to build up a store of unprocessed data when humans are available to feed the machine. The "extra" data could then be queued for access whenever one of the splitter CPU's has no available data to make work units from.

Great for weekends, or after outages. Not sure if this would be something that the project would actually want to run full time, as I understand the bottleneck for the actual science is in the post-home-user stages.

I'm imagining a couple really big USB external hard drives for "extra" data storage. These are getting pretty cheap.

FYI, limits have been raised...

- Matt

that's nice, but there is very little work to be split. please feed the splitters.

Please enjoy your Saturday night away from the lab.

I am not sure how this could be done, but some of us ache for a fairness doctrine as applied to downloaded WUs.

While the system was up for the last couple of weeks, I managed to snag 2 Astropulse and 8-9 Seti Enhanced WUs. Now the my last 6 Seti enhanced WUs are Pending Validation as the second computer is one of the machines that downloads 4-10 WUs per day. Granted most of them get returned inside the window of time, but still, I turn them around in 2 days, and wait 3-6 weeks for validation to happen.

Seti is out of WUs now so my Boinc is working on other projects JUST to stay busy
____________

I am not sure how this could be done, but some of us ache for a fairness doctrine as applied to downloaded WUs.

While the system was up for the last couple of weeks, I managed to snag 2 Astropulse and 8-9 Seti Enhanced WUs. Now the my last 6 Seti enhanced WUs are Pending Validation as the second computer is one of the machines that downloads 4-10 WUs per day. Granted most of them get returned inside the window of time, but still, I turn them around in 2 days, and wait 3-6 weeks for validation to happen.

Seti is out of WUs now so my Boinc is working on other projects JUST to stay busy

I think the "fairness" doctrine is known as "grab and growl". Grab what is there, and growl about it all you want, it will not change a thing.

how come the people with a low RAC are always ridiculing the people, usually high RAC users, who are almost out of work when they are reporting a problem.

i'm for a more fair work distribution method in which RAC is the basis. machines/users with a higher RAC should be rewarded more work, directly proportional to RAC, during the rationing periods.

then we all could listen to the "computationally challenged" users complain they don't have enough work...
____________

how come the people with a low RAC are always ridiculing the people, usually high RAC users, who are almost out of work when they are reporting a problem.

i'm for a more fair work distribution method in which RAC is the basis. machines/users with a higher RAC should be rewarded more work, directly proportional to RAC, during the rationing periods.

then we all could listen to the "computationally challenged" users complain they don't have enough work...

Fairness lies in the eye of the beholder.

I see the rationale for having as much seti work as your machine can handle, as the seti folks have had extended downtime over the last several months.

High RAC, low RAC, that all depends on just how many computers you can employ and the relative hotness of each of the machines.

Some of us only have one computer, and that one is several years old. If the High RAC users get all the WUs then we can be shut out.

A slightly better approach would reward users who consistently turn in units that pass validation, turn them in on time, and grade higher for short turn around time. People who fail to turn in units on time get throttled on how many work units they have at one time. People who have compute errors on their work get limited on how many WUs can be outstanding at one time. People who turn in work units well inside the time limit (lets assume that they return them inside 5 days, but sysadmin makes the decision) get rewarded with more WUs.


____________