Blitzed Again (Jul 02 2009)

Rather than speculating in a vacuum, I suggest any of you could simply try compressing a WU or two. You'd find that setiathome_enhanced WUs are moderately compressible, astropulse_v505 only slightly. If compressed downloads were implemented, it would be akin to adding another 25 MBits/second to the download bandwidth; enough to help short term, but hardly a permanent fix.

BOINC 5.x and later have libCurl which is prepared to decompress files sent with gzip compression. The download servers run Apache, which can be configured to gzip data being sent. All that would be needed is to set the warning that all users must upgrade to BOINC 5.x or later, give it long enough to be seen and adopted, then reconfigure the download servers.

The real issue is whether the download servers can handle the extra load of doing the compression; an earlier suggestion for a trial at SETI Beta wouldn't test that effectively. And although the 7zip format does compress better than gzip, it takes more memory and time to do the compression. It would also take project-specific coding to implement, rather than using the features BOINC already has.

Edit:

Correct me if I am wrong, but isn't it a text file that is sent each way?

Rather than speculating in a vacuum, I suggest any of you could simply try compressing a WU or two. You'd find that setiathome_enhanced WUs are moderately compressible, astropulse_v505 only slightly. If compressed downloads were implemented, it would be akin to adding another 25 MBits/second to the download bandwidth; enough to help short term, but hardly a permanent fix.

BOINC 5.x and later have libCurl which is prepared to decompress files sent with gzip compression. The download servers run Apache, which can be configured to gzip data being sent. All that would be needed is to set the warning that all users must upgrade to BOINC 5.x or later, give it long enough to be seen and adopted, then reconfigure the download servers.

The real issue is whether the download servers can handle the extra load of doing the compression; an earlier suggestion for a trial at SETI Beta wouldn't test that effectively. And although the 7zip format does compress better than gzip, it takes more memory and time to do the compression. It would also take project-specific coding to implement, rather than using the features BOINC already has.

Edit:

Correct me if I am wrong, but isn't it a text file that is sent each way?

The payload in Enhanced work is 256KB of nearly random data encoded in uue fashion, so can be sent as text. The payload in AP work is 8MB of pure 8 bit nearly random data. In both cases, it's the task of the science applications to ferret out the few cases where the data deviates from random noise.

Joe

it my opinion, any dual-core processor spends not a lot of time wrapping it up and compressing....a quad even less....


____________

I recommend Secunia PSI: http://secunia.com/vulnerability_scanning/personal/
Go Georgia Tech.

i'll still say "A byte Saved is a Byte that can be spent later." this concept is helping DD@H and H@H, when it comes to paying for bandwidth.


hey, if i were to win the lottery, i would donate a lot to boinc projects....
____________

I recommend Secunia PSI: http://secunia.com/vulnerability_scanning/personal/
Go Georgia Tech.

Lossless compression is usually done with the Lempel-Ziv-Welch (LZW) algorithm or a variation of it.
It works by examining the file and placing each string into a dictionary, if the same string is found again it is substituted with a shorter code, as the dictionary builds up the most popular strings get the shortest substitution codes and at the other end unique strings are not compressed.

As the Seti data files consist of a header followed by random data the header can be compressed considerably but the random data will only be compressed a very small amount. The header is about the same size for MB and AP tasks.
As the MB files are short, 350 Kbyte, they will compress more than the AP files which are 8 Mbyte. So seeing figures of 27% for MB and 11% for AP is probably close to what is expected.

Perhaps the problem is that of the distribution algorithm?
Looking at my 4 computers, the oldest process "normal" WU in either 34hrs or 10hrs, compared with the 3.5hrs or 1hr of the fastest. The other day the slowest got quite a number of slow work units, which it is still plodding through, and the fastest got a very large number of its fastest which it devoured without any problems.
Obviously on completion the WU are then returned, and if they are coming back very rapidly, and in very large numbers they will flood the available bandwidth, so why not ensure that the faster machines get the "bigger" WU, and the slower the "smaller"?

____________
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

I'm not so sure about compression. My understanding was that random noise wouldn't compress very much at all as there are no patterns in it to exploit to compress. Now a WU with a signal might compress because of the pattern of the signal. ...

it my opinion, any dual-core processor spends not a lot of time wrapping it up and compressing....a quad even less....

it my opinion, any dual-core processor spends not a lot of time wrapping it up and compressing....a quad even less....

The processing bottleneck would be at the server, not at the client. Now imagine having to compress ten to twenty tasks per second in order to feed the outbound queue. Similar for decompressing the result data.

... Shifting from a text encoding to binary (the way AP does it) could save 15% or more, save storage on disk, etc.

Of course, the science application would have to understand more than one encoding method.

it my opinion, any dual-core processor spends not a lot of time wrapping it up and compressing....a quad even less....

The processing bottleneck would be at the server, not at the client. Now imagine having to compress ten to twenty tasks per second in order to feed the outbound queue. Similar for decompressing the result data.

I agree that compression is tough.

... but what we're doing right now is taking the binary data and converting it to text. The mime type is x-setiathome which is legal but non-standard, and I can't tell how efficient it is by looking at it. If it's like Base64 it is 6 characters for four bytes, if it's like ASCII85, it's 5 characters for 4 bytes.

Then it converts back on the other end.

Shifting from a text encoding to binary (the way AP does it) could save 15% or more, save storage on disk, etc.

Of course, the science application would have to understand more than one encoding method.

This morning it came to my attention that we've been sending out workunits with the "application/x-troff-man" mime type. This was because files with numerical suffices (like workunits) are assumed to be man pages. This may have been causing some blockages at firewalls. I changed the mime type to "text/plain."

re mime type

This morning it came to my attention that we've been sending out workunits with the "application/x-troff-man" mime type. This was because files with numerical suffices (like workunits) are assumed to be man pages. This may have been causing some blockages at firewalls. I changed the mime type to "text/plain."

I get the feeling some Apache tables have been borked.

... Shifting from a text encoding to binary (the way AP does it) could save 15% or more, save storage on disk, etc.

Of course, the science application would have to understand more than one encoding method.

The code is already there as used in AP...

Just add an "if" statement? (And someone to code it.)

Happy crunchin',
Martin

well, hey that would be nice, but people are working on an opencl platform that will allow nvid and ati cards to crunch.... it would go much faster if ati weren't so "do it yourself...." which then if it came out, their goes the incentive for nvid.
____________

I recommend Secunia PSI: http://secunia.com/vulnerability_scanning/personal/
Go Georgia Tech.