Blitzed Again (Jul 02 2009)


log in

Advanced search

Message boards : Technical News : Blitzed Again (Jul 02 2009)

Previous · 1 · 2 · 3 · 4 · 5 · Next
Author Message
OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13625
Credit: 30,945,305
RAC: 20,415
United States
Message 914584 - Posted: 6 Jul 2009, 3:22:41 UTC - in response to Message 914561.



There's still a cost issue there. SETI@Home leases space from the University, and has to work within the constraints set forth by the University's dictates, which include power requirements for both the servers and the air conditioning to cool the servers. Then there's the issue of staff wages (they do try to get paid for their work).

I mention this because if the plan is to get other universities involved, you are effectively doubling the financial strain on the project. Other scientists at other universities will have to purchase servers (or look for donations), lease space, pay for their power usage, pay for the connection to the internet, etc. And of course those scientists will want to get paid as well.



How about rather than getting other uni's involved you get Nvidia involved. The deciding factor for me between an ATI card and an Nvidia card was the CUDA. Both cards play the games I want to play but only one will crunch. Amp up CUDA support messages in the SETI website and I'm sure that Nvidia would find hosting *all* the SETI boxes with a monster pipe a very profitable move. Win Win Win all round. Berkeley getts the boxes and their associated costs off site, we get WU to crunch, Nvidia gets advertising worth millions, the SETI team spends money on research rather than airconditioning and leasing space. Cheers Jason =:)


nVidia was only interested in getting the CUDA code working for marketing purposes. They are not interested in picking up the tab for the entire project when there's plenty of other CUDA capable projects out there that do not need help.
____________

Josef W. SegurProject donor
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4295
Credit: 1,065,441
RAC: 951
United States
Message 914602 - Posted: 6 Jul 2009, 4:00:45 UTC - in response to Message 914421.

... Shifting from a text encoding to binary (the way AP does it) could save 15% or more, save storage on disk, etc.

Of course, the science application would have to understand more than one encoding method.

The code is already there as used in AP...

Just add an "if" statement? (And someone to code it.)

Happy crunchin',
Martin

... and test it, and then time to deploy, and it'd be nice to give the optimized apps a chance to add it to their apps.

The actual coding is probably the easiest part.

Agreed, basically importing code from AP sources to the MB splitter and application. Building for all platforms could be problematic, note on the Applications page some haven't been updated since 5.12.

The x-setiathome format carries 6 bits per byte, I've never compared details to see if it's identical to UUencoding or Base64. The application code will use the same decoding if it's called Sun Binary, and of course the original data recorder used a Sun system.

The rate at which new builds can be tested at SETI Beta is fairly slow, so adapting the optimized apps is usually no problem. Obviously with incompatible WUs there would have to be a new application name, so no issues with running the new work on old apps.
Joe

zpm
Volunteer tester
Avatar
Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,601,367
RAC: 330
United States
Message 914608 - Posted: 6 Jul 2009, 4:26:42 UTC - in response to Message 914602.

as we are talking about the beta side ftm, the iup/down ld. server is down.....

i guess someone is playing with it...
____________

I recommend Secunia PSI: http://secunia.com/vulnerability_scanning/personal/
Go Georgia Tech.

Cameron
Avatar
Send message
Joined: 27 Nov 02
Posts: 69
Credit: 1,041,535
RAC: 205
Australia
Message 914611 - Posted: 6 Jul 2009, 4:43:47 UTC - in response to Message 914602.


Agreed, basically importing code from AP sources to the MB splitter and application. Building for all platforms could be problematic, note on the Applications page some haven't been updated since 5.12.

The x-setiathome format carries 6 bits per byte, I've never compared details to see if it's identical to UUencoding or Base64. The application code will use the same decoding if it's called Sun Binary, and of course the original data recorder used a Sun system.

The rate at which new builds can be tested at SETI Beta is fairly slow, so adapting the optimized apps is usually no problem. Obviously with incompatible WUs there would have to be a new application name, so no issues with running the new work on old apps.
Joe


Why are there two entrys for SPARC/Solaris and Linux/x86_64 in the Enhanced Table?

It doesn't change the fact of the statement and the Linux/x86_64 show two versions (5.12 and 5.28). It gives no indication as to the reason for this unlike the windows/x86 entries with CUDA support.

Country
Volunteer tester
Send message
Joined: 13 Jun 99
Posts: 5
Credit: 12,332,128
RAC: 0
Canada
Message 914631 - Posted: 6 Jul 2009, 6:42:35 UTC
Last modified: 6 Jul 2009, 6:59:58 UTC

I'm assuming we don't already us compression so if we are I guess we can disregard.

If we compressed with 7-Zip we can get about a 27% savings in bandwidth - which is huge. I took a bunch of seti files and compressed them at default settings to get that 27%

First I did the files individually, then I did a batch in 1 finished file of 7-Zip. This ended up being more or less the same so there seems to be no benefit in compressing a batch of seti files into 1 compressed file.

2nd I set 7-Zip to Ultra compress with produced more or less the same results as the default setting which either means default compression is Ultra mode or compression maxes out.

Lastly, I compressed all my seti at home files on my 2nd machine to test a different sent of work units and got a compression rate of 28%. So as we can see there is variation depending on how compressible the work units are.

Personally I'm all for uncompressing the files at my end if it means a 27% bandwidth savings for the Seti team. I didn't test the compression that's built into Boinc but the point is there are savings to be had.

Of course the other side of this is the time it takes to compress the files in the first place. Now for me it didn't take that much time but we're only talking about 60 megs of files not the gigs and gigs Seti@home has. This would tie up cpu time at there end as well as hard drive access time. I guess they would have to weight what is more important but bandwidth seems to be a major factor for them.

Country Waring
____________

Profile Borgholio
Avatar
Send message
Joined: 2 Aug 99
Posts: 653
Credit: 12,134,212
RAC: 3,452
United States
Message 914638 - Posted: 6 Jul 2009, 7:51:51 UTC - in response to Message 914631.

I, as well, attempted to compress several AP and MB workunits. Using the Windows XP built in zip compressor (compressed folders), I got 27 - 28% for MB and 1 - 3% for AP.
____________


You will be assimilated...bunghole!

Profile Gundolf Jahn
Send message
Joined: 19 Sep 00
Posts: 3184
Credit: 359,137
RAC: 27
Germany
Message 914645 - Posted: 6 Jul 2009, 8:16:23 UTC - in response to Message 914638.

I, as well, attempted to compress several AP and MB workunits. Using the Windows XP built in zip compressor (compressed folders), I got 27 - 28% for MB and 1 - 3% for AP.

Does anyone read this thread?

It was mentioned earlier that that results from the already used "compressing" algorithm for MB, namely UUencode or similar (effectively expanding 4 or 5 bytes to 6) while AP uses binary transfer.

Gruß,
Gundolf
____________
Computer sind nicht alles im Leben. (Kleiner Scherz)

SETI@home classic workunits 3,758
SETI@home classic CPU time 66,520 hours

Profile Borgholio
Avatar
Send message
Joined: 2 Aug 99
Posts: 653
Credit: 12,134,212
RAC: 3,452
United States
Message 914786 - Posted: 6 Jul 2009, 17:15:10 UTC - in response to Message 914645.

I, as well, attempted to compress several AP and MB workunits. Using the Windows XP built in zip compressor (compressed folders), I got 27 - 28% for MB and 1 - 3% for AP.

Does anyone read this thread?

It was mentioned earlier that that results from the already used "compressing" algorithm for MB, namely UUencode or similar (effectively expanding 4 or 5 bytes to 6) while AP uses binary transfer.

Gruß,
Gundolf


Yes I did read this thread, and the point is that compressing a MB workunit in .zip format gives you 27% better compression than whatever method is currently being used.
____________


You will be assimilated...bunghole!

Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar
Send message
Joined: 20 Dec 05
Posts: 1956
Credit: 10,406,693
RAC: 7,164
United States
Message 914791 - Posted: 6 Jul 2009, 17:21:45 UTC

Umm, reporting is on the fritz again, getting "couldn't connect to server" on the messages. It's been this way since 0700 Berkeley time...
____________
.

Profile Gary CharpentierProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 12693
Credit: 7,168,652
RAC: 14,923
United States
Message 914795 - Posted: 6 Jul 2009, 17:29:41 UTC - in response to Message 914786.

I, as well, attempted to compress several AP and MB workunits. Using the Windows XP built in zip compressor (compressed folders), I got 27 - 28% for MB and 1 - 3% for AP.

Does anyone read this thread?

It was mentioned earlier that that results from the already used "compressing" algorithm for MB, namely UUencode or similar (effectively expanding 4 or 5 bytes to 6) while AP uses binary transfer.

Gruß,
Gundolf


Yes I did read this thread, and the point is that compressing a MB workunit in .zip format gives you 27% better compression than whatever method is currently being used.

The point is then after that compression you have to uncompress it by 27% to transmit it as text!

____________

Country
Volunteer tester
Send message
Joined: 13 Jun 99
Posts: 5
Credit: 12,332,128
RAC: 0
Canada
Message 914822 - Posted: 6 Jul 2009, 18:04:41 UTC

I have a question - When seti sends me the files through the internet are they compressed and then I expand them to the text files that are on my computer? OR does seti@home compress the info into the text files and these text files are sent as is to my home and I can use them right away with no expanding them?

If I don't have to expand them and they send the text files as they appear in my seti data directory then clearly there is a more effective way to compress them with 7-Zip which could save bandwidth.

Country
____________

Profile Gundolf Jahn
Send message
Joined: 19 Sep 00
Posts: 3184
Credit: 359,137
RAC: 27
Germany
Message 914859 - Posted: 6 Jul 2009, 18:55:46 UTC - in response to Message 914822.

I have a question - When seti sends me the files through the internet are they compressed and then I expand them to the text files that are on my computer? OR does seti@home compress the info into the text files and these text files are sent as is to my home and I can use them right away with no expanding them?

If I don't have to expand them and they send the text files as they appear in my seti data directory then clearly there is a more effective way to compress them with 7-Zip which could save bandwidth.

Country

When SETI sends you the files through the Internet, they are expanded (to x-setiathome) and then you compress them (to binary). So, the way to save bandwidth would be to transfer them in binary format (like the AP tasks).

And since that has been said several times in this thread, I did ask if anyone reads it at all before posting.

Gruß,
Gundolf
____________
Computer sind nicht alles im Leben. (Kleiner Scherz)

SETI@home classic workunits 3,758
SETI@home classic CPU time 66,520 hours

zpm
Volunteer tester
Avatar
Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,601,367
RAC: 330
United States
Message 914872 - Posted: 6 Jul 2009, 19:14:21 UTC - in response to Message 914859.

well, hey you know that people on, in the western hemisphere jump to the end and don't read what has been said........
____________

I recommend Secunia PSI: http://secunia.com/vulnerability_scanning/personal/
Go Georgia Tech.

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 914892 - Posted: 6 Jul 2009, 19:37:45 UTC - in response to Message 914795.


Yes I did read this thread, and the point is that compressing a MB workunit in .zip format gives you 27% better compression than whatever method is currently being used.

The point is then after that compression you have to uncompress it by 27% to transmit it as text!

HTTP is 8 bit, you can transmit binary.

The SETI@HOME logo on this page was not sent as encoded text, it was sent binary.
____________

Country
Volunteer tester
Send message
Joined: 13 Jun 99
Posts: 5
Credit: 12,332,128
RAC: 0
Canada
Message 914898 - Posted: 6 Jul 2009, 19:49:07 UTC - in response to Message 914892.
Last modified: 6 Jul 2009, 19:49:46 UTC

Actually I did read the whole post but didn't fully understand it apparently but thank you for just implying I'm too lazy to read. Your new explanation makes more sense and I thank you for that, perhaps you should consider that next time.

Country
____________

Profile Borgholio
Avatar
Send message
Joined: 2 Aug 99
Posts: 653
Credit: 12,134,212
RAC: 3,452
United States
Message 914901 - Posted: 6 Jul 2009, 20:05:59 UTC - in response to Message 914795.


Yes I did read this thread, and the point is that compressing a MB workunit in .zip format gives you 27% better compression than whatever method is currently being used.

The point is then after that compression you have to uncompress it by 27% to transmit it as text!


So...don't transmit it as text then? Transmit as .zip and have it be uncompressed when it is returned to the Seti servers?
____________


You will be assimilated...bunghole!

Profile Gary CharpentierProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 12693
Credit: 7,168,652
RAC: 14,923
United States
Message 914911 - Posted: 6 Jul 2009, 20:24:37 UTC - in response to Message 914892.


Yes I did read this thread, and the point is that compressing a MB workunit in .zip format gives you 27% better compression than whatever method is currently being used.

The point is then after that compression you have to uncompress it by 27% to transmit it as text!

HTTP is 8 bit, you can transmit binary.

The SETI@HOME logo on this page was not sent as encoded text, it was sent binary.

And a considerable number of corporate gateways/virus/firewalls won't transmit binary through them if it isn't a jpeg. Or at least wouldn't when the original Seti@home was released. Today many more do and I agree that the MB work units would be much better transmitted as binary.

As to compression of random noise, that is after all what every work unit is unless ET is out there, if you understand the math behind compression, you will know that the result is likely to be bigger than what you started with. True I'm sure there is some non-random noise in the work unit, like the time, frequency, and location of the work unit, but it is such a small fraction of the data it won't make a difference.

____________

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 914930 - Posted: 6 Jul 2009, 20:50:36 UTC - in response to Message 914911.


HTTP is 8 bit, you can transmit binary.

The SETI@HOME logo on this page was not sent as encoded text, it was sent binary.

And a considerable number of corporate gateways/virus/firewalls won't transmit binary through them if it isn't a jpeg. Or at least wouldn't when the original Seti@home was released. Today many more do and I agree that the MB work units would be much better transmitted as binary.

As to compression of random noise, that is after all what every work unit is unless ET is out there, if you understand the math behind compression, you will know that the result is likely to be bigger than what you started with. True I'm sure there is some non-random noise in the work unit, like the time, frequency, and location of the work unit, but it is such a small fraction of the data it won't make a difference.

Not only do I understand that random binary does not compress, you'll probably find a post on this thread where I point that out.

... and there is the (server) CPU time to do the compression, which probably isn't worth it.

I wouldn't be surprised to find corporate firewalls that block .zip files as well, because they can be used to mask trojans coming in to the network.

Matt commented in a related post that work was being sent with a strange MIME type (suitable for MAN pages and that's about all) and he changed to to application/text, which is pretty harmless.

It seems to follow that BOINC doesn't use the (HTTP) MIME type at all, and they could just as easily tag .gif or .jpg on the end, send the right MIME type for an image, and that'd be that.
____________

zpm
Volunteer tester
Avatar
Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,601,367
RAC: 330
United States
Message 914953 - Posted: 6 Jul 2009, 21:25:00 UTC - in response to Message 914930.
Last modified: 6 Jul 2009, 21:25:20 UTC

theoretically speaking, if we were to get another server that only compressed and uncompressed the data, would that help?.....
____________

I recommend Secunia PSI: http://secunia.com/vulnerability_scanning/personal/
Go Georgia Tech.

Profile Gundolf Jahn
Send message
Joined: 19 Sep 00
Posts: 3184
Credit: 359,137
RAC: 27
Germany
Message 914954 - Posted: 6 Jul 2009, 21:28:25 UTC - in response to Message 914953.

But there is nothing to compress...

Previous · 1 · 2 · 3 · 4 · 5 · Next

Message boards : Technical News : Blitzed Again (Jul 02 2009)

Copyright © 2014 University of California