Bandwith solution suggestion

Message boards : Number crunching : Bandwith solution suggestion

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile Pilot
Avatar

Send message
Joined: 18 May 99
Posts: 534
Credit: 5,475,482
RAC: 0
Message 918129 - Posted: 15 Jul 2009, 17:54:17 UTC

Perhaps this has already been thought of, implemented or is not really viable, but it is just a suggestion. Would it be possible to only upload the first unique result of a work unit along with a checksum of the data and its results. All others clients would only have to report the calculated checksum for verification and credit.
IE:
1. Client completes a work unit, calculates the Checksum and sends it to BOINC. The client could then enter a state called “PENDING” until step 3 or 4 is reached.
2. BOINC checks database for prior submission of the Work Unit and a matching Checksum.
3. If a matching WU/Checksum is found, BOINC informs Client there is no need to send rest and the client is granted credit.
4. If no matching WU/Checksum is found, then BOINC informs the Client to send all.

When we finally figure it all out, all the rules will change and we can start all over again.
ID: 918129 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 7495
Credit: 91,191,024
RAC: 46,155
Australia
Message 918131 - Posted: 15 Jul 2009, 18:01:16 UTC - in response to Message 918129.  


Results are generally 22-33kB in size (mostly around 25kB).
MB Work Units are around 365kB in size, AP Work Units 8MB in size.
Even a huge reduction in the size of the result being returned won't have much of an impact on bandwidth, you've still got all the network communication that goes along with the upload process & just the sheer number of results to be returned.
So far most of the network problems (including not being able to upload (at least untill the present upload problems)) have been due to download bandwidth issues.
Grant
Darwin NT
ID: 918131 · Report as offensive
Profile Ageless
Avatar

Send message
Joined: 9 Jun 99
Posts: 13822
Credit: 3,269,733
RAC: 0
Netherlands
Message 918134 - Posted: 15 Jul 2009, 18:06:45 UTC - in response to Message 918129.  

Before you think too difficult scenarios, a possible fix for the waiting uploads is already checked in to Trunk and will (probably) show up in a next client.

David 10 July 2009
- client: restored code for project-wide backoff on file uploads and downloads.
I originally added this on 30 Sept 2005 and disabled it 2 weeks later because there were reports of problems.
However, we need this functionality (e.g. on GPU hosts with hundreds of files to upload, we need to back off after a few failures, not try all of them).
I added messages (<file_xfer_debug>) so you can see what's going on. Fixes #932.


All scenarios need a new client anyway and how are you going to persuade everyone (including all those not reading the forums) that the latest version is the best thing since sliced bread?
Jord

Ancient Astronaut Theorists suggest that in many ways, you can be considered an alien conspiracy!
ID: 918134 · Report as offensive
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11143
Credit: 83,862,528
RAC: 46,822
United Kingdom
Message 918136 - Posted: 15 Jul 2009, 18:11:33 UTC - in response to Message 918134.  

Before you think too difficult scenarios, a possible fix for the waiting uploads is already checked in to Trunk and will (probably) show up in a next client.

David 10 July 2009
- client: restored code for project-wide backoff on file uploads and downloads.
I originally added this on 30 Sept 2005 and disabled it 2 weeks later because there were reports of problems.
However, we need this functionality (e.g. on GPU hosts with hundreds of files to upload, we need to back off after a few failures, not try all of them).
I added messages (<file_xfer_debug>) so you can see what's going on. Fixes #932.

All scenarios need a new client anyway and how are you going to persuade everyone (including all those not reading the forums) that the latest version is the best thing since sliced bread?

I do hope we get a chance to test that in Beta (it was too late for v6.6.37) before it becomes recommended - get a chance to find and iron out whatever those unspecified "problems" were that it caused last time.
ID: 918136 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 7495
Credit: 91,191,024
RAC: 46,155
Australia
Message 918142 - Posted: 15 Jul 2009, 18:14:25 UTC - in response to Message 918134.  

All scenarios need a new client anyway and how are you going to persuade everyone (including all those not reading the forums) that the latest version is the best thing since sliced bread?

A few ntoices in the logs & popups from the system tray.
Older versions no longer supported- upgrade or move on.
Grant
Darwin NT
ID: 918142 · Report as offensive
Profile Ageless
Avatar

Send message
Joined: 9 Jun 99
Posts: 13822
Credit: 3,269,733
RAC: 0
Netherlands
Message 918153 - Posted: 15 Jul 2009, 18:25:35 UTC - in response to Message 918142.  

A few notices in the logs & popups from the system tray.
Older versions no longer supported- upgrade or move on.

Hooya, and get all the complaints that BOINC is being very annoying that way, where to turn it off and if that's not possible, where to get the client before that, that didn't do that. ;-)

Ricky wrote:
I do hope we get a chance to test that in Beta (it was too late for v6.6.37) before it becomes recommended - get a chance to find and iron out whatever those unspecified "problems" were that it caused last time.

I hear ya, fingers crossed.

I keep wondering why it was taken out in the first place.
and since you're good with searching through old emails and such... ;-)
Jord

Ancient Astronaut Theorists suggest that in many ways, you can be considered an alien conspiracy!
ID: 918153 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 918187 - Posted: 15 Jul 2009, 19:40:09 UTC - in response to Message 918136.  

I do hope we get a chance to test that in Beta (it was too late for v6.6.37) before it becomes recommended - get a chance to find and iron out whatever those unspecified "problems" were that it caused last time.

I can't do a build from the trunk (yet, I'm working on it) but it is something that can be tested right here, and this is a perfect time since SETI is struggling.

That is, the function can be tested, but the desired effect won't be noticed until it is widely deployed.
ID: 918187 · Report as offensive
zoom314
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 56774
Credit: 40,756,573
RAC: 5,083
United States
Message 918188 - Posted: 15 Jul 2009, 19:45:55 UTC
Last modified: 15 Jul 2009, 19:58:00 UTC

So far I just have to wait for the uploads to clear up due to a lack of available bandwidth, As so far there's no official response in Technical by Matt from the 14th as to why events with the upload server happened as they did. Some openness would be appreciated when they have the time.

S@H simply needs more Fiber in Its diet(or something to give It the runs maybe;)).
Pluto is still a planet

Beep! Beep!
ID: 918188 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 918195 - Posted: 15 Jul 2009, 20:03:07 UTC - in response to Message 918129.  

Perhaps this has already been thought of, implemented or is not really viable, but it is just a suggestion. Would it be possible to only upload the first unique result of a work unit along with a checksum of the data and its results. All others clients would only have to report the calculated checksum for verification and credit.
IE:
1. Client completes a work unit, calculates the Checksum and sends it to BOINC. The client could then enter a state called “PENDING” until step 3 or 4 is reached.
2. BOINC checks database for prior submission of the Work Unit and a matching Checksum.
3. If a matching WU/Checksum is found, BOINC informs Client there is no need to send rest and the client is granted credit.
4. If no matching WU/Checksum is found, then BOINC informs the Client to send all.

This is a clever idea.

The problem is: valid results are not necessarily identical, so the checksums will be different even on valid results.
ID: 918195 · Report as offensive
Profile Bill Walker
Avatar

Send message
Joined: 4 Sep 99
Posts: 3868
Credit: 2,697,159
RAC: 4
Canada
Message 918228 - Posted: 15 Jul 2009, 21:24:26 UTC - in response to Message 918195.  


This is a clever idea.

The problem is: valid results are not necessarily identical, so the checksums will be different even on valid results.


I thought about that, but is there some other indicator of validity (call it a Super Checksum) that takes up less bits than the full results package?


ID: 918228 · Report as offensive
Profile Voyager
Volunteer tester
Avatar

Send message
Joined: 2 Nov 99
Posts: 602
Credit: 3,264,813
RAC: 0
United States
Message 918235 - Posted: 15 Jul 2009, 21:36:37 UTC

a non-tech thought.would it help to upload oldest results first(those closest to due date)and work backwards through them?
ID: 918235 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 918243 - Posted: 15 Jul 2009, 22:00:15 UTC - in response to Message 918228.  


This is a clever idea.

The problem is: valid results are not necessarily identical, so the checksums will be different even on valid results.


I thought about that, but is there some other indicator of validity (call it a Super Checksum) that takes up less bits than the full results package?

I suspect that by the time you produce a description that accurately represents the result it'll turn out to be nearly the size of the result.
ID: 918243 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 918244 - Posted: 15 Jul 2009, 22:01:14 UTC - in response to Message 918235.  

a non-tech thought.would it help to upload oldest results first(those closest to due date)and work backwards through them?

If you defer those that don't need to be uploaded urgently, yes it would.
ID: 918244 · Report as offensive
John McLeod VII
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jul 99
Posts: 24806
Credit: 754,585
RAC: 65
United States
Message 918264 - Posted: 15 Jul 2009, 23:04:46 UTC - in response to Message 918129.  

Perhaps this has already been thought of, implemented or is not really viable, but it is just a suggestion. Would it be possible to only upload the first unique result of a work unit along with a checksum of the data and its results. All others clients would only have to report the calculated checksum for verification and credit.
IE:
1. Client completes a work unit, calculates the Checksum and sends it to BOINC. The client could then enter a state called “PENDING” until step 3 or 4 is reached.
2. BOINC checks database for prior submission of the Work Unit and a matching Checksum.
3. If a matching WU/Checksum is found, BOINC informs Client there is no need to send rest and the client is granted credit.
4. If no matching WU/Checksum is found, then BOINC informs the Client to send all.

The clients are not in communication with each other, and may not be in communication with the server before the upload starts. Testing to see if some other result was uploaded for that WU involves asking the DB server - this is NOT going to happen. What happens if the first task is invalid.


BOINC WIKI
ID: 918264 · Report as offensive
Profile Pilot
Avatar

Send message
Joined: 18 May 99
Posts: 534
Credit: 5,475,482
RAC: 0
Message 918269 - Posted: 15 Jul 2009, 23:16:21 UTC - in response to Message 918264.  

State 4 exist and the client is instructed to report all.
When we finally figure it all out, all the rules will change and we can start all over again.
ID: 918269 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 918273 - Posted: 15 Jul 2009, 23:31:05 UTC - in response to Message 918243.  


This is a clever idea.

The problem is: valid results are not necessarily identical, so the checksums will be different even on valid results.

I thought about that, but is there some other indicator of validity (call it a Super Checksum) that takes up less bits than the full results package?

I suspect that by the time you produce a description that accurately represents the result it'll turn out to be nearly the size of the result.

I took an MB result file of 25180 bytes, stripped out lines which the Validator ignores, and ended with a 776 byte file. Standalone test code derived from the Berkeley validator reports the stripped and unstripped versions "Strongly similar". The 776 bytes could be reduced much more if the XML were removed and numerical values sent as binary.

However, there would need to be significant changes to BOINC servers and the core client to handle the additional steps, saving the real result until told it isn't needed, etc. My firm belief is there's no bandwidth problem on the link going to SSL, rather it's a transaction rate problem. The smaller uploads would be no help for that, more likely make it worse.
                                                                Joe
ID: 918273 · Report as offensive
Profile Pilot
Avatar

Send message
Joined: 18 May 99
Posts: 534
Credit: 5,475,482
RAC: 0
Message 918278 - Posted: 15 Jul 2009, 23:38:18 UTC - in response to Message 918273.  

Thank you, I understand.
I had picked up from other threads that perhaps there needed to be additional Fiber or bandwidth to help overall performance.
When we finally figure it all out, all the rules will change and we can start all over again.
ID: 918278 · Report as offensive
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11143
Credit: 83,862,528
RAC: 46,822
United Kingdom
Message 918401 - Posted: 16 Jul 2009, 9:04:42 UTC

@ Joe, Ned,

Could you sanity-check my "data aggregator" suggestion at #918399, please?
ID: 918401 · Report as offensive
Profile MarkJProject Donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 08
Posts: 978
Credit: 33,086,194
RAC: 5,752
Australia
Message 918427 - Posted: 16 Jul 2009, 12:42:24 UTC - in response to Message 918401.  

@ Joe, Ned,

Could you sanity-check my "data aggregator" suggestion at #918399, please?


I wonder if the superhost (if it ever happens) would help this? It should be able to upload all its files in one chunk, thus reducing the number of network connections.
BOINC blog
ID: 918427 · Report as offensive
John McLeod VII
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jul 99
Posts: 24806
Credit: 754,585
RAC: 65
United States
Message 918435 - Posted: 16 Jul 2009, 13:15:28 UTC - in response to Message 918427.  

@ Joe, Ned,

Could you sanity-check my "data aggregator" suggestion at #918399, please?


I wonder if the superhost (if it ever happens) would help this? It should be able to upload all its files in one chunk, thus reducing the number of network connections.

It probably wouldn't help that much. Files are broken up into multiple packets, each of which have to make it through, and any of which may hit the floor.


BOINC WIKI
ID: 918435 · Report as offensive
1 · 2 · Next

Message boards : Number crunching : Bandwith solution suggestion


 
©2016 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.