Bandwith solution suggestion


log in

Advanced search

Message boards : Number crunching : Bandwith solution suggestion

1 · 2 · Next
Author Message
Profile Pilot
Avatar
Send message
Joined: 18 May 99
Posts: 534
Credit: 5,475,482
RAC: 0
Message 918129 - Posted: 15 Jul 2009, 17:54:17 UTC

Perhaps this has already been thought of, implemented or is not really viable, but it is just a suggestion. Would it be possible to only upload the first unique result of a work unit along with a checksum of the data and its results. All others clients would only have to report the calculated checksum for verification and credit.
IE:
1. Client completes a work unit, calculates the Checksum and sends it to BOINC. The client could then enter a state called “PENDING” until step 3 or 4 is reached.
2. BOINC checks database for prior submission of the Work Unit and a matching Checksum.
3. If a matching WU/Checksum is found, BOINC informs Client there is no need to send rest and the client is granted credit.
4. If no matching WU/Checksum is found, then BOINC informs the Client to send all.

____________
When we finally figure it all out, all the rules will change and we can start all over again.

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5819
Credit: 58,970,711
RAC: 48,077
Australia
Message 918131 - Posted: 15 Jul 2009, 18:01:16 UTC - in response to Message 918129.


Results are generally 22-33kB in size (mostly around 25kB).
MB Work Units are around 365kB in size, AP Work Units 8MB in size.
Even a huge reduction in the size of the result being returned won't have much of an impact on bandwidth, you've still got all the network communication that goes along with the upload process & just the sheer number of results to be returned.
So far most of the network problems (including not being able to upload (at least untill the present upload problems)) have been due to download bandwidth issues.
____________
Grant
Darwin NT.

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12300
Credit: 2,597,518
RAC: 1,084
Netherlands
Message 918134 - Posted: 15 Jul 2009, 18:06:45 UTC - in response to Message 918129.

Before you think too difficult scenarios, a possible fix for the waiting uploads is already checked in to Trunk and will (probably) show up in a next client.

David 10 July 2009
- client: restored code for project-wide backoff on file uploads and downloads.
I originally added this on 30 Sept 2005 and disabled it 2 weeks later because there were reports of problems.
However, we need this functionality (e.g. on GPU hosts with hundreds of files to upload, we need to back off after a few failures, not try all of them).
I added messages (<file_xfer_debug>) so you can see what's going on. Fixes #932.


All scenarios need a new client anyway and how are you going to persuade everyone (including all those not reading the forums) that the latest version is the best thing since sliced bread?
____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8499
Credit: 49,907,961
RAC: 51,045
United Kingdom
Message 918136 - Posted: 15 Jul 2009, 18:11:33 UTC - in response to Message 918134.

Before you think too difficult scenarios, a possible fix for the waiting uploads is already checked in to Trunk and will (probably) show up in a next client.

David 10 July 2009
- client: restored code for project-wide backoff on file uploads and downloads.
I originally added this on 30 Sept 2005 and disabled it 2 weeks later because there were reports of problems.
However, we need this functionality (e.g. on GPU hosts with hundreds of files to upload, we need to back off after a few failures, not try all of them).
I added messages (<file_xfer_debug>) so you can see what's going on. Fixes #932.

All scenarios need a new client anyway and how are you going to persuade everyone (including all those not reading the forums) that the latest version is the best thing since sliced bread?

I do hope we get a chance to test that in Beta (it was too late for v6.6.37) before it becomes recommended - get a chance to find and iron out whatever those unspecified "problems" were that it caused last time.

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5819
Credit: 58,970,711
RAC: 48,077
Australia
Message 918142 - Posted: 15 Jul 2009, 18:14:25 UTC - in response to Message 918134.

All scenarios need a new client anyway and how are you going to persuade everyone (including all those not reading the forums) that the latest version is the best thing since sliced bread?

A few ntoices in the logs & popups from the system tray.
Older versions no longer supported- upgrade or move on.
____________
Grant
Darwin NT.

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12300
Credit: 2,597,518
RAC: 1,084
Netherlands
Message 918153 - Posted: 15 Jul 2009, 18:25:35 UTC - in response to Message 918142.

A few notices in the logs & popups from the system tray.
Older versions no longer supported- upgrade or move on.

Hooya, and get all the complaints that BOINC is being very annoying that way, where to turn it off and if that's not possible, where to get the client before that, that didn't do that. ;-)

Ricky wrote:
I do hope we get a chance to test that in Beta (it was too late for v6.6.37) before it becomes recommended - get a chance to find and iron out whatever those unspecified "problems" were that it caused last time.

I hear ya, fingers crossed.

I keep wondering why it was taken out in the first place.
and since you're good with searching through old emails and such... ;-)
____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 918187 - Posted: 15 Jul 2009, 19:40:09 UTC - in response to Message 918136.

I do hope we get a chance to test that in Beta (it was too late for v6.6.37) before it becomes recommended - get a chance to find and iron out whatever those unspecified "problems" were that it caused last time.

I can't do a build from the trunk (yet, I'm working on it) but it is something that can be tested right here, and this is a perfect time since SETI is struggling.

That is, the function can be tested, but the desired effect won't be noticed until it is widely deployed.
____________

zoom314Project donor
Avatar
Send message
Joined: 30 Nov 03
Posts: 46305
Credit: 36,696,251
RAC: 5,305
Message 918188 - Posted: 15 Jul 2009, 19:45:55 UTC
Last modified: 15 Jul 2009, 19:58:00 UTC

So far I just have to wait for the uploads to clear up due to a lack of available bandwidth, As so far there's no official response in Technical by Matt from the 14th as to why events with the upload server happened as they did. Some openness would be appreciated when they have the time.

S@H simply needs more Fiber in Its diet(or something to give It the runs maybe;)).
____________
My Facebook, War Commander, 2015

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 918195 - Posted: 15 Jul 2009, 20:03:07 UTC - in response to Message 918129.

Perhaps this has already been thought of, implemented or is not really viable, but it is just a suggestion. Would it be possible to only upload the first unique result of a work unit along with a checksum of the data and its results. All others clients would only have to report the calculated checksum for verification and credit.
IE:
1. Client completes a work unit, calculates the Checksum and sends it to BOINC. The client could then enter a state called “PENDING” until step 3 or 4 is reached.
2. BOINC checks database for prior submission of the Work Unit and a matching Checksum.
3. If a matching WU/Checksum is found, BOINC informs Client there is no need to send rest and the client is granted credit.
4. If no matching WU/Checksum is found, then BOINC informs the Client to send all.

This is a clever idea.

The problem is: valid results are not necessarily identical, so the checksums will be different even on valid results.
____________

Profile Bill Walker
Avatar
Send message
Joined: 4 Sep 99
Posts: 3374
Credit: 2,082,070
RAC: 2,311
Canada
Message 918228 - Posted: 15 Jul 2009, 21:24:26 UTC - in response to Message 918195.


This is a clever idea.

The problem is: valid results are not necessarily identical, so the checksums will be different even on valid results.


I thought about that, but is there some other indicator of validity (call it a Super Checksum) that takes up less bits than the full results package?

____________

Profile Voyager
Volunteer tester
Avatar
Send message
Joined: 2 Nov 99
Posts: 602
Credit: 3,264,813
RAC: 0
United States
Message 918235 - Posted: 15 Jul 2009, 21:36:37 UTC

a non-tech thought.would it help to upload oldest results first(those closest to due date)and work backwards through them?

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 918243 - Posted: 15 Jul 2009, 22:00:15 UTC - in response to Message 918228.


This is a clever idea.

The problem is: valid results are not necessarily identical, so the checksums will be different even on valid results.


I thought about that, but is there some other indicator of validity (call it a Super Checksum) that takes up less bits than the full results package?

I suspect that by the time you produce a description that accurately represents the result it'll turn out to be nearly the size of the result.
____________

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 918244 - Posted: 15 Jul 2009, 22:01:14 UTC - in response to Message 918235.

a non-tech thought.would it help to upload oldest results first(those closest to due date)and work backwards through them?

If you defer those that don't need to be uploaded urgently, yes it would.
____________

John McLeod VII
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 15 Jul 99
Posts: 24520
Credit: 521,562
RAC: 92
United States
Message 918264 - Posted: 15 Jul 2009, 23:04:46 UTC - in response to Message 918129.

Perhaps this has already been thought of, implemented or is not really viable, but it is just a suggestion. Would it be possible to only upload the first unique result of a work unit along with a checksum of the data and its results. All others clients would only have to report the calculated checksum for verification and credit.
IE:
1. Client completes a work unit, calculates the Checksum and sends it to BOINC. The client could then enter a state called “PENDING” until step 3 or 4 is reached.
2. BOINC checks database for prior submission of the Work Unit and a matching Checksum.
3. If a matching WU/Checksum is found, BOINC informs Client there is no need to send rest and the client is granted credit.
4. If no matching WU/Checksum is found, then BOINC informs the Client to send all.

The clients are not in communication with each other, and may not be in communication with the server before the upload starts. Testing to see if some other result was uploaded for that WU involves asking the DB server - this is NOT going to happen. What happens if the first task is invalid.
____________


BOINC WIKI

Profile Pilot
Avatar
Send message
Joined: 18 May 99
Posts: 534
Credit: 5,475,482
RAC: 0
Message 918269 - Posted: 15 Jul 2009, 23:16:21 UTC - in response to Message 918264.

State 4 exist and the client is instructed to report all.
____________
When we finally figure it all out, all the rules will change and we can start all over again.

Josef W. SegurProject donor
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4247
Credit: 1,048,290
RAC: 293
United States
Message 918273 - Posted: 15 Jul 2009, 23:31:05 UTC - in response to Message 918243.


This is a clever idea.

The problem is: valid results are not necessarily identical, so the checksums will be different even on valid results.

I thought about that, but is there some other indicator of validity (call it a Super Checksum) that takes up less bits than the full results package?

I suspect that by the time you produce a description that accurately represents the result it'll turn out to be nearly the size of the result.

I took an MB result file of 25180 bytes, stripped out lines which the Validator ignores, and ended with a 776 byte file. Standalone test code derived from the Berkeley validator reports the stripped and unstripped versions "Strongly similar". The 776 bytes could be reduced much more if the XML were removed and numerical values sent as binary.

However, there would need to be significant changes to BOINC servers and the core client to handle the additional steps, saving the real result until told it isn't needed, etc. My firm belief is there's no bandwidth problem on the link going to SSL, rather it's a transaction rate problem. The smaller uploads would be no help for that, more likely make it worse.
Joe

Profile Pilot
Avatar
Send message
Joined: 18 May 99
Posts: 534
Credit: 5,475,482
RAC: 0
Message 918278 - Posted: 15 Jul 2009, 23:38:18 UTC - in response to Message 918273.

Thank you, I understand.
I had picked up from other threads that perhaps there needed to be additional Fiber or bandwidth to help overall performance.
____________
When we finally figure it all out, all the rules will change and we can start all over again.

Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8499
Credit: 49,907,961
RAC: 51,045
United Kingdom
Message 918401 - Posted: 16 Jul 2009, 9:04:42 UTC

@ Joe, Ned,

Could you sanity-check my "data aggregator" suggestion at #918399, please?

Profile MarkJProject donor
Volunteer tester
Avatar
Send message
Joined: 17 Feb 08
Posts: 939
Credit: 24,033,745
RAC: 74,788
Australia
Message 918427 - Posted: 16 Jul 2009, 12:42:24 UTC - in response to Message 918401.

@ Joe, Ned,

Could you sanity-check my "data aggregator" suggestion at #918399, please?


I wonder if the superhost (if it ever happens) would help this? It should be able to upload all its files in one chunk, thus reducing the number of network connections.
____________
BOINC blog

John McLeod VII
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 15 Jul 99
Posts: 24520
Credit: 521,562
RAC: 92
United States
Message 918435 - Posted: 16 Jul 2009, 13:15:28 UTC - in response to Message 918427.

@ Joe, Ned,

Could you sanity-check my "data aggregator" suggestion at #918399, please?


I wonder if the superhost (if it ever happens) would help this? It should be able to upload all its files in one chunk, thus reducing the number of network connections.

It probably wouldn't help that much. Files are broken up into multiple packets, each of which have to make it through, and any of which may hit the floor.
____________


BOINC WIKI

1 · 2 · Next

Message boards : Number crunching : Bandwith solution suggestion

Copyright © 2014 University of California