Bandwith solution suggestion

Message boards : Number crunching : Bandwith solution suggestion

Author	Message
Pilot Send message Joined: 18 May 99 Posts: 534 Credit: 5,160,706 RAC: 0	Message 918129 - Posted: 15 Jul 2009, 17:54:17 UTC
	Perhaps this has already been thought of, implemented or is not really viable, but it is just a suggestion. Would it be possible to only upload the first unique result of a work unit along with a checksum of the data and its results. All others clients would only have to report the calculated checksum for verification and credit. IE: 1. Client completes a work unit, calculates the Checksum and sends it to BOINC. The client could then enter a state called â€œPENDINGâ€ until step 3 or 4 is reached. 2. BOINC checks database for prior submission of the Work Unit and a matching Checksum. 3. If a matching WU/Checksum is found, BOINC informs Client there is no need to send rest and the client is granted credit. 4. If no matching WU/Checksum is found, then BOINC informs the Client to send all. ____________ When we finally figure it all out, all the rules will change and we can start all over again.
	ID: 918129 ·

Grant (SSSF) Send message Joined: 19 Aug 99 Posts: 5059 Credit: 40,817,641 RAC: 52,721	Message 918131 - Posted: 15 Jul 2009, 18:01:16 UTC - in response to Message 918129.
	Results are generally 22-33kB in size (mostly around 25kB). MB Work Units are around 365kB in size, AP Work Units 8MB in size. Even a huge reduction in the size of the result being returned won't have much of an impact on bandwidth, you've still got all the network communication that goes along with the upload process & just the sheer number of results to be returned. So far most of the network problems (including not being able to upload (at least untill the present upload problems)) have been due to download bandwidth issues. ____________ Grant Darwin NT.
	ID: 918131 ·

Ageless Send message Joined: 9 Jun 99 Posts: 11635 Credit: 1,999,039 RAC: 4,234	Message 918134 - Posted: 15 Jul 2009, 18:06:45 UTC - in response to Message 918129.
	Before you think too difficult scenarios, a possible fix for the waiting uploads is already checked in to Trunk and will (probably) show up in a next client. David 10 July 2009 - client: restored code for project-wide backoff on file uploads and downloads. I originally added this on 30 Sept 2005 and disabled it 2 weeks later because there were reports of problems. However, we need this functionality (e.g. on GPU hosts with hundreds of files to upload, we need to back off after a few failures, not try all of them). I added messages (<file_xfer_debug>) so you can see what's going on. Fixes #932. All scenarios need a new client anyway and how are you going to persuade everyone (including all those not reading the forums) that the latest version is the best thing since sliced bread? ____________ Jord - BOINC FAQ Service - BOINC User Wiki Real is just a matter of perception.
	ID: 918134 ·

Richard Haselgrove Volunteer tester Send message Joined: 4 Jul 99 Posts: 7458 Credit: 39,640,941 RAC: 33,302	Message 918136 - Posted: 15 Jul 2009, 18:11:33 UTC - in response to Message 918134.
	Before you think too difficult scenarios, a possible fix for the waiting uploads is already checked in to Trunk and will (probably) show up in a next client. David 10 July 2009 - client: restored code for project-wide backoff on file uploads and downloads. I originally added this on 30 Sept 2005 and disabled it 2 weeks later because there were reports of problems. However, we need this functionality (e.g. on GPU hosts with hundreds of files to upload, we need to back off after a few failures, not try all of them). I added messages (<file_xfer_debug>) so you can see what's going on. Fixes #932. All scenarios need a new client anyway and how are you going to persuade everyone (including all those not reading the forums) that the latest version is the best thing since sliced bread? I do hope we get a chance to test that in Beta (it was too late for v6.6.37) before it becomes recommended - get a chance to find and iron out whatever those unspecified "problems" were that it caused last time.
	ID: 918136 ·

Grant (SSSF) Send message Joined: 19 Aug 99 Posts: 5059 Credit: 40,817,641 RAC: 52,721	Message 918142 - Posted: 15 Jul 2009, 18:14:25 UTC - in response to Message 918134.
	All scenarios need a new client anyway and how are you going to persuade everyone (including all those not reading the forums) that the latest version is the best thing since sliced bread? A few ntoices in the logs & popups from the system tray. Older versions no longer supported- upgrade or move on. ____________ Grant Darwin NT.
	ID: 918142 ·

Ageless Send message Joined: 9 Jun 99 Posts: 11635 Credit: 1,999,039 RAC: 4,234	Message 918153 - Posted: 15 Jul 2009, 18:25:35 UTC - in response to Message 918142.
	A few notices in the logs & popups from the system tray. Older versions no longer supported- upgrade or move on. Hooya, and get all the complaints that BOINC is being very annoying that way, where to turn it off and if that's not possible, where to get the client before that, that didn't do that. ;-) Ricky wrote: I do hope we get a chance to test that in Beta (it was too late for v6.6.37) before it becomes recommended - get a chance to find and iron out whatever those unspecified "problems" were that it caused last time. I hear ya, fingers crossed. I keep wondering why it was taken out in the first place. and since you're good with searching through old emails and such... ;-) ____________ Jord - BOINC FAQ Service - BOINC User Wiki Real is just a matter of perception.
	ID: 918153 ·

1mp0£173 Volunteer tester Send message Joined: 3 Apr 99 Posts: 8423 Credit: 356,897 RAC: 0	Message 918187 - Posted: 15 Jul 2009, 19:40:09 UTC - in response to Message 918136.
	I do hope we get a chance to test that in Beta (it was too late for v6.6.37) before it becomes recommended - get a chance to find and iron out whatever those unspecified "problems" were that it caused last time. I can't do a build from the trunk (yet, I'm working on it) but it is something that can be tested right here, and this is a perfect time since SETI is struggling. That is, the function can be tested, but the desired effect won't be noticed until it is widely deployed. ____________
	ID: 918187 ·

VBobier Send message Joined: 30 Nov 03 Posts: 40795 Credit: 32,089,953 RAC: 19,313	Message 918188 - Posted: 15 Jul 2009, 19:45:55 UTC Last modified: 15 Jul 2009, 19:58:00 UTC
	So far I just have to wait for the uploads to clear up due to a lack of available bandwidth, As so far there's no official response in Technical by Matt from the 14th as to why events with the upload server happened as they did. Some openness would be appreciated when they have the time. S@H simply needs more Fiber in Its diet(or something to give It the runs maybe;)). ____________ BSG Anthem My Facebook page
	ID: 918188 ·

1mp0£173 Volunteer tester Send message Joined: 3 Apr 99 Posts: 8423 Credit: 356,897 RAC: 0	Message 918195 - Posted: 15 Jul 2009, 20:03:07 UTC - in response to Message 918129.
	Perhaps this has already been thought of, implemented or is not really viable, but it is just a suggestion. Would it be possible to only upload the first unique result of a work unit along with a checksum of the data and its results. All others clients would only have to report the calculated checksum for verification and credit. IE: 1. Client completes a work unit, calculates the Checksum and sends it to BOINC. The client could then enter a state called â€œPENDINGâ€ until step 3 or 4 is reached. 2. BOINC checks database for prior submission of the Work Unit and a matching Checksum. 3. If a matching WU/Checksum is found, BOINC informs Client there is no need to send rest and the client is granted credit. 4. If no matching WU/Checksum is found, then BOINC informs the Client to send all. This is a clever idea. The problem is: valid results are not necessarily identical, so the checksums will be different even on valid results. ____________
	ID: 918195 ·

Bill Walker Send message Joined: 4 Sep 99 Posts: 2861 Credit: 1,778,689 RAC: 216	Message 918228 - Posted: 15 Jul 2009, 21:24:26 UTC - in response to Message 918195.
	This is a clever idea. The problem is: valid results are not necessarily identical, so the checksums will be different even on valid results. I thought about that, but is there some other indicator of validity (call it a Super Checksum) that takes up less bits than the full results package? ____________
	ID: 918228 ·

Voyager Volunteer tester Send message Joined: 2 Nov 99 Posts: 602 Credit: 3,264,813 RAC: 0	Message 918235 - Posted: 15 Jul 2009, 21:36:37 UTC
	a non-tech thought.would it help to upload oldest results first(those closest to due date)and work backwards through them?
	ID: 918235 ·

1mp0£173 Volunteer tester Send message Joined: 3 Apr 99 Posts: 8423 Credit: 356,897 RAC: 0	Message 918243 - Posted: 15 Jul 2009, 22:00:15 UTC - in response to Message 918228.
	This is a clever idea. The problem is: valid results are not necessarily identical, so the checksums will be different even on valid results. I thought about that, but is there some other indicator of validity (call it a Super Checksum) that takes up less bits than the full results package? I suspect that by the time you produce a description that accurately represents the result it'll turn out to be nearly the size of the result. ____________
	ID: 918243 ·

1mp0£173 Volunteer tester Send message Joined: 3 Apr 99 Posts: 8423 Credit: 356,897 RAC: 0	Message 918244 - Posted: 15 Jul 2009, 22:01:14 UTC - in response to Message 918235.
	a non-tech thought.would it help to upload oldest results first(those closest to due date)and work backwards through them? If you defer those that don't need to be uploaded urgently, yes it would. ____________
	ID: 918244 ·

John McLeod VII Volunteer developer Volunteer tester Send message Joined: 15 Jul 99 Posts: 22343 Credit: 439,209 RAC: 396	Message 918264 - Posted: 15 Jul 2009, 23:04:46 UTC - in response to Message 918129.
	Perhaps this has already been thought of, implemented or is not really viable, but it is just a suggestion. Would it be possible to only upload the first unique result of a work unit along with a checksum of the data and its results. All others clients would only have to report the calculated checksum for verification and credit. IE: 1. Client completes a work unit, calculates the Checksum and sends it to BOINC. The client could then enter a state called â€œPENDINGâ€ until step 3 or 4 is reached. 2. BOINC checks database for prior submission of the Work Unit and a matching Checksum. 3. If a matching WU/Checksum is found, BOINC informs Client there is no need to send rest and the client is granted credit. 4. If no matching WU/Checksum is found, then BOINC informs the Client to send all. The clients are not in communication with each other, and may not be in communication with the server before the upload starts. Testing to see if some other result was uploaded for that WU involves asking the DB server - this is NOT going to happen. What happens if the first task is invalid. ____________ BOINC WIKI
	ID: 918264 ·

Pilot Send message Joined: 18 May 99 Posts: 534 Credit: 5,160,706 RAC: 0	Message 918269 - Posted: 15 Jul 2009, 23:16:21 UTC - in response to Message 918264.
	State 4 exist and the client is instructed to report all. ____________ When we finally figure it all out, all the rules will change and we can start all over again.
	ID: 918269 ·

Josef W. Segur Volunteer developer Volunteer tester Send message Joined: 30 Oct 99 Posts: 3932 Credit: 934,327 RAC: 15	Message 918273 - Posted: 15 Jul 2009, 23:31:05 UTC - in response to Message 918243.
	This is a clever idea. The problem is: valid results are not necessarily identical, so the checksums will be different even on valid results. I thought about that, but is there some other indicator of validity (call it a Super Checksum) that takes up less bits than the full results package? I suspect that by the time you produce a description that accurately represents the result it'll turn out to be nearly the size of the result. I took an MB result file of 25180 bytes, stripped out lines which the Validator ignores, and ended with a 776 byte file. Standalone test code derived from the Berkeley validator reports the stripped and unstripped versions "Strongly similar". The 776 bytes could be reduced much more if the XML were removed and numerical values sent as binary. However, there would need to be significant changes to BOINC servers and the core client to handle the additional steps, saving the real result until told it isn't needed, etc. My firm belief is there's no bandwidth problem on the link going to SSL, rather it's a transaction rate problem. The smaller uploads would be no help for that, more likely make it worse. Joe
	ID: 918273 ·

Pilot Send message Joined: 18 May 99 Posts: 534 Credit: 5,160,706 RAC: 0	Message 918278 - Posted: 15 Jul 2009, 23:38:18 UTC - in response to Message 918273.
	Thank you, I understand. I had picked up from other threads that perhaps there needed to be additional Fiber or bandwidth to help overall performance. ____________ When we finally figure it all out, all the rules will change and we can start all over again.
	ID: 918278 ·

Richard Haselgrove Volunteer tester Send message Joined: 4 Jul 99 Posts: 7458 Credit: 39,640,941 RAC: 33,302	Message 918401 - Posted: 16 Jul 2009, 9:04:42 UTC
	@ Joe, Ned, Could you sanity-check my "data aggregator" suggestion at #918399, please?
	ID: 918401 ·

MarkJ Volunteer tester Send message Joined: 17 Feb 08 Posts: 912 Credit: 14,549,661 RAC: 8,479	Message 918427 - Posted: 16 Jul 2009, 12:42:24 UTC - in response to Message 918401.
	@ Joe, Ned, Could you sanity-check my "data aggregator" suggestion at #918399, please? I wonder if the superhost (if it ever happens) would help this? It should be able to upload all its files in one chunk, thus reducing the number of network connections. ____________ BOINC blog
	ID: 918427 ·

John McLeod VII Volunteer developer Volunteer tester Send message Joined: 15 Jul 99 Posts: 22343 Credit: 439,209 RAC: 396	Message 918435 - Posted: 16 Jul 2009, 13:15:28 UTC - in response to Message 918427.
	@ Joe, Ned, Could you sanity-check my "data aggregator" suggestion at #918399, please? I wonder if the superhost (if it ever happens) would help this? It should be able to upload all its files in one chunk, thus reducing the number of network connections. It probably wouldn't help that much. Files are broken up into multiple packets, each of which have to make it through, and any of which may hit the floor. ____________ BOINC WIKI
	ID: 918435 ·

1 · 2 · Next

Message boards : Number crunching : Bandwith solution suggestion