Message boards :
Number crunching :
Bandwith solution suggestion
Message board moderation
Author | Message |
---|---|
Pilot Send message Joined: 18 May 99 Posts: 534 Credit: 5,475,482 RAC: 0 |
Perhaps this has already been thought of, implemented or is not really viable, but it is just a suggestion. Would it be possible to only upload the first unique result of a work unit along with a checksum of the data and its results. All others clients would only have to report the calculated checksum for verification and credit. IE: 1. Client completes a work unit, calculates the Checksum and sends it to BOINC. The client could then enter a state called “PENDING†until step 3 or 4 is reached. 2. BOINC checks database for prior submission of the Work Unit and a matching Checksum. 3. If a matching WU/Checksum is found, BOINC informs Client there is no need to send rest and the client is granted credit. 4. If no matching WU/Checksum is found, then BOINC informs the Client to send all. When we finally figure it all out, all the rules will change and we can start all over again. |
Grant (SSSF) Send message Joined: 19 Aug 99 Posts: 13720 Credit: 208,696,464 RAC: 304 |
Results are generally 22-33kB in size (mostly around 25kB). MB Work Units are around 365kB in size, AP Work Units 8MB in size. Even a huge reduction in the size of the result being returned won't have much of an impact on bandwidth, you've still got all the network communication that goes along with the upload process & just the sheer number of results to be returned. So far most of the network problems (including not being able to upload (at least untill the present upload problems)) have been due to download bandwidth issues. Grant Darwin NT |
Jord Send message Joined: 9 Jun 99 Posts: 15184 Credit: 4,362,181 RAC: 3 |
Before you think too difficult scenarios, a possible fix for the waiting uploads is already checked in to Trunk and will (probably) show up in a next client. David 10 July 2009 All scenarios need a new client anyway and how are you going to persuade everyone (including all those not reading the forums) that the latest version is the best thing since sliced bread? |
Richard Haselgrove Send message Joined: 4 Jul 99 Posts: 14649 Credit: 200,643,578 RAC: 874 |
Before you think too difficult scenarios, a possible fix for the waiting uploads is already checked in to Trunk and will (probably) show up in a next client. I do hope we get a chance to test that in Beta (it was too late for v6.6.37) before it becomes recommended - get a chance to find and iron out whatever those unspecified "problems" were that it caused last time. |
Grant (SSSF) Send message Joined: 19 Aug 99 Posts: 13720 Credit: 208,696,464 RAC: 304 |
All scenarios need a new client anyway and how are you going to persuade everyone (including all those not reading the forums) that the latest version is the best thing since sliced bread? A few ntoices in the logs & popups from the system tray. Older versions no longer supported- upgrade or move on. Grant Darwin NT |
Jord Send message Joined: 9 Jun 99 Posts: 15184 Credit: 4,362,181 RAC: 3 |
A few notices in the logs & popups from the system tray. Hooya, and get all the complaints that BOINC is being very annoying that way, where to turn it off and if that's not possible, where to get the client before that, that didn't do that. ;-) Ricky wrote: I do hope we get a chance to test that in Beta (it was too late for v6.6.37) before it becomes recommended - get a chance to find and iron out whatever those unspecified "problems" were that it caused last time. I hear ya, fingers crossed. I keep wondering why it was taken out in the first place. and since you're good with searching through old emails and such... ;-) |
1mp0£173 Send message Joined: 3 Apr 99 Posts: 8423 Credit: 356,897 RAC: 0 |
I do hope we get a chance to test that in Beta (it was too late for v6.6.37) before it becomes recommended - get a chance to find and iron out whatever those unspecified "problems" were that it caused last time. I can't do a build from the trunk (yet, I'm working on it) but it is something that can be tested right here, and this is a perfect time since SETI is struggling. That is, the function can be tested, but the desired effect won't be noticed until it is widely deployed. |
zoom3+1=4 Send message Joined: 30 Nov 03 Posts: 65709 Credit: 55,293,173 RAC: 49 |
So far I just have to wait for the uploads to clear up due to a lack of available bandwidth, As so far there's no official response in Technical by Matt from the 14th as to why events with the upload server happened as they did. Some openness would be appreciated when they have the time. S@H simply needs more Fiber in Its diet(or something to give It the runs maybe;)). The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's |
1mp0£173 Send message Joined: 3 Apr 99 Posts: 8423 Credit: 356,897 RAC: 0 |
Perhaps this has already been thought of, implemented or is not really viable, but it is just a suggestion. Would it be possible to only upload the first unique result of a work unit along with a checksum of the data and its results. All others clients would only have to report the calculated checksum for verification and credit. This is a clever idea. The problem is: valid results are not necessarily identical, so the checksums will be different even on valid results. |
Bill Walker Send message Joined: 4 Sep 99 Posts: 3868 Credit: 2,697,267 RAC: 0 |
I thought about that, but is there some other indicator of validity (call it a Super Checksum) that takes up less bits than the full results package? |
Voyager Send message Joined: 2 Nov 99 Posts: 602 Credit: 3,264,813 RAC: 0 |
a non-tech thought.would it help to upload oldest results first(those closest to due date)and work backwards through them? |
1mp0£173 Send message Joined: 3 Apr 99 Posts: 8423 Credit: 356,897 RAC: 0 |
I suspect that by the time you produce a description that accurately represents the result it'll turn out to be nearly the size of the result. |
1mp0£173 Send message Joined: 3 Apr 99 Posts: 8423 Credit: 356,897 RAC: 0 |
a non-tech thought.would it help to upload oldest results first(those closest to due date)and work backwards through them? If you defer those that don't need to be uploaded urgently, yes it would. |
John McLeod VII Send message Joined: 15 Jul 99 Posts: 24806 Credit: 790,712 RAC: 0 |
Perhaps this has already been thought of, implemented or is not really viable, but it is just a suggestion. Would it be possible to only upload the first unique result of a work unit along with a checksum of the data and its results. All others clients would only have to report the calculated checksum for verification and credit. The clients are not in communication with each other, and may not be in communication with the server before the upload starts. Testing to see if some other result was uploaded for that WU involves asking the DB server - this is NOT going to happen. What happens if the first task is invalid. BOINC WIKI |
Pilot Send message Joined: 18 May 99 Posts: 534 Credit: 5,475,482 RAC: 0 |
State 4 exist and the client is instructed to report all. When we finally figure it all out, all the rules will change and we can start all over again. |
Josef W. Segur Send message Joined: 30 Oct 99 Posts: 4504 Credit: 1,414,761 RAC: 0 |
I took an MB result file of 25180 bytes, stripped out lines which the Validator ignores, and ended with a 776 byte file. Standalone test code derived from the Berkeley validator reports the stripped and unstripped versions "Strongly similar". The 776 bytes could be reduced much more if the XML were removed and numerical values sent as binary. However, there would need to be significant changes to BOINC servers and the core client to handle the additional steps, saving the real result until told it isn't needed, etc. My firm belief is there's no bandwidth problem on the link going to SSL, rather it's a transaction rate problem. The smaller uploads would be no help for that, more likely make it worse. Joe |
Pilot Send message Joined: 18 May 99 Posts: 534 Credit: 5,475,482 RAC: 0 |
Thank you, I understand. I had picked up from other threads that perhaps there needed to be additional Fiber or bandwidth to help overall performance. When we finally figure it all out, all the rules will change and we can start all over again. |
Richard Haselgrove Send message Joined: 4 Jul 99 Posts: 14649 Credit: 200,643,578 RAC: 874 |
|
MarkJ Send message Joined: 17 Feb 08 Posts: 1139 Credit: 80,854,192 RAC: 5 |
@ Joe, Ned, I wonder if the superhost (if it ever happens) would help this? It should be able to upload all its files in one chunk, thus reducing the number of network connections. BOINC blog |
John McLeod VII Send message Joined: 15 Jul 99 Posts: 24806 Credit: 790,712 RAC: 0 |
@ Joe, Ned, It probably wouldn't help that much. Files are broken up into multiple packets, each of which have to make it through, and any of which may hit the floor. BOINC WIKI |
©2024 University of California
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.