Idea for Boinc.....

Just pondering....

I know that bandwidth conservation may not be an issue for other Boinc projects, but it is for Seti. At least right now.

I wonder if.....

With the increasing number of fast hosts and faster GPUs, cache sizes regardless of a user's settings are only going to increase.
Every time a host reports a completed WU or requests work, their entire cache list is sent along with the request. When cache sizes get into the 1000's of WUs, this can be a sizable transmission.

Would it be possible or practical for Boinc to transmit only information relating to what has changed in the list since the last host update, and not the entire list every time? And perhaps the entire list only every 10 completed requests to be sure everything stays in synch?

Would this be cumbersome to implement?
Would the bandwidth saved be worth the effort?
Just asking for trouble?

Just kitty thoughts.
____________
******

"Ask not, what your kitty can do for you. Ask what you can do for your kitty."

As it is kitten, so shall it be done.

I agree with this. Even when downloads/uploads are off, feeder is empty, etc, there is still a decent amount of traffic from just people trying to get new work. Perhaps there could be a quick comparison done between the host and the server of just the NUMBER of WU's on board, instead of sending the entire task list. Then, if there is a discrepancy (see: ghosts), the full file could be sent to see which files are missing. The server is already having to query the DB to compare task lists, surely it wouldn't be hard to just compare a number. I can't imagine it would add much if any extra load to the servers, and would surely save some bandwidth.

Then, it would just be:

Requesting new tasks, have 4250 on board.
ACK, 4250 on board, DB agrees.
Here's more tasks.

Or:

Requesting new tasks, have 4250 on board.
Error: should be 4275 on board.
Sending entire task list.
etc.
____________

Would it be possible or practical for Boinc to transmit only information relating to what has changed in the list since the last host update, and not the entire list every time? And perhaps the entire list only every 10 completed requests to be sure everything stays in synch?

Elegant idea. Can`t be that simple, can it.......?

Here`s hoping it is.

Regards,

Andy

It would surely require a client side update, as well as compatible server side changes, but I wouldn't think it would be TOO bad...
____________

The server code is prepared to handle requests which don't have those lists of tasks on hand, some very old versions of the core client don't send them. So it's certainly possible to have a new core client which only sends the lists once in every 5 or 10 requests, etc.

Note that requests without the lists wouldn't be useful for the resend lost work capability, but maybe having only a fraction of requests triggering the comparison would reduce the database load it causes to a negligible amount.

However, another possibility to consider is having the request gzipped before sending. That would reduce the size considerably, and the method of sending a request is very similar to uploading a result (which already has an optional gzip feature). A 19236 byte request listing 37 in progress compresses to 3301 bytes, for instance.

The server code is prepared to handle requests which don't have those lists of tasks on hand, some very old versions of the core client don't send them. So it's certainly possible to have a new core client which only sends the lists once in every 5 or 10 requests, etc.

Note that requests without the lists wouldn't be useful for the resend lost work capability, but maybe having only a fraction of requests triggering the comparison would reduce the database load it causes to a negligible amount.

However, another possibility to consider is having the request gzipped before sending. That would reduce the size considerably, and the method of sending a request is very similar to uploading a result (which already has an optional gzip feature). A 19236 byte request listing 37 in progress compresses to 3301 bytes, for instance.

Joe

Why not make the workunits bigger?
Every server at Seti would benefit from it.

I'll try again.

Why not make the workunits larger in datasize?
Every server at Seti would benefit from it.
Instead of about 300 Kbytes files per WU, increase it to let's say 1 Mbyte.
That will decrease the number of workunits by a third.
The network load for WU's surely must drop.

I don't believe it's a bandwidth problem but that network requests are coming in too often to SETI due to the big amount of WU's. The solution to this is to make fewer WU's by making them larger. The crunching time on computers beside SETI has no relevance to this problem.

Meeeooowwwrrr.

For over 12 yrs the data size has been the same and after we have processed them the results are stored on the science database, with up to 30 items of interest for each result.

If the data size were increased, do we still only look for a max of thirty items of interest and throw away any other items found,
OR do you have some magic way of storing these larger tasks (twice size, 60 items. etc.) that would be compatible with all the older data.

One AstroPulse task took 8 retries and 45 minutes to download.
And MB work DownLoads are very slow and need several retries, too.

Bigger tasks don't seem like a good idea, zipping them, if it's worth it?

(I noticed CPDN, also was slow and took several retries to DownLoad these (big)
WUs).
I also lowered my cache from 4 to 3 and 2 days, depending of the host.
When BOINC (6.12.33/34) asks for work and there is no work, a few minutes later
a new request is made and it takes some time, before it downloaded enough or
waits over an hour, to request again.
____________


Knight Who Says Ni N!, OUT numbered.................

For over 12 yrs the data size has been the same and after we have processed them the results are stored on the science database, with up to 30 items of interest for each result.

If the data size were increased, do we still only look for a max of thirty items of interest and throw away any other items found,
OR do you have some magic way of storing these larger tasks (twice size, 60 items. etc.) that would be compatible with all the older data.

I'm a bit confused now.
I have looked at the cricket graph and got this.

Graphs for gigabitethernet2_3

Values at last update:
Average bits in (for the day):
Cur: 92.94 Mbits/sec
Avg: 93.57 Mbits/sec
Max: 95.96 Mbits/sec
Average bits out (for the day):
Cur: 12.13 Mbits/sec
Avg: 12.55 Mbits/sec
Max: 13.70 Mbits/sec

Last updated at Fri Oct 7 14:05:18 2011

Is gigabitethernet2_3 only for SETI or it is for Berkley University and others?
To me it looks like they are sending 8 times more data then they are getting.
My contribution is getting 360 Kb and sending 30 kB per WU, that is i'm sending less data that i'm get.

Like you said it's indeed a bandwidth problem. To me it's seems there are other projects than SETI who maybe have to lookup their routines.