Brainstorming facilities for DIAL-UP BOINCers - pls contribute!

Message boards : Number crunching : Brainstorming facilities for DIAL-UP BOINCers - pls contribute!
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · 4 · Next

AuthorMessage
Profile [Cx]
Avatar

Send message
Joined: 25 Jul 05
Posts: 141
Credit: 25,742
RAC: 0
United Kingdom
Message 228059 - Posted: 8 Jan 2006, 20:03:14 UTC

I've just now taken to brainstorming ideas for features / facilities to request of Berkeley to add to BOINC to perhaps make the lives of dial-up BOINCers a bit easier.

Back over in the Sarcasm thread (darn, I should have copied the URL -- sorry!), I started an exchange with Tony (mmciastro), who's said he was on dial-up not long ago, and mentioned the discomfort of dealing with (uploading) completed WUs...

SO... I just thought I'd start a thread in the hopes that *specifically* DIAL-UP users (old and new) and those with insights to BOINCing without broadband would contribute their thoughts, ideas, etc.. and perhaps Berkeley may find this thread useful in creating updates for BOINC...

So, how about it? =)

.Cx. /#\\...]
ID: 228059 · Report as offensive
Profile Tern
Volunteer tester
Avatar

Send message
Joined: 4 Dec 03
Posts: 1122
Credit: 13,376,822
RAC: 44
United States
Message 228104 - Posted: 8 Jan 2006, 21:32:58 UTC

I'm sure there are a lot of things that can/should be done for dialup users, but the FIRST thing that MUST be done is simply having some way to _identify_ who is on dialup. This means having a field in the "general preferences", in each venue, for "Are you on dialup?".

Once you have that, then it should set return_results_immediately. Never having used BOINC on dialup, I don't know what else should be changed. The _basic_ idea is that dialup users should have "everything happen at once when they connect", with as few delays until the next connection as possible. Always-on users, it doesn't matter - delaying reporting, exponential fallback on retries, etc. is fine.
ID: 228104 · Report as offensive
Profile [Cx]
Avatar

Send message
Joined: 25 Jul 05
Posts: 141
Credit: 25,742
RAC: 0
United Kingdom
Message 228188 - Posted: 9 Jan 2006, 0:40:06 UTC - in response to Message 228104.  

@Bill Michael

Thanks for your input. Yes, that makes sense and sounds reasonable.
At a guess, I'd wager adding an identifier / flag to allow dial-up users to be distinguished from non-dial-up users shouldn't be too difficult to code in...
However, not being a part of the main design team, I can't speak for that.

The "everything happen at once" idea is fine, except that I see a potential problem:

Given a situation where a few/several completed WU's are ready to go when a dial-up user connects to UPLOAD, and BOINC tries to send them all, suppose something happens and they lose the signal/connection part-way through.

I'd suppose that multi-sending would "multi-thread" or share the upload bandwidth between all WUs attempting to be uploaded...

WU1:#..#..#..#..#..#..#..#..#| 90%
WU2:.#..#..#..#..#..#..#..#..| 80%
WU3:..#..#..#..#..#..#..#..#.| 80%

3 incomplete WU uploads -- requiring another attempt at reconnect
100% time wasted.

vs

WU1:########## 100%
WU2:..........########## 100%
WU3:....................#####| 50%

1 incomplete WU upload.
20% time wasted. (in this example)

Whereas if BOINC was "aware" of the dialup situation, and change its handling of uploads to suit, perhaps it could reduce the frustration over lost/incomplete/need-to-resend completed WUs..?? i.e. thinking of uploading WUs one at a time, to ensure whole WUs upload with greater frequency of success..?

Worst case scenario, couldn't disruption of the upload potentially lose the cruncher the WU and therefore the credits? Lost WUs would be a frustrating waste of time/CPU cycles, no?

ID: 228188 · Report as offensive
Profile ksnash

Send message
Joined: 28 Nov 99
Posts: 402
Credit: 528,725
RAC: 0
United States
Message 228194 - Posted: 9 Jan 2006, 1:08:28 UTC

I had the exact messages you put up. I do not have dial up. All I got was I have a sorry router. I had a Bellsouth Westell modem. Can't change. US Robotics 8514 serving 5 computers. I thought US Robotics was pretty good. The computers can download workunits but not return them properly. I got so many workunits trying to return that my internet connection was pretty much cutoff. I couldn't connect to anything on internet. If my equipment is so bad then would some people send me their equipment so I can continue to contribute. I had to stop the network connections within my home network.

If what I say is so bad why was it mentioned as problem in dev mail list by those who hate me so.
ID: 228194 · Report as offensive
Profile ksnash

Send message
Joined: 28 Nov 99
Posts: 402
Credit: 528,725
RAC: 0
United States
Message 228197 - Posted: 9 Jan 2006, 1:14:57 UTC - in response to Message 228194.  

I had the exact messages you put up. I do not have dial up. All I got was I have a sorry router. I had a Bellsouth Westell modem. Can't change. US Robotics 8514 serving 5 computers. I thought US Robotics was pretty good. The computers can download workunits but not return them properly. I got so many workunits trying to return that my internet connection was pretty much cutoff. I couldn't connect to anything on internet. If my equipment is so bad then would some people send me their equipment so I can continue to contribute. I had to stop the network connections within my home network.

If what I say is so bad why was it mentioned as problem in dev mail list by those who hate me so.


The only project that continued to have connection troubles from this was setiathome.

ID: 228197 · Report as offensive
Dorphas
Avatar

Send message
Joined: 16 May 99
Posts: 118
Credit: 8,007,247
RAC: 0
United States
Message 228198 - Posted: 9 Jan 2006, 1:17:39 UTC

all i can get right now where i live is dialup. i have no problems with it and boinc.i just keep a good cache and let them crunch away. when a wu is completed boinc tries to connect. when it doesn't because i am not online no problem. it will retry that wu later. and so on. lots of times i log on and have 20 or more wu to upload. they will not upload until the time boinc has set them to retry. once i log on i just manually send them and away they go. i get more wu downloaded and the cycle repeats. other than the slower speed of dialup i have no problems or issues with it and boinc.
ID: 228198 · Report as offensive
Profile Keck_Komputers
Volunteer tester
Avatar

Send message
Joined: 4 Jul 99
Posts: 1575
Credit: 4,152,111
RAC: 1
United States
Message 228222 - Posted: 9 Jan 2006, 2:19:18 UTC

Using the 'network activities suspended/enabled' menu item is a big help for intermittant connections. In fact I think if this could be automatically set as BOINC detects an internet connection the problems would be practically solved. The big problem with this idea is that BOINC does not detect internet connections reliably.
BOINC WIKI

BOINCing since 2002/12/8
ID: 228222 · Report as offensive
John McLeod VII
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jul 99
Posts: 24806
Credit: 790,712
RAC: 0
United States
Message 228225 - Posted: 9 Jan 2006, 2:40:23 UTC - in response to Message 228222.  

Using the 'network activities suspended/enabled' menu item is a big help for intermittant connections. In fact I think if this could be automatically set as BOINC detects an internet connection the problems would be practically solved. The big problem with this idea is that BOINC does not detect internet connections reliably.

It is also a nearly impossible task to detect whether a network is really connected. At one time I had a LAN whose link to the outside world was a router/switch/firewall attached to a POTS modem. It was absoloutely impossible for a computer to programatically figure out the state of that modem. It was easy to figure out that the computer was attached to a network at 100MBits, but it was not possible to detect that modem. It was supposed to auto dial (and it did, finishing just after most programs gave up on getting a response). Similar problems can occur with other setups where a link that is not visible from the local computer is normally not connected to the network.


BOINC WIKI
ID: 228225 · Report as offensive
Jack Gulley

Send message
Joined: 4 Mar 03
Posts: 423
Credit: 526,566
RAC: 0
United States
Message 228247 - Posted: 9 Jan 2006, 3:11:17 UTC

Actually this whole subject is one of the major problems I have with BOINC, it tries to "delay" things happening, when from basic Queuing Theory that is not what you want to do in the name of efficiency. You want to get as much through the pipe as fast as you can. It is not the objective of Queuing Theory and computers to play fair and waste time on the chance that someone else might need to use it. This "fair use" actually wastes bandwidth and time. There is no provision in Queuing Theory for being fair!

Although you can be fair, as long as it does not reduce efficiency.

The only justification I have seen in some of the design of BOINC is so that "the server" is not overloaded. It can not be that the Network is over loaded, as that has a built in self limiting design, and in BOINC/Seti's case, is no where near being overloaded. Even when it was handling the smaller load from Seti Classic at the same time. All that BOINC does is delay the peak load until later and later and try to smooth it out. Wasting bits of bandwidth here and there. The end result is worse than what happens without all the built in delays.

Sorry, but the design objective of a computer or a network is to get as much done as fast as possible. If it is more efficient to allow one process or group of processes to hog the resources first, then the rest will just have to wait in the name of efficiency. Any attempt to make Queuing Theory "play nice and fair" will fail, because its objective is efficient use of resources. The TCP/IP protocol is a contention protocol that handles such overloads on networks, is based on Queuing Theory, and there is no reason to try to tweak it like BOINC does. It drops the last requests if it has to, and TCP/IP will retry again several times before erring out. But because a request is dropped and errors out, there is no reason that it can't be retried after a short delay, and not have to wait 10 minutes or more.

Queuing Theory says the first in with a request gets to use the resource first. If it needs to make another request, it should not have to wait before it can make another request. It should have the right to compete again, if not just continue with its additional requests, if it is more efficient to allow it to do so. Consider the simple case of only one system needing to make a series of requests. Why should it have to wait between requests if it is the only one needing the resource? The bandwidth is being wasted and the one system is being delayed in its operations. A system has no way to know if another will be contending for a resource so it should just ignore that possibility and try anyway.

The server will limit this contention process by the number of transfers it allows to proceed at one time. If well designed it will dynamically adjust this number based on how much of its bandwidth and buffer space is being used by the current requests.

There is really no need to treat DSL/CABLE users different than Dial-Up. Both should automatically attempt to upload a result if there is a connection present, without any delay. Once uploaded, its Report should automatically be allowed to follow as soon as a connection can be made. No artificial delays should be imposed. If the server needs a delay between receiving an upload and its Report, then the server should provide for that delay within its own structure, not impose an arbitrary delay in the HOST sending the data.

The current design as I understand it does restrict a BOINC host from trying to send/receive more than two result at a time. This is a practical limitation, as it reduces the resources required by the server to handle requests from a large number of Hosts. And it limits how long an individual transfer will take on a Host. This is one of the practical deviations from pure Queuing Theory that takes into consideration error recovery and the lost efficiency caused by errors. Trade offs here can be quite subjective and impossible to prove, as long as they are reasonable. But the Retry back-off is a bit of a problem. Too long, should never be over an hour at most. HOSTs making a large number of request when the server is down does not hurt anything, as its not doing anything anyway. When it is busy, these request are small and take up little bandwidth to receive, respond to and ignore.

I agree that there should be a Try all transfers now button that cancels all Retry back-offs. Really, there should be two such buttons, Try all upload now and Try all download now. This is necessary for the Dial-up users and useful for Broadband users as well. No reason to even consider limiting it to Dial-up users, so no need to have some way of telling what type of connection a HOST is currently using. Another option would be for the program to detect the presence of an Internet connection and permission to use it and just start sending/ receiving. But Dial-up users might object to that if they are connecting to do some other online work.

The issue of too many request all at the same time should be handled by the limits of the network bandwidth and the limits of the server. The server does a good job of this to a point. It just does not allow a HOST to send anything if it is too busy to receive it. The only flaw I see in how the BOINC server handles this part of it, is there is currently no explicit "go away, to busy right now" response from the server. Instead, the HOST just has to set there and timeout with an error and then back-off and retry later. Actually, the information is there and the BOINC Host should be allowed to use it (RWIN=0). This way, if the server is to busy, that transfer can be set to Retry and the next allowed to connect without having to wait so long.

Adding the Try Upload|Download Now buttons and allowing the Reports to upload without delay would go a long way in fixing BOINC's problems in this area.

There is one problem with changing the way Reports are handled. It would cause some of the add-on programs that try to track results being sent back to miss seeing results. This can and should be handled with a log file, possibly a settable option, that records the information from each WU when it is completed. It would need to include the Project and application name, WU name, download time, start time, stop time, execution time, completion status, and have a field filled in about the WU by the application (such as for Seti the Arc angle or location information, and results information).

After writing this, I am starting to wonder if they have taken Queuing Theory out of Computer Science and System Design courses, or worse, turned it into a "Feel Good about it" course.
ID: 228247 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 228260 - Posted: 9 Jan 2006, 4:02:14 UTC - in response to Message 228222.  

Using the 'network activities suspended/enabled' menu item is a big help for intermittant connections. In fact I think if this could be automatically set as BOINC detects an internet connection the problems would be practically solved. The big problem with this idea is that BOINC does not detect internet connections reliably.

I agree that setting is a big help, here's how I use it:

1. When my host is offline I don't want to have BOINC wasting CPU time trying to make connection. I also don't want it to build up exponential delays for any network activities.

2. When I am online but doing mail fetch, web browsing, etc. it prevents BOINC from delaying those operations. Note that this means I disagree it would be desirable to have BOINC detect when I'm online and automatically enable itself. I leave network activities suspended until I'm ready to take a break.

3. For any project for which BOINC both has uploads to do and is likely to fetch new work, I set "No new work". After that I enable BOINC network activity and go brew a cup of tea or whatever. When I come back the uploads are normally done, and setting "Allow new work" does one single Scheduler contact to both report the uploads and ask for new work.
===========

My suggestion is to have transitioning from network activity suspended to enabled cause a deferral of Scheduler contact if there are uploads ready to be sent. That deferral might be as simple as one minute, or some refinement to tailor it to the amount of uploads might be incorporated.

I think it should apply not only to changing the state manually but also when the transition is done for the time settings in preferences.
                                                        Joe

ID: 228260 · Report as offensive
Profile AlecStaar
Avatar

Send message
Joined: 16 Dec 05
Posts: 260
Credit: 44,472
RAC: 0
United States
Message 228265 - Posted: 9 Jan 2006, 4:13:13 UTC
Last modified: 9 Jan 2006, 4:30:42 UTC

A 'redirect' from here, about this topic, because Cx from this thread url below (first one) posted about THIS one, here:

"Hiya. :) Found your thread. In a similar vein, I've started a thread more specifically for those who take the time to connect via DIAL-UP."

http://setiathome.berkeley.edu/forum_thread.php?id=26801

That all said & quoted? Well, here I am:

SOME INFO. ON SYSTEM TUNING WHICH CAN HELP OVERALL PERFORMANCE OF THIS PROJECT VIA OS TUNING which, even for dialup folks it can help!

(However, it does entail some reading from the url's below I post & considering the diff.'s between dialup &/or cable/dsl connected users):


=======================================================================

APK System Performance Tuneup:
==========================================


http://www.avatar.demon.nl/APKTuneup.html

APK Online Security & Speed Tuneup:
==========================================


http://www.avatar.demon.nl/APK.html

:)

(The 2nd link I originally posted here's GREAT for improving your online speed, which can help (as well as making your system more secure online), but the 1st & NEWER url link above is MORE about tuning/optimization performance tricks/hacks/tips/techniques of your OS + apps which run LOCALLY to your system, & not just tuning IP performance... which the 2nd link above I posted was largely about, but you CAN use many of its settings on dialup as well: I have in the past in fact, a year or two ago, when I was on dialup (on cablemodem now again though thank goodness))

See, when you 'tune' those BOTH? ALL ELSE GAINS...

* Food for thought, but, a ton to read between them BOTH!

Again - both pages for tuning speed &/or security online provide link URL's to Microsoft & other credible online sources for reading what it is you're tuning & how it can affect you for better performance!

IIRC, the .reg files there (saves you time manually editing your registry via regedit.exe) also have the link URL's in them as well as tips on EACH merged setting from MS themselves on what it is you are actually tuning &/or affecting also... it was HOURS of work editing them for that, but worth it imo & those of others that have used them online for years now for this very purpose.

(And, the pages also 'warn' about what you MAY LOSE the ability to do if some settings are applied, in SOME circumstances by doing the trimming of services etc. for example, which for a stand-alone rig (i.e. - no home network LAN usually which the 2nd URL warns what to avoid tuning/trimming on in that case) is not too much usually IF anything @ all - but, this depends on how you use your machine... e.g. - if you run your own system for website development using IIS? You won't be cutting off its services (at least the World Wide Web Publishing one for example)).

ALSO - IF you read my profile, it has a LOAD of tips on how to get the most out of your system... no matter what type it is!

I.E.- Especially IF you run a Windows NT-based OS type (more modern ones like 2000/XP/Server 2003)... @ least for this project & yes, other things.

E.G. - I do some fairly 'radical' things for this project in particular to show more performance & gain from it.

Things like running SETI on a solid-state ramdisk to max out disk based access this project does, but also thing for gaining more CPU (potentially @ least) cycles available by killing off background apps (trayicons, services, etc. & even Explorer.exe shell) while it runs, & it helps!

APK

P.S.=> Once again, on ALL of the above - The tips noted & tricks I use to gain for performance of this project's processing's NOT a substitute for better hardware by any means (@ best, a 20% gain overall of your system's operations), but they DO help you get the BEST out of what YOU own, making you get the best out of what you've got hardware & OS setup-wise!

And, again above all:

Don't overlook using the optimized boinc*.exe & seti4.x.exe clients that either Trux OR Crunch3r have provided... they DO rock (as I mentioned earlier & many times here on these forums), & have DOUBLED my system's abilities in per-unit worktime outputs!

Hope this all helps! apk
http://torry.net/authorsmore.php?id=1781

"The object's hull is made of SOLID neutronium: A single StarShip cannot combat it!" quote Mr. Spock, Star Trek original series, episode title: "The Doomsday Machine"
ID: 228265 · Report as offensive
John McLeod VII
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jul 99
Posts: 24806
Credit: 790,712
RAC: 0
United States
Message 228300 - Posted: 9 Jan 2006, 5:15:46 UTC - in response to Message 228247.  

Actually this whole subject is one of the major problems I have with BOINC, it tries to "delay" things happening, when from basic Queuing Theory that is not what you want to do in the name of efficiency. You want to get as much through the pipe as fast as you can. It is not the objective of Queuing Theory and computers to play fair and waste time on the chance that someone else might need to use it. This "fair use" actually wastes bandwidth and time. There is no provision in Queuing Theory for being fair!

Although you can be fair, as long as it does not reduce efficiency.

The only justification I have seen in some of the design of BOINC is so that "the server" is not overloaded. It can not be that the Network is over loaded, as that has a built in self limiting design, and in BOINC/Seti's case, is no where near being overloaded. Even when it was handling the smaller load from Seti Classic at the same time. All that BOINC does is delay the peak load until later and later and try to smooth it out. Wasting bits of bandwidth here and there. The end result is worse than what happens without all the built in delays.

Sorry, but the design objective of a computer or a network is to get as much done as fast as possible. If it is more efficient to allow one process or group of processes to hog the resources first, then the rest will just have to wait in the name of efficiency. Any attempt to make Queuing Theory "play nice and fair" will fail, because its objective is efficient use of resources. The TCP/IP protocol is a contention protocol that handles such overloads on networks, is based on Queuing Theory, and there is no reason to try to tweak it like BOINC does. It drops the last requests if it has to, and TCP/IP will retry again several times before erring out. But because a request is dropped and errors out, there is no reason that it can't be retried after a short delay, and not have to wait 10 minutes or more.

Queuing Theory says the first in with a request gets to use the resource first. If it needs to make another request, it should not have to wait before it can make another request. It should have the right to compete again, if not just continue with its additional requests, if it is more efficient to allow it to do so. Consider the simple case of only one system needing to make a series of requests. Why should it have to wait between requests if it is the only one needing the resource? The bandwidth is being wasted and the one system is being delayed in its operations. A system has no way to know if another will be contending for a resource so it should just ignore that possibility and try anyway.

The server will limit this contention process by the number of transfers it allows to proceed at one time. If well designed it will dynamically adjust this number based on how much of its bandwidth and buffer space is being used by the current requests.

There is really no need to treat DSL/CABLE users different than Dial-Up. Both should automatically attempt to upload a result if there is a connection present, without any delay. Once uploaded, its Report should automatically be allowed to follow as soon as a connection can be made. No artificial delays should be imposed. If the server needs a delay between receiving an upload and its Report, then the server should provide for that delay within its own structure, not impose an arbitrary delay in the HOST sending the data.

The current design as I understand it does restrict a BOINC host from trying to send/receive more than two result at a time. This is a practical limitation, as it reduces the resources required by the server to handle requests from a large number of Hosts. And it limits how long an individual transfer will take on a Host. This is one of the practical deviations from pure Queuing Theory that takes into consideration error recovery and the lost efficiency caused by errors. Trade offs here can be quite subjective and impossible to prove, as long as they are reasonable. But the Retry back-off is a bit of a problem. Too long, should never be over an hour at most. HOSTs making a large number of request when the server is down does not hurt anything, as its not doing anything anyway. When it is busy, these request are small and take up little bandwidth to receive, respond to and ignore.

I agree that there should be a Try all transfers now button that cancels all Retry back-offs. Really, there should be two such buttons, Try all upload now and Try all download now. This is necessary for the Dial-up users and useful for Broadband users as well. No reason to even consider limiting it to Dial-up users, so no need to have some way of telling what type of connection a HOST is currently using. Another option would be for the program to detect the presence of an Internet connection and permission to use it and just start sending/ receiving. But Dial-up users might object to that if they are connecting to do some other online work.

The issue of too many request all at the same time should be handled by the limits of the network bandwidth and the limits of the server. The server does a good job of this to a point. It just does not allow a HOST to send anything if it is too busy to receive it. The only flaw I see in how the BOINC server handles this part of it, is there is currently no explicit "go away, to busy right now" response from the server. Instead, the HOST just has to set there and timeout with an error and then back-off and retry later. Actually, the information is there and the BOINC Host should be allowed to use it (RWIN=0). This way, if the server is to busy, that transfer can be set to Retry and the next allowed to connect without having to wait so long.

Adding the Try Upload|Download Now buttons and allowing the Reports to upload without delay would go a long way in fixing BOINC's problems in this area.

There is one problem with changing the way Reports are handled. It would cause some of the add-on programs that try to track results being sent back to miss seeing results. This can and should be handled with a log file, possibly a settable option, that records the information from each WU when it is completed. It would need to include the Project and application name, WU name, download time, start time, stop time, execution time, completion status, and have a field filled in about the WU by the application (such as for Seti the Arc angle or location information, and results information).

After writing this, I am starting to wonder if they have taken Queuing Theory out of Computer Science and System Design courses, or worse, turned it into a "Feel Good about it" course.

If it were the network bandwidth that was the problem, you would be correct. However, that it NOT the bottleneck. The bottleneck is opening a link to the database (slow). Once this is done, it is cheap to do several operations at the same time using that connection. The connection is closed at the end of the session, I believe for security purposes. Thus the more items that can be reported or requested at the same time, the lower the load on the server.


BOINC WIKI
ID: 228300 · Report as offensive
John McLeod VII
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jul 99
Posts: 24806
Credit: 790,712
RAC: 0
United States
Message 228302 - Posted: 9 Jan 2006, 5:17:11 UTC

In reading this thread I was hoping for a list of problems that modem users were having with the work scheduler and other aspects.


BOINC WIKI
ID: 228302 · Report as offensive
Profile mr.kjellen
Volunteer tester
Avatar

Send message
Joined: 4 Jan 01
Posts: 195
Credit: 71,324,196
RAC: 0
Sweden
Message 228377 - Posted: 9 Jan 2006, 10:11:17 UTC - in response to Message 228302.  

Hi,
I tried to configure a service install for my mom who wants to participate but don't want to have to actually do anything.
When I set it to automatically connect, boinc interpreted the slow connedtion procedure as a failed attempt and would not log off after completed reporting of results according to preferences. Not good of course.
Setting preferences to 'Ask before connecting' didnt work either, because no manager is run. (no graphical presentation of dialog).

I gave up and signed her up for climate prediction instead. I can report that manualy when im up there visiting since I have quite a lot mort time to work with on CPN.

/Anton
ID: 228377 · Report as offensive
Jack Gulley

Send message
Joined: 4 Mar 03
Posts: 423
Credit: 526,566
RAC: 0
United States
Message 228381 - Posted: 9 Jan 2006, 10:51:24 UTC - in response to Message 228300.  

The bottleneck is opening a link to the database (slow). Once this is done, it is cheap to do several operations at the same time using that connection. The connection is closed at the end of the session, I believe for security purposes. Thus the more items that can be reported or requested at the same time, the lower the load on the server.

I have no problem with reporting multiple results in one session as that reduces the number of data base sessions required and a small amount of network overhead, hence more efficient. But, if you upload a result and open a session with the data base, would it not be even more efficient to also proceed with accepting the Report(s) at that time and doing the acknowledge, and then processing any request for downloading. Do it all in one session.

This would help the Dial-up user, and as it would reduce the number of sessions opened and closed to handle a single result being completed it would help the server. Broadband users benefit as there is no advantage in threating them different.

The only reason to require separate sessions for uploading and Reporting, would be fear of a data base crash at the very end of the process and losing a result because of it. (The server loses its record of the upload and report yet the BOINC HOST gets the acknowledgment and deletes the result record.) But compared to results being lost by users hitting ABORT, their disk crashes and other mistakes, server data base crashes hitting that very small window would be a minor problem.

Second point is a question. Does the server open a data base session before the upload actually starts or does it allow the upload to proceed to 100% and then open the data base session?

Deferring the opening of the data base session would reduce the server overhead a little by not wasting an Open/timoutClose for dropped Dial-up communications links, and at the same time reduce overhead for its problems with dropping sessions, by having fewer data base sessions open at one time.

If it is currently uploading segments of an upload directly into the data base by opening it first, then either additional memory or some other temporary storage may be needed to buffer the upload.
ID: 228381 · Report as offensive
Profile Keck_Komputers
Volunteer tester
Avatar

Send message
Joined: 4 Jul 99
Posts: 1575
Credit: 4,152,111
RAC: 1
United States
Message 228390 - Posted: 9 Jan 2006, 12:07:10 UTC

@Jack
Only the scheduler RPC opens the database. Uploading a result does not, it just stores the file on the server. That is why seperating uploading and reporting can save traffic on the server.
BOINC WIKI

BOINCing since 2002/12/8
ID: 228390 · Report as offensive
Jack Gulley

Send message
Joined: 4 Mar 03
Posts: 423
Credit: 526,566
RAC: 0
United States
Message 228408 - Posted: 9 Jan 2006, 13:11:40 UTC - in response to Message 228390.  

@Jack
Only the scheduler RPC opens the database. Uploading a result does not, it just stores the file on the server. That is why separating uploading and reporting can save traffic on the server.

Thanks, that knowledge relieves many of my concerns, some not stated for lack of information about how the servers are actually programmed (and lack of time to find out). Some of the descriptions of "dropped connections" and their effect on data base access had me worried that serious design problems existed. Will have to sleep on that useful bit of information.

I am now very interested in seeing what happens on the next outage and recovery. I have seen NO "Couldn't connect" and "No Scheduler responded" messages as of 9am Sunday morning Berkeley time when the problem with the campus network/router appears to have been fixed. There could have been a hardware problem behind a lot of these problems seen in past recovery.

Those types of errors should be very abnormal and only seen when a router/link is dropping enough packets that TCP/IP retires also fail, or there are serious problems in the frontend server and its ability to handle of the low level TCP/IP connection traffic. I have never seen a load on the inbound Ethernet link that could cause even a loaded P-II server to drop connection attempts on a 100Mb link. (Well maybe an old Windows 98 unpatched server running with bad disk drives.) And have never seen BOINC manage to drive the outbound link above 90% were such errors could occur because the server could not send any response at all back.
ID: 228408 · Report as offensive
Profile Tern
Volunteer tester
Avatar

Send message
Joined: 4 Dec 03
Posts: 1122
Credit: 13,376,822
RAC: 44
United States
Message 228511 - Posted: 9 Jan 2006, 21:05:33 UTC

I fully understand the THEORY that reporting the results later, by reporting multiple results at the same time, reduces the load. I still question how many times it _really_ reports multiple results and not one at a time, only later. On my systems, with a relatively small cache, it uploads one, runs the next for a while, then requests more work and reports the first one, repeats. Never reports or downloads more than one at a time, after the initial cache-fill.

Regardless, if someone is on dialup, all the reporting should be done during the same connection as the uploading, not wait possibly a week for the next connection, or have to be done manually.
ID: 228511 · Report as offensive
Profile thepossum1
Avatar

Send message
Joined: 15 Apr 00
Posts: 106
Credit: 213,585
RAC: 0
United States
Message 228655 - Posted: 10 Jan 2006, 1:59:51 UTC

Tried to reply to this and hit the post reply button just as the servers went down.

I'm dial up and have had only one "problem" that I've been able to resolve. I run my computer 24/7. I used to have the BOINC manager set for network activity based on preferences BUT, I was finding that when I'd try to disconnect and go offline, the greedy beggar wanted to keep that connection going--basically not allowing the modem to close the connection and I wasn't able to shut it down from the task manager either--it took a reboot to straighten it out. Solution--the manager can stay based on preferences as long as I'm online. When I'm finished and want to disconnect, I have to go in the command and set it to network activity suspended THEN disconnect. When I go back online, wait till the home page comes up then reset the network to preferences.Then I'll update--it uploads all finished WU's and downloads new ones--usually 2 in each direction at the same time. I've not had a single problem since I began doing it this way nor have I had any dropped connections. Works for me anyway.
Happy Crunching!

ID: 228655 · Report as offensive
Jack Gulley

Send message
Joined: 4 Mar 03
Posts: 423
Credit: 526,566
RAC: 0
United States
Message 228702 - Posted: 10 Jan 2006, 4:00:42 UTC - in response to Message 228511.  

I fully understand the THEORY that reporting the results later

I think this is a case where the design has been focused on special situations, and the Theory for dealing with them, not normal operation. It helps during recovery from outages by reducing the number of sessions opened. Or in cases like dial-up users.

I also have to agree that it should be done as soon as all results are uploaded. That also works in the special cases they are trying to deal with.
ID: 228702 · Report as offensive
1 · 2 · 3 · 4 · Next

Message boards : Number crunching : Brainstorming facilities for DIAL-UP BOINCers - pls contribute!


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.