I'm not really disagreeing with you, but are not some of the problems that Seti see's, due to their large user base, problems that other project are liable to see in the future. Even if the other projects user base does not increase the average computer performance will increase and possibly put their systems under the same stress's that Seti see's today.
Therefore the changes, improvements that Seti possibly needs from BOINC, will help the other projects in the future.

In the last month or so we have seen Einstein fall over a couple of times just after Seti has hit a problem. Possibly due to hosts trying to filling their caches from a backup project. Einstein is probably the most similar to Seti, popular and usually stable.

If only there were a way to distribute the server, like the server hands out large batches of files to various computers, then the crunchers look to the volunteer servers.

BOINC, but in reverse.

I didn't mean to sound attacked, just as I don't see in your reply a sense that you sound attacked.

I agree with you about use of the retry for stuck uploads. But stuck uploads are something almost unique to SETI. The only time I see it with the large number of other projects I've attached to is when the project itself is down (which does happen episodically). In the case of SETI it happens regularly -- largely because of the inadequate ratio of users/workstations to project resources. At one point the SETI stuck upload scenario was simply a result of the scheduled maintenance and communications recovery cycle (maintenance of 4 to 6 hours, and recovery of about double that). My approach to that was to simply clear reported work to SETI either late Monday night or early Tuesday morning and suspend project processing. That way I'd have no stuck uploads to bother me and thus avoided generating traffic to the overloaded SETI resources.

Over the past several months though, the stuck uploads problem for SETI has become more frequent, as the increased workload as not found a matching increased resource level at the server. As this problem has increased, no doubt some users have responded (in frustration) to using the retry option to upload workunits. What I have done is instead to simply reduce the amount of SETI processing I generate, in effect load balancing, by focusing on other projects. I will note, that (not with SETI), but with other projects, I do use the retry button to clear 'stuck uploads' once those other projects come back online -- the thing is, with those other projects the situation is very much an 'either/or' condition. Once those projects are back online, using the dreaded retry button works and is only used once to clear the upload and so it works perhaps as intended.

So we are in at least some agreement here. That being said, perhaps I was sounding reactive to what I saw as a bit of 'blame the user' and fix the problem by reducing the users options. Now, what I'd like to see in the client itself, instead of masking the retry button (which as I noted above has a selective constructive use), is a clearing of client specific (rather than project specific) problems which have cropped up in the client - especially with the 6.x iterations.

In terms of a client configuraton change, one thing I would like to see in terms of network communication is the ability to turn off Network activity at the project specific level. That is a change which might be of help to SETI. I'd note that when the Climate change project has had issues, they suggest turning off network communications -- but recognize that approach only works for single project configurations.

I don't know why you're feeling so attacked by my simple comment. I was merely suggesting that hitting retry causes the problem to be worse. I'm not certain why you felt it necessary to point the fingers at the BOINC developers over my comment, or that you are calling the users "silly", which is your words, not mine. I don't think users are "silly", but I do think that users who try to force comms by hitting retry are part of the problem. Its a logical conclusion to come to and there's no reason for getting defensive over it.

---

See my other replies to both you and Ned. I do agree that my use of the term 'silly' was ill-advised as it certainly seemed to touch a nerve for both you and Ned. I was reacting to the view that the SETI specific problem regarding dropped connections is (at least in part) a user problem to be resolved by a change in the BOINC client. My own sense is that the number of users hitting the retry button (in part out of frustration with the irregularity and frequency of dropped connections) is a much smaller piece of the problem than conditions and limitations on the project side.

Regarding 'frustrated and defensive' -- perhaps, though amongst the three of us (Ned, you and I) that seems to be something not unique to me....

As I stated previously - its your words, not mine, that users are "silly". I don't think they're "silly" at all, but I do think forcing comms when the servers are already dropping connections is a bad thing, and that perhaps trying to control the traffic by not allowing people to do this would be a good thing. Like Ned's stoplight analogy, its not silly at all to try to control the traffic better. Is it silly to want to reduce collisions? That's effectively what's happening on an overloaded network, and I'd like to see less collisions, not more, and certainly not more caused purposely.

Yet you are getting frustrated and defensive over my single comment anyway.

---

So isn't this actually a problem in that boinc is not scalable by design?

Instead of looking for bandaids, the boinc developers should be moving to a distributed system, where intermediate nodes could be used to group results. I think the add-ons to the seti-classic provided this ability, which was not embraced when boinc started up. Or better yet, perhaps a bit-torrent type of structure would be better? (Don't know, just throwing out discussion points)

Or better yet, perhaps a bit-torrent type of structure would be better? (Don't know, just throwing out discussion points)

The problem still remains- you still have to get the data to & from the Seti servers to those other computers. And through the same connection, the present problem would still remain.

---

Grant
Darwin NT

So isn't this actually a problem in that boinc is not scalable by design?

Instead of looking for bandaids, the boinc developers should be moving to a distributed system, where intermediate nodes could be used to group results. I think the add-ons to the seti-classic provided this ability, which was not embraced when boinc started up. Or better yet, perhaps a bit-torrent type of structure would be better? (Don't know, just throwing out discussion points)

Bit-torrent has been suggested many times, and the answer is still the same. Bit-torrent is great when you want to distribute the same thing to lots of clients as every client becomes a sub-server. But here, you are supplying different data to each client so the bit-torrent model just does not work.

F.

---

Yes, we are in agreement, Einstein and SETI were my first two projects. These days, I've move most of my CPU and all my GPU cycles to other projects. I have more cycles for Einstein in part because their problems are 'digital' - either they are online or they are not which is not the case here where everything can be working, just simply not 'fast enough' for the user load.

The larger projects are more likely to bounce up against communications problems than the 'small guys' since often enough the issue here appears to be 'pipewidth'.

Then again, work units for Einstein are much longer than they are here which probably helps with the upload/download condition.

Folks at Climate 'solved' the problem a similar way, they have VERY long cycle work units, and to address the 'where are my credits population' their design includes daily work progress (and credit) 'trickles'. That means their outages (which do happen) have very little impact on the end user (except for a delay in seeing credits, not a loss of work or credits).

The Milkyway project encountered problems, particularly in downloading work, largely as a function of allowing optimized 3rd party applications -- which vastly improved the efficiency of work unit processing. This problem got worse once the folks working on optimization developed an ATI-GPU application which works with the same work units and VERY fast. It has taken their VERY limited project resources several months to deal with this -- in the past month or so they have -- with significantly longer work units -- so they have enough work for people to download without pounding the server.

I think a very real problem here is that there are a large number of SETI only users (that's NOT the users 'fault' or the projects 'fault') and so they have nowhere to turn when SETI is (as it has been for a long time) periodically problematic. Perhaps that increases the 'temperature' of some of the discussions over here.

I'm not really disagreeing with you, but are not some of the problems that Seti see's, due to their large user base, problems that other project are liable to see in the future. Even if the other projects user base does not increase the average computer performance will increase and possibly put their systems under the same stress's that Seti see's today.
Therefore the changes, improvements that Seti possibly needs from BOINC, will help the other projects in the future.

In the last month or so we have seen Einstein fall over a couple of times just after Seti has hit a problem. Possibly due to hosts trying to filling their caches from a backup project. Einstein is probably the most similar to Seti, popular and usually stable.

---

So isn't this actually a problem in that boinc is not scalable by design?

Instead of looking for bandaids, the boinc developers should be moving to a distributed system, where intermediate nodes could be used to group results. I think the add-ons to the seti-classic provided this ability, which was not embraced when boinc started up. Or better yet, perhaps a bit-torrent type of structure would be better? (Don't know, just throwing out discussion points)

Scalability requires one prerequisite: the money to do the scaling. The fiber link plus a few more servers at Berkeley would "scale" just like buying space off-campus would "scale" -- with the additional management to keep the offsite "tanks" filled.

When you suggest intermediate nodes, there is still a need of some sort to distribute work from the central site to the nodes themselves.

user <--> berkeley and user <--> offsite node <--> berkeley

The common part is the link in and out of Berkeley. You've moved the problem, but it's still there.

The only advantage is that you could optimize the use of the Berkeley link, but the same thing can be done without the offsite node, optimizing the clients directly.

---

So the queue idea is the best one, it does not eliminate the problem, but it manages it in a way that nobody freaks out, they know they will get a turn.

I too want to see traffic flowing smoothly -- for all BOINC projects. I would rather not see SETI specific traffic flow issues dictate the configuration of the BOINC client as the SETI specific traffic flow issues are in fact SETI project specific.

Traffic flow issues for SETI may well be something attributed to users -- in that there are too many of them for the available resource and performance level that this project has. Solving that issue is not a case (in my view) of changing the BOINC client, but something more local.

That being said, I realize that much of the BOINC client development has SETI specific roots, resources and influence and so the possibility that an effort to SETI specific issues might well bleed into design changes for the BOINC client.

As long as we're talking about optimizing throughput, and not "controlling silly users" -- because the goal is not to get those unruly users under control, but to make things work.

I'm also opposed to BOINC doing anything that is not useful for other projects, but the goal for BOINC is to make distributed computing something small projects with little funding affordable.

Traffic control for SETI's 100 megabit link could help a small project running off of a home DSL line and just one server.

I don't think it's the number of clients, I think it's the number of clients times the number of retries.

---

OK -- we have a fair degree of agreement. I guess where I differ is in your view some of that problem source, I see the number of user manually generated retries as likely a very small piece of the problem, a piece of the problem yes, but just a very small piece, and one which happens because of the resource constraints we agree exist, particularly with SETI and its large user community.

Regarding the user retries -- if some folks are using a script to generate retries in the Transfer area of the BOINC client, then I agree, THAT could be a real problem source -- automated pounding of the server is not a good thing.

As to *really* small projects with only slower DSL available -- well the deal there is that those projects would need to be REALLY small in terms of user count. Actually, I think that points out the constraint here with SETI -- the ratio of users (or more accurately workstations) to the resources available to the project. There is always a project side constraint at the high end -- be it the pipeline (as most folks here see it for SETI), the available servers, or the available tech support. So when there are too many workstations doing work on any given projects, that constraint will be hit. At this point in time, (at least for my project set -- and I will readily admit there are bunch of other projects I don't work with) the only project with too many users (or too much processing power) relative to the project is SETI -- which of course is by far the largest project in terms of user available processing power.

Actually, one of my other projects also has run into a constraint -- it isn't the servers or the pipe, it is that they lack the tech resources to generate and analyze new work -- that's the Malaria project.

As I noted before, for a time this past year, the MilkyWay folks had a similar issue -- too much processing power wanting work. For that project, the problem manifested not so much as a function of stuck uploads, but rather that getting new work was quite problematic -- which resulted in some folks setting up scripts to hit the server for new work. Fortunately, that project (with its very small staff resources) resolved the problem by producing work units which are 4 to 8 times longer than they were earlier. So no one (even those running the optimized client for ATI 48xx GPU's) is 'work starved'.

One thing of interest -- the angst over in the Milkyway project was somewhat similar to some of what happens here -- because that project also has folks who are operating in single project mode. Over there, a major factor is the lack of ATI GPU support in any other BOINC project -- those folks have nowhere else to turn to even if they wanted to.

As long as we're talking about optimizing throughput, and not "controlling silly users" -- because the goal is not to get those unruly users under control, but to make things work.

I'm also opposed to BOINC doing anything that is not useful for other projects, but the goal for BOINC is to make distributed computing something small projects with little funding affordable.

Traffic control for SETI's 100 megabit link could help a small project running off of a home DSL line and just one server.

I don't think it's the number of clients, I think it's the number of clients times the number of retries.

---

OK -- we have a fair degree of agreement. I guess where I differ is in your view some of that problem source, I see the number of user manually generated retries as likely a very small piece of the problem, a piece of the problem yes, but just a very small piece, and one which happens because of the resource constraints we agree exist, particularly with SETI and its large user community.

I think the manually triggered retries are a very very small percentage of the whole, I think 180,000 computers retrying multiple work units with the default (only) backoff parameters is at fault.

As work backs up, those same clients have more uploads, and continue to beat on the servers trying to push through work.

If we can optimize throughput in general, then those who do watch too closely will never see the backlog that scares them now.

My DSL example was just to point out that unlimited bandwidth is almost never available, and server capacity also factors into this.

---

See my other replies to both you and Ned. I do agree that my use of the term 'silly' was ill-advised as it certainly seemed to touch a nerve for both you and Ned.

The only nerve it touched was the suggestive comment that we are "blaming the users" when the problem lies elsewhere, and I don't feel that's an accurate accusation of my views...

I was reacting to the view that the SETI specific problem regarding dropped connections is (at least in part) a user problem to be resolved by a change in the BOINC client. My own sense is that the number of users hitting the retry button (in part out of frustration with the irregularity and frequency of dropped connections) is a much smaller piece of the problem than conditions and limitations on the project side.

...it may be a small subset of users, but grown adults should know better and not try to force the situation. Allowing the option to remain only allows abuse of that option that quite a few people do.

... and without an actual survey of the users, it may be far more people than you know hitting the retry button. And its just as likely to be far fewer than I realize. Regardless, as long as the abuse continues, its only making it worse for everyone else, so why allow it to continue?

Regarding 'frustrated and defensive' -- perhaps, though amongst the three of us (Ned, you and I) that seems to be something not unique to me....

I'm not certain about that. I've shown no reason to be defensive, though slightly insulted over the "users are the problem"/"Users are not the problem, fix the real issue" implication. ...and I'm the furthest from frustrated because none of these Panic threads or server problems frustrate me. I don't get frustrated like many around here seem to. The only thing I try to do is change people's expectations and reduce other's frustration. Most often while doing this, people tend to turn their angst toward me because I suddenly embody their barrier to progress through my opposing stance - often with labels such as "apologist", "purist" and "sycophant" are amonth the most popular. I guess its far easier to visualize your angst against something that can actually respond to you and argue - unlike the servers who don't argue and don't fight back.

Then the suggestions ensue about my own frustration, defensiveness, and angst. Typical reversal of the issues, but false nonetheless.

---

Hmm -- OK -- so instead of disabling the manual retry, you are suggesting that the default backoff parameters is a problem source (for SETI in particular with its constraints). Is that settable (at a project level) in any current iteration of the client? Does one have to manually tweak a config file to change this?

I agree that unlimited bandwidth is essentially never available, but haven't seen bandwidth as a constraint (or the gating constraint) for any other project.

I'm certainly one of those who watched too closely -- what backed me off of that was attaching to many other projects and dropping the SETI resource share so these days, to the extent that I continue to watch too closely -- it isn't regarding the SETI project. For SETI, to paraphrase the great Rex Harrison, I've grown accustomed to its pace.

I think the manually triggered retries are a very very small percentage of the whole, I think 180,000 computers retrying multiple work units with the default (only) backoff parameters is at fault.

As work backs up, those same clients have more uploads, and continue to beat on the servers trying to push through work.

If we can optimize throughput in general, then those who do watch too closely will never see the backlog that scares them now.

My DSL example was just to point out that unlimited bandwidth is almost never available, and server capacity also factors into this.

---

Whoops -- so instead of the phrase 'silly users' (my bad choice of words which garnered a reaction), it is 'grown adults who should know better'. Seems to me that you've reintroduced a variant of my phrase here and in doing so are blaming the users. Probably not your intent.

The only nerve it touched was the suggestive comment that we are "blaming the users" when the problem lies elsewhere, and I don't feel that's an accurate accusation of my views...

...it may be a small subset of users, but grown adults should know better and not try to force the situation. Allowing the option to remain only allows abuse of that option that quite a few people do.

---

Whoosh...:D

---

"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.

Hmm -- OK -- so instead of disabling the manual retry, you are suggesting that the default backoff parameters is a problem source (for SETI in particular with its constraints).

I think there is value in disabling the manual retry, but ultimately, you can't control people, users are never going to do what you want, and in this case, I don't think they are the source of the problem.

When I say "client" I'm talking about the BOINC program on users' machines. If BOINCstats can be trusted, there are 180,000 of those, with multiple work units, and I don't think the backoff parameters can be tuned currently at all.

---

Whoosh...:D

Doubley Whhosh...:D

Then again, I didn't perceive *myself* to be defensive either....

To your point about frustration levels -- I agree, that won't solve problems and may generate more heat than light. We (those who post here) are most inclined to engage in back and forth that might portray angst of one form or another simply because, by posting here, we are demonstrating a level of interest that the vast majority of SETI users simply don't have. Your participation as a moderator indicates that you too have a level of interest far above that of the vast majority of SETI users.

There was a time when the issues that SETI has regarding work flow did bother me -- for me it was worst way back doing the switch over from Classic to BOINC many years ago (I joined SETI classic over 8 years ago). Then there were times over the intervening years when things got much better - creating something of an expectation level, and then not so good causing - some degree of angst. My approach (not one that others should feel is particularly special or required), was as I've noted before, simply to spread the CPU (and GPU) wealth around. Once SETI dropped down my list, the periodic (perhaps frequent) problems with it, to which (noting your point about expectations) I have become accustomed, no longer caused me that frustration level. I realize that essentially nothing I do will effect changes (I don't have large bucks to buy a fiber connection for SETI), and so to the extent I participate, it is to join in the community experience.

Fortunately, for me, I *am* attracted to the research that a number of the other projects engage in (the SETI project reflects an astrophysics interest which also has value in Einstein and MilkyWay, being a cancer survivor provides interest in projects like Rosetta, POEM, WorldGrid, GRID GPU and the like, and an interest in climate change makes the Climate project attractive). So, the struggles that SETI has while of interest, don't frustrate me all that much.

Now the back and forth of 'I'm right, you're wrong' -- well I suspect that can cause angst, frustration, and defensiveness even among the best of us -- even, dare I say it, moderators such as yourself. Again, I don't mean that in a negative way, just voicing my view of the human condition.



And that level of interest means that even you just might be subject to feelings of defensiveness, angst, or being insulted even when that was not the intent of the original post. (insert appropriate mantra here).

I'd not characterize you as an apoligist, or purist or sycophant. The term I'd use, and NOT in pejorative manner, is advocate.

I'm not certain about that. I've shown no reason to be defensive, though slightly insulted over the "users are the problem"/"Users are not the problem, fix the real issue" implication. ...and I'm the furthest from frustrated because none of these Panic threads or server problems frustrate me. I don't get frustrated like many around here seem to. The only thing I try to do is change people's expectations and reduce other's frustration. Most often while doing this, people tend to turn their angst toward me because I suddenly embody their barrier to progress through my opposing stance - often with labels such as "apologist", "purist" and "sycophant" are amonth the most popular. I guess its far easier to visualize your angst against something that can actually respond to you and argue - unlike the servers who don't argue and don't fight back.

Then the suggestions ensue about my own frustration, defensiveness, and angst. Typical reversal of the issues, but false nonetheless.

---