Somewhere in the middle of the Work Unit problem thread, I threw in a few off-topic musings. It got a couple of favourable reactions, so I thought I'd pick them out and give them a thread of their own.

From what Joe was saying, it seems plausible that the current very high rate of -9 units ISN'T just bad luck with a heavy dose of RFI interference, ISN'T a problem with blanking the radar, but IS the result of setting the app sensitivity too high via the splitter parameters.

In which case, Matt & Co are fighting the wrong fire.

If they could fix the CAUSE of the current high demand for WUs, caches would start to fill up again, the work request rate would die back, the servers (and the staff!) would cool down, and they'd be able to sort out the remaining server configuration issues in (relative) peace. Instead, they're trying to increase the server speed, and probably throwing away perfectly good data in the process.

The beer is leaking out of the barrels, and all they're doing is digging a bigger drain.

I wish there was a better (trusted, moderated) feedback channel to the project developers/operational staff. I tried posting a quiet note in Technical News, but it got rapidly swamped in the Credit Wars - IMHO, very bad netiquette to post that sort of stuff in a news forum.

A similar thing happened with the app_info.xml bug in May. We, the denizens of the fora, had pretty much sussed out what the problem was and which bit of the server was broken by about midday Friday UTC. It was clear that the project staff were still in the dark about the true situation on Monday afternoon, CA time, some 80 hours later (I can provide sources for that assertion). If the staff could have a quick way of picking up on hard, technical facts gleaned by us crunchers, without having to wade through all the dross, I'm sure it would be helpful to them. As it is, I bet every single thread on these message boards errors out with a -9 when the staff try to read them.

I wonder if there could be an inverse of the Technical News board? Posts could only be made, or could be moved into it, by a 'Technical Moderator' - one of a group of people with sufficient technical / hardware / networking / programming experience (there are enough of them active on the boards) to recognise a significant point and allow it to float to the surface.

Just a thought. Comments?

Judging by Matt's post in Splitsville (Technical News), I wasn't too wide of the mark:

So here's the deal. Getting multibeam data out to the public is having its ups and downs. Thanks to some helpful poking and prodding from various users we uncovered a problem with the splitter causing it to generate workunits with bogus triplet thresholds. The result: about 50% of the workunits sent out were overflowing quickly and returning, creating network clogs on our already-overwhelmed servers. And about 2.5% of the workunits were sent out with impossibly low threshholds, causing clients to spin on ridiculously slow calculations. The mystery here is why these aren't also immediately overflowing (with such thresholds they should report a lot of garbage right away). This may have to do when/where the client checks for overflow - it may take several hours to reach 0.001% done, but then the hope is these clients will then finally be bursting with data and returning the results home.

This was actually a problem in beta that got fixed, but now somehow resurfaced, which is also a mystery. CVS out of sync? Some stupid code put in to check for config overrides on the command line? Unfortunately the splitter guru is on vacation, so we had to make our best attempt to understand the code and patch it ourselves. Jeff just did so and put the fixed version on line and we're watching the thresholds. So far so good.

Meanwhile, we're back to yesterday's problem of just not having enough throughput from the workunit file server, so that's the main bottleneck right now, and there's not much we can do about it except wait for the current artificial demand (caused by the excessive overflows) to die down and see if we catch up.

- Matt

From what I can read on the boards, Joe had diagnosed the problem by 16:25 UTC, and Matt became aware of it around 16:33 UTC - unfortunately, on different days. At least the time gap was only 24 hours this time, not the 80 hours we were in limbo in May.

Let me stress that the channel of communication that I'm groping towards would be for very rare occurrences indeed - twice in three months - so it would need to be very, very tightly moderated indeed to keep the signal-to-noise ratio high. Then, and only then, I feel it could be some use to the lab staff.

Again, comments anyone?

---

You've got a good idea, but it wouldn't really take someone "technical", just someone that is able to read the various threads and recognize common connections. During this "problem" I've noticed a dozen or more threads that had related posts in them. They would be worded differently, etc. but they seemed to all be related to the common "problem". Someone that could recognize those commonalities and either move or copy them to a common thread, a completely different forum area, and/or directly to the staff. If I understand your post correctly, you're suggesting a restricted forum only a moderator could post in, where the common problem posts could be moved to and sorta "collated" into just the problems without all the "fluff" that goes into an open forum post.
Good idea. It seems like it would be very useful in gaining insights into the next problem that occurs.

---

Jim

Some people plan their life out and look back at the wealth they've had.
Others live life day by day and look back at the wealth of experiences and enjoyment they've had.

The model I suppose we are looking at is like the Change Log on the BOINC/Core Client Board, where Kathryn and Jord as Forum moderator and Project administrators are the only ones that post.

Once a problem has been spotted, in other threads, by the 'good volunteer(s)' and is posted in this restricted thread, then everyone will know the info has been or will be shortly passed to Berkeley. Any further posting, by us commoners, will have to be in the original thread.

Andy

Let me stress that the channel of communication that I'm groping towards would be for very rare occurrences indeed - twice in three months - so it would need to be very, very tightly moderated indeed to keep the signal-to-noise ratio high. Then, and only then, I feel it could be some use to the lab staff.

Again, comments anyone?

The key is in the moderation or filtering: there is an effectively infinite amount of noise and relatively little signal.

... and I think the response time you noted is actually quite good, given the number of people working at the project and the "shields" they have to have to prevent being overwhelmed by the kind of noise we can produce.

In a sense, Joe probably does have an "out of band" path to get to the project, as do a few others, and that probably is close to what you're suggesting.

---

Let me stress that the channel of communication that I'm groping towards would be for very rare occurrences indeed - twice in three months - so it would need to be very, very tightly moderated indeed to keep the signal-to-noise ratio high. Then, and only then, I feel it could be some use to the lab staff.

Again, comments anyone?

The key is in the moderation or filtering: there is an effectively infinite amount of noise and relatively little signal.

... and I think the response time you noted is actually quite good, given the number of people working at the project and the "shields" they have to have to prevent being overwhelmed by the kind of noise we can produce.

In a sense, Joe probably does have an "out of band" path to get to the project, as do a few others, and that probably is close to what you're suggesting.

One project I run has an "Urgent Problems" thread. It is tightly moderated (off-topic & non-useful msg's are moved to another thread), but is highly useful in finding out what is going on. On average a problem shows up in two or three technical or crunching threads before someone adds "posted to urgent problem thread" which the project admin keeps a fairly close eye on.

---

Boinc Button Abuser In Training >My Shrubbers<

If the thread was set up like the Change Log I mentioned earlier only the designated persons can post in it.

Might be a good idea for Tech News and some of the stickies, to keep out the noise, also. An example would be only the optimisers, Simon, Alex, Joe, Crunch3r et al could post in Optimized app thread.

Andy

If the thread was set up like the Change Log I mentioned earlier only the designated persons can post in it.

Might be a good idea for Tech News and some of the stickies, to keep out the noise, also. An example would be only the optimisers, Simon, Alex, Joe, Crunch3r et al could post in Optimized app thread.

Andy

This would make it a lot easier for the project scientists and admins (Eric, Matt, et.al.) to check up. I've noticed that especially Eric will roam the forums looking for problems, but if they were in a hurry, they could just drop in anytime and check the one forum or thread and be able to tell whether everything were going right or not. Copy the relavent information to the post and put a link to the thread it came from in case they need background info that might have been posted leading up to it.
Really sounds like it would be useful in getting help with situations like we just had. I feel that if they could have had the whole picture to start with they might not have spent so much time trying to alleviate a problem that the system was causing itself. As it was, the info was spread throughout so many different threads, with new threads starting all the time they were swamped with irrelevent information.

---

Jim

Some people plan their life out and look back at the wealth they've had.
Others live life day by day and look back at the wealth of experiences and enjoyment they've had.

OK. I've deliberately held back from posting in this thread for 24 hours, to let the discussion take its own course, but y'all seem to be done now, so perhaps it's time to move on to stage 2.

Reading what you've all said, my initial proposal was perhaps over-complex. The revised suggestion which follows perhaps has an advantage in that, so far as I can see, everything I'm about to say can be accommodated in the existing server code / user status flags etc.: the only thing that would need to change is the way we use the existing tools.

So, I propose:
--------------------
We ask the mods to set up an Urgent Problems for Berkeley thread (as per ohiomike's suggestion/example) - either in Technical News or Number Crunching, I don't think it matters which.

The thread is to be kept sticky and locked (as per Andy's example of the BOINC 'Change Log' thread). Only moderators (by which term, I would include project staff) could post in / move posts into the thread.

The thread would have two initial posts: One, to explain what it's for and how to use it (and when not to use it!), and the other as an example of the sort of problem reporting that would be most useful: e.g., saying "I think the server's broken" is unhelpful, but saying "I think the server's XYZ widget is broken, because three people have reported ABC behaviour here" is more useful.

We ask the mods to officially sanction the use of the red-x mechanism to flag up problem reports. Once red-x'd, it would be the moderators' decision whether to 'upgrade' the report.
--------------------

How does that sound?

If nobody objects (or comes up with a better idea), I'll try pitching it to the setimods mailing list: and I'm happy to have a first attempt at drafting the opening posts.

Two questions: Does anybody know whether the mods have a tool for copying posts, instead of moving them? If not, we might encourage people to post a duplicate of their problem report: one for moving into the Berkeley thread, and the other to remain in place in the original problem discussion thread.

And should we limit it to strictly urgent (crunch-threatening) problems? I'm wondering about Alinator's recent query about web-based preference choices not sticking. It's not 'urgent' in the same sense as the splitter bug, but I'm sure it's annoying, and it can only be solved by project staff action.

The key is in the moderation or filtering: there is an effectively infinite amount of noise and relatively little signal.

... and I think the response time you noted is actually quite good, given the number of people working at the project and the "shields" they have to have to prevent being overwhelmed by the kind of noise we can produce.

In a sense, Joe probably does have an "out of band" path to get to the project, as do a few others, and that probably is close to what you're suggesting.

Ned,

I didn't want you to think I've ignored your comments, even though I haven't referred to them in my other post.

In fact, in the 'splitter bug' case, there was some fairly assertive use of the back channels, and not just by Joe. I think it worked, and I think that's what kept the time interval down to 24 hours.

But by their very nature, the back channels are secretive and feedback-free zones, and we'll never really know. They rely both on people knowing how to invoke them, and on the relevant intermediaries being available and willing to help. The advantage of the 'red-x' system I'm proposing now, is that it's already geared up for rapid response and full time-zone coverage.

Sounds good to me... very good ideas :D Thanks Richard for paying attention to this issue.

Two questions: Does anybody know whether the mods have a tool for copying posts, instead of moving them? If not, we might encourage people to post a duplicate of their problem report: one for moving into the Berkeley thread, and the other to remain in place in the original problem discussion thread.

And should we limit it to strictly urgent (crunch-threatening) problems? I'm wondering about Alinator's recent query about web-based preference choices not sticking. It's not 'urgent' in the same sense as the splitter bug, but I'm sure it's annoying, and it can only be solved by project staff action.

My answer to Question one: [facetious_mode]Yes, It's called the clipboard.[/facetious_mode]

My Answer to Question two: 'Blockers' & 'Showstoppers' [ & direct developments only, drawn from other threads or posted by staff/ mods, not discussion - keep that for the normal threads], the other threads can handle the less urgent mainstream stuff like 'I'm not getting work' or 'when will a vista version of such and such..', or 'Oh No my RAC...', as that is what tends to get the most noise.

---

"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.

My Answer to Question two: 'Blockers' & 'Showstoppers' [ & direct developments only, drawn from other threads or posted by staff/ mods, not discussion - keep that for the normal threads], the other threads can handle the less urgent mainstream stuff like 'I'm not getting work' or 'when will a vista version of such and such..', or 'Oh No my RAC...', as that is what tends to get the most noise.

Sorry, I should have made myself clearer.

The thread should only be used for issues at the Berkeley end of things, and then only for issues which affect all users, or a signficant sub-class of users - such as the May bug, which affected every user of the 'anonymous platform mechanism', not just third-party optimised apps. And it should be clear from the problem reports what the reporting user has done to investigate the problem, and why they believe it's a Berkeley issue. I'll try to cover both those points in my draft opening posts.

And I absolutely agree that we must keep client errors, and user complaints, out of the proposed thread. I, for one, will have no compunction about red-xing any such posts - no matter which moderator has moved them in there!

All good then, no argument here :D

---

"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.

As long as only mods are able to update/add to the thread, as you propose, then yes. Something like this is certainly needed. It should make Berkeley’s life a little easier if they have one place only to look for possible issues instead of parsing dozens of general discussion threads. They would be more likely to look in one place even when they are very busy; which is usually the case, hopefully improving response times to problems that they and only they can resolve.

The only negative (playing devil’s advocate here) is that I can see some crying about “Why hasn’t my problem been listed in this thread yet?”. Knowledgeable mods with thick skins could make this work.

Two questions: Does anybody know whether the mods have a tool for copying posts, instead of moving them? If not, we might encourage people to post a duplicate of their problem report: one for moving into the Berkeley thread, and the other to remain in place in the original problem discussion thread.

As far as I can see, there's no way to 'copy' posts, only move them. If we actually use the Clipboard to copy someone's post, we would have to post it in our name which would make it look like it came from us instead of the user (and would probably cause too much confusion).

The problem with encouraging people to post duplicates is that we will have a ton of duplicate posts from users in anticipation that their problem is legit, when in fact it may not be a Berkeley problem. We will then have to wade through twice as many posts just to find the real problem ones. How will they know when to post a duplicate of a legit problem? Certainly the less tech-savvy won't know before hand.

---

...
As far as I can see, there's no way to 'copy' posts, only move them. If we actually use the Clipboard to copy someone's post, we would have to post it in our name which would make it look like it came from us instead of the user (and would probably cause too much confusion).
...

Are you then able to leave a 'placeholder' post saying something like 'Post by _name_ moved to Important Berkeley Issues thread...' and a link ?

---

"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.

Two questions: Does anybody know whether the mods have a tool for copying posts, instead of moving them? If not, we might encourage people to post a duplicate of their problem report: one for moving into the Berkeley thread, and the other to remain in place in the original problem discussion thread.

As far as I can see, there's no way to 'copy' posts, only move them. If we actually use the Clipboard to copy someone's post, we would have to post it in our name which would make it look like it came from us instead of the user (and would probably cause too much confusion).

The problem with encouraging people to post duplicates is that we will have a ton of duplicate posts from users in anticipation that their problem is legit, when in fact it may not be a Berkeley problem. We will then have to wade through twice as many posts just to find the real problem ones. How will they know when to post a duplicate of a legit problem? Certainly the less tech-savvy won't know before hand.

I can see a solution, the mod posts and explains the problem (or even copies and pastes the entire original post) then add a link to the post/thread. The staff can read the mod's overview, then if they need more detail, follow the link to the original post/thread. I use "bbcode extra" which makes it very simple to add links, etc. to a post.

---

Jim

Some people plan their life out and look back at the wealth they've had.
Others live life day by day and look back at the wealth of experiences and enjoyment they've had.

I agree with what has been proposed, but would like to add Persistent problems to the list.

The example being the windows access errors that occasionally occur when it should be a -9 overflow, which I know Joe has examined and hopefully cured.

Also a question are we talking Seti only, or as Seti is the test bed for BOINC is it going to also going to cover BOINC problems. I'm sure those of us that actively crunch for Beta, sometimes what seems a Seti problem turns into a BOINC fault.

Talking Beta, are we proposing similar thread there, or will the one here cover both sites. I'd vote for keeping them separate.

Also, thinking laterally, what about problems identified on the Q&A boards.

Apologies, if you think I am expanding this too far.
Andy

I agree with what has been proposed, but would like to add Persistent problems to the list.

The example being the windows access errors that occasionally occur when it should be a -9 overflow, which I know Joe has examined and hopefully cured.

Also a question are we talking Seti only, or as Seti is the test bed for BOINC is it going to also going to cover BOINC problems. I'm sure those of us that actively crunch for Beta, sometimes what seems a Seti problem turns into a BOINC fault.

Talking Beta, are we proposing similar thread there, or will the one here cover both sites. I'd vote for keeping them separate.

Also, thinking laterally, what about problems identified on the Q&A boards.

Apologies, if you think I am expanding this too far.
Andy

I would suggest not just a sticky thread, but a complete new forum with the said limits imposed. This would allow separate threads where the admins and mods could discuss a certain problem between themselves if needed (to ask for more information, etc.) without outside interference, then when that certain issue were resolved, just mark that problem as "cured" and let that thread die. Then create a new one for the next problem. This way the staff could have one place that everyone could look at to see what was done to correct their problem, etc.

One for the entire project would obviously be easier for the admins and staff to check.

---

Jim

Some people plan their life out and look back at the wealth they've had.
Others live life day by day and look back at the wealth of experiences and enjoyment they've had.

First, Richard, thanks for initiative on this problem, which I'll summarize as providing a useful condensation of end-user knowledge for better utility to the actual staff.

The condensation and noise removal seem crucial to making it useful to the staff, but I think the thoughts so far may not have considered that displaying the resulting condensate publicly should also be useful to the users attempting to contribute.

To be specific, if I think I have an insight, I can check the moderated feedback thread. If my thought is already there, I stand easy. If not, I try to bring it up in a way that could get it there: I attempt an appropriate post, use of red-x, or whatever the means adopted is. If my issue is in the thread, but I think is crucially wrong on something for which I have personal expertise, I try much harder to find the place and way to alter the consensus. If nonsense is out in the public forums, but not making into the moderated thread, I can spare myself the effort of trying to correct it.

This transparency of moderated consensus is the element that is most crucially missing in the previous back-channel system. Much respect though I have for Joe Segur, I'd not readily know what he was saying.

I'm bringing all this up in an attempt to sway slightly the initial way this scheme gets described, and to re-emphasize how crucially important good moderation--which filters noise, but is ready to pass on possibly well-informed contradiction.

---

One addition that could be made if the single project-wide forum were adopted would be a stickied locked notice in each forum with links to that forum and the "tech notes" forum, etc. saying "If you are experiencing problems, look here to see if it has been reported." This would be locked so people wouldn't be wading through tons of "junk" to find the link. A simple thread with the one post with the links. This would make it easier for people to see what was going on, and it might cut down on the duplicate threads that show up here. If people see an issue listed, they can click the link to the original thread(s) and find out exactly what is being worked on, and post there with further information, etc.

---

Jim

Some people plan their life out and look back at the wealth they've had.
Others live life day by day and look back at the wealth of experiences and enjoyment they've had.