When XML stats will be ON again?

International statistics are not working :-(
--
Get up, stand up! Don't give up the fight!

Credits will make everybody feel high! ;-)

I emailed Jeff about this yesterday since my site hadn't gotten an update in over 48 hours. He said they had turned off the XML export to lighten the load on the database server. Personally, I think there must be something else very wrong if a "select * from teams" takes any more than 0.5 seconds to execute... but then again, what do I know?

Anyway, Berkeley is working on getting a replicated database up and running which will handle the XML generation and (I suspect) a lot of the website. When this is done, they will turn the XML back on. I requested that they turn it on at least once per day but that obviously wasn't done.

Oh well, I guess we will live - barely.


---------------------------------------
- A member of The Knights Who Say NI!
Possibly the best stats site in the universe:
http://boinc-kwsn.no-ip.info

As far as I remember SETI Classic had to move all statistics demands on another server. At times, there were so many ppl, who wanted to see their stats that it was impossible for website to work efiiciently (thus making it impossible to download S@H client for example)
BOINC@Poland: Polish crossproject team.

http://mchl.republika.pl/boincatpoland/

- Join -

This still doesn't answer the question when we can expect the XML to be turned on again!

> This still doesn't answer the question when we can expect the XML to be turned
> on again!
>
>
>

> "When" is supposed to refer to a timeframe instead of circumstances!

I can tell you precisely ...


Later ...
<p>
For BOINC Documentation: Click Me!

> I can tell you precisely ...
>
>
> Later ...

Brilliant, you are telling precisely nothing.
What's wrong with the question? Why the sarcasm?

> > I can tell you precisely ...
> >
> >
> > Later ...
>
> Brilliant, you are telling precisely nothing.
> What's wrong with the question? Why the sarcasm?
>

Because nobody knows the exact date. Just hope it'll be soon.
They don't delay work on project on purpose.
BOINC@Poland: Polish crossproject team.

http://mchl.republika.pl/boincatpoland/

- Join -

> > > I can tell you precisely ...
> > >
> > >
> > > Later ...
> >
> > Brilliant, you are telling precisely nothing.
> > What's wrong with the question? Why the sarcasm?
> >
>
> Because nobody knows the exact date. Just hope it'll be soon.
> They don't delay work on project on purpose.

I am a system engineer myself. I never do maintenance without informing my clients what is going on and how long I estimates it is going to take.
I don't ask the team for guarentees, but is it a one day job, a three day job, or will it take a whole week?

That has nothing to do with delaying a job. That is serving your clients, your customers if you like, because basically that is what we are. We are their clients. We, in general, are asked to help the project, or SCIENCE whatever you prefer, by donating CPU time.
The least the team can do is giving us their best estimate instead of "just wait and hope when certain services are back", what is basiccaly what you are writing: "Just hope it'll be soon".

The reactions on this forum should give you an idea that a lot of s@h users are thinking that way. The only thing asked for is information. I am not shouting or calling names, just asking for some info. That can never result in the kind of answers I, and others who asked the same question, received in this thread. Unless of course you think some decency is to much to ask for when you are dealing with a person you don't know!?

> > > > I can tell you precisely ...
> > > >
> > > >
> > > > Later ...
> > >
> > > Brilliant, you are telling precisely nothing.
> > > What's wrong with the question? Why the sarcasm?
> > >
> >
> > Because nobody knows the exact date. Just hope it'll be soon.
> > They don't delay work on project on purpose.
>
> I am a system engineer myself. I never do maintenance without informing my
> clients what is going on and how long I estimates it is going to take.
> I don't ask the team for guarentees, but is it a one day job, a three day job,
> or will it take a whole week?
>
> That has nothing to do with delaying a job. That is serving your clients, your
> customers if you like, because basically that is what we are. We are their
> clients. We, in general, are asked to help the project, or SCIENCE whatever
> you prefer, by donating CPU time.
> The least the team can do is giving us their best estimate instead of "just
> wait and hope when certain services are back", what is basiccaly what you are
> writing: "Just hope it'll be soon".
>
> The reactions on this forum should give you an idea that a lot of s@h users
> are thinking that way. The only thing asked for is information. I am not
> shouting or calling names, just asking for some info. That can never result in
> the kind of answers I, and others who asked the same question, received in
> this thread. Unless of course you think some decency is to much to ask for
> when you are dealing with a person you don't know!?
>
>
>
>

Bravo maybe that will get some attention from the "powers to be" I have said for the most part the same thing and get sarcastic answers have seen that more than a few people have left the project because of the answers they get, I don't want to be one of them but i think the project leaders should understand that we DONATE our time and energy and should be treated with a little respect and paciants because all of are not pc engineers. All i ask for is post the info in a timely maner.

> > I can tell you precisely ...
> >
> >
> > Later ...
>
> Brilliant, you are telling precisely nothing.
> What's wrong with the question? Why the sarcasm?

Actually, sarcasm was not intended. You asked, then demanded an answer to a question that cannot be answered. especially, since the actual occurance is dependent on circumstances, which you don't accept as a boundry to the answer.

We are on the "bleeding edge" of a new technology. SETI@Home Classic created a whole new class of computing using hundreds of thousands, even millions, of personal computers to create a virtual Super-computer. Then proved that this can and will work for a lot of computing problems.

To follow up that act, we are again creating another new class of computing that is a major extension, if not a completely new class, where the capability that was latent in the old class is now used to process many different projects on the interconnected parts of that super-computer. As a matter of fact, one of the most significant improvement is to "conserve" the assets by never using more than is necessary to produce a result.

Minor, but also significant is the addition of traceability of the results so that the science produced meets the required standards of the academic profession where any expiriment must demonstrate the data, the mechanisms to produce the results, and then the conclusions.

Ok, because we are in a brand new region we are finding problems. early on in Beta we found a lot, but we mostly were finding them in the client applications. Now, the basic science application is pertty stable and only a few tweaks have needed to be made. The GUI, as expected when out "in the wild" has developed several new problems. Yet, even for that, it has proved to be pretty resilent and workable.

Now we are working on the cross-platform GUI and the back end. The back end unfortunately has been proving to be much harder to get working to a desired level. Again, this was something that I think was expected. The problems have been harder than expected to solve. The real sticking points seem to be that, at least with SETI@Home, the volume of transactions has been higher than expected and harder than expected to cope with.

Couple that with hardware issues and stir ...

SO, because the database is a sore spot and is central to the problems we have seen the turning off of the pending credit and the result pages, and now the XML output. All of these are rational choices to try to get well ... If you have a cold that you want to cure, you don't go out ice fishing in Minnesota in your undies ...

My answer remains ... later ...
<p>
For BOINC Documentation: Click Me!

> > > I can tell you precisely ...
> > >
> > >
> > > Later ...
> >
> > Brilliant, you are telling precisely nothing.
> > What's wrong with the question? Why the sarcasm?
>
> Actually, sarcasm was not intended. You asked, then demanded an answer to a
> question that cannot be answered. especially, since the actual occurance is
> dependent on circumstances, which you don't accept as a boundry to the
> answer.
>
> We are on the "bleeding edge" of a new technology. SETI@Home Classic created
> a whole new class of computing using hundreds of thousands, even millions, of
> personal computers to create a virtual Super-computer. Then proved that this
> can and will work for a lot of computing problems.
>
> To follow up that act, we are again creating another new class of computing
> that is a major extension, if not a completely new class, where the capability
> that was latent in the old class is now used to process many different
> projects on the interconnected parts of that super-computer. As a matter of
> fact, one of the most significant improvement is to "conserve" the assets by
> never using more than is necessary to produce a result.
>
> Minor, but also significant is the addition of traceability of the results so
> that the science produced meets the required standards of the academic
> profession where any expiriment must demonstrate the data, the mechanisms to
> produce the results, and then the conclusions.
>
> Ok, because we are in a brand new region we are finding problems. early on in
> Beta we found a lot, but we mostly were finding them in the client
> applications. Now, the basic science application is pertty stable and only a
> few tweaks have needed to be made. The GUI, as expected when out "in the
> wild" has developed several new problems. Yet, even for that, it has proved
> to be pretty resilent and workable.
>
> Now we are working on the cross-platform GUI and the back end. The back end
> unfortunately has been proving to be much harder to get working to a desired
> level. Again, this was something that I think was expected. The problems
> have been harder than expected to solve. The real sticking points seem to be
> that, at least with SETI@Home, the volume of transactions has been higher than
> expected and harder than expected to cope with.
>
> Couple that with hardware issues and stir ...
>
> SO, because the database is a sore spot and is central to the problems we have
> seen the turning off of the pending credit and the result pages, and now the
> XML output. All of these are rational choices to try to get well ... If you
> have a cold that you want to cure, you don't go out ice fishing in Minnesota
> in your undies ...
>
> My answer remains ... later ...
> <p>
> For BOINC Documentation: Click Me!
>
>
>
>


Actually, sarcasm was not intended.

If you
> have a cold that you want to cure, you don't go out ice fishing in Minnesota
> in your undies ...

No Sarcasm..............Really
-------------------------------

My guess is, that we won't be seeing results, and pending credits pages, sooner than a separate machine is put up to handle the web pages (and stats maybe). This means need for obtaining and setting-up new hardware which is usually a lengthy process. I just hope we won't have to wait for XMLs so long.

P.S. I dont like to be called a 'client' or 'customer' of BOINC/S@H. I prefer the word 'volunteer'.
BOINC@Poland: Polish crossproject team.

http://mchl.republika.pl/boincatpoland/

- Join -

Taking a quick look on Predictor@home shows that it's not always possible to set a timeline.

"9/13/04 - Update of the past week:
The servers have not been delivered and Dell can't give us an estimated time of delivery yet; they seem to have some sort of problem with our order. We are looking into this and hope it will be resolved as quickly as possible.
9/29/04 - Update of the past week:
Our newly ordered servers still haven't arrived. After checking with Dell, the only answer we seem to be getting is that the machines are still in production test :-( We will keep on bugging Dell to provide us with a delivery estimate."


So the only exact answer would be "When it's done".

Actually I really do think as an individual I've done more then my best to support this project over the years...(by all means don't misinterpret, this has no pretention!).
The reason,... because I really BELIEVE in this good cause and in the benefit of the results for all of mankind even if we don't find any signal.
In the worst case we will have gathered a lot of information about the radiofrequency distributions in space.

I'm NO part of a team, for that is my choice.
By acting in this way I did invest seriously in keeping upto date my computer infrastructure and I do have a very huge electricity bill.
I don't want to brag, so my computers are hidden, and I won't use this forum to make the amounts and the costs public, for this aspect is not in question...

What do I want :

1. Reasonable briefing of WHAT is actually happening. (e.g. Why can't this be done..., why is this down..., why is that broken,...) - in short a very REASONABLE public relation towards the public, especially for newcomers wich could get otherwise lost for the sake forever.

2. At least the XML files should get back on line, so that follow up of our work has at least a minimum visuality / feedback and a personal satisfaction that things are running in the right direction.

Greetings to all idealists-crunchers all over this small grain of sand in the desert of the universe, from Belgium...

I thinks the last posts all sums it up pretty neat.

1) There is a huge demand for visibility of results. Therefor, when the XML is turned of there should be a clear explanation what is happening and an educated guess of how long it will take.

2) The educated guess is nothing more than a guess. When e.g. Dell cannot deliver the hardware, just tell the public Dell is not delivering. That's is better than saying nothing or "later". Believe me, I know about hardware delivery problems. :-(

3) In this case a database replica is being restored on a seperate machine.
This means some questions can be answered:
- Is the machine available?
- How long to install and configure this seperate machine to accomodate the database when everything goes as expected?
- How long will the restore of the database take?
- How long will the complete setup take under expected conditions?

>>"All of these are rational choices to try to get well "
4) I have and will never questioned the decisions made with regards to maintenance of the servers. Only the persons working on the servers can make these decisions.
All I wanted to know is do I have to check stats every day to see if they are up because it can be fixed quickly, or is it useless to check before next Tuesday next (as an example) because that is how long it will take at least.

5) No disrespect to the team is meant in any way.
I started with BOINC last July and am not really into "the scene" because I just enjoy myself with the competition between me and my friens within the team, and the position of our team in the world ranking. This means the XML is a big part of our fun in spending CPU time on S@H.

> We are on the "bleeding edge" of a new technology.

This is NO BLEEDING EDGE, most of it is COMMON GROUND it is NOT to be compared with e.g. the finding of the existance of the electron...Multiprocessing and multicomputing have been on the teachinglist of numerous universities all over the world!!!

> Ok, because we are in a brand new region we are finding problems. early on in
> Beta we found a lot, but we mostly were finding them in the client
> applications. Now, the basic science application is pertty stable and only a
> few tweaks have needed to be made.

A FEW???? tweaks - you're joking - I call this MAJOR MISTAKES / in industry you get FIRED for a LOT LESS -
Even now some ELEMENTARY FEATURES like the amount of posts are NOT CORRECTED.

> Now we are working on the cross-platform GUI and the back end. The back end
> unfortunately has been proving to be much harder to get working to a desired
> level. Again, this was something that I think was expected. The problems
> have been harder than expected to solve. The real sticking points seem to be
> that, at least with SETI@Home, the volume of transactions has been higher than
> expected and harder than expected to cope with.

They could IN ADVANCE have made an ESTIMATE WITH AN ERROR MARGIN of the amounts of traffic based on the experience with Seti@Home Classic. So they have done A VERY LOUSY ESTIMATE, even SEVERELY UNDERESTIMATED the needed hardware...


> Couple that with hardware issues and stir ...

Sorry, but this is NOT a milkshake,...
In my view and long career in science everything has to do with careful planning, funds, manpower and an extreme bookkeeping view of the possibilities of which goal can be achieved. If this cannot be realistically implemented in advance, you're a dreamer and have real "wild west adventures" like this one in some instances. Most of the people don't have the slightest idea what it means to run an operation like this, within the limits of technical capability and funds. (You shurely don't have to be a programmer on this one!)

> SO, because the database is a sore spot and is central to the problems we have
> seen the turning off of the pending credit and the result pages,

This proves THAT THE ABOVE ANALYSIS is right, THOUGH IT MAY BE NOT their fault because of the LACK of FUNDS to do better (i.e. to have money to buy the hardware needed to cope with this kind of grown-above-the-head-project).
I have seen companies go down due to their overwhelming success.

The turning off of Pending credits, ..., ..., XML etc. is NO CURE it is REALLY THE SIGN OF THE DESEASE.
They should have calculated this in and GET 100% MORE HARDWARE THAN NEEDED before the real life startup. Elevators are calculated with a severe safety factor, otherwise you or I wouldn't step inside. What goes up can and will fall down eventually.
Murphy's law states things will break at the worst possible moment. Even a piece of foam can be lethal, or a weak joint can cause a rocket to explode.

No hard feelings, but a little disappointed,... ethousiasm should not darken the scientific view.

HP / Phd. Chemistry-Physics-Informatics.

I do agree with H P (Mr. Linux) New and Marco Berkhout! I also donated 5.722 years of computer time to SETI@Home because I am interested in the project AND the competition - call me childish or not. This project cost me 2 computers so far and a couple of hard disks, since both are not intended to run at 100% CPU for 24/7 and years, but I still support it.

The minimum you can ask under these circumstances is to get some information. I am aware that we are presently beta- or even alpha-testers. That's o.k., because someone has to do it, even to the risk that something goes wrong. But I do want to know what it is!

Oh, my ... what heat ... well, let me try to take them in order:

> > Brilliant, you are telling precisely nothing.
> > What's wrong with the question? Why the sarcasm?
======
If you
> have a cold that you want to cure, you don't go out ice fishing in Minnesota
> in your undies ...

No Sarcasm..............Really
-------------------------------

Nope, neither the first one or the second. If you wish, you have my permission to interpret it however you wish. I think both of them are observations. And I still maintain that the first answers were as accurate as they could be. They were also dismissed as being inadequate because you did not want one that was based on dependencies. When, in this case, it is most likely going to be a dependency issue.

So, re sarcasm: "I know you believe you understand what you think I said, but I'm not sure you realize that what you heard is not what I meant."

So, we have a back log of work. The stats were turned off to get the back-log down. A rational decision. They will be turned on when the can be turned on. That is going to be later. Turning them on before the backlog is gone does not make sense. That is my opinion, yours is different. Yet for all the threats of people that they are going to leave as they hate [insert pet hate here] and they will never come back. Well, the place is still crowded. Work is being done, and slowly but surely the problems are being fixed, could it be faster? Probably. But they are being fixed. Am I pleased with the speed? No, but being pragmatic and looking at the problems my call is that they are being addressed about as fast as anyone could expect.

And I stick with my statement about "bleeding edge". A Lexus and a Yugo are both cars made with standard off the shelf parts ... which one do you want to spend $60,000 on? Taking standard parts and assembling them is not always as straight forward as it seems. In this case we have a "mature" and proven technology of distributed computing and we are taking it in a new direction. Most of the multi-processing work that you speak of is either related to a loosely coupled supercomputer architecture with homogenous internal components to build a computing environment, or is a purpose built machine, or is a software process developed to process a problem on uncoupled hetrogeneous machines.

What do I mean by all the jargon? For those that do not follow this subject area, lets talk about them a bit before we continue.

Historically, top end computers are the most expensive. In the past, high powered computers relied on expensive engineering and fast components to make the fastest machines possible. Seymore Cray made his name in several companies by being just about the best at that game. But high cost started people looking for alternatives. The most successful of these is the loosely coupled model with a large number of identical computers being gathered in one room. Today these are usually created with 1,000 to 5,000 PCs of one form or another.

The problems to be worked then have to be segmented in such a way that the individual nodes can produce a result. In those cases where cross-communication of partial solutions can be allowed with high-speed data busses that connect adjacent nodes in a "Hyper-cube" or other inteconnection mechanism. This development of computing architecture has, in fact. created a new problem. Not all problems that need a supercomputer can be neatly sliced up in this way to be processed and we don't have much of an industry building supercomputers the old way ...

SETI@Home pretty much pioneered the concept of an uncoupled (because our nodes do not under any circumstances commnicate with each other) supercomputer based on part time allocations of resources by a volunteer/participant to the project of choice. Thier success lead to a fair number of other projects to begin using this architecture.

Ok, back on topic ...

BOINC is not the same as the original SETI@Home or any other first generation project. All prior projects used a purpose-built concept for the implementation of the back-end processing with a proprietary, project oriented, software package installed on the participants computer. Fast forward to BOINC and we have a new paradign with a generalized back end architecture and a generalized software package for the participant's computer.

This is revolutionary and, to be honest, I am suprised at how well it is working, but I digress ...

There has never been an implementation before where muliple projects are using 50% or more of the application developed to support a specific science application (ok, it is a made up figure, but that does not invalidate the argument). We truly are on the bleeding edge. The magic is in totality of the system. Granted it uses standard, off the wall, I mean shelf, participants and their computers, but the assembly and architecture is revolutionary.

And some of it does not work well. Exactly why, I don't know, and to a certain degree I don't care. Why? Because I have the luxury now of contributing to 3 projects each and every day. When Predictor@Home comes back, it will be 4. Problems with SETI@Home, no sweat, I have plenty of work from the other projects (heck only 9 more days and I will finish my first cp.net model).

Ok, major mistakes? Sure, and you have never made one? Besides this is a volunteer, low cost, don't break the bank effort. And it is doing reasonably well. You might not want to think so. But you have the option now to vote with your feet and go elsewhere. If the SETI@Home team is evil incarnate and you want to fire them, you have that option. Why not exercise it?

In a long ago series I did an extensive analysis of the database schema and it was not to my liking. And I said why and backed it up with as much research as I could. And I did that discussion in the public for all to read and complain about. And I too said that it appeared that load testing was not done. But you know what? Even if they had done load testing it may not have made any difference. [emphaisis mine]. A trueism of relational databases is that you can never properly load test because real data rarely matches the test data.

If I do a load test I make up records that match the proportions of data that I expect in the proportions that I expect them to arrive and to populate the tables. Even if I get most of that right the tests may go well and I go happy into live with a schema that cannot survive the data streams. And one of the most critical aspects of the relational design is in the proportions of data in the tables.

I asserted then, and again now, that even if I had had complete freedom, and I had been the one to design the schema it is very probable that the database that I designed would also have had performance problems. They would have been different problems, but the same end result may have occurred. And unlike many, as smart as I am, and as much as I know, and as many projects I have worked on, I have never, not once, gotten it all perfect the first time. I know many that post here have never had a failed project and they freely voice the opinion that the developers should be drawn and quartered (and what kind of bed would you like with that?), as a way to atone for the errors of their ways.

But I digress ...

>> Couple that with hardware issues and stir ...
> Sorry, but this is NOT a milkshake,...

Well, I have had milkshakes that came out about as bad. Your mileage may vary.

> In my view and long career in science everything has to do with careful planning, funds,
> manpower and an extreme bookkeeping view of the possibilities of which goal can be achieved.

Ok, so which part are you going to do? Predictor is looking for a post doc, are you going to go there and show us how it is done? Or do you have a product that you have developed that is better than BOINC?

I am always amazed at the people who claim that if only they had been in charge we would be alright. I have, in my days as a Beta tester, and even today, a person that sees problems and I send an e-mail with a statement of the problem to the appropriate person (when I know who that person is). I have critiqued parts of the system publically, and privately. Sometimes I get listened to. Many times I do not, or the problem has to be deferred.

I lurk on the optimization mailing list because it is interesting to see a group of people taking the time to try to write code that we all may be using in the near future. I took a different corner of the BOINC universe, firstly because no one likes to write documentation (spelled it right this time), and the stuff we have, ahem, stinks. Others have taken the tack to provide us with third party tools (BOINC View is my favorite). Others are doing signatures and statictics sites. You guys here seem, to me (this my opinion) to think that you could do better than the staff at UCB. The source code is there... when will I see the improved product? Have you ordered the new servers on your Diner's Club? When will they be in?

Yes, turning off the Pending Credit page is a symptom. Turning off the XML stats is another. I can probably give you a few more, but all of that is beside the point. We are operational. Just like the aircraft carriers I saided on ... if it starts to sink there is no place to go.

On one cruise I had a tester casualty. We started repairs and the maintenance offecer came by to ask what we were doing and I told him we would be back up in an houh. He asked how. I told him we were putting in two parts I had bought in Radio Shack before we sailed and that is how we were going to do it. He was agast, but I asked him; which do you want, a bench that is down for 6 months, or a Radio Shack part that does not fail (we had already done this 3 time previously on other cards in that bench). He chose to stay on-line with my $1.29 worth of parts I bought before we sailed.

I am neither a blind cheerleader, nor a rabid critic. And I am not implying that I see you either way.
<p>
For BOINC Documentation: Click Me!

> Oh, my ... what heat ... well, let me try to take them in order:

Well said!!! (All of it)


[b][i]...only a fool never makes a mistake![b][i]