So... another week another minor server crisis. This one was brewing for a while - we've been getting memory errors/upsets on our main internal file server (which hosts, among other things, all the files that make up the SETI@home web site). We got replacement memory, and were hoping for a quiescent moment to swap it out, but after two crashes in one day (on Tuesday) I just went ahead and did the swap.

So far so good (i.e. no further crashes), except we're still getting memory upsets in the server log. I only replaced 2 of the faulty DIMMs (which were noted as faulty by the motherboard), but maybe others need replacing as well.

In the meantime I found that project recovery today was significantly slowed by the result web pages on our site, so those are turned off at the moment (as I'm writing this).

Meanwhile other tasks this week included cleaning up the lab (the fire marshall is visiting today) and resurrecting SERENDIP code I haven't touched in over a decade. I got it to compile, now I'm just removing the non-fatal compiler warnings one by one. We'll use this code to help process Kepler data (which happens to be in a similar format to our old SERENDIP data). Maybe I'll even get back to analyzing the SERENDIP IV data set (also over a decade old and it may be worth taking another look at it with this code).

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Thanks for the update Matt.


I have a "cure" for fire marshals - pm me for info ;-)

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

Thanks for the update Matt,

Claggy

@ Matt - I've been getting random memory errors on my home server for the last month, and bluescreen lockups, even with ECC memory.

Turned out to be a voltage regulator failure on the motherboard (affecting the memory termination voltage only) - the same memory is working fine in a replacement motherboard.

Might be worth checking voltages, if that motherboard has the right degree of instrumentation.

@ Matt - I've been getting random memory errors on my home server for the last month, and bluescreen lockups, even with ECC memory.

Turned out to be a voltage regulator failure on the motherboard (affecting the memory termination voltage only) - the same memory is working fine in a replacement motherboard.

Might be worth checking voltages, if that motherboard has the right degree of instrumentation.

It also might be worth looking into seeing what the voltages the power supply are putting out. As was discovered/documented in my lengthy capacitor replacement thread, whilst bulged caps were part of the problem, the root of the problem was the power supply. After opening it up, every single capacitor in there is bulged and nearly to burst/ooze stage.

While it isn't very feasible to open the power supply up, you should either be able to see what the board reports for the voltages, or just back-probe connectors with a voltmeter and see what the rails are putting out.

---

Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)

Matt: Ur the MAN. Thanks for the communication(s)..

For days I have languished over my falling RAC; I am finally reading here that all is finally well in the land of Oz. Mr. Wizard: why, oh why does the server status say all is well in Oz, but in reality no project server can be found, either via proxy or direct?

---

Is this why you have such a BIG problem keeping me full of SETI work? I've gone from 4,000+ units per day in late Jan (when I joined) thru mid Feb down to 2,400 and have often run compeletly out of work lately.

---

"results ready to send" = 1, and the creation rate is below 1 per second, NTM that only four "tapes" are "hung"...

---

.

Hello, from Albany, CA!...

You getting a visit from this fire marshal?

http://www.youtube.com/watch?v=PlLPogmB8M8

---

That is exactly my point!! I went from 75K+RAC/day to ZERO. "Project has no tasks available." This does not keep my 5 systems happy at all. Now it is Friday; all personel have left for the weekend; and SETI is sitting idle (it seems).

---

Of course they left for the weekend. It is the weekend.

Downloading several WU right now.

---

That is exactly my point!! I went from 75K+RAC/day to ZERO. "Project has no tasks available." This does not keep my 5 systems happy at all. Now it is Friday; all personel have left for the weekend; and SETI is sitting idle (it seems).

Your main problem could be the BOINC version that you're using, check out the Top Hosts table and see what most of the main setups are using (work is flowing here at a good rate). ;)

Cheers.

---

That is exactly my point!! I went from 75K+RAC/day to ZERO. "Project has no tasks available." This does not keep my 5 systems happy at all. Now it is Friday; all personel have left for the weekend; and SETI is sitting idle (it seems).

Your main problem could be the BOINC version that you're using, check out the Top Hosts table and see what most of the main setups are using (work is flowing here at a good rate). ;)

Cheers.

He's running 6.12.34, the most recent recommended version of the BOINC client... and well represented in the "top hosts" table

---

.

Hello, from Albany, CA!...

Hi Folks,
I've re-installed the Lunatics apps:-) And once again I'm back to haveing ALL GPU tasks running in high priority mode.

Not only S@H but also E@H.

This doesnt apparently occur with CPU tasks.

Is this 'normal' behaviour?

Also since boinc is configured to use 100% resources in both S@H and E@H it 'normally' works at 50% for each, now however its persistantly running E@H GPU tasks non stop. Unless I suspend E@H, let S@H utilse both CPU & GPU then resume E@H.. If I then suspend S@H to let E@H CPU tasks get a look in [they take a long time and expire faster than S@H ones] Then I'm back to square one with E@H hogging the GPU..

So how does one adjust the priority of the GPU tasks. I dont know if this HP mode is detrimental to my GPU cards or not.. But I'm anyway wondering why its gone into hyperdrive as soon as Lunatics is installed

Another thing, it doesnt pick up dumped tasks and restart them, it grabs a new task and does that..
I see several tasks with various % done waiting to be restarted..
I guess they will eventually get to the top of the queue but I suspect only after all other WU are done..

Regards,

---

Cliff,
Been there, Done that, Still no damm T shirt!

That is exactly my point!! I went from 75K+RAC/day to ZERO. "Project has no tasks available." This does not keep my 5 systems happy at all. Now it is Friday; all personel have left for the weekend; and SETI is sitting idle (it seems).

Your main problem could be the BOINC version that you're using, check out the Top Hosts table and see what most of the main setups are using (work is flowing here at a good rate). ;)

Cheers.

He's running 6.12.34, the most recent recommended version of the BOINC client... and well represented in the "top hosts" table

Funny that, I see more of the later 6.10.xx versions myself there.

Cheers.

---

That is exactly my point!! I went from 75K+RAC/day to ZERO. "Project has no tasks available." This does not keep my 5 systems happy at all. Now it is Friday; all personel have left for the weekend; and SETI is sitting idle (it seems).

Your main problem could be the BOINC version that you're using, check out the Top Hosts table and see what most of the main setups are using (work is flowing here at a good rate). ;)

Cheers.

He's running 6.12.34, the most recent recommended version of the BOINC client... and well represented in the "top hosts" table

Funny that, I see more of the later 6.10.xx versions myself there.

Cheers.

yes, but 6.12.34 is in there - so it isn't a problem with the client version... There's just more 6.10.60 clients out there...

---

.

Hello, from Albany, CA!...

That is exactly my point!! I went from 75K+RAC/day to ZERO. "Project has no tasks available." This does not keep my 5 systems happy at all. Now it is Friday; all personel have left for the weekend; and SETI is sitting idle (it seems).

Your main problem could be the BOINC version that you're using, check out the Top Hosts table and see what most of the main setups are using (work is flowing here at a good rate). ;)

Cheers.

He's running 6.12.34, the most recent recommended version of the BOINC client... and well represented in the "top hosts" table

Funny that, I see more of the later 6.10.xx versions myself there.

Cheers.

yes, but 6.12.34 is in there - so it isn't a problem with the client version... There's just more 6.10.60 clients out there...

The issue with the 6.12.xx BOINC versions is the long back-off times compared to 6.10.58 or .60. Many of the larger/faster rigs that have stepped back to 6.10.60 are experiencing significantly less problems with workfetch since they "downgraded". There are several threads in Number Crunching that address that issue.

---

Donald
Infernal Optimist / Submariner, retired

Hi Cliff,
The problem with E@H seems to mainly involve two things they are doing.

First is the way E@H sets their deadlines. They attach -extremely- short deadlines on all their tasks. This causes BOINC to place their tasks' execution priority far ahead of all the other applications' tasks so that the E@H tasks don't expire. This is great for E@H, but gives what one might almost call an "unfair" advantage to their project. This is great for E@H, but makes it difficult for the other projects to either get equal run time or to get the resource distribution you desire.

Second is the rather large number of tasks E@H sends you every time BOINC requests new tasks. The sudden addition of so many longer-running tasks overloads the system and, combined with the short deadlines, gives their tasks priority over the other applications.

I have found that adjusting the Einstein resource allocations to a lower number for E@H helps, but it's -very- difficult to get the right mix of allocations to get E@H to request new tasks on a low allocation versus grabbing too much execution time on slightly higher allocations. I have for some time been experimenting with various ways to get the project mix back to what I prefer on my machines, but E@H's rather unusual combination of deadlines and download limits still interferes with both BOINC's processing distribution and new task request programming. Despite my best efforts, I have to periodically manually intervene to get the mix back to what I want.

The easiest manual intervention method I have found is the following:
1. Select the "Projects" tab.
2. Select (highlight) all projects.
3. Select "No new tasks".
4. Select (highlight) all projects I do not want to change.
5. Select "Suspend".
6. Select (highlight) the project that is now short on new tasks.
7. Select "Allow new tasks"
8. If the low-task project doesn't immediately request new tasks, select "Update".

Watch the project closely so you don't get too many tasks at once. As soon as enough tasks to get things back on track begin to download, reverse the process via the "Projects" tab as follows:
1. Select (highlight) the project that is downloading the new tasks.
2. Select "No new tasks"
3. Select (highlight) all suspended projects.
4. Select "Resume"
5. Select (highlight) all projects.
6. Select "Allow new tasks".

This is a rough way to get around it, but so far it's the only work-around I've found.

Perhaps the only true solution is for BOINC to issue some new "guidelines" to the various Project Administrators on deadlines and quotas and then modify BOINC's programming to handle any accidental variance from these guidelines.

As to your question about the "high priority" status on a running task, it appears to be related more to the deadline date. It seems that a task is placed in "high priority" status if the remaining time versus the deadline place it in jeopardy of not completing in time. I have not found evidence that this has anything to do with physical load on the GPU, but I may be wrong. Perhaps someone else knows the answer to this.

Finally, Cliff, the "dumped" task problem is again related to deadlines. If one or more tasks either have not yet started, or get dangerously close to not completing by their deadlines based on remaining processing time, BOINC will stop running one or more tasks that are not so time-critical and begin running the "at risk" tasks. Don't worry, as long as there are not too many tasks on your machine to complete in time, BOINC will resume computation on the "dumped" task as soon as the "at risk" tasks complete.

I hope this helps, Cliff, and I invite additional insight from anyone who may have found an easier way to deal with this problem. Just like Cliff and the rest of us experiencing this problem, I sure would like any help on how to get BOINC back to running smoothly without so much manual intervention.