Panic Mode On (77) Server Problems?

I can track batches of shorties that my 24 core box is processing via its UPS load stats. It's only about a 2% difference drop in power when running shorties, but at that machines power level it is apparent.
Snapshot of the past few weeks.
http://www.hal6000.com/seti/images/ups_load-month.png

It's after noon in Berkeley and I haven't seen any indication that the fellas in the lab have changed anything.

I assume they know there is a problem with Uploads.

I've got computers showing "project backoff" times of up-to almost six hours and similarly sad "retry" times.

The repeating wave on the incoming side of the cricket graph makes me think there is "something" that doesn't like "something" and whatever is broken is sneezing at semi-regular intervals; not merely that the communications are simply slow because of an over-abundance of data. (shortie storm, AP distribution, the usual suspects)

The "sawtooth" shape of the thing (without correlation with the "outflow") makes me wonder. I hope it's making someone else wonder, too.

I don't care, of course, until there are so many un-uploaded (and therefore unreported) work units in all of our queues that all of the Clients need manual intervention to be able to "Update." We (who read here) will be fine, but I'd wager a fair number of crunchers aren't going to be willing to babysit the process for hours (retry now / update, retry now / update) to prevent the "report" queue and client_state files from becoming unmanageable for the Client.

I've noticed that several of mine have refused to auto "Update" while there are hundreds of Uploads stuck. Me? I keep-up with it and try to mitigate the damage. Thank goodness I had a situation last night where I could babysit one of my rigs every few minutes for about 3 hours "retrying" then "updating" of something like 1,500 tasks (and that isn't even a particularly fast machine). I suspect that there are a lot of people who won't do that.

It really isn't important whether the work gets reported now or tomorrow or a week from next Thursday, but it would be a crying shame to have all that crunching eventually time-out and go to waste because it has "broken" the Client's ability to deal with the backlog.

You might want to check out cacti it is one of the pieces of software I use.

Well I hope that the guys work out what they didn't get right during the last outage during tomorrow's outage as this past week has been very annoying.

Cheers.
____________

I have two WU' downloading and it seems they've been downloading for days...when I hit retry it gets a few bytes then just craps out...not
a problem with any other projects, just SETI. If they're using some kind of bandwidth throttling they should turn it off, or check to see if they have a failed NIC or something else in the pipeline.

Well the servers are up, it took all my results and reported them successfully but the cricket graphs are still dead. Did you upgrade their backbone so our stuff isn't being piped through that link? I thought the new gigabit switch was only internal.

Edit: Oops, my bad, seems like I just happen to turn on internet access just at the point the servers came up. Looks like the cricket graph is now starting to go up.
____________
"Life is just nature's way of keeping meat fresh." - The Doctor

No luck getting much to download here - says project has no tasks available. Must have to just wait it out for a bit.
____________

Not downloading here either. :(

Well overall I'm mostly good for CPU work. One machine has lots to keep it going for close to a week. Unfortunately, by brand new ATI 7870 GPU has nothing to impress me with, my ATI 4870 GPU will be out of work before I go to bed tonight, and my ATI 6300 (Turks) is also twiddling its thumbs. Ohh Well.
____________

No luck getting much to download here - says project has no tasks available.

Well, just got 4 CPU WU, but the download was no better than the last 4-5 days.
____________

It seems that the guys got onto what ever it was that was causing problems since the last outage as all but my lowly P.O.S. Vista rig are back up to full (it surprised me though that that P.O.S. was able to report well over 400 in its first 2 hits at the servers once they came back up).

Now if the guys can just restrain themselves to just 1 new tape at a time until the AP's are done things will go well.

Cheers.
____________

What's interesting is all the WU's i've been getting the last 4-5 days are all due on or around Oct. 5th, and all run high priority.
____________