Panic Mode On (78) Server Problems?

Anybody got any idea whether we're being haunted by Astropulse ghosts, to the same degree?

I think it's getting to the point that I need to put out an urgent APB, reluctant though I am to do so on a Sunday - just to get a lid put on the situation, until they can look at it tomorrow or Tuesday.

The question is - is stopping the MB splitters enough, or should I ask them to stop AP as well?

With splitters stopped, and no new work entering the system, we could allow work fetch and fish for resends of the work we've already supposedly got.

As Marcel said..."Just shoot up here amongst us, one of us has got to have some relief."

:)

There are going to 10s of thousands of timeouts when we run out of shorties & all the VLAR resends start going to NVidia GPUs.

Overnight i got a few resends on my systems, but it's still mostly timeouts. There's been the odd HTTP error, couldn't reach Scheduler, and when it was reached Project has no tasks available.
But mostly it's still timeouts.
Been that way pretty much since a few hours after the last weekly outage, and i notice the database queries are still over 1,000/s where less than 800/s is the usual number.
____________
Grant
Darwin NT.

According to AP Ghosts: on my host http://setiathome.berkeley.edu/show_host_detail.php?hostid=5553346 which only do AP, there are till now no ghost units at all. My other host which is setup as MB only have some, around 100. And only for this host i do manual suspend of network communication (once a day open for 2-3 hours) cause i dont wanna put extra load on the schedular because of NNT.
____________

What I'm seeing now.

This is about MB units, I don't do APs.

Now that the "ready to send" queue has dried up, I've been able to get schedule requests completed even when allowing new tasks. Not getting any new units but I would expect that with the ready queue running on vapors. However Ghost Detector shows I currently have 196 ghost units that hadn't been downloaded but assigned and I should be getting those, eventually. All ATI, almost all shorties BTW.

Problem is that's nearly half of my "in progress" units. And the vast majority of the ones I do have are shorties. I'm guessing I'm down to a couple of days worth of units locally right now.

Que sera sera.
____________
"Life is just nature's way of keeping meat fresh." - The Doctor

I've just had a note back from Eric:

I've stopped the splitters and doubled the httpd timeout...

I think we're going to need to at least temporarily go back
to restricting workunits in progress on a per host basis and per RPC
basis, regardless of what complaints we get about people being unable
to keep their hosts busy.

The splitters are already showing red/orange on the server status page, and 'ready to send' is as near zero as makes no difference (there'll always be a few errors and timeouts to resend). So I'm going to turn off NNT and see what happens - let's see if we can help get this beast back under control.

"In too many cases, it's not a good idea to run multiple projects,
on 1 host."

What data do you have to support that assumption?
And your personal experiences are too small a data point to be
statistically significant. I offset them with mine of running 8 projects without
having any of the problems you have.

At the moment I've been out of SETI gpu work for a while, running Einstein instead. I'm down to 56 cpu APs (about 8 days worth) that are alternating with LHC and Rosetta. When the servers first started screwing up this weekend I went NNT on SETI. Call it foresight, experience, luck, whatever, but I have 0 ghosts and continuous work to do on my cruncher. When the lab boys get this little snafu fixed I'll be waiting because I have what seems to be in short supply around here

patience.

I had set NNT so to run down my cache to switch off my machine. When I looked after it had finished crunching I had 20 or so ghosts, so I unset NNT for a while to try and see if I could get them. I now appear to have 329 tasks I haven't' got.

This is much more than my cache is set for,will I still get them?

PS I have received none so far as well.

Well there was a dip in the cricket graphs and about then I got 20 members of my Ghost Army downloaded. However now that the cricket graphs are back up, I'm getting scheduler timeouts again as it's trying to get my ghosts and report newly done units. :(
____________
"Life is just nature's way of keeping meat fresh." - The Doctor