My vote was for shutting down the splitters for a week (or 2 or how ever long it takes), and just have people process resends until such time as the Validation & Assimilation backlog's have cleared.
Not started to clear, but fully cleared.

This would alienate all those users who are not following these forums making them quit or switch to other projects permanently. Loss of users would help the server congestion but hurt the science progress.

I think letting the backlogs clear at the start of a Tuesday downtime would make a big difference. Especially if they also trigger the validation of all those results that have missed validation for various reasons over the last weeks and are now waiting for the deadlines. The resend cycle wouldn't clear but they are a small percentage of all the tasks. The huge 'Workunits waiting for assimilation' backlog that is now 3.5 million and still rising would clear.

Those workunits waiting for assimilation must have corresponding result rows still in the database at least for the canonical result but probably for all the results because I have never seen any workunit in the website show part of the results deleted while the workunit still exists. The number or results waiting for assimilation is not shown on SSP, so I guess those results may be still counted in the validation queue. It this is the case, then those may explain over 7 million of the current 12 million result validation queue!

Once we have the new NAS device up and running, bump up the limits &...

When the problem is the database not fitting in RAM, the disk performance increase won't fix the problem. It only reduces the magnitude of the consequences a bit.

---

My vote was for shutting down the splitters for a week (or 2 or how ever long it takes), and just have people process resends until such time as the Validation & Assimilation backlog's have cleared.
Not started to clear, but fully cleared.

Pull all BLC35 files and then restart the splitters with 100 + 100 serverside limits again.
Once we have the new NAS device up and running, bump up the limits & reintroduce the BLC35 files and then use them to stress test the system. If it fails again then it's fundraising time for new database servers that are capable of handling the load (that really needs to be done anyway in order to meet the projects goals of many more crunchers returning much more work).

+1
I think this is a great idea. We will all still get work... just make it the resends, until db reaches a good size.
p.s. The idea of processing data without a wingman, or having a bad result put in over my good result is BS and worthless. I love Seti, but I don't want feel good theater, I want SCIENCE! so I'm NNT until it gets better.

The Splitters have fallen off again, most requests are receiving 'Project has No tasks...' again. Caches are falling.... one is down by 50% already, and so it continues.

Oh, the problem with failed Uploads has also returned. Probably has something to do with returning around 60 to 70 completed tasks every 5 minutes.

Now they have apparently switched to 'initial replication 1': 3861450832

So no more risk of bad results returned first making good results returned later fail, but also no chance whatsoever of catching the bad results.

---

Once we have the new NAS device up and running, bump up the limits &...

When the problem is the database not fitting in RAM, the disk performance increase won't fix the problem. It only reduces the magnitude of the consequences a bit.

Depending on how much better they perform, the need for all of it to fit in RAM may not arise (although that is rather wishful thinking- i am expecting the new storage to be significantly faster than the exiting storage, however i don't expect it to be significant enough.).
Or they could be replaced by an AFA (All Flash Array), negating the need for the entire thing to fit in RAM. Or a new server with more RAM. Or better yet, both.

---

Grant
Darwin NT

Results received in last hour = 197,095
just a matter of time now, probably not long.

Results received in last hour = 197,095
just a matter of time now, probably not long.

Already getting "Project has no tasks available" messages, i think Tbar posted similarly in another thread. Caches running down.
Not surprising considering the return rate & the increasing Validation & Assimilation backlogs- both have reached new record highs.

---

Grant
Darwin NT

Or they could be replaced by an AFA (All Flash Array), negating the need for the entire thing to fit in RAM.

Setiathome Boinc database running from flash would burn out the flash in a short time!

---

If we are no longer validating the WUs properly... some of mine don't even have a wingman... why is Results returned and awaiting validation number growing in the status??

edit: Ville - I love your Pluto pic.

Or they could be replaced by an AFA (All Flash Array), negating the need for the entire thing to fit in RAM.

Setiathome Boinc database running from flash would burn out the flash in a short time!

After several decades.
Yes, if you were to use consumer/client SSDs they would die rather quickly, however SSDs designed for enterprise use will last an extremely long time, under much heavier use than Seti provides.

For example, DWPD (Drive Writes Per Day, where the entire capacity of the drive is written to in a 24hr period). Consumer drivers are rated at around .1 to .5, Enterprise drives are rated as high as 3 DWPD, some specialised write drives even higher.
And of course with multiple drives in an array or pool, even udner the heaviest of loads, they will never come close to their rated maximum DWPD limit.

---

Grant
Darwin NT

Ville - I love your Pluto pic.

I got the idea of using a Pluto pic from you ;)

---

If we are no longer validating the WUs properly... some of mine don't even have a wingman... why is Results returned and awaiting validation number growing in the status??

My theory is that the Transitioner isn't (hasn't) marked all those returns as 'ready to validate' - I think the bulk of them have been sitting there untouched since the December troubles.

Eric replied - very late on Thursday night, his time -

I will see if I can figure out a transitioner trick tomorrow, in which case I will revert to standard replication.

(I suggested that Matt might have a script for that - I think we've done it before)

The first machine is now out of work, https://setiathome.berkeley.edu/results.php?hostid=6796479
The Next machine's cache is down by 60%, it will be out soon, https://setiathome.berkeley.edu/results.php?hostid=6813106
The load is still above 190k, and the Splitters can't keep up, https://setiathome.berkeley.edu/show_server_status.php

If we are no longer validating the WUs properly... some of mine don't even have a wingman... why is Results returned and awaiting validation number growing in the status??

If my theory about results belonging to workunits waiting for assimilation being shown as waiting for validation is correct, then we could have about 7.5 million of the 12.2 million results there being ones that have been validated but not assimilated yet. And that is growing fast.

The 'Workunits waiting for assimilation' is a supposed to be close to zero in normal situation because workunits get assimilated immediately after they have been validated. But for more than a week now that number has been steadily growing. Recently by about 30000 per hour. The Astropulse number has also been growing the last couple of hours. There is some serious performance problem in assimilation.

---

The first machine is now out of work

You are crunching too fast. My caches are nearly full in both machines.

---

Because I'm nowhere near the bulk of my computers I've had to resort to using the web options page to set don't do any SETI work - the first time I've had to resort to this sort of thing due to actions by SETI :-(

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

There is a noise bombing window in blc35 at around 58692_07 and _08. Those are probably causing the current high return rate.

---

Eric replied - very late on Thursday night, his time -

Is he aware of the assimilation problem?

---

Eric replied - very late on Thursday night, his time -

Is he aware of the assimilation problem?

He didn't mention it, but I would expect so, yes: that's an unambiguous figure on the face of the SSP (and the more complete figures which, I presume, they have access to via internal monitoring).

The 'results awaiting validation' and 'workunits awaiting validation' figures are also unambiguous, but they are unusual - why are they so different? The first usually hovers around 4 million, but recently it's been 12 million. Why?

The rise started when the 'in progress' limit was raised - an obvious direct connection, no alarm bells. But why is it still so high? That needs explanation, and I've suggested a possible way of finding out the answer. Lets hope it works, else someone is going to have to come up with another suggestion.

If this is an attempt to reduce the amount of work sitting around waiting to be validated it's not working as the number has increased from about 11,500,000 last night to about 12,250,000 this morning.

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?