Yeah Matt is really missed in times like these and he would've had those MBv7's put to bed long ago.

Wait, I've been out of the loop for a while. Matt left?

Yeah Matt has been over at the Breakthrough Listen project for a couple of years now.

Or maybe just at the Breakthrough Listen office down on Campus - that's where I found him back in July.

https://i.imgur.com/Lninw9X.jpg

And now and again out at Parkes.

Cheers.

. . Not recently I don't think :(

Stephen

? ?

A word of warning- if you do manage to sore some work, be prepared to have to Retry pending transfers a few 100 (it feels like a thousand) times.
The download servers are borked as well as everything else.

. . been there, done that, over and over and over and ... oh what the heck ...

Stephen

:(

Just ran across something new.

I looked at my pending page and found a workunit at randon.
It said "completed, waiting validation.

But when I clicked on it to see the status, I found this


https://setiathome.berkeley.edu/workunit.php?wuid=3863183873

Reported by 3 units and validated.

Something is borked.

---

Dave

That's not unusual. The first 2 tasks did not match but were close. The 3rd was close enough to validate the first 2.

---

Reported by 3 units and validated.

Something is borked.

Side effect of the minimum 3 quorum for early overflows and the flakey AMD card problem.

---

Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)

I am getting a very small flow of Seti@Home tasks.

So I have NNTed both E@H and WCG in hopes of sucking more down :)
Some of the weather tasks run more than a half a dayso it will take a while to widdle about half of them down.

Tom

---

A proud member of the OFA (Old Farts Association).

I guess I was not clear.

Looking at my pending page, it says that the workunit is "completed, waiting validation"

YET

looking at the workunit, it says validated.

Just ran across something new.

I looked at my pending page and found a workunit at randon.
It said "completed, waiting validation.

But when I clicked on it to see the status, I found this


https://setiathome.berkeley.edu/workunit.php?wuid=3863183873

Reported by 3 units and validated.

Something is borked.

---

Dave

Just ran across something new.

I looked at my pending page and found a workunit at randon.
It said "completed, waiting validation.

But when I clicked on it to see the status, I found this


https://setiathome.berkeley.edu/workunit.php?wuid=3863183873

Reported by 3 units and validated.

Something is borked.

. . Not sure what your issue is with that WU. Normally a WU will linger in the system for approx 24 hours after validation, currently with the problems everyone has been discussing and are very concerned about they are hanging about for much longer. That unit has only just validated so I would not expect to wave goodbye to it for a day or 3 yet ...

. . OH, maybe you missed the discussion of the change that was introduced because of NAVI cards whereby overflow results are being triple checked for validation.

{edit}
. . The misleading listing on the your stats page may be due to the lag with the replica database??

Stephen

:(

Yes, things are borked. That is what this thread has been discussing for the past two weeks. Also the replica database is 8000 seconds behind. So what you see on your page is already 2 hours old.

There is nothing normal about the current situation so no reason to expect normal classifications. I would just not worry about it since there is nothing you can do on your end to change anything.

---

Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)

There will also be weirdness from the replica db not being caught up with the main db.

Tired of this current snafu. Shut my crunchers down and will hold off until SETI has a couple of days of normal work flow. Good crunching, all.

---

I am seeing no reductions in the size of the database with all the task counts at all time highs. Nothing is going to happen until we fall below the magic 20M number.

We fell below that at 02:50 UTC, but nothing has happened yet at the splitters.

Assimilation queue seems to be slowly going down now. Two steps down, one up.

---

Assimilation queue seems to be slowly going down now. Two steps down, one up.

Until we can get "Results returned and awaiting validation" down to around 3.5 million (given the present amount of Work in progress- so 7 million to go), and the "Workunits waiting for assimilation" back down to 0 (3.7 million to go), any new work just causes those numbers to climb.
And ideally we'd want the purge numbers to be within about 500k of the In progress numbers (i think that was the general ball park when things were working).

---

Grant
Darwin NT

Until we can get "Results returned and awaiting validation" down to around 3.5 million (given the present amount of Work in progress- so 7 million to go), and the "Workunits waiting for assimilation" back down to 0 (3.7 million to go), any new work just causes those numbers to climb.

If the underlying problem is not fixed, the numbers will just start growing again no matter how low they were driven.

Apparently the splitters are occasionally running in so short bursts that the SSP can't catch them. I got a small bunch of freshly split _0s and _1s. Mostly noise bombs.

---

If the underlying problem is not fixed, the numbers will just start growing again no matter how low they were driven.

Yep.
It appears we've just about finished all the BLC35 noise bombs***. And there is now a fix for the AMD RX 5000 card issues.
While the increased serverside limits didn't help things, it was those 2 issues that really brought things undone- as the way to stop dodgy results getting in to the science database was require more than 1 wingman to verify a noisy WU result. Combined with files that were producing almost nothing but noise bombs, the size of the database exploded as the hardware just couldn't keep up with the load. And there may have been other performance related issues that have contributed to the initial database rapid expansion & the corresponding excruciatingly slow recovery.


Having said that, it shows that we really do need new hardware in order to meet (not too distant) future workloads (let alone the continuing upload & download server issues).


Edit-
*** Having said that, there's still a big heap of them still to come (there were that many noisy files there).

---

Grant
Darwin NT

Yeaaaaah, a lot of tasks for for CPU and CPU+GPU are now waiting :)

Edit: Except for the replica, which is now 5,91 hours behind, and it's getting worse for each update of the SSP. :-(

Fun time, I just config'd graphs for replica:
https://munin.kiska.pw/munin/Munin-Node/Munin-Node/replica_setiathome.html

This should make Grant happy :D

Yeaaaaah, a lot of tasks for for CPU and CPU+GPU are now waiting :)

It's nice to get work, but it would have been nicer (given how things are at present) for the backlogs to be a few more million down before that happened.

---

Grant
Darwin NT

This should make Grant happy :D

Very nice.
Now, if the "Results returned and awaiting validation" were on the same graph as the "Results out in the field" for both for MB & AP it'd be perfect (they're the same order of magnitude as each other- millions for MB and hundreds of thousands for AP, whereas the Assimilation & Deletion numbers are (when things aren't broken) usually around 0 so with the values in their millions there it makes it harder to see what's been going on with the smaller values).

Oh, and the "Workunits waiting for db purging" and "Results waiting for db purging" could also go on the "Results returned and awaiting validation" and "Results out in the field" graph (or have their own).
Pretty please. Pretty please with a cherry on top.

---

Grant
Darwin NT

This should make Grant happy :D

Very nice.
Now, if the "Results returned and awaiting validation" were on the same graph as the "Results out in the field" for both for MB & AP it'd be perfect (they're the same order of magnitude as each other- millions for MB and hundreds of thousands for AP, whereas the Assimilation & Deletion numbers are (when things aren't broken) usually around 0 so with the values in their millions there it makes it harder to see what's been going on with the smaller values).

Oh, and the "Workunits waiting for db purging" and "Results waiting for db purging" could also go on the "Results returned and awaiting validation" and "Results out in the field" graph (or have their own).
Pretty please. Pretty please with a cherry on top.

Once it starts populating :D
https://munin.kiska.pw/munin/Munin-Node/Munin-Node/results_setiathomev8_in_progress_validation.html

Remind me to do the other stuff later