I thought it was on disk as well

It is, and that's the problem. Until it is read in to memory then any access is extremely slow. If it doesn't all fit in to memory, then it has to be read from (and written to) disk instead- which is orders of magnitude slower than memory, particularly for random reads & write.

---

Grant
Darwin NT

My oldest valid is from 18 July WU 3056429782

Apples and Oranges, I think...
Likewise, your oldest valid has only been valid for 6 hours, as that's when the wingman reported that would compare against two previous results to resolve an inconclusive.

Agreed. That one should be purged at 4 Nov 2018, 01:32 UTC - 24 hours after it was validated. I'll be asleep in bed, but if anyone wants to check, they can work out how long the backlog is at that time.

edit : ah my misunderstanding. thought you guys were talking about valids, not pendings

My oldest pending was returned 19 July. WU 3058299862

My oldest valid is from 18 July WU 3056429782

. . Sorry Keith but that valid is from the 3rd November 2018. Prior to that it was an inconclusive. Before that it was a pending because you had two dud wingmen that timed out. And back to my pet gripe .... funny how it keeps turning up.

Stephen

:(

If it can't fit into RAM where does it go?

. . Umm, it is on disk. That is what makes access to it slow ...
Stephen

I thought it was on disk as well

. . If it cannot fit into RAM it is ONLY on disk, ... hence slow ...

Stephen

. .

LOL. You only have 16 tasks outstanding in the database. I have 55,000 outstanding in the database.

Hey... be nice... my mac mini is proud of every one of the tasks it does. It may be old and slow but it has done a bunch of WUs over many years.

But I'm still not sure why you see more than 24 hours of valids? Maybe your machine spits them out so fast that you have a long list of pending... just wanting around on the partner machine (probably something slow like mine).

It wouldn't cost the db much to show me more than 24 hours of history.

edit : ah my misunderstanding. thought you guys were talking about valids, not pendings

. . In that case it is not a matter of purging the database as they will remain until the last lame duck wingman completes the task or times out and someone else does. And that could be years ... :( {as in where one dud wingman times out on a task only to have it pass to another delinquent host and then another ... etc}. Which brings me back to an old bug bear of mine about deadlines ... :)

Stephen

. . Message from earlier got stuck when my PC crashed (again) due to temp problem.

:(

Stephen has said it in a way I get now. It (A task and it's associated WUs) is purged 24 hours after the slowest valid on the task. It can't be any closer than that or I'd never see any stats. I'm usually the slow one. Of course when the system has a hiccup db purge is usually the thing last done, so that is a part of it too.

I've often wondered how they pick due dates for the WUs, but that might be a discussion for another thread. I'm sure it is a delicate balance between managing the size of the database, and making sure the slower machines get a chance to be a part of the seti project.

Is your outstanding usually this high? or the result of finishing more WUs than usual due to the recent noise bombs??

OK, poor choice in examples. Not relative to the original discussion. I do have a tendency to accumulate outstanding pendings that DON'T CLEAR after the last validating task is validated and past the deadline. And the database misses it. Then have to wait the database to pick it up again after the original deadline on the third or fourth wingman.

I would say my task load is maybe 20-30% higher due to all the noise bombs recently. The point I was trying to get across is when the purgers and validators are still trying to clear the backlog after the outage, the website is unusable for me to view tasks because of the long waits and timeouts. As I said, the slope in the purge line finally turned downward yesterday which made my attempt to find my oldest tasks even possible.

---

Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)

@ Unixchick:

I've often wondered how they pick due dates for the WUs, but that might be a discussion for another thread. I'm sure it is a delicate balance between managing the size of the database, and making sure the slower machines get a chance to be a part of the seti project.

Yes, the deadlines are tuned in accordance with the estimated runtimes of the tasks, and (for SETI data, not Astropulse) with a very strong presumption that even the slowest devices should be able to complete tasks and contribute to the search.

I don't think we've exhumed it since you started contributing to the boards, but there's a 10-year-old thread - Estimates and Deadlines revisited - which might give you some insight into the thought processes involved.

OK, poor choice in examples. Not relative to the original discussion. I do have a tendency to accumulate outstanding pendings that DON'T CLEAR after the last validating task is validated and past the deadline. And the database misses it. Then have to wait the database to pick it up again after the original deadline on the third or fourth wingman.

I would say my task load is maybe 20-30% higher due to all the noise bombs recently. The point I was trying to get across is when the purgers and validators are still trying to clear the backlog after the outage, the website is unusable for me to view tasks because of the long waits and timeouts. As I said, the slope in the purge line finally turned downward yesterday which made my attempt to find my oldest tasks even possible.

. . In that respect you have my sympathy. For the last few months I have been monitoring my inconclusive rate looking for any pattern, reviewing page after page of valid tasks (twice per day) to identify the oldest one listed so I know which inconclusive tasks to count, then reviewing page after page of inconclusives (looking at each task individually) sorting out the current ones from many that are months old due to dud wingmen. On the slow machines it isn't too hard, but on my "good" machine it can take forever with only 2000 or so results. With your numbers I wouldn't even try, I'd go nuts.

Stephen

The results received in the last hour has hit 150K and all is running fine, and there only appears to be some minor lag in db purging. I think we will have a mild panic Monday morning as the files to split will get low as we seem to be going through them quicker. (1800-2000/day as a rough guess)

and a throw back to another topic:
Thank you Richard for the link to the thread about WU deadline dates. Some good work on that thread. Enjoyable reading.

and a throw back to another topic:
Thank you Richard for the link to the thread about WU deadline dates. Some good work on that thread. Enjoyable reading.

. . But looking at the data Joe Segur posted over 10 years ago with hardware much less powerful than that currently being used, deadlines should be no more than one month ( about 4 weeks) yet today they are still set to 2 months or even longer. As you say, a lot of good work, but is anybody reading it (apart from Richard).

Stephen

? ?

The results received in the last hour has hit 150K and all is running fine, and there only appears to be some minor lag in db purging. I think we will have a mild panic Monday morning as the files to split will get low as we seem to be going through them quicker. (1800-2000/day as a rough guess)

. . Less than 1800 channels left so I hope they have another dataset to load when the guys get to work :)

. . Or we will be running on empty by the end o f day ...

Stephen

<fingers crossed>

. . But looking at the data Joe Segur posted over 10 years ago with hardware much less powerful than that currently being used, deadlines should be no more than one month ( about 4 weeks) yet today they are still set to 2 months or even longer. As you say, a lot of good work, but is anybody reading it (apart from Richard).

Ten years ago, we didn't support Android devices, either.

Ten years ago, we didn't support Android devices, either.

. . And how is that working out? :)

Stephen

:)

About this well :-)

Android (ARM processor)			8.00 (armv6-neon)		192 GigaFLOPS
Android (ARM processor)			8.00 (armv6-neon-nopie)		22 GigaFLOPS
Android (ARM processor)			8.00 (armv6-vfp)		64 GigaFLOPS
Android (ARM processor)			8.00 (armv6-vfp-nopie)		15 GigaFLOPS
Android (ARM processor)			8.00 (armv7-neon)		204 GigaFLOPS
Android (ARM processor)			8.00 (armv7-neon-nopie)		31 GigaFLOPS
Android (ARM processor)			8.00 (armv7-vfpv3)		207 GigaFLOPS
Android (ARM processor)			8.00 (armv7-vfpv3-nopie)	29 GigaFLOPS
Android (ARM processor)			8.00 (armv7-vfpv3d16)		192 GigaFLOPS
Android (ARM processor)			8.00 (armv7-vfpv3d16-nopie)	28 GigaFLOPS
Android (ARM processor)			8.00 (armv7-vfpv4)		195 GigaFLOPS
Android (ARM processor)			8.00 (armv7-vfpv4-nopie)	21 GigaFLOPS
Android (Intel/AMD x86 processor)	8.00 (nopie)			3 GigaFLOPS
Android (Intel/AMD x86 processor)	8.00 (pie)			80 GigaFLOPS
Android (ARM64 processor)		8.00 (arm64-neon)		133 GigaFLOPS
Android (ARM64 processor)		8.00 (arm64-vfpv4)		118 GigaFLOPS
Android (ARM64 processor)		8.01				716 GigaFLOPS
		
									2250 GigaFLOPS

See we have new BLC04 work posted now. Should slow down the returns and slow the drop in RAC from the fast, low-paying BLC01.

---

Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)

About this well :-)

Android (ARM processor)			8.00 (armv6-neon)		192 GigaFLOPS
Android (ARM processor)			8.00 (armv6-neon-nopie)		22 GigaFLOPS
Android (ARM processor)			8.00 (armv6-vfp)		64 GigaFLOPS
Android (ARM processor)			8.00 (armv6-vfp-nopie)		15 GigaFLOPS
Android (ARM processor)			8.00 (armv7-neon)		204 GigaFLOPS
Android (ARM processor)			8.00 (armv7-neon-nopie)		31 GigaFLOPS
Android (ARM processor)			8.00 (armv7-vfpv3)		207 GigaFLOPS
Android (ARM processor)			8.00 (armv7-vfpv3-nopie)	29 GigaFLOPS
Android (ARM processor)			8.00 (armv7-vfpv3d16)		192 GigaFLOPS
Android (ARM processor)			8.00 (armv7-vfpv3d16-nopie)	28 GigaFLOPS
Android (ARM processor)			8.00 (armv7-vfpv4)		195 GigaFLOPS
Android (ARM processor)			8.00 (armv7-vfpv4-nopie)	21 GigaFLOPS
Android (Intel/AMD x86 processor)	8.00 (nopie)			3 GigaFLOPS
Android (Intel/AMD x86 processor)	8.00 (pie)			80 GigaFLOPS
Android (ARM64 processor)		8.00 (arm64-neon)		133 GigaFLOPS
Android (ARM64 processor)		8.00 (arm64-vfpv4)		118 GigaFLOPS
Android (ARM64 processor)		8.01				716 GigaFLOPS
		
									2250 GigaFLOPS

. . Those numbers seem good(ish) but apart from highlighting the need for 'pie' (whatever that is) how does that compare to say an i3 or an AMD8350?

Stephen

?

See we have new BLC04 work posted now. Should slow down the returns and slow the drop in RAC from the fast, low-paying BLC01.

. . Well good news (for me) but bad news for you I think, those Blc04 tapes are also GBT-New Format. Run times only a few seconds longer than the Blc01 tapes.

Stephen

:)

Since all data split for more than a year has been the "new" format cancel your thought about anything we are seeing today being a "new format" because it isn't.
It's the data contained within the tape that is having an impact on the run-time -sometimes we get data that is much faster to process than others.

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

. . Those numbers seem good(ish) but apart from highlighting the need for 'pie' (whatever that is) how does that compare to say an i3 or an AMD8350?

ARM CPUs are computationally roughly on par with P4 CPUs from many years ago, but use way, way, waaay less power.

---

Grant
Darwin NT