We have to help SETI if we all want to really crunch signals! My head is empty right now... I mean jeez.. 1Mil of WU waiting for validation!

Any Ideas?

---

We have to help SETI if we all want to really crunch signals! My head is empty right now... I mean jeez.. 1Mil of WU waiting for validation!

Any Ideas?

Validation@Home

Probably not sexy enough for a new project though =/

---

Sure... like I said in this post: Scrap the credits, both on Seti Classic and here. :)

Then I don't see anyone wory about it anymore. Quick, slick and clean solution. :)

---

I should reiterate:

A quick rough estimate shows me that we have over 50 million results to validate in SETI classic (it was upwards to 100s of millions at various points). Of course, nobody notices this as they get credit in SETI classic before the results are validated.

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

I should reiterate:

A quick rough estimate shows me that we have over 50 million results to validate in SETI classic (it was upwards to 100s of millions at various points). Of course, nobody notices this as they get credit in SETI classic before the results are validated.

- Matt

Know what's really worrying me? We sent our new guy out for pizza and beer an hour ago and he ain't back yet. Funny how one event puts the others in perspective. Credits?? What credits? Where's my @#%$@ pizza?!? Onwards and upwards, Matt, keep after it! :)
_______
* Will Admin your system for food

Validation@Home

Sounds like a good idea.. If only I knew anything about programming.

---

- cJ

Validation@Home

Sounds like a good idea.. If only I knew anything about programming.

The problem is that the current process is file system bound, not CPU bound, so the extra CPU horsepower will not help.

---

BOINC WIKI

What Should be done is have an adjustable system. Measure the different type loads on the different servers and when one gets overloaded and another is going idle have it automatically move some of the load to the other server. Just looking at Ware-Wulf system for Beowulf clusters on Linux. It allows you to control how processes and resources are allocated amongst the individual beowulf clusters. No node is defined as any particular server and any node can assume the role of any needed process or resource. You can move some of the files to other servers; Can't you?

---

Moving the data to a different raid rack would take a few days and the project would have to be down the whole time.

Moving small files has a lot of overhead, the maximum throughput even for a gigabit switch is still something like 125MB/s.

It is just a nasty problem.

---

----- Rom
BOINC Development Team, U.C. Berkeley
My Blog

Moving the data to a different raid rack would take a few days and the project would have to be down the whole time.

Moving small files has a lot of overhead, the maximum throughput even for a gigabit switch is still something like 125MB/s.

It is just a nasty problem.

It does seem to me that there must be a large amount of overhead processing all these things as individual files in the a system. Has anyone done a comparison of storing this data in LOBs in the DB. It looks like there is a DB record for every WU and result. A RDBS should always be better at handling creates, updates, & deletes than a file system. I mean the file system is dealing with directoies, space allocations, and security that just doesn't seem necessary.

Just my thoughts.

---

It is just a nasty problem.

[font='courier,courier new']I think you win "Understatement of the Year" :-\\

Lotsa luck, Rom.[/font]

---

It does seem to me that there must be a large amount of overhead processing all these things as individual files in the a system. Has anyone done a comparison of storing this data in LOBs in the DB. It looks like there is a DB record for every WU and result. A RDBS should always be better at handling creates, updates, & deletes than a file system. I mean the file system is dealing with directoies, space allocations, and security that just doesn't seem necessary.

Sorry, but what is a LOBs and how would it help?

---

May this Farce be with You

A.k.a. BLOB ...

The point that is missed is that the database content is also written to disk. So, you still have the data written to disk. Now the difference is that you have the RDBMS doing the file searching. It does not mean that it is any more efficient. It just sounds like it will be faster.

Granted "Longhorn" was supposed to "eliminate" the file system ... Ha! ... Win95 was supposed to eliminate DOS too ...

---

Granted "Longhorn" was supposed to "eliminate" the file system ... Ha! ... Win95 was supposed to eliminate DOS too ...

How does one eliminate a file system and still have a functioning computer? Doesn't make sense to me. But what do I know.

Jim

---

I don't know the details of how the validator works, but make a program that will pair up say 20 work units and all their partners, compress them, then send them out to our computers to actually validate, and then send back the numbers only.

---

- cJ

Granted "Longhorn" was supposed to "eliminate" the file system ... Ha! ... Win95 was supposed to eliminate DOS too ...

How does one eliminate a file system and still have a functioning computer? Doesn't make sense to me. But what do I know.

Doesn't make sense to me either!

It could likely be just MS marketing-speak and obfuscation. MS are very fond of obscuring the 'workings' to give 'extra gloss' and the impression of there being more than what there actually is in reality.


There are abstract ideals of merging database-like functionality directly into the filesystem, rather than having a database doing filesystem-like manipulations on top of the filesystem. This is one of the ideals for ReiserFS.

(If you check the ReiserFS web pages, there's lots of description for a 'unified namespace'. Essentially, just as a good kernel is not seen by the user, the filesystem becomes seamlessly part of the overall system. The user then no longer need think of 'files'. Their work and data are safely stored with database-like accessibility.)

Regards,
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

A.k.a. BLOB ...

The point that is missed is that the database content is also written to disk. So, you still have the data written to disk. Now the difference is that you have the RDBMS doing the file searching. It does not mean that it is any more efficient. It just sounds like it will be faster.

Granted "Longhorn" was supposed to "eliminate" the file system ... Ha! ... Win95 was supposed to eliminate DOS too ...

Actually, Paul....

The idea of storing them as a BLOB isn't a bad idea because the problem is simply dealing with a directory that is full of files, not the files themselves.

So, if they were stored as a BLOB instead, they'd all be in one file.

But, as Matt pointed out, any proposal to make a change, BLOBs, a bigger directory fanout, multiple results per file, or any other idea means making a change to all of the programs that use those files (transitioners, validators, assimilators, etc.).

So, the project has to be taken down while those components are altered, then all of the files have to be moved to the new format.

The conversion process would be subject to all of the same struggles we're having right now, and validation would stop while it's being done.

Yeah, you could do something else, like put in code to look in the new place, and then the old place. I think it'd be a good idea to store part of the path in the database so that the directory assignments happened in one spot (and the fanout could change in just one spot).

But most of these just move the problem around, and at great expense.

So, it makes the most sense to stay with the current course and try to clean up the directories before something more dramatic is tried.

---

I don't know the details of how the validator works, but make a program that will pair up say 20 work units and all their partners, compress them, then send them out to our computers to actually validate, and then send back the numbers only.

But then we'd need a validator-validator, and the thing making the validation@home work units would still have to get those work units from the same overstressed file system we've got now.

I don't know where Matt and his accomplices are on their attempts to fix the problem, they may not have found and deleted all of the orphans.

---

I don't know where Matt and his accomplices are on their attempts to fix the problem, they may not have found and deleted all of the orphans.

Considering that the number is still getting bigger, either deleting the files didn't help out, or there is a different problem.

---

- cJ

I don't know where Matt and his accomplices are on their attempts to fix the problem, they may not have found and deleted all of the orphans.

Considering that the number is still getting bigger, either deleting the files didn't help out, or there is a different problem.

The technical news said they'd do one or more outages to delete files, and I don't think we saw the one we expected yesterday.

---