SETI@HOME -- A Victim of it's Own Success?!?


log in

Advanced search

Message boards : Number crunching : SETI@HOME -- A Victim of it's Own Success?!?

1 · 2 · 3 · Next
Author Message
w1hueProject donor
Volunteer tester
Send message
Joined: 4 Aug 00
Posts: 48
Credit: 1,804,615
RAC: 1,009
United States
Message 1340931 - Posted: 26 Feb 2013, 4:22:34 UTC

It's now taking over 1 day for my machines to successfully download a WU! I was considering adding the SETI project to a new machine, but why bother?
____________

Speedy
Volunteer tester
Avatar
Send message
Joined: 26 Jun 04
Posts: 678
Credit: 5,922,634
RAC: 3,922
New Zealand
Message 1340937 - Posted: 26 Feb 2013, 5:09:07 UTC

You ask why bother? I would do it because you want to help with the science or you would like to find out whether or not there are other people in the world apart from humans.

Part of the reason for your computer taking a day to successfully download one work unit may have been because the system was recovering from lab wide car appears over the weekend and this servers needed to catch up with the demand of the of the projects large user base.
____________

Live in NZ y not join Smile City?

ClaggyProject donor
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4140
Credit: 33,543,011
RAC: 25,043
United Kingdom
Message 1340987 - Posted: 26 Feb 2013, 10:58:13 UTC - in response to Message 1340931.

It's now taking over 1 day for my machines to successfully download a WU! I was considering adding the SETI project to a new machine, but why bother?

Did your machine run of work while that Wu was downloading?

If it didn't, then it's not a problem,

If it did, then increase you cache size a little so you have a few more cached tasks in hand.

Claggy

Profile Paul D Harris
Volunteer tester
Send message
Joined: 1 Dec 99
Posts: 1123
Credit: 33,598,472
RAC: 0
United States
Message 1340988 - Posted: 26 Feb 2013, 11:02:15 UTC

Same problem here why bother? It has been over 10 years now a new profile or design is needed or formula or whatever.
____________

Profile Blurf
Volunteer tester
Send message
Joined: 2 Sep 06
Posts: 7576
Credit: 6,971,809
RAC: 3,184
United States
Message 1341075 - Posted: 26 Feb 2013, 21:16:16 UTC - in response to Message 1340988.

Same problem here why bother? It has been over 10 years now a new profile or design is needed or formula or whatever.


Neat idea Paul but that'll cost money....no money, no major changes.
____________


w1hueProject donor
Volunteer tester
Send message
Joined: 4 Aug 00
Posts: 48
Credit: 1,804,615
RAC: 1,009
United States
Message 1341155 - Posted: 27 Feb 2013, 4:27:18 UTC - in response to Message 1340937.

You ask why bother? I would do it because you want to help with the science or you would like to find out whether or not there are other people in the world apart from humans.

As for the science, the only real science, as far as I am concerned, is with the "AstroPulse" sub-project -- and for the most part, I only accept AP WUs.

As far as finding out if "intelligent beings are out there", I am already 99.999% sure that there are. But as far as ever receiving any signal from "them", I am just about as sure that we will NOT. Why? Mainly because the probability that other "people" with technology similar to ours exist within a reasonable radius (say within 100 light-years or so) at this particular sliver of time in the overall age of the universe is mighty, mighty slim.

That is why I dedicate most of my computer time to projects that have more of a chance of providing some "real" science.

So have at it, all you geeks out there expecting to receive a SETI signal within your life times! :-)
____________

robertmiles
Send message
Joined: 16 Jan 12
Posts: 29
Credit: 721,244
RAC: 629
United States
Message 1341179 - Posted: 27 Feb 2013, 6:24:20 UTC

I've found that Astropulse workunits take days to download the input files to my computers, then only a few hours to run.

Therefore, this seems like time to ask: If your download server had enough GPUs, would it put less load on the server to just run all the Astropulse workunits than to send them anywhere?

robertmiles
Send message
Joined: 16 Jan 12
Posts: 29
Credit: 721,244
RAC: 629
United States
Message 1341185 - Posted: 27 Feb 2013, 6:35:19 UTC

Another idea: Is the largest input file of Astropulse workunits compressed in any way? If not, would compressing reduce its size enough to be worthwhile, after you add a decompression program to the list of files needed for each workunit?

Ianab
Volunteer tester
Send message
Joined: 11 Jun 08
Posts: 673
Credit: 12,626,479
RAC: 6,014
New Zealand
Message 1341220 - Posted: 27 Feb 2013, 9:48:29 UTC - in response to Message 1341185.

Issue with the project using GPUs to process the work units is they would need thousands of them. A whole room full, needing maybe a megawatt of power, and the same to keep them cool. The project would need to build an actual supercomputer.

The project might take all day to get the work to you, but it's still handling 30-40,000 AP units every day. Imagine the rack of computers that would be needed to do this all stacked up in one place.

As for the compression, my understanding is the data is more like "white noise". A normal compression (zip) doesn't have much effect on it. A "lossy" compression might loose the faint data we are actually looking for, or introduce false data.

So, good questions, but the answers are negative.

That leaves the current situation, where the project is in the fortunate situation of having more volunteers than it can effectively supply work too 100% of the time. Arguably a good situation for the project, but a bit frustrating for the volunteers. Hence the "Victim of it's own success" comment. But, not being able to supply all the work we want is not a failure for the project. It's not even in their "mission statement". If they merely supply all the work that their hardware and net link can handle, then they are doing well.

Leaves us with 2 things we can do.

1 - choose a backup project. Set it for a lower priority, then if you have trouble get SETI work, your machine fetches work from whatever other projects you have chosen.
2 - Raise some more funds so they can buy better servers and routers to allow for more work units to be processed.

Just remember the Project NEVER promised that there would be a steady or unlimited supply of work units. That's just what is happening now.

Frustrating to the guys that are in competition trying to achieve high RACs, but that's not actually a priority of the project.

Ian

Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8631
Credit: 51,504,211
RAC: 48,053
United Kingdom
Message 1341222 - Posted: 27 Feb 2013, 10:01:53 UTC - in response to Message 1341220.

Issue with the project using GPUs to process the work units is they would need thousands of them. A whole room full, needing maybe a megawatt of power, and the same to keep them cool. The project would need to build an actual supercomputer.

The project might take all day to get the work to you, but it's still handling 30-40,000 AP units every day. Imagine the rack of computers that would be needed to do this all stacked up in one place.

As for the compression, my understanding is the data is more like "white noise". A normal compression (zip) doesn't have much effect on it. A "lossy" compression might loose the faint data we are actually looking for, or introduce false data.

So, good questions, but the answers are negative.

That leaves the current situation, where the project is in the fortunate situation of having more volunteers than it can effectively supply work too 100% of the time. Arguably a good situation for the project, but a bit frustrating for the volunteers. Hence the "Victim of it's own success" comment. But, not being able to supply all the work we want is not a failure for the project. It's not even in their "mission statement". If they merely supply all the work that their hardware and net link can handle, then they are doing well.

Leaves us with 2 things we can do.

1 - choose a backup project. Set it for a lower priority, then if you have trouble get SETI work, your machine fetches work from whatever other projects you have chosen.
2 - Raise some more funds so they can buy better servers and routers to allow for more work units to be processed.

Just remember the Project NEVER promised that there would be a steady or unlimited supply of work units. That's just what is happening now.

Frustrating to the guys that are in competition trying to achieve high RACs, but that's not actually a priority of the project.

Ian

Well said - I think that sums it up nicely.

The one other thing we can usefully do is to keep our equipment (as far as possible) in good working order, and within limits of temperature etc. so that they return accurate results. That would ensure that the limited resources of bandwidth would be used as much as possible for science, rather than for correcting errors by crunching extra copies.

Wedge009
Volunteer tester
Avatar
Send message
Joined: 3 Apr 99
Posts: 339
Credit: 148,777,340
RAC: 192,369
Australia
Message 1341240 - Posted: 27 Feb 2013, 12:19:18 UTC - in response to Message 1341185.

Another idea: Is the largest input file of Astropulse workunits compressed in any way? If not, would compressing reduce its size enough to be worthwhile, after you add a decompression program to the list of files needed for each workunit?

I've been wanting to ask about this as well. I know it's been raised before and I recall back then that the answer was something along the lines of 'there would be too much stress on the servers to compress every work-unit'. Given that the servers are a lot more powerful now than they were back then, is this still an infeasible suggestion? I'll understand if it is not, but I am just wondering if it is still not even remotely possible. Given AP's 8 MiB WU size vs MB's ~300 KiB, it's clear to anyone that AP WUs consume a lot of what little available bandwidth there is.
____________
Soli Deo Gloria

Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8631
Credit: 51,504,211
RAC: 48,053
United Kingdom
Message 1341249 - Posted: 27 Feb 2013, 12:46:53 UTC - in response to Message 1341240.

Another idea: Is the largest input file of Astropulse workunits compressed in any way? If not, would compressing reduce its size enough to be worthwhile, after you add a decompression program to the list of files needed for each workunit?

I've been wanting to ask about this as well. I know it's been raised before and I recall back then that the answer was something along the lines of 'there would be too much stress on the servers to compress every work-unit'. Given that the servers are a lot more powerful now than they were back then, is this still an infeasible suggestion? I'll understand if it is not, but I am just wondering if it is still not even remotely possible. Given AP's 8 MiB WU size vs MB's ~300 KiB, it's clear to anyone that AP WUs consume a lot of what little available bandwidth there is.

The usual answer I give - and it's been raised multiple times - is "try one and see".

Even using LZMA compression, I don't think anyone has got above ~3% compression - except if you happen to try a B3_P1 task with the stuck bit. The ~50% compression you get then is a sure indicator that the job will overflow with 100% blanking in a very few seconds.

In other words, it isn't just the server load, but a cost-benefit balance between server load and minimal gain.

Wedge009
Volunteer tester
Avatar
Send message
Joined: 3 Apr 99
Posts: 339
Credit: 148,777,340
RAC: 192,369
Australia
Message 1341253 - Posted: 27 Feb 2013, 12:55:21 UTC

No worries. I suppose I should have done that first - for some reason, I had the impression that the 8 MiB file would compress well. But maybe because I'm thinking of the XML part of the WU, which is text-based, whereas the bulk of the WU is binary data, right?
____________
Soli Deo Gloria

juan BFBProject donor
Volunteer tester
Avatar
Send message
Joined: 16 Mar 07
Posts: 5395
Credit: 305,821,125
RAC: 322,493
Brazil
Message 1341271 - Posted: 27 Feb 2013, 14:09:29 UTC - in response to Message 1341253.
Last modified: 27 Feb 2013, 14:10:16 UTC

No worries. I suppose I should have done that first - for some reason, I had the impression that the 8 MiB file would compress well. But maybe because I'm thinking of the XML part of the WU, which is text-based, whereas the bulk of the WU is binary data, right?

Not be so sure, the compression of a highly random AP 8MB file will not produce a real gain in their size, and need a aditional work from the allready busy SETI servers to do that, remember what happens when AP splitters start.

Why they realy need is a way to use the 1 Gbps link (today they paid for 1 Gbps but could use only 100Mbps - something that makes no sense) or Split the Project in 2 diferent 100 Mbps (or more) links, one for AP and one for MB.
____________

N9JFE David SProject donor
Volunteer tester
Avatar
Send message
Joined: 4 Oct 99
Posts: 11960
Credit: 14,637,464
RAC: 12,524
United States
Message 1341275 - Posted: 27 Feb 2013, 14:24:22 UTC - in response to Message 1340937.

... the system was recovering from lab wide car appears over the weekend ...

Does this mean something in NZ-ese that it doesn't in American? :-)

____________
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.


Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8631
Credit: 51,504,211
RAC: 48,053
United Kingdom
Message 1341280 - Posted: 27 Feb 2013, 15:09:39 UTC - in response to Message 1341253.
Last modified: 27 Feb 2013, 15:10:11 UTC

No worries. I suppose I should have done that first - for some reason, I had the impression that the 8 MiB file would compress well. But maybe because I'm thinking of the XML part of the WU, which is text-based, whereas the bulk of the WU is binary data, right?

Yes, that's right. And in the case of AP, the compressible XML part is a very, very small fraction of the file.

Actually, that reminded me of a discussion we had a while back about MB ('setiathome_enhanced') data files. As befits a long-term longitudinal study like ours, the basic format specification for a MB WU hasn't changed since the original Classic project was launched. And, I was surprised to remind myself, the project is already two-thirds of the age of the internet itself (or, to be strict, of the World Wide Web, if we date that - as most people do - from the release of the Mosaic browser in 1993).

Back in those early days, not every ISP, or even every long-distance or intercontinental link, was fully transparent to 8-bit binary data. In order to allow as many people as possible to join in, even over those difficult links, the data format SETI chose was a 6-bit binary coding, so that every character can be represented by a printable byte in the 128-character ASCII set (the first 32 non-printable control characters being too risky to use on the comms lines of the era).

Fourteen years later, I think we can trust 8-bit data ;). In theory, the MB application is fully compatible with the 8-bit binary data format used by AP files: some months ago, Eric put a test of this theory on his ever-lengthening Beta 'to-do' list. If that works out, a simple change to the splitter configuration will allow the generation of smaller (denser) data files, without any extra compression/decompression stages - that way, we get almost the same benefit as compression, practically for free. But we do have to find time to test it first...

Profile Mr. KevvyProject donor
Volunteer tester
Avatar
Send message
Joined: 15 May 99
Posts: 726
Credit: 76,389,161
RAC: 82,838
Canada
Message 1341322 - Posted: 27 Feb 2013, 18:36:02 UTC - in response to Message 1341280.

Fourteen years later, I think we can trust 8-bit data ;). In theory, the MB application is fully compatible with the 8-bit binary data format used by AP files: some months ago, Eric put a test of this theory on his ever-lengthening Beta 'to-do' list. If that works out, a simple change to the splitter configuration will allow the generation of smaller (denser) data files, without any extra compression/decompression stages - that way, we get almost the same benefit as compression, practically for free. But we do have to find time to test it first...


Nice... had no idea they were using 6-bit characters. If there is a time to make the change, this would be it: the AP units are apparently having a "shorty storm" that's plugging up the bandwidth and aborting downloads for everyone, and it doesn't seem to be going away.
____________
“Never doubt that a small group of thoughtful, committed citizens can change the world; indeed, it's the only thing that ever has.”
--- Margaret Mead

Profile Paul D Harris
Volunteer tester
Send message
Joined: 1 Dec 99
Posts: 1123
Credit: 33,598,472
RAC: 0
United States
Message 1341531 - Posted: 28 Feb 2013, 6:05:01 UTC - in response to Message 1341155.

You ask why bother? I would do it because you want to help with the science or you would like to find out whether or not there are other people in the world apart from humans.

As for the science, the only real science, as far as I am concerned, is with the "AstroPulse" sub-project -- and for the most part, I only accept AP WUs.

As far as finding out if "intelligent beings are out there", I am already 99.999% sure that there are. But as far as ever receiving any signal from "them", I am just about as sure that we will NOT. Why? Mainly because the probability that other "people" with technology similar to ours exist within a reasonable radius (say within 100 light-years or so) at this particular sliver of time in the overall age of the universe is mighty, mighty slim.

That is why I dedicate most of my computer time to projects that have more of a chance of providing some "real" science.

So have at it, all you geeks out there expecting to receive a SETI signal within your life times! :-)


If they do exist they may not use radio waves or micro waves technology. They may use something entirely different such as light waves or mental telepathy or some other organic technology that SETI cannot detect.
____________

Profile Mark WyzenbeekProject donor
Avatar
Send message
Joined: 28 Jun 99
Posts: 91
Credit: 1,667,258
RAC: 1,252
United States
Message 1341567 - Posted: 28 Feb 2013, 7:45:28 UTC - in response to Message 1340931.

It's now taking over 1 day for my machines to successfully download a WU! I was considering adding the SETI project to a new machine, but why bother?

Because Seti is at the bleeding edge of distributed computing.
____________
The Universe is not only stranger than you imagine, it's stranger than you can imagine.

SETI@home classic workunits 1,405 CPU time 57,318 hours

robertmiles
Send message
Joined: 16 Jan 12
Posts: 29
Credit: 721,244
RAC: 629
United States
Message 1341572 - Posted: 28 Feb 2013, 7:48:35 UTC

Another idea for AstroPulse workunits: I assume that the load on the server varies with time of day or night. Are you able to set the server so that it will try to schedule the retries for incomplete downloads of the larger files to be preferably during the less busy times?

1 · 2 · 3 · Next

Message boards : Number crunching : SETI@HOME -- A Victim of it's Own Success?!?

Copyright © 2014 University of California