Moving on... (Apr 08 2013)


log in

Advanced search

Message boards : Technical News : Moving on... (Apr 08 2013)

Previous · 1 · 2 · 3 · Next
Author Message
Profile Gary Charpentier
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 12043
Credit: 6,371,515
RAC: 8,627
United States
Message 1355389 - Posted: 10 Apr 2013, 17:34:32 UTC - in response to Message 1355323.

A "NTPCKR" dedicated server, is the way to go.

Theres is no point in having all the result seating unchecked on the database

They aren't
http://setiathome.berkeley.edu/ntpckr.php
It just isn't ready for prime time ...

____________

Sten-Arne
Volunteer tester
Send message
Joined: 1 Nov 08
Posts: 3327
Credit: 18,961,665
RAC: 25,549
Sweden
Message 1355404 - Posted: 10 Apr 2013, 18:23:02 UTC - in response to Message 1355389.
Last modified: 10 Apr 2013, 18:28:26 UTC

A "NTPCKR" dedicated server, is the way to go.

Theres is no point in having all the result seating unchecked on the database

They aren't
http://setiathome.berkeley.edu/ntpckr.php
It just isn't ready for prime time ...


NTPCKR have been sitting "disabled" on the SSP for ages. Check the thread http://setiathome.berkeley.edu/forum_forum.php?id=830 I'd say it isn't running at all, and haven't done anything useful for more than a year, maybe even two years. There are millions upon millions of results just sitting there, waiting to be checked by the NTPCKR. For all I know, we may have hundreds of signs of ET in those results, but as long as the NTPCKR isn't doing any analyzing, we will never know. We're crunching and crunching millions upon millions of results, just to be added to the extremely large cache of not yet analyzed results.

Many people participating here, believe that there is only one step to finding ET with SETI@Home, and that is crunching the tasks sent out, and send back the results. That's only part one. NTPCKR, is where the results are analyzed, and I'm sure there are more parts than that before anyone would say that ET's been found. All we have been doing here this far, for more than a decade, is part one of the project. Part two haven't even really started yet.

As I said, we may already have found ET, but unless the tasks are analyzed, we will never know...


ntpckr_small_sig1 georgem Disabled
ntpckr_small_sig2 georgem Disabled
ntpckr_small_sig3 georgem Disabled
ntpckr_small_sig4 georgem Disabled

rfi_small_sig1 georgem Disabled
rfi_small_sig2 georgem Disabled
rfi_small_sig3 georgem Disabled
rfi_small_sig4 georgem Disabled

____________

Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 26 May 99
Posts: 6782
Credit: 24,438,086
RAC: 26,983
United Kingdom
Message 1355413 - Posted: 10 Apr 2013, 18:48:11 UTC - in response to Message 1355404.

A "NTPCKR" dedicated server, is the way to go.

Theres is no point in having all the result seating unchecked on the database

They aren't
http://setiathome.berkeley.edu/ntpckr.php
It just isn't ready for prime time ...


NTPCKR have been sitting "disabled" on the SSP for ages. Check the thread http://setiathome.berkeley.edu/forum_forum.php?id=830 I'd say it isn't running at all, and haven't done anything useful for more than a year, maybe even two years. There are millions upon millions of results just sitting there, waiting to be checked by the NTPCKR. For all I know, we may have hundreds of signs of ET in those results, but as long as the NTPCKR isn't doing any analyzing, we will never know. We're crunching and crunching millions upon millions of results, just to be added to the extremely large cache of not yet analyzed results.

Many people participating here, believe that there is only one step to finding ET with SETI@Home, and that is crunching the tasks sent out, and send back the results. That's only part one. NTPCKR, is where the results are analyzed, and I'm sure there are more parts than that before anyone would say that ET's been found. All we have been doing here this far, for more than a decade, is part one of the project. Part two haven't even really started yet.

As I said, we may already have found ET, but unless the tasks are analyzed, we will never know...


ntpckr_small_sig1 georgem Disabled
ntpckr_small_sig2 georgem Disabled
ntpckr_small_sig3 georgem Disabled
ntpckr_small_sig4 georgem Disabled

rfi_small_sig1 georgem Disabled
rfi_small_sig2 georgem Disabled
rfi_small_sig3 georgem Disabled
rfi_small_sig4 georgem Disabled

As discussed a while ago there is an "offline" nitpicker that it being used, just not talked about for some reason.
____________


Today is life, the only life we're sure of. Make the most of today.

Profile HAL9000
Volunteer tester
Avatar
Send message
Joined: 11 Sep 99
Posts: 3841
Credit: 106,371,043
RAC: 89,850
United States
Message 1355414 - Posted: 10 Apr 2013, 18:51:56 UTC - in response to Message 1355085.
Last modified: 10 Apr 2013, 18:59:55 UTC

Are you saying that you are not able to acquire enough data from Arecibo to continue to distribute work at the improved rates?
And if so, would more drives for data shuttle service help?
Or is it a limit on the rate that you are able to record data or the time allowed to do so?


Right... our observation time is the bottleneck in this case. When we are able to use the telescope we can record at the rates we desire, and we are easily able to shuttle all the data back to UCB on the drives we have.

- Matt

One you guys get data from Arecibo. Do you have to go to the colo to plug in drives or do you just copy the raw data the server(s) over the remote admin interface?
____________
SETI@home classic workunits: 93,865 CPU time: 863,447 hours

Join the BP6/VP6 User Group today!

Sten-Arne
Volunteer tester
Send message
Joined: 1 Nov 08
Posts: 3327
Credit: 18,961,665
RAC: 25,549
Sweden
Message 1355416 - Posted: 10 Apr 2013, 19:05:24 UTC - in response to Message 1355413.


As discussed a while ago there is an "offline" nitpicker that it being used, just not talked about for some reason.


Oh so it's one "NTPCKR" for Dummies (we normal simple crunchers), and it never runs, and one "NTPCKR" for Scientists and others deemed worthy to know about what it finds, and it runs in secret.

The class society have reached SETI :-)

Good to know.....
____________

Profile Gary Charpentier
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 12043
Credit: 6,371,515
RAC: 8,627
United States
Message 1355443 - Posted: 10 Apr 2013, 21:04:42 UTC - in response to Message 1355416.


As discussed a while ago there is an "offline" nitpicker that it being used, just not talked about for some reason.


Oh so it's one "NTPCKR" for Dummies (we normal simple crunchers), and it never runs, and one "NTPCKR" for Scientists and others deemed worthy to know about what it finds, and it runs in secret.

The class society have reached SETI :-)

Good to know.....

The public ntpckr IIRC was to be used with http://boinc.berkeley.edu/trac/wiki/BossaIntro. The public one, if it runs, will generate waterfall plots. Humans eyeballs need to look at the waterfall plots to figure out if it is BS or ET. Seti doesn't have the $$$ to do the training to teach you how the tell the difference yet or to implement it real time for 10,000 eyeballs to look at each one in parallel. However that doesn't mean that the data isn't being looked at from time to time.

____________

Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 26 May 99
Posts: 6782
Credit: 24,438,086
RAC: 26,983
United Kingdom
Message 1355467 - Posted: 10 Apr 2013, 22:09:11 UTC - in response to Message 1355443.


As discussed a while ago there is an "offline" nitpicker that it being used, just not talked about for some reason.


Oh so it's one "NTPCKR" for Dummies (we normal simple crunchers), and it never runs, and one "NTPCKR" for Scientists and others deemed worthy to know about what it finds, and it runs in secret.

The class society have reached SETI :-)

Good to know.....

The public ntpckr IIRC was to be used with http://boinc.berkeley.edu/trac/wiki/BossaIntro. The public one, if it runs, will generate waterfall plots. Humans eyeballs need to look at the waterfall plots to figure out if it is BS or ET. Seti doesn't have the $$$ to do the training to teach you how the tell the difference yet or to implement it real time for 10,000 eyeballs to look at each one in parallel. However that doesn't mean that the data isn't being looked at from time to time.

I see, can you give us mere mortals a link to the thread that contains this information?
____________


Today is life, the only life we're sure of. Make the most of today.

Josef W. Segur
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4196
Credit: 1,028,796
RAC: 262
United States
Message 1355480 - Posted: 10 Apr 2013, 23:18:13 UTC - in response to Message 1355299.

Hey, weren't we supposed to start seeing some data from somewhere else, Green Bank I think? Whatever happened with that?

See Andrew Siemion's posts in the New paper on SETI observations of the Kepler field thread. They still intend that first 40 Terabytes of Green Bank data to be processed here, but haven't had time to get splitters coded and tested.

With the Arecibo data more or less fully occupying the old download capacity, there also wasn't much motivation to add that other data source. But one of the GPUUG donation drive targets was getting enough drives and shipping cases to bring GBT data to Berkeley, which I took as an indication they intend to record more data there.
Joe

Profile BassieXp
Volunteer tester
Send message
Joined: 5 Jun 05
Posts: 8
Credit: 269,338
RAC: 0
Netherlands
Message 1355483 - Posted: 10 Apr 2013, 23:26:24 UTC - in response to Message 1355413.


As discussed a while ago there is an "offline" nitpicker that it being used, just not talked about for some reason.


I've recently watched the 10yr anniversary video's again and from what i could tell.
The way they do it now is that there's a machine, server, desktop or something which run's intermittent , where the nitpicker is a piece of software which should be able to run the results as they are comming in.

For those who are interessted in the video i believe is was in Eric Korpela's presentation. (just looked it up it's around 53m:45s).
Just on the subject the presentation from Jeff Cobb and Matt Lebofsky is also verry interesting as it's about the backend in 2009.
http://setiathome.berkeley.edu/sah_10yr_anniversary.php
____________

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38153
Credit: 555,809,175
RAC: 598,436
United States
Message 1355572 - Posted: 11 Apr 2013, 7:22:12 UTC - in response to Message 1355414.

Are you saying that you are not able to acquire enough data from Arecibo to continue to distribute work at the improved rates?
And if so, would more drives for data shuttle service help?
Or is it a limit on the rate that you are able to record data or the time allowed to do so?


Right... our observation time is the bottleneck in this case. When we are able to use the telescope we can record at the rates we desire, and we are easily able to shuttle all the data back to UCB on the drives we have.

- Matt

One you guys get data from Arecibo. Do you have to go to the colo to plug in drives or do you just copy the raw data the server(s) over the remote admin interface?

Given the amount of inbound traffic shown on the Cricket graphs from time to time, some of us are assuming it is due to data being transferred to the colo servers from the lab.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Cosmic_Ocean
Avatar
Send message
Joined: 23 Dec 00
Posts: 2233
Credit: 8,424,413
RAC: 4,073
United States
Message 1355581 - Posted: 11 Apr 2013, 8:17:34 UTC - in response to Message 1355483.

I've recently watched the 10yr anniversary video's again and from what i could tell.
The way they do it now is that there's a machine, server, desktop or something which run's intermittent , where the nitpicker is a piece of software which should be able to run the results as they are comming in.

Pretty much. But first it has to get through 13 years of back-log before it can analyze in real-time as more results come in.

In the past, the reason it hasn't run in a while is because either the dedicated machine for it got re-purposed to replace a failing/dead machine or two or the database server didn't have enough disk I/O or spare memory capacity to handle the extra load.. or both simultaneously.

Once the database gets tweaked some more to be able to handle some more additional heavy load, and there is a dedicated machine to run it, I'm sure it will happen.

Though just as a thought, could get a dedicated machine for it, load it with CPUs and RAM, and make an offline database from one of the weekly back-ups and let it chew on that locally for a couple months, and then when it gets caught up, let it catch up on the main database for a week or so, and then once it is current, it can do real-time with ease.
____________

Linux laptop uptime: 1484d 22h 42m
Ended due to UPS failure, found 14 hours after the fact

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38153
Credit: 555,809,175
RAC: 598,436
United States
Message 1355582 - Posted: 11 Apr 2013, 8:25:51 UTC - in response to Message 1355581.
Last modified: 11 Apr 2013, 8:30:22 UTC

I've recently watched the 10yr anniversary video's again and from what i could tell.
The way they do it now is that there's a machine, server, desktop or something which run's intermittent , where the nitpicker is a piece of software which should be able to run the results as they are comming in.

Pretty much. But first it has to get through 13 years of back-log before it can analyze in real-time as more results come in.

In the past, the reason it hasn't run in a while is because either the dedicated machine for it got re-purposed to replace a failing/dead machine or two or the database server didn't have enough disk I/O or spare memory capacity to handle the extra load.. or both simultaneously.

Once the database gets tweaked some more to be able to handle some more additional heavy load, and there is a dedicated machine to run it, I'm sure it will happen.

Though just as a thought, could get a dedicated machine for it, load it with CPUs and RAM, and make an offline database from one of the weekly back-ups and let it chew on that locally for a couple months, and then when it gets caught up, let it catch up on the main database for a week or so, and then once it is current, it can do real-time with ease.

Another thought that I believe has been bandied about before, would be to farm out the ntpckr process to us crunchers as a separate set of tasks, such as MB and AP are.

Could have a choice of MB, AP, or NP......

I think the Seti processing collective could munch through the ntpckr backlog in rather short order.

This would, of course, require a lot of work to write the programming to split, distribute, and process the data on our remote hosts.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Highlander
Avatar
Send message
Joined: 5 Oct 99
Posts: 143
Credit: 31,058,002
RAC: 121
Germany
Message 1355590 - Posted: 11 Apr 2013, 9:18:43 UTC

It's a pity that ALMA records data at frequency range from 30 to 950 GHz. In theory this would be a fantastic data-source (data are public available). But vy nice for "day-dreaming".
____________

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38153
Credit: 555,809,175
RAC: 598,436
United States
Message 1355592 - Posted: 11 Apr 2013, 9:21:29 UTC - in response to Message 1355590.

It's a pity that ALMA records data at frequency range from 30 to 950 GHz. In theory this would be a fantastic data-source (data are public available). But vy nice for "day-dreaming".

Well, I have no doubt that Eric will make other arrangements should the data being recorded at Arecibo be insufficient to further his search, and ours.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

contalis
Send message
Joined: 5 Dec 01
Posts: 24
Credit: 4,807,912
RAC: 207
United States
Message 1357022 - Posted: 15 Apr 2013, 1:43:27 UTC - in response to Message 1355582.

Another thought that I believe has been bandied about before, would be to farm out the ntpckr process to us crunchers as a separate set of tasks, such as MB and AP are.


My understanding is that the ntpckr needs to be processed locally, though IDK if that's speculation by others or from official sources.

contalis
Send message
Joined: 5 Dec 01
Posts: 24
Credit: 4,807,912
RAC: 207
United States
Message 1357023 - Posted: 15 Apr 2013, 1:45:00 UTC - in response to Message 1355443.

The public one, if it runs, will generate waterfall plots. Humans eyeballs need to look at the waterfall plots to figure out if it is BS or ET. Seti doesn't have the $$$ to do the training to teach you how the tell the difference yet or to implement it real time for 10,000 eyeballs to look at each one in parallel


isn't this what Seti Live is doing: http://setilive.org/
____________

Profile BMaytum
Volunteer tester
Avatar
Send message
Joined: 3 Apr 99
Posts: 98
Credit: 3,425,015
RAC: 2,774
United States
Message 1357629 - Posted: 16 Apr 2013, 22:28:09 UTC - in response to Message 1357023.
Last modified: 16 Apr 2013, 22:31:26 UTC

The public one, if it runs, will generate waterfall plots. Humans eyeballs need to look at the waterfall plots to figure out if it is BS or ET. ...


isn't this what Seti Live is doing: http://setilive.org/


I'd guess that's what Seti Live is/was attempting to do. Admittedly I didn't spend much time there or at the(ir?) related http://setiquest.org/join-the-quest/software/readme website. Initially I thought I might grab SetiQuest's open source SonATA "waterfall image" software to try on one of my (non-Ubuntu) linux PCs. But I chose not to after inferring from some blogs there that the initial multi-hundreds or thousands of eyeballs looking at waterfalls starting in 2011(?) had declined exponentially since then. I also wondered how they had blown through the $100,000 TED funding already....

Anyway, right after the recent successful S@H colocation, I too thought: "Great, now the SSL scientists can get on with NTPCKR so we longtime crunchers can see some candidates". But in reading the most recent posts in this thread, I'm beginning to wonder what's really going to happen to digesting the millions of S@H results, and when?
____________
Sabertooth Z77, i7-3770K@4.2GHz, GTX680, W8.1Pro x64
P5N32-E SLI, C2D E8400@3Ghz, GTX580, Win7SP1Pro x64 & PCLinuxOS2014 x86

wulf 21
Send message
Joined: 18 Apr 09
Posts: 75
Credit: 12,838,367
RAC: 3,823
Germany
Message 1357711 - Posted: 17 Apr 2013, 7:29:19 UTC - in response to Message 1357022.

Another thought that I believe has been bandied about before, would be to farm out the ntpckr process to us crunchers as a separate set of tasks, such as MB and AP are.


My understanding is that the ntpckr needs to be processed locally, though IDK if that's speculation by others or from official sources.


Exactly my thought.

The reason why the raw data crunching is so perfectly fitted for distributed computing is that a lot of processing time is required for a fairly low amount of data. Your computer is running thousands of tests on the same 0.36 KB of data. (a lot of different frequencies/resolutions/chirp rates). Then just a handful of hits are sent back.

Ntpckr on the other hand needs to do a fairly low amount of processing on a high amount of data. For a single result all that needs to be done is deciding which pixel(s) it applies to (a decision which a local server can do faster than sending it out to a user).

Then, to start scoring, you need ALL the results that have ever been watched at this pixel. And all that needs to be done is calculating a score from the values that are in the results table, which again can be done really fast.

And then again for the RFI checking, you need ALL the results that have been watched about the time, the result you want to check have been watched. And the simple test that has to be done is if there were a lot of results on the same frequency before and after the result (i.e. in different spots of the sky) or a lot of results at the same time but at other frequencys (i.e. the signal is broadband). Both is considered RFI. So, just 2 fast tests on a lot of data.

Summing up, if you would do persistency checking or RFI checking as a distributed computing task, preparing the WUs for the users, scheduling them and sending them over the internet would take more time then just doing the checks locally. It's just not fitted tor distributed computing.

Mary
Send message
Joined: 22 Mar 12
Posts: 1
Credit: 241,650
RAC: 0
Message 1380898 - Posted: 14 Jun 2013, 1:39:51 UTC

Why did my screen turn red with green flags all over it about a week or so ago?

N9JFE David S
Volunteer tester
Avatar
Send message
Joined: 4 Oct 99
Posts: 10698
Credit: 13,380,244
RAC: 14,893
United States
Message 1381124 - Posted: 14 Jun 2013, 15:48:35 UTC

Hey, Matt!

We haven't heard from you in a couple months now. Is everything going so well you don't need to tell us anything?

How about an update on some other stuff? You guys must be working on something now that the servers are more or less behaving themselves. Maybe you can tell us what's up with the nitpckrs and the other things at the bottom of the server page.

____________
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.


Previous · 1 · 2 · 3 · Next

Message boards : Technical News : Moving on... (Apr 08 2013)

Copyright © 2014 University of California