Moving on... (Apr 08 2013)

Message boards : Technical News : Moving on... (Apr 08 2013)
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · Next

AuthorMessage
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1355413 - Posted: 10 Apr 2013, 18:48:11 UTC - in response to Message 1355404.  

A "NTPCKR" dedicated server, is the way to go.

Theres is no point in having all the result seating unchecked on the database

They aren't
http://setiathome.berkeley.edu/ntpckr.php
It just isn't ready for prime time ...


NTPCKR have been sitting "disabled" on the SSP for ages. Check the thread http://setiathome.berkeley.edu/forum_forum.php?id=830 I'd say it isn't running at all, and haven't done anything useful for more than a year, maybe even two years. There are millions upon millions of results just sitting there, waiting to be checked by the NTPCKR. For all I know, we may have hundreds of signs of ET in those results, but as long as the NTPCKR isn't doing any analyzing, we will never know. We're crunching and crunching millions upon millions of results, just to be added to the extremely large cache of not yet analyzed results.

Many people participating here, believe that there is only one step to finding ET with SETI@Home, and that is crunching the tasks sent out, and send back the results. That's only part one. NTPCKR, is where the results are analyzed, and I'm sure there are more parts than that before anyone would say that ET's been found. All we have been doing here this far, for more than a decade, is part one of the project. Part two haven't even really started yet.

As I said, we may already have found ET, but unless the tasks are analyzed, we will never know...


ntpckr_small_sig1 georgem Disabled
ntpckr_small_sig2 georgem Disabled
ntpckr_small_sig3 georgem Disabled
ntpckr_small_sig4 georgem Disabled

rfi_small_sig1 georgem Disabled
rfi_small_sig2 georgem Disabled
rfi_small_sig3 georgem Disabled
rfi_small_sig4 georgem Disabled

As discussed a while ago there is an "offline" nitpicker that it being used, just not talked about for some reason.
ID: 1355413 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1355414 - Posted: 10 Apr 2013, 18:51:56 UTC - in response to Message 1355085.  
Last modified: 10 Apr 2013, 18:59:55 UTC

Are you saying that you are not able to acquire enough data from Arecibo to continue to distribute work at the improved rates?
And if so, would more drives for data shuttle service help?
Or is it a limit on the rate that you are able to record data or the time allowed to do so?


Right... our observation time is the bottleneck in this case. When we are able to use the telescope we can record at the rates we desire, and we are easily able to shuttle all the data back to UCB on the drives we have.

- Matt

One you guys get data from Arecibo. Do you have to go to the colo to plug in drives or do you just copy the raw data the server(s) over the remote admin interface?
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1355414 · Report as offensive
Profile Gary Charpentier Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 25 Dec 00
Posts: 30593
Credit: 53,134,872
RAC: 32
United States
Message 1355443 - Posted: 10 Apr 2013, 21:04:42 UTC - in response to Message 1355416.  


As discussed a while ago there is an "offline" nitpicker that it being used, just not talked about for some reason.


Oh so it's one "NTPCKR" for Dummies (we normal simple crunchers), and it never runs, and one "NTPCKR" for Scientists and others deemed worthy to know about what it finds, and it runs in secret.

The class society have reached SETI :-)

Good to know.....

The public ntpckr IIRC was to be used with http://boinc.berkeley.edu/trac/wiki/BossaIntro. The public one, if it runs, will generate waterfall plots. Humans eyeballs need to look at the waterfall plots to figure out if it is BS or ET. Seti doesn't have the $$$ to do the training to teach you how the tell the difference yet or to implement it real time for 10,000 eyeballs to look at each one in parallel. However that doesn't mean that the data isn't being looked at from time to time.

ID: 1355443 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1355467 - Posted: 10 Apr 2013, 22:09:11 UTC - in response to Message 1355443.  


As discussed a while ago there is an "offline" nitpicker that it being used, just not talked about for some reason.


Oh so it's one "NTPCKR" for Dummies (we normal simple crunchers), and it never runs, and one "NTPCKR" for Scientists and others deemed worthy to know about what it finds, and it runs in secret.

The class society have reached SETI :-)

Good to know.....

The public ntpckr IIRC was to be used with http://boinc.berkeley.edu/trac/wiki/BossaIntro. The public one, if it runs, will generate waterfall plots. Humans eyeballs need to look at the waterfall plots to figure out if it is BS or ET. Seti doesn't have the $$$ to do the training to teach you how the tell the difference yet or to implement it real time for 10,000 eyeballs to look at each one in parallel. However that doesn't mean that the data isn't being looked at from time to time.

I see, can you give us mere mortals a link to the thread that contains this information?
ID: 1355467 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 1355480 - Posted: 10 Apr 2013, 23:18:13 UTC - in response to Message 1355299.  

Hey, weren't we supposed to start seeing some data from somewhere else, Green Bank I think? Whatever happened with that?

See Andrew Siemion's posts in the New paper on SETI observations of the Kepler field thread. They still intend that first 40 Terabytes of Green Bank data to be processed here, but haven't had time to get splitters coded and tested.

With the Arecibo data more or less fully occupying the old download capacity, there also wasn't much motivation to add that other data source. But one of the GPUUG donation drive targets was getting enough drives and shipping cases to bring GBT data to Berkeley, which I took as an indication they intend to record more data there.
                                                                   Joe
ID: 1355480 · Report as offensive
Profile BassieXp
Volunteer tester

Send message
Joined: 5 Jun 05
Posts: 14
Credit: 1,408,518
RAC: 0
Netherlands
Message 1355483 - Posted: 10 Apr 2013, 23:26:24 UTC - in response to Message 1355413.  


As discussed a while ago there is an "offline" nitpicker that it being used, just not talked about for some reason.


I've recently watched the 10yr anniversary video's again and from what i could tell.
The way they do it now is that there's a machine, server, desktop or something which run's intermittent , where the nitpicker is a piece of software which should be able to run the results as they are comming in.

For those who are interessted in the video i believe is was in Eric Korpela's presentation. (just looked it up it's around 53m:45s).
Just on the subject the presentation from Jeff Cobb and Matt Lebofsky is also verry interesting as it's about the backend in 2009.
http://setiathome.berkeley.edu/sah_10yr_anniversary.php
ID: 1355483 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1355572 - Posted: 11 Apr 2013, 7:22:12 UTC - in response to Message 1355414.  

Are you saying that you are not able to acquire enough data from Arecibo to continue to distribute work at the improved rates?
And if so, would more drives for data shuttle service help?
Or is it a limit on the rate that you are able to record data or the time allowed to do so?


Right... our observation time is the bottleneck in this case. When we are able to use the telescope we can record at the rates we desire, and we are easily able to shuttle all the data back to UCB on the drives we have.

- Matt

One you guys get data from Arecibo. Do you have to go to the colo to plug in drives or do you just copy the raw data the server(s) over the remote admin interface?

Given the amount of inbound traffic shown on the Cricket graphs from time to time, some of us are assuming it is due to data being transferred to the colo servers from the lab.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1355572 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1355581 - Posted: 11 Apr 2013, 8:17:34 UTC - in response to Message 1355483.  

I've recently watched the 10yr anniversary video's again and from what i could tell.
The way they do it now is that there's a machine, server, desktop or something which run's intermittent , where the nitpicker is a piece of software which should be able to run the results as they are comming in.

Pretty much. But first it has to get through 13 years of back-log before it can analyze in real-time as more results come in.

In the past, the reason it hasn't run in a while is because either the dedicated machine for it got re-purposed to replace a failing/dead machine or two or the database server didn't have enough disk I/O or spare memory capacity to handle the extra load.. or both simultaneously.

Once the database gets tweaked some more to be able to handle some more additional heavy load, and there is a dedicated machine to run it, I'm sure it will happen.

Though just as a thought, could get a dedicated machine for it, load it with CPUs and RAM, and make an offline database from one of the weekly back-ups and let it chew on that locally for a couple months, and then when it gets caught up, let it catch up on the main database for a week or so, and then once it is current, it can do real-time with ease.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1355581 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1355582 - Posted: 11 Apr 2013, 8:25:51 UTC - in response to Message 1355581.  
Last modified: 11 Apr 2013, 8:30:22 UTC

I've recently watched the 10yr anniversary video's again and from what i could tell.
The way they do it now is that there's a machine, server, desktop or something which run's intermittent , where the nitpicker is a piece of software which should be able to run the results as they are comming in.

Pretty much. But first it has to get through 13 years of back-log before it can analyze in real-time as more results come in.

In the past, the reason it hasn't run in a while is because either the dedicated machine for it got re-purposed to replace a failing/dead machine or two or the database server didn't have enough disk I/O or spare memory capacity to handle the extra load.. or both simultaneously.

Once the database gets tweaked some more to be able to handle some more additional heavy load, and there is a dedicated machine to run it, I'm sure it will happen.

Though just as a thought, could get a dedicated machine for it, load it with CPUs and RAM, and make an offline database from one of the weekly back-ups and let it chew on that locally for a couple months, and then when it gets caught up, let it catch up on the main database for a week or so, and then once it is current, it can do real-time with ease.

Another thought that I believe has been bandied about before, would be to farm out the ntpckr process to us crunchers as a separate set of tasks, such as MB and AP are.

Could have a choice of MB, AP, or NP......

I think the Seti processing collective could munch through the ntpckr backlog in rather short order.

This would, of course, require a lot of work to write the programming to split, distribute, and process the data on our remote hosts.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1355582 · Report as offensive
Highlander
Avatar

Send message
Joined: 5 Oct 99
Posts: 167
Credit: 37,987,668
RAC: 16
Germany
Message 1355590 - Posted: 11 Apr 2013, 9:18:43 UTC

It's a pity that ALMA records data at frequency range from 30 to 950 GHz. In theory this would be a fantastic data-source (data are public available). But vy nice for "day-dreaming".
- Performance is not a simple linear function of the number of CPUs you throw at the problem. -
ID: 1355590 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1355592 - Posted: 11 Apr 2013, 9:21:29 UTC - in response to Message 1355590.  

It's a pity that ALMA records data at frequency range from 30 to 950 GHz. In theory this would be a fantastic data-source (data are public available). But vy nice for "day-dreaming".

Well, I have no doubt that Eric will make other arrangements should the data being recorded at Arecibo be insufficient to further his search, and ours.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1355592 · Report as offensive
contalis

Send message
Joined: 5 Dec 01
Posts: 28
Credit: 27,237,533
RAC: 73
United States
Message 1357022 - Posted: 15 Apr 2013, 1:43:27 UTC - in response to Message 1355582.  

Another thought that I believe has been bandied about before, would be to farm out the ntpckr process to us crunchers as a separate set of tasks, such as MB and AP are.


My understanding is that the ntpckr needs to be processed locally, though IDK if that's speculation by others or from official sources.
ID: 1357022 · Report as offensive
contalis

Send message
Joined: 5 Dec 01
Posts: 28
Credit: 27,237,533
RAC: 73
United States
Message 1357023 - Posted: 15 Apr 2013, 1:45:00 UTC - in response to Message 1355443.  

The public one, if it runs, will generate waterfall plots. Humans eyeballs need to look at the waterfall plots to figure out if it is BS or ET. Seti doesn't have the $$$ to do the training to teach you how the tell the difference yet or to implement it real time for 10,000 eyeballs to look at each one in parallel


isn't this what Seti Live is doing: http://setilive.org/
ID: 1357023 · Report as offensive
Profile BMaytum
Volunteer tester
Avatar

Send message
Joined: 3 Apr 99
Posts: 104
Credit: 4,382,041
RAC: 2
United States
Message 1357629 - Posted: 16 Apr 2013, 22:28:09 UTC - in response to Message 1357023.  
Last modified: 16 Apr 2013, 22:31:26 UTC

The public one, if it runs, will generate waterfall plots. Humans eyeballs need to look at the waterfall plots to figure out if it is BS or ET. ...


isn't this what Seti Live is doing: http://setilive.org/


I'd guess that's what Seti Live is/was attempting to do. Admittedly I didn't spend much time there or at the(ir?) related http://setiquest.org/join-the-quest/software/readme website. Initially I thought I might grab SetiQuest's open source SonATA "waterfall image" software to try on one of my (non-Ubuntu) linux PCs. But I chose not to after inferring from some blogs there that the initial multi-hundreds or thousands of eyeballs looking at waterfalls starting in 2011(?) had declined exponentially since then. I also wondered how they had blown through the $100,000 TED funding already....

Anyway, right after the recent successful S@H colocation, I too thought: "Great, now the SSL scientists can get on with NTPCKR so we longtime crunchers can see some candidates". But in reading the most recent posts in this thread, I'm beginning to wonder what's really going to happen to digesting the millions of S@H results, and when?
Sabertooth Z77, i7-3770K@4.2GHz, GTX680, W8.1Pro x64
P5N32-E SLI, C2D E8400@3Ghz, GTX580, Win7SP1Pro x64 & PCLinuxOS2015 x64
ID: 1357629 · Report as offensive
wulf 21

Send message
Joined: 18 Apr 09
Posts: 93
Credit: 26,337,213
RAC: 43
Germany
Message 1357711 - Posted: 17 Apr 2013, 7:29:19 UTC - in response to Message 1357022.  

Another thought that I believe has been bandied about before, would be to farm out the ntpckr process to us crunchers as a separate set of tasks, such as MB and AP are.


My understanding is that the ntpckr needs to be processed locally, though IDK if that's speculation by others or from official sources.


Exactly my thought.

The reason why the raw data crunching is so perfectly fitted for distributed computing is that a lot of processing time is required for a fairly low amount of data. Your computer is running thousands of tests on the same 0.36 KB of data. (a lot of different frequencies/resolutions/chirp rates). Then just a handful of hits are sent back.

Ntpckr on the other hand needs to do a fairly low amount of processing on a high amount of data. For a single result all that needs to be done is deciding which pixel(s) it applies to (a decision which a local server can do faster than sending it out to a user).

Then, to start scoring, you need ALL the results that have ever been watched at this pixel. And all that needs to be done is calculating a score from the values that are in the results table, which again can be done really fast.

And then again for the RFI checking, you need ALL the results that have been watched about the time, the result you want to check have been watched. And the simple test that has to be done is if there were a lot of results on the same frequency before and after the result (i.e. in different spots of the sky) or a lot of results at the same time but at other frequencys (i.e. the signal is broadband). Both is considered RFI. So, just 2 fast tests on a lot of data.

Summing up, if you would do persistency checking or RFI checking as a distributed computing task, preparing the WUs for the users, scheduling them and sending them over the internet would take more time then just doing the checks locally. It's just not fitted tor distributed computing.
ID: 1357711 · Report as offensive
Mary

Send message
Joined: 22 Mar 12
Posts: 1
Credit: 363,143
RAC: 0
United States
Message 1380898 - Posted: 14 Jun 2013, 1:39:51 UTC

Why did my screen turn red with green flags all over it about a week or so ago?
ID: 1380898 · Report as offensive
David S
Volunteer tester
Avatar

Send message
Joined: 4 Oct 99
Posts: 18352
Credit: 27,761,924
RAC: 12
United States
Message 1381124 - Posted: 14 Jun 2013, 15:48:35 UTC

Hey, Matt!

We haven't heard from you in a couple months now. Is everything going so well you don't need to tell us anything?

How about an update on some other stuff? You guys must be working on something now that the servers are more or less behaving themselves. Maybe you can tell us what's up with the nitpckrs and the other things at the bottom of the server page.

David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.

ID: 1381124 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1381662 - Posted: 15 Jun 2013, 23:42:16 UTC

We haven't heard from you in a couple months now.

Yes, I wonder why??
ID: 1381662 · Report as offensive
Alinator
Volunteer tester

Send message
Joined: 19 Apr 05
Posts: 4178
Credit: 4,647,982
RAC: 0
United States
Message 1382251 - Posted: 17 Jun 2013, 19:47:48 UTC - in response to Message 1381662.  

We haven't heard from you in a couple months now.

Yes, I wonder why??


SAH firefighting, other SSL duties to perform, music, a life.....

Not necessarily in that order. :-D
ID: 1382251 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 1382293 - Posted: 17 Jun 2013, 22:13:16 UTC

I apologize for the lack of updates. I was out of the lab for a month, then out of town the past two weekends.. So I've fallen out of the habit. I'm around (mostly) this summer so I'll try to get a random update out at least this week and then more afterward... There's always *something* to report...

- Matt
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 1382293 · Report as offensive
Previous · 1 · 2 · 3 · Next

Message boards : Technical News : Moving on... (Apr 08 2013)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.