Kerned density estimation instead of histograms


log in

Advanced search

Message boards : SETI@home Science : Kerned density estimation instead of histograms

Author Message
Profile Fox
Send message
Joined: 31 Jul 99
Posts: 4
Credit: 1,892,177
RAC: 0
Italy
Message 974525 - Posted: 27 Feb 2010, 18:27:48 UTC

Hello

I have a question (or a suggestion) for you. In the window of seti software it is possible to see an histogram. This is for the search of a particular signal with gaussian shape (I suppose...). Why you don't use the Kernel Density Estimation (KDE) to do that? The KDE is a powerful statitical method similar to the histogram but:
- it doesn't need the bins and the bin size
- it doesn't have the bin position problem (I mean where you have to start your binning)
- it converge fastly than a histogram to the real probability density function



Tnx for the attention and sorry for the poor english.
Bye



____________

PhonAcq
Send message
Joined: 14 Apr 01
Posts: 1622
Credit: 22,356,438
RAC: 6,174
United States
Message 974772 - Posted: 28 Feb 2010, 14:33:35 UTC

I can't reply for the project, but your suggestion is like many good ones I've seen on these message boards. It is very likely there aren't enough staff to consider each such suggestion, in light of the other priorities of the project.

I wonder, then, wouldn't it be good to have a snapshot of the science database available to us for download from time to time. Then people with sufficient interest could process it any way we like. If something really keen pops up, the results could be given back to the project.

Making the snapshots available wouldn't take too much effort, because they are essentially back-up copies I suppose. There would be a bandwidth demand, but very few people might be interested in getting a copy. Even so, downloads could possibly be limited to be during the weekly maintenance period.

It might make sense to try something like this for a while to see if something results! A sort of distributed computing at a higher level.

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 8489
Credit: 4,188,747
RAC: 1,720
United Kingdom
Message 976847 - Posted: 9 Mar 2010, 0:04:20 UTC - in response to Message 974525.

... seti software it is possible to see an histogram. This is for the search of a particular signal with gaussian shape (I suppose...). Why you don't use the Kernel Density Estimation (KDE) to do that? The KDE is a powerful statitical method similar to the histogram but:
- it doesn't need the bins and the bin size
- it doesn't have the bin position problem (I mean where you have to start your binning)
- it converge fastly than a histogram to the real probability density function

How is that more useful for the signal analysis than the present code?

(The present bins are assumed to be narrow enough to find any wanted gaussians.)

Does the KDE discriminate better than fixed histogram bins to more accurately find gaussians that are noisy and still reject random noise that might look like a noisy gaussian?

Is the KDE easy to compute?


Thanks, could be interesting?...

Regards,
Martin

____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Profile Fox
Send message
Joined: 31 Jul 99
Posts: 4
Credit: 1,892,177
RAC: 0
Italy
Message 982187 - Posted: 21 Mar 2010, 18:03:31 UTC - in response to Message 976847.


How is that more useful for the signal analysis than the present code?
(The present bins are assumed to be narrow enough to find any wanted gaussians.)
Does the KDE discriminate better than fixed histogram bins to more accurately find gaussians that are noisy and still reject random noise that might look like a noisy gaussian?
Is the KDE easy to compute?
Thanks, could be interesting?...
Regards,
Martin



Unfortunately I cannot reply to all of your question here, also because it is necessary a lot of math (and some steps are still unclear also for me). What can I do is start to partially reply to your first question, for that I suggest to see the example that is show here: http://school.maths.uwa.edu.au/~duongt/seminars/intro2kde/


and still reject random noise that might look like a noisy gaussian?

Does an histogram do that?

In general the calculation of the KDE is more intensive but more accurate and approximate better the true probability density function (look here for more example and discussion: www-hermes.desy.de/notes/pub/TALK/sgliske.tpsh09.pdf ).


An interest example wrote in python could be found here: http://jpktd.blogspot.com/2009/03/using-gaussian-kernel-density.html
it is really similar to the code that i use to show and fit my data.





____________

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 8489
Credit: 4,188,747
RAC: 1,720
United Kingdom
Message 982947 - Posted: 24 Mar 2010, 3:05:26 UTC - in response to Message 982187.
Last modified: 24 Mar 2010, 3:06:22 UTC


and still reject random noise that might look like a noisy gaussian?

Does an histogram do that?

In general the calculation of the KDE is more intensive but more accurate and approximate better the true probability density function...

The issue is whether a more accurate probability density function gives any advantage for the s@h analysis and search over what is already being done.

I suspect that whatever additional accuracy might be gained using KDE, that better accuracy is then wasted when the thresholding is automatically adjusted to allow for the noise floor for each WU.

Sorry, but there will have to be some clear and big advantage over the present system for there to be any interest in making a new type of analysis and search.


Regardless, it's always good to look at new ideas!

Keep searchin',
Martin
____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Message boards : SETI@home Science : Kerned density estimation instead of histograms

Copyright © 2014 University of California