Kerned density estimation instead of histograms

Message boards : SETI@home Science : Kerned density estimation instead of histograms

Author	Message
Fox Send message Joined: 31 Jul 99 Posts: 4 Credit: 781,844 RAC: 1,969	Message 974525 - Posted: 27 Feb 2010, 18:27:48 UTC
	Hello I have a question (or a suggestion) for you. In the window of seti software it is possible to see an histogram. This is for the search of a particular signal with gaussian shape (I suppose...). Why you don't use the Kernel Density Estimation (KDE) to do that? The KDE is a powerful statitical method similar to the histogram but: - it doesn't need the bins and the bin size - it doesn't have the bin position problem (I mean where you have to start your binning) - it converge fastly than a histogram to the real probability density function Tnx for the attention and sorry for the poor english. Bye ____________
	ID: 974525 ·

PhonAcq Send message Joined: 14 Apr 01 Posts: 1620 Credit: 20,890,839 RAC: 4,792	Message 974772 - Posted: 28 Feb 2010, 14:33:35 UTC
	I can't reply for the project, but your suggestion is like many good ones I've seen on these message boards. It is very likely there aren't enough staff to consider each such suggestion, in light of the other priorities of the project. I wonder, then, wouldn't it be good to have a snapshot of the science database available to us for download from time to time. Then people with sufficient interest could process it any way we like. If something really keen pops up, the results could be given back to the project. Making the snapshots available wouldn't take too much effort, because they are essentially back-up copies I suppose. There would be a bandwidth demand, but very few people might be interested in getting a copy. Even so, downloads could possibly be limited to be during the weekly maintenance period. It might make sense to try something like this for a while to see if something results! A sort of distributed computing at a higher level.
	ID: 974772 ·

ML1 Volunteer tester Send message Joined: 25 Nov 01 Posts: 7141 Credit: 3,687,899 RAC: 958	Message 976847 - Posted: 9 Mar 2010, 0:04:20 UTC - in response to Message 974525.
	... seti software it is possible to see an histogram. This is for the search of a particular signal with gaussian shape (I suppose...). Why you don't use the Kernel Density Estimation (KDE) to do that? The KDE is a powerful statitical method similar to the histogram but: - it doesn't need the bins and the bin size - it doesn't have the bin position problem (I mean where you have to start your binning) - it converge fastly than a histogram to the real probability density function How is that more useful for the signal analysis than the present code? (The present bins are assumed to be narrow enough to find any wanted gaussians.) Does the KDE discriminate better than fixed histogram bins to more accurately find gaussians that are noisy and still reject random noise that might look like a noisy gaussian? Is the KDE easy to compute? Thanks, could be interesting?... Regards, Martin ____________ Mandriva Linux A user friendly OS! See new freedom Mageia2 The Future is what We make IT (GPLv3)
	ID: 976847 ·

Fox Send message Joined: 31 Jul 99 Posts: 4 Credit: 781,844 RAC: 1,969	Message 982187 - Posted: 21 Mar 2010, 18:03:31 UTC - in response to Message 976847.
	How is that more useful for the signal analysis than the present code? (The present bins are assumed to be narrow enough to find any wanted gaussians.) Does the KDE discriminate better than fixed histogram bins to more accurately find gaussians that are noisy and still reject random noise that might look like a noisy gaussian? Is the KDE easy to compute? Thanks, could be interesting?... Regards, Martin Unfortunately I cannot reply to all of your question here, also because it is necessary a lot of math (and some steps are still unclear also for me). What can I do is start to partially reply to your first question, for that I suggest to see the example that is show here: http://school.maths.uwa.edu.au/~duongt/seminars/intro2kde/ and still reject random noise that might look like a noisy gaussian? Does an histogram do that? In general the calculation of the KDE is more intensive but more accurate and approximate better the true probability density function (look here for more example and discussion: www-hermes.desy.de/notes/pub/TALK/sgliske.tpsh09.pdf ). An interest example wrote in python could be found here: http://jpktd.blogspot.com/2009/03/using-gaussian-kernel-density.html it is really similar to the code that i use to show and fit my data. ____________
	ID: 982187 ·

ML1 Volunteer tester Send message Joined: 25 Nov 01 Posts: 7141 Credit: 3,687,899 RAC: 958	Message 982947 - Posted: 24 Mar 2010, 3:05:26 UTC - in response to Message 982187. Last modified: 24 Mar 2010, 3:06:22 UTC
	and still reject random noise that might look like a noisy gaussian? Does an histogram do that? In general the calculation of the KDE is more intensive but more accurate and approximate better the true probability density function... The issue is whether a more accurate probability density function gives any advantage for the s@h analysis and search over what is already being done. I suspect that whatever additional accuracy might be gained using KDE, that better accuracy is then wasted when the thresholding is automatically adjusted to allow for the noise floor for each WU. Sorry, but there will have to be some clear and big advantage over the present system for there to be any interest in making a new type of analysis and search. Regardless, it's always good to look at new ideas! Keep searchin', Martin ____________ Mandriva Linux A user friendly OS! See new freedom Mageia2 The Future is what We make IT (GPLv3)
	ID: 982947 ·

Message boards : SETI@home Science : Kerned density estimation instead of histograms