Eric's Weekly Post #3 (Warning, science enclosed. Do not fold or bend.)


log in

Advanced search

Message boards : SETI@home Staff Blog : Eric's Weekly Post #3 (Warning, science enclosed. Do not fold or bend.)

1 · 2 · 3 · 4 . . . 6 · Next
Author Message
Eric Korpela
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 3 Apr 99
Posts: 1076
Credit: 7,771,718
RAC: 7,157
United States
Message 518296 - Posted: 16 Feb 2007, 19:26:20 UTC
Last modified: 16 Feb 2007, 19:27:46 UTC

OK, so I missed a week. Nobody's perfect.

This week, I've been busting my brain about some interference we are seeing in the multibeam data recorder. Anyone who runs the beta may have noticed that both Astropulse and SETI@home multibeam work units have been hitting overflow conditions fairly regularly. Regularly enough that a lot of machines were hitting their 100 workunit per day limit. So I
set about the task of figuring out what was going on. I'll ask you to forgive me in advance because this might get a bit technical...

I grabbed one of the recorder data files that showed this effect and did a little spectral analysis to see if I could figure something out. This is what I saw.



This is what we call a waterfall plot of about 0.42 seconds of data from one of the data recorder channels. The vertical axis is frequency difference in MHz from 1.420 GHz. The horizontal axis is time. Generally it looks OK. The darkish stripe at zero frequency is due to a high-pass filter that cuts out any constant voltage offset in the system. The gradual darkening at the top and bottom are the high and low frequency edges of our bandpass filter. You might notice a slightly brighter stripe at about 0.5 MHz that is due to the Hydrogen line that Kevin spends his time making maps of.

What I didn't expect to see is the 7 or so vertical lines that cross zero frequency. That's our problem. Pulsed interference of some sort. From the width I can see that the pulses have time scales of 20 to 50 microseconds or so. They aren't entirely regularly spaced, but they are close enough and strong enough to register as pulses in astropulse and as triplets in SETI@home. I looked at all of the receivers and I saw something similar in each.

The next thing to check was whether the interference appeared at the same time in all the bands. So I did a complex sum of all of the channels to see if the signals were coherent, meaning they have the same phase and timing in all the channels.

Spectral analysis of the result showed this:



That convinced me that there is some source of interference that affects all of the channels in nearly the same way. It's also more common than you would think if you just looked at one channel. So I did the obvious thing and subtracted the summed waveform from the channel to see if that would remove the pulses. It did. Perfectly.

That is, until I tried to turn the subtracted channel into a work unit. In order to do that, I needed to turn my floating point data back into the 1 bit samples. That restored the pulses. It took a little work to figure out why. I was subtracting a waveform with values between -1 and 1 from values that were either -1 or 1. Then I would take the result and convert it to -1 or 1 depending upon whether it was less than zero. So if I subtracted 1 from 1, I would get zero, which is not less than zero so the result would be 1. 1-1=1. Josh is exploring ways to fix this. The one I'm currently working with is: if the result of the subtraction is between -0.5 and 0.5, then assign a random value. So far it seems to work pretty well, I'll be testing it out in beta next week.
Another option would be to use 2 bit samples in the work units. That also appears to work well, but it doubles the size of the workunit files. So we'll try the randomizer first.

Next we'll try to figure out where these pulses are coming from. Since they are coherent across channels and polarizations I suspect that that are somewhere in the electronics chain after the mixer. Maybe even in the data recorder itself. We'll be talking with engineers at Arecibo in order to find out if we can get rid of them at the source.

Dan suggested this may be the first time that an RFI waveform has been removed from a channel by using multibeam data. I doubt it, but it's certainly the first time we've done it, and that's good enough for me.

--
Eric
____________

Profile Clyde C. Phillips, III
Send message
Joined: 2 Aug 00
Posts: 1851
Credit: 5,955,047
RAC: 0
United States
Message 518304 - Posted: 16 Feb 2007, 20:00:24 UTC

It looks like something's happening about 100 times per second, but the frequency varies a total of about 2:1.
____________

Josef W. Segur
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4134
Credit: 1,003,611
RAC: 234
United States
Message 518308 - Posted: 16 Feb 2007, 20:26:11 UTC - in response to Message 518294.

OK, so I missed a week. Nobody's perfect.

'Tis a gem of a post, well worth waiting an extra week.

The second image link has an extra term, this works better.
Joe

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 7939
Credit: 4,007,943
RAC: 706
United Kingdom
Message 518312 - Posted: 16 Feb 2007, 20:32:40 UTC

OK, so an outsider's completely wild guess:

Those FFTs look like the pulses are rounded, and they are symmetrically centred mid-band, so I'd suspect first signal wiring pickup or voltage rail disturbance near/on the ADC(s) in the data recorder, or pickup on the way to the datarecorder.

Do those pulses match HDD activity?!

A second wild shot is if the datarecorder CPU is overloaded and is missing picking up or responding to a proportion of its interrupts.


Thanks for the detail. This looks set to be a good science story.

Regards,
Martin

____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Profile Byron Leigh Hatch @ team Carl Sagan
Volunteer tester
Avatar
Send message
Joined: 5 Jul 99
Posts: 2777
Credit: 11,755,233
RAC: 426
Canada
Message 518344 - Posted: 16 Feb 2007, 21:54:48 UTC

Hi Eric ,

thank you very much for your post

Best Wishes
Byron
:-)

Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15988
Credit: 683,158
RAC: 113
United States
Message 518415 - Posted: 16 Feb 2007, 23:17:52 UTC


@ Eric . . . what a Wonderful Portrait of a Day in the Life.

> it's really worth being in Science, isn't it Sir?

< Thank You Immensely.




____________
BOINC Wiki . . .

Science Status Page . . .

Profile Byron Leigh Hatch @ team Carl Sagan
Volunteer tester
Avatar
Send message
Joined: 5 Jul 99
Posts: 2777
Credit: 11,755,233
RAC: 426
Canada
Message 518429 - Posted: 16 Feb 2007, 23:31:51 UTC - in response to Message 518296.






Dan suggested this may be the first time that an RFI waveform has been removed from a channel by using multibeam data. I doubt it, but it's certainly the first time we've done it, and that's good enough for me.

-- Eric


Eric ,

the "normal" Multibeam WU in beta testing usually on average has an angle range of somewhere between 3 and 0.03

this __Multibeam - 9 result_overflow WU__ that my computer processed has a true angle range of : 135.086062 ?

is there any signifcance re: true angle range of : 135.086062 ?

- Byron


Astro
Volunteer tester
Avatar
Send message
Joined: 16 Apr 02
Posts: 8026
Credit: 600,015
RAC: 0
Message 518438 - Posted: 16 Feb 2007, 23:45:56 UTC
Last modified: 16 Feb 2007, 23:55:16 UTC

do the signals from each head travel independently to the recorders or are they multiplexed and then separated later?

Profile Clyde C. Phillips, III
Send message
Joined: 2 Aug 00
Posts: 1851
Credit: 5,955,047
RAC: 0
United States
Message 518827 - Posted: 17 Feb 2007, 17:44:58 UTC - in response to Message 518429.






Dan suggested this may be the first time that an RFI waveform has been removed from a channel by using multibeam data. I doubt it, but it's certainly the first time we've done it, and that's good enough for me.

-- Eric


Eric ,

the "normal" Multibeam WU in beta testing usually on average has an angle range of somewhere between 3 and 0.03

this __Multibeam - 9 result_overflow WU__ that my computer processed has a true angle range of : 135.086062 ?

is there any signifcance re: true angle range of : 135.086062 ?

- Byron




Byron, I saw someone mentioning one of these extremely long angle range units a year or so ago. On that one it appeared that the recorder was turned off for several hours, the scope then moved most of that angle and then the recorder turned back on. Probably that was the way the cookie crumbled when somebody divided the tape into workunits. There are probably 255 other units with similar ID numbers that have that long angle.

____________

Odysseus
Volunteer tester
Avatar
Send message
Joined: 26 Jul 99
Posts: 1786
Credit: 3,773,708
RAC: 0
Canada
Message 518930 - Posted: 17 Feb 2007, 22:29:46 UTC - in response to Message 518827.

Byron, I saw someone mentioning one of these extremely long angle range units a year or so ago. On that one it appeared that the recorder was turned off for several hours, the scope then moved most of that angle and then the recorder turned back on. Probably that was the way the cookie crumbled when somebody divided the tape into workunits. There are probably 255 other units with similar ID numbers that have that long angle.

We have been seeing batches of Multibeam WUs with extremely high ARs in Beta, often returning “–9 overflows”; the highest I noticed was aroud 75°, with start and end RAs nearly 6 h apart.
____________

Profile Clyde C. Phillips, III
Send message
Joined: 2 Aug 00
Posts: 1851
Credit: 5,955,047
RAC: 0
United States
Message 519364 - Posted: 18 Feb 2007, 19:02:28 UTC

Somewhere, records of telescope (receiver) position at time of day were available for SetiClassic. The same might be true for Alfa. Every few seconds the position in declination and right ascension were given. But with the units with long angle there was a big discontinuity of several hours and many tens of degrees, as I remember. The receiver was probably turned off several hours during which the receiver moved that big angle. I'd have my doubts that any of those 256 discontinuous units would be any good. But to search for those long interruptions might be a bigger waste of time than to crunch 256 worthless units.
____________

Eric Korpela
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 3 Apr 99
Posts: 1076
Credit: 7,771,718
RAC: 7,157
United States
Message 519888 - Posted: 19 Feb 2007, 17:24:17 UTC - in response to Message 518429.



this __Multibeam - 9 result_overflow WU__ that my computer processed has a true angle range of : 135.086062 ?

is there any signifcance re: true angle range of : 135.086062 ?



Hi Byron,

The high angle range workunits are the result of a different problem. Apparently the telescope sends a bad coordinate packet on rare occasion. I'm going to fix that one by checking whether the distance between any two coordinates exceeds the maximum rate the telescope can move in that amount of time.

Eric
____________

Eric Korpela
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 3 Apr 99
Posts: 1076
Credit: 7,771,718
RAC: 7,157
United States
Message 519890 - Posted: 19 Feb 2007, 17:29:21 UTC - in response to Message 518312.


Those FFTs look like the pulses are rounded, and they are symmetrically centred mid-band, so I'd suspect first signal wiring pickup or voltage rail disturbance near/on the ADC(s) in the data recorder, or pickup on the way to the datarecorder.


My bet is fluctuations in the ground plane or a loose connection somewhere.



Do those pulses match HDD activity?!



That's another one we can check on. The files are temporarily stored on a RAM disk before being transferred to the HDD. We should be able to run for a period with no HDD activity.


A second wild shot is if the datarecorder CPU is overloaded and is missing picking up or responding to a proportion of its interrupts.


This is probably not the case, but worth checking....

Thanks,

Eric
____________

Eric Korpela
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 3 Apr 99
Posts: 1076
Credit: 7,771,718
RAC: 7,157
United States
Message 519893 - Posted: 19 Feb 2007, 17:34:33 UTC - in response to Message 518438.

do the signals from each head travel independently to the recorders or are they multiplexed and then separated later?


Each signal travels separately, although there are plenty of opportunities for crosstalk in electronic systems along the way. It's a long way from the telescope to the recorder.

Eric
____________

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 7939
Credit: 4,007,943
RAC: 706
United Kingdom
Message 520012 - Posted: 19 Feb 2007, 21:19:14 UTC - in response to Message 519893.

Each signal travels separately, although there are plenty of opportunities for crosstalk in electronic systems along the way. It's a long way from the telescope to the recorder.

That and the data recorder itself must make for a good scientific paper...


Another thought: Is the interference consistent throughout all the recording period or is it time of day dependant, or runs for short periods? (Mobile phones, portable phones, security radios...?)

Good luck for the glitch hunt.

Regards,
Martin
____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Profile Michael Buckingham
Volunteer tester
Avatar
Send message
Joined: 21 Aug 99
Posts: 4508
Credit: 2,676,597
RAC: 0
United States
Message 520343 - Posted: 20 Feb 2007, 10:00:09 UTC
Last modified: 20 Feb 2007, 10:01:33 UTC

Have you figured out what the pulse repitition interval and the pulse duration is?

Personally, my opinion is any pulse repitition less than 100usecs would or should indicate a pulse doppler radar? Or something with a fairly predictable duty cycle...it doesn't even have to be pulse constant...

Is there a weather radar near by? Perhaps your picking up sidelobes or back lobes getting into the sidelobes of the receiving antenna? Sidelobe poke through or sibelobe of a separate transmitter getting into the sidelobe of the receiver? I am not actually suggesting radar, but even perhaps unshielded equipment nearby?






____________


http://www.mikesbawx.org/photo/

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 7939
Credit: 4,007,943
RAC: 706
United Kingdom
Message 520373 - Posted: 20 Feb 2007, 13:06:03 UTC - in response to Message 520343.

Have you figured out what the pulse repitition interval and the pulse duration is?

Personally, my opinion is any pulse repitition less than 100usecs would or should indicate a pulse doppler radar? Or something with a fairly predictable duty cycle...it doesn't even have to be pulse constant...

I half thought that but the spacing is more of the order of 10ms, and variable (100 pulses per second) which is more like the clicks from a mobile phone or HDD activity...

There are FFT waterfall plots posted earlier in the thread.

Capacitive pickup across cables or ground bounce can be... "difficult"...

Happy tracing!

Regards,
Martin
____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Profile Misfit
Volunteer tester
Avatar
Send message
Joined: 21 Jun 01
Posts: 21782
Credit: 2,507,328
RAC: 2,995
United States
Message 520727 - Posted: 21 Feb 2007, 2:51:31 UTC - in response to Message 520343.

Is there a weather radar near by? Perhaps your picking up sidelobes or back lobes getting into the sidelobes of the receiving antenna? Sidelobe poke through or sibelobe of a separate transmitter getting into the sidelobe of the receiver? I am not actually suggesting radar, but even perhaps unshielded equipment nearby?

Many years ago very quiet night I could pick up the airport radar at Lindberg Field (or possibly the North Island Naval Station) on my CB radio. Gotta watch out for those harmonics.
____________

Profile littlegreenmanfrommars
Volunteer tester
Avatar
Send message
Joined: 28 Jan 06
Posts: 1410
Credit: 934,158
RAC: 0
Australia
Message 520840 - Posted: 21 Feb 2007, 7:45:40 UTC
Last modified: 21 Feb 2007, 7:50:25 UTC

Just a thought Eric...

There seems to be a "ghost" of the central group of vertical lines at about the 0.12 to 0.17 Mhz range of your graph. To me, that looks possibly like a harmonic frequency, caused usually by a signal diffracting between two objects. If so, you may need to research more than one possible source of the RFI.

To illustrate my point, I once experienced the same effect in London, England, while listening to a radio station transmitting on 1.214 MHz. I could receive a weaker, but perfectly clear signal on 310 KHz, a fourth harmonic of the true signal.

Of course, I could be well clear of the mark, but I thought it was worth a mention.
____________

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 7939
Credit: 4,007,943
RAC: 706
United Kingdom
Message 520912 - Posted: 21 Feb 2007, 13:05:53 UTC - in response to Message 520840.
Last modified: 21 Feb 2007, 13:16:05 UTC

Just a thought Eric...

There seems to be a "ghost" of the central group of vertical lines at about the 0.12 to 0.17 Mhz range of your graph. To me, that looks possibly like a harmonic frequency, caused usually by a signal diffracting between two objects. If so, you may need to research more than one possible source of the RFI.

To illustrate my point, I once experienced the same effect in London, England, while listening to a radio station transmitting on 1.214 MHz. I could receive a weaker, but perfectly clear signal on 310 KHz, a fourth harmonic of the true signal...

Errrr... I didn't know that "sub"-harmonics existed!... [edit] (Note: They don't! However, you can see an interferometric effect whereby the amplitude of a signal gets modulated at the difference ferquency from the modulating/interfering signal. This is different from generating a (higher frequency) harmonic from non-perfectly-sinusoidal signals or from FFT decomposition of one signal. Corrections welcomed!) [/edit]

More seriously, you may well have had that (strong) signal breaking through on the mixer stage alternate frequency of one of the IF stages in your receiver. Was it a non-superhetrodyn receiver with only one (or no) IF stages even?

"Ghosting" as commonly seen on TV pictures where you get multipath reflection is from the same frequency signal arriving via multiple paths. You get the same signal but displaced in time. Hence, for an analogue modulated TV picture that is scanned across the screen horizontally, you get multiple identical pictures overlayed but slightly displaced horizontally.


If what you notice are indeed "ghosts" but time delayed, then that would suggest multipath pickup. The "multipath" part could be ANY different route, whether external or just different routes through the pickup path in the affected equipment.

It could just be that the interference source produces that repeating pattern and that it "just looks like that".

Who knows?

Until Eric tracks it down.

It would be interesting to see what more of the WUs show and how consistent/repeatable it is, and so on...

Much better to cure the source rather than to try to remove the symptoms.

(Hey, we could make a reality-TV show out of "What next!"... :-) )


Happy hunting!
Martin

____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

1 · 2 · 3 · 4 . . . 6 · Next

Message boards : SETI@home Staff Blog : Eric's Weekly Post #3 (Warning, science enclosed. Do not fold or bend.)

Copyright © 2014 University of California