Data Chat

Message boards : Number crunching : Data Chat
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 8 · 9 · 10 · 11 · 12 · 13 · 14 . . . 34 · Next

AuthorMessage
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 2004978 - Posted: 31 Jul 2019, 6:58:02 UTC

The last bit of the file name is usually a number from 006 through 125 . Sometimes we get only a portion of the files like lately we have been getting up to 025 or so. Sometimes there is a gap in the numbers around the 90s. This sequence is missing many of the files.
I see 6-14, 24,26, 32, 45 and so on... seems like many gaps in the numbering system.

I don't understand why? did they not record bits? did the inbetween numbers have bad data? Is this just a normal thing I am wrongly feeling weird about??
Mainly curiosity though... just a big why are the files named this way and what does it mean?

The files returned rate is normal, so the data isn't short or noisy.
ID: 2004978 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2004981 - Posted: 31 Jul 2019, 7:05:52 UTC - in response to Message 2004978.  

The last bit of the file name is usually a number from 006 through 125 . Sometimes we get only a portion of the files like lately we have been getting up to 025 or so. Sometimes there is a gap in the numbers around the 90s. This sequence is missing many of the files.
I see 6-14, 24,26, 32, 45 and so on... seems like many gaps in the numbering system.

I don't understand why? did they not record bits? did the inbetween numbers have bad data? Is this just a normal thing I am wrongly feeling weird about??
Mainly curiosity though... just a big why are the files named this way and what does it mean?

The files returned rate is normal, so the data isn't short or noisy.

Haven't a clue about the naming other than the target name which is from a stellar catalogue. The rest is unknown to me. That question needs answering by Jeff Cobb or Matt Lebofsky of Breakthrough Listen.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2004981 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 2005027 - Posted: 31 Jul 2019, 15:10:01 UTC - in response to Message 2004627.  

The key trigger is "report the same result twice". The easiest way I can think of to invoke that is "don't process ACKs for the next report".

And now dinner calls...
Well, the final dinner turned out to be 'on American water', rather than 'soil':



And a fine round-off to my trip it was too. Many thanks to Brian, our host here. Too busy attending to the fresh salmon to think about coding, but I still think the line quoted is the easiest. I'll have to close down soon to pack for the flight back tonight, but we can pick this up again after maintenance Tuesday, when I'm back home.

Richard, I'm sorry the salmon interfered with your thinking. IMO the sockeye salmon you had on Sat. eve is my favorite specie but arguably chinook is the best.
ID: 2005027 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2005097 - Posted: 1 Aug 2019, 1:08:12 UTC - in response to Message 2004978.  
Last modified: 1 Aug 2019, 1:40:44 UTC

The last bit of the file name is usually a number from 006 through 125 . Sometimes we get only a portion of the files like lately we have been getting up to 025 or so. Sometimes there is a gap in the numbers around the 90s. This sequence is missing many of the files.
I see 6-14, 24,26, 32, 45 and so on... seems like many gaps in the numbering system.
understand why? did they not record bits? did the inbetween numbers have bad data? Is this just a normal thing I am wrongly feeling weird about??
Mainly curiosity though... just a big why are the files named this way and what does it mean?
The files returned rate is normal, so the data isn't short or noisy.


blc32_2bit_guppi_58643_73551_HIP68184_0093

Part 1 - Breakthrough Listen (Capture or channel?) with the number of the receiver channel/recorder which sourced the tape. Originally this was a single digit number for channels 0-7, but with the increase in recording channels this has gone to two digits 00-67.
Part 2 - guppi is the file format and is an acronym for Greenbank Ultimate Pulsar Processing Instrument. The 2-bit part refers to the data format at some point in the tapes origin, again Eric has explained that as well, but this generally confuses a lot of people because we have been receiving our WUs in 4 bit format for a couple of years now. This enables processing with finer resolution and better noise rejection.
Part 3 - The date of the recording in modified Julian format.
Part 4 - The daily offset in seconds past midnight for the beginning, I presume, of the recorded segment. (a number from 00000 to 86399)
Part 5 - The object being observed according to whichever star/object catalogue is being used.
Part 6 - The tape sequence number for the data recorded on that receiver in that session.

. . The gaps in the tape sequence could be caused by the telescope being retasked or reconfigured, a major noise event making the data unusable or many other issues that could crop up.

Stephen

. .
ID: 2005097 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2005100 - Posted: 1 Aug 2019, 1:58:44 UTC - in response to Message 2005097.  

Thanks for the primer, Stephen.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2005100 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 2005117 - Posted: 1 Aug 2019, 4:55:31 UTC

Thank you Stephen. I need make a faq of all the good answers to my many many questions. Thank you.
ID: 2005117 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 2005184 - Posted: 1 Aug 2019, 16:21:23 UTC

They recorded Aricebo data last night, so hopefully that will be given to us to split soon. I'm also expecting more Greenbank data for day 58643, as it usually takes us weeks (months?) to process what they can record in a day.
ID: 2005184 · Report as offensive
Kiska
Volunteer tester

Send message
Joined: 31 Mar 12
Posts: 302
Credit: 3,067,762
RAC: 0
Australia
Message 2005193 - Posted: 1 Aug 2019, 17:01:55 UTC - in response to Message 2005097.  

The last bit of the file name is usually a number from 006 through 125 . Sometimes we get only a portion of the files like lately we have been getting up to 025 or so. Sometimes there is a gap in the numbers around the 90s. This sequence is missing many of the files.
I see 6-14, 24,26, 32, 45 and so on... seems like many gaps in the numbering system.
understand why? did they not record bits? did the inbetween numbers have bad data? Is this just a normal thing I am wrongly feeling weird about??
Mainly curiosity though... just a big why are the files named this way and what does it mean?
The files returned rate is normal, so the data isn't short or noisy.


blc32_2bit_guppi_58643_73551_HIP68184_0093

Part 1 - Breakthrough Listen (Capture or channel?) with the number of the receiver channel/recorder which sourced the tape. Originally this was a single digit number for channels 0-7, but with the increase in recording channels this has gone to two digits 00-67.


Refer to: https://github.com/UCBerkeleySETI/breakthrough/blob/master/GBT/waterfall.md or https://setiathome.berkeley.edu/forum_thread.php?id=80380#1823582

aka BLC(x) is from a compute node there are just more of them than before

Although it hasn't been updated since 2017 this quote is still relevant since they actually did it
The VEGAS instrument at GBT is a big digitizer. It samples at 20 gigasamples / s which gives 10 GHz usable bandwidth. Right now we only run one of VEGAS’s eight ROACH boards, so we get 1/8 of the total bandwidth (about 1.25 GHz). These data come over 10 gigabit ethernet through a network switch to the BL compute infrastructure. Breakthrough Listen will eventually duplicate the existing compute infrastructure by a factor 8, allowing the whole 10 GHz bandwidth to be recorded.
ID: 2005193 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 2005198 - Posted: 1 Aug 2019, 17:17:23 UTC - in response to Message 2005193.  

Thanks for the post Kiska... You would think Keith would remember I posted this 3 years ago since I am on his own team........lol
ID: 2005198 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2005206 - Posted: 1 Aug 2019, 17:48:36 UTC - in response to Message 2005198.  

Thanks for the post Kiska... You would think Keith would remember I posted this 3 years ago since I am on his own team........lol

But . . . but . . . I haven't been a member on the team for 3 years. Haha.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2005206 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 2005215 - Posted: 1 Aug 2019, 18:31:48 UTC - in response to Message 2005206.  

Thanks for the post Kiska... You would think Keith would remember I posted this 3 years ago since I am on his own team........lol

But . . . but . . . I haven't been a member on the team for 3 years. Haha.


excuses....excuses.... <insert_nerd_laugh>hahaha</insert_nerd_laugh>

https://youtu.be/RQ9o7XvhH1k
ID: 2005215 · Report as offensive
Profile Sebastian M. Bobrecki
Volunteer tester

Send message
Joined: 7 Feb 02
Posts: 23
Credit: 38,375,443
RAC: 0
Poland
Message 2005477 - Posted: 3 Aug 2019, 8:39:48 UTC - in response to Message 2005097.  
Last modified: 3 Aug 2019, 8:42:58 UTC

...
blc32_2bit_guppi_58643_73551_HIP68184_0093
...
Part 3 - The date of the recording in modified Julian format.
Part 4 - The daily offset in seconds past midnight for the beginning, I presume, of the recorded segment. (a number from 00000 to 86399)
...
One correction. The Part 4 is not a offset. It's a fraction of day. It is calculated by dividing given number of seconds pas midnight by number of seconds in the day (witch is normally 86400, but can be 86399 or 86401 if leap seconds are required ). And back to get the number of seconds you have to multiply this fraction by number of seconds in day. For example in: blc11_2bit_guppi_58692_01635_HIP80413_0117 there is 01635 as fractional part. So to get number of seconds past midnight just do:

86400s x 0.01635 = 1412.64s = 23m 32.64s

Thanks to that there is sub-second accuracy.
ID: 2005477 · Report as offensive
Kiska
Volunteer tester

Send message
Joined: 31 Mar 12
Posts: 302
Credit: 3,067,762
RAC: 0
Australia
Message 2005489 - Posted: 3 Aug 2019, 11:57:51 UTC - in response to Message 2005477.  
Last modified: 3 Aug 2019, 11:58:13 UTC

...
blc32_2bit_guppi_58643_73551_HIP68184_0093
...
Part 3 - The date of the recording in modified Julian format.
Part 4 - The daily offset in seconds past midnight for the beginning, I presume, of the recorded segment. (a number from 00000 to 86399)
...
One correction. The Part 4 is not a offset. It's a fraction of day. It is calculated by dividing given number of seconds pas midnight by number of seconds in the day (witch is normally 86400, but can be 86399 or 86401 if leap seconds are required ). And back to get the number of seconds you have to multiply this fraction by number of seconds in day. For example in: blc11_2bit_guppi_58692_01635_HIP80413_0117 there is 01635 as fractional part. So to get number of seconds past midnight just do:

86400s x 0.01635 = 1412.64s = 23m 32.64s

Thanks to that there is sub-second accuracy.


I'll quote from Zalster from 2016 from this thread: https://setiathome.berkeley.edu/forum_thread.php?id=80380#1823582
From Eric we got
blc3_2bit_guppi_57451_19304_HIP62472_0003

blc3: Breakthrough Listen
2bit: bits per real sample (4 bit per complex sample)
guppi: File format (Greenbank Ultimate Pulsar Processing Instrument)
57451: Modified Julian Date
19304: Seconds past midnight
HIP62472: Object Name
0003: Observation sequence number


Using your example: blc11_2bit_guppi_58692_01635_HIP80413_0117
Would be 1635 seconds past midnight or 00:27:15(whichever timezone)
ID: 2005489 · Report as offensive
Profile Sebastian M. Bobrecki
Volunteer tester

Send message
Joined: 7 Feb 02
Posts: 23
Credit: 38,375,443
RAC: 0
Poland
Message 2005713 - Posted: 4 Aug 2019, 15:25:35 UTC - in response to Message 2005489.  

...
Using your example: blc11_2bit_guppi_58692_01635_HIP80413_0117
Would be 1635 seconds past midnight or 00:27:15(whichever timezone)
If this is the case, than this means that this is not a Julian date. Following wikipedia: ... Ordinarily calculating the fractional portion of the JD is straightforward; the number of seconds that have elapsed in the day divided by the number of seconds in a day, 86,400. ...
ID: 2005713 · Report as offensive
Kiska
Volunteer tester

Send message
Joined: 31 Mar 12
Posts: 302
Credit: 3,067,762
RAC: 0
Australia
Message 2005726 - Posted: 4 Aug 2019, 17:20:22 UTC - in response to Message 2005713.  
Last modified: 4 Aug 2019, 17:27:56 UTC

...
Using your example: blc11_2bit_guppi_58692_01635_HIP80413_0117
Would be 1635 seconds past midnight or 00:27:15(whichever timezone)
If this is the case, than this means that this is not a Julian date. Following wikipedia: ... Ordinarily calculating the fractional portion of the JD is straightforward; the number of seconds that have elapsed in the day divided by the number of seconds in a day, 86,400. ...


It isn't, its MJD(Modified Julian Date)
http://scienceworld.wolfram.com/astronomy/ModifiedJulianDate.html

Again using your example:
blc11_2bit_guppi_58692_01635_HIP80413_0117

This was recorded 28th July 2019 at 00:27:15UTC
ID: 2005726 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2005784 - Posted: 5 Aug 2019, 0:05:42 UTC
Last modified: 5 Aug 2019, 0:06:54 UTC

. . Hi people,

. . I am about to make myself very unpopular here by pointing out that you are all wrong. You are discussing the format of the Julian date which is the first segment ONLY of the two. And you are all correct that there is a calculation involved but it does NOT give sub-second accuracy, it gives sub-day accuracy.

. . 58692_01635

. . In Julian date that time would be one 7 digit number plus a decimal fraction component derived by the calculations you are discussing for the fraction of the day after noon. That is, the above time would be represented as a single date segment of 2458692.519.

. . In strict Modified Julian it would still be a single number of 58692.019. (calculated from midnight not noon, and I limited the accuracy to 3 decimal places)

. . Those calculations derive a fraction of a day not a fraction of a second.

. . So clearly this format provides the MJD for midnight with the added segment which is, as Eric stated, the offset in seconds from the stated MJD at midnight. This gives greater accuracy for the recording time to the second rather than a fraction of a day.

Stephen

<shrug>
ID: 2005784 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 2006012 - Posted: 6 Aug 2019, 15:01:04 UTC

It's maintenance shutdown day.

We have a nice long queue of Green Bank files to split, so I'm not expecting any new files. I'm guessing they are hoarding the Aricebo files as we haven't gotten one in a few days. Maybe they are saving them for WOW??
ID: 2006012 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 2006325 - Posted: 8 Aug 2019, 13:28:25 UTC

My last post didn't age well. We didn't get a shutdown and they just split 24jl19aa .
ID: 2006325 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2006395 - Posted: 8 Aug 2019, 23:52:35 UTC - in response to Message 2006325.  

My last post didn't age well. We didn't get a shutdown and they just split 24jl19aa .

. . Have you not heard of the Phantom of the Forums?

. . When you write about something happening, not happening or just being difficult, the Phantom makes it go away :)

. . "I haven't been able to get new work for hours!"

. . Two minutes later ... whooosh! 50 downloaded tasks ...

. . He(She?) is at work constantly ...

Stephen

:)
ID: 2006395 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2006872 - Posted: 11 Aug 2019, 23:45:31 UTC - in response to Message 2006826.  
Last modified: 11 Aug 2019, 23:45:51 UTC

Time to add new tapes. The runtimes of most of the blc11 tasks, are pretty short.
Only 3178 total channels left to do, as of now.

. . Sadly it is only 5pm Sunday in sunny Berkeley CA. The guys will probably not be in the lab for another 15 hours (very roughly speaking), but the good news is that there is enough work to last until then :)

Stephen

:)
ID: 2006872 · Report as offensive
Previous · 1 . . . 8 · 9 · 10 · 11 · 12 · 13 · 14 . . . 34 · Next

Message boards : Number crunching : Data Chat


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.