SETI@home II Data Recorder

Program Requirements

The start of data flow on the multiple cards are syncronized to within a microsecond.

The recorder is sensitive to user input and should gracefully stop when told to do so. It should also stop gracefully upon reciept of SIGKILL or SIGTERM signals.

It keeps a log of it's activities and errors.

The recorder should gracefully handle the following exceptions.

• lack of radio data from one or more EDT cards
• lack of (or unusual?) telescope data
• data buffer overrun
• any IO difficulties such as a full disk

Data Rate

This rate is achieved by acquireing data on 2 EDT cards simultaneously, each of which is acquiring at 8x the DR1 rate.

The sample rate is always 1/2 that of the clock rate. This is because the frontend HW divides the clockrate by 2 in order to insure that the clock waveform is square, rather than rectangular.

Note that our average data is not 10MBps. We estimate that we will being taking data 1/3 of the the time allocated to astronomy and astronomy gets 1/2 of of the total time.. So our average data rate will be 10/6 = 1.7MBps. Let's round this up to 2MBps for comfort. That will be 173GB per day or ~63TB in a year.

At 173GB per day, how will we get the data to SSL?

• Over the Internet? At 2MBps we could maybe, barely, do this. According to Arun, AO has an OC3 fiber to the University of Puerto Rico, which operates the "GigaPOP" on the island for the UPR system and funnels traffic (over another OC3 I believe) to Internet2 via AMPATH in Miami. Because of the gang up at UPR, Arun reports only getting sustained transfers speeds of no more than around 3MBytes/s to mainland I2 sites.
• Use tapes? Fine, but the tapes are expensive. We could use a small number and rotate them if we have storage at SSL for the 63TB/year. Matt reports being able to reuse a tape ~4x before problems start. Note that the current HW writes to SDT at 4MBps, good by a factor of 2.
• Use disks? Maybe, but we would have to have storage at SSL for the 63TB/year.

On Samples and Buffers

Regarding DR2. The front end HW packs 1 bit complex samples into 16 bit words, from 16 data sources at a sample rate of 2.5e6 samples/s (just like DR1 except 16x the data rate). This data rate is borne by 2 EDT cards, each of which sees 8x the DR1 data rate. So, instead of filling a 1MB buffer in ~1.678s, that buffer is now filled in ~0.2098s. This means that the 13.42s of the 128K FFT takes 64 1MB DR2 buffers on each of the 2 cards.

These buffer counts become important in the frequency stepping logic.

Blocks

Data Portion

Here is a mapping of frontend inputs to bits in a 16 bit EDT word. (Yes, N=1 and N=0 are in the correct order.)

                                    Bank N 
                                    ------
    
    Input Channel         EDT bits          plotter beam file       VGC channel
       N=1  N=0
    -------------         --------          -----------------       -----------
        1 or 9              4,5                     N_2                 N 0
        2 or 10             6,7                     N_3                 N 1
        3 or 11             12,13                   N_6                 N 2
        4 or 12             14,15                   N_7                 N 3 
        5 or 13             0,1                     N_0                 N 4
        6 or 14             2,3                     N_1                 N 5
        7 or 15             8,9                     N_4                 N 6
        8 or 16             10,11                   N_5                 N 7

Master : channels 1-8 (beamset 1)
Slave  : channles 9-16 (beamset 0)
           

Headers

• Tape name.
• Record Type.
• Receiver ID.
• Card ID.
• Block sequence number. This is a count of membufs commited to a diskbuf.
• Data sequence number. The actual sequence number as obtained from the EDT driver.
• Missed buffer flag. The appearance of this flag being true will indicate a disconnect between the previous data portion and the current one.
• Time in UT. This is the time that the data portion recording was completed.
• Telescope coordinates. In a multibeam scenario, the coordinates will be those of the central beam.
• Center frequency. In a multiband scenario, this will be the center frequency of the entire superband.
• Number of membufs configured.
• Number of diskbufs configured.
• entire scramnet blocks? Of which types?

main() thread logic

	establish communication with EDT drivers
	quiet the data source via EDT function bit 3 = 0
	block signals of interest
	init thread structures
	start threads
		threads will:
		allocate buffers and associated objects
		start EDT
		set thread state to ready
	install signal handler
	unblock signals
	wait for thread state to turn ready
	start the data source via EDT function bit 3 = 1
	wait for threads to join
	exit
	

Coordinating Multiple Cards

Data from multiple cards will be coalated on tape. The current disk buffer will be memory mapped to make this easy. Each card's disk buffer thread will be given an initial offset for each new disk buffer. Data will be memcpy'ed from each card's membuf along with a header to that offset and and a new offset obtained by advancing the current offset by the size of membuf+header. Exception to handle: one of the multiple cards stalls in an unexpected way - all cards should probably stop in this situation.

Data from multiple cards will not be coalated on tape. Data from each card will be recorded on separate tapes.

Buffer Management

	n = 0
	lock MemBufMutex[n]
	.
	.
	.
        while(1) {
		wait for MemBuf[n] to return
		place time in TimeBuf[n]
		unlock MembufMutex[n]
		n = Max_n ? 0 : n + 1
		lock MemBufMutex[n]
	}
        

• The disk thread is ahead of the membuf thread. In practice this should not happen, but if it does, we idle on the current ring segment until the membuf thread is once again ahead of the disk thread. The data stream should remain contiguous under this condition.
• The membuf thread has gotten an entire ring ahead of the disk buffer, ie an overflow condition. In this case we enter an indication of an overflow condition into the data stream and continue normally with the current ring buffer segment.
• The membuf thread has passed the diskbuf thread while the data were being copied and thus the copied data are corrupt. In this case we back up the disk buffer write pointer to undo the corrupt write, enter an indication of an overflow condition into the data stream and continue with the next ring buffer segment.

 
The logic is:
        while(1) {
		lock MemBufrMutex[n]
		ThisTimeBuffer = TimeBuffer[n]
		unlock MemBufMutex[n]

		if(ThisTimeBuffer == PriorTimeBuffer)
			continue      // we have passed MemBuf
		if(TimeBuffer[n] - PriorTimeBuffer > BufferAcquireTime)
			indicate buffer overrun

		DiskLocSave = current DiskBuffer location
		format header
		move header+data to DiskBuffer

		lock MemBufMutex[n]
		WriteTimeBuffer = TimeBuffer[n]
		unlock MemBufMutex[n]

		if(TimeBuffer[n] != ThisTimeBuffer)
			back DiskBuf location to DiskLocSave 	
		else
			if(DiskBuff full)
				AdvanceDiskBuf()

		n = Max_n ? 0 : n + 1
	}
        

	sync DiskBuf memmap
	unmap DiskBuf
	close DiskBuf file
	mark DiskBuf file for tape processing
	while(1) {
		if(DiskBuf area is full) sleep 1
	} 
	open next DiskBuf file
	memmap new DiskBuf
	determine new DiskBuf offsets
        

Memory Mapping the Disk Buffers

First, the data recorder mmaps the portion of disk buffer that it is writting to. This portion is held in real memory via mlock()

Second, all disk buffer files are held in a ramdisk file system. The ramdisk is set up via an entry such as the following in /etc/fstab:

    tmpfs                   /ramdisk                tmpfs   size=1280m      0 0
    

Pointing and Time Acquisition

Current time will be obtained from ssystem time, which will be kept accurate via NTP.

SCRAMNET Fields

Field                               Type    Meaning                                         Transforms
--------------------------          ------  ---------------------------------
agcData.st.cblkMCur.dat.posAz               uncorrected Az encoder reading                  *= * 0.0001 gives degrees  
                                            (as much as 2 arcmin wrong, ie 1 beam)
-----------------------------------------------------------------------------------------------------------------------
agcData.st.cblkMCur.dat.posGr               uncorrected ZA encoder reading                  *= * 0.0001 gives degrees
                                            (as much as 2 arcmin wrong, ie 1 beam)
-----------------------------------------------------------------------------------------------------------------------
agcData.st.cblkMCur.dat.timeMs              time (s past midnight) of encoder readings
-----------------------------------------------------------------------------------------------------------------------
alfa.first_bias                             alfa amp bias supply 1 : must be "on" for alfa
                                            ** This may be an IDLE point - I need a range
                                            of volatages from Phil.
-----------------------------------------------------------------------------------------------------------------------
alfa.second_bias                            alfa amp bias supply 2 : must be "on" for alfa
                                            ** This may be an IDLE point - I need a range
                                            of volatages from Phil.
-----------------------------------------------------------------------------------------------------------------------
alfa.motor_position                         TBD
                                            Does this give the parallactic angle, ie
                                            the rotation angle?
-----------------------------------------------------------------------------------------------------------------------
if1Data.st.synI.freqHz[0]                   first LO (normally set to mix the requested     += our LO gives recorder center freq
                                            rf freq to 250 Mhz.
                                            ** IDLE if there are no observable frequencies.
-----------------------------------------------------------------------------------------------------------------------
if1Data.st.synI.ampDb[0]                    ?
-----------------------------------------------------------------------------------------------------------------------
if1Data.st.rfFreq                           ?
-----------------------------------------------------------------------------------------------------------------------
if1Data.st.if1FrqMhz                        ?
-----------------------------------------------------------------------------------------------------------------------
if2Data.st.stat1.useAlfa            bool    input selector for the distribution 
                                            amplifiers at 250Mhz.  Ie, ALFA is ON.
                                            ** IDLE if 0.
-----------------------------------------------------------------------------------------------------------------------
ttData.st.slv[0].tickMsg.position           turret position (alfa's position is at          *= 1./ ( (4096. * 210. / 5.0) / (360.) )
                                            26.64.. probably test +/- .1 degrees            gives degrees
                                            ** IDLE if not in ALFA position - check
                                            for off by more than 1 degree.
-----------------------------------------------------------------------------------------------------------------------
pntData.st.x.pl.curP.raJ                    last requested RA (DO NOT USE)
-----------------------------------------------------------------------------------------------------------------------
pntData.st.x.pl.curP.decJ                   last requested Dec (DO NOT USE)
-----------------------------------------------------------------------------------------------------------------------
pntData.st.x.pl.tm.mjd                      MJD of requested time?
-----------------------------------------------------------------------------------------------------------------------
pntData.st.x.pl.tm.ut1Frac                  MJD fractional of requested time
-----------------------------------------------------------------------------------------------------------------------
????????????????                            Bandpass filter.  
                                            ** This is an IDLE point via observable
                                            frequency check.
-----------------------------------------------------------------------------------------------------------------------

Frequency Stepping

If stepping is indicated, we do so on a specific data boundries, those being the boundries between data points that will be in discrete FFTs at our longest transform length. The goal is that we do not change frequency band during any FFT. The logic is:

	foreach(longest-FFT-discrete-chunk) {
		sky_angle = telescope movement since last check
	        obs_angle = telescope movement since we started stepping
		if(sky_angle < 0.1*BeamRes && obs_angle  < 0.5*BeamRes)
			if(! at max frequency step) {
				step frequency(+/-)
			} else {
				go to zero step
			}
		}
	}
	

Concerning the operation on the condition variable. The condition in question is that we have acquired N buffers of data and thus should check for telescope tracking and perhaps program the synthesizer. This condition is waitied on in the synthesizer thread and is signaled in the data acquisition thread. During normal operations, the synthesizer thread will first obtain the associated mutex and then wait on the condition variable, thus releasing the mutex. Sometime later the data acquisition thread will see that the condition is met and will obtain the mutex, signal the variable, and release the mutiex (note that these three actions are discrete, ie not atomic). Upon waking the synthesizer thread thus re-obtains the mutex (perhaps spinning for a ~microsecond until the data acquisition thread releases it). The synthesizer thread then maskes sure the the condition is indeed met (via a boolean) and then does its thing. It must do its thing quickly enough so that it can re-obtain the mutex and re-wait on the condition (releasing the mutex) before the data acquisition thread sees that the condition is once again met and signals accordingly. If the synthesizer thread is late, it will miss the signal. Not good, but also not terrible.

The manual for controlling the synthesizer is here. Of note is the difference in interpreting the 10 digit frequency command between synthesizer models. On the PTS 500, the low order digit is 0.1Hz while on the PTS 3200 the low order digit is 1Hz. This is accounted for by a config option. Either PTS500 or PTS3200 must be specified as the synth_model in the config file.

Data Acquisition Enabling

• In MemBufInit for each data stream we initialize the EDT card and immediately set the disarm bit in the EDT PCD_FUNCT register and flush the EDT fifo.
• In main(), after returning from StartThreads(), we drop into a loop that we do not leave until all threads have reported ready. Upon leaving the loop, we set the arm bit in the EDT PCD_FUNCT register for all data streams. EDT begins to acquire data. However...
• In DiskBufInit() for each data stream we set DiskBuf[i].Idle to true. This means that even after enabling via EDT, we will be in an idle state (see below) until the IdleWatch thread has a chance to evaluate the engineering and observatory values.

Data Acquisition Idling

• VGC values too high
• ALFA not on (see below for the determination of this).

The logic of idling is as follows. There is a dedicated "idle watch" thread. Within it's control structure are the limit values used to determine whether or not idling is in order. Every N seconds, the idle watch thread wakes up and compares these limits to the current data collected by other execution threads (serializing the latter). If one or more values vioaltes the idling threshold for that value, data taking is idled if not already so. If no thresholds are violated, data taking is re-enabled if previously idled. Data taking is idled and de-idled by toggling the boolean DiskBuf[].Idle. As implied by this reference, idling is a function of the diskbuf thread(s). When DiskBuf[].Idle == true, the local MissingBufs boolean is set to true and no data are written to disk. Note that EDT and membuf[] continue to acquire data, as do the engeneering routines. But none of it is written to disk.

QuickLook Data

The interval between quick looks is a parameter in the configuration file and is given in seconds. In addition to this interval, a quick look is limited to a single center frequency. This frequency is also a parameter in the configuration file. A third and final parameter in the configuration file is the length of a quick look file, given in a count of 1MB buffers.

The synth and diskbuf threads interact to produce quicklook files. The synth thread turns the quick look flag on (== true) and diskbuf thread 0 turns it off (== false).

Serialization

• MemBuf_t.Mutex[]. This ring of mutexes protect associated metadata in the EDT managed ring buffers and gates access to the data itself. Competitor threads for these mutexes are MemBufFunc() and DiskBufFunc().
  MemBufFunc() acquires before calling edt_wait_for_buffers() and releases after edt returns and the stamping of the metadata.
  DiskBufFunc() acquires before reading the metadata associated with the first/next buffer and releases after copying out that metadata. DiskBufFunc() then procedes to copy out the data itself. Note that the data itself, while access gated, is not protected (although we do check for overwrites).
  The initial acquisition is important and is done in MemBufInit(), on behalf of MemBufFunc().
• Synth_t.DataMutex. This mutex protects the global copy of the current synthesizer frequency and the time at with that frequency was set. Competitor threads are MemBufFunc() (reads) and SynthThread() (writes).
  The initial acquisition/release is done in SynthInit() in order to program the initial frequency.
• Synth_t.CheckMutex. This is the mutex associated with the synth step check condition variable Synth_t.CheckNowCond. This condition variable is used by MemBufFunc() to signal SynthThread() that it is time to check for telescope movement and possible frequency stepping.
• Log_t.Mutex. This mutex protects the various log devices so that messages do not become garbled. All threads are competitors for this mutex.
• FrameHeader_t.Mutex. Currently unused.

Tape Recording

Hardware and OS Considerations

A note on PCI.

32 bit, 33 MHz PCI has a data throughput of 90MB/s (plus overhead makes the true throughput 133MB/s. A 64 bit bus would double this and a 66 MHz bus would double it again. On a single 32 bit, 33 MHz bus, our data would cross it once for the DMA to memory, again for the transfer from memory to disk, and a third and forth time if the data are then written to tape. So, 4 x 10MB/s = 40MB/s as the minimum bandwidth.

A note on SCSI.

Ultra SCSI is 20MHz and ultra2 SCSI is 40MHz.

A note on DLT.

From the Quantum site:

Product	Capacity	xfer speed	interface	MTBF
SDLT 320	160/320 GB	16/32 MB/s	LVD ultra 2 SCSI or HVD ultra SCSI	250,000 hours
SDLT 600	300/600 GB	36/72 MB/s	ultra 160 SCSI or Gb fibre in enterprise libraries	250,000 hours

The data will appear in the EDT ring buffer as a series of 16 bit words. Each word will have data from all 8 sources (nominally beams) that a single EDT card collects from. Within each word, the data will appear as 8 contiguous pairs of bits. Each pair of bits is the real/imaginary complex sample from the source.

EDT Issues

• The 1GB/Linux issue.
• Q: Will edt_wait_for_buffers() return only for the current buffer or for all buffers not yet returned?

Multithreading Under Linux

• Scheduling. The driving reason behind making the datarecorder a threaded application is so that the code that waits for the EDT buffers and timestamps them can run unfettered by other activities. Thus we need for this code to have a scheduling policy such that if is it runable, it runs, pre-empting anything else that might be on the CPU. We will use pthreads scheduling attributes to achieve this goal.

  The datarecorder will set the scope, policy, and priority attributes. The scope will be PTHREAD_SCOPE_SYSTEM. Thus, the datarecorder threads are scheduled against all other threads in the system, including the OS itself. The policy will be SCHED_FIFO. This policy results in threads of a higher priority preempting threads of a lower priority and threads of a given priority are placed at the end of a FIFO when they block. The thread that waits for and timestamps the EDT buffers will be given the highest possible priority as given by a call to sched_get_priority_max(SCHED_FIFO). Likewise, a call to sched_get_priority_min(SCHED_FIFO) obtains the minimum priority and this assigned to all other datarecorder threads. Note that the fact that the policy of all datarecorder threads is SCHED_FIFO means that they will trump most other application threads in the system even though they have a low priority within this policy (check this out and explain why).

• Signals. When a signal is generated, it will be sent to one and only one thread within the process. The OS selects the thread to receive an asynchonous signal (like a SIGTERM) in an arbitrary manner. Any thread not blocking the signal may receive it. If all threads are blocking the siganal, it will be pending and the the first thread to unblock the signal will receive it.

  A new thread inherits the signal mask of the thread that created it. That mask can then be changed and only the thread that made the change will be affected. Signal masks are inherited and thread specific. The action to take upon receiving a signal, by contrast, is shared. Any thread calling sigaction() affects the action taken by all threads not blocking the signal.

  The datarecorder will need to handle the asynchronous signals SIGINT and SIGTERM. This is because the EDT driver remains in an open state and is no longer usable if the process terminates without issueing an edt_close(). The simple way to do this is to block these signals prior to thread creation in the main() routine and then install the handler. After thread creation main() can then unblock these signals. Thus the main() thread will be selected to handle them. The handler will set the global trigger variables which tell the various threads to gracefully shut down. They will do so and then join main(). The process will then exit cleanly. Care must be taken to check the trigger variable within all loops in the non-main() threads. Note that multiple threads in a process show up in a ps listing as seperate processes. When issuing a SIGTERM via the kill command, the main() thread must be specified. This is generally the one with the lowest process ID.

Usage Notes

• The device drivers for the cards must be loaded and the cards themselves must be configured. This usaually happens automatically upon bootup via /etc/tc.2/S99EDT. Occasionally, this must be done manually. The most common reason for the need to manually load is that the recorder process ends ungracefully, leaving the cards/drivers in an odd state. To manually unload and then load:

  		/etc/init.d/edt stop
  		/etc/init.d/edt start
  		

• If you are testing without an external clock, the clock rate on the cards must be set. Use "set_clock unit rate":

  		set_clock 0 1000000
  		set_clock 1 1000000
  		

• Run dr2!

Testing

Run dr2 in very verbose mode and check the output for:

• Correct thread scheduing
• Correct configuration
• Datastream initialization
• Config/disarming/arming of EDT cards
• Circular progression of ring buffers (check index and pointers)
• Synchronization memory buffer access between the MEM and DSK threads [last checked 5/31/05]
• Synthesiser stepping: rate and frequency
• Mmap advancement
• Disk file advancement
• Orderly shutdown (via trigger, SIGINT, SIGTERM, SIGALRM)

Run strings on a disk file and check the header for:

• Config info
• Time
• Coordinates

Inject a known signal into the data stream and od -x and dr2_reader to check the disk files.

Also test for:

• Error handling
• Synchronized data collection between EDT cards
• Does data collection obey the external clock? [last checked 5/31/05]

For testing at AO (say, with the -obs option):

• Log onto serendip4.naic.edu.
• From serendip4, log onto aosun.
• From aosun, scp down any new SW.
• From aosun, log onto aolc{1,2,3,4}
• Run the tests.

Current Hardware Setup

Storage: HW RAID with the following logical volumes:

• vol 0 : 2 mirrowed 18GB drives
• vol 1 : 1 18GB drives
• vol 2 : 1 18GB drives

Firmware revs:

• BIOS A06
• embedded server management firmware: 5.22
• system backplane firmware: 5.35
• remote assistant card: 2.0

Bugs and Caveats

If the disk buffer that dr2 is trying to open is never available (ie, marked as .done), dr2 will retry forever. There is no timeout.

The ReadyTime[] associated with the membufs are not protected with a mutex. Perhaps they should be but the thought is that if there were a read/write collision the value read would be garbage and an overrun detection would result. These are only written to in one place so a write collision will never happen.

I should be using the function gettimeofday() rather than the depricated ftime(). This should not be a operational concern.

Sometimes, due to a slowdown in getting disk buffers to tape, we can have "adjacent" buffers that are offset in time. You can see this in the log via different corrected diffs acros the disk buffers. Eg: Wed Apr 12 14:39:12 2006 : DSK[0] : FrameSeq is 947 DataSeq is 1530 IdleCount is 23 Corrected Diff is 560 Wed Apr 12 14:39:12 2006 : DSK[1] : FrameSeq is 839 DataSeq is 1530 IdleCount is 23 Corrected Diff is 668

If the program is stopped very early with a control-C, the machine can hang. Is this because of closing an unopen edt device?

Sometimes DSK[1] starts 1 buffer behind DSK[2], as in :

If the script that puts disk buffers to tape (and then deletes those disk buffers) gets behind, the recorder can get into a situation where the next buffer to open is available for one data stream, but not the other. In this case, consecutive blocks on tape do not follow the pattern of one data stream followed by the other, sync'ed in time. This is norticed in log entries such as:

DSK[1] : FrameSeq is 250623   DataSeq is 250670   IdleCount is 47   Corrected Diff is 0
DSK[0] : FrameSeq is 250622   DataSeq is 250670   IdleCount is 48   Corrected Diff is 0

DSK[1] : New disk buffer /ramdisk/diskbuf_ds1_2 is now open
DSK[0] : Disk buffer /ramdisk/diskbuf_ds0_2.done is marked done but still exists

DSK[1] : FrameSeq is 250633   DataSeq is 250680   IdleCount is 47   Corrected Diff is 0
DSK[0] : Disk buffer /ramdisk/diskbuf_ds0_2.done is marked done but still exists

DSK[1] : FrameSeq is 250643   DataSeq is 250690   IdleCount is 47   Corrected Diff is 0
DSK[0] : Disk buffer /ramdisk/diskbuf_ds0_2.done is marked done but still exists

DSK[1] : FrameSeq is 250653   DataSeq is 250700   IdleCount is 47   Corrected Diff is 0
DSK[0] : New disk buffer /ramdisk/diskbuf_ds0_2 is now open

DSK[0] : FrameSeq is 250634   DataSeq is 250710   IdleCount is 48   Corrected Diff is 28
DSK[1] : FrameSeq is 250663   DataSeq is 250710   IdleCount is 47   Corrected Diff is 0

At mmap advancement time, the acquire time can be tagged as a little longer than normal. When this happens, the next buffer's acquire time is a little shorter. When this happens, the mean of the 2 times is always the expected acquire time. Following the 2 odd times, the acquire time returns to normal until the next mmap advancement. When this happens, it happens at every mmap advancement. It only happens with the buffers associated with data stream 1 (never ds 0).

On close examination, I find that the timestamps for ds 1 are always a bit behind ds 0. Why? The lag stays consistent through the run of the program. The lag seems to always be either ~0.014 seconds or ~0.043 seconds. If the lag is the former, the odd acquires times occur. If it's the latter, the odd acquire times do not occur.

Here's my guess as to what is happening. The EDT drivers are running at the same priority as the diskbuf threads (which do the mmap). Ds 0 sees the need for and triggers an mmap advancement before ds 1. If the lag between ds 0 and ds 1 is tiny, the mmap for ds 0 is in progress when EDT 1 wants to report that a buffer is ready. So there is a slight delay in the report (ie a return from edt_wait_for_buffers()). But EDT has not stopped - it is acquiring the next buffer, which then gets reported "early". So I don't not think that the actual data acquisition is being affected.

Here is some log output showing the odd acquire times.

Sun May 14 13:49:02 2006 : DSK[0] : Advancing mmap...
Sun May 14 13:49:02 2006 : MEM[1] : ringbuf num is 2 ringbuf pointer is 0xb02c4000 ringbuf time is 1147639742.344000 freq is 150000000 freqtime is 1147639
734.559000
Sun May 14 13:49:02 2006 : DSK[1] : ringbuf num is 2 ringbuf pointer is 0xb02c4000 ringbuf time is 1147639742.344000
Sun May 14 13:49:02 2006 : DSK[1] : ThisReadySec is 1147639742.344000  LastReadySec is 1147639742.114000  Diff is 0.230000
1147639741.904000 1147639742.114000 1147639742.344000 1147639741.694000 2
Sun May 14 13:49:02 2006 : DSK[1] : FrameSeq is 8   DataSeq is 31   IdleCount is 23   Corrected Diff is 0
Sun May 14 13:49:02 2006 : DSK[1] : Advancing mmap...
Sun May 14 13:49:02 2006 : MEM[0] : ringbuf num is 3 ringbuf pointer is 0xb04c8000 ringbuf time is 1147639742.520000 freq is 150000000 freqtime is 1147639
734.559000
Sun May 14 13:49:02 2006 : DSK[0] : ringbuf num is 3 ringbuf pointer is 0xb04c8000 ringbuf time is 1147639742.520000
Sun May 14 13:49:02 2006 : DSK[0] : ThisReadySec is 1147639742.520000  LastReadySec is 1147639742.310000  Diff is 0.210000
1147639741.891000 1147639742.100000 1147639742.310000 1147639742.520000 3
Sun May 14 13:49:02 2006 : DSK[0] : FrameSeq is 9   DataSeq is 32   IdleCount is 23   Corrected Diff is 0
Sun May 14 13:49:02 2006 : MEM[1] : ringbuf num is 3 ringbuf pointer is 0xb01c2000 ringbuf time is 1147639742.533000 freq is 150000000 freqtime is 1147639
734.559000
Sun May 14 13:49:02 2006 : DSK[1] : ringbuf num is 3 ringbuf pointer is 0xb01c2000 ringbuf time is 1147639742.533000
Sun May 14 13:49:02 2006 : DSK[1] : ThisReadySec is 1147639742.533000  LastReadySec is 1147639742.344000  Diff is 0.189000
1147639741.904000 1147639742.114000 1147639742.344000 1147639742.533000 3
Sun May 14 13:49:02 2006 : DSK[1] : FrameSeq is 9   DataSeq is 32   IdleCount is 23   Corrected Diff is 0
Sun May 14 13:49:02 2006 : MEM[0] : ringbuf num is 0 ringbuf pointer is 0xb38d6000 ringbuf time is 1147639742.730000 freq is 150000000 freqtime is 1147639
734.559000
Sun May 14 13:49:02 2006 : DSK[0] : ringbuf num is 0 ringbuf pointer is 0xb38d6000 ringbuf time is 1147639742.730000
Sun May 14 13:49:02 2006 : DSK[0] : ThisReadySec is 1147639742.730000  LastReadySec is 1147639742.520000  Diff is 0.210000
1147639742.730000 1147639742.100000 1147639742.310000 1147639742.520000 0
Sun May 14 13:49:02 2006 : DSK[0] : FrameSeq is 10   DataSeq is 33   IdleCount is 23   Corrected Diff is 0
Sun May 14 13:49:02 2006 : MEM[1] : ringbuf num is 0 ringbuf pointer is 0xb10cd000 ringbuf time is 1147639742.743000 freq is 150000000 freqtime is 1147639
734.559000
Sun May 14 13:49:02 2006 : DSK[1] : ringbuf num is 0 ringbuf pointer is 0xb10cd000 ringbuf time is 1147639742.743000
Sun May 14 13:49:02 2006 : DSK[1] : ThisReadySec is 1147639742.743000  LastReadySec is 1147639742.533000  Diff is 0.210000

We do not check for required config items.

stepping is done via the EDT "done counter", not the "frame counter". This means that we might step during an idle period.

If we ever run the recorder such that it does not exit between tapes, the cone and frame counters will overflow.

The first LO as it appears in the header might not be in sync with the second LO and sky frequency, the latter being always in sync with the data (unless the first LO changes mid buffer).

According to Phil P., pointing (uncorrected) can be off as much as 500 arcsec in Az along with 100 arcsec in Za. But does this effect the stepping differential (typically 2.5 minutes for pulsar work)?

                                TO DO
M is ntp not correcting post boot? - it is correcting just takes a while
M watch /home/online/vw/etc/Pnt/ for changed in ut2ut1.dat and maybe obsPosition.dat.
M log rotation and retention
M enable ntp over boot (iptables)
M get quicklooks coming to SSL
M scram monitor

get pointing correction model - via scram?  see PNT block
the MISSED bool is not making it inbto headers
doc Dan's console notes
do I have all code (new files?) CVS'ed
am I always sending synth strings every 13s?


segfault on no trigger on inital startup
checkin my change to skip.c ?
why is there often an idle count diff twixt DSI 0 and 1?
why does dd not obtain the desired amount of data from a buffer or tape (blocksize?)
are we getting bad azzaInit messages?
idle.C - local string overwrite?
we can go idle mid-quicklook - not good
be less verbose
write operating instructions
console messages
Mikial - best alfashm.h location?
Arun - system backups?
timestamps to recorder log
sendmail security
clean code
doc monallsm and phil's screens
close remaining crons
is there a delay in 1st synth? see headers
are statics handled by phil's code?
stinit




minidev
------------------------
re-init EDT on idel/de-idle ?                                                   DEFFERED or CANCELLED
cannot kill if EDT not acquiring data                                           DEFFERED
why does buf_1 sometines start 1 behind buf_0                                   DEFFERED
is there a wrap issue when converting Ra to hours (and Dec to degrees?)         **
is there a wrap issue with sky angle?                                           **
touser+toconsole....                                                            DONE (but kind of a hack)
static azzza init call?                                                         *
better time func (gettimeofday) vs ftime())                                     *
idle string commas                                                              *
utils like synthcmd                                                             *
reasonable defaults?                                                            *
check data types                                                                *
always_step to config
is there a disk1 race condition?
are there any bad data at SayToExit() time? (from out-of-sync mutexes?)
"thread" vs "function"
int vs long
struct constructors?
required configs
clean up and make consistant all error checking (doc).  Poorly handled:
    - not being able to open a disk buffer because the handler is not 
    keeping up
remove deprecated variables
clean up make file
all pthread_exit()'s in place
proper data buffer flushing?
EdtCntl() inline?
edt_wait_buffers_timed() ?
sleep between scram reads?
i vs dsi
ring vs mem
no disk thread = no arming = panic!
tape serial numbers?

end to end tests
------------------------
figure out what beam is what  - plot all 16 beams                               **
record tapes and analyse with seti@home and astropulse:                         **
test with dispersed pulses in noise (astropulse)                                **
test with gaussian                                                              **
test with seti@home pulses                                                      **
hydrogen plots
data interleave on tape OK?                                                     **
sq wave test (across buffers and across beamsets)                               **


tune swap device priority
map PCI slot to EDT unit number                                                 **



remote control and monitoring
------------------------
monitoring:
    - snapshots of data:                                                |
        what are the engineering values?   RA and DEC?                  | All this is done via logging,
        do we see the hydrogen line at the right place?                 | header data, and quicklook data.
            get paul demorest hydrogen compare thing working (AO data   | 
checker).
        scram net data monitoring                                       | Done via log (idle reason) and headers.
    - what mode is the data recorder in:
        taking data?  not taking data?
        waiting for tape change?  for how long?                         | DO, via tape recorder log
        how full is the current tape?                                   | DO, via tape recorder log
        buffer overflow?                                                | Done via the log and headers. 
        when were we last taking data                                   | Done via the log and headers.?
        why aren't we taking data                                       | Done via the log and headers.?
        what is the name of the tape in the drive?                      | DO
    - we need to be able to look at scram net data here in berkeley     | DEFERRED, although we will have quicklook
        so we can diagnose why/why we aren't taking data.               | data.

control:
    - we need to be able to control our synthesizer from berkeley


prep
------------------------
dev/test system with jig                                            DEFERRED
back up setirec2 to CD                                              **
final system dd                                                     **
print code and design doc                                           **
emails over to setirec2                                             **

at AO
------------------------
headers from right place?
do we need pointing corrections?
play with dbuff len and dbuff num
strip down procs
    audit.d
    whatis

Links

Notes
-----

Our beam resolution with ALFA is 3 arcmin.

-----------------------------------------------------------------------
WU frequency boundry / subband number calculation:

 min WU freq =

         2.5 * (n - 1/16)
         ----------------    +    (n < 128 ? 1420 : 1418.5)
              256


 max WU freq =

         2.5 * (n + 15/16)
         -----------------   +    (n < 128 ? 1420 : 1418.5)
              256


 where n is the sub_band number (ie, "WU number").

-----------------------------------------------------------------------


Which power spectrum bin will a signal show up in?

lets say we have a 1KHz signal in a 1.25MHz band  and we run a 16K FFT on it.

The power spectrum runs:
    0-------------------max_pos|max_abs_neg----------------min_abs_neg

so, if the freq turns up as a positive freq, it would be:

    .1/1.25 x  8192             = bin 655
    
or as a negative freq:

    16384 - (.1/1.25 x  8192)   = bin 15729


-----
Here is the scramnet map.  Note that not all block types are broadcast at all times
(correct?).  For example, PNT blocks are sometimes not broadcast.  Why?

The app executes read_scram() which returns a  struct SCRAMNET *.


ra  = scram->pntData.st.x.pl.curP.raJ;
dec = scram->pntData.st.x.pl.curP.decJ;
mjd  = scram->pntData.st.x.pl.tm.mjd + scram->pntData.st.x.pl.tm.ut1Frac;

struct SCRAMNET is defined in scram/aoui/scram.h  as:
struct SCRAMNET {
  int sock;                 /* multicast socket */
  int lock;                 /* true if update blocked for reading */
  struct sockaddr_in from;  /* read address */
  struct BIGENOUGH in;      /* input buffer */
  SCRM_B_AGC agcData;       /* data */
  SCRM_B_TT ttData;
  SCRM_B_PNT pntData;
  SCRM_B_TIE tieData;
  SCRM_B_IF1 if1Data;
  SCRM_B_IF2 if2Data;
  struct EXECSHM exec;
  struct WAPPSHM wapp[4];
  struct ALFASHM alfa;
};

scramread() reads a block from a socket into filed "in", a BIGENOUGH struct which
is a union of all possible block types amd a filed called "magic" which indicates
the type of block just read.  scramread then memcpy's "in" to the appropriate
typed struct within SCRAMNET.

Let's assume we are looking for pointing data.  We look for scram->in.magic == PNT.
If this is so then we go to the SCRM_B_PNT struct.  We are now at :
        scram->pntData

In here we fine a field 
        PNT_STATE       st; 
We are now at
        scram->pntData.st

In here we find a field 
        PNT_X_STATE      x;     /* for point xform*/
We are now at
        scram->pntData.st.x
        
In here we find a field 
        PNT_X_PL        pl;  
We are now at
        scram->pntData.st.x.pl
        
In here we find a fields 
        PNT_X_CUR_PNT   curP;           /* current point (64*int)*/
and
        PNT_X_TIME      tm;             /* for this pipelined point(14*int)*/
We are now at
        scram->pntData.st.x.curP
and
        scram->pntData.st.x.tm
        
In curP we find fields: 
        double          raJ;             /* ra J2000 back converted from az,za*/
        double          decJ;            /* dec J2000 back converted from az,za*/
and in tm we find field:
        int     mjd;                    /* modified julian day for ut1Frac*/
So we have:
        scram->pntData.st.x.pl.curP.raJ;
and
        scram->pntData.st.x.pl.curP.decJ;
and
        scram->pntData.st.x.pl.tm.mjd  
and
        scram->pntData.st.x.pl.tm.ut1Frac;
        
locations:
    struct type         header file
    -------------       -------------------------
    SCRAMNET            scram/aoui/scram.h
    BIGENOUGH           scram/aoui/scram.h
    SCRM_B_PNT          scram/phil/scrmBlkPnt.h
    PNT_STATE           scram/phil/pntProgState.h
    PNT_X_STATE         scram/phil/pntProgState.h
    PNT_X_PL            scram/phil/pntProgState.h
    PNT_X_CUR_PNT       scram/phil/pntProgState.h
    PNT_X_TIME          scram/phil/pntProgState.h


More scramnet notes:

datum we need       struct          header file                 commnets
-------------       -----------     ---------------             -------------------
az, za, time        PNT_STATE       scram/phil/pntProgState.h   RA/Dec are *pointing commands*.
                                                                Az/Za are encoder readings.
                                                                Time should be correct for
                                                                these readings.
turret position     TT   
pointing model                                                  This is in a file that
                                                                Phil updates 2x/yr or so.
                                                                It's our reciever config.
                                                                Auto-email on update?
rotation angle      EXECSHM ?       ?
LO setting          IF1 (IFLO?)     
bandpass filter                                                 This is a boolean that
                                                                Phil has to add to some
                                                                bitmap(?).  Endian issues
                                                                with bit maps?
is alfa on?                                                     There are 2 pieces of data
                                                                that together indicate this.
                                                                Phil has to get back to us
                                                                on this.



All scramnet blocks are brodacast 1/s, right?  Cetainly true for PNT.

/share/galfa/gsr2.0/phil/gen/pnt  is where to find alfabmpos.pro, an
IDL routine that takes az, za, juldate, and (optionally) rotation
angle of the center beam of alfa and computes the ra and dec of all
7 beams.


/share/galfa/gsr2.0/procs/init/hdr/ is where to find
M1_HDR, which does time correction and calls ALFABMPOS


path to file on block to read LO below, 

soon phil will add a bit to read whether
bandpass filter is in our out.   
current bandpass filter is 1440/100 MHz.

note:  better to read LO from this block, as the block that galfa uses
only works when GUI is in use. 

note2: best to use Az, El, 

monitor programs:
go to phil
    64  16:49   cd ~phil
    65  16:49   cd vw
    66  16:49   cd h
    67  16:49   ls
    68  16:50   ls IF1*
    69  16:50   ls if1*
    70  16:51   more if1ProgState.h
    
    

    /home/phil/vw/datatk/shm/Mon  contains

.make.state      If/              Setups/          monProg.sh*
ARTS/            Imakefile        Ter/             monProg.sh.old*
Agc/             Makefile         Tie/             monProgNew.sh*
Alfa/            Pnt/             Tur/             monProgTest.sh*

/home/phil/vw/Solaris/bin    contains

agcLogD*        getTm.h*        logif*          pntMonGet*      terMonGet*      
tieMonSum1*
agcMonGet*      if1MonGet*      mshmD*          pntMonGetN*     tieLogD*        
tieMonSum1New*
agcMonTest*     if2MonGet*      mshmDump*       rcvMonGet*      tieMonGet*      
turLogD*
alfmMonGet*     ifMonGet*       pntDbgGet*      tdtrkfilter*    tieMonSum*      
turMonGet*


http://www2.naic.edu/~phil/

----------------------------------------------------------------

Current disk layout of setirec2.

Summary:    Disk sda contains the system and no swap space or free space.
            Disk sdb contains 1GB of swap and 8GB of free space.
            Disk sdc
[root@setirec2 ~]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2              8483368   6229836   1822592  78% /
/dev/sda1               256667     18461    224954   8% /boot
none                    770768         0    770768   0% /dev/shm
tmpfs                  1310720         0   1310720   0% /ramdisk


[root@setirec2 ~]# cat /proc/swaps
Filename                        Type            Size    Used    Priority
/dev/sdc1                       partition       1052216 0       -1
/dev/sdb1                       partition       1052216 0       -2


[root@setirec2 ~]# fdisk -l
Disk /dev/sda: 9105 MB, 9105023488 bytes
255 heads, 63 sectors/track, 1106 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sda1   *         1        33    265041   83  Linux
/dev/sda2            34      1106   8618872+  83  Linux

Disk /dev/sdb: 9105 MB, 9105023488 bytes
255 heads, 63 sectors/track, 1106 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sdb1   *         1       131   1052226   82  Linux swap

Disk /dev/sdc: 18.2 GB, 18210036736 bytes
255 heads, 63 sectors/track, 2213 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sdc1   *         1       131   1052226   82  Linux swap


----------------------------------------------------------------