Data transfer bottleneck

Message boards : Nebula : Data transfer bottleneck
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile David Anderson
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 13 Feb 99
Posts: 173
Credit: 502,653
RAC: 0
Message 1878025 - Posted: 12 Jul 2017, 21:47:09 UTC

Eric Korpela recently reported an anomaly that turned out to be a bug in how I was building indices. I fixed this and re-ran the pipeline, and the multiplet/pixel results are now online.

But waterfall plots aren't working properly. I haven't finished moving the signal data from Hannover back to Berkeley, in part because the machine we're storing it on (jansky) has a flaky network interface card.

These data transfers are now a bottleneck. I can run the pipeline in about a day, but it takes many days to transfer the signals. I'm using rsync over ssh. I talked with Matt Lebofsky, who says he knows how to speed things up by a significant factor. I'll be looking into those once the jansky hardware situation is fixed.
ID: 1878025 · Report as offensive

Message boards : Nebula : Data transfer bottleneck


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.