Message boards :
Nebula :
Scoring a million pixels
Message board moderation
Author | Message |
---|---|
David Anderson Send message Joined: 13 Feb 99 Posts: 173 Credit: 502,653 RAC: 0 |
Yesterday I tried scoring 1M pixels (previously I'd been scoring 100K or so at a time). The scoring itself was successful. However, the result was a directory containing 3M files. A script then moves these files into a 1-level hierarchy of 4096 directories. This script ran very slowly, like 1 file every 2-3 seconds. Apparently the filesystem on the Atlas machine does not implement directories in an efficient (constant-time) way. Surprising. Anyway, I changed things so that the Condor jobs that score pixels put the output files directly into the hierarchy, so we no longer have super-big directories. Hopefully this will fix things. |
Raistmer Send message Joined: 16 Jun 01 Posts: 6325 Credit: 106,370,077 RAC: 121 |
Hello David Any news how Nebula development going? Did you able to score whole dataset already? If not what is current highest value of pixels processed at one pass? SETI apps news We're not gonna fight them. We're gonna transcend them. |
©2024 University of California
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.