Scoring a million pixels

Message boards : Nebula : Scoring a million pixels
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile David Anderson
Volunteer moderator
Project administrator
Project developer
Avatar

Send message
Joined: 13 Feb 99
Posts: 99
Credit: 390,229
RAC: 0
Message 1833609 - Posted: 2 Dec 2016, 8:05:07 UTC
Last modified: 2 Dec 2016, 8:06:02 UTC

Yesterday I tried scoring 1M pixels (previously I'd been scoring 100K or so at a time).

The scoring itself was successful. However, the result was a directory containing 3M files. A script then moves these files into a 1-level hierarchy of 4096 directories. This script ran very slowly, like 1 file every 2-3 seconds. Apparently the filesystem on the Atlas machine does not implement directories in an efficient (constant-time) way. Surprising.

Anyway, I changed things so that the Condor jobs that score pixels put the output files directly into the hierarchy, so we no longer have super-big directories. Hopefully this will fix things.
ID: 1833609 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 5820
Credit: 76,159,447
RAC: 52,330
Russia
Message 1839128 - Posted: 31 Dec 2016, 14:53:00 UTC - in response to Message 1833609.  

Hello David

Any news how Nebula development going?
Did you able to score whole dataset already? If not what is current highest value of pixels processed at one pass?
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1839128 · Report as offensive

Message boards : Nebula : Scoring a million pixels


 
©2017 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.