64KB Output filesize limitation

Message boards : Number crunching : 64KB Output filesize limitation
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Lint trap

Send message
Joined: 30 May 03
Posts: 871
Credit: 28,092,319
RAC: 0
United States
Message 1217630 - Posted: 13 Apr 2012, 13:15:45 UTC

This wu is producing large output files. Three pc's, so far, have identical -131 error text.

I wonder if any valid science is being lost, or is this most likely just 'noise' of no value ?

My stdoutdae.txt said:

12-Apr-2012 14:19:22 [SETI@home] Computation for task 29se11aa.21883.2382.12.10.109_2 finished
12-Apr-2012 14:19:22 [SETI@home] Output file 29se11aa.21883.2382.12.10.109_2_0 for task 29se11aa.21883.2382.12.10.109_2 exceeds size limit.
12-Apr-2012 14:19:22 [SETI@home] File size: 68146.000000 bytes.  Limit: 65536.000000 bytes



Lt

ID: 1217630 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1217635 - Posted: 13 Apr 2012, 14:03:05 UTC - in response to Message 1217630.  
Last modified: 13 Apr 2012, 14:38:04 UTC

I downloaded the data file, and it's 404 KB in size (compared to the normal 367 KB)

There are a vast number of <coordinate_t> elements in the data header, but otherwise - on a relatively casual glance - it looks normal.

I have a number of other abnormally-sized data files in my cache:

404 KB from
29se11aa.2171.2382
29se11aa.21883.2382 (like yours)

392 KB from
29se11aa.12895.2484
29se11aa.20678.2484

all downloaded between about 16:30 and 20:00 on Wednesday 11 April.

That does sound like another splitter problem worth drawing to Jeff's attention - anyone like to add some extra observations while I go out on a short errand? Then I'll do a full report when I get back.

Edit - put in fuller identification numbers, not all sequences from that tape are affected.
ID: 1217635 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1217652 - Posted: 13 Apr 2012, 15:13:17 UTC
Last modified: 13 Apr 2012, 15:15:56 UTC

I had one from that same 29se11aa.21883 group a few days ago. Which I didn't think much of at the time. Checking the file it was 404KB like the one you checked as well.

Looking at the tasks on my systems it looks like it can be further narrowed down to 29se11aa.21883.2382.12.10

I found these hanging out on one of my systems.
04/11/2012  10:15 AM           413,626 29se11aa.21883.2382.12.10.212
04/11/2012  10:15 AM           413,626 29se11aa.21883.2382.12.10.204
04/11/2012  10:15 AM           413,627 29se11aa.21883.2382.12.10.230
04/11/2012  10:15 AM           413,628 29se11aa.21883.2382.12.10.221
04/11/2012  10:15 AM           413,628 29se11aa.21883.2382.12.10.219
04/11/2012  10:15 AM           413,628 29se11aa.21883.2382.12.10.209

Where other tasks in the 29se11aa.21883.10980. & 29se11aa.21883.11798 groups are the normal 375,222 byte size.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1217652 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1217655 - Posted: 13 Apr 2012, 15:25:11 UTC

The ones I've found have all been shorties - 14-day deadlines, typical of VHAR.

In fact, they seem to be VVHAR - the one Lt linked has AR=55, I've just looked at an AR=57 here.
ID: 1217655 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1217660 - Posted: 13 Apr 2012, 15:50:44 UTC - in response to Message 1217655.  
Last modified: 13 Apr 2012, 15:52:28 UTC

The ones I've found have all been shorties - 14-day deadlines, typical of VHAR.

In fact, they seem to be VVHAR - the one Lt linked has AR=55, I've just looked at an AR=57 here.

The current tasks & the one previous one I had are all 55.95.

I found one where my wing mate has successfully completed the task. So it looks like at least some of them will be OK.
http://setiathome.berkeley.edu/workunit.php?wuid=968984952

Unless this is one of those cases where it is an issue when using the opt apps. Looking at the 4 that Lint trap had. The stock app seems to have completed the tasks OK & the opt app ended up with the -131.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1217660 · Report as offensive
Profile Lint trap

Send message
Joined: 30 May 03
Posts: 871
Credit: 28,092,319
RAC: 0
United States
Message 1217703 - Posted: 13 Apr 2012, 17:16:13 UTC - in response to Message 1217660.  


Unless this is one of those cases where it is an issue when using the opt apps. Looking at the 4 that Lint trap had. The stock app seems to have completed the tasks OK & the opt app ended up with the -131.



By completed OK, you mean the output filesize didn't exceed the limit ?

If it's just a case of the Opt-apps being too wordy, perhaps Lunatics could check for that and somehow limit the output text to stay under 64KB, or, the limit could be changed ?


Lt
ID: 1217703 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 1217706 - Posted: 13 Apr 2012, 17:23:36 UTC - in response to Message 1217655.  

The ones I've found have all been shorties - 14-day deadlines, typical of VHAR.

In fact, they seem to be VVHAR - the one Lt linked has AR=55, I've just looked at an AR=57 here.

Hmm, for the one HAL9000 linked, the first two coordinate sets are

      <coordinate_t>
        <time>2455834.2627723</time>
        <ra>15.211767558555</ra>
        <dec>18.111415397339</dec>
      </coordinate_t>
      <coordinate_t>
        <time>2455834.4264586</time>
        <ra>19.048461812708</ra>
        <dec>17.949663297908</dec>
      </coordinate_t>


Almost all the AR=56 is in that pair, they're nearly 4 hours apart in both time and ra.

The 368 coordinate sets make the header section 58562 bytes, so adding a few signals to that size header can easily make a result file over 65536.
                                                                  Joe
ID: 1217706 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1217716 - Posted: 13 Apr 2012, 17:44:49 UTC - in response to Message 1217703.  


Unless this is one of those cases where it is an issue when using the opt apps. Looking at the 4 that Lint trap had. The stock app seems to have completed the tasks OK & the opt app ended up with the -131.

By completed OK, you mean the output filesize didn't exceed the limit ?

If it's just a case of the Opt-apps being too wordy, perhaps Lunatics could check for that and somehow limit the output text to stay under 64KB, or, the limit could be changed ?

No, it wouldn't help.

The wordy Lunatics stuff is in stderr_txt, which gets sent to the server by a different channel when you report the task.

The 64KB limit is on the uploaded result file which one would expect to be identical - that's the intention, anyway, otherwise the results won't validate.

Unless there's something subtle like the stock app producing a file with Unix-style line endings (CR only), and the Opti apps using Windows line endings (CR-LF). In a file with a lot of short lines, that can make a big difference.
ID: 1217716 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1217719 - Posted: 13 Apr 2012, 17:50:12 UTC - in response to Message 1217706.  

The ones I've found have all been shorties - 14-day deadlines, typical of VHAR.

In fact, they seem to be VVHAR - the one Lt linked has AR=55, I've just looked at an AR=57 here.

Hmm, for the one HAL9000 linked, the first two coordinate sets are

      <coordinate_t>
        <time>2455834.2627723</time>
        <ra>15.211767558555</ra>
        <dec>18.111415397339</dec>
      </coordinate_t>
      <coordinate_t>
        <time>2455834.4264586</time>
        <ra>19.048461812708</ra>
        <dec>17.949663297908</dec>
      </coordinate_t>


Almost all the AR=56 is in that pair, they're nearly 4 hours apart in both time and ra.

The 368 coordinate sets make the header section 58562 bytes, so adding a few signals to that size header can easily make a result file over 65536.
                                                                  Joe

Lt's first report, relating to a 404 KB data file, was only 2.5KB over the limit. So the second set I listed, with 392 KB data files, will almost certainly be OK, unless they find a large number of signals. But we're certainly on the margin.
ID: 1217719 · Report as offensive
Profile Lint trap

Send message
Joined: 30 May 03
Posts: 871
Credit: 28,092,319
RAC: 0
United States
Message 1217725 - Posted: 13 Apr 2012, 18:06:38 UTC - in response to Message 1217716.  


Unless this is one of those cases where it is an issue when using the opt apps. Looking at the 4 that Lint trap had. The stock app seems to have completed the tasks OK & the opt app ended up with the -131.

By completed OK, you mean the output filesize didn't exceed the limit ?

If it's just a case of the Opt-apps being too wordy, perhaps Lunatics could check for that and somehow limit the output text to stay under 64KB, or, the limit could be changed ?

No, it wouldn't help.

The wordy Lunatics stuff is in stderr_txt, which gets sent to the server by a different channel when you report the task.

The 64KB limit is on the uploaded result file which one would expect to be identical - that's the intention, anyway, otherwise the results won't validate.

Unless there's something subtle like the stock app producing a file with Unix-style line endings (CR only), and the Opti apps using Windows line endings (CR-LF). In a file with a lot of short lines, that can make a big difference.



Thanks for re-explaining that! I always have trouble with the distinctions. It's clearer now.

Lt

ID: 1217725 · Report as offensive
AndrewM
Volunteer tester

Send message
Joined: 5 Jan 08
Posts: 369
Credit: 34,275,196
RAC: 0
Australia
Message 1217755 - Posted: 13 Apr 2012, 19:08:30 UTC

Some extra observations:

These finished with -131 errors:

29se11aa.21883.2382.12.10.68_0_0
29se11aa.21883.2382.12.10.65_1_0
29se11aa.21883.2382.12.10.43_0_0
29se11aa.14713.2382.7.10.160_0_0
29se11aa.14713.2382.7.10.143_0_0

This is in cache at 404Kb

29se11aa.29304.2382.9.10.230


Cheers
AndrewM
ID: 1217755 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1217776 - Posted: 13 Apr 2012, 19:51:45 UTC - in response to Message 1217716.  


Unless this is one of those cases where it is an issue when using the opt apps. Looking at the 4 that Lint trap had. The stock app seems to have completed the tasks OK & the opt app ended up with the -131.

By completed OK, you mean the output filesize didn't exceed the limit ?

If it's just a case of the Opt-apps being too wordy, perhaps Lunatics could check for that and somehow limit the output text to stay under 64KB, or, the limit could be changed ?

No, it wouldn't help.

The wordy Lunatics stuff is in stderr_txt, which gets sent to the server by a different channel when you report the task.

The 64KB limit is on the uploaded result file which one would expect to be identical - that's the intention, anyway, otherwise the results won't validate.

Unless there's something subtle like the stock app producing a file with Unix-style line endings (CR only), and the Opti apps using Windows line endings (CR-LF). In a file with a lot of short lines, that can make a big difference.

If it would help anything. I could setup a system to process the 7 files I saved with each app and then zip up the result files for someone to grab and look over.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1217776 · Report as offensive
Profile Mark Wyzenbeek
Avatar

Send message
Joined: 28 Jun 99
Posts: 134
Credit: 6,203,079
RAC: 0
United States
Message 1217939 - Posted: 14 Apr 2012, 3:37:02 UTC - in response to Message 1217776.  

I had a task end in error with a -131. A search brought me here. At least now I know what the error means. If it doesn't reoccur I won't worry about it. Thanks
The Universe is not only stranger than you imagine, it's stranger than you can imagine.

SETI@home classic workunits 1,405 CPU time 57,318 hours
ID: 1217939 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 1217974 - Posted: 14 Apr 2012, 4:38:52 UTC - in response to Message 1217776.  

...
Unless there's something subtle like the stock app producing a file with Unix-style line endings (CR only), and the Opti apps using Windows line endings (CR-LF). In a file with a lot of short lines, that can make a big difference.

If it would help anything. I could setup a system to process the 7 files I saved with each app and then zip up the result files for someone to grab and look over.

I did do an offline comparison run with the 29se11aa.21883.2382.12.10.217 WU. Stock 6.03 produced a 66019 bytes result file, AK_v8b2 65758 bytes. IOW, both would have incurred a -131 error.

Both apps produce LF only line endings. The 261 byte difference is because 6.03 was built in 2008 with an older version of the workunit header structure so has a few more lines of header in the result file. That makes the observed cases where stock did not error but optimized did even more puzzling.

With the coordinates obviously wrong in some ways, the positions of any found signals wouldn't be known properly. IMO it is preferable to error out the WUs rather than assimilate bad results, but not a big deal since 99.9999...% of the signals in the science database are no more meaningful.
                                                                 Joe
ID: 1217974 · Report as offensive
Profile Lint trap

Send message
Joined: 30 May 03
Posts: 871
Credit: 28,092,319
RAC: 0
United States
Message 1218549 - Posted: 15 Apr 2012, 2:21:07 UTC



Thanks to everyone for your investigative efforts! It's good to know nothing important is being lost.

Lt


ID: 1218549 · Report as offensive

Message boards : Number crunching : 64KB Output filesize limitation


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.