64KB Output filesize limitation


log in

Advanced search

Message boards : Number crunching : 64KB Output filesize limitation

Author Message
Profile Lint trap
Send message
Joined: 30 May 03
Posts: 856
Credit: 24,550,260
RAC: 11,306
United States
Message 1217630 - Posted: 13 Apr 2012, 13:15:45 UTC

This wu is producing large output files. Three pc's, so far, have identical -131 error text.

I wonder if any valid science is being lost, or is this most likely just 'noise' of no value ?

My stdoutdae.txt said:

12-Apr-2012 14:19:22 [SETI@home] Computation for task 29se11aa.21883.2382.12.10.109_2 finished 12-Apr-2012 14:19:22 [SETI@home] Output file 29se11aa.21883.2382.12.10.109_2_0 for task 29se11aa.21883.2382.12.10.109_2 exceeds size limit. 12-Apr-2012 14:19:22 [SETI@home] File size: 68146.000000 bytes. Limit: 65536.000000 bytes



Lt

Richard Haselgrove
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8275
Credit: 44,939,496
RAC: 13,639
United Kingdom
Message 1217635 - Posted: 13 Apr 2012, 14:03:05 UTC - in response to Message 1217630.
Last modified: 13 Apr 2012, 14:38:04 UTC

I downloaded the data file, and it's 404 KB in size (compared to the normal 367 KB)

There are a vast number of <coordinate_t> elements in the data header, but otherwise - on a relatively casual glance - it looks normal.

I have a number of other abnormally-sized data files in my cache:

404 KB from
29se11aa.2171.2382
29se11aa.21883.2382 (like yours)

392 KB from
29se11aa.12895.2484
29se11aa.20678.2484

all downloaded between about 16:30 and 20:00 on Wednesday 11 April.

That does sound like another splitter problem worth drawing to Jeff's attention - anyone like to add some extra observations while I go out on a short errand? Then I'll do a full report when I get back.

Edit - put in fuller identification numbers, not all sequences from that tape are affected.

Profile HAL9000
Volunteer tester
Avatar
Send message
Joined: 11 Sep 99
Posts: 3567
Credit: 97,955,075
RAC: 79,022
United States
Message 1217652 - Posted: 13 Apr 2012, 15:13:17 UTC
Last modified: 13 Apr 2012, 15:15:56 UTC

I had one from that same 29se11aa.21883 group a few days ago. Which I didn't think much of at the time. Checking the file it was 404KB like the one you checked as well.

Looking at the tasks on my systems it looks like it can be further narrowed down to 29se11aa.21883.2382.12.10

I found these hanging out on one of my systems.

04/11/2012 10:15 AM 413,626 29se11aa.21883.2382.12.10.212 04/11/2012 10:15 AM 413,626 29se11aa.21883.2382.12.10.204 04/11/2012 10:15 AM 413,627 29se11aa.21883.2382.12.10.230 04/11/2012 10:15 AM 413,628 29se11aa.21883.2382.12.10.221 04/11/2012 10:15 AM 413,628 29se11aa.21883.2382.12.10.219 04/11/2012 10:15 AM 413,628 29se11aa.21883.2382.12.10.209

Where other tasks in the 29se11aa.21883.10980. & 29se11aa.21883.11798 groups are the normal 375,222 byte size.
____________
SETI@home classic workunits: 93,865 CPU time: 863,447 hours

Join the BP6/VP6 User Group today!

Richard Haselgrove
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8275
Credit: 44,939,496
RAC: 13,639
United Kingdom
Message 1217655 - Posted: 13 Apr 2012, 15:25:11 UTC

The ones I've found have all been shorties - 14-day deadlines, typical of VHAR.

In fact, they seem to be VVHAR - the one Lt linked has AR=55, I've just looked at an AR=57 here.

Profile HAL9000
Volunteer tester
Avatar
Send message
Joined: 11 Sep 99
Posts: 3567
Credit: 97,955,075
RAC: 79,022
United States
Message 1217660 - Posted: 13 Apr 2012, 15:50:44 UTC - in response to Message 1217655.
Last modified: 13 Apr 2012, 15:52:28 UTC

The ones I've found have all been shorties - 14-day deadlines, typical of VHAR.

In fact, they seem to be VVHAR - the one Lt linked has AR=55, I've just looked at an AR=57 here.

The current tasks & the one previous one I had are all 55.95.

I found one where my wing mate has successfully completed the task. So it looks like at least some of them will be OK.
http://setiathome.berkeley.edu/workunit.php?wuid=968984952

Unless this is one of those cases where it is an issue when using the opt apps. Looking at the 4 that Lint trap had. The stock app seems to have completed the tasks OK & the opt app ended up with the -131.
____________
SETI@home classic workunits: 93,865 CPU time: 863,447 hours

Join the BP6/VP6 User Group today!

Profile Lint trap
Send message
Joined: 30 May 03
Posts: 856
Credit: 24,550,260
RAC: 11,306
United States
Message 1217703 - Posted: 13 Apr 2012, 17:16:13 UTC - in response to Message 1217660.


Unless this is one of those cases where it is an issue when using the opt apps. Looking at the 4 that Lint trap had. The stock app seems to have completed the tasks OK & the opt app ended up with the -131.



By completed OK, you mean the output filesize didn't exceed the limit ?

If it's just a case of the Opt-apps being too wordy, perhaps Lunatics could check for that and somehow limit the output text to stay under 64KB, or, the limit could be changed ?


Lt

Josef W. Segur
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4134
Credit: 1,003,719
RAC: 231
United States
Message 1217706 - Posted: 13 Apr 2012, 17:23:36 UTC - in response to Message 1217655.

The ones I've found have all been shorties - 14-day deadlines, typical of VHAR.

In fact, they seem to be VVHAR - the one Lt linked has AR=55, I've just looked at an AR=57 here.

Hmm, for the one HAL9000 linked, the first two coordinate sets are

<coordinate_t> <time>2455834.2627723</time> <ra>15.211767558555</ra> <dec>18.111415397339</dec> </coordinate_t> <coordinate_t> <time>2455834.4264586</time> <ra>19.048461812708</ra> <dec>17.949663297908</dec> </coordinate_t>


Almost all the AR=56 is in that pair, they're nearly 4 hours apart in both time and ra.

The 368 coordinate sets make the header section 58562 bytes, so adding a few signals to that size header can easily make a result file over 65536.
Joe

Richard Haselgrove
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8275
Credit: 44,939,496
RAC: 13,639
United Kingdom
Message 1217716 - Posted: 13 Apr 2012, 17:44:49 UTC - in response to Message 1217703.


Unless this is one of those cases where it is an issue when using the opt apps. Looking at the 4 that Lint trap had. The stock app seems to have completed the tasks OK & the opt app ended up with the -131.

By completed OK, you mean the output filesize didn't exceed the limit ?

If it's just a case of the Opt-apps being too wordy, perhaps Lunatics could check for that and somehow limit the output text to stay under 64KB, or, the limit could be changed ?

No, it wouldn't help.

The wordy Lunatics stuff is in stderr_txt, which gets sent to the server by a different channel when you report the task.

The 64KB limit is on the uploaded result file which one would expect to be identical - that's the intention, anyway, otherwise the results won't validate.

Unless there's something subtle like the stock app producing a file with Unix-style line endings (CR only), and the Opti apps using Windows line endings (CR-LF). In a file with a lot of short lines, that can make a big difference.

Richard Haselgrove
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8275
Credit: 44,939,496
RAC: 13,639
United Kingdom
Message 1217719 - Posted: 13 Apr 2012, 17:50:12 UTC - in response to Message 1217706.

The ones I've found have all been shorties - 14-day deadlines, typical of VHAR.

In fact, they seem to be VVHAR - the one Lt linked has AR=55, I've just looked at an AR=57 here.

Hmm, for the one HAL9000 linked, the first two coordinate sets are

<coordinate_t> <time>2455834.2627723</time> <ra>15.211767558555</ra> <dec>18.111415397339</dec> </coordinate_t> <coordinate_t> <time>2455834.4264586</time> <ra>19.048461812708</ra> <dec>17.949663297908</dec> </coordinate_t>


Almost all the AR=56 is in that pair, they're nearly 4 hours apart in both time and ra.

The 368 coordinate sets make the header section 58562 bytes, so adding a few signals to that size header can easily make a result file over 65536.
Joe

Lt's first report, relating to a 404 KB data file, was only 2.5KB over the limit. So the second set I listed, with 392 KB data files, will almost certainly be OK, unless they find a large number of signals. But we're certainly on the margin.

Profile Lint trap
Send message
Joined: 30 May 03
Posts: 856
Credit: 24,550,260
RAC: 11,306
United States
Message 1217725 - Posted: 13 Apr 2012, 18:06:38 UTC - in response to Message 1217716.


Unless this is one of those cases where it is an issue when using the opt apps. Looking at the 4 that Lint trap had. The stock app seems to have completed the tasks OK & the opt app ended up with the -131.

By completed OK, you mean the output filesize didn't exceed the limit ?

If it's just a case of the Opt-apps being too wordy, perhaps Lunatics could check for that and somehow limit the output text to stay under 64KB, or, the limit could be changed ?

No, it wouldn't help.

The wordy Lunatics stuff is in stderr_txt, which gets sent to the server by a different channel when you report the task.

The 64KB limit is on the uploaded result file which one would expect to be identical - that's the intention, anyway, otherwise the results won't validate.

Unless there's something subtle like the stock app producing a file with Unix-style line endings (CR only), and the Opti apps using Windows line endings (CR-LF). In a file with a lot of short lines, that can make a big difference.



Thanks for re-explaining that! I always have trouble with the distinctions. It's clearer now.

Lt

AndrewM
Volunteer tester
Send message
Joined: 5 Jan 08
Posts: 361
Credit: 33,872,609
RAC: 5
Australia
Message 1217755 - Posted: 13 Apr 2012, 19:08:30 UTC

Some extra observations:

These finished with -131 errors:

29se11aa.21883.2382.12.10.68_0_0
29se11aa.21883.2382.12.10.65_1_0
29se11aa.21883.2382.12.10.43_0_0
29se11aa.14713.2382.7.10.160_0_0
29se11aa.14713.2382.7.10.143_0_0

This is in cache at 404Kb

29se11aa.29304.2382.9.10.230


Cheers
____________
AndrewM

Profile HAL9000
Volunteer tester
Avatar
Send message
Joined: 11 Sep 99
Posts: 3567
Credit: 97,955,075
RAC: 79,022
United States
Message 1217776 - Posted: 13 Apr 2012, 19:51:45 UTC - in response to Message 1217716.


Unless this is one of those cases where it is an issue when using the opt apps. Looking at the 4 that Lint trap had. The stock app seems to have completed the tasks OK & the opt app ended up with the -131.

By completed OK, you mean the output filesize didn't exceed the limit ?

If it's just a case of the Opt-apps being too wordy, perhaps Lunatics could check for that and somehow limit the output text to stay under 64KB, or, the limit could be changed ?

No, it wouldn't help.

The wordy Lunatics stuff is in stderr_txt, which gets sent to the server by a different channel when you report the task.

The 64KB limit is on the uploaded result file which one would expect to be identical - that's the intention, anyway, otherwise the results won't validate.

Unless there's something subtle like the stock app producing a file with Unix-style line endings (CR only), and the Opti apps using Windows line endings (CR-LF). In a file with a lot of short lines, that can make a big difference.

If it would help anything. I could setup a system to process the 7 files I saved with each app and then zip up the result files for someone to grab and look over.
____________
SETI@home classic workunits: 93,865 CPU time: 863,447 hours

Join the BP6/VP6 User Group today!

Profile Mark Wyzenbeek
Avatar
Send message
Joined: 28 Jun 99
Posts: 84
Credit: 1,520,034
RAC: 705
United States
Message 1217939 - Posted: 14 Apr 2012, 3:37:02 UTC - in response to Message 1217776.

I had a task end in error with a -131. A search brought me here. At least now I know what the error means. If it doesn't reoccur I won't worry about it. Thanks
____________
The Universe is not only stranger than you imagine, it's stranger than you can imagine.

SETI@home classic workunits 1,405 CPU time 57,318 hours

Josef W. Segur
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4134
Credit: 1,003,719
RAC: 231
United States
Message 1217974 - Posted: 14 Apr 2012, 4:38:52 UTC - in response to Message 1217776.

...
Unless there's something subtle like the stock app producing a file with Unix-style line endings (CR only), and the Opti apps using Windows line endings (CR-LF). In a file with a lot of short lines, that can make a big difference.

If it would help anything. I could setup a system to process the 7 files I saved with each app and then zip up the result files for someone to grab and look over.

I did do an offline comparison run with the 29se11aa.21883.2382.12.10.217 WU. Stock 6.03 produced a 66019 bytes result file, AK_v8b2 65758 bytes. IOW, both would have incurred a -131 error.

Both apps produce LF only line endings. The 261 byte difference is because 6.03 was built in 2008 with an older version of the workunit header structure so has a few more lines of header in the result file. That makes the observed cases where stock did not error but optimized did even more puzzling.

With the coordinates obviously wrong in some ways, the positions of any found signals wouldn't be known properly. IMO it is preferable to error out the WUs rather than assimilate bad results, but not a big deal since 99.9999...% of the signals in the science database are no more meaningful.
Joe

Profile Lint trap
Send message
Joined: 30 May 03
Posts: 856
Credit: 24,550,260
RAC: 11,306
United States
Message 1218549 - Posted: 15 Apr 2012, 2:21:07 UTC



Thanks to everyone for your investigative efforts! It's good to know nothing important is being lost.

Lt


Message boards : Number crunching : 64KB Output filesize limitation

Copyright © 2014 University of California