download_fanout and cc_config

Message boards : Number crunching : download_fanout and cc_config
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile Keith T.
Volunteer tester
Avatar

Send message
Joined: 23 Aug 99
Posts: 962
Credit: 537,293
RAC: 9
United Kingdom
Message 567004 - Posted: 14 May 2007, 9:11:37 UTC
Last modified: 14 May 2007, 9:17:27 UTC

If you have stalled downloads that have shown up in your results page but are stuck in the Transfers queue, and are familiar with BOINC and managing files at the OS level, you may like to try this.

1. Create a cc_config.xml file http://boinc.berkeley.edu/client_msgs.php in your BOINC directory. Add the tag <file_xfer_debug>1</file_xfer_debug> to cc_config.xml in the <log_flags> section.

2. Go to the Advanced menu of BOINC Manager and click Read config file.

3. Go to the Transfers tab of BOINC Manager, select one of your stalled downloads and click Retry Now. Go to the Messages tab and highlight the line that contains [---] [file_xfer_debug] PERS_FILE_XFER::start_xfer(): URL: http://setiboincdata.ssl.berkeley.edu/sah/download_fanout/. Copy this into Notepad.

4. Open http://boinc2.ssl.berkeley.edu/sah/download_fanout/ using your favourite web-browser. Scolll down until you find the matching sub-directory e.g 73, c3, 3ae, 3d3. Open the sub-directory. Search within the page for the matching file name. (You may have to do a partial search as the file names appear truncated in the browser view, but this should narrow down your search to a few lines.)

5. Right click the appropriate file, and download to a suitable location on your computer. (I chose "\\Program Files\\BOINC\\projects".)

6. Move the file to "\\Program Files\\BOINC\\projects\\setiathome.berkeley.edu" using Windows Explorer (or equivalent); you will probably be prompted to over-write a 0 byte file with one of 354 KB.

7. Go back to the Transfers tab of BOINC Manager, select the same task and click Retry Now again.

8. Back to Messages tab and you should now see "[SETI@home] File xxx exists already, skipping download", the file should also go from the Transfers queue.

9. Run the task normally.

10. Repeat steps 3-8 as necessary.

It worked for me. HTH.


Sir Arthur C Clarke 1917-2008
ID: 567004 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 567015 - Posted: 14 May 2007, 10:02:24 UTC - in response to Message 567004.  
Last modified: 14 May 2007, 10:16:57 UTC

Thanks Keith,
Tried it differently which worked for me. Instead of using the cc_config.xml approach:

1. Obtain file/foldername from the stuck workunit in the client_state.xml
2. visit the http://boinc2.ssl.berkeley.edu/sah/download_fanout/
3. download the correct file (locating by the sequence you described), overwriting the zero length one
4. "retry now"

Have 2 workunits crunching now :D [ Sadly my other machine has no downloads allocated, oh well, ones enough 'til they get back and kick those servers :D ]

"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 567015 · Report as offensive
Profile Keith T.
Volunteer tester
Avatar

Send message
Joined: 23 Aug 99
Posts: 962
Credit: 537,293
RAC: 9
United Kingdom
Message 567024 - Posted: 14 May 2007, 10:33:18 UTC - in response to Message 567015.  

Thanks Keith,
Tried it differently which worked for me. Instead of using the cc_config.xml approach:

1. Obtain file/foldername from the stuck workunit in the client_state.xml
2. visit the http://boinc2.ssl.berkeley.edu/sah/download_fanout/
3. download the correct file (locating by the sequence you described), overwriting the zero length one
4. "retry now"

Have 2 workunits crunching now :D [ Sadly my other machine has no downloads allocated, oh well, ones enough 'til they get back and kick those servers :D ]


Glad it worked, your method is probaly simpler for those that do'nt already have a cc_config file. It also involves less load on the server as you only need to re-try once for each file instead of twice.

I have also set SETI and SETI Beta to No new tasks for a few days until network traffic quietens down.

While you are waiting for more SETI tasks, why not add Rosetta or another project. I like Rosetta because it has a quorum of 1 so you get instant credit for valid results!, the graphics are very good too.
Sir Arthur C Clarke 1917-2008
ID: 567024 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 567032 - Posted: 14 May 2007, 10:58:49 UTC

Well done Keith - splendid workround, worked a treat.

But a word of warning - it can become horribly addictive!

I had a few stuck, so I downloaded them manually and started crunching SETI again. As soon as the download queue, was empty, the beast contacted the scheduler again and got allocated some more. So I downloaded a few of those as well: all with the shortest deadlines (4+ days). Which on my rig go immediately into EDF. And some of them were -9 overflows, so now they're in the upload queue instead. So I need to download some more....

Ah well, back to the asylum, LOL.
ID: 567032 · Report as offensive
Profile Julian Ellis
Volunteer tester

Send message
Joined: 12 Dec 04
Posts: 22
Credit: 365,438
RAC: 0
United Kingdom
Message 567039 - Posted: 14 May 2007, 11:44:25 UTC

It is a lot easier if you just use proxy 128.32.18.173 this will allow download but not uploads, please turn off the proxy when downloads complete so others can use it.

ID: 567039 · Report as offensive
Conrad Human
Volunteer tester

Send message
Joined: 17 Nov 00
Posts: 67
Credit: 2,009,224
RAC: 0
South Africa
Message 567071 - Posted: 14 May 2007, 12:27:39 UTC

i have done a bulk replace of
http://setiboincdata.ssl.berkeley.edu/sah/download_fanout/
to
http://boinc2.ssl.berkeley.edu/sah/download_fanout/
in
client_state.xml

bussy downloading WU's

ID: 567071 · Report as offensive
JasonTsao

Send message
Joined: 12 Mar 01
Posts: 2
Credit: 2,466,348
RAC: 0
Taiwan
Message 567076 - Posted: 14 May 2007, 12:41:34 UTC - in response to Message 567015.  

Thanks Keith,
Tried it differently which worked for me. Instead of using the cc_config.xml approach:

1. Obtain file/foldername from the stuck workunit in the client_state.xml
2. visit the http://boinc2.ssl.berkeley.edu/sah/download_fanout/
3. download the correct file (locating by the sequence you described), overwriting the zero length one
4. "retry now"

Have 2 workunits crunching now :D [ Sadly my other machine has no downloads allocated, oh well, ones enough 'til they get back and kick those servers :D ]

Thanks Jason and Keith.
I can get wu to work.
ID: 567076 · Report as offensive
gomeyer
Volunteer tester

Send message
Joined: 21 May 99
Posts: 488
Credit: 50,370,425
RAC: 0
United States
Message 567097 - Posted: 14 May 2007, 13:10:27 UTC - in response to Message 567071.  

i have done a bulk replace of
http://setiboincdata.ssl.berkeley.edu/sah/download_fanout/
to
http://boinc2.ssl.berkeley.edu/sah/download_fanout/
in
client_state.xml


Just a suggestion, but don't forget to return it to normal once all available work has been downloaded.
ID: 567097 · Report as offensive
Greg Beach
Avatar

Send message
Joined: 7 Jun 99
Posts: 23
Credit: 4,978,313
RAC: 0
Canada
Message 567100 - Posted: 14 May 2007, 13:17:24 UTC - in response to Message 567039.  

It is a lot easier if you just use proxy 128.32.18.173 this will allow download but not uploads, please turn off the proxy when downloads complete so others can use it.



Thanks for the tip. Worked like a charm and I've turned off the proxy.
ID: 567100 · Report as offensive
Profile Frosted
Avatar

Send message
Joined: 11 Jul 99
Posts: 83
Credit: 3,898,641
RAC: 0
Canada
Message 567103 - Posted: 14 May 2007, 13:20:10 UTC - in response to Message 567039.  

It is a lot easier if you just use proxy 128.32.18.173 this will allow download but not uploads, please turn off the proxy when downloads complete so others can use it.


WOW! that's awsome!!
ID: 567103 · Report as offensive
Profile Y & J
Volunteer tester

Send message
Joined: 14 Nov 01
Posts: 15
Credit: 215,639
RAC: 0
United States
Message 567104 - Posted: 14 May 2007, 13:25:43 UTC - in response to Message 567004.  

If you have stalled downloads that have shown up in your results page but are stuck in the Transfers queue, and are familiar with BOINC and managing files at the OS level, you may like to try this.


Thanks Keith

This worked out great. I have never looked or even played with Boinc codes, but your instructions worked the first time. I usually just let Boinc crunch away. I was able to download all five wu's on SETI, but SETI beta has the scheduler disabled so I can not clean that wu up. I will have to start to look into Boinc coding more now.


[color= blue][u]SETI@home classic workunits = 5,906 with CPU time of 60,377 hours[/u][/color]
ID: 567104 · Report as offensive
Profile JerWA

Send message
Joined: 3 Apr 99
Posts: 13
Credit: 4,262,442
RAC: 0
United States
Message 567173 - Posted: 14 May 2007, 15:01:59 UTC - in response to Message 567039.  
Last modified: 14 May 2007, 15:03:36 UTC

It is a lot easier if you just use proxy 128.32.18.173 this will allow download but not uploads, please turn off the proxy when downloads complete so others can use it.

Worked a charm to get the WUs stuck in the download queue clear (very fast transfers too). Thanks, helps keep things going while we're waiting for someone to smack the servers around.

Now if only there was a way to clear the 16 WUs stuck in upload.
ID: 567173 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 567179 - Posted: 14 May 2007, 15:09:06 UTC
Last modified: 14 May 2007, 15:09:35 UTC

The proxy server setting work to get my downloads. I also noticed very fast transfers. How does this work? I have never tried or had to use a proxy server in Boinc before.
And is there such a thing as a proxy server that will handle the uploads?
Thanx for the work around.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 567179 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 567191 - Posted: 14 May 2007, 15:23:51 UTC
Last modified: 14 May 2007, 15:24:36 UTC

I think we ought to be a bit cautious about the proxy. The IP address given resolves to galileo.SSL.Berkeley.EDU: a few days ago Tony (Astro) posted here:
they've blocked the galileo loophole. That addy went through the campus network which they paid dearly for(in cash). Not to mention that it'd do NO good this time as all outgoing work need to be written to thumper so it knows who's got what. there's no sneaking around this time.

It looks as if they unblocked the galileo loophole - deliberately or otherwise.

If what Tony says about the cost of the Campus network is still true, then anyone who used that proxy owes them (another) donation - and please make doubly sure that you remove the proxy information as soon as possible after use.
ID: 567191 · Report as offensive
Astro
Volunteer tester
Avatar

Send message
Joined: 16 Apr 02
Posts: 8026
Credit: 600,015
RAC: 0
Message 567201 - Posted: 14 May 2007, 15:36:29 UTC
Last modified: 14 May 2007, 15:37:44 UTC

I know people snuck around the normal path using galileo one time previously. It used to go through the much more expensive campus network. They saw the traffic and killed it. Looks like the loophole is back open. I view the previous use of this as selfish and self serving. Having NO respect for the project and it's needs($$$). Just because it works, doesn't make it the right thing to do. This only applies if it's still on the campus network (which it used to be, but I don't know that now). They're required to use the campus network and have been using it for low traffic things like these message boards.

I await word from them, before solidifying my position on this issue. I won't be using it. Until I know it's not costing them an arm and a leg.
ID: 567201 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 567212 - Posted: 14 May 2007, 15:50:24 UTC

The final part of my tracert looks like this:

23 177 ms 177 ms 177 ms calren-trcust.plalca01.transitrail.net [137.164.131.254]
24 178 ms 185 ms 177 ms ucb--svl-dc1-egm.cenic.net [137.164.23.66]
25 178 ms 177 ms 176 ms g3-17.inr-202-reccev.Berkeley.EDU [128.32.0.35]
26 177 ms 177 ms 177 ms g6-2.inr-230-spr.Berkeley.EDU [128.32.255.114]
27 177 ms 177 ms 177 ms ssl-f1.SSL.Berkeley.EDU [169.229.0.218]
28 179 ms 178 ms 178 ms galileo.SSL.Berkeley.EDU [128.32.18.173]

inr-230 Sproul looks, according to the cricket graphs, very much like a primary campus network. I think we should stay off it.

Anyway, it's all academic (sorry) now. The cricket graphs also show that the proper route has just started up again, and with a vengeance - Cur: 55.21 Mbits/sec.
ID: 567212 · Report as offensive
KB7RZF
Volunteer tester
Avatar

Send message
Joined: 15 Aug 99
Posts: 9549
Credit: 3,308,926
RAC: 2
United States
Message 567265 - Posted: 14 May 2007, 17:16:54 UTC - in response to Message 567236.  

It is a lot easier if you just use proxy 128.32.18.173 this will allow download but not uploads, please turn off the proxy when downloads complete so others can use it.



Thanks, works like a charm (not that fast, but enough)

How ever any donations is out of the question, it's quite enough I participate in this thankless drivel for free ...

Did you expect to be compensated for volunteering? ROFLMAO Ok.
ID: 567265 · Report as offensive
Profile criton
Avatar

Send message
Joined: 28 Feb 00
Posts: 131
Credit: 13,351,000
RAC: 2
United Kingdom
Message 567270 - Posted: 14 May 2007, 17:22:33 UTC

thanks for the work round julian with the proxy server
worked a treat 4 computers working again crunching for setti.


greenfinger
ID: 567270 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 567300 - Posted: 14 May 2007, 18:04:29 UTC - in response to Message 567071.  

i have done a bulk replace of
http://setiboincdata.ssl.berkeley.edu/sah/download_fanout/
to
http://boinc2.ssl.berkeley.edu/sah/download_fanout/
in
client_state.xml

bussy downloading WU's

Excellent! You identified what was failing (the redirect) and found the correct workaround method.
                                                               Joe
ID: 567300 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 567461 - Posted: 14 May 2007, 23:10:49 UTC

Oh yeah this.. Sorry but this doesn't work anymore.

- Matt
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 567461 · Report as offensive
1 · 2 · Next

Message boards : Number crunching : download_fanout and cc_config


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.