I can't Get Boinc to upload


log in

Advanced search

Questions and Answers : Preferences : I can't Get Boinc to upload

Author Message
Ben
Send message
Joined: 10 Oct 07
Posts: 7
Credit: 11,711
RAC: 0
Canada
Message 914310 - Posted: 5 Jul 2009, 12:27:22 UTC
Last modified: 5 Jul 2009, 12:31:50 UTC

I have had to cancel completed work units because they were past their due date because I can't seem to upload.

I have a modem that has a serious fire wall.

I want to open a port for BOINC but I don't know the settings.

What is the port range that I should use and what is the host port?

Also is it a TCP port or a UDP port?

These are common error messages that i get


7/5/2009 5:32:58 AM SETI@home Temporarily failed upload of 18se08aa.14688.40263.15.8.41_0_0: HTTP error
7/5/2009 5:32:58 AM SETI@home Backing off 13 min 55 sec on upload of 18se08aa.14688.40263.15.8.41_0_0
7/5/2009 5:33:00 AM Internet access OK - project servers may be temporarily down.

Ben
Send message
Joined: 10 Oct 07
Posts: 7
Credit: 11,711
RAC: 0
Canada
Message 914311 - Posted: 5 Jul 2009, 12:30:10 UTC
Last modified: 5 Jul 2009, 12:31:40 UTC

These are common error messages that i get


7/5/2009 5:32:58 AM SETI@home Temporarily failed upload of 18se08aa.14688.40263.15.8.41_0_0: HTTP error
7/5/2009 5:32:58 AM SETI@home Backing off 13 min 55 sec on upload of 18se08aa.14688.40263.15.8.41_0_0
7/5/2009 5:33:00 AM Internet access OK - project servers may be temporarily down.

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12280
Credit: 2,567,190
RAC: 657
Netherlands
Message 914327 - Posted: 5 Jul 2009, 14:19:05 UTC - in response to Message 914311.

We all get these errors at this time, as the bandwidth into Seti is used up by lots of downloads. Give it time, eventually everything will work.

You shouldn't need to open any ports in your modem/router's firewall. BOINC uses TCP port 80 and 443 for communications with the internet.
____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

Ben
Send message
Joined: 10 Oct 07
Posts: 7
Credit: 11,711
RAC: 0
Canada
Message 914329 - Posted: 5 Jul 2009, 14:31:07 UTC - in response to Message 914327.

Thank you I will endeavor to endure my backlog. I just want to help as much as my tiny computer can.

Profile Mr. KevvyProject donor
Volunteer tester
Avatar
Send message
Joined: 15 May 99
Posts: 688
Credit: 71,737,372
RAC: 78,222
Canada
Message 914340 - Posted: 5 Jul 2009, 15:15:16 UTC - in response to Message 914311.

This problem with the upload server seems to happen every couple of days. Eventually the WUs get through; however it takes so many retries that on a fast/multiprocessor PC, queued uploads accumulate faster than they are uploaded so the client doesn't request more WU's and eventually runs out of them.

I merely manually retry the upload repeatedly until the queue is gone. Some of them take ten or more retries unfortunately.

The problem seems to be that the confirmation of completed upload never gets from the server to the client; I see many of the uploads get to 100% but they still fail and are retried. When this happens, with hundreds of clients repeating their uploads over and over, the server gets swamped and then new uploads won't even start properly. Hoping that it will finally get fixed eventually!
____________
“Never doubt that a small group of thoughtful, committed citizens can change the world; indeed, it's the only thing that ever has.”
--- Margaret Mead

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12280
Credit: 2,567,190
RAC: 657
Netherlands
Message 914346 - Posted: 5 Jul 2009, 15:36:22 UTC - in response to Message 914340.

This problem with the upload server seems to happen every couple of days....

When this happens, with hundreds of clients repeating their uploads over and over, the server gets swamped and then new uploads won't even start properly. Hoping that it will finally get fixed eventually!

It's a problem with bandwidth, more than a problem with the server itself. The server and the database on it can easily handle the amount it is getting in. It's the line into the server that's the problem, it is only 100Mbit and over this line all downloads, uploads and reports go out of and in to Seti.

As the graph shows, the bandwidth has been pegged at almost max for many hours now. And until Seti can get a new line in/out of the building, this won't change much.

So don't blame it on the server. It's not at fault here.
____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

Profile Mr. KevvyProject donor
Volunteer tester
Avatar
Send message
Joined: 15 May 99
Posts: 688
Credit: 71,737,372
RAC: 78,222
Canada
Message 914350 - Posted: 5 Jul 2009, 15:41:22 UTC - in response to Message 914346.
Last modified: 5 Jul 2009, 15:42:58 UTC

As the graph shows, the bandwidth has been pegged at almost max for many hours now. And until Seti can get a new line in/out of the building, this won't change much.


But why is the bandwidth maxed? Because the clients are retrying their uploads over and over, they go to 100% and then fail because they don't get the confirmation, so they retry.

Again, it took ten tries or more when I manually retried the uploads before they would complete. And much of the time, rather than failing at 0% they uploaded completely to 100% and then failed. So with hundreds of clients doing this simultaneously with the same WU's to be uploaded, all the bandwidth is being wasted on repeated failed uploads. Seen this happen dozens of times. <<shrug>>
____________
“Never doubt that a small group of thoughtful, committed citizens can change the world; indeed, it's the only thing that ever has.”
--- Margaret Mead

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12280
Credit: 2,567,190
RAC: 657
Netherlands
Message 914354 - Posted: 5 Jul 2009, 15:56:47 UTC - in response to Message 914350.

But why is the bandwidth maxed?

Because there are a lot of downloads going out at the same time, over the same pipeline. Some of them are Astropulse (AP), which are 8MB tasks, uncompressed data. You only need 2 simultaneous AP tasks on download to clog up a 100Mbit line. (2 x 8MB x 8bit = 128Mbit) for as long as that download takes; so then it depends on what speed hosts can download with how long such a clog up takes.

You don't have to try to upload, do retries and all that. BOINC will take care of that all by itself and when there's a lull in downloads, some uploads can get through.
____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

Ben
Send message
Joined: 10 Oct 07
Posts: 7
Credit: 11,711
RAC: 0
Canada
Message 914510 - Posted: 5 Jul 2009, 22:53:23 UTC - in response to Message 914346.

It's a problem with bandwidth, more than a problem with the server itself. The server and the database on it can easily handle the amount it is getting in. It's the line into the server that's the problem, it is only 100Mbit and over this line all downloads, uploads and reports go out of and in to Seti.

So when do you think you will be able to get a better line into the building? 100mb/s seems a little slow to me. I can get more then that on my PC although I use my computer at home with minimal highspeed internet, surely you can afford to have a hardware upgrade. perhaps a second line, or even have a T1 line installled just for your project? Although I do not have any disposable income I am sure that you could use this issue to raise funds from those who do.

John McLeod VII
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 15 Jul 99
Posts: 24232
Credit: 519,558
RAC: 57
United States
Message 914519 - Posted: 5 Jul 2009, 23:33:50 UTC - in response to Message 914510.

It's a problem with bandwidth, more than a problem with the server itself. The server and the database on it can easily handle the amount it is getting in. It's the line into the server that's the problem, it is only 100Mbit and over this line all downloads, uploads and reports go out of and in to Seti.

So when do you think you will be able to get a better line into the building? 100mb/s seems a little slow to me. I can get more then that on my PC although I use my computer at home with minimal highspeed internet, surely you can afford to have a hardware upgrade. perhaps a second line, or even have a T1 line installled just for your project? Although I do not have any disposable income I am sure that you could use this issue to raise funds from those who do.

Actually, there are bottlenecks everywhere. Not too long ago, the worst bottle necks were on the servers.
____________


BOINC WIKI

OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13564
Credit: 29,762,423
RAC: 16,596
United States
Message 914582 - Posted: 6 Jul 2009, 3:17:58 UTC - in response to Message 914510.

So when do you think you will be able to get a better line into the building? 100mb/s seems a little slow to me. I can get more then that on my PC although I use my computer at home with minimal highspeed internet, surely you can afford to have a hardware upgrade. perhaps a second line, or even have a T1 line installled just for your project? Although I do not have any disposable income I am sure that you could use this issue to raise funds from those who do.


A T1 line is only 1.5 Megabits. The 100 Megabit line is far faster than T1. I seriously doubt your High Speed Internet at home is 100 Megabits as even the fastest Verizon FIOS service is only at 50 Megabits.

Since SETI has to go through Berkeley, and Berkeley doesn't want to pay for the upgrade, and SETI has little or no government funding, they don't really have the money to pay for a faster pipe (last estimated at over $100,000 to install).

Personally, I'm not certain a faster pipe will make the problems go away. The servers have to be able to handle the load, and the servers are dropping connections because there's just too many hosts looking for and returning work.
____________

John McLeod VII
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 15 Jul 99
Posts: 24232
Credit: 519,558
RAC: 57
United States
Message 914601 - Posted: 6 Jul 2009, 4:00:03 UTC - in response to Message 914582.

So when do you think you will be able to get a better line into the building? 100mb/s seems a little slow to me. I can get more then that on my PC although I use my computer at home with minimal highspeed internet, surely you can afford to have a hardware upgrade. perhaps a second line, or even have a T1 line installled just for your project? Although I do not have any disposable income I am sure that you could use this issue to raise funds from those who do.


A T1 line is only 1.5 Megabits. The 100 Megabit line is far faster than T1. I seriously doubt your High Speed Internet at home is 100 Megabits as even the fastest Verizon FIOS service is only at 50 Megabits.

Since SETI has to go through Berkeley, and Berkeley doesn't want to pay for the upgrade, and SETI has little or no government funding, they don't really have the money to pay for a faster pipe (last estimated at over $100,000 to install).

Personally, I'm not certain a faster pipe will make the problems go away. The servers have to be able to handle the load, and the servers are dropping connections because there's just too many hosts looking for and returning work.

Having a Gigabit pipe will definitely push the bottlenecks back onto the servers.

I know that I have 100Mb internally, but only 3Mb externally (not shared).
____________


BOINC WIKI

OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13564
Credit: 29,762,423
RAC: 16,596
United States
Message 914605 - Posted: 6 Jul 2009, 4:15:29 UTC - in response to Message 914601.

So when do you think you will be able to get a better line into the building? 100mb/s seems a little slow to me. I can get more then that on my PC although I use my computer at home with minimal highspeed internet, surely you can afford to have a hardware upgrade. perhaps a second line, or even have a T1 line installled just for your project? Although I do not have any disposable income I am sure that you could use this issue to raise funds from those who do.


A T1 line is only 1.5 Megabits. The 100 Megabit line is far faster than T1. I seriously doubt your High Speed Internet at home is 100 Megabits as even the fastest Verizon FIOS service is only at 50 Megabits.

Since SETI has to go through Berkeley, and Berkeley doesn't want to pay for the upgrade, and SETI has little or no government funding, they don't really have the money to pay for a faster pipe (last estimated at over $100,000 to install).

Personally, I'm not certain a faster pipe will make the problems go away. The servers have to be able to handle the load, and the servers are dropping connections because there's just too many hosts looking for and returning work.

Having a Gigabit pipe will definitely push the bottlenecks back onto the servers.

I know that I have 100Mb internally, but only 3Mb externally (not shared).


Agreed. And if the servers can't handle all the connections simultaneously, then its going to create a new bottleneck.

Me? I'm running mostly 1000Mb (or 1Gigabit) speeds internally, and my Comcast HSI is at 22Megabits.
____________

MuylaerB
Avatar
Send message
Joined: 27 May 99
Posts: 3
Credit: 1,263,619
RAC: 1,180
Belgium
Message 914826 - Posted: 6 Jul 2009, 18:06:48 UTC - in response to Message 914354.

A simple fix for this is sending the data compressed...
____________

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12280
Credit: 2,567,190
RAC: 657
Netherlands
Message 914846 - Posted: 6 Jul 2009, 18:26:10 UTC - in response to Message 914826.

Perhaps so, but that has been tried already. I asked about that in another thread a week ago, and got this answer:

Richard Haselgrove wrote:
I don't think that would help much, Jord.

AP data files are in binary format, and contain essentially random data - which is, by definition, almost incompressible.

I've just tried a random file preserved from earlier tests. The original 8,196 KB data file came down to 8,125 KB with WinZip, and 7,145 KB with 7-zip. Given the added processing load on the servers, I can't see it making a significant difference.


So as you can see, there is not much to compress.
____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13564
Credit: 29,762,423
RAC: 16,596
United States
Message 914913 - Posted: 6 Jul 2009, 20:26:57 UTC - in response to Message 914846.

Not to mention that we have no idea what SETI's server's CPU utilization looks like. Compression may not be much for a home computer that stands mostly idle, but a server's CPU utilization that's primarily maxed out (or close to it) may be stealing CPU cycles away from other important tasks.
____________

John McLeod VII
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 15 Jul 99
Posts: 24232
Credit: 519,558
RAC: 57
United States
Message 914976 - Posted: 6 Jul 2009, 22:16:22 UTC - in response to Message 914913.

Not to mention that we have no idea what SETI's server's CPU utilization looks like. Compression may not be much for a home computer that stands mostly idle, but a server's CPU utilization that's primarily maxed out (or close to it) may be stealing CPU cycles away from other important tasks.

Actually, based on the tech news, we do have a fair idea of what the server CPU utilization looks like - nearly maxed out.
____________


BOINC WIKI

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12280
Credit: 2,567,190
RAC: 657
Netherlands
Message 914995 - Posted: 6 Jul 2009, 22:51:24 UTC - in response to Message 914913.
Last modified: 6 Jul 2009, 23:13:49 UTC

Oh, I don't know, that could perhaps be solved by putting a dedicated server between the splitter and the upload server, but since the APs don't compress much it's really not much use.

Would compressing the result files matter, though?
I don't know how big AP results are (don't run them), but The Multibeam results are 26-35KB and mostly text, so easily compressible. I'm just not sure if it matters if the file is 35KB or 7KB compressed, that difference is negligible on most high-speed connections.

And then there's that niggle about the sched_request_*.xml files. If they're large, due to a large amount of cache you got, those are not getting through easily enough either. It does matter if you're trying to reach the scheduler with a 7KB file instead of a 1MB file.

Edit: Ah funny... From Matt's post:
From what Eric tells me compressing workunits only helps multibeam, and only by about 20%. Almost not worth considering, since that will get us 5-10 Mbits back, and we need something like 50.
____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

Questions and Answers : Preferences : I can't Get Boinc to upload

Copyright © 2014 University of California