SETI server backing off downloads problem


log in

Advanced search

Questions and Answers : Web site : SETI server backing off downloads problem

Previous · 1 · 2 · 3 · Next
Author Message
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 26 May 99
Posts: 6819
Credit: 24,594,515
RAC: 26,645
United Kingdom
Message 1270795 - Posted: 12 Aug 2012, 12:11:57 UTC

No I suspect a lot of crunchers don't want "one at a time"

Obviously this is so much better, 27 tasks started.



____________


Today is life, the only life we're sure of. Make the most of today.

Profile ignorance is no excuse
Avatar
Send message
Joined: 4 Oct 00
Posts: 9529
Credit: 44,433,274
RAC: 0
Korea, North
Message 1270831 - Posted: 12 Aug 2012, 15:05:52 UTC - in response to Message 1270795.

As I explained years ago to a user that wanted 30 WU's downloading at a time, YOu are taking the space of 15 standard users. If everyone did what you do the servers would be overwhelmed.

Since most of us aren't running out of work as fast as we get it, it doesn't make much sense to overwhelm to transfer more than 2 at a time which is the standard rate. Getting all your transfers done at the same time doesn't help finish work faster. all it does is help clog an already overwhelmed pipeline.

For those that insist on flooding the servers, so be it. Nobody says you can't. However, it's pretty inconsiderate to all the others that want more work as well.
____________
In a rich man's house there is no place to spit but his face.
Diogenes Of Sinope

End terrorism by building a school

Andrzej Mackiewicz
Send message
Joined: 25 Apr 12
Posts: 29
Credit: 7,217,261
RAC: 27
Australia
Message 1270853 - Posted: 12 Aug 2012, 16:51:42 UTC - in response to Message 1270831.

Yes. What I'm hoping for is to allow SETI to set that max dl's setting in everyone's config by default, should that setting become per-project. That would at least temporarily relieve the strain on SETI servers until people change it for no good reason.

As you point out, if we could do this, nobody need change their setting above 1 as they can't finish calculating one work unit faster than they can download one work unit. Admittedly dialup users may not be able to download as fast as they can compute but then more connections certainly wouldn't even help those users..

Petition to change max downloads in BOINC to be per-project? ;)

Andrzej Mackiewicz
Send message
Joined: 25 Apr 12
Posts: 29
Credit: 7,217,261
RAC: 27
Australia
Message 1270855 - Posted: 12 Aug 2012, 16:52:11 UTC - in response to Message 1270795.

lol :)

Andrzej Mackiewicz
Send message
Joined: 25 Apr 12
Posts: 29
Credit: 7,217,261
RAC: 27
Australia
Message 1280219 - Posted: 5 Sep 2012, 11:05:51 UTC

Guess not enough people are interested to put it forward to the BOINC coders

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12259
Credit: 2,552,534
RAC: 773
Netherlands
Message 1280283 - Posted: 5 Sep 2012, 15:27:23 UTC - in response to Message 1280219.

No one else wants to be the messenger, no. So why don't you sign up to the BOINC development email list and ask for this yourself? Do write the whole request in the email, there's nothing more annoying to the developers than doing a request and pointing to a (big) thread on some forums somewhere.

The BOINC Dev email list requires registration in order to be able to post to it.
____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

Andrzej Mackiewicz
Send message
Joined: 25 Apr 12
Posts: 29
Credit: 7,217,261
RAC: 27
Australia
Message 1280309 - Posted: 5 Sep 2012, 16:37:04 UTC - in response to Message 1280283.

No, I was more thinking of a large list of users backing the idea.. But I appreciate your telling me to do it myself ^_^

Andrzej Mackiewicz
Send message
Joined: 25 Apr 12
Posts: 29
Credit: 7,217,261
RAC: 27
Australia
Message 1280406 - Posted: 5 Sep 2012, 21:05:42 UTC - in response to Message 1270678.

@BilBg

I just re-read this thread so I can draft an email to try and get this issue addressed and noticed two things:

1. You are asking SETI for up to 10 simultaneous downloads? Nice contributing to the problem with 5 times the potential simultaneous requests being sent to SETI servers for no benefit whatsoever mate.
2. The <max_file_xfers_per_project>1</max_file_xfers_per_project> you kindly pointed out (and I cleverly overlooked) is exactly what is needed- you will not have any reason to complain if you have it set to 1 download at a time..

Your CPU/GPU is very frigging unlikely to be able to crunch data faster than you can download the next work unit. Even if it has to wait for 1 minute while the next work unit is downloading, it still beats being told to back off for 1 hour+!

So yeah, my idea is already in practice on BOINC's end; we just need to tell/force people's max_file_xfers_per_project to 1. No need to email Dave ^_^

Akira
Send message
Joined: 11 Sep 12
Posts: 2
Credit: 68,797
RAC: 0
United States
Message 1282477 - Posted: 11 Sep 2012, 2:32:16 UTC - in response to Message 1280406.
Last modified: 11 Sep 2012, 2:32:47 UTC

I think what they are inquiring about, at least I am, is that why is it taking many minutes, in some cases hours, to download a few Kbytes, or Mbytes of information? It took me about 40min to download a 51KB size file before BOINC started downloading other files for SETI@HOME. Hours to download what looks like the CUDA executable files at roughly 10KB total?

I understand this funded by limited grants and/or donations as it does cost money to host/maintain/upgrade the bandwidth, software and hardware; but 10KB/s? That is roughly 128Kbps or 2, maybe 3 times as fast as dial-up. These files could be downloaded in mere seconds and the idle processing power of my computer could already be crunching the numbers. =)

I would have posted a screenshot but it does not appear we can upload a picture to the message boards but to only link the image from another website, which I do not have one to host the image. =)

Profile BilBg
Volunteer tester
Avatar
Send message
Joined: 27 May 07
Posts: 2570
Credit: 5,873,301
RAC: 2,516
Bulgaria
Message 1282479 - Posted: 11 Sep 2012, 2:42:44 UTC - in response to Message 1282477.


The line at SETI is 100 Mbps and is saturated almost all the time by ~ 200 000 computers asking for work:

http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=%2Frouter-interfaces%2Finr-250%2Fgigabitethernet2_3;view=Octets


____________



- ALF - "Find out what you don't do well ..... then don't do it!" :)

Akira
Send message
Joined: 11 Sep 12
Posts: 2
Credit: 68,797
RAC: 0
United States
Message 1282487 - Posted: 11 Sep 2012, 3:02:14 UTC - in response to Message 1282479.

Ahhhh. Yeah, That would do it. Wish I had the $ to donate a GB link or two. Of course that would fill up soon enough, I'm sure. =)

I remember the days when our ISP upgraded to its 4th T1 and we thought we would never have saturated the 3rd one. Then we got our first DS3... and then our 3rd... and so on. You never think you would exceed X bandwidth so why get Y. I was sad to see it sold off and parted but hey, such the life of a business. Someone waves enough $$ and the owner will sell!

We had our graph moved to internal network access after the first DS3 because you had those, like me =), that would email or call and complain that your 60% saturated! When are you going to increase your bandwidth? You are now 60.5% saturated... 61% saturated...!

Thanks for the link to the graph at SETI. It's always interesting to know how much bandwidth is being used for a project.

Andrzej Mackiewicz
Send message
Joined: 25 Apr 12
Posts: 29
Credit: 7,217,261
RAC: 27
Australia
Message 1282538 - Posted: 11 Sep 2012, 6:31:15 UTC - in response to Message 1282479.

Thanks for those numbers BilBg. Certainly made me think..

Do you know if the 200,000 is SETI's active member count or an approximated count of simultaneous users based on observed statistics?

If it's the approximate number of active members, this single download request per project idea could still help; if it's an approximation of how many users are simultaneously requesting work given that we have more than that many users, I think my idea can't help sadly, as that calculates to half a byte per second per user even if we could maintain connections to them all simultaneously...

CRAAAAPPP XD

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12259
Credit: 2,552,534
RAC: 773
Netherlands
Message 1282594 - Posted: 11 Sep 2012, 11:20:36 UTC - in response to Message 1282538.

Active users (individual user IDs that got credit): http://boincstats.com/en/stats/0/project/detail/user, around 142,000
Active hosts (computers that got granted credit): http://boincstats.com/en/stats/0/project/detail/host, around 212,000
____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

Andrzej Mackiewicz
Send message
Joined: 25 Apr 12
Posts: 29
Credit: 7,217,261
RAC: 27
Australia
Message 1283110 - Posted: 13 Sep 2012, 3:53:40 UTC - in response to Message 1282594.

Thanks for that mate.

212000 active hosts / 24 hours = approx. 8834 computers making requests per hour

100 Megabit SETI connection / 8834 average required connections = 1.5 KB/s download speed theoretically available per user

Everyone asking for 2-4+ downloads at a time from SETI is only hurting their chances of finishing a transfer. 4 simultaneous downloads utilising 1.5KB/s potentially available bandwidth in total will take forever and can result in your being told to Back Off due to there being bugger-all bandwidth available for you.

If everyone only used one download connection to SETI at any given time, the server should be able to cope with giving them their tiny share of bandwidth and will finish sending the work unit in 6 mins or so even at 1.5KB/s.

Those downloading 2-4+ WU's at a time from SETI are essentially cutting their own throats as 1.5KB/s is not hardly enough to support 4 simultaneous downloads.

Sound?

John McLeod VII
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 15 Jul 99
Posts: 24095
Credit: 517,832
RAC: 152
United States
Message 1285304 - Posted: 19 Sep 2012, 1:50:12 UTC - in response to Message 1283110.

Thanks for that mate.

212000 active hosts / 24 hours = approx. 8834 computers making requests per hour

100 Megabit SETI connection / 8834 average required connections = 1.5 KB/s download speed theoretically available per user

Everyone asking for 2-4+ downloads at a time from SETI is only hurting their chances of finishing a transfer. 4 simultaneous downloads utilising 1.5KB/s potentially available bandwidth in total will take forever and can result in your being told to Back Off due to there being bugger-all bandwidth available for you.

If everyone only used one download connection to SETI at any given time, the server should be able to cope with giving them their tiny share of bandwidth and will finish sending the work unit in 6 mins or so even at 1.5KB/s.

Those downloading 2-4+ WU's at a time from SETI are essentially cutting their own throats as 1.5KB/s is not hardly enough to support 4 simultaneous downloads.

Sound?

Not really. If I only download one at a time, that just makes them serial rather than parallel. The same amount of data has to come through the pipe in a day. In other words, I can take a minute total with 4 slots used, or 4 minutes with 1 slot used. The total amount of data is the same. The real problem is the amount of data that can be shoveled through the pipe in a day. Insisting that everyone download a single file at a time just moves the problem, and not not by much. It does not change the problem.
____________


BOINC WIKI

Andrzej Mackiewicz
Send message
Joined: 25 Apr 12
Posts: 29
Credit: 7,217,261
RAC: 27
Australia
Message 1285341 - Posted: 19 Sep 2012, 5:04:41 UTC - in response to Message 1285304.

I understand and agree; the total bandwidth available doesn't change. By the same token, if I am downloading 4 files with a potential available bandwidth of 1.5KB/s, it takes me WAY longer to get a usable work unit. That being the problem; we don't always have work ready to do.

There is no sense in anybody trying to download more than one file at a time when bandwidth is so scarce. Just like you say, the bandwidth available to you is the same for 1 or 4 transfers- though you want a complete work unit ASAP, not 4 partials.

Also it takes more CPU and a little more bandwidth to service 4 times the number of connections; it's not exactly the same load on the server. I submit that any positive change is better than no change at all, and everyone limiting themselves to 1 download at a time will do this.

John McLeod VII
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 15 Jul 99
Posts: 24095
Credit: 517,832
RAC: 152
United States
Message 1287211 - Posted: 23 Sep 2012, 20:19:56 UTC - in response to Message 1285341.

I understand and agree; the total bandwidth available doesn't change. By the same token, if I am downloading 4 files with a potential available bandwidth of 1.5KB/s, it takes me WAY longer to get a usable work unit. That being the problem; we don't always have work ready to do.

There is no sense in anybody trying to download more than one file at a time when bandwidth is so scarce. Just like you say, the bandwidth available to you is the same for 1 or 4 transfers- though you want a complete work unit ASAP, not 4 partials.

Also it takes more CPU and a little more bandwidth to service 4 times the number of connections; it's not exactly the same load on the server. I submit that any positive change is better than no change at all, and everyone limiting themselves to 1 download at a time will do this.

It actually makes no difference overall whether everyone loads 20 files at once or one file at once. It can actually make things worse instead of better.

(Just for the example, we will assume that the server is limited to 4 downloads at once. Each user wishes to download 5 files. Each subsequent user starts downloading files one time space apart. 5 users. Server limits transfers to 4 max)

Example

Time 0. First user starts downloading first file.
3 slots free.
Time 1. First user starts downloading second file.
Second user starts downloading first file.
2 slots free
Time 2. First user starts downloading third file.
Second user starts downloading second file.
Third user starts downloading third file.
1 slot free.
Time 3. First user starts downloading fourth file.
Second user starts downloading third file.
Third user starts downloading second file.
Fourth user starts downloading first file.
Time 4. First user starts downloading fifth file.
Second user starts downloading fourth file.
Third user starts downloading third file.
Fourth user starts downloading second file.
Fifth user blocked from downloading.
Time 5. Second user starts downloading fifth file.
Third user starts downloading fourth file.
Fourth user starts downloading third file.
Fifth user starts downloading first file.
Time 6. Third user starts downloading fifth file.
Fourth user starts downloading fourth file.
Fifth user starts downloading second file.
One slot free.
Time 7. Fourth user starts downloading fifth file.
Fifth user starts downloading third file.
2 slots free.
Time 8. Fifth user starts downloading fourth file.
3 slots free.
Time 9. Fifth user starts downloading fifth file.

Now for allowing 4 at once from the client.

Time 1. First user starts first file
First user starts second file
First user starts third file
First user starts fourth file
Time 2. First user starts fifth file
Second user starts first file
Second user starts second file
Second user starts third file
Time 3. Second user starts fourth file
Second user starts fifth file
Third user starts first file
Third user starts second file
Time 4. Third user starts third file
Third user starts fourth file
Third user starts fifth file
Fourth user starts first file
Time 5. Fourth user starts second file
Fourth user starts third file
Fourth user starts fourth file
Fourth user starts fifth file
Fifth user blocked from downloading
Time 6. Fifth user starts first file
Fifth user starts second file
Fifth user starts third file
Fifth user starts fourth file
Time 7. Fifth user starts fifth file
3 slots unused
Time 8. Not used
Time 9. Not used

Admittedly, if there is a constant need for more bandwidth than can be supplied, there will be no recovery, but it makes no difference at all to the server if each host starts one file upload at a time, or multiple files at a time. If only one is started at a time by each host, it merely means that the second file will be requested during the second time slot - when there are even more requests to be handled. If bandwidth is saturated, the only fix is more bandwidth, or smaller files. Some savings can be realized if shared files are downloaded to one external host and shared from there. However, the typical savings would be 50% as each file is shared between 2 hosts for download, and is not shared for upload.
____________


BOINC WIKI

Andrzej Mackiewicz
Send message
Joined: 25 Apr 12
Posts: 29
Credit: 7,217,261
RAC: 27
Australia
Message 1287316 - Posted: 24 Sep 2012, 4:48:15 UTC - in response to Message 1287211.

The server can always manage its bandwidth to ensure people aren't stalling however if it reaches its maximum simultaneous connections it is totally unable to service any more requests until someone is finished, which, when you consider the number of users and the number of potential simultaneous requests results in a DDOS-like effect where the server is essentially rendered non-communicable at times by the sheer number of connections, not the bandwidth being consumed strictly.

You will always get a usable work unit faster if you're using all available bandwidth for the one transfer; how is that going to "actually make things worse" exactly?

John McLeod VII
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 15 Jul 99
Posts: 24095
Credit: 517,832
RAC: 152
United States
Message 1287870 - Posted: 26 Sep 2012, 4:02:20 UTC - in response to Message 1287316.

The server can always manage its bandwidth to ensure people aren't stalling however if it reaches its maximum simultaneous connections it is totally unable to service any more requests until someone is finished, which, when you consider the number of users and the number of potential simultaneous requests results in a DDOS-like effect where the server is essentially rendered non-communicable at times by the sheer number of connections, not the bandwidth being consumed strictly.

You will always get a usable work unit faster if you're using all available bandwidth for the one transfer; how is that going to "actually make things worse" exactly?

Actually, you don't because of latency. While waiting for acknowledgement of a packet for file A, file B can certainly be downloading a packet. The point is that it makes no difference at all how many files each user attempts to download at a time as long as the server limits itself to somewhat less than the total bandwidth and fails to start files if it is saturated. This is something that the server needs to do, not the client.
____________


BOINC WIKI

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12259
Credit: 2,552,534
RAC: 773
Netherlands
Message 1287906 - Posted: 26 Sep 2012, 6:45:48 UTC - in response to Message 1287316.
Last modified: 26 Sep 2012, 6:46:01 UTC

You will always get a usable work unit faster if you're using all available bandwidth for the one transfer; how is that going to "actually make things worse" exactly?

In other words, Seti has to lock down right now, not allow anyone else to sign up, as their bandwidth is already over the max that the database can take at any one second.

So they'll have to whittle down the amount of computers to just under DDOS attacks and the maximum of 1.5KB/sec. So that means culling the flock to a maximum of 19,999 computers and then only send out Seti Enhanced work, not those pesky 8MB Astropulse, as those will limit the bandwidth used to just 2 computers at a time, and allowing only 2 computers is just silly, right?

;-)
____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

Previous · 1 · 2 · 3 · Next

Questions and Answers : Web site : SETI server backing off downloads problem

Copyright © 2014 University of California