Panic Mode On (8) Server problems

Message boards : Number crunching : Panic Mode On (8) Server problems
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 15 · Next

AuthorMessage
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 797663 - Posted: 14 Aug 2008, 6:03:56 UTC


For a while there i thought the download storm was easing, but it was just a small lull & it's back to full bore. At least the uploads have dropped away significantly.
Grant
Darwin NT
ID: 797663 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 797671 - Posted: 14 Aug 2008, 6:26:39 UTC - in response to Message 797663.  


For a while there i thought the download storm was easing, but it was just a small lull & it's back to full bore. At least the uploads have dropped away significantly.

I am getting both up and downloads through......albeit after a few retries due to the bandwidth being saturated....

There is now some work built up in the ready to send cache, and the splitters are running at a good pace.
I everything holds together this should calm down nicely by morning.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 797671 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 797680 - Posted: 14 Aug 2008, 7:02:56 UTC - in response to Message 797671.  
Last modified: 14 Aug 2008, 7:27:17 UTC

I everything holds together this should calm down nicely by morning.

Hmm, might be a bit longer than that.
Seem to be quite a few short Work Units in the mix, combined with lots of empty caches, combined with some people probably upping the size of their cache.
Could be quite a few hours yet before it settles down (it's been 12 hours so far...).



EDIT- make that a lot of short Work Units. My first few allocations were mostly "normal" run time Work Units (60min or so) with a few shorties thrown in.
But the last few allocations (will be quite a while before they actually download from the looks of things) are mostly shorties with only a few normal run time Work Units in the mix.
Grant
Darwin NT
ID: 797680 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 797682 - Posted: 14 Aug 2008, 7:10:13 UTC - in response to Message 797680.  

I everything holds together this should calm down nicely by morning.

Hmm, might be a bit longer than that.
Seem to be quite a few short Work Units in the mix, combined with lots of empty caches, combined with some people probably upping the size of their cache.
Could be quite a few hours yet before it settles down (it's been 12 hours so far...).

Yeah, I have noticed the itty bitty WUs in the mix of what the Frozen Penny has been downloading......but it still has been getting some longer WUs too.......

I think the folks who may be upping their caches are a very small percentage of users.....but even if folks are not upping their caches, there are many that still have to be refilled after the spat of no work being available, so I guess I may be being optimistic.

At least things are flowing at the moment.

"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 797682 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 797715 - Posted: 14 Aug 2008, 8:24:22 UTC


Some random thoughts;
Hopefully there won't be too many more short Work Units to come.
Results in Progress was up to around 3.5 million. It did drop below 2 million.
It's taken over 13 hours to get that up to 2.5 million; so that'd be another day & a bit at the present rate to get things back to normal.
Bring on the longer Work Units...
Grant
Darwin NT
ID: 797715 · Report as offensive
Profile littlegreenmanfrommars
Volunteer tester
Avatar

Send message
Joined: 28 Jan 06
Posts: 1410
Credit: 934,158
RAC: 0
Australia
Message 797741 - Posted: 14 Aug 2008, 10:23:09 UTC
Last modified: 14 Aug 2008, 10:24:09 UTC

All settling down nicely here.
The laptop had some struggles up and downloading, but only remaining issue is export of XML to the stats sites. I reckon that should happen overnight.

PANIC OFF, I'd say
ID: 797741 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 797757 - Posted: 14 Aug 2008, 12:06:26 UTC - in response to Message 797741.  

PANIC OFF, I'd say

Nah, not untill the network traffic drops off to more normal levels for a few hours (at least).
Let the panic continue.

Grant
Darwin NT
ID: 797757 · Report as offensive
Profile Andy Lee Robinson
Avatar

Send message
Joined: 8 Dec 05
Posts: 630
Credit: 59,973,836
RAC: 0
Hungary
Message 797761 - Posted: 14 Aug 2008, 12:12:48 UTC - in response to Message 797741.  

All settling down nicely here.
The laptop had some struggles up and downloading, but only remaining issue is export of XML to the stats sites. I reckon that should happen overnight.

PANIC OFF, I'd say


Still a struggle here.. it's embarrassing..
Much of the bandwidth is used by failed transfers, so only a few percent actually get to completion.

So for what should be a 300k download becomes a 1Mb download per WU.

Any outages with the bandwidth requirements so precariously balanced will cause another cascade of failed transfers and longer and longer recovery time.

If they can't get another 100Mb line, then I vote they should firewall off parts of the net for a while to allow traffic to stabilise without carrying the burden of so many retries.

Perhaps even/odd ip addresses allowed in turn would help.


ID: 797761 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 797773 - Posted: 14 Aug 2008, 12:49:36 UTC - in response to Message 797761.  
Last modified: 14 Aug 2008, 12:51:22 UTC

Much of the bandwidth is used by failed transfers, so only a few percent actually get to completion.

So for what should be a 300k download becomes a 1Mb download per WU.

Having a look at my transfer queue there were a few where it would start to transfer & then stop & then start from scratch again when it next tried. But they would have been only a couple of percent of my total downloads at most & very few of them got more than 15% before timing out.

Though they may need to take a look at things as far as the time out on the download is concerend.
Most will atttempt to download for about 2 minutes before timing out. Yet some will only attempt to download for 4 seconds, others for only one second, before timing out. With this heavy traffic often it's been taking about 5-10 seconds for a download just to start & often it's only at 1-2kB/s & gradually picks up pace as is goes (although some never get much faster than 2kB/s).
Grant
Darwin NT
ID: 797773 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 65746
Credit: 55,293,173
RAC: 49
United States
Message 797801 - Posted: 14 Aug 2008, 14:47:07 UTC

Is anybody seeing this:
8/14/2008 7:36:19 AM|SETI@home|[file_xfer] Started upload of file 17ap08ae.451.17659.3.8.26_0_0
8/14/2008 7:36:41 AM||Project communication failed: attempting access to reference site
8/14/2008 7:36:41 AM|SETI@home|[file_xfer] Temporarily failed upload of 17ap08ae.451.17659.3.8.26_0_0: connect() failed
8/14/2008 7:36:41 AM|SETI@home|Backing off 2 min 27 sec on upload of file 17ap08ae.451.17659.3.8.26_0_0
8/14/2008 7:36:43 AM||Access to reference site succeeded - project servers may be temporarily down.
8/14/2008 7:39:10 AM|SETI@home|[file_xfer] Started upload of file 17ap08ae.451.17659.3.8.26_0_0
8/14/2008 7:39:32 AM||Project communication failed: attempting access to reference site
8/14/2008 7:39:32 AM|SETI@home|[file_xfer] Temporarily failed upload of 17ap08ae.451.17659.3.8.26_0_0: connect() failed
8/14/2008 7:39:32 AM|SETI@home|Backing off 6 min 21 sec on upload of file 17ap08ae.451.17659.3.8.26_0_0
8/14/2008 7:39:33 AM||Access to reference site succeeded - project servers may be temporarily down.
8/14/2008 7:41:42 AM|SETI@home|[file_xfer] Started upload of file 14mr08aa.20841.19704.6.8.241_0_0
8/14/2008 7:42:04 AM||Project communication failed: attempting access to reference site
8/14/2008 7:42:04 AM|SETI@home|[file_xfer] Temporarily failed upload of 14mr08aa.20841.19704.6.8.241_0_0: connect() failed

Hopefully the Outage today will fix this today as I can't upload a thing.
The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's
ID: 797801 · Report as offensive
Profile dnolan
Avatar

Send message
Joined: 30 Aug 01
Posts: 1228
Credit: 47,779,411
RAC: 32
United States
Message 797804 - Posted: 14 Aug 2008, 14:55:59 UTC - in response to Message 797801.  
Last modified: 14 Aug 2008, 15:04:05 UTC

Is anybody seeing this:
8/14/2008 7:41:42 AM|SETI@home|[file_xfer] Started upload of file 14mr08aa.20841.19704.6.8.241_0_0
8/14/2008 7:42:04 AM||Project communication failed: attempting access to reference site
8/14/2008 7:42:04 AM|SETI@home|[file_xfer] Temporarily failed upload of 14mr08aa.20841.19704.6.8.241_0_0: connect() failed

Hopefully the Outage today will fix this today as I can't upload a thing.


Yup, I'm seeing it, too.

-Dave

[Edit] Oh, and what outage? Was there an announcement somewhere that there's going to be an outage today?
ID: 797804 · Report as offensive
Profile Logan
Volunteer tester
Avatar

Send message
Joined: 26 Jan 07
Posts: 743
Credit: 918,353
RAC: 0
Spain
Message 797805 - Posted: 14 Aug 2008, 15:03:02 UTC - in response to Message 797804.  

Is anybody seeing this:
8/14/2008 7:41:42 AM|SETI@home|[file_xfer] Started upload of file 14mr08aa.20841.19704.6.8.241_0_0
8/14/2008 7:42:04 AM||Project communication failed: attempting access to reference site
8/14/2008 7:42:04 AM|SETI@home|[file_xfer] Temporarily failed upload of 14mr08aa.20841.19704.6.8.241_0_0: connect() failed

Hopefully the Outage today will fix this today as I can't upload a thing.


Yup, I'm seeing it, too.

-Dave



The same here...
Logan.

BOINC FAQ Service (Ahora, también disponible en Español/Now available in Spanish)
ID: 797805 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 65746
Credit: 55,293,173
RAC: 49
United States
Message 797809 - Posted: 14 Aug 2008, 15:06:10 UTC - in response to Message 797804.  

Is anybody seeing this:
8/14/2008 7:41:42 AM|SETI@home|[file_xfer] Started upload of file 14mr08aa.20841.19704.6.8.241_0_0
8/14/2008 7:42:04 AM||Project communication failed: attempting access to reference site
8/14/2008 7:42:04 AM|SETI@home|[file_xfer] Temporarily failed upload of 14mr08aa.20841.19704.6.8.241_0_0: connect() failed

Hopefully the Outage today will fix this today as I can't upload a thing.


Yup, I'm seeing it, too.

-Dave

[Edit] Oh, and what outage? Was there an announcement somewhere that there's going to be an outage today?

It's early, So oops! My concentration slipped a bit.
The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's
ID: 797809 · Report as offensive
Profile dnolan
Avatar

Send message
Joined: 30 Aug 01
Posts: 1228
Credit: 47,779,411
RAC: 32
United States
Message 797811 - Posted: 14 Aug 2008, 15:07:44 UTC - in response to Message 797809.  


It's early, So oops! My concentration slipped a bit.


No problem, just wasn't sure if I'd missed something....

-Dave
ID: 797811 · Report as offensive
Profile Blurf
Volunteer tester

Send message
Joined: 2 Sep 06
Posts: 8962
Credit: 12,678,685
RAC: 0
United States
Message 797825 - Posted: 14 Aug 2008, 15:31:42 UTC

8/14-11:30am EST-downloaded a bunch of new WU's-can't upload. Already emailed Eric


ID: 797825 · Report as offensive
Profile arkayn
Volunteer tester
Avatar

Send message
Joined: 14 May 99
Posts: 4438
Credit: 55,006,323
RAC: 0
United States
Message 797861 - Posted: 14 Aug 2008, 16:46:52 UTC

All of my upload on the iMac have went through now, of course it did not help that my internet connection died last night.

All the lights were on the modem indicating a connection but I had to restart the modem to get a connection.

ID: 797861 · Report as offensive
Profile [B^S] madmac
Volunteer tester
Avatar

Send message
Joined: 9 Feb 04
Posts: 1175
Credit: 4,754,897
RAC: 0
United Kingdom
Message 797870 - Posted: 14 Aug 2008, 17:03:04 UTC

I am having trouble uploading two wus except the ones before and after have uploaded, it must be my timing just hitting berkerleys server at the wrong time. Well I will just sit and wait for them to upload, it is mainly http errors and connect errors. Even had trouble with the scheduler, got project servers may be temporaily down
ID: 797870 · Report as offensive
Profile Logan
Volunteer tester
Avatar

Send message
Joined: 26 Jan 07
Posts: 743
Credit: 918,353
RAC: 0
Spain
Message 797874 - Posted: 14 Aug 2008, 17:11:23 UTC

Well, I only have now 11 wu's pending of upload.

The thing is running little by little. (a few minutes ago I had 40 or more...)


Best regards.
Logan.

BOINC FAQ Service (Ahora, también disponible en Español/Now available in Spanish)
ID: 797874 · Report as offensive
Profile Geek@Play
Volunteer tester
Avatar

Send message
Joined: 31 Jul 01
Posts: 2467
Credit: 86,146,931
RAC: 0
United States
Message 797880 - Posted: 14 Aug 2008, 17:22:17 UTC - in response to Message 797874.  

Well, I only have now 11 wu's pending of upload.

The thing is running little by little. (a few minutes ago I had 40 or more...)


Best regards.


Yes......my uploads are starting to trickle in to Berkeley also.

Boinc....Boinc....Boinc....Boinc....
ID: 797880 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 797893 - Posted: 14 Aug 2008, 17:53:13 UTC


Downloads appear to be going through on the 1st attempt in most cases (10-20kB/s), and some of my uploads are starting to trickle though.

At the present rate of travel, about 12 more hours & the number of results In Progress will back to where it was before the outage occured. Then the traffic should finally drop down to more normal levels.
Hopefully.
Grant
Darwin NT
ID: 797893 · Report as offensive
Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 15 · Next

Message boards : Number crunching : Panic Mode On (8) Server problems


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.