Panic Mode On (77) Server Problems?

Message boards : Number crunching : Panic Mode On (77) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 22 · Next

AuthorMessage
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1290626 - Posted: 3 Oct 2012, 8:02:57 UTC

What with servers? Can't upload...
ID: 1290626 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1290627 - Posted: 3 Oct 2012, 8:08:02 UTC - in response to Message 1290626.  
Last modified: 3 Oct 2012, 8:10:40 UTC

What with servers? Can't upload...

Been that way after the weekly ouage for the last 3 weeks, this one just seems worse than the last couple.
And even if you can upload everything the most likely response for a request for work from the Scheduler is "Project has no taks available", "No tasks sent" or "Timeout reached".


EDIT- and if you do happen to get some work, it's all shorties. So you'll process it faster than you can return it, let alone get new work. If that were possible.
Grant
Darwin NT
ID: 1290627 · Report as offensive
Profile Mike Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 34253
Credit: 79,922,639
RAC: 80
Germany
Message 1290636 - Posted: 3 Oct 2012, 8:59:01 UTC

No upload problems here.



With each crime and every kindness we birth our future.
ID: 1290636 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 1290638 - Posted: 3 Oct 2012, 9:01:39 UTC - in response to Message 1290636.  

No upload problems here.

Consider yourself lucky then. ;)

Cheers.
ID: 1290638 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1290640 - Posted: 3 Oct 2012, 9:08:36 UTC

120 downloads and 110+ uploads not moving even with an application of "retry"

Have set NNT on all 3 machines, no point adding to the problems.
ID: 1290640 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1290642 - Posted: 3 Oct 2012, 9:14:55 UTC - in response to Message 1290640.  
Last modified: 3 Oct 2012, 9:15:19 UTC

120 downloads and 110+ uploads not moving even with an application of "retry"

I expect to be out of CPU work on both systems when i get up in the morning not being here to hit retry endlessly overnight.
And when i finally do clear the backlog, i can't get enough work to even top up the cache, it just keeps getting smaller.


Have set NNT on all 3 machines, no point adding to the problems.

Won't make any difference- all those backedup uploads will stop any Scheduler requests from occuring unless a WU is near it's deadline.
Grant
Darwin NT
ID: 1290642 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1290645 - Posted: 3 Oct 2012, 9:19:53 UTC - in response to Message 1290642.  
Last modified: 3 Oct 2012, 9:20:24 UTC

120 downloads and 110+ uploads not moving even with an application of "retry"

I expect to be out of CPU work on both systems when i get up in the morning not being here to hit retry endlessly overnight.
And when i finally do clear the backlog, i can't get enough work to even top up the cache, it just keeps getting smaller.


Have set NNT on all 3 machines, no point adding to the problems.

Won't make any difference- all those backedup uploads will stop any Scheduler requests from occuring unless a WU is near it's deadline.


Yes but if and when the log jam breaks I can report and upload without bothering to ask for more.
ID: 1290645 · Report as offensive
JohnDK Crowdfunding Project Donor*Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 28 May 00
Posts: 1222
Credit: 451,243,443
RAC: 1,127
Denmark
Message 1290673 - Posted: 3 Oct 2012, 10:40:43 UTC - in response to Message 1290638.  

No upload problems here.

Consider yourself lucky then. ;)

Cheers.

Yes seems Mike is the only one without any problems.
ID: 1290673 · Report as offensive
DesO

Send message
Joined: 2 Feb 12
Posts: 144
Credit: 2,624,617
RAC: 0
United Kingdom
Message 1290681 - Posted: 3 Oct 2012, 11:00:48 UTC

Hi

Its seems nearly certain the folks at SETI have no idea a problem exists.

To expidite solution I have contacted a SETI sponsor (Planetary Society) with the aim of getting attention drawn to the situation ;

"Hi

I’m one of the many tens of thousands of people who contribute time and energy crunching data for SETI at home and work.

For over a week there have been severe upload and download problems effecting every participant and the
situation is becoming worse. People are running out of work and only a reduced number of the smallest work packages are being processed.

No one from SETI has contacted anyone via the SETI forum or through announcement.

There is a worrying silence and some of us think they have no idea a problem exists.

There is no direct communication line to SETI for members so those who give a dam can do nothing at this time.

I have contacted your organisation, as a sponsor of SETI, to ask if you have communication lines to SETI and then if so please use them to
begin a handling for the current situation and also to provide some contact for the folks who are wondering where the situation is going.

Many Thanks Des"
ID: 1290681 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1290686 - Posted: 3 Oct 2012, 11:14:15 UTC
Last modified: 3 Oct 2012, 11:16:35 UTC

Hardly Planetary Society can do anything...
The only possibility is to pay for gigabyte ethernet link for SETI...

Meantime cant' upload to SETI beta too :(
ID: 1290686 · Report as offensive
DesO

Send message
Joined: 2 Feb 12
Posts: 144
Credit: 2,624,617
RAC: 0
United Kingdom
Message 1290691 - Posted: 3 Oct 2012, 11:24:01 UTC - in response to Message 1290686.  

Hardly Planetary Society can do anything...
The only possibility is to pay for gigabyte ethernet link for SETI...

Meantime cant' upload to SETI beta too :(



I contacted the PS because as they are SETI sponsors so they will have ability to contact SETI directly to alert them to the current problem. I dont think anyone in the forum has a direct line to SETI.

D
ID: 1290691 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1290706 - Posted: 3 Oct 2012, 11:53:09 UTC - in response to Message 1290691.  

Hardly Planetary Society can do anything...
The only possibility is to pay for gigabyte ethernet link for SETI...

Meantime cant' upload to SETI beta too :(



I contacted the PS because as they are SETI sponsors so they will have ability to contact SETI directly to alert them to the current problem. I dont think anyone in the forum has a direct line to SETI.

D

I believe several people do indeed have ways of contacting the LAB, I have left a message with one and am waiting a reply.
ID: 1290706 · Report as offensive
Profile shizaru
Volunteer tester
Avatar

Send message
Joined: 14 Jun 04
Posts: 1130
Credit: 1,967,904
RAC: 0
Greece
Message 1290709 - Posted: 3 Oct 2012, 12:13:17 UTC - in response to Message 1290691.  

I dont think anyone in the forum has a direct line to SETI.


I'm pretty sure a few people do, but I don't think that's the point. I'm guessing you're thinking Google Inc. when instead you should be thinking Google Garage Days:) But even THAT is not the point.

Point is the last time I ran out of work to crunch was probably 18 months ago. And that was because the servers were shut down. Probably Feb. 2011?

Anyway!
If you find yourself running out of work on a laptop (which I see is your highest cruncher) then I'm sure we can help you get your settings and Boinc versions sorted out so it doesn't happen again. Just create a thread here in number crunching. Simple:)
ID: 1290709 · Report as offensive
DesO

Send message
Joined: 2 Feb 12
Posts: 144
Credit: 2,624,617
RAC: 0
United Kingdom
Message 1290710 - Posted: 3 Oct 2012, 12:19:55 UTC - in response to Message 1290706.  

Glad to hear some communication is going on.

Best D
ID: 1290710 · Report as offensive
Profile Donald L. Johnson
Avatar

Send message
Joined: 5 Aug 02
Posts: 8240
Credit: 14,654,533
RAC: 20
United States
Message 1290737 - Posted: 3 Oct 2012, 14:22:32 UTC - in response to Message 1290691.  
Last modified: 3 Oct 2012, 14:30:18 UTC

Hardly Planetary Society can do anything...
The only possibility is to pay for gigabyte ethernet link for SETI...

Meantime cant' upload to SETI beta too :(


I contacted the PS because as they are SETI sponsors so they will have ability to contact SETI directly to alert them to the current problem. I dont think anyone in the forum has a direct line to SETI.

D

You are incorrect. Anyone can contact the folks who run this SETI@Home (not simply SETI) project, directly, by Private Message, through their postings in these Forums. Many of the folks with "Volunteer Tester" and "Volunteer Developer" tags under their names also have direct email contacts with Eric and Matt and Jeff. They ARE aware of the problems with uploads, downloads, and scheduler requests.

A BIG part of the problem is the bottleneck at the 100 MBs data link to the servers. This IS being worked, but as has been said repeatedly, the fix involves organisational politics at the University of California. Another big part of the problem is the "shortie storm" we have been going through the past 2-3 weeks. We ALWAYS have up/download problems during shortie storms, for the reasons given in other posts, but this is one of the longest storms I've seen, and because of the continuing improvement of the computing power of the participants, one of the most severe.

This SETI@Home project does NOT recieve major funding from the Planetary Society (or any other pro-Space group), they are more involved with the SETI Institute and the Allen Telescope Array. This project gets much of its funding from donations from the participants. It operates on a shoe-string budget, and the staff are all part-time, sharing their time and efforts with other projects at the Space Studies Lab.

Nothing happens quickly here, but as Bernie said, eventually it gets fixed. In the meantime, patience is not just a virtue, it is a requirement.
Donald
Infernal Optimist / Submariner, retired
ID: 1290737 · Report as offensive
Profile James Sotherden
Avatar

Send message
Joined: 16 May 99
Posts: 10436
Credit: 110,373,059
RAC: 54
United States
Message 1290753 - Posted: 3 Oct 2012, 14:49:31 UTC

Last week I couldnt buy a download. This week I cant sell an upload.

I havent seen this long of a shorty storm either.

We will get work when we get it.
[/quote]

Old James
ID: 1290753 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1290767 - Posted: 3 Oct 2012, 15:08:28 UTC

Donald, whilst I do not doubt you. The tone of Matt's post yesterday did not suggest to me he was aware of any problems at all. Which again I agree is surprising as normally someone at the lab knows as soon, and sometimes before, we do.

Unless of course this is just an overload of the system because everything IS working.
ID: 1290767 · Report as offensive
Profile Slavac
Volunteer tester
Avatar

Send message
Joined: 27 Apr 11
Posts: 1932
Credit: 17,952,639
RAC: 0
United States
Message 1290795 - Posted: 3 Oct 2012, 15:58:33 UTC - in response to Message 1290786.  

Patience folks, patience.

I'm working on a few new bits of hardware that will help out our downloads sticking issues. We should be launching these shortly I hope once I get the specs approved.

There are several of us 'behind the scenes' types who also crunch and know what issues we're facing. We're trying to address and fix these as soon as possible but due to a lack of manpower in the lab, things are necessarily slow going.

One piece of hardware I'm hoping to get will be a load balancer which will more evenly distribute tasks to our download and upload servers. Currently as people have noticed, you will ping off of a dead or overloaded server which constantly mucks up the works. In the near future I'm hoping we can implement a very beefy balancer with proper software and hardware which will alleviate this issue significantly.

If you really want to help us fix these issues, keep reporting your performance and consider donating to the project in whatever capacity you're comfortable with.

In short, patience. We've switched over one server's tasks (mostly) to George and have switched from Apache (which has caused us a load of issues) to Nginx. Add the time spent switching all of our processes over to a new switch, dealing with a crashy server and the push to V7, you can see where our small staff is overwhelmed. Our DL/UL issues will be sorted soon.


Executive Director GPU Users Group Inc. -
brad@gpuug.org
ID: 1290795 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1290805 - Posted: 3 Oct 2012, 16:20:27 UTC - in response to Message 1290795.  

Patience folks, patience.

I'm working on a few new bits of hardware that will help out our downloads sticking issues. We should be launching these shortly I hope once I get the specs approved.

There are several of us 'behind the scenes' types who also crunch and know what issues we're facing. We're trying to address and fix these as soon as possible but due to a lack of manpower in the lab, things are necessarily slow going.

One piece of hardware I'm hoping to get will be a load balancer which will more evenly distribute tasks to our download and upload servers. Currently as people have noticed, you will ping off of a dead or overloaded server which constantly mucks up the works. In the near future I'm hoping we can implement a very beefy balancer with proper software and hardware which will alleviate this issue significantly.

If you really want to help us fix these issues, keep reporting your performance and consider donating to the project in whatever capacity you're comfortable with.

In short, patience. We've switched over one server's tasks (mostly) to George and have switched from Apache (which has caused us a load of issues) to Nginx. Add the time spent switching all of our processes over to a new switch, dealing with a crashy server and the push to V7, you can see where our small staff is overwhelmed. Our DL/UL issues will be sorted soon.

Great work on the download servers - they seem to be running much better today, with nginx.

But what goes down, must come back up again, and is seems Bruno is having difficulty keeping up...

03-Oct-2012 16:23:35 [SETI@home] [http] [ID#45620] Info: Trying 208.68.240.16...
03-Oct-2012 16:23:56 [SETI@home] [http] [ID#45620] Info: Timed out
03-Oct-2012 16:23:56 [SETI@home] [http] [ID#45620] Info: Failed connect to setiboincdata.ssl.berkeley.edu:80; No error
03-Oct-2012 16:23:56 [SETI@home] [http] [ID#45620] Info: Closing connection #1
03-Oct-2012 16:23:56 [SETI@home] [http] HTTP error: Couldn't connect to server

03-Oct-2012 16:27:43 [SETI@home] [http] [ID#45635] Info: Trying 208.68.240.16...
03-Oct-2012 16:28:04 [SETI@home] [http] [ID#45635] Info: Timed out
03-Oct-2012 16:28:04 [SETI@home] [http] [ID#45635] Info: Failed connect to setiboincdata.ssl.berkeley.edu:80; No error
03-Oct-2012 16:28:04 [SETI@home] [http] [ID#45635] Info: Closing connection #0
03-Oct-2012 16:28:04 [SETI@home] [http] HTTP error: Couldn't connect to server

03-Oct-2012 16:28:06 [SETI@home] [http] [ID#45636] Info: Trying 208.68.240.16...
03-Oct-2012 16:28:27 [SETI@home] [http] [ID#45636] Info: Timed out
03-Oct-2012 16:28:27 [SETI@home] [http] [ID#45636] Info: Failed connect to setiboincdata.ssl.berkeley.edu:80; No error
03-Oct-2012 16:28:27 [SETI@home] [http] [ID#45636] Info: Closing connection #0
03-Oct-2012 16:28:27 [SETI@home] [http] HTTP error: Couldn't connect to server

03-Oct-2012 16:29:51 [SETI@home] [http] [ID#45644] Info: Trying 208.68.240.16...
03-Oct-2012 16:30:13 [SETI@home] [http] [ID#45644] Info: Timed out
03-Oct-2012 16:30:13 [SETI@home] [http] [ID#45644] Info: Failed connect to setiboincdata.ssl.berkeley.edu:80; No error
03-Oct-2012 16:30:13 [SETI@home] [http] [ID#45644] Info: Closing connection #0
03-Oct-2012 16:30:13 [SETI@home] [http] HTTP error: Couldn't connect to server

03-Oct-2012 16:31:40 [SETI@home] [http] [ID#45650] Info: Trying 208.68.240.16...
03-Oct-2012 16:31:40 [SETI@home] [http] [ID#45650] Info: Connected to setiboincdata.ssl.berkeley.edu (208.68.240.16) port 80 (#1)
03-Oct-2012 16:31:40 [SETI@home] [http] [ID#45650] Info: Connected to setiboincdata.ssl.berkeley.edu (208.68.240.16) port 80 (#1)
03-Oct-2012 16:31:40 [SETI@home] [http] [ID#45650] Sent header to server: POST /sah_cgi/file_upload_handler HTTP/1.1
03-Oct-2012 16:31:40 [SETI@home] [http] [ID#45650] Sent header to server: User-Agent: BOINC client (windows_intelx86 7.0.36)
03-Oct-2012 16:31:40 [SETI@home] [http] [ID#45650] Sent header to server: Host: setiboincdata.ssl.berkeley.edu
03-Oct-2012 16:31:40 [SETI@home] [http] [ID#45650] Sent header to server: Accept: */*
03-Oct-2012 16:31:40 [SETI@home] [http] [ID#45650] Sent header to server: Accept-Encoding: deflate, gzip
03-Oct-2012 16:31:40 [SETI@home] [http] [ID#45650] Sent header to server: Content-Type: application/x-www-form-urlencoded
03-Oct-2012 16:31:40 [SETI@home] [http] [ID#45650] Sent header to server: Content-Length: 285
03-Oct-2012 16:31:40 [SETI@home] [http] [ID#45650] Sent header to server:
03-Oct-2012 16:32:27 [SETI@home] [http] [ID#45650] Info: Recv failure: Connection was reset
03-Oct-2012 16:32:27 [SETI@home] [http] [ID#45650] Info: Closing connection #1
03-Oct-2012 16:32:27 [SETI@home] [http] HTTP error: Failure when receiving data from the peer
03-Oct-2012 16:32:28 [SETI@home] Temporarily failed upload of 25jl12ab.14581.6611.6.10.139_0_0: transient HTTP error

03-Oct-2012 16:32:29 [SETI@home] [http] [ID#45654] Info: Trying 208.68.240.16...
03-Oct-2012 16:32:50 [SETI@home] [http] [ID#45654] Info: Timed out
03-Oct-2012 16:32:50 [SETI@home] [http] [ID#45654] Info: Failed connect to setiboincdata.ssl.berkeley.edu:80; No error
03-Oct-2012 16:32:50 [SETI@home] [http] [ID#45654] Info: Closing connection #0
ID: 1290805 · Report as offensive
tbret
Volunteer tester
Avatar

Send message
Joined: 28 May 99
Posts: 3380
Credit: 296,162,071
RAC: 40
United States
Message 1290814 - Posted: 3 Oct 2012, 16:37:55 UTC - in response to Message 1290795.  



Patience folks, patience.



Have you ever been a passenger in a car and noticed an ugly situation developing that you aren't sure the driver has seen?

It's much, much easier to relax if the driver just mutters, "I see 'em."

Thank you for reporting that the lab is aware.
ID: 1290814 · Report as offensive
Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 22 · Next

Message boards : Number crunching : Panic Mode On (77) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.