Panic Mode On (77) Server Problems?

Message boards : Number crunching : Panic Mode On (77) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · 13 . . . 22 · Next

AuthorMessage
Profile ivan
Volunteer tester
Avatar

Send message
Joined: 5 Mar 01
Posts: 783
Credit: 348,560,338
RAC: 223
United Kingdom
Message 1290863 - Posted: 3 Oct 2012, 18:45:02 UTC - in response to Message 1290805.  

Great work on the download servers - they seem to be running much better today, with nginx.

About the time you posted that, I was having great difficulties with uploads, but downloads were very slick (given that the particular machine is currently only getting 11 WUs per request). A few hours later, things are sticking both ways...
ID: 1290863 · Report as offensive
Profile Area 51
Avatar

Send message
Joined: 31 Jan 04
Posts: 965
Credit: 42,193,520
RAC: 0
United Kingdom
Message 1290864 - Posted: 3 Oct 2012, 18:45:04 UTC

I seem to get bogged down downloading - and uploading! I get 2,000 shorties, churn through them in quite short order, and spend the next 4 days trying to upload the results, and of course, I have to upload & report before I can anything new. One thing I have noticed though. When something goes up/down the pipe - it flies - really flies. There isn't even time for a progress bar during the transmission.

ID: 1290864 · Report as offensive
Profile Slavac
Volunteer tester
Avatar

Send message
Joined: 27 Apr 11
Posts: 1932
Credit: 17,952,639
RAC: 0
United States
Message 1290888 - Posted: 3 Oct 2012, 19:44:28 UTC - in response to Message 1290864.  

The plan right now, pending specs, is building a dedicated upload and download server soon. This one will be specifically slated for nothing but replacing our two remaining old servers. Combine that with a load balancer, the new switch, George and the jbod array, we should be heading in the right direction.

Now if only I had a large stack of money for more bandwidth. One day maybe.


Executive Director GPU Users Group Inc. -
brad@gpuug.org
ID: 1290888 · Report as offensive
MikeN

Send message
Joined: 24 Jan 11
Posts: 319
Credit: 64,719,409
RAC: 85
United Kingdom
Message 1290921 - Posted: 3 Oct 2012, 21:03:28 UTC

Well uploads have suddenly picked up dramatically. Cricket graph shows them doubling over the last hourand mine have gone through. However, schedular now hard to contact and I have started getting large numbers of timed out no response errors (>50 so far). Looks like the guys might have turned off 'resend ghost units' to free up some bandwdth or memory storage.
ID: 1290921 · Report as offensive
Profile ivan
Volunteer tester
Avatar

Send message
Joined: 5 Mar 01
Posts: 783
Credit: 348,560,338
RAC: 223
United Kingdom
Message 1290922 - Posted: 3 Oct 2012, 21:05:48 UTC - in response to Message 1290888.  

Now if only I had a large stack of money for more bandwidth. One day maybe.

I had an idle thought -- I don't remember exactly what the difficulty is in getting a 1 Gbps link down to the campus boundary, but I was wondering if there were a parallel unused "dark fibre" to the existing 100 Mbps link that could be channel-bonded to it to give 200 Mbps. "We" (the UK LCG community) made heavy use of such technology with multiple 1 Gbps links in our data centres until a recent Government windfall enabled most of us to upgrade to 10 Gbps links...
ID: 1290922 · Report as offensive
Profile Mike Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 34253
Credit: 79,922,639
RAC: 80
Germany
Message 1290931 - Posted: 3 Oct 2012, 21:37:53 UTC - in response to Message 1290922.  
Last modified: 3 Oct 2012, 21:38:30 UTC

Now if only I had a large stack of money for more bandwidth. One day maybe.

I had an idle thought -- I don't remember exactly what the difficulty is in getting a 1 Gbps link down to the campus boundary, but I was wondering if there were a parallel unused "dark fibre" to the existing 100 Mbps link that could be channel-bonded to it to give 200 Mbps. "We" (the UK LCG community) made heavy use of such technology with multiple 1 Gbps links in our data centres until a recent Government windfall enabled most of us to upgrade to 10 Gbps links...


Eric mentioned once biggest part is political IIRC.


With each crime and every kindness we birth our future.
ID: 1290931 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1290934 - Posted: 3 Oct 2012, 21:43:11 UTC - in response to Message 1290691.  

I contacted the PS because as they are SETI sponsors so they will have ability to contact SETI directly to alert them to the current problem. I dont think anyone in the forum has a direct line to SETI.

You don't appear to understand.
The Plantetary Socieity has no relationship whatsoever with Seti@Home.
Niether does the Seti Institute.

Seti@home is a project being run in the Space Science Laboratory & the Universtiy of Berkeley.
It has no direct affiliations with any other Seti organisations.

Grant
Darwin NT
ID: 1290934 · Report as offensive
Kevin Benfield

Send message
Joined: 29 Dec 03
Posts: 39
Credit: 30,085,439
RAC: 0
United Kingdom
Message 1290936 - Posted: 3 Oct 2012, 21:48:19 UTC - in response to Message 1289368.  

Only getting GPU units, not getting any at all for CPU, been like this for a couple of days.
Is anyone getting CPU units ?
ID: 1290936 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1290937 - Posted: 3 Oct 2012, 21:51:11 UTC - in response to Message 1290888.  
Last modified: 3 Oct 2012, 21:56:37 UTC

The plan right now, pending specs, is building a dedicated upload and download server soon. This one will be specifically slated for nothing but replacing our two remaining old servers. Combine that with a load balancer, the new switch, George and the jbod array, we should be heading in the right direction.


Will this also help with the Scheduler issues?
"Project has no tasks available" & "No tasks sent" have been common responses to work requests for a long time now. But over the last few weeks "Timeout was reached" has become very common, often 4 in 5 resposes to work requests.
And now that i've been able to upload all that backlogged work that is the only response i've been getting on one of my machines as i try to report 75 tasks & get new work. My other machine has been getting some work, but it's mostly "No tasks sent" with the odd "Project has no tasks available".

EDIT- oh, i forgot the "Couldn't connect to server" error that occasionally (but more & more frequently) pops up when trying to report or request new work.
Grant
Darwin NT
ID: 1290937 · Report as offensive
tbret
Volunteer tester
Avatar

Send message
Joined: 28 May 99
Posts: 3380
Credit: 296,162,071
RAC: 40
United States
Message 1290938 - Posted: 3 Oct 2012, 21:51:46 UTC - in response to Message 1290934.  
Last modified: 3 Oct 2012, 21:52:38 UTC



The Plantetary Socieity has no relationship whatsoever with Seti@Home.
Niether does the Seti Institute.





You can certainly understand the confusion, then, since The Planetary Society gets top billing:

SETI Sponsor Page
ID: 1290938 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1290939 - Posted: 3 Oct 2012, 21:51:59 UTC - in response to Message 1290936.  

Only getting GPU units, not getting any at all for CPU, been like this for a couple of days.
Is anyone getting CPU units ?

That will happen untill the GPU cache is full, then you will start getting work for the CPU again.
Grant
Darwin NT
ID: 1290939 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1290940 - Posted: 3 Oct 2012, 21:54:52 UTC - in response to Message 1290938.  

You can certainly understand the confusion, then, since The Planetary Society gets top billing:

It's easier to say there is no relationship, than to try to point out the difference between being a founding sponsor & actually being involved in the operation of the project.
Grant
Darwin NT
ID: 1290940 · Report as offensive
JohnDK Crowdfunding Project Donor*Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 28 May 00
Posts: 1222
Credit: 451,243,443
RAC: 1,127
Denmark
Message 1290947 - Posted: 3 Oct 2012, 22:15:32 UTC

Wouldn't a WU limit help big time right now?
ID: 1290947 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1290953 - Posted: 3 Oct 2012, 22:31:52 UTC - in response to Message 1290947.  


Wouldn't a WU limit help big time right now?

Not that much.
Grant
Darwin NT
ID: 1290953 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1290954 - Posted: 3 Oct 2012, 22:32:31 UTC - in response to Message 1290947.  
Last modified: 3 Oct 2012, 22:33:31 UTC

Wouldn't a WU limit help big time right now?

Probably but the "big" crunchers wouldn't like it. Personally for my small part I have set NNT and will wait out the storm. I was hoping to hit 10 million soon, but that can wait.

Edit - Actually looking at the numbers I might make it after all
ID: 1290954 · Report as offensive
dpatter3

Send message
Joined: 3 Sep 03
Posts: 8
Credit: 13,660,454
RAC: 0
United States
Message 1290955 - Posted: 3 Oct 2012, 22:37:31 UTC

After trying all day to upload 80 WU they suddenly went through in the last 30 minutes. Don't know if the "powers that be" tweaked something but all is well.
ID: 1290955 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1290960 - Posted: 3 Oct 2012, 22:51:44 UTC - in response to Message 1290934.  

I contacted the PS because as they are SETI sponsors so they will have ability to contact SETI directly to alert them to the current problem. I dont think anyone in the forum has a direct line to SETI.

You don't appear to understand.
The Plantetary Socieity has no relationship whatsoever with Seti@Home.
Niether does the Seti Institute.

Seti@home is a project being run in the Space Science Laboratory & the Universtiy of Berkeley.
It has no direct affiliations with any other Seti organisations.

Mind you, the University of California at Berkeley is host to a lot more generic SETI projects, over and above the specific SETI@home we work on here.

http://seti.berkeley.edu/
ID: 1290960 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 1290981 - Posted: 3 Oct 2012, 23:41:35 UTC - in response to Message 1290960.  

Well at least this morning here my uploads are going faster than I can produce them and maybe in around an hour, or 2, I might get to try out the download side of things. ;)

Cheers.
ID: 1290981 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1290982 - Posted: 3 Oct 2012, 23:47:12 UTC - in response to Message 1290922.  

Now if only I had a large stack of money for more bandwidth. One day maybe.

I had an idle thought -- I don't remember exactly what the difficulty is in getting a 1 Gbps link down to the campus boundary, but I was wondering if there were a parallel unused "dark fibre" to the existing 100 Mbps link that could be channel-bonded to it to give 200 Mbps. "We" (the UK LCG community) made heavy use of such technology with multiple 1 Gbps links in our data centres until a recent Government windfall enabled most of us to upgrade to 10 Gbps links...

I've been reading Matt's posts for a few years now and if I recall, the problem is getting a 1gbit fibre line "up the hill", which as the crow flies is something like 2.5 miles. It has to be buried, and the last time I heard an estimate or rough figure for that, it was something like US$80,000.

The Hurricane Electronics Internet connection IS gigabit down on the campus, but the router down there does not do gigabit, the link running up the hill does not do gigabit, and I don't remember if the router in the lab can do it. I think it can.

For both getting a new line up the hill and changing out the equipment down on the campus, it is a political nightmare full of red tape, strings, and loop-holes. Even if they got enough donations that were ear-marked for either of these two things, those in charge of the finances don't have to use it for what it was ear-marked for, especially if there's something they deem more important at the time.


I know last year the SSL building finally got a gigabit link, but it is for all the other projects in the building, as well as administrative uses. Uploads and downloads for S@H are required to run only on the HE link. The staff does use the other connection for sending the 50gb "tapes" to and from off-site storage, and this forum that you're reading runs off of that link as well.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1290982 · Report as offensive
Profile MusicGod
Avatar

Send message
Joined: 7 Dec 02
Posts: 97
Credit: 24,782,870
RAC: 0
United States
Message 1290983 - Posted: 3 Oct 2012, 23:49:31 UTC

Here come a Sh*tload of Shorties>>>>>>>
ID: 1290983 · Report as offensive
Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · 13 . . . 22 · Next

Message boards : Number crunching : Panic Mode On (77) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.