Panic Mode On (50) Server problems?

Message boards : Number crunching : Panic Mode On (50) Server problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 4 · 5 · 6 · 7 · 8 · 9 · 10 · Next

AuthorMessage
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13742
Credit: 208,696,464
RAC: 304
Australia
Message 1128215 - Posted: 16 Jul 2011, 8:03:09 UTC


Uhoh! Upload traffic is tapering off again, so expect some upload difficulties shortly. Or maybe no response from the Scheduler.
Grant
Darwin NT
ID: 1128215 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22216
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1128216 - Posted: 16 Jul 2011, 8:04:44 UTC

MB splitters will shut themselves down when they run out of tapes to split, and as I type this they are almost to that point, so unless someone breaks into the lab over the weekend its going to be a very empty stream by Monday morning.
Still, that's what caches are all about - carry you over the data outages.

(And if all else fails hope a reserve project will pick up the torch)
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1128216 · Report as offensive
Profile Cliff Harding
Volunteer tester
Avatar

Send message
Joined: 18 Aug 99
Posts: 1432
Credit: 110,967,840
RAC: 67
United States
Message 1128404 - Posted: 16 Jul 2011, 18:15:30 UTC

Starting to have upload problems again, haven't been able for approx an 1hr. Cricket shows something is slowing down.


I don't buy computers, I build them!!
ID: 1128404 · Report as offensive
Silvester the furious
Avatar

Send message
Joined: 19 Nov 10
Posts: 79
Credit: 1,734,928
RAC: 0
United States
Message 1128414 - Posted: 16 Jul 2011, 18:47:59 UTC - in response to Message 1128215.  


Uhoh! Upload traffic is tapering off again, so expect some upload difficulties shortly. Or maybe no response from the Scheduler.



"The hardest thing in the world to understand is the income tax."
--Albert Einstein

DON'T TREAD ON ME!
ID: 1128414 · Report as offensive
Profile soft^spirit
Avatar

Send message
Joined: 18 May 99
Posts: 6497
Credit: 34,134,168
RAC: 0
United States
Message 1128423 - Posted: 16 Jul 2011, 19:23:23 UTC

well they have obviously been struggling with servers this week.. the status page seems.. out dated. An update on how they are doing would be really nice,
and call us for help if we can.

in the mean time,

Janice
ID: 1128423 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1128427 - Posted: 16 Jul 2011, 19:27:26 UTC - in response to Message 1128423.  

well they have obviously been struggling with servers this week.. the status page seems.. out dated. An update on how they are doing would be really nice,
and call us for help if we can.

in the mean time,

Status page updated at 19:20.
But, reaching the upload server is spotty at best right now.
Suspect the pipe is clogged with AP work going outbound.

So enjoy your panic anyway...LOL.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1128427 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 1128443 - Posted: 16 Jul 2011, 20:31:47 UTC - in response to Message 1128430.  
Last modified: 16 Jul 2011, 20:32:22 UTC

Before you blame the APs have you looked at how many of your tasks are shorties? I was watching my transfers tab and by the time one managed to upload I had two more show up waiting to try. It's been that way since at least early last night.

Oh, have I mentioned lately [b]I hate shorties storms!!![b] ?


PROUD MEMBER OF Team Starfire World BOINC
ID: 1128443 · Report as offensive
Kevin Olley

Send message
Joined: 3 Aug 99
Posts: 906
Credit: 261,085,289
RAC: 572
United Kingdom
Message 1128448 - Posted: 16 Jul 2011, 20:38:14 UTC

It ain't all bad.

My downloads MB are now coming in faster than I can process them, download queue is reducing at a fast rate.

Stop press.

Upload speed has suddenly increased.


Kevin


ID: 1128448 · Report as offensive
Profile soft^spirit
Avatar

Send message
Joined: 18 May 99
Posts: 6497
Credit: 34,134,168
RAC: 0
United States
Message 1128451 - Posted: 16 Jul 2011, 20:39:30 UTC

things do not match as far as what is up and down and what symptoms are appearing.. I think they re-shuffled some servers and did not note the page yet.
As well Jocelyn has been showing off which usually updates the info.

Some things just do not add up with what we are getting. It is low priority
but.. I think the page needs the entries corrected.

Of course I could be wrong.. but either way I can panic :D
Janice
ID: 1128451 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1128473 - Posted: 16 Jul 2011, 21:34:20 UTC - in response to Message 1128451.  

things do not match as far as what is up and down and what symptoms are appearing.. I think they re-shuffled some servers and did not note the page yet.
As well Jocelyn has been showing off which usually updates the info.

Some things just do not add up with what we are getting. It is low priority
but.. I think the page needs the entries corrected.

Of course I could be wrong.. but either way I can panic :D

Jocelyn has not been doing so well keeping up lately.
I think when the replica DB is down, all inquiries go directly to Carolyn, the master DB. And she is up to the task.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1128473 · Report as offensive
Highlander
Avatar

Send message
Joined: 5 Oct 99
Posts: 167
Credit: 37,987,668
RAC: 16
Germany
Message 1128476 - Posted: 16 Jul 2011, 21:49:40 UTC
Last modified: 16 Jul 2011, 21:51:52 UTC

I also process the WUs faster than i can download. Seems that my 4 day cache is not enough, but i have no intention to increase this. Other projects like also some computing cycles :-).

I finally know, why the download pipe is so clogged: Look :)
- Performance is not a simple linear function of the number of CPUs you throw at the problem. -
ID: 1128476 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13742
Credit: 208,696,464
RAC: 304
Australia
Message 1128481 - Posted: 16 Jul 2011, 22:02:04 UTC - in response to Message 1128476.  
Last modified: 16 Jul 2011, 22:03:13 UTC

I also process the WUs faster than i can download. Seems that my 4 day cache is not enough,

It's not so much the cache not being large enough, it's not being able to fill the cache up that leads to running out of work.

It's either no work is being allocated, or it's being allocated but the traffic makes it difficult to download. Something in the middle of those 2 would be nice.
Grant
Darwin NT
ID: 1128481 · Report as offensive
Treasurer

Send message
Joined: 13 Dec 05
Posts: 109
Credit: 1,569,762
RAC: 0
Germany
Message 1128483 - Posted: 16 Jul 2011, 22:02:46 UTC

All MBs are split. Open your caches wide people! Free APs for everyone!
ID: 1128483 · Report as offensive
Profile Donald L. Johnson
Avatar

Send message
Joined: 5 Aug 02
Posts: 8240
Credit: 14,654,533
RAC: 20
United States
Message 1128575 - Posted: 17 Jul 2011, 2:19:55 UTC - in response to Message 1128473.  
Last modified: 17 Jul 2011, 2:52:19 UTC

things do not match as far as what is up and down and what symptoms are appearing.. I think they re-shuffled some servers and did not note the page yet.
As well Jocelyn has been showing off which usually updates the info.

Some things just do not add up with what we are getting. It is low priority
but.. I think the page needs the entries corrected.

Of course I could be wrong.. but either way I can panic :D

Jocelyn has not been doing so well keeping up lately.
I think when the replica DB is down, all inquiries go directly to Carolyn, the master DB. And she is up to the task.

I think Carolyn has been carrying the whole load since jocelyn started having storage/disc drive problems in June(?). Besides Jocelyn not being able to keep up with Carolyn, the Forums have been slow loading, and the number of master database queries shown on the Status page seems to have been running much higher (800-1000/sec) than I recall before June (Yeah, it's only about 650 as I type. It's a week-end.).

I think Jocelyn is still not working properly, so they were using her for a real-time back-up, but not for any account or website-related functions, until they took her off-line last week. Wish somebody would give us an update on Monday afternoon.
Donald
Infernal Optimist / Submariner, retired
ID: 1128575 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1128597 - Posted: 17 Jul 2011, 4:19:20 UTC

Everything seems to be working fine for me. Uploads go through on the first try. Scheduler requests go through on the first try, and when I do get an AP task, it downloads on the first try. Maybe I'm lucky, or maybe things are working properly.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1128597 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13742
Credit: 208,696,464
RAC: 304
Australia
Message 1128631 - Posted: 17 Jul 2011, 6:53:16 UTC - in response to Message 1128597.  

Maybe I'm lucky, or maybe things are working properly.

You're just lucky.

Grant
Darwin NT
ID: 1128631 · Report as offensive
Profile Uli
Volunteer tester
Avatar

Send message
Joined: 6 Feb 00
Posts: 10923
Credit: 5,996,015
RAC: 1
Germany
Message 1128636 - Posted: 17 Jul 2011, 7:02:32 UTC

This maybe a noval idea, but I have been thinking of reducing my cache in Seti.

Mind you, it could be mos, until I make a move.
Pluto will always be a planet to me.

Seti Ambassador
Not to late to order an Anni Shirt
ID: 1128636 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 1128713 - Posted: 17 Jul 2011, 13:32:36 UTC - in response to Message 1128636.  

I don't have to worry about reducing my cache, SETI has taken care of that for me! :-) I have two APs running now and have managed to beg, borrow or steal two more APs and two MBs to do later. I have no CUDA work of any kind though.


Please, someone, go in and load another tape!


PROUD MEMBER OF Team Starfire World BOINC
ID: 1128713 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1128718 - Posted: 17 Jul 2011, 13:51:31 UTC - in response to Message 1128713.  

I don't have to worry about reducing my cache, SETI has taken care of that for me! :-) I have two APs running now and have managed to beg, borrow or steal two more APs and two MBs to do later. I have no CUDA work of any kind though.


Please, someone, go in and load another tape!

Loading more worksets right now is not gonna change much.
Cricket shows bandwidth maxxed out. Work is going out as fast as it can. Difference lately is a lot more of it is AP, which chews up more bandwidth than even shorty MB storms, I think.
If you check the Scarecrow graphs for AP, in progress is climbing at pretty good rate.
I have MB set as my primary, but accepting AP if none is available, and some are coming through now and again.
Not as fast as I am sending them out however.

Not sure if the demand for AP WUs is gonna taper off as smaller caches and slower crunchers are satisfied or not.

If not, bandwidth is the primary bottleneck when the servers are all running up to snuff. It cannot be addressed soon enough.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1128718 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 1128771 - Posted: 17 Jul 2011, 15:49:18 UTC - in response to Message 1128718.  

Well, it looks like someone heard me. I'm showing another tape splitting but it's, of course, not fast enough to keep up with us and I haven't been able to grab any yet. I have a couple of APs sitting around waiting for a free CPU. I might play with the reschedule and run them on my poor cold GPU.


PROUD MEMBER OF Team Starfire World BOINC
ID: 1128771 · Report as offensive
Previous · 1 . . . 4 · 5 · 6 · 7 · 8 · 9 · 10 · Next

Message boards : Number crunching : Panic Mode On (50) Server problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.