Panic Mode On (80) Server Problems?

Message boards : Number crunching : Panic Mode On (80) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · 13 . . . 25 · Next

AuthorMessage
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22160
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1324945 - Posted: 5 Jan 2013, 15:40:24 UTC

The fact that Matt has sent out a message saying that the outage has been cancelled indicates that the outage for the repairs to the SSL-wide power systems has been cancelled for this weekend, and that the whole of the SSL will be breathing a communal sigh of relief. One of two things has happened - the uni has not been able to undertake the repairs this weekend for whatever reason, or the uni has found a way to conduct the repairs without shutting down the whole of the SSL. I suspect the former is the more probable, so there will be a planned outage for the work to be done, and there will be notice of it being done so the whole lab can prepared for it.

(We are looking forward to version four of a similar outage where I work.)
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1324945 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1324951 - Posted: 5 Jan 2013, 15:59:21 UTC

Looks like there can be some outage on Monday, though.
IST Service Status writes:
Scheduled Outage Space Sciences Laboratory

Outage Type: SCHEDULED OUTAGE
Date Submitted: Monday, January 7, 2013
Outage Start/End Time: 0630 – 0700
Groups Impacted: Space Sciences Laboratory
Equipment: sut1ds, sslringsut1fes

Description: Users on the PPCS-SSL-net_169.229.155.240/29 subnet will be without network connectivity while a switchport is reconfigured.

But perhaps that it only affects the forums, and not the data carrier.
ID: 1324951 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1324954 - Posted: 5 Jan 2013, 16:01:08 UTC - in response to Message 1324945.  

The fact that Matt has sent out a message saying that the outage has been cancelled indicates that the outage for the repairs to the SSL-wide power systems has been cancelled for this weekend, and that the whole of the SSL will be breathing a communal sigh of relief. One of two things has happened - the uni has not been able to undertake the repairs this weekend for whatever reason, or the uni has found a way to conduct the repairs without shutting down the whole of the SSL. I suspect the former is the more probable, so there will be a planned outage for the work to be done, and there will be notice of it being done so the whole lab can prepared for it.

(We are looking forward to version four of a similar outage where I work.)

If they have to do any transformer work, total outage is likely.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1324954 · Report as offensive
Profile Donald L. Johnson
Avatar

Send message
Joined: 5 Aug 02
Posts: 8240
Credit: 14,654,533
RAC: 20
United States
Message 1324957 - Posted: 5 Jan 2013, 16:05:14 UTC - in response to Message 1324951.  

Looks like there can be some outage on Monday, though.
IST Service Status writes:
Scheduled Outage Space Sciences Laboratory

Outage Type: SCHEDULED OUTAGE
Date Submitted: Monday, January 7, 2013
Outage Start/End Time: 0630 – 0700
Groups Impacted: Space Sciences Laboratory
Equipment: sut1ds, sslringsut1fes

Description: Users on the PPCS-SSL-net_169.229.155.240/29 subnet will be without network connectivity while a switchport is reconfigured.

But perhaps that it only affects the forums, and not the data carrier.

Only half an hour allotted, and before normal working hours at Berkeley - should be minimal impact on us, probably just the forums and front page.
Donald
Infernal Optimist / Submariner, retired
ID: 1324957 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1324958 - Posted: 5 Jan 2013, 16:07:27 UTC - in response to Message 1324957.  

Looks like there can be some outage on Monday, though.
IST Service Status writes:
Scheduled Outage Space Sciences Laboratory

Outage Type: SCHEDULED OUTAGE
Date Submitted: Monday, January 7, 2013
Outage Start/End Time: 0630 – 0700
Groups Impacted: Space Sciences Laboratory
Equipment: sut1ds, sslringsut1fes

Description: Users on the PPCS-SSL-net_169.229.155.240/29 subnet will be without network connectivity while a switchport is reconfigured.

But perhaps that it only affects the forums, and not the data carrier.

Only half an hour allotted, and before normal working hours at Berkeley - should be minimal impact on us, probably just the forums and front page.

Yeah, OK.
I am an electrician, and know what we can do, LOL.


"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1324958 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22160
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1325024 - Posted: 5 Jan 2013, 18:53:33 UTC

This is an ITS outage, not a power outage, and it only affects (one of) the out of building switches, so the servers will be able to carry on, but there may be no out of building connectivity depending on which switch they are working on and how that switch is being reconfigured.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1325024 · Report as offensive
Keith White
Avatar

Send message
Joined: 29 May 99
Posts: 392
Credit: 13,035,233
RAC: 22
United States
Message 1325032 - Posted: 5 Jan 2013, 19:15:13 UTC - in response to Message 1324930.  
Last modified: 5 Jan 2013, 19:21:09 UTC

I'm concern that they haven't powered down for the scheduled electrical work. I'm concerned because I could see that the staff was told the work was delayed but sometime today someone else will simply cut the power, leading to a mess that'll take days to restore.

Keith...

I am sure they have it covered.
The work was rescheduled.
I am sure we shall see a notice when they are gonna do it.
They're not simply gonna cut out the line to the SSL without notice.

I take it you haven't worked with contractors, utilities or a university maintenance department before. It makes those 6 hour windows for the cable guy seem downright convenient. ;)

Edit: Okay, they did canceled it. Now back to your previously scheduled grumblings already in progress.
"Life is just nature's way of keeping meat fresh." - The Doctor
ID: 1325032 · Report as offensive
.clair.

Send message
Joined: 4 Nov 04
Posts: 1300
Credit: 55,390,408
RAC: 69
United Kingdom
Message 1325036 - Posted: 5 Jan 2013, 19:36:58 UTC - in response to Message 1325024.  

which switch they are working on and how that switch is being reconfigured.

That can be :-
software
firmware
hardware
reboot
steel toecap boot ....
hammer
wire cuters
trash IT
ID: 1325036 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22160
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1325042 - Posted: 5 Jan 2013, 19:51:33 UTC

Just now downloads are back to their normal - about as fast as a snail with arthritis moving across coarse glass paper...

Unlike when there were 4,000,000 awaiting download when I couldn't get any, but downloads were fast.

I just wonder if they were trying to see what happens if you overfill the tanks, and at what point the downloads "work well". Keep going down, its a long way below the current level.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1325042 · Report as offensive
Profile ivan
Volunteer tester
Avatar

Send message
Joined: 5 Mar 01
Posts: 783
Credit: 348,560,338
RAC: 223
United Kingdom
Message 1325050 - Posted: 5 Jan 2013, 20:36:39 UTC - in response to Message 1325042.  

Just now downloads are back to their normal - about as fast as a snail with arthritis moving across coarse glass paper...

Scheduler response times are still measured in seconds though, here at home at least. I wish they'd start to increase the limits a bit though, my faster machines can't last too long a break at 100 GPU WUs/machine.
ID: 1325050 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1325058 - Posted: 5 Jan 2013, 20:54:57 UTC - in response to Message 1325050.  

I wish they'd start to increase the limits a bit though, my faster machines can't last too long a break at 100 GPU WUs/machine.

Or 100 per CPU.

Grant
Darwin NT
ID: 1325058 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1325069 - Posted: 5 Jan 2013, 21:56:42 UTC

Just now downloads are back to their normal - about as fast as a snail with arthritis moving across coarse glass paper...


Hmm that fast!! I was getting speeds of 350K now I'm lucky if it 3K!!
ID: 1325069 · Report as offensive
fscheel

Send message
Joined: 13 Apr 12
Posts: 73
Credit: 11,135,641
RAC: 0
United States
Message 1325255 - Posted: 6 Jan 2013, 13:29:58 UTC

I notice this morning that my pc 6738288 has marked around 100 tasks as abandoned. What caused this and is there anything I need to do? It hasn't abandoned anymore in the past 6-7 hours and appears to be working. Boinc tasks is reporting about 200 tasks on the pc but the web site reports about 120.

thanks...Frank
ID: 1325255 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1325263 - Posted: 6 Jan 2013, 13:38:26 UTC - in response to Message 1325255.  

I notice this morning that my pc 6738288 has marked around 100 tasks as abandoned. What caused this and is there anything I need to do? It hasn't abandoned anymore in the past 6-7 hours and appears to be working. Boinc tasks is reporting about 200 tasks on the pc but the web site reports about 120.

thanks...Frank

Random task abandonment has been reported before. Nothing you can or need to do on your end. Just keep crunching, and if the servers pick up the problem, they might resend the missing tasks. If not, no worries, they'll get crunched by somebody sooner or later.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1325263 · Report as offensive
fscheel

Send message
Joined: 13 Apr 12
Posts: 73
Credit: 11,135,641
RAC: 0
United States
Message 1325270 - Posted: 6 Jan 2013, 13:45:32 UTC - in response to Message 1325263.  

I notice this morning that my pc 6738288 has marked around 100 tasks as abandoned. What caused this and is there anything I need to do? It hasn't abandoned anymore in the past 6-7 hours and appears to be working. Boinc tasks is reporting about 200 tasks on the pc but the web site reports about 120.

thanks...Frank

Random task abandonment has been reported before. Nothing you can or need to do on your end. Just keep crunching, and if the servers pick up the problem, they might resend the missing tasks. If not, no worries, they'll get crunched by somebody sooner or later.



Thanks
ID: 1325270 · Report as offensive
MikeN

Send message
Joined: 24 Jan 11
Posts: 319
Credit: 64,719,409
RAC: 85
United Kingdom
Message 1325314 - Posted: 6 Jan 2013, 15:07:28 UTC - in response to Message 1325270.  

I notice this morning that my pc 6738288 has marked around 100 tasks as abandoned. What caused this and is there anything I need to do? It hasn't abandoned anymore in the past 6-7 hours and appears to be working. Boinc tasks is reporting about 200 tasks on the pc but the web site reports about 120.

thanks...Frank

Random task abandonment has been reported before. Nothing you can or need to do on your end. Just keep crunching, and if the servers pick up the problem, they might resend the missing tasks. If not, no worries, they'll get crunched by somebody sooner or later.



Thanks

I think you need to abandon all the tasks on your PC by resetting the project. If BOINC has marked them as abandoned you will not get credit for them, nor will any results you return count. I would also reboot the PC as there may be some sort of problem with it.

This happened to me three times in two days over Christmas, always with the same PC. Everytime it got 100 CPU tasks, a couple of hours later they would all be marked abandoned. In the end I suspended CPU processing until I could get back to work to reboot he system and check there was nothing wrong with the PC. It has been fine since.
ID: 1325314 · Report as offensive
Profile Cliff Harding
Volunteer tester
Avatar

Send message
Joined: 18 Aug 99
Posts: 1432
Credit: 110,967,840
RAC: 67
United States
Message 1325326 - Posted: 6 Jan 2013, 16:16:57 UTC

Looks like things are grinding to a halt again. Can't get any GPU d/ls because I have 2 AP tasks that have been stuck for several hours and retries don't help much as it quickly drops to under 1.0 kb then goes into retry status.


I don't buy computers, I build them!!
ID: 1325326 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1325476 - Posted: 7 Jan 2013, 10:21:23 UTC - in response to Message 1325326.  


Whatever was causing the MB validators to get bogged down appears to have cleared- the backlog is now declining instead of growing. Also the AP work appears to have pretty much cleared, so hopefully the download traffic will settle down shortly.
And whatever they did to the splitters after that last outage appears to have worked- i can't remember a time when the splitters were able to pump out 60 WU/s time after time, but now the Ready to send buffer is back down to normal levels, that's what they're doing. 50/s is the worst so far, for a quite a while there they were struggling to do 20/s.
Grant
Darwin NT
ID: 1325476 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22160
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1325623 - Posted: 7 Jan 2013, 22:24:45 UTC

The current round of sick download performance was happening before the APs started to move, so you can forget them as the culprit - the root cause is a very poorly though through and implemented download scheduling algorithm that can do nothing apart from cause problems.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1325623 · Report as offensive
.clair.

Send message
Joined: 4 Nov 04
Posts: 1300
Credit: 55,390,408
RAC: 69
United Kingdom
Message 1325661 - Posted: 8 Jan 2013, 1:58:31 UTC - in response to Message 1325623.  

- the root cause is a very poorly though through and implemented download scheduling algorithm that can do nothing apart from cause problems.

Iz that inside info ?
If they have found another gremlin with its neck stuck in a bottle,
the hIT men can have a smashing time with it,
unless the Trainman says otherwize :(
ID: 1325661 · Report as offensive
Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · 13 . . . 25 · Next

Message boards : Number crunching : Panic Mode On (80) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.