Panic Mode On (40) Server problems

Message boards : Number crunching : Panic Mode On (40) Server problems
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 4 · 5 · 6 · 7 · 8 · 9 · 10 . . . 11 · Next

AuthorMessage
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14660
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1042709 - Posted: 16 Oct 2010, 16:17:56 UTC - in response to Message 1042707.  

The "Retry Now" button is the only thing that has gotten me any WU's this morning. I now have no transfers pending, but when I try a manual update all I get is the "Not reporting or requesting tasks" message from the server. My preferences are set to try and keep about three days work on hand. Strange.

Any thoughts?

The "Not reporting or requesting tasks" message comes from your local client, not from the server.
ID: 1042709 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51470
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1042715 - Posted: 16 Oct 2010, 16:27:31 UTC - in response to Message 1042707.  
Last modified: 16 Oct 2010, 16:36:49 UTC

The "Retry Now" button is the only thing that has gotten me any WU's this morning. I now have no transfers pending, but when I try a manual update all I get is the "Not reporting or requesting tasks" message from the server. My preferences are set to try and keep about three days work on hand. Strange.

Any thoughts?

Check the exchanges between SciManSteve and myself earlier in this thread.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1042715 · Report as offensive
Profile ajinbc
Avatar

Send message
Joined: 15 Mar 06
Posts: 484
Credit: 318,444
RAC: 0
Canada
Message 1042717 - Posted: 16 Oct 2010, 16:38:40 UTC - in response to Message 1042715.  

Thanks, I already did.

Huge thank you for the "Retry Now" tip/solution to my pending downloads. I've got enough work for about six hours now. Hopefully I'll get some more WU's later today.


ID: 1042717 · Report as offensive
Profile James Sotherden
Avatar

Send message
Joined: 16 May 99
Posts: 10436
Credit: 110,373,059
RAC: 54
United States
Message 1042732 - Posted: 16 Oct 2010, 17:28:26 UTC

I have got a ton of work stuck in download land since last night. Probally get a new crop of ghosts out of it. I knew I should have left my NNT button alone:)
[/quote]

Old James
ID: 1042732 · Report as offensive
Profile Gundolf Jahn

Send message
Joined: 19 Sep 00
Posts: 3184
Credit: 446,358
RAC: 0
Germany
Message 1042739 - Posted: 16 Oct 2010, 17:38:11 UTC - in response to Message 1042732.  

Probally get a new crop of ghosts out of it.

If they are in the Transfers tab, they can't be ghosts! Obviously, your client has received the list of assigned tasks from the server.

Gruß,
Gundolf
ID: 1042739 · Report as offensive
Profile James Sotherden
Avatar

Send message
Joined: 16 May 99
Posts: 10436
Credit: 110,373,059
RAC: 54
United States
Message 1042751 - Posted: 16 Oct 2010, 18:15:21 UTC

Thanks Gundolf, I forgot that little detail.
[/quote]

Old James
ID: 1042751 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13797
Credit: 208,696,464
RAC: 304
Australia
Message 1042758 - Posted: 16 Oct 2010, 18:22:58 UTC - in response to Message 1042621.  

Not sure what's happening, but inbound traffic is way up. DDoS attack? When everyone hits the per machine limit the inbound traffic ramps up, but outbound is still pegged & i expect it'll be another day till everyone hits the limits.


Inbound traffic is still way above normal. My downloads are also taking several attempts before they finally complete (5kB/s at best, some as slow as 1kB/s). And i notice that the assimilators are no longer making headway- the queue is growning. Last time that happened was 2 (or 3?) weeks ago & they had to shut down the forums & some other services to get things working again.
Grant
Darwin NT
ID: 1042758 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 35564
Credit: 261,360,520
RAC: 489
Australia
Message 1042817 - Posted: 16 Oct 2010, 21:35:33 UTC - in response to Message 1042665.  

DLs?

Scheduler request completed: got 0 new tasks
Message from server: No work sent
Message from server: This computer has reached a limit on tasks in progress


IIRC, Jeff mentioned that we'll have this week after the outage a/the high limit.

Hmm.. it's look like not.
At least not at my machines..

Someone know why we have only the small limit?

Well I've received a lot more work this time than I got at this same time last week.

Cheers.
ID: 1042817 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 35564
Credit: 261,360,520
RAC: 489
Australia
Message 1042819 - Posted: 16 Oct 2010, 21:37:29 UTC - in response to Message 1042758.  

Not sure what's happening, but inbound traffic is way up. DDoS attack? When everyone hits the per machine limit the inbound traffic ramps up, but outbound is still pegged & i expect it'll be another day till everyone hits the limits.


Inbound traffic is still way above normal. My downloads are also taking several attempts before they finally complete (5kB/s at best, some as slow as 1kB/s). And i notice that the assimilators are no longer making headway- the queue is growning. Last time that happened was 2 (or 3?) weeks ago & they had to shut down the forums & some other services to get things working again.

Yes it's just like pulling teeth from a hen.

Cheers.
ID: 1042819 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51470
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1042821 - Posted: 16 Oct 2010, 21:40:47 UTC - in response to Message 1042819.  

Not sure what's happening, but inbound traffic is way up. DDoS attack? When everyone hits the per machine limit the inbound traffic ramps up, but outbound is still pegged & i expect it'll be another day till everyone hits the limits.


Inbound traffic is still way above normal. My downloads are also taking several attempts before they finally complete (5kB/s at best, some as slow as 1kB/s). And i notice that the assimilators are no longer making headway- the queue is growning. Last time that happened was 2 (or 3?) weeks ago & they had to shut down the forums & some other services to get things working again.

Yes it's just like pulling teeth from a hen.

Cheers.

Eric will be looking into it when he can.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1042821 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14660
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1042832 - Posted: 16 Oct 2010, 21:54:59 UTC - in response to Message 1042821.  

Eric will be looking into it when he can.

Not until Monday, I trust.

With the cricket graphs still pegged out at maximum, there's absolutely no point in him sacrificing a family weekend to speed up any other part of the system.
ID: 1042832 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 35564
Credit: 261,360,520
RAC: 489
Australia
Message 1042933 - Posted: 17 Oct 2010, 0:59:18 UTC - in response to Message 1042832.  

Maybe I should have spoken a bit sooner as not long after my last post the dam burst.

All 3 pc's are now happy campers with plenty of work.

Cheers.
ID: 1042933 · Report as offensive
Profile SciManStev Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Jun 99
Posts: 6656
Credit: 121,090,076
RAC: 0
United States
Message 1042937 - Posted: 17 Oct 2010, 1:12:09 UTC - in response to Message 1042933.  

Even my rig, which would not request work, seems to have a decent backlog of tasks to download. I feel much better now, as when there is available bandwith, I have wu's to download.

It seemed to pick up speed as the day drew on. The less I looked at it, the better it did. Hmmmmm...

Steve
Warning, addicted to SETI crunching!
Crunching as a member of GPU Users Group.
GPUUG Website
ID: 1042937 · Report as offensive
Profile soft^spirit
Avatar

Send message
Joined: 18 May 99
Posts: 6497
Credit: 34,134,168
RAC: 0
United States
Message 1042944 - Posted: 17 Oct 2010, 1:22:51 UTC

hitting project limits here.

So done.. for now.
Janice
ID: 1042944 · Report as offensive
Profile SciManStev Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Jun 99
Posts: 6656
Credit: 121,090,076
RAC: 0
United States
Message 1042965 - Posted: 17 Oct 2010, 2:02:44 UTC

From the days earlier situation of not being able to request work, I have gone to plenty of wu's in the download cache with at least 80 AP units. I do the AP units in about 7 hours each, and with 5 cores crunching on them, bring em on! My chiller won't start cycling on and off tonight!

Steve
Warning, addicted to SETI crunching!
Crunching as a member of GPU Users Group.
GPUUG Website
ID: 1042965 · Report as offensive
DJStarfox

Send message
Joined: 23 May 01
Posts: 1066
Credit: 1,226,053
RAC: 2
United States
Message 1042986 - Posted: 17 Oct 2010, 3:07:12 UTC

This is a classic computer science problem. N computers request data from a server at a poisson rate. If request succeeds, computer waits for big random some time. If request fails, computer tries again in 11 seconds + small random time.

Expected outcome: Server overload.
ID: 1042986 · Report as offensive
Lionel

Send message
Joined: 25 Mar 00
Posts: 680
Credit: 563,640,304
RAC: 597
Australia
Message 1043006 - Posted: 17 Oct 2010, 4:17:11 UTC - in response to Message 1042544.  


already have 1 machine on at "limit on taks in progress"...

how about switching it off and letting us fill our caches up before the next outage...


Don't know which of your machines you are talking about but..........

2 of your machines show nearly 5000 wu on board. Allowing some for ghosts, just one of your machines has enough work to last for the entire 3 day outage for all 4 of my machines. I do about 4000 total for the 3 day outage.

How about leave some for us..........


re allocated wus...

one machine has ~1k wus
one still downloading has ~720 wus with about 300 still coming in
one with ~1600 wus

the rest are just ghosts ...

in terms of throughput, the above is just enough to cover the 3 day outage on 1/2 boxes only...the third will run dry by end of day 2 ... my caches normally run with ~6k-7k wus per machine to avoid seti outages/downtime but what i have noticed is that since the 3 day outage was implemented, the caches have been dwindling and pre last outage they ran dry and have not been able to refill over 2 successive cycles ... (i don't think its working very well) ...

at the moment all the boxes are asking seti for more work about every 1-2 minutes and getting "reached a limit...." ... i know i'm not the only one that's in this position or feeling pain

my pending credit is now at 857k+ ... it used to sit around 130k...








ID: 1043006 · Report as offensive
Profile Sutaru Tsureku
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 1043035 - Posted: 17 Oct 2010, 8:08:01 UTC - in response to Message 1042817.  
Last modified: 17 Oct 2010, 8:09:58 UTC

DLs?

Scheduler request completed: got 0 new tasks
Message from server: No work sent
Message from server: This computer has reached a limit on tasks in progress


IIRC, Jeff mentioned that we'll have this week after the outage a/the high limit.

Hmm.. it's look like not.
At least not at my machines..

Someone know why we have only the small limit?

Well I've received a lot more work this time than I got at this same time last week.

Cheers.


We have currently:
 40 WUs CPU/Core
320 WUs GPU


But I'm confused - why?
Jeff mentioned that this week we will have the high limit. But it's not.

-----------------------------------------------------------

The one and only problem is the traffic bandwidth (max. ~ 95 Mbits/sec) between S@h and us.
So why not solve this problem?

What give it to buy new servers, if the memebers can't reach them?

How is the currently status? We'll see one day the double speed or much more?

If more and more people will upgrade their systems with GPUs.. one day noone can reach S@h..
OTOH, more and more people will leave S@h - because they can't reach S@h..
But always, the possible traffic bandwidth is too low!
ID: 1043035 · Report as offensive
Blake Bonkofsky
Volunteer tester
Avatar

Send message
Joined: 29 Dec 99
Posts: 617
Credit: 46,383,149
RAC: 0
United States
Message 1043039 - Posted: 17 Oct 2010, 8:23:45 UTC - in response to Message 1043035.  

From my understanding, I believe that if the new servers work out as planned, the old ones can be turned into machines to do the science on the back end, without the need for a full 3 day shutdown. If that is the case, the bandwidth will not be an issue, as the project can run longer than 4 days a week. I know I read in one of the threads on here that it is at least being considered.
ID: 1043039 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 35564
Credit: 261,360,520
RAC: 489
Australia
Message 1043047 - Posted: 17 Oct 2010, 8:53:47 UTC - in response to Message 1043035.  

DLs?

Scheduler request completed: got 0 new tasks
Message from server: No work sent
Message from server: This computer has reached a limit on tasks in progress


IIRC, Jeff mentioned that we'll have this week after the outage a/the high limit.

Hmm.. it's look like not.
At least not at my machines..

Someone know why we have only the small limit?

Well I've received a lot more work this time than I got at this same time last week.

Cheers.


We have currently:
 40 WUs CPU/Core
320 WUs GPU


But I'm confused - why?
Jeff mentioned that this week we will have the high limit. But it's not.

-----------------------------------------------------------

The one and only problem is the traffic bandwidth (max. ~ 95 Mbits/sec) between S@h and us.
So why not solve this problem?

What give it to buy new servers, if the memebers can't reach them?

How is the currently status? We'll see one day the double speed or much more?

If more and more people will upgrade their systems with GPUs.. one day noone can reach S@h..
OTOH, more and more people will leave S@h - because they can't reach S@h..
But always, the possible traffic bandwidth is too low!

Well I don't know where you are getting your figures from for this week but I have about 30% more work against this time last week.

Cheers.
ID: 1043047 · Report as offensive
Previous · 1 . . . 4 · 5 · 6 · 7 · 8 · 9 · 10 . . . 11 · Next

Message boards : Number crunching : Panic Mode On (40) Server problems


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.