Panic Mode On (47) Server problems?

Author	Message
AndrewM Volunteer tester Send message Joined: 5 Jan 08 Posts: 354 Credit: 33,583,383 RAC: 5,583	Message 1112111 - Posted: 1 Jun 2011, 18:20:47 UTC
I wonder if data storage is being moved off Thumper? It was a temporary solution/measure after all
ID: 1112111 ·

[seti.international] Philip J. Fry Volunteer tester Send message Joined: 6 Apr 07 Posts: 6570 Credit: 50,807,522 RAC: 20,504	Message 1112637 - Posted: 3 Jun 2011, 13:07:02 UTC Last modified: 3 Jun 2011, 13:08:35 UTC
I guess I don't need to mention the current probs with UL/DL.. I'm surprised about the following message of BOINC which come from time to time since ~ 00:42 UTC: 'Scheduler request failed: Timeout was reached' - after ~ 60 secs, IIRC normal this happened ~ 600 secs in past.. My machine (BOINC DEV-V6.12.28) is the only one which have current probs to reach the scheduler? *- Best regards!* -** Sutaru Tsureku, team seti.international founder. - Optimize your PC for higher RAC. - SETI@home needs your help. - ____________ >Das Deutsche Cafe. The German Cafe.<
ID: 1112637 ·

Richard Haselgrove Volunteer tester Send message Joined: 4 Jul 99 Posts: 7461 Credit: 39,740,352 RAC: 34,408	Message 1112647 - Posted: 3 Jun 2011, 13:22:32 UTC - in response to Message 1112637.
No, you won't be the only one getting timeouts. I've probably got some too, though I can't be bothered to look. We were discussing 'impossible deadlines' yesterday, linked to the resending of lost VLAR results. Every 'lost result' event almost always starts with a server timeout - they're common, especially when the comms link is busy. For those who haven't got it bookmarked yet, here's a live view of the Cricket Graph. When it's all green like that, you can be sure that the scheduler is issuing tasks, and the download servers are working flat out. Beyond that, it's just the luck of hitting a quiet millisecond in the conversation, so you can get your request in. I've managed it a few times this morning: each time, I've got mostly shorties. The congestion will last until they hit a tape with lower ARs: or they run out of APs to split: or people turn down their cache requests a bit. Until then, sit back and enjoy the show.
ID: 1112647 ·

DJStarfox Send message Joined: 23 May 01 Posts: 1040 Credit: 485,117 RAC: 71	Message 1112648 - Posted: 3 Jun 2011, 13:24:07 UTC
Very strange DL issues.... A new connection gets about 64k then stalls. Repeat for all connections. I think the DL server just needs a reboot.
ID: 1112648 ·

Richard Haselgrove Volunteer tester Send message Joined: 4 Jul 99 Posts: 7461 Credit: 39,740,352 RAC: 34,408	Message 1112650 - Posted: 3 Jun 2011, 13:25:35 UTC - in response to Message 1112648.
I think the DL server just needs a reboot. I think it just needs a rest and a long, cool drink.
ID: 1112650 ·

HAL9000 Volunteer tester Send message Joined: 11 Sep 99 Posts: 3279 Credit: 76,913,836 RAC: 78,483	Message 1112657 - Posted: 3 Jun 2011, 13:37:33 UTC - in response to Message 1112650.
I think the DL server just needs a reboot. I think it just needs a rest and a long, cool drink. A liquid cooled OC48 network connection... I would be OK with that as well. :) ____________ SETI@home classic workunits: 93,865 CPU time: 863,447 hours Join the BP6/VP6 User Group today!
ID: 1112657 ·

John Neale Volunteer tester Send message Joined: 16 Mar 00 Posts: 403 Credit: 1,950,207 RAC: 3,694	Message 1112673 - Posted: 3 Jun 2011, 14:35:09 UTC - in response to Message 1112647.
For those who haven't got it bookmarked yet, here's a live view of the Cricket Graph. Where's the duck? ;) ____________
ID: 1112673 ·

Helli Volunteer tester Send message Joined: 15 Dec 99 Posts: 692 Credit: 66,780,021 RAC: 2,335	Message 1112676 - Posted: 3 Jun 2011, 14:47:04 UTC - in response to Message 1112673. Last modified: 3 Jun 2011, 14:47:45 UTC
For those who haven't got it bookmarked yet, here's a live view of the Cricket Graph. Where's the duck? ;) Yup, it's time for Richard's Rubber Ducky. :D May i post the Link, Richard? Helli
ID: 1112676 ·

Richard Haselgrove Volunteer tester Send message Joined: 4 Jul 99 Posts: 7461 Credit: 39,740,352 RAC: 34,408	Message 1112677 - Posted: 3 Jun 2011, 14:50:42 UTC - in response to Message 1112673.

ID: 1112677 ·

msattler Volunteer tester Send message Joined: 9 Jul 00 Posts: 34178 Credit: 345,230,701 RAC: 551,876	Message 1112680 - Posted: 3 Jun 2011, 14:58:02 UTC - in response to Message 1112677.
Hellooooooo, ducky! ____________ ****** "Ask not, what your kitty can do for you. Ask what you can do for your kitty." As it is kitten, so shall it be done.
ID: 1112680 ·

Richard Haselgrove Volunteer tester Send message Joined: 4 Jul 99 Posts: 7461 Credit: 39,740,352 RAC: 34,408	Message 1112687 - Posted: 3 Jun 2011, 15:19:30 UTC - in response to Message 1112680.
Hellooooooo, ducky! I'd better not post a sailor picture, then :P
ID: 1112687 ·

msattler Volunteer tester Send message Joined: 9 Jul 00 Posts: 34178 Credit: 345,230,701 RAC: 551,876	Message 1112688 - Posted: 3 Jun 2011, 15:21:01 UTC - in response to Message 1112687.
Hellooooooo, ducky! I'd better not post a sailor picture, then :P LOL....not for me. I'll stick with the ducky. ____________ ****** "Ask not, what your kitty can do for you. Ask what you can do for your kitty." As it is kitten, so shall it be done.
ID: 1112688 ·

Dave Send message Joined: 29 Mar 02 Posts: 774 Credit: 23,193,139 RAC: 0	Message 1112799 - Posted: 3 Jun 2011, 20:46:58 UTC
Floating on a sea of green? Surely floating on the blue line. In a green atmosphere... Maybe we have already found alien life - in the form of a duck that breaths green air!!!
ID: 1112799 ·

SciManStev Volunteer tester Send message Joined: 20 Jun 99 Posts: 4297 Credit: 63,037,613 RAC: 73,512	Message 1112819 - Posted: 3 Jun 2011, 21:44:12 UTC
I really like that duck! Steve ____________ Warning, addicted to SETI crunching! Crunching as a member of GPU Users Group. GPUUG Website
ID: 1112819 ·

Richard Haselgrove Volunteer tester Send message Joined: 4 Jul 99 Posts: 7461 Credit: 39,740,352 RAC: 34,408	Message 1112840 - Posted: 3 Jun 2011, 23:45:53 UTC - in response to Message 1112637.
I'm surprised about the following message of BOINC which come from time to time since ~ 00:42 UTC: 'Scheduler request failed: Timeout was reached' - after ~ 60 secs, IIRC normal this happened ~ 600 secs in past.. I tested this with my 420M laptop, also running v6.12.28: 04/06/2011 00:33:22 \| SETI@home \| Sending scheduler request: To fetch work. 04/06/2011 00:33:22 \| SETI@home \| Reporting 7 completed tasks, requesting new tasks for CPU and NVIDIA GPU 04/06/2011 00:33:22 \| SETI@home \| [sched_op] CPU work request: 142292.84 seconds; 0.00 CPUs 04/06/2011 00:33:22 \| SETI@home \| [sched_op] NVIDIA GPU work request: 18929.68 seconds; 0.00 GPUs 04/06/2011 00:35:22 \| SETI@home \| Scheduler request completed: got 48 new tasks 04/06/2011 00:35:22 \| SETI@home \| [sched_op] estimated total CPU task duration: 157672 seconds 04/06/2011 00:35:22 \| SETI@home \| [sched_op] estimated total NVIDIA GPU task duration: 21611 seconds So extactly two minutes, but no timeout. Of the 48 tasks, 39 were shorties.
ID: 1112840 ·

Grant (SSSF) Send message Joined: 19 Aug 99 Posts: 5059 Credit: 40,956,343 RAC: 52,815	Message 1112863 - Posted: 4 Jun 2011, 1:56:21 UTC - in response to Message 1112840.
6 days down & now into the 7th day of full network load. Usually the Results returned per hour are around 50,000 or so. Since the network storage problem outage it hasn't been below 60,000. For the last day it's been above 70,000 & it almost hit 80,000 at one stage. The hardware is certainly taking a beating, and no end in sight. The last batch of work i got still contained a lot of shorties. ____________ Grant Darwin NT.
ID: 1112863 ·

tbret Send message Joined: 28 May 99 Posts: 1812 Credit: 125,282,715 RAC: 411,161	Message 1112890 - Posted: 4 Jun 2011, 5:01:52 UTC - in response to Message 1112863.
6 days down & now into the 7th day of full network load. Usually the Results returned per hour are around 50,000 or so. Since the network storage problem outage it hasn't been below 60,000. For the last day it's been above 70,000 & it almost hit 80,000 at one stage. The hardware is certainly taking a beating, and no end in sight. The last batch of work i got still contained a lot of shorties. I wonder how much damage the project would suffer if the air conditioning went away under these circumstances? Currently, "project backoff: 05:16:00" ____________
ID: 1112890 ·

tbret Send message Joined: 28 May 99 Posts: 1812 Credit: 125,282,715 RAC: 411,161	Message 1112896 - Posted: 4 Jun 2011, 5:42:18 UTC - in response to Message 1112863.
The hardware is certainly taking a beating, and no end in sight. The last batch of work i got still contained a lot of shorties. Oooops. Grant, you and I forgot that the hardware is only working about 1/10th as hard as it was probably designed-for. ____________
ID: 1112896 ·

Grant (SSSF) Send message Joined: 19 Aug 99 Posts: 5059 Credit: 40,956,343 RAC: 52,815	Message 1112922 - Posted: 4 Jun 2011, 7:31:06 UTC - in response to Message 1112890.
I wonder how much damage the project would suffer if the air conditioning went away under these circumstances? No damage, but everything would shutdown. Grant, you and I forgot that the hardware is only working about 1/10th as hard as it was probably designed-for. Actually, due to the lack of the network storage, parts of it will be running pretty much at their limits. ____________ Grant Darwin NT.
ID: 1112922 ·

Dave Send message Joined: 29 Mar 02 Posts: 774 Credit: 23,193,139 RAC: 0	Message 1112925 - Posted: 4 Jun 2011, 7:47:40 UTC - in response to Message 1112819. Last modified: 4 Jun 2011, 7:50:37 UTC
I really like that duck! Steve & for those of you that remember Orville on BBC1: "I 'aaate that duck" ;)!
ID: 1112925 ·

Author

Message

AndrewM
Volunteer tester
Send message
Joined: 5 Jan 08
Posts: 354
Credit: 33,583,383
RAC: 5,583

I wonder if data storage is being moved off Thumper? It was a temporary solution/measure after all

ID: 1112111 ·

[seti.international] Philip J. Fry

Volunteer tester
Avatar

Send message
Joined: 6 Apr 07
Posts: 6570
Credit: 50,807,522
RAC: 20,504

I guess I don't need to mention the current probs with UL/DL..

I'm surprised about the following message of BOINC which come from time to time since ~ 00:42 UTC: 'Scheduler request failed: Timeout was reached' - after ~ 60 secs, IIRC normal this happened ~ 600 secs in past..

My machine (BOINC DEV-V6.12.28) is the only one which have current probs to reach the scheduler?

- Best regards! - Sutaru Tsureku, team seti.international founder. - Optimize your PC for higher RAC. - SETI@home needs your help. -
____________

>Das Deutsche Cafe. The German Cafe.<

ID: 1112637 ·

Richard Haselgrove

Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 7461
Credit: 39,740,352
RAC: 34,408

No, you won't be the only one getting timeouts. I've probably got some too, though I can't be bothered to look.

We were discussing 'impossible deadlines' yesterday, linked to the resending of lost VLAR results. Every 'lost result' event almost always starts with a server timeout - they're common, especially when the comms link is busy.

For those who haven't got it bookmarked yet, here's a live view of the Cricket Graph.

When it's all green like that, you can be sure that the scheduler is issuing tasks, and the download servers are working flat out. Beyond that, it's just the luck of hitting a quiet millisecond in the conversation, so you can get your request in. I've managed it a few times this morning: each time, I've got mostly shorties. The congestion will last until they hit a tape with lower ARs: or they run out of APs to split: or people turn down their cache requests a bit. Until then, sit back and enjoy the show.

ID: 1112647 ·

DJStarfox
Send message
Joined: 23 May 01
Posts: 1040
Credit: 485,117
RAC: 71

Very strange DL issues....

A new connection gets about 64k then stalls. Repeat for all connections. I think the DL server just needs a reboot.

ID: 1112648 ·

Richard Haselgrove

Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 7461
Credit: 39,740,352
RAC: 34,408

I think the DL server just needs a reboot.

I think it just needs a rest and a long, cool drink.

ID: 1112650 ·

HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 3279
Credit: 76,913,836
RAC: 78,483

I think the DL server just needs a reboot.

I think it just needs a rest and a long, cool drink.

A liquid cooled OC48 network connection... I would be OK with that as well. :)
____________
SETI@home classic workunits: 93,865 CPU time: 863,447 hours

Join the BP6/VP6 User Group today!

ID: 1112657 ·

John Neale
Volunteer tester
Avatar

Send message
Joined: 16 Mar 00
Posts: 403
Credit: 1,950,207
RAC: 3,694

For those who haven't got it bookmarked yet, here's a live view of the Cricket Graph.

Where's the duck? ;)

____________

ID: 1112673 ·

Helli

Volunteer tester
Avatar

Send message
Joined: 15 Dec 99
Posts: 692
Credit: 66,780,021
RAC: 2,335

For those who haven't got it bookmarked yet, here's a live view of the Cricket Graph.

Where's the duck? ;)

Yup, it's time for Richard's Rubber Ducky. :D

May i post the Link, Richard?

Helli

ID: 1112676 ·

Richard Haselgrove

Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 7461
Credit: 39,740,352
RAC: 34,408

ID: 1112677 ·

msattler

Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 34178
Credit: 345,230,701
RAC: 551,876

Hellooooooo, ducky!
____________
******

"Ask not, what your kitty can do for you. Ask what you can do for your kitty."

As it is kitten, so shall it be done.

ID: 1112680 ·

Richard Haselgrove

Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 7461
Credit: 39,740,352
RAC: 34,408

Hellooooooo, ducky!

I'd better not post a sailor picture, then :P

ID: 1112687 ·

msattler

Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 34178
Credit: 345,230,701
RAC: 551,876

Hellooooooo, ducky!

I'd better not post a sailor picture, then :P

LOL....not for me.
I'll stick with the ducky.
____________
******

"Ask not, what your kitty can do for you. Ask what you can do for your kitty."

As it is kitten, so shall it be done.

ID: 1112688 ·

Dave

Send message
Joined: 29 Mar 02
Posts: 774
Credit: 23,193,139
RAC: 0

Floating on a sea of green?

Surely floating on the blue line.

In a green atmosphere...

Maybe we have already found alien life - in the form of a duck that breaths green air!!!

ID: 1112799 ·

SciManStev

Volunteer tester

Send message
Joined: 20 Jun 99
Posts: 4297
Credit: 63,037,613
RAC: 73,512

I really like that duck!

Steve
____________
Warning, addicted to SETI crunching!
Crunching as a member of GPU Users Group.
GPUUG Website

ID: 1112819 ·

Richard Haselgrove

Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 7461
Credit: 39,740,352
RAC: 34,408

I'm surprised about the following message of BOINC which come from time to time since ~ 00:42 UTC: 'Scheduler request failed: Timeout was reached' - after ~ 60 secs, IIRC normal this happened ~ 600 secs in past..

I tested this with my 420M laptop, also running v6.12.28:

So extactly two minutes, but no timeout. Of the 48 tasks, 39 were shorties.

ID: 1112840 ·

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5059
Credit: 40,956,343
RAC: 52,815

6 days down & now into the 7th day of full network load.
Usually the Results returned per hour are around 50,000 or so. Since the network storage problem outage it hasn't been below 60,000. For the last day it's been above 70,000 & it almost hit 80,000 at one stage.
The hardware is certainly taking a beating, and no end in sight. The last batch of work i got still contained a lot of shorties.
____________
Grant
Darwin NT.

ID: 1112863 ·

tbret

Send message
Joined: 28 May 99
Posts: 1812
Credit: 125,282,715
RAC: 411,161

6 days down & now into the 7th day of full network load.
Usually the Results returned per hour are around 50,000 or so. Since the network storage problem outage it hasn't been below 60,000. For the last day it's been above 70,000 & it almost hit 80,000 at one stage.
The hardware is certainly taking a beating, and no end in sight. The last batch of work i got still contained a lot of shorties.

I wonder how much damage the project would suffer if the air conditioning went away under these circumstances?

Currently, "project backoff: 05:16:00"

____________

ID: 1112890 ·

tbret

Send message
Joined: 28 May 99
Posts: 1812
Credit: 125,282,715
RAC: 411,161

The hardware is certainly taking a beating, and no end in sight. The last batch of work i got still contained a lot of shorties.

Oooops.

Grant, you and I forgot that the hardware is only working about 1/10th as hard as it was probably designed-for.
____________

ID: 1112896 ·

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5059
Credit: 40,956,343
RAC: 52,815

I wonder how much damage the project would suffer if the air conditioning went away under these circumstances?

No damage, but everything would shutdown.

Grant, you and I forgot that the hardware is only working about 1/10th as hard as it was probably designed-for.

Actually, due to the lack of the network storage, parts of it will be running pretty much at their limits.
____________
Grant
Darwin NT.

ID: 1112922 ·

Dave

Send message
Joined: 29 Mar 02
Posts: 774
Credit: 23,193,139
RAC: 0

I really like that duck!

Steve

& for those of you that remember Orville on BBC1:

"I 'aaate that duck" ;)!

ID: 1112925 ·