Panic Mode On (80) Server Problems?

Message boards : Number crunching : Panic Mode On (80) Server Problems?

To post messages, you must log in.

Previous · 1 . . . 21 · 22 · 23 · 24 · 25 · Next

AuthorMessage
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11143
Credit: 83,845,598
RAC: 46,300
United Kingdom
Message 1332468 - Posted: 29 Jan 2013, 10:10:20 UTC - in response to Message 1332467.  

But yes, there is a problem.

Yep, Scheduler borked again.
"Couldn't connect to server" once again the standard response.

The server status page froze at 08:30 UTC - once that happens, there's usually no scheduler service until the staff get to the lab and restart things.

Which, since it's Tuesday, means not until after maintenance.

And since 'ready to send' was below high water mark when the page froze, and the splitters were running, we'll probably have a big bloat of tasks to work off when things are working again.
ID: 1332468 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 7495
Credit: 91,178,764
RAC: 46,088
Australia
Message 1332469 - Posted: 29 Jan 2013, 10:17:30 UTC - in response to Message 1332467.  

But yes, there is a problem.

Yep, Scheduler borked again.
"Couldn't connect to server" once again the standard response.


Make that the only response.
The last few times the Scheduler was playing up hitting rerty a few hundred times would eventually report the work done & get a bit more, but not this time. Dead as a dodo.
Grant
Darwin NT
ID: 1332469 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 2871
Credit: 10,622,529
RAC: 342
United States
Message 1332470 - Posted: 29 Jan 2013, 10:18:02 UTC

Well without a proxy, downloads are still questionable and fail often.. but I picked a proxy from the list and the 3 APs I had in my download queue were screaming in at 75-100KB/sec each.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1332470 · Report as offensive
juan BFP
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 5847
Credit: 330,565,742
RAC: 7,841
Panama
Message 1332474 - Posted: 29 Jan 2013, 11:09:28 UTC

29/01/2013 09:05:19 | SETI@home | Scheduler request failed: Couldn't connect to server
29/01/2013 09:05:21 | | Internet access OK - project servers may be temporarily down.

Again? I´m tired...
ID: 1332474 · Report as offensive
MikeN

Send message
Joined: 24 Jan 11
Posts: 314
Credit: 44,990,399
RAC: 11,037
United Kingdom
Message 1332497 - Posted: 29 Jan 2013, 13:26:27 UTC

Just to add insult to injury, SETI decided to declare all 180 tasks on my main cruncher 'abandoned' at 2AM this morning (UK time). After I rebooted and reset the project I have not been able to connect to SETI to get any new tasks, so it is now eating its way through Einstein and Cosmology and will probably stay that way until after the weekly outage, probably about another 8-9 hours:((
ID: 1332497 · Report as offensive
Profile Ex
Volunteer tester
Avatar

Send message
Joined: 12 Mar 12
Posts: 2895
Credit: 2,080,451
RAC: 420
United States
Message 1332518 - Posted: 29 Jan 2013, 15:23:53 UTC
Last modified: 29 Jan 2013, 15:24:37 UTC

I don't seem to be able to upload tasks or get any at the moment.

I know it's Tuesday AM over in Cali, but isn't it too early for the server to be down?

I guess it's good I just bumped up my caches yesterday.
-Dave #2

3.2.0-33
ID: 1332518 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45949
Credit: 815,445,142
RAC: 124,478
United States
Message 1332520 - Posted: 29 Jan 2013, 15:27:27 UTC - in response to Message 1332518.  

I don't seem to be able to upload tasks or get any at the moment.

I know it's Tuesday AM over in Cali, but isn't it too early for the server to be down?

I guess it's good I just bumped up my caches yesterday.

Servers crashed last night. Bookmark the Cricket graph for future reference.
Hopefully they'll be back up later today after the usual maintenance outage.
Always remember.....kitties are all Angels with fur.

Have made friends in this life.
Most were cats.
ID: 1332520 · Report as offensive
Profile Ex
Volunteer tester
Avatar

Send message
Joined: 12 Mar 12
Posts: 2895
Credit: 2,080,451
RAC: 420
United States
Message 1332522 - Posted: 29 Jan 2013, 15:33:39 UTC - in response to Message 1332520.  

I don't seem to be able to upload tasks or get any at the moment.

I know it's Tuesday AM over in Cali, but isn't it too early for the server to be down?

I guess it's good I just bumped up my caches yesterday.

Servers crashed last night. Bookmark the Cricket graph for future reference.
Hopefully they'll be back up later today after the usual maintenance outage.

Thanks and thanks!
-Dave #2

3.2.0-33
ID: 1332522 · Report as offensive
TPCBF

Send message
Joined: 18 May 99
Posts: 54
Credit: 2,731,128
RAC: 837
United States
Message 1332532 - Posted: 29 Jan 2013, 16:24:47 UTC - in response to Message 1331704.  

Good to see I am not the only "old timer" still patiently crunching.
Altho with SETI being a bit squiffy I am afriad I have switched most of my CPU time to World Community Grid for the time being.
Now that explains why WCG went tits up yesterday as well... <LOL> (well, actually a rather sad situation)

Ralf
ID: 1332532 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45949
Credit: 815,445,142
RAC: 124,478
United States
Message 1332536 - Posted: 29 Jan 2013, 16:33:52 UTC - in response to Message 1332532.  

Good to see I am not the only "old timer" still patiently crunching.
Altho with SETI being a bit squiffy I am afriad I have switched most of my CPU time to World Community Grid for the time being.
Now that explains why WCG went tits up yesterday as well... <LOL> (well, actually a rather sad situation)

Ralf

Actually, my CPUs are still all crunching Seti. It's the GPUs that quickly ran out and are now idling until the servers come back online. I sometimes run Einstein as a backup, but for now the kitties will just wait it out on standby.
Always remember.....kitties are all Angels with fur.

Have made friends in this life.
Most were cats.
ID: 1332536 · Report as offensive
Richard HaselgroveProject Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11143
Credit: 83,845,598
RAC: 46,300
United Kingdom
Message 1332580 - Posted: 29 Jan 2013, 22:48:44 UTC - in response to Message 1332522.  

I don't seem to be able to upload tasks or get any at the moment.

I know it's Tuesday AM over in Cali, but isn't it too early for the server to be down?

I guess it's good I just bumped up my caches yesterday.

Servers crashed last night. Bookmark the Cricket graph for future reference.
Hopefully they'll be back up later today after the usual maintenance outage.

Thanks and thanks!

Bookmark the server status page while you're at it, and pay special attention to the [As of xxx] time in the top-left corner.

And since 'ready to send' was below high water mark when the page froze, and the splitters were running, we'll probably have a big bloat of tasks to work off when things are working again.

Now -

Results ready to send:1,444,950
ID: 1332580 · Report as offensive
Rolf

Send message
Joined: 16 Jun 09
Posts: 114
Credit: 7,817,146
RAC: 0
Switzerland
Message 1332586 - Posted: 29 Jan 2013, 23:12:01 UTC - in response to Message 1332580.  

Results ready to send:1,444,950

AND

30.01.2013 00:01:01 | SETI@home | Scheduler request completed: got 0 new tasks
30.01.2013 00:01:01 | SETI@home | Project has no tasks available

SORRY! But messages like this only put more confusion to the user's brain than anything else does! Or is it because of my bad English?
When I started with SETI June 16th 1999 (my first account), I thought SETI would be easy understanding and it would not need a lot of button-pushing.
No it's confusing (see above) and my retry-button is almost kaput!
ID: 1332586 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 5431
Credit: 66,416,873
RAC: 12,792
Russia
Message 1332598 - Posted: 30 Jan 2013, 0:16:31 UTC - in response to Message 1332359.  

Been not watching closely over the traditional Australia Day long weekend chaos, and my machines were crunching when I looked occasionally. If I had stuck transfers I just put this retryMainTransfers.cmd in my scheduled tasks for every 20 mins or so:

@ECHO OFF
boinccmd --get_file_transfers > mainxfers.txt
FOR /F "tokens=1,2" %%i IN (mainxfers.txt) DO (
 IF "%%i" EQU "name:" echo %%j
 IF "%%i" EQU "name:" boinccmd --file_transfer http://setiathome.berkeley.edu/ %%j retry
)

Thanks! Will try
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1332598 · Report as offensive
Profile Ex
Volunteer tester
Avatar

Send message
Joined: 12 Mar 12
Posts: 2895
Credit: 2,080,451
RAC: 420
United States
Message 1332636 - Posted: 30 Jan 2013, 3:06:43 UTC - in response to Message 1332586.  

Results ready to send:1,444,950

AND

30.01.2013 00:01:01 | SETI@home | Scheduler request completed: got 0 new tasks
30.01.2013 00:01:01 | SETI@home | Project has no tasks available

SORRY! But messages like this only put more confusion to the user's brain than anything else does! Or is it because of my bad English?
When I started with SETI June 16th 1999 (my first account), I thought SETI would be easy understanding and it would not need a lot of button-pushing.
No it's confusing (see above) and my retry-button is almost kaput!


...Every Tuesday morning (Pacific time) we begin a four hour data distribution outage for database and systems maintenance. The upload/download servers will be offline during this time. Afterwards you may experience connectivity issues for several more hours as the servers catch up with demand. 15 Jan 2013, 17:39:39 UTC

This is nothing new...
-Dave #2

3.2.0-33
ID: 1332636 · Report as offensive
Profile Qui-Gon
Volunteer tester
Avatar

Send message
Joined: 15 May 99
Posts: 2929
Credit: 10,358,480
RAC: 7,583
United States
Message 1332639 - Posted: 30 Jan 2013, 3:17:29 UTC - in response to Message 1332636.  

Results ready to send:1,444,950

AND

30.01.2013 00:01:01 | SETI@home | Scheduler request completed: got 0 new tasks
30.01.2013 00:01:01 | SETI@home | Project has no tasks available

SORRY! But messages like this only put more confusion to the user's brain than anything else does! Or is it because of my bad English?
When I started with SETI June 16th 1999 (my first account), I thought SETI would be easy understanding and it would not need a lot of button-pushing.
No it's confusing (see above) and my retry-button is almost kaput!


...Every Tuesday morning (Pacific time) we begin a four hour data distribution outage for database and systems maintenance. The upload/download servers will be offline during this time. Afterwards you may experience connectivity issues for several more hours as the servers catch up with demand. 15 Jan 2013, 17:39:39 UTC

This is nothing new...

You are correct, Ex, that this is nothing new, but it is happening after I downloaded some work, and while the Server Status page shows 1.4 million "Results" ready to send. It is still confusing and misleading to send a message that there is no work available when so much is shown on the Status page. (And I've been here a while, too.)
ID: 1332639 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 2871
Credit: 10,622,529
RAC: 342
United States
Message 1332642 - Posted: 30 Jan 2013, 3:21:23 UTC
Last modified: 30 Jan 2013, 3:31:33 UTC

I like whatever got fixed during today's maintenance (if that even happened).

For the past few hours, every scheduler contact attempt results in a reply within 3 seconds, and I've been getting a lot of "got 1 new tasks" to go with them, and the AP starts to download, and goes through completion without a single hiccup at 10-15KB/sec. This is more like it.

Oh.. maybe that's because I still have a proxy enabled. Oops. Weird though, because the last time I was using a proxy, I would get HTTP error 417 for scheduler requests, and uploads wouldn't even go through, but everything is working perfectly with this proxy. Weird.


Also.. Holy Ready to Send buffer..

[As of 30 Jan 2013, 3:20:09 UTC]
Results ready to send  1,162,432  12,917 0m

Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1332642 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 7495
Credit: 91,178,764
RAC: 46,088
Australia
Message 1332686 - Posted: 30 Jan 2013, 7:10:10 UTC - in response to Message 1332642.  


AP waiting to validate & assimilate have cleared, the Scheduler is working & the huge Ready to Send buffer is rapidly shrinking down to a more normal size.
*fingers crossed*
Grant
Darwin NT
ID: 1332686 · Report as offensive
rob smithProject Donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 13338
Credit: 154,962,661
RAC: 118,095
United Kingdom
Message 1332689 - Posted: 30 Jan 2013, 7:20:10 UTC - in response to Message 1332639.  


You are correct, Ex, that this is nothing new, but it is happening after I downloaded some work, and while the Server Status page shows 1.4 million "Results" ready to send. It is still confusing and misleading to send a message that there is no work available when so much is shown on the Status page. (And I've been here a while, too.)


The message about "no work available" is certainly confusing, but its a standard message. The work is sent out in small batches of 100, once a batch has been assigned there is a short pause, and if you request work during that pause you get the message about "no work available". Confusing, but correct.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1332689 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 2871
Credit: 10,622,529
RAC: 342
United States
Message 1332690 - Posted: 30 Jan 2013, 7:25:55 UTC
Last modified: 30 Jan 2013, 7:26:39 UTC

Actually, it is 200, and I don't know if it is still every 2 seconds or not. Used to be 100 every 2 seconds for the feeder. If it runs out before the next refill interval, you get "no work available." Mayhaps it should be updated to say "project has no work available at the moment."


Also, it looks like MB is about to run out of tapes to split. I think it would be an interesting experiment to run with AP-only for a day or two to see how Cricket looks. It will probably just stay maxed out though. MB-only just about fills it when the limits are not enforced.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1332690 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 7495
Credit: 91,178,764
RAC: 46,088
Australia
Message 1332692 - Posted: 30 Jan 2013, 7:33:14 UTC - in response to Message 1332690.  


Probably tempting fate here, but even though downloads are as crappy as they have ever been, and it was a longer than usual outage, the Scheduler responses are coming through within 5 seconds in most cases.
Grant
Darwin NT
ID: 1332692 · Report as offensive
Previous · 1 . . . 21 · 22 · 23 · 24 · 25 · Next

Message boards : Number crunching : Panic Mode On (80) Server Problems?


 
©2016 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.