Panic Mode On (109) Server Problems?

Message boards : Number crunching : Panic Mode On (109) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 11 · 12 · 13 · 14 · 15 · 16 · 17 . . . 36 · Next

AuthorMessage
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1909748 - Posted: 31 Dec 2017, 4:07:07 UTC - in response to Message 1909742.  

I'm down to my last 2 gpu tasks on the Linux cruncher. Guess it's going to do Einstein for the rest of the evening.

Even my C2D is struggling to get work today.
Usually it doesn't have a problem, even when the i7 does.
Grant
Darwin NT
ID: 1909748 · Report as offensive
Profile Jeff Buck Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 11 Feb 00
Posts: 1441
Credit: 148,764,870
RAC: 0
United States
Message 1909753 - Posted: 31 Dec 2017, 4:46:08 UTC

This seems eerily like the "Database slowness" (as Eric called it) back on 10 Nov.

We're having some as yet unexplained slowness with the our BOINC database. There don't seem to be any hardware issues. Temperatures are running normal and all the drives seem good. Yet for some reason the query that fills the "ready to send" queue is running about 10 times slower than it normally does.

Until I get it fixed, it means that on average we're sending out 3 workunits a second rather than 30+.
That was a holiday weekend, too.
ID: 1909753 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1909755 - Posted: 31 Dec 2017, 5:09:02 UTC

Can't connect to server for the last 10 minutes.
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | Sending scheduler request: To fetch work.
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | Reporting 2 completed tasks
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | [http] HTTP_OP::init_post(): http://setiboinc.ssl.berkeley.edu/sah_cgi/cgi
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | [http] [ID#1] Info:    Trying 208.68.240.126...
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | [http] [ID#1] Info:  Connected to setiboinc.ssl.berkeley.edu (208.68.240.126) port 80 (#16)
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | [http] [ID#1] Sent header to server: POST /sah_cgi/cgi HTTP/1.1
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | [http] [ID#1] Sent header to server: Host: setiboinc.ssl.berkeley.edu
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | [http] [ID#1] Sent header to server: User-Agent: BOINC client (x86_64-pc-linux-gnu 7.8.3)
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | [http] [ID#1] Sent header to server: Accept: */*
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | [http] [ID#1] Sent header to server: Accept-Encoding: deflate, gzip
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | [http] [ID#1] Sent header to server: Content-Type: application/x-www-form-urlencoded
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | [http] [ID#1] Sent header to server: Accept-Language: en_US
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | [http] [ID#1] Sent header to server: Content-Length: 36896
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | [http] [ID#1] Sent header to server: Expect: 100-continue
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | [http] [ID#1] Sent header to server:
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | 
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | [http] [ID#1] Received header from server: HTTP/1.1 100 Continue
Sat 30 Dec 2017 09:05:59 PM PST | SETI@home | [http] [ID#1] Info:  We are completely uploaded and fine
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: HTTP/1.1 500 Internal Server Error
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: Date: Sun, 31 Dec 2017 05:05:59 GMT
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: Server: Apache/2.2.15 (Scientific Linux)
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: Content-Length: 647
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: Connection: close
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: Content-Type: text/html; charset=iso-8859-1
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server:
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: <html><head>
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: <title>500 Internal Server Error</title>
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: </head><body>
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: <h1>Internal Server Error</h1>
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: <p>The server encountered an internal error or
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: misconfiguration and was unable to complete
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: your request.</p>
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: <p>Please contact the server administrator,
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server:  boincadm@ssl.berkeley.edu and inform them of the time the error occurred,
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: and anything you might have done that may have
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: caused the error.</p>
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: <p>More information about this error may be available
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: in the server error log.</p>
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: <hr>
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: <address>Apache/2.2.15 (Scientific Linux) Server at setiboinc.ssl.berkeley.edu Port 80</address>
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Received header from server: </body></html>
Sat 30 Dec 2017 09:06:40 PM PST |  | [http_xfer] [ID#1] HTTP: wrote 647 bytes
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | [http] [ID#1] Info:  Closing connection 16
Sat 30 Dec 2017 09:06:40 PM PST | SETI@home | Scheduler request failed: HTTP internal server error

Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1909755 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1909767 - Posted: 31 Dec 2017, 6:44:56 UTC - in response to Message 1909753.  

This seems eerily like the "Database slowness" (as Eric called it) back on 10 Nov.

We're having some as yet unexplained slowness with the our BOINC database. There don't seem to be any hardware issues. Temperatures are running normal and all the drives seem good. Yet for some reason the query that fills the "ready to send" queue is running about 10 times slower than it normally does.

Until I get it fixed, it means that on average we're sending out 3 workunits a second rather than 30+.
That was a holiday weekend, too.


. . I guess the servers have a calender with the holidays marked on it ? A shame they are union servers :)

Stephen

:)
ID: 1909767 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1909768 - Posted: 31 Dec 2017, 6:48:15 UTC - in response to Message 1909755.  
Last modified: 31 Dec 2017, 6:48:33 UTC

Can't connect to server for the last 10 minutes.

There are almost always issues with the web site & scheduler around this time of day for anything from just 10min to 45min.
Grant
Darwin NT
ID: 1909768 · Report as offensive
Ghia
Avatar

Send message
Joined: 7 Feb 17
Posts: 238
Credit: 28,911,438
RAC: 50
Norway
Message 1909996 - Posted: 1 Jan 2018, 9:38:20 UTC

Hi, everyone...wish you a Happy New S@H Year !

Just a small question : I have the usual 5 minutes back-off time after a manual update request. Is there supposed to be an automatic scheduler request when that time runs out ? I seem to remember that used to be the case, but now the time just runs out and nothing happens. It may take over 30 minutes before a new scheduler request is sent to the server.
This doesn't have any negative consequences, of course...just finding it weird. I haven't done any changes to Boinc or S@H settings for many months...the only thing out of the ordinary is that my system was down for a week before Christmas when my monitor died.

...Ghia...
Humans may rule the world...but bacteria run it...
ID: 1909996 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1909999 - Posted: 1 Jan 2018, 9:51:01 UTC - in response to Message 1909996.  

Hi, everyone...wish you a Happy New S@H Year !

Just a small question : I have the usual 5 minutes back-off time after a manual update request. Is there supposed to be an automatic scheduler request when that time runs out ?

It depends.
If your cache is full, then it won't make another request till the next WU has been completed and uploaded.
But if your cache isn't full, even if another WU hasn't been completed, then after the 5min 3 sec delay it will ask for work again (although that also depends on your Store up to an additional x days setttings). The larger that value, the longer it will wait (ie the more WUs that will have to be returned) before requesting more work.
Grant
Darwin NT
ID: 1909999 · Report as offensive
Ghia
Avatar

Send message
Joined: 7 Feb 17
Posts: 238
Credit: 28,911,438
RAC: 50
Norway
Message 1910007 - Posted: 1 Jan 2018, 11:31:04 UTC - in response to Message 1909999.  
Last modified: 1 Jan 2018, 11:41:42 UTC

Hi, everyone...wish you a Happy New S@H Year !

Just a small question : I have the usual 5 minutes back-off time after a manual update request. Is there supposed to be an automatic scheduler request when that time runs out ?

It depends.
If your cache is full, then it won't make another request till the next WU has been completed and uploaded.
But if your cache isn't full, even if another WU hasn't been completed, then after the 5min 3 sec delay it will ask for work again (although that also depends on your Store up to an additional x days setttings). The larger that value, the longer it will wait (ie the more WUs that will have to be returned) before requesting more work.


My system is modest (CPU i7-4770K, GPU GeForce 780Ti), and my in progress is isusally 160-180.
Settings are not greedy :
Store at least 1.1 days of work
Store up to an additional 1 days of work

Excerpt from log :
01/01/2018 10:15:45 | SETI@home | update requested by user
01/01/2018 10:15:45 | SETI@home | Sending scheduler request: Requested by user.
01/01/2018 10:15:45 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
01/01/2018 10:15:49 | SETI@home | Scheduler request completed: got 1 new tasks
.
.
01/01/2018 11:18:44 | SETI@home | Finished upload of 31dc07ad.25668.72.3.30.201_0_r839218346_0
01/01/2018 11:22:52 | SETI@home | Sending scheduler request: To report completed tasks.
01/01/2018 11:22:52 | SETI@home | Reporting 16 completed tasks
01/01/2018 11:22:52 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
01/01/2018 11:22:58 | SETI@home | Scheduler request completed: got 15 new tasks
01/01/2018 11:23:00 | SETI@home | Started download of 28mr07ak.1010.1299.7.34.159
.
.
01/01/2018 12:22:58 | SETI@home | Finished upload of 28mr07ak.7480.6616.5.32.49_0_r2027700034_0
01/01/2018 12:24:00 | SETI@home | Sending scheduler request: To report completed tasks.
01/01/2018 12:24:00 | SETI@home | Reporting 20 completed tasks
01/01/2018 12:24:00 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
01/01/2018 12:24:06 | SETI@home | Scheduler request completed: got 2 new tasks
01/01/2018 12:24:08 | SETI@home | Started download of 25ja07ac.4170.10706.16.43.83.vlar

As you can see, the last two requests were more than 1 hour apart, with 20 WUs reported. Is that reasonable ?

Edit : Shouldn't have posted to the panic thread...moderator, please feel free to move it...
Humans may rule the world...but bacteria run it...
ID: 1910007 · Report as offensive
Profile Kissagogo27 Special Project $75 donor
Avatar

Send message
Joined: 6 Nov 99
Posts: 715
Credit: 8,032,827
RAC: 62
France
Message 1910008 - Posted: 1 Jan 2018, 11:39:39 UTC
Last modified: 1 Jan 2018, 11:41:53 UTC

Hi, the additionnal work setting set the time between 2 server call , 1 day means server request every 24h ( without any completed WU report)

u can set 0.01day for request every 15mn
ID: 1910008 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1910020 - Posted: 1 Jan 2018, 13:56:42 UTC - in response to Message 1910008.  

u can set 0.01day for request every 15mn
Or 0.00 for 'whenever needed'. Subject to server backoffs, of course.
ID: 1910020 · Report as offensive
Ghia
Avatar

Send message
Joined: 7 Feb 17
Posts: 238
Credit: 28,911,438
RAC: 50
Norway
Message 1910021 - Posted: 1 Jan 2018, 14:01:23 UTC - in response to Message 1910020.  

u can set 0.01day for request every 15mn
Or 0.00 for 'whenever needed'. Subject to server backoffs, of course.

I see. Tnx, all :-) I just don't know why it worked differently earlier, haha.
Humans may rule the world...but bacteria run it...
ID: 1910021 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1910023 - Posted: 1 Jan 2018, 14:23:01 UTC - in response to Message 1910021.  

I just don't know why it worked differently earlier, haha.
The definitions of 'Store at least' and 'up to an additional' were substantially revised with the launch of BOINC v7, around 2012. If you've been used to BOINC v6 and earlier - it's changed.
ID: 1910023 · Report as offensive
Ghia
Avatar

Send message
Joined: 7 Feb 17
Posts: 238
Credit: 28,911,438
RAC: 50
Norway
Message 1910028 - Posted: 1 Jan 2018, 15:08:00 UTC - in response to Message 1910023.  

I just don't know why it worked differently earlier, haha.
The definitions of 'Store at least' and 'up to an additional' were substantially revised with the launch of BOINC v7, around 2012. If you've been used to BOINC v6 and earlier - it's changed.

No...count me as a newbie :-) I crunched for a while back in the late nineties, but my current Boinc/Seti account is from February 2017.
Humans may rule the world...but bacteria run it...
ID: 1910028 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11358
Credit: 29,581,041
RAC: 66
United States
Message 1910078 - Posted: 1 Jan 2018, 20:08:55 UTC

Does anyone know if we will have an outrage tomorrow or will it be Wednesday as it was last week?
ID: 1910078 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1910079 - Posted: 1 Jan 2018, 20:14:47 UTC - in response to Message 1910078.  

I'd put money on Wednesday
ID: 1910079 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1910102 - Posted: 1 Jan 2018, 21:53:21 UTC

I had the same question. Anyone heard anything official from staff?
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1910102 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1910105 - Posted: 1 Jan 2018, 22:14:08 UTC

If I had to lay odds, I would say the outage would be tomorrow as usual.
Most folks that have been on holiday are back to work tomorrow, including myself.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1910105 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1910106 - Posted: 1 Jan 2018, 22:17:33 UTC - in response to Message 1910102.  

I had the same question. Anyone heard anything official from staff?
They'll tell you on Thursday.
ID: 1910106 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1910107 - Posted: 1 Jan 2018, 22:18:00 UTC - in response to Message 1910105.  

That would be my guess too. Just a normal holiday shortened 4-day work week.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1910107 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1910108 - Posted: 1 Jan 2018, 22:18:56 UTC - in response to Message 1910106.  

I had the same question. Anyone heard anything official from staff?
They'll tell you on Thursday.

Ha Ha LOL True enough sometimes.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1910108 · Report as offensive
Previous · 1 . . . 11 · 12 · 13 · 14 · 15 · 16 · 17 . . . 36 · Next

Message boards : Number crunching : Panic Mode On (109) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.