We are recovering from a database crash


log in

Advanced search

Message boards : News : We are recovering from a database crash

1 · 2 · 3 · 4 . . . 5 · Next
Author Message
Jeff Cobb
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 1 Mar 99
Posts: 110
Credit: 40,367
RAC: 0
United States
Message 1298434 - Posted: 24 Oct 2012, 22:56:13 UTC

The main boinc database suffered a crash this morning. We are back up and running now and are catching up on work distribution.
____________

Jeff Cobb
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 1 Mar 99
Posts: 110
Credit: 40,367
RAC: 0
United States
Message 1298438 - Posted: 24 Oct 2012, 23:01:40 UTC

A few more details...

The machine that the database was on completely hung up and required a hard reset. This introduced corruption into the database on restart.
Luckily, the replica database was OK and caught up. We switched the master/replica relationship between the two machines and are still
recovering (using yesterday's backup) the "new" replica. Once that recovery is complete, we will restart replication.
____________

Profile [seti.international] Dirk SadowskiProject donor
Volunteer tester
Avatar
Send message
Joined: 6 Apr 07
Posts: 7061
Credit: 59,996,253
RAC: 21,513
Germany
Message 1298478 - Posted: 25 Oct 2012, 2:18:40 UTC

Jeff, thanks for the news!


* Best regards! :-) * Sutaru Tsureku, team seti.international founder. * Optimize your PC for higher RAC. * SETI@home needs your help. *
____________
BR



>Das Deutsche Cafe. The German Cafe.<

Profile UliProject donor
Volunteer tester
Avatar
Send message
Joined: 6 Feb 00
Posts: 9711
Credit: 5,455,637
RAC: 4,398
Germany
Message 1298552 - Posted: 25 Oct 2012, 7:04:03 UTC

Thank you Jeff.
____________
Pluto will always be a planet to me.
Order your 15th Seti Anniversary Shirt today. Just PM me for details.
Cash Donation Specialist

Seti Ambassador

Profile S@NL Etienne Dokkum
Volunteer tester
Avatar
Send message
Joined: 11 Jun 99
Posts: 159
Credit: 15,834,828
RAC: 18,048
Netherlands
Message 1298555 - Posted: 25 Oct 2012, 7:32:45 UTC

thank you for the update Jeff.
____________

Profile Chris SProject donor
Volunteer tester
Avatar
Send message
Joined: 19 Nov 00
Posts: 31452
Credit: 12,176,760
RAC: 28,998
United Kingdom
Message 1298566 - Posted: 25 Oct 2012, 8:55:45 UTC

Many thanks for the heads up Jeff, appreciated.

N9JFE David SProject donor
Volunteer tester
Avatar
Send message
Joined: 4 Oct 99
Posts: 11162
Credit: 13,950,924
RAC: 12,479
United States
Message 1298618 - Posted: 25 Oct 2012, 13:08:54 UTC - in response to Message 1298438.

A few more details...

The machine that the database was on completely hung up and required a hard reset. This introduced corruption into the database on restart.

Thanks for the info. This caused even the whole web site to go down?

____________
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.


WezH
Volunteer tester
Send message
Joined: 19 Aug 99
Posts: 85
Credit: 3,471,667
RAC: 10,103
Finland
Message 1298632 - Posted: 25 Oct 2012, 15:14:57 UTC

XML stats hasn't been updated since 23-Oct-2012.

Profile Ronald R CODNEY
Avatar
Send message
Joined: 19 Nov 11
Posts: 87
Credit: 420,497
RAC: 0
United States
Message 1298801 - Posted: 26 Oct 2012, 0:07:28 UTC

Sounds like the gremlins & goblins got a head start, eh?

k1agp
Send message
Joined: 17 Nov 08
Posts: 14
Credit: 128,051
RAC: 0
United States
Message 1301049 - Posted: 1 Nov 2012, 18:27:51 UTC - in response to Message 1298438.


Are things running normal at this point in time? Thank you k1agp

Profile Brother Frank
Send message
Joined: 10 Dec 11
Posts: 26
Credit: 15,142,410
RAC: 0
United States
Message 1301097 - Posted: 1 Nov 2012, 22:54:15 UTC - in response to Message 1301049.

I don't think so. Yesterday I noticed that none of my computers have been able to send completed work in. I have 40 to 60 completed jobs sitting in my task list for each of my two desktops and my powerful notebook and my RAC number has not been modified for a couple of days now on either of those machines. This afternoon I started to get long Communication deferrals of 3 hours or more. That started a little earlier with a 60 minute delay. Things looked ok when the site first came up out of the maintenance period. I haven't been able to find anything posted on this when I search all the message threads, but it appears that something is very wrong with the upload process. I am continuing to process the work I have hoping that it will resolve itself. I notice a number of transient http errors and messages saying the internet connection seems to be down. My computers are reporting jobs completed, but on my more active machines, nothing gets out. My less powerful notebooks seem to post eventually, but they too are having problems with reporting out.
____________
Frank Elliott,Member of Carepages.com,a chronic illness support site. Was FrankLivingFully there.Free user name & pw needed. My Google+ Profile is:
https://profiles.google.com/u/0/10871372137584 Science,SF,Space,Astronomy,Medicine,Psyc Topics.

Profile MilVetRetired
Send message
Joined: 24 Apr 12
Posts: 12
Credit: 4,894,397
RAC: 5,752
United States
Message 1301230 - Posted: 2 Nov 2012, 11:55:17 UTC - in response to Message 1301097.

I have four puters sitting idel due to no comms with the servers. My normal RAC is around 12000 but is dropping because of this issue.

http://seti.military-veteran.com/

Les73gtx
Send message
Joined: 13 Feb 03
Posts: 1
Credit: 7,781,460
RAC: 0
United States
Message 1301236 - Posted: 2 Nov 2012, 12:07:36 UTC

I have computers with many work units to report and are sitting Idle today .... and have recieved no work for the last two days. I there a problem somewhere? Server status looks fine but I am not able to do communications with the project
____________

N9JFE David SProject donor
Volunteer tester
Avatar
Send message
Joined: 4 Oct 99
Posts: 11162
Credit: 13,950,924
RAC: 12,479
United States
Message 1301252 - Posted: 2 Nov 2012, 13:19:47 UTC - in response to Message 1301236.

Brother Frank, MilVetRetired, and Les73gtx (and anyone else having trouble with communication), I suggest you check the Number Crunching forum on this message board. I'd start with the thread "Panic Mode On 77[?] Server Problems?", but skip ahead to the last day or two.

____________
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.


Richard
Send message
Joined: 23 Aug 08
Posts: 3
Credit: 2,088,124
RAC: 2,696
United States
Message 1301695 - Posted: 3 Nov 2012, 15:31:45 UTC - in response to Message 1298434.

Good Morning:
I have noticed that "Seti" has been down going on three days. Do you have any idea as to when you will be back on line. Have many completions I need to upload. Have a great weekend and thank you.

Richard Le Vine

Edu Fontana
Send message
Joined: 17 Oct 12
Posts: 1
Credit: 18,097
RAC: 0
Brazil
Message 1301735 - Posted: 3 Nov 2012, 17:15:22 UTC - in response to Message 1298434.

Hi Jeff,

There is more than a week I don't receive data for processing in my pc.
This incident with the Main Boinc Datase is the problem?
Could you/SETI send me datas to process?
Thanks a lot,

Eduardo Fontana
São Paulo, Brazil

Profile Colin Steadman
Avatar
Send message
Joined: 5 Aug 99
Posts: 25
Credit: 3,985,652
RAC: 0
United Kingdom
Message 1301819 - Posted: 3 Nov 2012, 21:31:15 UTC - in response to Message 1301735.

Are you guys still having issues with this incident. I'm getting very few tasks and my PC is sitting idle 90% of the time for the last few days.

Is it me, or is it something your end:

02/11/2012 18:17:10 | SETI@home | Reporting 19 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
02/11/2012 18:19:28 | SETI@home | Computation for task 03oc12ad.29816.12337.140733193388040.10.211_1 finished
02/11/2012 18:19:30 | SETI@home | Started upload of 03oc12ad.29816.12337.140733193388040.10.211_1_0
02/11/2012 18:19:45 | SETI@home | Finished upload of 03oc12ad.29816.12337.140733193388040.10.211_1_0
02/11/2012 18:22:22 | SETI@home | Scheduler request failed: Timeout was reached
02/11/2012 18:22:36 | | Project communication failed: attempting access to reference site
02/11/2012 18:22:38 | | Internet access OK - project servers may be temporarily down.
02/11/2012 18:35:17 | SETI@home | Sending scheduler request: To fetch work.
02/11/2012 18:35:17 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
02/11/2012 18:40:33 | SETI@home | Scheduler request failed: Timeout was reached
02/11/2012 18:40:48 | | Project communication failed: attempting access to reference site
02/11/2012 18:40:50 | | Internet access OK - project servers may be temporarily down.
02/11/2012 19:11:04 | SETI@home | Sending scheduler request: To fetch work.
02/11/2012 19:11:04 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
02/11/2012 19:16:13 | SETI@home | Scheduler request failed: Timeout was reached
02/11/2012 19:16:28 | | Project communication failed: attempting access to reference site
02/11/2012 19:16:30 | | Internet access OK - project servers may be temporarily down.
02/11/2012 20:04:37 | | Suspending computation - an exclusive app is running
02/11/2012 20:04:37 | | Suspending network activity - an exclusive app is running
02/11/2012 20:27:38 | | Resuming computation
02/11/2012 20:27:38 | | Resuming network activity
02/11/2012 20:27:38 | SETI@home | Sending scheduler request: To fetch work.
02/11/2012 20:27:38 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
02/11/2012 20:32:50 | SETI@home | Scheduler request failed: Timeout was reached
02/11/2012 20:33:04 | | Project communication failed: attempting access to reference site
02/11/2012 20:33:07 | | Internet access OK - project servers may be temporarily down.
02/11/2012 20:59:08 | SETI@home | update requested by user
02/11/2012 20:59:10 | SETI@home | Sending scheduler request: Requested by user.
02/11/2012 20:59:10 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
02/11/2012 21:04:18 | SETI@home | Scheduler request failed: Timeout was reached
02/11/2012 21:04:32 | | Project communication failed: attempting access to reference site
02/11/2012 21:04:34 | | Internet access OK - project servers may be temporarily down.
03/11/2012 00:52:32 | SETI@home | Sending scheduler request: To fetch work.
03/11/2012 00:52:32 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
03/11/2012 00:57:47 | SETI@home | Scheduler request failed: Timeout was reached
03/11/2012 00:58:01 | | Project communication failed: attempting access to reference site
03/11/2012 00:58:03 | | Internet access OK - project servers may be temporarily down.
03/11/2012 04:23:47 | SETI@home | Sending scheduler request: To fetch work.
03/11/2012 04:23:47 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
03/11/2012 04:29:11 | SETI@home | Scheduler request failed: Timeout was reached
03/11/2012 04:29:26 | | Project communication failed: attempting access to reference site
03/11/2012 04:29:28 | | Internet access OK - project servers may be temporarily down.
03/11/2012 07:28:45 | SETI@home | Sending scheduler request: To fetch work.
03/11/2012 07:28:45 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
03/11/2012 07:33:56 | SETI@home | Scheduler request failed: Timeout was reached
03/11/2012 07:34:10 | | Project communication failed: attempting access to reference site
03/11/2012 07:34:12 | | Internet access OK - project servers may be temporarily down.
03/11/2012 07:35:07 | SETI@home | Fetching scheduler list
03/11/2012 07:35:09 | SETI@home | Master file download succeeded
03/11/2012 07:35:14 | SETI@home | Sending scheduler request: To fetch work.
03/11/2012 07:35:14 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
03/11/2012 07:40:26 | SETI@home | Scheduler request failed: Timeout was reached
03/11/2012 07:40:41 | | Project communication failed: attempting access to reference site
03/11/2012 07:40:43 | | Internet access OK - project servers may be temporarily down.
03/11/2012 07:42:08 | SETI@home | Sending scheduler request: To fetch work.
03/11/2012 07:42:08 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
03/11/2012 07:42:30 | SETI@home | Scheduler request failed: Couldn't connect to server
03/11/2012 07:42:45 | | Project communication failed: attempting access to reference site
03/11/2012 07:42:46 | | Internet access OK - project servers may be temporarily down.
03/11/2012 07:46:23 | SETI@home | Sending scheduler request: To fetch work.
03/11/2012 07:46:23 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
03/11/2012 07:51:30 | SETI@home | Scheduler request failed: Timeout was reached
03/11/2012 07:51:46 | | Project communication failed: attempting access to reference site
03/11/2012 07:51:48 | | Internet access OK - project servers may be temporarily down.
03/11/2012 07:56:30 | SETI@home | Sending scheduler request: To fetch work.
03/11/2012 07:56:30 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
03/11/2012 08:01:42 | SETI@home | Scheduler request failed: Timeout was reached
03/11/2012 08:01:56 | | Project communication failed: attempting access to reference site
03/11/2012 08:01:58 | | Internet access OK - project servers may be temporarily down.
03/11/2012 08:12:56 | SETI@home | Sending scheduler request: To fetch work.
03/11/2012 08:12:56 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
03/11/2012 08:18:10 | SETI@home | Scheduler request failed: Timeout was reached
03/11/2012 08:18:25 | | Project communication failed: attempting access to reference site
03/11/2012 08:18:27 | | Internet access OK - project servers may be temporarily down.
03/11/2012 08:49:10 | SETI@home | Sending scheduler request: To fetch work.
03/11/2012 08:49:10 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
03/11/2012 08:54:24 | SETI@home | Scheduler request failed: Timeout was reached
03/11/2012 08:54:38 | | Project communication failed: attempting access to reference site
03/11/2012 08:54:40 | | Internet access OK - project servers may be temporarily down.
03/11/2012 09:44:55 | SETI@home | Sending scheduler request: To fetch work.
03/11/2012 09:44:55 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
03/11/2012 09:50:07 | SETI@home | Scheduler request failed: Timeout was reached
03/11/2012 09:50:22 | | Project communication failed: attempting access to reference site
03/11/2012 09:50:24 | | Internet access OK - project servers may be temporarily down.
03/11/2012 11:29:34 | SETI@home | Sending scheduler request: To fetch work.
03/11/2012 11:29:34 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
03/11/2012 11:34:43 | SETI@home | Scheduler request failed: Timeout was reached
03/11/2012 11:34:58 | | Project communication failed: attempting access to reference site
03/11/2012 11:35:00 | | Internet access OK - project servers may be temporarily down.
03/11/2012 14:36:44 | SETI@home | Sending scheduler request: To fetch work.
03/11/2012 14:36:44 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
03/11/2012 14:41:56 | SETI@home | Scheduler request failed: Timeout was reached
03/11/2012 14:42:10 | | Project communication failed: attempting access to reference site
03/11/2012 14:42:12 | | Internet access OK - project servers may be temporarily down.
03/11/2012 18:00:56 | SETI@home | Sending scheduler request: To report completed tasks.
03/11/2012 18:00:56 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
03/11/2012 18:06:04 | SETI@home | Scheduler request failed: Timeout was reached
03/11/2012 18:06:18 | | Project communication failed: attempting access to reference site
03/11/2012 18:06:20 | | Internet access OK - project servers may be temporarily down.
03/11/2012 21:07:39 | SETI@home | Sending scheduler request: To report completed tasks.
03/11/2012 21:07:39 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
03/11/2012 21:12:57 | SETI@home | Scheduler request failed: Timeout was reached
03/11/2012 21:13:11 | | Project communication failed: attempting access to reference site
03/11/2012 21:13:13 | | Internet access OK - project servers may be temporarily down.
03/11/2012 21:14:28 | SETI@home | Fetching scheduler list
03/11/2012 21:14:29 | SETI@home | Master file download succeeded
03/11/2012 21:14:34 | SETI@home | Sending scheduler request: To report completed tasks.
03/11/2012 21:14:34 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
03/11/2012 21:19:53 | SETI@home | Scheduler request failed: Timeout was reached
03/11/2012 21:20:07 | | Project communication failed: attempting access to reference site
03/11/2012 21:20:09 | | Internet access OK - project servers may be temporarily down.
03/11/2012 21:21:14 | SETI@home | Sending scheduler request: To report completed tasks.
03/11/2012 21:21:14 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI
03/11/2012 21:24:19 | SETI@home | update requested by user
03/11/2012 21:26:22 | SETI@home | Scheduler request failed: Timeout was reached
03/11/2012 21:26:36 | | Project communication failed: attempting access to reference site
03/11/2012 21:26:39 | | Internet access OK - project servers may be temporarily down.
03/11/2012 21:30:15 | SETI@home | Sending scheduler request: To report completed tasks.
03/11/2012 21:30:15 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI

____________

The Mom
Send message
Joined: 16 Aug 10
Posts: 4
Credit: 3,936,932
RAC: 1,344
United States
Message 1301839 - Posted: 3 Nov 2012, 22:15:20 UTC

I'm having same issue as others who are reporting no uploads or downloads for the last few days. I currently have 45 completed units but when I click "Update", I eventually get a "communication deferred xx:xx:xx" I checked the server status and it looks like most all of them are up and running. The only ones "not running" are ntpckr_small_sig1, ntpckr_small_sig2, ntpckr_small_sig3, and ntpckr_small_sig4. The only ones "disabled" are rfi_small_sig1, rfi_small_sig2, rfi_small_sig3, and rfi_small_sig4. So, why no communication?

Dick Keeler
Send message
Joined: 15 May 99
Posts: 2
Credit: 6,426,317
RAC: 3,828
Cyprus
Message 1301971 - Posted: 4 Nov 2012, 5:18:32 UTC

The last message on the Message board stating "We are recovering from a database crash" is dated the 24 Oct. No messages since. I too cannot send completed units or received new ones. Perhaps everyone at Seti has gone on holiday and forgot to mention that to all of their volunteer users. Guess we will have to wait and see what happens next.
____________

Profile ivan
Volunteer tester
Avatar
Send message
Joined: 5 Mar 01
Posts: 602
Credit: 135,497,456
RAC: 131,481
United Kingdom
Message 1302065 - Posted: 4 Nov 2012, 13:08:57 UTC - in response to Message 1301971.

One trick some of us have found useful is to go into the Projects tab in boincmgr, select seti@home and then click on No New Tasks and then Update. Seems that the timeouts occur when the scheduler is asking for new tasks. Click on Allow New Tasks once the scheduler request completes, so you get more work when the blockage clears.
____________

1 · 2 · 3 · 4 . . . 5 · Next

Message boards : News : We are recovering from a database crash

Copyright © 2014 University of California