We are recovering from a database crash


log in

Advanced search

Message boards : News : We are recovering from a database crash

1 · 2 · 3 · 4 . . . 5 · Next
Author Message
Jeff Cobb
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 1 Mar 99
Posts: 110
Credit: 40,367
RAC: 0
United States
Message 1298434 - Posted: 24 Oct 2012, 22:56:13 UTC

The main boinc database suffered a crash this morning. We are back up and running now and are catching up on work distribution.
____________

Jeff Cobb
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 1 Mar 99
Posts: 110
Credit: 40,367
RAC: 0
United States
Message 1298438 - Posted: 24 Oct 2012, 23:01:40 UTC

A few more details...

The machine that the database was on completely hung up and required a hard reset. This introduced corruption into the database on restart.
Luckily, the replica database was OK and caught up. We switched the master/replica relationship between the two machines and are still
recovering (using yesterday's backup) the "new" replica. Once that recovery is complete, we will restart replication.
____________

Profile [seti.international] Dirk SadowskiProject donor
Volunteer tester
Avatar
Send message
Joined: 6 Apr 07
Posts: 7101
Credit: 60,864,596
RAC: 17,228
Germany
Message 1298478 - Posted: 25 Oct 2012, 2:18:40 UTC

Jeff, thanks for the news!


* Best regards! :-) * Sutaru Tsureku, team seti.international founder. * Optimize your PC for higher RAC. * SETI@home needs your help. *
____________
BR

SETI@home Needs your Help ... $10 & U get a Star!

Team seti.international

Das Deutsche Cafe. The German Cafe.

Profile UliProject donor
Volunteer tester
Avatar
Send message
Joined: 6 Feb 00
Posts: 9839
Credit: 5,465,758
RAC: 206
Germany
Message 1298552 - Posted: 25 Oct 2012, 7:04:03 UTC

Thank you Jeff.
____________
Pluto will always be a planet to me.
Order your 15th Seti Anniversary Shirt today. Just PM me for details.
Cash Donation Specialist

Seti Ambassador

Profile S@NL Etienne Dokkum
Volunteer tester
Avatar
Send message
Joined: 11 Jun 99
Posts: 165
Credit: 16,972,265
RAC: 20,698
Netherlands
Message 1298555 - Posted: 25 Oct 2012, 7:32:45 UTC

thank you for the update Jeff.
____________

Profile Chris SProject donor
Volunteer tester
Avatar
Send message
Joined: 19 Nov 00
Posts: 32036
Credit: 13,715,041
RAC: 27,905
United Kingdom
Message 1298566 - Posted: 25 Oct 2012, 8:55:45 UTC

Many thanks for the heads up Jeff, appreciated.

N9JFE David SProject donor
Volunteer tester
Avatar
Send message
Joined: 4 Oct 99
Posts: 11932
Credit: 14,609,767
RAC: 12,441
United States
Message 1298618 - Posted: 25 Oct 2012, 13:08:54 UTC - in response to Message 1298438.

A few more details...

The machine that the database was on completely hung up and required a hard reset. This introduced corruption into the database on restart.

Thanks for the info. This caused even the whole web site to go down?

____________
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.


WezH
Volunteer tester
Send message
Joined: 19 Aug 99
Posts: 111
Credit: 4,386,529
RAC: 27,922
Finland
Message 1298632 - Posted: 25 Oct 2012, 15:14:57 UTC

XML stats hasn't been updated since 23-Oct-2012.

Profile Ronald R CODNEY
Avatar
Send message
Joined: 19 Nov 11
Posts: 87
Credit: 420,497
RAC: 0
United States
Message 1298801 - Posted: 26 Oct 2012, 0:07:28 UTC

Sounds like the gremlins & goblins got a head start, eh?

k1agp
Send message
Joined: 17 Nov 08
Posts: 14
Credit: 128,051
RAC: 0
United States
Message 1301049 - Posted: 1 Nov 2012, 18:27:51 UTC - in response to Message 1298438.


Are things running normal at this point in time? Thank you k1agp

Profile Brother Frank
Send message
Joined: 10 Dec 11
Posts: 26
Credit: 15,142,410
RAC: 0
United States
Message 1301097 - Posted: 1 Nov 2012, 22:54:15 UTC - in response to Message 1301049.

I don't think so. Yesterday I noticed that none of my computers have been able to send completed work in. I have 40 to 60 completed jobs sitting in my task list for each of my two desktops and my powerful notebook and my RAC number has not been modified for a couple of days now on either of those machines. This afternoon I started to get long Communication deferrals of 3 hours or more. That started a little earlier with a 60 minute delay. Things looked ok when the site first came up out of the maintenance period. I haven't been able to find anything posted on this when I search all the message threads, but it appears that something is very wrong with the upload process. I am continuing to process the work I have hoping that it will resolve itself. I notice a number of transient http errors and messages saying the internet connection seems to be down. My computers are reporting jobs completed, but on my more active machines, nothing gets out. My less powerful notebooks seem to post eventually, but they too are having problems with reporting out.
____________
Frank Elliott,Member of Carepages.com,a chronic illness support site. Was FrankLivingFully there.Free user name & pw needed. My Google+ Profile is:
https://profiles.google.com/u/0/10871372137584 Science,SF,Space,Astronomy,Medicine,Psyc Topics.

Profile MilVetRetiredProject donor
Send message
Joined: 24 Apr 12
Posts: 13
Credit: 5,217,337
RAC: 6,954
United States
Message 1301230 - Posted: 2 Nov 2012, 11:55:17 UTC - in response to Message 1301097.

I have four puters sitting idel due to no comms with the servers. My normal RAC is around 12000 but is dropping because of this issue.

http://seti.military-veteran.com/

Les73gtx
Send message
Joined: 13 Feb 03
Posts: 1
Credit: 7,781,460
RAC: 0
United States
Message 1301236 - Posted: 2 Nov 2012, 12:07:36 UTC

I have computers with many work units to report and are sitting Idle today .... and have recieved no work for the last two days. I there a problem somewhere? Server status looks fine but I am not able to do communications with the project
____________

N9JFE David SProject donor
Volunteer tester
Avatar
Send message
Joined: 4 Oct 99
Posts: 11932
Credit: 14,609,767
RAC: 12,441
United States
Message 1301252 - Posted: 2 Nov 2012, 13:19:47 UTC - in response to Message 1301236.

Brother Frank, MilVetRetired, and Les73gtx (and anyone else having trouble with communication), I suggest you check the Number Crunching forum on this message board. I'd start with the thread "Panic Mode On 77[?] Server Problems?", but skip ahead to the last day or two.

____________
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.


Richard
Send message
Joined: 23 Aug 08
Posts: 3
Credit: 2,213,640
RAC: 2,593
United States
Message 1301695 - Posted: 3 Nov 2012, 15:31:45 UTC - in response to Message 1298434.

Good Morning:
I have noticed that "Seti" has been down going on three days. Do you have any idea as to when you will be back on line. Have many completions I need to upload. Have a great weekend and thank you.

Richard Le Vine

Edu Fontana
Send message
Joined: 17 Oct 12
Posts: 1
Credit: 18,097
RAC: 0
Brazil
Message 1301735 - Posted: 3 Nov 2012, 17:15:22 UTC - in response to Message 1298434.

Hi Jeff,

There is more than a week I don't receive data for processing in my pc.
This incident with the Main Boinc Datase is the problem?
Could you/SETI send me datas to process?
Thanks a lot,

Eduardo Fontana
São Paulo, Brazil

Profile Colin Steadman
Avatar
Send message
Joined: 5 Aug 99
Posts: 25
Credit: 3,985,652
RAC: 0
United Kingdom
Message 1301819 - Posted: 3 Nov 2012, 21:31:15 UTC - in response to Message 1301735.

Are you guys still having issues with this incident. I'm getting very few tasks and my PC is sitting idle 90% of the time for the last few days.

Is it me, or is it something your end:

02/11/2012 18:17:10 | SETI@home | Reporting 19 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 02/11/2012 18:19:28 | SETI@home | Computation for task 03oc12ad.29816.12337.140733193388040.10.211_1 finished 02/11/2012 18:19:30 | SETI@home | Started upload of 03oc12ad.29816.12337.140733193388040.10.211_1_0 02/11/2012 18:19:45 | SETI@home | Finished upload of 03oc12ad.29816.12337.140733193388040.10.211_1_0 02/11/2012 18:22:22 | SETI@home | Scheduler request failed: Timeout was reached 02/11/2012 18:22:36 | | Project communication failed: attempting access to reference site 02/11/2012 18:22:38 | | Internet access OK - project servers may be temporarily down. 02/11/2012 18:35:17 | SETI@home | Sending scheduler request: To fetch work. 02/11/2012 18:35:17 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 02/11/2012 18:40:33 | SETI@home | Scheduler request failed: Timeout was reached 02/11/2012 18:40:48 | | Project communication failed: attempting access to reference site 02/11/2012 18:40:50 | | Internet access OK - project servers may be temporarily down. 02/11/2012 19:11:04 | SETI@home | Sending scheduler request: To fetch work. 02/11/2012 19:11:04 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 02/11/2012 19:16:13 | SETI@home | Scheduler request failed: Timeout was reached 02/11/2012 19:16:28 | | Project communication failed: attempting access to reference site 02/11/2012 19:16:30 | | Internet access OK - project servers may be temporarily down. 02/11/2012 20:04:37 | | Suspending computation - an exclusive app is running 02/11/2012 20:04:37 | | Suspending network activity - an exclusive app is running 02/11/2012 20:27:38 | | Resuming computation 02/11/2012 20:27:38 | | Resuming network activity 02/11/2012 20:27:38 | SETI@home | Sending scheduler request: To fetch work. 02/11/2012 20:27:38 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 02/11/2012 20:32:50 | SETI@home | Scheduler request failed: Timeout was reached 02/11/2012 20:33:04 | | Project communication failed: attempting access to reference site 02/11/2012 20:33:07 | | Internet access OK - project servers may be temporarily down. 02/11/2012 20:59:08 | SETI@home | update requested by user 02/11/2012 20:59:10 | SETI@home | Sending scheduler request: Requested by user. 02/11/2012 20:59:10 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 02/11/2012 21:04:18 | SETI@home | Scheduler request failed: Timeout was reached 02/11/2012 21:04:32 | | Project communication failed: attempting access to reference site 02/11/2012 21:04:34 | | Internet access OK - project servers may be temporarily down. 03/11/2012 00:52:32 | SETI@home | Sending scheduler request: To fetch work. 03/11/2012 00:52:32 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 03/11/2012 00:57:47 | SETI@home | Scheduler request failed: Timeout was reached 03/11/2012 00:58:01 | | Project communication failed: attempting access to reference site 03/11/2012 00:58:03 | | Internet access OK - project servers may be temporarily down. 03/11/2012 04:23:47 | SETI@home | Sending scheduler request: To fetch work. 03/11/2012 04:23:47 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 03/11/2012 04:29:11 | SETI@home | Scheduler request failed: Timeout was reached 03/11/2012 04:29:26 | | Project communication failed: attempting access to reference site 03/11/2012 04:29:28 | | Internet access OK - project servers may be temporarily down. 03/11/2012 07:28:45 | SETI@home | Sending scheduler request: To fetch work. 03/11/2012 07:28:45 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 03/11/2012 07:33:56 | SETI@home | Scheduler request failed: Timeout was reached 03/11/2012 07:34:10 | | Project communication failed: attempting access to reference site 03/11/2012 07:34:12 | | Internet access OK - project servers may be temporarily down. 03/11/2012 07:35:07 | SETI@home | Fetching scheduler list 03/11/2012 07:35:09 | SETI@home | Master file download succeeded 03/11/2012 07:35:14 | SETI@home | Sending scheduler request: To fetch work. 03/11/2012 07:35:14 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 03/11/2012 07:40:26 | SETI@home | Scheduler request failed: Timeout was reached 03/11/2012 07:40:41 | | Project communication failed: attempting access to reference site 03/11/2012 07:40:43 | | Internet access OK - project servers may be temporarily down. 03/11/2012 07:42:08 | SETI@home | Sending scheduler request: To fetch work. 03/11/2012 07:42:08 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 03/11/2012 07:42:30 | SETI@home | Scheduler request failed: Couldn't connect to server 03/11/2012 07:42:45 | | Project communication failed: attempting access to reference site 03/11/2012 07:42:46 | | Internet access OK - project servers may be temporarily down. 03/11/2012 07:46:23 | SETI@home | Sending scheduler request: To fetch work. 03/11/2012 07:46:23 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 03/11/2012 07:51:30 | SETI@home | Scheduler request failed: Timeout was reached 03/11/2012 07:51:46 | | Project communication failed: attempting access to reference site 03/11/2012 07:51:48 | | Internet access OK - project servers may be temporarily down. 03/11/2012 07:56:30 | SETI@home | Sending scheduler request: To fetch work. 03/11/2012 07:56:30 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 03/11/2012 08:01:42 | SETI@home | Scheduler request failed: Timeout was reached 03/11/2012 08:01:56 | | Project communication failed: attempting access to reference site 03/11/2012 08:01:58 | | Internet access OK - project servers may be temporarily down. 03/11/2012 08:12:56 | SETI@home | Sending scheduler request: To fetch work. 03/11/2012 08:12:56 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 03/11/2012 08:18:10 | SETI@home | Scheduler request failed: Timeout was reached 03/11/2012 08:18:25 | | Project communication failed: attempting access to reference site 03/11/2012 08:18:27 | | Internet access OK - project servers may be temporarily down. 03/11/2012 08:49:10 | SETI@home | Sending scheduler request: To fetch work. 03/11/2012 08:49:10 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 03/11/2012 08:54:24 | SETI@home | Scheduler request failed: Timeout was reached 03/11/2012 08:54:38 | | Project communication failed: attempting access to reference site 03/11/2012 08:54:40 | | Internet access OK - project servers may be temporarily down. 03/11/2012 09:44:55 | SETI@home | Sending scheduler request: To fetch work. 03/11/2012 09:44:55 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 03/11/2012 09:50:07 | SETI@home | Scheduler request failed: Timeout was reached 03/11/2012 09:50:22 | | Project communication failed: attempting access to reference site 03/11/2012 09:50:24 | | Internet access OK - project servers may be temporarily down. 03/11/2012 11:29:34 | SETI@home | Sending scheduler request: To fetch work. 03/11/2012 11:29:34 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 03/11/2012 11:34:43 | SETI@home | Scheduler request failed: Timeout was reached 03/11/2012 11:34:58 | | Project communication failed: attempting access to reference site 03/11/2012 11:35:00 | | Internet access OK - project servers may be temporarily down. 03/11/2012 14:36:44 | SETI@home | Sending scheduler request: To fetch work. 03/11/2012 14:36:44 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 03/11/2012 14:41:56 | SETI@home | Scheduler request failed: Timeout was reached 03/11/2012 14:42:10 | | Project communication failed: attempting access to reference site 03/11/2012 14:42:12 | | Internet access OK - project servers may be temporarily down. 03/11/2012 18:00:56 | SETI@home | Sending scheduler request: To report completed tasks. 03/11/2012 18:00:56 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 03/11/2012 18:06:04 | SETI@home | Scheduler request failed: Timeout was reached 03/11/2012 18:06:18 | | Project communication failed: attempting access to reference site 03/11/2012 18:06:20 | | Internet access OK - project servers may be temporarily down. 03/11/2012 21:07:39 | SETI@home | Sending scheduler request: To report completed tasks. 03/11/2012 21:07:39 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 03/11/2012 21:12:57 | SETI@home | Scheduler request failed: Timeout was reached 03/11/2012 21:13:11 | | Project communication failed: attempting access to reference site 03/11/2012 21:13:13 | | Internet access OK - project servers may be temporarily down. 03/11/2012 21:14:28 | SETI@home | Fetching scheduler list 03/11/2012 21:14:29 | SETI@home | Master file download succeeded 03/11/2012 21:14:34 | SETI@home | Sending scheduler request: To report completed tasks. 03/11/2012 21:14:34 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 03/11/2012 21:19:53 | SETI@home | Scheduler request failed: Timeout was reached 03/11/2012 21:20:07 | | Project communication failed: attempting access to reference site 03/11/2012 21:20:09 | | Internet access OK - project servers may be temporarily down. 03/11/2012 21:21:14 | SETI@home | Sending scheduler request: To report completed tasks. 03/11/2012 21:21:14 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI 03/11/2012 21:24:19 | SETI@home | update requested by user 03/11/2012 21:26:22 | SETI@home | Scheduler request failed: Timeout was reached 03/11/2012 21:26:36 | | Project communication failed: attempting access to reference site 03/11/2012 21:26:39 | | Internet access OK - project servers may be temporarily down. 03/11/2012 21:30:15 | SETI@home | Sending scheduler request: To report completed tasks. 03/11/2012 21:30:15 | SETI@home | Reporting 20 completed tasks, requesting new tasks for CPU and NVIDIA and ATI

____________

The Mom
Send message
Joined: 16 Aug 10
Posts: 4
Credit: 4,024,893
RAC: 1,052
United States
Message 1301839 - Posted: 3 Nov 2012, 22:15:20 UTC

I'm having same issue as others who are reporting no uploads or downloads for the last few days. I currently have 45 completed units but when I click "Update", I eventually get a "communication deferred xx:xx:xx" I checked the server status and it looks like most all of them are up and running. The only ones "not running" are ntpckr_small_sig1, ntpckr_small_sig2, ntpckr_small_sig3, and ntpckr_small_sig4. The only ones "disabled" are rfi_small_sig1, rfi_small_sig2, rfi_small_sig3, and rfi_small_sig4. So, why no communication?

Dick Keeler
Send message
Joined: 15 May 99
Posts: 2
Credit: 6,578,495
RAC: 1,770
Cyprus
Message 1301971 - Posted: 4 Nov 2012, 5:18:32 UTC

The last message on the Message board stating "We are recovering from a database crash" is dated the 24 Oct. No messages since. I too cannot send completed units or received new ones. Perhaps everyone at Seti has gone on holiday and forgot to mention that to all of their volunteer users. Guess we will have to wait and see what happens next.
____________

Profile ivan
Volunteer tester
Avatar
Send message
Joined: 5 Mar 01
Posts: 621
Credit: 142,886,250
RAC: 145,701
United Kingdom
Message 1302065 - Posted: 4 Nov 2012, 13:08:57 UTC - in response to Message 1301971.

One trick some of us have found useful is to go into the Projects tab in boincmgr, select seti@home and then click on No New Tasks and then Update. Seems that the timeouts occur when the scheduler is asking for new tasks. Click on Allow New Tasks once the scheduler request completes, so you get more work when the blockage clears.
____________

1 · 2 · 3 · 4 . . . 5 · Next

Message boards : News : We are recovering from a database crash

Copyright © 2014 University of California