Panic Mode On (101) Server Problems?

Message boards : Number crunching : Panic Mode On (101) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 27 · Next

AuthorMessage
WezH
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 571
Credit: 66,124,963
RAC: 0
Finland
Message 1735671 - Posted: 20 Oct 2015, 9:44:49 UTC - in response to Message 1735660.  
Last modified: 20 Oct 2015, 9:44:49 UTC

Well, 14 hours it did stay online... Not good...
ID: 1735671 · Report as offensive
Profile Sutaru Tsureku
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7104
Credit: 147,329,936
RAC: 1,017
Germany
Message 1735678 - Posted: 20 Oct 2015, 10:34:08 UTC
Last modified: 20 Oct 2015, 10:34:08 UTC

...so does this mean, it would be 'healthier' for the replica database, that we wouldn't use the forum? ;-)
ID: 1735678 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 4812
Credit: 553,017,527
RAC: 1,261,356
United States
Message 1735681 - Posted: 20 Oct 2015, 11:19:07 UTC - in response to Message 1735678.  
Last modified: 20 Oct 2015, 11:19:07 UTC

...so does this mean, it would be 'healthier' for the replica database, that we wouldn't use the forum? ;-)

It probably means whatever was done to the Forum last year succeeded in not only screwing up the 'Resend Lost Tasks' feature but also hit the Replica database as well. The Solution? Turn off Resend Lost Tasks And the Replica. Well, it's Not the solution I would have chose, but, it appears that's the one we are stuck with. It would be nice to discover Why the Resend feature Suddenly caused problems with the Forum when it worked Fine for years, and also why the Forum now seems to hate the Replica as well.
Any suggestions?
;-)
ID: 1735681 · Report as offensive
Profile Zombu2
Volunteer tester

Send message
Joined: 24 Feb 01
Posts: 1615
Credit: 49,315,423
RAC: 0
United States
Message 1735688 - Posted: 20 Oct 2015, 13:03:52 UTC
Last modified: 20 Oct 2015, 13:03:52 UTC

i suggest fixing it
I came down with a bad case of i don't give a crap
ID: 1735688 · Report as offensive
Profile Siran d'Vel'nahr
Volunteer tester
Avatar

Send message
Joined: 23 May 99
Posts: 7001
Credit: 22,207,835
RAC: 85,904
United States
Message 1735691 - Posted: 20 Oct 2015, 13:30:23 UTC - in response to Message 1735688.  
Last modified: 20 Oct 2015, 13:30:23 UTC

i suggest fixing it

And just what do you think they do when a problem arises, sit on their collective hands and watch everyone gripe about it here? :|
CAPT Siran d'Vel'nahr XO - L L & P _\\//
USS Vre'kasht NCC-33187

Windows 10 OS? "What a piece of junk!" - L. Skywalker

ID: 1735691 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 13129
Credit: 148,874,322
RAC: 175,633
United Kingdom
Message 1735703 - Posted: 20 Oct 2015, 14:38:55 UTC - in response to Message 1735681.  
Last modified: 20 Oct 2015, 14:38:55 UTC

It probably means whatever was done to the Forum last year succeeded in not only screwing up the 'Resend Lost Tasks' feature but also hit the Replica database as well. The Solution? Turn off Resend Lost Tasks And the Replica. Well, it's Not the solution I would have chose, but, it appears that's the one we are stuck with. It would be nice to discover Why the Resend feature Suddenly caused problems with the Forum when it worked Fine for years, and also why the Forum now seems to hate the Replica as well.
Any suggestions?
;-)

Well, there's a partial explanation in the description for the attempted fix last night:

web: don't accidentally write to read-only DB replica
BOINC lets you maintain a read-only DB replica,
and certain read-only web pages (like top_users.php)
use the replica in preferences to the master.
But it turns out that these pages aren't actually read-only
because of the too-clever forum_preferences logic.
So sometimes it would add forum_preferences records to the replica,
leading to errors [when] it later copies the same item from the master.

Solution: skip the forum preferences logic if we're using the replica.

98bfcdb2d32719f1dbfa06bc55b58e5d28a7ae61

If your php coding skills are up to writing less-clever forum_preferences logic, I'm sure they would appreciate your (or anybody else's) assistance.
ID: 1735703 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 4812
Credit: 553,017,527
RAC: 1,261,356
United States
Message 1735709 - Posted: 20 Oct 2015, 15:03:01 UTC - in response to Message 1735703.  
Last modified: 20 Oct 2015, 15:09:45 UTC

It probably means whatever was done to the Forum last year succeeded in not only screwing up the 'Resend Lost Tasks' feature but also hit the Replica database as well. The Solution? Turn off Resend Lost Tasks And the Replica. Well, it's Not the solution I would have chose, but, it appears that's the one we are stuck with. It would be nice to discover Why the Resend feature Suddenly caused problems with the Forum when it worked Fine for years, and also why the Forum now seems to hate the Replica as well.
Any suggestions?
;-)

Well, there's a partial explanation in the description for the attempted fix last night:

web: don't accidentally write to read-only DB replica
BOINC lets you maintain a read-only DB replica,
and certain read-only web pages (like top_users.php)
use the replica in preferences to the master.
But it turns out that these pages aren't actually read-only
because of the too-clever forum_preferences logic.
So sometimes it would add forum_preferences records to the replica,
leading to errors [when] it later copies the same item from the master.

Solution: skip the forum preferences logic if we're using the replica.

98bfcdb2d32719f1dbfa06bc55b58e5d28a7ae61

If your php coding skills are up to writing less-clever forum_preferences logic, I'm sure they would appreciate your (or anybody else's) assistance.

I don't think the forum_preferences are very clever. I still have the problem where BOINC tells me my Host location is different from the one on the Website. I just restarted a host a little while ago and it says it's location is none when it is actually listed as School on the Website. It's been set School for a while but keeps reverting to none on reboot. I can set it to something else, then back to School and it will be fine...for a while. If it can't even read the location right, how clever can it be?
Tue 20 Oct 2015 10:11:08 AM EDT | | Starting BOINC client version 7.2.33 for x86_64-pc-linux-gnu
Tue 20 Oct 2015 10:11:08 AM EDT | | Data directory: /home/tbar/BOINC
Tue 20 Oct 2015 10:11:08 AM EDT | | Host name: TBarsIntel
Tue 20 Oct 2015 10:11:08 AM EDT | | OS: Linux: 3.13.0-65-generic
Tue 20 Oct 2015 10:11:08 AM EDT | | Local time is UTC -4 hours
Tue 20 Oct 2015 10:11:08 AM EDT | SETI@home | URL http://setiathome.berkeley.edu/; Computer ID 7258715; resource share 100
Tue 20 Oct 2015 10:11:08 AM EDT | | Host location: none
Tue 20 Oct 2015 10:11:08 AM EDT | | General prefs: using your defaults
Tue 20 Oct 2015 10:11:08 AM EDT | | Reading preferences override file
ID: 1735709 · Report as offensive
Profile Ageless
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 14867
Credit: 4,087,624
RAC: 894
Netherlands
Message 1735713 - Posted: 20 Oct 2015, 15:08:56 UTC - in response to Message 1735709.  
Last modified: 20 Oct 2015, 15:08:56 UTC

I don't think the forum_preferences are very clever. I still have the problem where BOINC tells me my Host location is different from the one on the Website.

You don't tell what the location of your computer is on the forum preferences, but on the web preferences and project preferences.

https://setiathome.berkeley.edu/edit_forum_preferences_form.php is the forum preferences.
Jord

According to Giorgo of the Ancient Astronaut Theorists I do not help with tech questions via private message. He's right: please use the forums for that.
ID: 1735713 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 4812
Credit: 553,017,527
RAC: 1,261,356
United States
Message 1735714 - Posted: 20 Oct 2015, 15:12:49 UTC - in response to Message 1735713.  
Last modified: 20 Oct 2015, 15:21:55 UTC

That's strange, because I can change it the way I ALWAYS have and it will respond the same way it always has. Here, http://setiathome.berkeley.edu/prefs.php?subset=project and Here, http://setiathome.berkeley.edu/show_host_detail.php?hostid=7258715
Tue 20 Oct 2015 11:15:46 AM EDT | SETI@home | New computer location: home
Tue 20 Oct 2015 11:16:57 AM EDT | SETI@home | New computer location: school

Now it's set to School again. See how long that lasts.
All my Hosts have different Locations, the Mac uses None, the other uses School. But it keeps reverting to None for some reason.
ID: 1735714 · Report as offensive
Profile Ageless
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 14867
Credit: 4,087,624
RAC: 894
Netherlands
Message 1735715 - Posted: 20 Oct 2015, 15:21:16 UTC - in response to Message 1735714.  
Last modified: 20 Oct 2015, 15:21:16 UTC

You do not set the location of the computer in the forum preferences. Those preferences are for these message boards and for the way that you want to be notified on subscribed threads.

You do set the location in the project preferences and separately on the details of your computer, but both those pages have N O T H I N G to do with the forum preferences. Click the link I gave in my previous post. See where you end up at.
Jord

According to Giorgo of the Ancient Astronaut Theorists I do not help with tech questions via private message. He's right: please use the forums for that.
ID: 1735715 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 4812
Credit: 553,017,527
RAC: 1,261,356
United States
Message 1735716 - Posted: 20 Oct 2015, 15:25:38 UTC - in response to Message 1735715.  
Last modified: 20 Oct 2015, 15:25:38 UTC

You do not set the location of the computer in the forum preferences. Those preferences are for these message boards and for the way that you want to be notified on subscribed threads.

You do set the location in the project preferences and separately on the details of your computer, but both those pages have N O T H I N G to do with the forum preferences. Click the link I gave in my previous post. See where you end up at.

I see, different preferences. So....the Project Preferences are also messed up?
;-)
ID: 1735716 · Report as offensive
Profile betreger
Avatar

Send message
Joined: 29 Jun 99
Posts: 9435
Credit: 25,341,311
RAC: 21,409
United States
Message 1735724 - Posted: 20 Oct 2015, 20:41:50 UTC
Last modified: 20 Oct 2015, 20:41:50 UTC

We are back from today's outrage and do not have the replica database working, this is not a cause for PANIC BUT instead it is a cause for concern.
ID: 1735724 · Report as offensive
Profile Ageless
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 14867
Credit: 4,087,624
RAC: 894
Netherlands
Message 1735727 - Posted: 20 Oct 2015, 21:13:31 UTC

If all is well, the 'modified' message is gone now.

David Anderson wrote:
Fixed; this bug was introduced in an older commit that wasn't on S@h yet.

Jord

According to Giorgo of the Ancient Astronaut Theorists I do not help with tech questions via private message. He's right: please use the forums for that.
ID: 1735727 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 13129
Credit: 148,874,322
RAC: 175,633
United Kingdom
Message 1735738 - Posted: 20 Oct 2015, 21:33:38 UTC - in response to Message 1735727.  

If all is well, the 'modified' message is gone now.

David Anderson wrote:
Fixed; this bug was introduced in an older commit that wasn't on S@h yet.

Or perhaps:

web: initialize post.modified to zero, not current time

Looking better, anyway.
ID: 1735738 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 10
United States
Message 1735740 - Posted: 20 Oct 2015, 21:36:41 UTC

I should point out after the bug was fixed, the replica was still several days behind, and thus contained some broken commits that it hadn't gotten to yet, hence the continuing crashes even after the fix was implemented.

To solve that problem, and speed things along, I'm recreating the replica from scratch with the backup done during the outage today. Should be on line later this afternoon. THEN we'll see if everything is working well....

- Matt
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 1735740 · Report as offensive
Profile Mike Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 31833
Credit: 74,702,181
RAC: 23,823
Germany
Message 1735748 - Posted: 20 Oct 2015, 21:41:37 UTC

Thanks for the update Matt.
With each crime and every kindness we birth our future.
ID: 1735748 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 13129
Credit: 148,874,322
RAC: 175,633
United Kingdom
Message 1735750 - Posted: 20 Oct 2015, 21:43:24 UTC - in response to Message 1735740.  

I should point out after the bug was fixed, the replica was still several days behind, and thus contained some broken commits that it hadn't gotten to yet, hence the continuing crashes even after the fix was implemented.

To solve that problem, and speed things along, I'm recreating the replica from scratch with the backup done during the outage today. Should be on line later this afternoon. THEN we'll see if everything is working well....

- Matt

I wondered about that, but surely the replica was up-to-date before it went offline the first time.

Then, assuming that it was offline, was the bug so bad that the replica could still be updated in error, despite (a) being read-only, and (b) being offline? Without that, and with the bug being reported as caused by changes to the replica only, I couldn't see how new errors could creep into the system.
ID: 1735750 · Report as offensive
Profile Ageless
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 14867
Credit: 4,087,624
RAC: 894
Netherlands
Message 1735755 - Posted: 20 Oct 2015, 22:02:39 UTC
Last modified: 20 Oct 2015, 22:06:41 UTC

Sniff, something on my system just threw away close to two years of Seti statistics. I have an old backup of the statistics* file, but that dates back to 22 October 2014. A year's worth of data lost. :-(

<save_stats_days> was set to 1461 days, so it should've kept the data. My earliest 'time stamp' is now Fri, 16 Oct 2015 00:00:00 GMT.

The stats file for Pirates@home is still intact and showing all those entries.

Edit: I see Einstein's data got thrown out as well. Everything that contacted homebase on the 16th got reset.
Jord

According to Giorgo of the Ancient Astronaut Theorists I do not help with tech questions via private message. He's right: please use the forums for that.
ID: 1735755 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6115
Credit: 98,517,676
RAC: 46,023
Russia
Message 1735766 - Posted: 20 Oct 2015, 22:38:05 UTC - in response to Message 1735755.  

Sniff, something on my system just threw away close to two years of Seti statistics. I have an old backup of the statistics* file, but that dates back to 22 October 2014. A year's worth of data lost. :-(

<save_stats_days> was set to 1461 days, so it should've kept the data. My earliest 'time stamp' is now Fri, 16 Oct 2015 00:00:00 GMT.

The stats file for Pirates@home is still intact and showing all those entries.

Edit: I see Einstein's data got thrown out as well. Everything that contacted homebase on the 16th got reset.

Had similar issue before on 2 hosts already. At some moment data log disappeared for SETI project while remained for others. number of days long enough to cover next 5 years...
ID: 1735766 · Report as offensive
OTS
Volunteer tester

Send message
Joined: 6 Jan 08
Posts: 360
Credit: 20,533,537
RAC: 0
United States
Message 1735798 - Posted: 21 Oct 2015, 0:23:40 UTC

Replica DB is back on line and only 3,540 seconds behind. That has to be some sort of record. Boincstats.com shows enough credit issued since the last daily update to create a number 2 in my "Best Ten Days".
ID: 1735798 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 27 · Next

Message boards : Number crunching : Panic Mode On (101) Server Problems?


 
©2019 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.