Panic Mode On (101) Server Problems?

Message boards : Number crunching : Panic Mode On (101) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 27 · Next

AuthorMessage
Profile Sutaru Tsureku
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 1735678 - Posted: 20 Oct 2015, 10:34:08 UTC
Last modified: 20 Oct 2015, 10:34:08 UTC

...so does this mean, it would be 'healthier' for the replica database, that we wouldn't use the forum? ;-)
ID: 1735678 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1735681 - Posted: 20 Oct 2015, 11:19:07 UTC - in response to Message 1735678.  
Last modified: 20 Oct 2015, 11:19:07 UTC

...so does this mean, it would be 'healthier' for the replica database, that we wouldn't use the forum? ;-)

It probably means whatever was done to the Forum last year succeeded in not only screwing up the 'Resend Lost Tasks' feature but also hit the Replica database as well. The Solution? Turn off Resend Lost Tasks And the Replica. Well, it's Not the solution I would have chose, but, it appears that's the one we are stuck with. It would be nice to discover Why the Resend feature Suddenly caused problems with the Forum when it worked Fine for years, and also why the Forum now seems to hate the Replica as well.
Any suggestions?
;-)
ID: 1735681 · Report as offensive
Profile Zombu2
Volunteer tester

Send message
Joined: 24 Feb 01
Posts: 1615
Credit: 49,315,423
RAC: 0
United States
Message 1735688 - Posted: 20 Oct 2015, 13:03:52 UTC
Last modified: 20 Oct 2015, 13:03:52 UTC

i suggest fixing it
I came down with a bad case of i don't give a crap
ID: 1735688 · Report as offensive
Profile Siran d'Vel'nahr
Volunteer tester
Avatar

Send message
Joined: 23 May 99
Posts: 7379
Credit: 44,181,323
RAC: 238
United States
Message 1735691 - Posted: 20 Oct 2015, 13:30:23 UTC - in response to Message 1735688.  
Last modified: 20 Oct 2015, 13:30:23 UTC

i suggest fixing it

And just what do you think they do when a problem arises, sit on their collective hands and watch everyone gripe about it here? :|
CAPT Siran d'Vel'nahr - L L & P _\\//
Winders 11 OS? "What a piece of junk!" - L. Skywalker
"Logic is the cement of our civilization with which we ascend from chaos using reason as our guide." - T'Plana-hath
ID: 1735691 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1735703 - Posted: 20 Oct 2015, 14:38:55 UTC - in response to Message 1735681.  
Last modified: 20 Oct 2015, 14:38:55 UTC

It probably means whatever was done to the Forum last year succeeded in not only screwing up the 'Resend Lost Tasks' feature but also hit the Replica database as well. The Solution? Turn off Resend Lost Tasks And the Replica. Well, it's Not the solution I would have chose, but, it appears that's the one we are stuck with. It would be nice to discover Why the Resend feature Suddenly caused problems with the Forum when it worked Fine for years, and also why the Forum now seems to hate the Replica as well.
Any suggestions?
;-)

Well, there's a partial explanation in the description for the attempted fix last night:

web: don't accidentally write to read-only DB replica
BOINC lets you maintain a read-only DB replica,
and certain read-only web pages (like top_users.php)
use the replica in preferences to the master.
But it turns out that these pages aren't actually read-only
because of the too-clever forum_preferences logic.
So sometimes it would add forum_preferences records to the replica,
leading to errors [when] it later copies the same item from the master.

Solution: skip the forum preferences logic if we're using the replica.

98bfcdb2d32719f1dbfa06bc55b58e5d28a7ae61

If your php coding skills are up to writing less-clever forum_preferences logic, I'm sure they would appreciate your (or anybody else's) assistance.
ID: 1735703 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1735709 - Posted: 20 Oct 2015, 15:03:01 UTC - in response to Message 1735703.  
Last modified: 20 Oct 2015, 15:09:45 UTC

It probably means whatever was done to the Forum last year succeeded in not only screwing up the 'Resend Lost Tasks' feature but also hit the Replica database as well. The Solution? Turn off Resend Lost Tasks And the Replica. Well, it's Not the solution I would have chose, but, it appears that's the one we are stuck with. It would be nice to discover Why the Resend feature Suddenly caused problems with the Forum when it worked Fine for years, and also why the Forum now seems to hate the Replica as well.
Any suggestions?
;-)

Well, there's a partial explanation in the description for the attempted fix last night:

web: don't accidentally write to read-only DB replica
BOINC lets you maintain a read-only DB replica,
and certain read-only web pages (like top_users.php)
use the replica in preferences to the master.
But it turns out that these pages aren't actually read-only
because of the too-clever forum_preferences logic.
So sometimes it would add forum_preferences records to the replica,
leading to errors [when] it later copies the same item from the master.

Solution: skip the forum preferences logic if we're using the replica.

98bfcdb2d32719f1dbfa06bc55b58e5d28a7ae61

If your php coding skills are up to writing less-clever forum_preferences logic, I'm sure they would appreciate your (or anybody else's) assistance.

I don't think the forum_preferences are very clever. I still have the problem where BOINC tells me my Host location is different from the one on the Website. I just restarted a host a little while ago and it says it's location is none when it is actually listed as School on the Website. It's been set School for a while but keeps reverting to none on reboot. I can set it to something else, then back to School and it will be fine...for a while. If it can't even read the location right, how clever can it be?
Tue 20 Oct 2015 10:11:08 AM EDT | | Starting BOINC client version 7.2.33 for x86_64-pc-linux-gnu
Tue 20 Oct 2015 10:11:08 AM EDT | | Data directory: /home/tbar/BOINC
Tue 20 Oct 2015 10:11:08 AM EDT | | Host name: TBarsIntel
Tue 20 Oct 2015 10:11:08 AM EDT | | OS: Linux: 3.13.0-65-generic
Tue 20 Oct 2015 10:11:08 AM EDT | | Local time is UTC -4 hours
Tue 20 Oct 2015 10:11:08 AM EDT | SETI@home | URL http://setiathome.berkeley.edu/; Computer ID 7258715; resource share 100
Tue 20 Oct 2015 10:11:08 AM EDT | | Host location: none
Tue 20 Oct 2015 10:11:08 AM EDT | | General prefs: using your defaults
Tue 20 Oct 2015 10:11:08 AM EDT | | Reading preferences override file
ID: 1735709 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1735713 - Posted: 20 Oct 2015, 15:08:56 UTC - in response to Message 1735709.  
Last modified: 20 Oct 2015, 15:08:56 UTC

I don't think the forum_preferences are very clever. I still have the problem where BOINC tells me my Host location is different from the one on the Website.

You don't tell what the location of your computer is on the forum preferences, but on the web preferences and project preferences.

https://setiathome.berkeley.edu/edit_forum_preferences_form.php is the forum preferences.
ID: 1735713 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1735714 - Posted: 20 Oct 2015, 15:12:49 UTC - in response to Message 1735713.  
Last modified: 20 Oct 2015, 15:21:55 UTC

That's strange, because I can change it the way I ALWAYS have and it will respond the same way it always has. Here, http://setiathome.berkeley.edu/prefs.php?subset=project and Here, http://setiathome.berkeley.edu/show_host_detail.php?hostid=7258715
Tue 20 Oct 2015 11:15:46 AM EDT | SETI@home | New computer location: home
Tue 20 Oct 2015 11:16:57 AM EDT | SETI@home | New computer location: school

Now it's set to School again. See how long that lasts.
All my Hosts have different Locations, the Mac uses None, the other uses School. But it keeps reverting to None for some reason.
ID: 1735714 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1735715 - Posted: 20 Oct 2015, 15:21:16 UTC - in response to Message 1735714.  
Last modified: 20 Oct 2015, 15:21:16 UTC

You do not set the location of the computer in the forum preferences. Those preferences are for these message boards and for the way that you want to be notified on subscribed threads.

You do set the location in the project preferences and separately on the details of your computer, but both those pages have N O T H I N G to do with the forum preferences. Click the link I gave in my previous post. See where you end up at.
ID: 1735715 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1735716 - Posted: 20 Oct 2015, 15:25:38 UTC - in response to Message 1735715.  
Last modified: 20 Oct 2015, 15:25:38 UTC

You do not set the location of the computer in the forum preferences. Those preferences are for these message boards and for the way that you want to be notified on subscribed threads.

You do set the location in the project preferences and separately on the details of your computer, but both those pages have N O T H I N G to do with the forum preferences. Click the link I gave in my previous post. See where you end up at.

I see, different preferences. So....the Project Preferences are also messed up?
;-)
ID: 1735716 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 1735724 - Posted: 20 Oct 2015, 20:41:50 UTC
Last modified: 20 Oct 2015, 20:41:50 UTC

We are back from today's outrage and do not have the replica database working, this is not a cause for PANIC BUT instead it is a cause for concern.
ID: 1735724 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1735727 - Posted: 20 Oct 2015, 21:13:31 UTC

If all is well, the 'modified' message is gone now.

David Anderson wrote:
Fixed; this bug was introduced in an older commit that wasn't on S@h yet.

ID: 1735727 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1735738 - Posted: 20 Oct 2015, 21:33:38 UTC - in response to Message 1735727.  

If all is well, the 'modified' message is gone now.

David Anderson wrote:
Fixed; this bug was introduced in an older commit that wasn't on S@h yet.

Or perhaps:

web: initialize post.modified to zero, not current time

Looking better, anyway.
ID: 1735738 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 1735740 - Posted: 20 Oct 2015, 21:36:41 UTC

I should point out after the bug was fixed, the replica was still several days behind, and thus contained some broken commits that it hadn't gotten to yet, hence the continuing crashes even after the fix was implemented.

To solve that problem, and speed things along, I'm recreating the replica from scratch with the backup done during the outage today. Should be on line later this afternoon. THEN we'll see if everything is working well....

- Matt
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 1735740 · Report as offensive
Profile Mike Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 34255
Credit: 79,922,639
RAC: 80
Germany
Message 1735748 - Posted: 20 Oct 2015, 21:41:37 UTC

Thanks for the update Matt.


With each crime and every kindness we birth our future.
ID: 1735748 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1735750 - Posted: 20 Oct 2015, 21:43:24 UTC - in response to Message 1735740.  

I should point out after the bug was fixed, the replica was still several days behind, and thus contained some broken commits that it hadn't gotten to yet, hence the continuing crashes even after the fix was implemented.

To solve that problem, and speed things along, I'm recreating the replica from scratch with the backup done during the outage today. Should be on line later this afternoon. THEN we'll see if everything is working well....

- Matt

I wondered about that, but surely the replica was up-to-date before it went offline the first time.

Then, assuming that it was offline, was the bug so bad that the replica could still be updated in error, despite (a) being read-only, and (b) being offline? Without that, and with the bug being reported as caused by changes to the replica only, I couldn't see how new errors could creep into the system.
ID: 1735750 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1735755 - Posted: 20 Oct 2015, 22:02:39 UTC
Last modified: 20 Oct 2015, 22:06:41 UTC

Sniff, something on my system just threw away close to two years of Seti statistics. I have an old backup of the statistics* file, but that dates back to 22 October 2014. A year's worth of data lost. :-(

<save_stats_days> was set to 1461 days, so it should've kept the data. My earliest 'time stamp' is now Fri, 16 Oct 2015 00:00:00 GMT.

The stats file for Pirates@home is still intact and showing all those entries.

Edit: I see Einstein's data got thrown out as well. Everything that contacted homebase on the 16th got reset.
ID: 1735755 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1735766 - Posted: 20 Oct 2015, 22:38:05 UTC - in response to Message 1735755.  

Sniff, something on my system just threw away close to two years of Seti statistics. I have an old backup of the statistics* file, but that dates back to 22 October 2014. A year's worth of data lost. :-(

<save_stats_days> was set to 1461 days, so it should've kept the data. My earliest 'time stamp' is now Fri, 16 Oct 2015 00:00:00 GMT.

The stats file for Pirates@home is still intact and showing all those entries.

Edit: I see Einstein's data got thrown out as well. Everything that contacted homebase on the 16th got reset.

Had similar issue before on 2 hosts already. At some moment data log disappeared for SETI project while remained for others. number of days long enough to cover next 5 years...
ID: 1735766 · Report as offensive
OTS
Volunteer tester

Send message
Joined: 6 Jan 08
Posts: 369
Credit: 20,533,537
RAC: 0
United States
Message 1735798 - Posted: 21 Oct 2015, 0:23:40 UTC

Replica DB is back on line and only 3,540 seconds behind. That has to be some sort of record. Boincstats.com shows enough credit issued since the last daily update to create a number 2 in my "Best Ten Days".
ID: 1735798 · Report as offensive
David S
Volunteer tester
Avatar

Send message
Joined: 4 Oct 99
Posts: 18352
Credit: 27,761,924
RAC: 12
United States
Message 1735806 - Posted: 21 Oct 2015, 0:58:42 UTC

Does any of the latest forum tinkering explain why the bit.ly shortcuts in my signature suddenly don't work? I changed one of them to its actual URL, but I don't have any characters left to change the others. The bit.ly URLs do still work on their own.
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.

ID: 1735806 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 27 · Next

Message boards : Number crunching : Panic Mode On (101) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.