Fall Cleaning (Oct 25 2007)

Message boards : Technical News : Fall Cleaning (Oct 25 2007)
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 666498 - Posted: 25 Oct 2007, 20:30:00 UTC

For some reason I'm in the "deal with boring, nagging sysadmin tasks" zone this week, so that's mostly what I've been working on. Gotta ride the wave when it happens, you know? Nothing really interesting there to report. Writing scripts, updating our UPS plans, cleaning up and improving our internal alert system... stuff like that.

Last night the logical log on our primary science database filled up. This is the log that is used by the secondary to keep in sync with the primary. When the log is full, the primary halts all connections as a protective measure, as the secondary will lose track of future updates. What does all this mean for you? Well, with the primary effectively offline the assimilators and splitters were blocked, and we ran out of work to send this morning. We spotted this quickly enough, but apparently we need better alerts and some automatic logical log rotation system. We're still getting the feel of this informix database replication stuff.

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 666498 · Report as offensive
Profile Dr. C.E.T.I.
Avatar

Send message
Joined: 29 Feb 00
Posts: 16019
Credit: 794,685
RAC: 0
United States
Message 666500 - Posted: 25 Oct 2007, 20:35:04 UTC - in response to Message 666498.  

For some reason I'm in the "deal with boring, nagging sysadmin tasks" zone this week, so that's mostly what I've been working on. Gotta ride the wave when it happens, you know? Nothing really interesting there to report. Writing scripts, updating our UPS plans, cleaning up and improving our internal alert system... stuff like that.

Last night the logical log on our primary science database filled up. This is the log that is used by the secondary to keep in sync with the primary. When the log is full, the primary halts all connections as a protective measure, as the secondary will lose track of future updates. What does all this mean for you? Well, with the primary effectively offline the assimilators and splitters were blocked, and we ran out of work to send this morning. We spotted this quickly enough, but apparently we need better alerts and some automatic logical log rotation system. We're still getting the feel of this informix database replication stuff.

- Matt


Thank You Matt - and to the Rest of the Berkeley Staff - Thanks


BOINC Wiki . . .

Science Status Page . . .
ID: 666500 · Report as offensive
DJStarfox

Send message
Joined: 23 May 01
Posts: 1066
Credit: 1,226,053
RAC: 2
United States
Message 666699 - Posted: 26 Oct 2007, 2:29:24 UTC - in response to Message 666498.  
Last modified: 26 Oct 2007, 2:30:09 UTC

Any good DBA managing a busy, replicated database is always watching his transaction log files. It's amazing how fast they can fill up; I've seen 8 GB disappear in 5 minutes when a certain user is connected! Unfortunately, I've never worked with Informix, so I couldn't offer any strategies to help.
ID: 666699 · Report as offensive
JLDun
Volunteer tester
Avatar

Send message
Joined: 21 Apr 06
Posts: 573
Credit: 196,101
RAC: 0
United States
Message 666753 - Posted: 26 Oct 2007, 3:46:15 UTC - in response to Message 666699.  

Any good DBA managing a busy, replicated database is always watching his transaction log files. It's amazing how fast they can fill up; I've seen 8 GB disappear in 5 minutes when a certain user is connected! Unfortunately, I've never worked with Informix, so I couldn't offer any strategies to help.

Would that be NEZ? ;-p
ID: 666753 · Report as offensive
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 20147
Credit: 7,508,002
RAC: 20
United Kingdom
Message 666943 - Posted: 26 Oct 2007, 12:26:45 UTC - in response to Message 666699.  

Any good DBA managing a busy, replicated database is always watching his transaction log files. ...

An even better one is elsewhere doing useful work whilst a check script (eg logwatch?) is doing the checking, ready to send an alert for anything 'unusual'...

Regards,
Martin

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 666943 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 667051 - Posted: 26 Oct 2007, 16:21:12 UTC - in response to Message 666943.  

I think the issue is that we are still getting a feel for informix replication, such as it is. Documentation isn't so great. And having done it easily and relatively seamlessly with mysql, there's still an "astonishment" factor with informix replication. So things are going up and down as we test, and logs filling up isn't exactly the main thing on our radar yet.

- Matt

Any good DBA managing a busy, replicated database is always watching his transaction log files. ...

An even better one is elsewhere doing useful work whilst a check script (eg logwatch?) is doing the checking, ready to send an alert for anything 'unusual'...

Regards,
Martin


-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 667051 · Report as offensive
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 20147
Credit: 7,508,002
RAC: 20
United Kingdom
Message 667066 - Posted: 26 Oct 2007, 16:49:28 UTC - in response to Message 667051.  
Last modified: 26 Oct 2007, 16:52:10 UTC

I think the issue is that we are still getting a feel for informix replication, such as it is. Documentation isn't so great. And having done it easily and relatively seamlessly with mysql, there's still an "astonishment" factor with informix...

There's always more to learn.

I'm very glad that I'm not the one on the sharp end of a million on-line remote 'experts'!

(Matt alias "Neo"? ;-) )


(And I fully expect a few 'giggles' for myself when I build a mere six server system soonish. A weekend is likely not long enough... And neither will be the patch leads... :-( All good fun! )

Happy crunchin',

Regards,
Martin

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 667066 · Report as offensive

Message boards : Technical News : Fall Cleaning (Oct 25 2007)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.