Oopsie (Oct 06 2009)


log in

Advanced search

Message boards : Technical News : Oopsie (Oct 06 2009)

Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1384
Credit: 74,079
RAC: 0
United States
Message 938002 - Posted: 6 Oct 2009, 22:43:01 UTC

Quick post-weekly-outage wrapup: everything went fine, albeit a little slow given recent events. The replica recovery is going on now. Hopefully it'll continue along safely overnight and we can turn the replica back on sometime tomorrow.

One hilarious note. All our server reboots over the weekend dislodged several instances of sendmail, which then went on to send forth unexpectedly large queues of cronjob/server related e-mails to me, Jeff, and Eric. We're talking about 35,000 e-mails, all of which went through the lab spam firewall first, thus clobbering everybody's e-mail in the entire Space Lab for about 24 hours. Fun.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Pablo_ARG
Send message
Joined: 13 Aug 03
Posts: 12
Credit: 2,041,544
RAC: 0
Argentina
Message 938008 - Posted: 6 Oct 2009, 23:07:58 UTC

The weekend outage is finally repaired.

Thanks for the update.

PD: 35.000 e-mails in two days ? ... servers cries loud :)
____________

Profile LiliKrist
Volunteer tester
Avatar
Send message
Joined: 12 Aug 09
Posts: 333
Credit: 143,167
RAC: 0
Indonesia
Message 938026 - Posted: 7 Oct 2009, 1:20:48 UTC

^U^
____________


N = R x fp x ne x fl x fi x fc x L

Profile Johnney Guinness
Volunteer tester
Avatar
Send message
Joined: 11 Sep 06
Posts: 2950
Credit: 2,306,418
RAC: 0
Ireland
Message 938035 - Posted: 7 Oct 2009, 2:19:15 UTC
Last modified: 7 Oct 2009, 2:20:04 UTC

I start sweating if my mail box says "You have 5 new messages".

With 35,000 emails, you must have thought you were Mr. Popular for a few moments Matt...LOL

John.
____________

Profile Uli
Volunteer tester
Avatar
Send message
Joined: 6 Feb 00
Posts: 9360
Credit: 4,933,775
RAC: 3,481
Germany
Message 938041 - Posted: 7 Oct 2009, 2:54:05 UTC

Thanks Matt, that explains a few things.
____________
Pluto will always be a planet to me.

Cash Donation Specialist

John McLeod VII
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 15 Jul 99
Posts: 23702
Credit: 493,921
RAC: 159
United States
Message 938042 - Posted: 7 Oct 2009, 2:55:14 UTC - in response to Message 938002.

Quick post-weekly-outage wrapup: everything went fine, albeit a little slow given recent events. The replica recovery is going on now. Hopefully it'll continue along safely overnight and we can turn the replica back on sometime tomorrow.

One hilarious note. All our server reboots over the weekend dislodged several instances of sendmail, which then went on to send forth unexpectedly large queues of cronjob/server related e-mails to me, Jeff, and Eric. We're talking about 35,000 e-mails, all of which went through the lab spam firewall first, thus clobbering everybody's e-mail in the entire Space Lab for about 24 hours. Fun.

- Matt

THAT explains the delayed emails to the BOINC Help Desk and the S@H moderators email list.
____________


BOINC WIKI

Profile Arvid Almstrom
Avatar
Send message
Joined: 23 Mar 00
Posts: 98
Credit: 136,195,994
RAC: 9,380
Australia
Message 938053 - Posted: 7 Oct 2009, 3:48:52 UTC


I do like the 35,000 e-mails, no doubt causing all sorts of problems for the administrators, maybe your were not the only once dragged back to work over the weekend.

One another note, the main page for technical notes is getting quite large. I was on a slow wireless broadband link stuck on 2G, when I discovered that this page is huge on a slow connection. Any chance of truncating say the first couple years. If anyone would like access to the older stuff maybe a second page which includes the removed comments.

Arvid
____________
Arvid Almstrom

Josef W. Segur
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4140
Credit: 1,004,550
RAC: 250
United States
Message 938068 - Posted: 7 Oct 2009, 5:24:48 UTC - in response to Message 938053.

...
One another note, the main page for technical notes is getting quite large. I was on a slow wireless broadband link stuck on 2G, when I discovered that this page is huge on a slow connection. Any chance of truncating say the first couple years. If anyone would like access to the older stuff maybe a second page which includes the removed comments.

Arvid

If you get to the bottom of that page, you discover that several older years have already been split off into separate archives. Being on dial-up, I'd also find it easier to handle if it were split again. But I'd put it in the "nice to have" class, something for a day when more important work is blocked for some reason.
Joe

Speedy
Volunteer tester
Avatar
Send message
Joined: 26 Jun 04
Posts: 623
Credit: 4,777,336
RAC: 5,270
New Zealand
Message 938085 - Posted: 7 Oct 2009, 8:27:48 UTC
Last modified: 7 Oct 2009, 8:32:20 UTC

Thank you to the people came up to the lab to try and the servers working again. Am I correct in saying that all the fresh tapes full of data that are left have been loaded into the splitters? Thanks for reading
____________

Live in NZ y not join Smile City?

John McCallum
Avatar
Send message
Joined: 5 Dec 04
Posts: 805
Credit: 270,586
RAC: 172
United Kingdom
Message 938126 - Posted: 7 Oct 2009, 14:19:27 UTC

'We're talking about 35,000 e-mails'and every one or most will have to be checked,at least the message titles to ensure that nothing important is missed
some one is going to be very popular lol
____________
Old enough to know better(but)still young enough not to care

Speedy
Volunteer tester
Avatar
Send message
Joined: 26 Jun 04
Posts: 623
Credit: 4,777,336
RAC: 5,270
New Zealand
Message 938184 - Posted: 7 Oct 2009, 20:50:46 UTC

I was taking a look at http://setiathome.berkeley.edu/sah_status.html & noticed that a few ap & mb splitters are disabled. I also noticed that there's a new Program running on bruno called Vote_Monitor. Does anyone know what the vote monitor does? I couldn't ever imagine e-mails in my in box were the e-mails telling Matt Jeff & Eric that the the servers were looking up?
____________

Live in NZ y not join Smile City?

Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1384
Credit: 74,079
RAC: 0
United States
Message 938187 - Posted: 7 Oct 2009, 20:56:26 UTC - in response to Message 938184.

We have a lot of splitters listed in our back-end configuration file - more than we usually need (or can handle) - which is why several are disabled at any given time. If we need to start more (or less), it's an easy toggle within the config file.

The vote monitor is a script that regularly check the state of and handles elections for user banishment on the forums.

I was taking a look at http://setiathome.berkeley.edu/sah_status.html & noticed that a few ap & mb splitters are disabled. I also noticed that there's a new Program running on bruno called Vote_Monitor. Does anyone know what the vote monitor does? I couldn't ever imagine e-mails in my in box were the e-mails telling Matt Jeff & Eric that the the servers were looking up?


- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile 52 Aces
Avatar
Send message
Joined: 7 Jan 02
Posts: 494
Credit: 12,232,651
RAC: 848
United States
Message 938192 - Posted: 7 Oct 2009, 21:25:03 UTC - in response to Message 938187.

The vote monitor is a script that regularly check the state of and handles elections for user banishment on the forums.


Cool, how many fake logons under spoofed IP & MACs must I create to ban someone ? j/k ;-)

OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13307
Credit: 27,892,984
RAC: 16,255
United States
Message 938209 - Posted: 7 Oct 2009, 21:55:05 UTC - in response to Message 938192.

The vote monitor is a script that regularly check the state of and handles elections for user banishment on the forums.


Cool, how many fake logons under spoofed IP & MACs must I create to ban someone ? j/k ;-)


LOL

It won't work though. In order to vote you must have a bit set on your account indicating that you are a moderator for SETI@Home. You'd have to hack the server where user accounts are stored, find the appropriate flag and set it.

Even then, the Admin(s) can override a ban vote and reverse it immediately, then quickly remove the flagged bit from the offending account. At that point, there's a strong chance that the account with the bit set would be reviewed and possibly perma-banned for such an infraction.
____________

Profile 52 Aces
Avatar
Send message
Joined: 7 Jan 02
Posts: 494
Credit: 12,232,651
RAC: 848
United States
Message 938224 - Posted: 7 Oct 2009, 22:46:19 UTC - in response to Message 938209.

Thx for the tech-splanation. That said, you have software & a thread running just for ban concensus !! ;-)

Good thing I don't have a vote, cuz I'd vote to turn off the vote_monitor and run another instance of upload_server ;-) (I know, that one is probably wire bound anyhow)



OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13307
Credit: 27,892,984
RAC: 16,255
United States
Message 938225 - Posted: 7 Oct 2009, 22:58:42 UTC - in response to Message 938224.

Thx for the tech-splanation. That said, you have software & a thread running just for ban concensus !! ;-)

Good thing I don't have a vote, cuz I'd vote to turn off the vote_monitor and run another instance of upload_server ;-) (I know, that one is probably wire bound anyhow)


If there are no active bans pending, then the process doesn't use any CPU cycles, thus negating a need to turn it off and create another upload_server instance. ;-)
____________

Message boards : Technical News : Oopsie (Oct 06 2009)

Copyright © 2014 University of California