Oopsie (Oct 06 2009)

Message boards : Technical News : Oopsie (Oct 06 2009)
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 938002 - Posted: 6 Oct 2009, 22:43:01 UTC

Quick post-weekly-outage wrapup: everything went fine, albeit a little slow given recent events. The replica recovery is going on now. Hopefully it'll continue along safely overnight and we can turn the replica back on sometime tomorrow.

One hilarious note. All our server reboots over the weekend dislodged several instances of sendmail, which then went on to send forth unexpectedly large queues of cronjob/server related e-mails to me, Jeff, and Eric. We're talking about 35,000 e-mails, all of which went through the lab spam firewall first, thus clobbering everybody's e-mail in the entire Space Lab for about 24 hours. Fun.

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 938002 · Report as offensive
Pablo_ARG

Send message
Joined: 13 Aug 03
Posts: 12
Credit: 2,041,544
RAC: 0
Argentina
Message 938008 - Posted: 6 Oct 2009, 23:07:58 UTC

The weekend outage is finally repaired.

Thanks for the update.

PD: 35.000 e-mails in two days ? ... servers cries loud :)
ID: 938008 · Report as offensive
Profile LiliKrist
Volunteer tester
Avatar

Send message
Joined: 12 Aug 09
Posts: 333
Credit: 143,167
RAC: 0
Indonesia
Message 938026 - Posted: 7 Oct 2009, 1:20:48 UTC

^U^


N = R x fp x ne x fl x fi x fc x L
ID: 938026 · Report as offensive
Profile Johnney Guinness
Volunteer tester
Avatar

Send message
Joined: 11 Sep 06
Posts: 3093
Credit: 2,652,287
RAC: 0
Ireland
Message 938035 - Posted: 7 Oct 2009, 2:19:15 UTC
Last modified: 7 Oct 2009, 2:20:04 UTC

I start sweating if my mail box says "You have 5 new messages".

With 35,000 emails, you must have thought you were Mr. Popular for a few moments Matt...LOL

John.
ID: 938035 · Report as offensive
Profile Uli
Volunteer tester
Avatar

Send message
Joined: 6 Feb 00
Posts: 10923
Credit: 5,996,015
RAC: 1
Germany
Message 938041 - Posted: 7 Oct 2009, 2:54:05 UTC

Thanks Matt, that explains a few things.
Pluto will always be a planet to me.

Seti Ambassador
Not to late to order an Anni Shirt
ID: 938041 · Report as offensive
John McLeod VII
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jul 99
Posts: 24806
Credit: 790,712
RAC: 0
United States
Message 938042 - Posted: 7 Oct 2009, 2:55:14 UTC - in response to Message 938002.  

Quick post-weekly-outage wrapup: everything went fine, albeit a little slow given recent events. The replica recovery is going on now. Hopefully it'll continue along safely overnight and we can turn the replica back on sometime tomorrow.

One hilarious note. All our server reboots over the weekend dislodged several instances of sendmail, which then went on to send forth unexpectedly large queues of cronjob/server related e-mails to me, Jeff, and Eric. We're talking about 35,000 e-mails, all of which went through the lab spam firewall first, thus clobbering everybody's e-mail in the entire Space Lab for about 24 hours. Fun.

- Matt

THAT explains the delayed emails to the BOINC Help Desk and the S@H moderators email list.


BOINC WIKI
ID: 938042 · Report as offensive
Profile Arvid Almstrom
Avatar

Send message
Joined: 23 Mar 00
Posts: 98
Credit: 137,331,372
RAC: 0
Australia
Message 938053 - Posted: 7 Oct 2009, 3:48:52 UTC


I do like the 35,000 e-mails, no doubt causing all sorts of problems for the administrators, maybe your were not the only once dragged back to work over the weekend.

One another note, the main page for technical notes is getting quite large. I was on a slow wireless broadband link stuck on 2G, when I discovered that this page is huge on a slow connection. Any chance of truncating say the first couple years. If anyone would like access to the older stuff maybe a second page which includes the removed comments.

Arvid
Arvid Almstrom
ID: 938053 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 938068 - Posted: 7 Oct 2009, 5:24:48 UTC - in response to Message 938053.  

...
One another note, the main page for technical notes is getting quite large. I was on a slow wireless broadband link stuck on 2G, when I discovered that this page is huge on a slow connection. Any chance of truncating say the first couple years. If anyone would like access to the older stuff maybe a second page which includes the removed comments.

Arvid

If you get to the bottom of that page, you discover that several older years have already been split off into separate archives. Being on dial-up, I'd also find it easier to handle if it were split again. But I'd put it in the "nice to have" class, something for a day when more important work is blocked for some reason.
                                                                Joe
ID: 938068 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1639
Credit: 12,921,799
RAC: 89
New Zealand
Message 938085 - Posted: 7 Oct 2009, 8:27:48 UTC
Last modified: 7 Oct 2009, 8:32:20 UTC

Thank you to the people came up to the lab to try and the servers working again. Am I correct in saying that all the fresh tapes full of data that are left have been loaded into the splitters? Thanks for reading
ID: 938085 · Report as offensive
John McCallum
Avatar

Send message
Joined: 5 Dec 04
Posts: 877
Credit: 599,458
RAC: 8
United Kingdom
Message 938126 - Posted: 7 Oct 2009, 14:19:27 UTC

'We're talking about 35,000 e-mails'and every one or most will have to be checked,at least the message titles to ensure that nothing important is missed
some one is going to be very popular lol
Old enough to know better(but)still young enough not to care
ID: 938126 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1639
Credit: 12,921,799
RAC: 89
New Zealand
Message 938184 - Posted: 7 Oct 2009, 20:50:46 UTC

I was taking a look at http://setiathome.berkeley.edu/sah_status.html & noticed that a few ap & mb splitters are disabled. I also noticed that there's a new Program running on bruno called Vote_Monitor. Does anyone know what the vote monitor does? I couldn't ever imagine e-mails in my in box were the e-mails telling Matt Jeff & Eric that the the servers were looking up?
ID: 938184 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 938187 - Posted: 7 Oct 2009, 20:56:26 UTC - in response to Message 938184.  

We have a lot of splitters listed in our back-end configuration file - more than we usually need (or can handle) - which is why several are disabled at any given time. If we need to start more (or less), it's an easy toggle within the config file.

The vote monitor is a script that regularly check the state of and handles elections for user banishment on the forums.

I was taking a look at http://setiathome.berkeley.edu/sah_status.html & noticed that a few ap & mb splitters are disabled. I also noticed that there's a new Program running on bruno called Vote_Monitor. Does anyone know what the vote monitor does? I couldn't ever imagine e-mails in my in box were the e-mails telling Matt Jeff & Eric that the the servers were looking up?


- Matt
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 938187 · Report as offensive
Profile 52 Aces
Avatar

Send message
Joined: 7 Jan 02
Posts: 497
Credit: 14,261,068
RAC: 67
United States
Message 938192 - Posted: 7 Oct 2009, 21:25:03 UTC - in response to Message 938187.  

The vote monitor is a script that regularly check the state of and handles elections for user banishment on the forums.


Cool, how many fake logons under spoofed IP & MACs must I create to ban someone ? j/k ;-)

ID: 938192 · Report as offensive
OzzFan Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Apr 02
Posts: 15691
Credit: 84,761,841
RAC: 28
United States
Message 938209 - Posted: 7 Oct 2009, 21:55:05 UTC - in response to Message 938192.  

The vote monitor is a script that regularly check the state of and handles elections for user banishment on the forums.


Cool, how many fake logons under spoofed IP & MACs must I create to ban someone ? j/k ;-)


LOL

It won't work though. In order to vote you must have a bit set on your account indicating that you are a moderator for SETI@Home. You'd have to hack the server where user accounts are stored, find the appropriate flag and set it.

Even then, the Admin(s) can override a ban vote and reverse it immediately, then quickly remove the flagged bit from the offending account. At that point, there's a strong chance that the account with the bit set would be reviewed and possibly perma-banned for such an infraction.
ID: 938209 · Report as offensive
Profile 52 Aces
Avatar

Send message
Joined: 7 Jan 02
Posts: 497
Credit: 14,261,068
RAC: 67
United States
Message 938224 - Posted: 7 Oct 2009, 22:46:19 UTC - in response to Message 938209.  

Thx for the tech-splanation. That said, you have software & a thread running just for ban concensus !! ;-)

Good thing I don't have a vote, cuz I'd vote to turn off the vote_monitor and run another instance of upload_server ;-) (I know, that one is probably wire bound anyhow)



ID: 938224 · Report as offensive
OzzFan Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Apr 02
Posts: 15691
Credit: 84,761,841
RAC: 28
United States
Message 938225 - Posted: 7 Oct 2009, 22:58:42 UTC - in response to Message 938224.  

Thx for the tech-splanation. That said, you have software & a thread running just for ban concensus !! ;-)

Good thing I don't have a vote, cuz I'd vote to turn off the vote_monitor and run another instance of upload_server ;-) (I know, that one is probably wire bound anyhow)


If there are no active bans pending, then the process doesn't use any CPU cycles, thus negating a need to turn it off and create another upload_server instance. ;-)
ID: 938225 · Report as offensive

Message boards : Technical News : Oopsie (Oct 06 2009)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.