Rains it Pours (May 04 2009)

Message boards : Technical News : Rains it Pours (May 04 2009)
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 891328 - Posted: 4 May 2009, 22:27:44 UTC

The weekend was a little bumpy. The mysql database was showing signs of trouble Saturday. Eric was the only one paying attention at the time, so he restarted the database. Everything seemed fine, except he made some posts of the forum and then they all disappeared. This is still a mystery (the cause, the exact effects, and if it still a problem). Eric is trying to recreate and diagnose.

But we were still getting web scraped to death. I played a gig Saturday night, getting home around 1:30am. I noticed the lingering problems at that point and blocked a couple more IP addresses and kicked off the long queries. Things more or less recovered on their own after that (except for the validators, which I fixed in the morning).

So this is getting to be a regular problem, which I partially addressed this morning. I dug through the php code and quickly figured out how to get a couple of the offensive long queries to point at the replica database. This seemed to be quite helpful, but the replica is still behind due to the other problems mentioned above. So people are seeing about a day in the past when checking out their current results on our web site. It's confusing, but not the worst tragedy in the world, and it's a problem that will correct itself shortly. It'll all be caught up after the outage tomorrow.

To keep things interesting, we seem to be in a middle of a spate of weird workunits - ones where the data isn't kosher and therefore returning quickly. Eric is also on top of that one. In the meantime, our outgoing traffic is a bit pegged.

Less than three weeks until the anniversary. I'm getting my powerpoint together now. And I couldn't think of a worthy thread title theme this month, so how about apt titles for a change?

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 891328 · Report as offensive
Dena Wiltsie
Volunteer tester

Send message
Joined: 19 Apr 01
Posts: 1628
Credit: 24,230,968
RAC: 26
United States
Message 891331 - Posted: 4 May 2009, 22:40:14 UTC - in response to Message 891328.  

Would it be possible to divide the data into two parts. One part external IP address may access and the other part blocked from all external access?
ID: 891331 · Report as offensive
Profile Virtual Boss*
Volunteer tester
Avatar

Send message
Joined: 4 May 08
Posts: 417
Credit: 6,440,287
RAC: 0
Australia
Message 891332 - Posted: 4 May 2009, 22:40:27 UTC - in response to Message 891328.  

Thanks for the update Matt.

And THANKS for the teams vigilance over the weekend.

May the deluge ease off and give you all a well earned breather.
ID: 891332 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 891346 - Posted: 4 May 2009, 23:19:06 UTC

Thank you for your hard work over the weekend.
To keep things interesting, we seem to be in a middle of a spate of weird workunits - ones where the data isn't kosher. and therefore returning quickly.

Out of interest was this a AP or MB tape and what was the problem with the data had it skipped a process before it went through the splitters? Thanks in advance.
ID: 891346 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 891454 - Posted: 5 May 2009, 2:44:40 UTC

How come tapes of 0.00GB in sizes apear in the splitting queue from time to time? E.g tape 31mr09aa is at the bottom of the current splitter queue
ID: 891454 · Report as offensive
Profile Misfit
Volunteer tester
Avatar

Send message
Joined: 21 Jun 01
Posts: 21804
Credit: 2,815,091
RAC: 0
United States
Message 891489 - Posted: 5 May 2009, 5:00:41 UTC - in response to Message 891328.  

The weekend was a little bumpy. The mysql database was showing signs of trouble Saturday. Eric was the only one paying attention at the time, so he restarted the database. Everything seemed fine, except he made some posts of the forum and then they all disappeared. This is still a mystery (the cause, the exact effects, and if it still a problem). Eric is trying to recreate and diagnose.

Well the forum went down again right before the posts disappeared. Perhaps those posts were never backed up and the forum returned to the state it was in right before the scheduled outage. The glorified spam thread in the Cafe would support this.

To keep things interesting, we seem to be in a middle of a spate of weird workunits - ones where the data isn't kosher and therefore returning quickly. Eric is also on top of that one. In the meantime, our outgoing traffic is a bit pegged.

Three straight days of MB VLARs at BETA.
me@rescam.org
ID: 891489 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 66292
Credit: 55,293,173
RAC: 49
United States
Message 891502 - Posted: 5 May 2009, 6:09:24 UTC - in response to Message 891328.  

How about "Control in the midst of Khaos!" As in Get Smart's Khaos.. ;)
Savoir-Faire is everywhere!
The T1 Trust, T1 Class 4-4-4-4 #5550, America's First HST

ID: 891502 · Report as offensive
Profile KWSN Ekky Ekky Ekky
Avatar

Send message
Joined: 25 May 99
Posts: 944
Credit: 52,956,491
RAC: 67
United Kingdom
Message 891523 - Posted: 5 May 2009, 8:20:50 UTC

Phew, nice to see those WUs return after going AWOL; congratulations for all your hard work, you clever people.

ID: 891523 · Report as offensive
Profile DaveB
Volunteer tester

Send message
Joined: 5 Sep 03
Posts: 11
Credit: 475,872
RAC: 0
Message 891582 - Posted: 5 May 2009, 12:59:15 UTC

Hi, I dont like to say this but there is a minor problem with the team lists. It expires around the 6000 place mark. I am worried this may be a symptom of a more important database problem.


ID: 891582 · Report as offensive

Message boards : Technical News : Rains it Pours (May 04 2009)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.