Panic Mode On (95) Server Problems?

Message boards : Number crunching : Panic Mode On (95) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 4 · 5 · 6 · 7 · 8 · 9 · 10 . . . 22 · Next

AuthorMessage
Profile JaundicedEye
Avatar

Send message
Joined: 14 Mar 12
Posts: 5375
Credit: 30,870,693
RAC: 1
United States
Message 1639862 - Posted: 10 Feb 2015, 22:18:00 UTC - in response to Message 1639847.  
Last modified: 10 Feb 2015, 22:28:04 UTC

And we're back. Still no AP database though. Everything AP is still dead. 11 days now, since the last non resend AP.

It's dead Jim, truly dead.

I seem to recall a song in the '70s about a "Dead Skunk in the Middle of the Road"........

Edit: Louden Wainright III....https://www.youtube.com/watch?v=EaN7xuAIjXI

:Dg

"Sour Grapes make a bitter Whine." <(0)>
ID: 1639862 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 1639880 - Posted: 10 Feb 2015, 22:45:16 UTC - in response to Message 1639847.  

And we're back. Still no AP database though. Everything AP is still dead. 11 days now, since the last non resend AP.

It's dead Jim, truly dead.

Look on the bright side, after spending 10 days or so crunching APs and getting 0 credit now we get to crunch MB and receive something. If you set your goals low enough you are never disappointed.
ID: 1639880 · Report as offensive
Phil Burden

Send message
Joined: 26 Oct 00
Posts: 264
Credit: 22,303,899
RAC: 0
United Kingdom
Message 1639895 - Posted: 10 Feb 2015, 23:03:28 UTC

Well, another outage done, and all is NOT well. "unable to connect to server" <sigh>

P.
ID: 1639895 · Report as offensive
Eric Findley
Avatar

Send message
Joined: 28 Mar 03
Posts: 72
Credit: 8,674,945
RAC: 0
United States
Message 1639899 - Posted: 10 Feb 2015, 23:09:56 UTC - in response to Message 1638947.  

got a notice from utilities company saying I use 11% more power than neighbors[sigh, big brother world).

Slowing things down til AP works again.

Use 11% more. So what.

Don't you pay more?

Absolutely I pay more........I guess their point is I'm not GREEN ENOUGH...

And they know where they can put that argument.

:D...g

Gotta love it. My notice says I use 63% more!!! Must be the UPS I run to protect against their crappy power ... I mean, I'm only running 3 PCs .... Jeez.

My electric company keeps telling me that I should get rid of my second frig?
ID: 1639899 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1639916 - Posted: 10 Feb 2015, 23:39:40 UTC - in response to Message 1639880.  

Look on the bright side, after spending 10 days or so crunching APs and getting 0 credit now we get to crunch MB and receive something. If you set your goals low enough you are never disappointed.

"I found that if you have goals, you might not reach them, but if you have none, then you are never disappointed, and I gotta tell ya, it feels.. phenomenal." - Peter La Fleur

So.. if instead of questioning whether the glass is half-empty or half-full.. take the opportunity to drink what's left and enjoy it, and then when the glass is empty, it can only get better from there, because refilling the glass in any amount is better than it being empty.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1639916 · Report as offensive
Profile JaundicedEye
Avatar

Send message
Joined: 14 Mar 12
Posts: 5375
Credit: 30,870,693
RAC: 1
United States
Message 1639948 - Posted: 11 Feb 2015, 0:43:39 UTC - in response to Message 1639899.  

My electric company keeps telling me that I should get rid of my second frig?


If that's the beer frig, no way.........oops, wrong thread.

:Dg

"Sour Grapes make a bitter Whine." <(0)>
ID: 1639948 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 1639960 - Posted: 11 Feb 2015, 1:12:49 UTC - in response to Message 1639916.  

So.. if instead of questioning whether the glass is half-empty or half-full.. take the opportunity to drink what's left and enjoy it, and then when the glass is empty, it can only get better from there, because refilling the glass in any amount is better than it being empty.

You are describing MB.
ID: 1639960 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 1639967 - Posted: 11 Feb 2015, 1:26:55 UTC

To add insult to injury it seems MB is not validating for me since todays outage.
I shall crunch on.
ID: 1639967 · Report as offensive
Profile cliff
Avatar

Send message
Joined: 16 Dec 07
Posts: 625
Credit: 3,590,440
RAC: 0
United Kingdom
Message 1639973 - Posted: 11 Feb 2015, 1:44:33 UTC

OK, so NOW what's the bleeding problem??
cannot upload reports on completed WU, cannot get replacement WU for those completed.

Stats page shows servers working, but then its been know to tell porkies:-/

Regards,
Cliff,
Been there, Done that, Still no damm T shirt!
ID: 1639973 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1639998 - Posted: 11 Feb 2015, 2:33:59 UTC - in response to Message 1639973.  

Don't know what's wrong. Nobody seems to be reporting issues. SSP shows the project is up and running but I can't report and can't get any tasks.

74742 SETI@home 2/10/2015 6:50:35 PM Scheduler request failed: HTTP internal server error


The router outboard show the project basically flatlined:

http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=%2Frouter-interfaces%2Finr-211%2Fgigabitethernet6_17;ranges=d%3Aw;view=Octets

Expected continuing problems with AP but now there are problems with MB. I'm about to run out of Seti GPU tasks and soon even CPU tasks. Guess my other projects, Einstein and MilkyWay will get the attention. Anybody have an idea what is going on???? This is not the normal slow recovery after project maintenance.

Cheers, Keith
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1639998 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 1640009 - Posted: 11 Feb 2015, 3:03:39 UTC

I got home, set no new work, updated, and all tasks on both rigs were reported, then set both back to allow work again.

Now I just hope that they fix whatever is not working very soon or my main rig is going to be grabbing another batch of GPU backup work.

Cheers.
ID: 1640009 · Report as offensive
Profile Oz
Avatar

Send message
Joined: 6 Jun 99
Posts: 233
Credit: 200,655,462
RAC: 212
United States
Message 1640022 - Posted: 11 Feb 2015, 3:37:12 UTC
Last modified: 11 Feb 2015, 3:38:24 UTC

Don't worry, it will be fixed during next week's maintenance.
Member of the 20 Year Club



ID: 1640022 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 1640023 - Posted: 11 Feb 2015, 3:43:34 UTC

With no AP for some time and MB seemingly borked panic may follow in 24 hours or so.
ID: 1640023 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1640026 - Posted: 11 Feb 2015, 3:55:46 UTC - in response to Message 1639960.  

So.. if instead of questioning whether the glass is half-empty or half-full.. take the opportunity to drink what's left and enjoy it, and then when the glass is empty, it can only get better from there, because refilling the glass in any amount is better than it being empty.

You are describing MB.

I wasn't specifically referring to MB being what can re-fill the empty glass (cache), but it IS an option if you absolutely must have something in your cache.

I, on the other hand, will continue waiting for AP to be fixed. To each their own.

And yeah.. I was going to ask why nobody has said anything about there basically being nothing for data coming out to us. SSP shows the RTS buffer is at its high-water mark, but nothing's going out. Odd..
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1640026 · Report as offensive
Profile JaundicedEye
Avatar

Send message
Joined: 14 Mar 12
Posts: 5375
Credit: 30,870,693
RAC: 1
United States
Message 1640032 - Posted: 11 Feb 2015, 4:33:58 UTC

WOW! The glass is really EMPTY.......

Please, sir, may I have some more?

:D(((g

"Sour Grapes make a bitter Whine." <(0)>
ID: 1640032 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 1640035 - Posted: 11 Feb 2015, 5:05:14 UTC

Today's maintenace was a real dandy.
The whole thing is going down fast, I'm sure this will get attention from the staff.
ID: 1640035 · Report as offensive
Profile JaundicedEye
Avatar

Send message
Joined: 14 Mar 12
Posts: 5375
Credit: 30,870,693
RAC: 1
United States
Message 1640041 - Posted: 11 Feb 2015, 5:43:37 UTC - in response to Message 1640035.  

Today's maintenace was a real dandy.
The whole thing is going down fast, I'm sure this will get attention from the staff.

Yes, a major flogging of the offending server is definitely in order.
40 lashes and no dinner!

"Sour Grapes make a bitter Whine." <(0)>
ID: 1640041 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13735
Credit: 208,696,464
RAC: 304
Australia
Message 1640045 - Posted: 11 Feb 2015, 5:54:58 UTC - in response to Message 1640041.  
Last modified: 11 Feb 2015, 5:56:23 UTC

Server status page is showing the usual green for MB to be running (AP is still dead), however it's not working.
All Scheduler requests result in "Scheduler request failed: couldn't connect to server" messages, and then ever increasing project backoffs. At least we can still upload (for now).
Grant
Darwin NT
ID: 1640045 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1640049 - Posted: 11 Feb 2015, 6:25:09 UTC - in response to Message 1640048.  

Crunching away on Beta until they get things sorted out...
ID: 1640049 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 1640062 - Posted: 11 Feb 2015, 7:25:10 UTC - in response to Message 1639714.  

Let me try to better-explain this.

Two results are reported. Validator compares them and picks a canonical result (typically the first one reported, not necessarily the _0 task).

Once validated, the WU moves on to assimilation. Once assimilation happens (the data in the results are entered into the science database), the WU file itself, and the result files that got returned are no longer needed, so they move on to the file deleter.

File deleter runs pretty much once it is told it has something that needs deletion, much like validation runs once there are two results to compare.

"results/WUs waiting for purge" is what we see on our tasks pages until they get deleted. That has a 24-hour delay starting from when that data was assimilated. Once DB purge happens, us end-users can't see anything about those WUs/tasks anymore (the staff can, if they do the right queries in the right places and so forth, but there's never any need to, so they don't).


As far as I know, the data we see on the task pages here on the website comes from the BOINC database. Basically, that's just a DB that keeps track of users, hosts, WUs and tasks that are in-play, and it also does keep track of who participated (hostID) on what WUs, so that eventually when the contents of the science DBs are analyzed, if something interesting is found, they know who to contact to give credit to for crunching the WU that found the result.

Regarding the huge backlog of purging for MB.. I don't know what the issue is on that one, but generally when there's a large backlog like that, it is gone after the outage (sometimes, tasks that were assimilated less than 24 hours ago are purged during the maintenance, other times, only the ones older than 24 hours when the purge was manually run). We'll see what happens in about.. 10-12 hours from now.

Thanks for the explanation.

@Grant, I was referring to the fact that there was over 3 million results waiting to be deleted there was no backlog in the servers weren't running behind. I think the purging totals are higher moment because there are/were shorter work units going through. As I write
11 Feb 2015, 7:20:03 UTC looks like parts of the server running about 3 hours behind in displaying information.
ID: 1640062 · Report as offensive
Previous · 1 . . . 4 · 5 · 6 · 7 · 8 · 9 · 10 . . . 22 · Next

Message boards : Number crunching : Panic Mode On (95) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.