Kryten woes (Jan 31 2007)

Our usual database backup outage yesterday was a bit longer than usual because we were doing some experimenting with a new network route. More info to come about that at a later date. Let's just say this is something we've been working on for a year and once it comes to fruition we can freely discuss it.

Anyway, there's a usual period of catch-up during which kryten (the upload server) drops TCP connections. Usually the rate of dropped connections decreases within an hour or so. Not the case this time. While most transactions were being served the past 20 hours, there were still a non-zero amount of dropped connections as of this morning.

This was due to our old nemesis - the dropped NFS mount issue. To restate once again: kryten loses random NFS mounts around the network - this has something to do with its multiple ethernet connections but we still haven't really tracked down the exact cause. Since a simple reboot fixes the problem, this isn't exactly a crisis compared to other things. And since we were uploading results just fine for the most part during the evening, no alarms went off. Plus, frankly, this problem isn't very high priority as it will sort of just "time out" at some point in the future (kryten will be eventually replaced I imagine).

[edit: we are now doing some more network testing, so the upload/servers will be going up and down for brief periods of time over the next hour or two]

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

This was noticed by several people having issues finishing uploads and or downloads that were pending during the outage. I had a couple of machines that seemed to stall those transfers, while other transfers were fine (might be a bug in BOINC we have found?). It's almost like the one transfer had a DNS cache that would not let go. A stop and start of the BOINC program resolved this issue.

So another thing for you to look into. The clients are all 5.8 and above on my systems that exhibited this error, but it looks like 5.4 clients did, also.

So Matt,

I see all 4 of the validators are running as processes on Kryten (the system you mentioned). Kryten has 4 CPUs so this should be ok depending on RAM bandwidth, etc.

a. Would the NFS dropping cause one (or more) of the validator instances to claim it was "not running"

b. Would spreading the validator instances across multiple servers slow it down, speed it up, or be the same in performance?

Thanks,
=Ben

So Matt,

I see all 4 of the validators are running as processes on Kryten (the system you mentioned). Kryten has 4 CPUs so this should be ok depending on RAM bandwidth, etc.

a. Would the NFS dropping cause one (or more) of the validator instances to claim it was "not running"

b. Would spreading the validator instances across multiple servers slow it down, speed it up, or be the same in performance?

Thanks,
=Ben

a. Would the NFS dropping cause one (or more) of the validator instances to claim it was "not running."

b. Would spreading the validator instances across multiple servers slow it down, speed it up, or be the same in performance?

Update: Had to reboot it for the second time today. Sigh.

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Downloads don't seem to be a problem, inasmuch as they (intermittently) actually download. That at least means I haven't run out of work here.
However, uploads to S@h are a continuing problem.
BOINC also reports no connection.
This is all going to be part of the process of getting things going again. Patience!
The thing that worries me is my router seems to be misbehaving quite badly, and I have had to reboot it numerous times per day since the outage began. There seems to be nothing wrong, configuration-wise, so it makes me wonder if BOINC is doing something to my system as it tries to access the project.
Whatever happens, I take my hat off to the people at S@h, as they obviously are working their bums off to fix poor old Kryten.
____________

It's got to the stage where work has run out. I hope you get it sorted soon.

It's got to the stage where work has run out. I hope you get it sorted soon.

Increase the size of your cache in your preferences.

I have it set to connect every 0.5 days.
Is this too often for a system online 24/7?

I have it set to connect every 0.5 days.
Is this too often for a system online 24/7?

In that way you won't have many WU to crunch if servers are down.

It's a bit tricky with that WU-cache.
The more days you set - the more WU's you will get.
Lets say you have a computer that finnished a WU in 2 hrs. It will return the result immidiatly.

I've set the work-cache to 10 days and that gives me aprox 6 days of work with optimized clients.

Kiva

Thanks for that.
I've now set prefs for 10 days.
All I need now is to actually download the work units it's listed.
I suspect there is a network problem somewhere between my computer and seti's

I had got the list of WUs OK, but just could not download them.
I've now started using a webcache and managed to download them all.
It appears to be an issue on blueyonder network in UK that is cause of problem.
People are reporting similar problems with other sites, such as Google, etc.
Thanks for all help, at least I'll not be short of WUs now.

I'm not sure this is the correct thread to reply to, but it is my theory that those who have more or less constant internet access should each pick a short queue, such as 1 day or less. Doing so would reduce the temporary storage requirements at CentralCommand because the results would come back faster (rather than be pending on a client machine for 10 days). For the case that Seti is down, we should also pick a backup project to do. For example, on one of my machines I select Einstein and give it an 8 resource share. This way Seti gets most of my cpu time (my choice); I 'minimize' the drag on the seti computers (at least storage); and I'm mostly covered for times kryten or someone is ill.