upload issues and other news (Oct 5, 2007)

Matt is still away on his well deserved vacation so I will summarize the week.

Last weekend we had 3 servers go down, as Eric described in the previous tech note. Two of these were attached to a UPS that malfunctioned. Not good, but at least we understand what happened. The third machine, bruno, crashes every week or two and hangs on reboot for reasons we have yet to understand. Our best guess at this time is that the fiber connection to the disk array that holds the upload directory is sometimes throwing garbage onto the bus that the machine cannot gracefully handle. This is an old fiber array that we would like to phase out anyway, so we thought about different storage devices that we currently have that could hold the uploads. We came up with the underutilized disk space on the master science database machine, thumper. This could have the added benefit of hosting the assimilators on the same machine that hosts the back end science database. Eric ran a script that gradually migrated the uploads over to thumper.

This worked fine until the migration reach a critical point, at which time the loads on the two download machines shot up to the 80-100 range (they are usually at 5 or less). The high loads were because each instance of the file_upload_handler was taking a long time to write the uploaded results over to thumper. To make a long story short, it turns out that the volume on thumper that held the new upload directory was getting slammed by the uploads. It was running at nearly 100% utilization (local disk, not network, utilization). This was, and still is, a bit surprising. The volume on bruno is software RAID50 and on thumper the volume is software RAID5, the latter having 2 more spindles than each of the RAID50 mirrors on bruno. At any rate, we are migrating back to the fiber array on bruno and have already seen download performance normalize. We'll have to figure this one out...

The other systems news of the week involves database replication on both of our production databases. The seti_boinc database (users, hosts, teams, recent results) replica was lost to a machine crash. We restored from the master and the replica is once again running normally. We are getting very close to having a replica of the back end science database. The initial data load is nearly complete. We will turn on replication either over the weekend or early next week.

Over in science development we are getting the splitter ready to handle the radar blanking signal that will be embedded in all new data once Arecibo comes back on line later this month.

-- Jeff

Thank you for the update Jeff.
____________

Play Fallen Sword!

I think that damn military radar has been the source of the -9 errors and noise in a lot of the WU in the last 6 months or so.

I think that damn military radar has been the source of the -9 errors and noise in a lot of the WU in the last 6 months or so.

It is in *all* of the multibeam data. As I recall, the linefeed array was lower down on the structure so the radar signal was blocked by a nearby hill. The multibeam array is positioned higher so it receives the signal. The splitters have been configured to remove most of it, but some still gets through. Right now it's been a tradeoff between the amount of radar signal in the work units and the amount of good data that is not split due to the possibility of radar contamination. When they get the new setup working they can remove the radar signal and *only* the radar signal.

When they get the new setup working they can remove the radar signal and *only* the radar signal.

I know I've seen an explanation before, but how can SETI 'be sure' they've removed the "d*** military radar" from the WU's? Without going through every second of WU data before it's sent?

I'm suggesting making an occulting screen probably something like that of a drive-in movie theatre but that just may be too expensive and on someone elses property. Also the probability of hurricanes and tornadoes. Maybe a mesh, rather than a light-opaque structure would be just as effective.

If you have pending problems with garbage on the fibre cables you could check the mass points on the system.

One problem could be that the shielding of the fibre cables is directly connected on both ends of the cable. Its strongly recommended to connect ONLY one point directly with the mass point and the other over a spezial surge protector.

Another problem could be that the mass points of the server array is to long. If you connect too many servers over a larger distance with the same mass point the equalizing current of the mass wire could also cause serious problems on the sending and receiving modules. (You know these problems if you ever connected your hifi with the television....)

These two problems can occour and cause such problems. Mostly this problems are not taken care of at the construction of the servers, cause of the increased costs of building.

I already had such problems..... maybe i could help.....


Excuse me for the english.... its a long time since last using it ;)

Absorbing the signal would be preferable to reflecting it for the radar op's sake! hehe Unless it were reflected at an angle so as not to bounce straight back at them!

Well I'd send it back to them amplified a few billion times... but that's me!

Problem with any amount of terrestrial rfi, it massively outweighs those we are looking for. Blocking rfi is difficult because of diffraction and reradiation, and the pylons will also resonate and reradiate all kinds of frequencies and pick up much more than just the radar. A huge multiwalled Faraday cage might help.

Damned inconvenient to have one within line-of-sight-and-a-bit-more near Arecibo. The power of these things can be 10s of kW so even when not facing the antenna, leakage could still be considerable.

Those responsible for the radar should also be responsible for it not interfering with their neighbours, and a radar absorber/attenuator/blocker or switcher offer would be more effective closer to the source.

Saving grace is that the signal can be neutralized somewhat, or skipped over but there can still be annoying traces.

Ideally would need to build a really big mountain around the dish and turn it
into a large tube, or move to the far side of the Moon and recruit a few convenient craters.

Don't worry about sending it back to them. Their software will just remove it as ground clutter.

Gary

I'm suggesting making an occulting screen probably something like that of a drive-in movie theatre but that just may be too expensive and on someone elses property. Also the probability of hurricanes and tornadoes. Maybe a mesh, rather than a light-opaque structure would be just as effective.

Hmm, good suggestion. It wouldn't have to be very big, just big enough to block the radar signal from the receiving antennas. Unless there's reflections from some other structure. It would have to be made of something opaque to the freq of the radar. Maybe the sort of radar absorbing material they use on the stealth and other planes. Absorbing the signal would be preferable to reflecting it for the radar op's sake! hehe Unless it were reflected at an angle so as not to bounce straight back at them!

Sorry, but you are confusing electrical signal cables with fiber optic cables. The grounding advice you gave is well applied to electrical cables, but fiber optics are immune to RFI and other electical interference, and do not require shielding which is earthed (grounded).

:) I know that fibre cables normally have no shielding, but sometimes the rodent protection of the cables is connected to the sending and the receiving modul, or to the server rack. This can cause such problems, not because of electrical interference in the cable itself, but in the sending or receiving modul.


I just wanted to mention it, cause i had such problems twice (with fibre optic) and it took me 2 months to locate the problem.

I used to live about a wavelength from a commercial 10kW AM band radio transmitter. Everything was hot! Of course powerline likes to get into everything too. Sometimes it is easier to shield the powerlines or move them away from the equipment or orient them at right angles to the data cables.

I also remember something recent about the power being redone near the lab building. Check on the timeframe of when the problem started and when that work was done. Perhaps some powerline got moved too close to the data cable maybe even on another floor of the building. New construction might be ENT rather than EMT. Or they could have induced a ground loop in the construction or worse not tied something to ground that should be.

Finally, despite the name Fibre Channel, not all Fibre Channel is carried on fibre optic cables. Copper cable alternatives are available on short runs by some vendors.

I suppose if the 2 interconnected pieces of equipment were not on the same ground bus there could be some differential between them thus inducing some current flow from one to the other.

All too common.
Earth loops can be an nightmare to track down & resolve.

Hi,

Something is still messed up

From the log of one of my machines

10/7/2007 1:07:03 AM|SETI@home|Scheduler RPC succeeded [server version 511]
10/7/2007 1:07:03 AM|SETI@home|Message from server: No work sent
10/7/2007 1:07:03 AM|SETI@home|Message from server: (won't finish in time) Computer on 85.4% of time, BOINC on 100.0% of that, this project gets 100.0% of that

Say What ?? the machine is on 24/7 and only runs seti@home

The closest deadline is 0ct 30 for 6 less than 5.5hr chunks everything else is
not due till Nov. Unless the scheduler is messed up and is ignoring deadlines
the oct deadlines should be completed early monday.

So WHY the won't finish in time and computer on errors ???

I agree here, something is messed up...

10/8/2007 6:59:57 AM|SETI@home|Sending scheduler request: Requested by user
10/8/2007 6:59:57 AM|SETI@home|Requesting 700834 seconds of new work, and reporting 16 completed tasks
10/8/2007 7:00:13 AM||Project communication failed: attempting access to reference site
10/8/2007 7:00:16 AM||Access to reference site succeeded - project servers may be temporarily down.
10/8/2007 7:00:18 AM|SETI@home|Scheduler request failed: server returned nothing (no headers, no data)
10/8/2007 7:00:18 AM|SETI@home|Deferring communication for 1 min 0 sec
10/8/2007 7:00:18 AM|SETI@home|Reason: scheduler request failed
10/8/2007 7:01:18 AM|SETI@home|Sending scheduler request: Requested by user
10/8/2007 7:01:18 AM|SETI@home|Requesting 700832 seconds of new work, and reporting 16 completed tasks
10/8/2007 7:02:53 AM|SETI@home|Scheduler request failed: HTTP service unavailable
10/8/2007 7:02:53 AM|SETI@home|Deferring communication for 1 min 0 sec
10/8/2007 7:02:53 AM|SETI@home|Reason: scheduler request failed

at this point, I set "no new tasks"...

10/8/2007 7:03:13 AM|SETI@home|Sending scheduler request: Requested by user
10/8/2007 7:03:13 AM|SETI@home|Reporting 16 tasks
10/8/2007 7:04:33 AM|SETI@home|Scheduler request failed: HTTP service unavailable
10/8/2007 7:04:33 AM|SETI@home|Deferring communication for 1 min 0 sec
10/8/2007 7:04:33 AM|SETI@home|Reason: scheduler request failed
10/8/2007 7:05:33 AM|SETI@home|Sending scheduler request: Requested by user


This behavior is with 5.10.20 (x64)... I'm NOT getting the same with 5.8.16 or 15 (x86) (on different computers in the "farm".) I finally got through on the next try...
____________
_______

Something is still messed up

From the log of one of my machines

10/7/2007 1:07:03 AM|SETI@home|Scheduler RPC succeeded [server version 511]
10/7/2007 1:07:03 AM|SETI@home|Message from server: No work sent
10/7/2007 1:07:03 AM|SETI@home|Message from server: (won't finish in time) Computer on 85.4% of time, BOINC on 100.0% of that, this project gets 100.0% of that

Say What ?? the machine is on 24/7 and only runs seti@home

The closest deadline is 0ct 30 for 6 less than 5.5hr chunks everything else is
not due till Nov. Unless the scheduler is messed up and is ignoring deadlines
the Oct deadlines should be completed early Monday.

So WHY the won't finish in time and computer on errors ???

Sorry, but you are confusing electrical signal cables with fiber optic cables. The grounding advice you gave is well applied to electrical cables, but fiber optics are immune to RFI and other electical interference, and do not require shielding which is earthed (grounded).

:) I know that fibre cables normally have no shielding, but sometimes the rodent protection of the cables is connected to the sending and the receiving modul, or to the server rack. This can cause such problems, not because of electrical interference in the cable itself, but in the sending or receiving modul.


I just wanted to mention it, cause i had such problems twice (with fibre optic) and it took me 2 months to locate the problem.

I suppose if the 2 interconnected pieces of equipment were not on the same ground bus there could be some differential between them thus inducing some current flow from one to the other. Fair enough. When you don't have an obvious solution to a problem, you must look for other obscure causes.

I suppose if the 2 interconnected pieces of equipment were not on the same ground bus there could be some differential between them thus inducing some current flow from one to the other.

All too common.
Earth loops can be an nightmare to track down & resolve.

The seti@home - Server actual very unstable!
What the Matter?

The seti@home - Server actual very unstable!
What the Matter?

• Set BOINC to cache more work
  
• Connect to more than one BOINC project so you'll always have work from someone
  
• Donate some cash so SETI can get better servers.
  

It is amazing how well things work given the size of the project and the project funding.
____________