upload issues and other news (Oct 5, 2007)

Matt is still away on his well deserved vacation so I will summarize the week.

Last weekend we had 3 servers go down, as Eric described in the previous tech note. Two of these were attached to a UPS that malfunctioned. Not good, but at least we understand what happened. The third machine, bruno, crashes every week or two and hangs on reboot for reasons we have yet to understand. Our best guess at this time is that the fiber connection to the disk array that holds the upload directory is sometimes throwing garbage onto the bus that the machine cannot gracefully handle. This is an old fiber array that we would like to phase out anyway, so we thought about different storage devices that we currently have that could hold the uploads. We came up with the underutilized disk space on the master science database machine, thumper. This could have the added benefit of hosting the assimilators on the same machine that hosts the back end science database. Eric ran a script that gradually migrated the uploads over to thumper.

This worked fine until the migration reach a critical point, at which time the loads on the two download machines shot up to the 80-100 range (they are usually at 5 or less). The high loads were because each instance of the file_upload_handler was taking a long time to write the uploaded results over to thumper. To make a long story short, it turns out that the volume on thumper that held the new upload directory was getting slammed by the uploads. It was running at nearly 100% utilization (local disk, not network, utilization). This was, and still is, a bit surprising. The volume on bruno is software RAID50 and on thumper the volume is software RAID5, the latter having 2 more spindles than each of the RAID50 mirrors on bruno. At any rate, we are migrating back to the fiber array on bruno and have already seen download performance normalize. We'll have to figure this one out...

The other systems news of the week involves database replication on both of our production databases. The seti_boinc database (users, hosts, teams, recent results) replica was lost to a machine crash. We restored from the master and the replica is once again running normally. We are getting very close to having a replica of the back end science database. The initial data load is nearly complete. We will turn on replication either over the weekend or early next week.

Over in science development we are getting the splitter ready to handle the radar blanking signal that will be embedded in all new data once Arecibo comes back on line later this month.

-- Jeff

Thank you for the update Jeff.
____________

I think that damn military radar has been the source of the -9 errors and noise in a lot of the WU in the last 6 months or so.

I think that damn military radar has been the source of the -9 errors and noise in a lot of the WU in the last 6 months or so.

It is in *all* of the multibeam data. As I recall, the linefeed array was lower down on the structure so the radar signal was blocked by a nearby hill. The multibeam array is positioned higher so it receives the signal. The splitters have been configured to remove most of it, but some still gets through. Right now it's been a tradeoff between the amount of radar signal in the work units and the amount of good data that is not split due to the possibility of radar contamination. When they get the new setup working they can remove the radar signal and *only* the radar signal.

When they get the new setup working they can remove the radar signal and *only* the radar signal.

I know I've seen an explanation before, but how can SETI 'be sure' they've removed the "d*** military radar" from the WU's? Without going through every second of WU data before it's sent?

I'm suggesting making an occulting screen probably something like that of a drive-in movie theatre but that just may be too expensive and on someone elses property. Also the probability of hurricanes and tornadoes. Maybe a mesh, rather than a light-opaque structure would be just as effective.

If you have pending problems with garbage on the fibre cables you could check the mass points on the system.

One problem could be that the shielding of the fibre cables is directly connected on both ends of the cable. Its strongly recommended to connect ONLY one point directly with the mass point and the other over a spezial surge protector.

Another problem could be that the mass points of the server array is to long. If you connect too many servers over a larger distance with the same mass point the equalizing current of the mass wire could also cause serious problems on the sending and receiving modules. (You know these problems if you ever connected your hifi with the television....)

These two problems can occour and cause such problems. Mostly this problems are not taken care of at the construction of the servers, cause of the increased costs of building.

I already had such problems..... maybe i could help.....


Excuse me for the english.... its a long time since last using it ;)

Absorbing the signal would be preferable to reflecting it for the radar op's sake! hehe Unless it were reflected at an angle so as not to bounce straight back at them!

Well I'd send it back to them amplified a few billion times... but that's me!

Problem with any amount of terrestrial rfi, it massively outweighs those we are looking for. Blocking rfi is difficult because of diffraction and reradiation, and the pylons will also resonate and reradiate all kinds of frequencies and pick up much more than just the radar. A huge multiwalled Faraday cage might help.

Damned inconvenient to have one within line-of-sight-and-a-bit-more near Arecibo. The power of these things can be 10s of kW so even when not facing the antenna, leakage could still be considerable.

Those responsible for the radar should also be responsible for it not interfering with their neighbours, and a radar absorber/attenuator/blocker or switcher offer would be more effective closer to the source.

Saving grace is that the signal can be neutralized somewhat, or skipped over but there can still be annoying traces.

Ideally would need to build a really big mountain around the dish and turn it
into a large tube, or move to the far side of the Moon and recruit a few convenient craters.

Don't worry about sending it back to them. Their software will just remove it as ground clutter.

Gary

I'm suggesting making an occulting screen probably something like that of a drive-in movie theatre but that just may be too expensive and on someone elses property. Also the probability of hurricanes and tornadoes. Maybe a mesh, rather than a light-opaque structure would be just as effective.

Hmm, good suggestion. It wouldn't have to be very big, just big enough to block the radar signal from the receiving antennas. Unless there's reflections from some other structure. It would have to be made of something opaque to the freq of the radar. Maybe the sort of radar absorbing material they use on the stealth and other planes. Absorbing the signal would be preferable to reflecting it for the radar op's sake! hehe Unless it were reflected at an angle so as not to bounce straight back at them!

Sorry, but you are confusing electrical signal cables with fiber optic cables. The grounding advice you gave is well applied to electrical cables, but fiber optics are immune to RFI and other electical interference, and do not require shielding which is earthed (grounded).

:) I know that fibre cables normally have no shielding, but sometimes the rodent protection of the cables is connected to the sending and the receiving modul, or to the server rack. This can cause such problems, not because of electrical interference in the cable itself, but in the sending or receiving modul.


I just wanted to mention it, cause i had such problems twice (with fibre optic) and it took me 2 months to locate the problem.