upload issues and other news (Oct 5, 2007)


log in

Advanced search

Message boards : Technical News : upload issues and other news (Oct 5, 2007)

1 · 2 · Next
Author Message
Jeff Cobb
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 1 Mar 99
Posts: 110
Credit: 40,367
RAC: 0
United States
Message 654763 - Posted: 5 Oct 2007, 21:54:54 UTC

Matt is still away on his well deserved vacation so I will summarize the week.

Last weekend we had 3 servers go down, as Eric described in the previous tech note. Two of these were attached to a UPS that malfunctioned. Not good, but at least we understand what happened. The third machine, bruno, crashes every week or two and hangs on reboot for reasons we have yet to understand. Our best guess at this time is that the fiber connection to the disk array that holds the upload directory is sometimes throwing garbage onto the bus that the machine cannot gracefully handle. This is an old fiber array that we would like to phase out anyway, so we thought about different storage devices that we currently have that could hold the uploads. We came up with the underutilized disk space on the master science database machine, thumper. This could have the added benefit of hosting the assimilators on the same machine that hosts the back end science database. Eric ran a script that gradually migrated the uploads over to thumper.

This worked fine until the migration reach a critical point, at which time the loads on the two download machines shot up to the 80-100 range (they are usually at 5 or less). The high loads were because each instance of the file_upload_handler was taking a long time to write the uploaded results over to thumper. To make a long story short, it turns out that the volume on thumper that held the new upload directory was getting slammed by the uploads. It was running at nearly 100% utilization (local disk, not network, utilization). This was, and still is, a bit surprising. The volume on bruno is software RAID50 and on thumper the volume is software RAID5, the latter having 2 more spindles than each of the RAID50 mirrors on bruno. At any rate, we are migrating back to the fiber array on bruno and have already seen download performance normalize. We'll have to figure this one out...

The other systems news of the week involves database replication on both of our production databases. The seti_boinc database (users, hosts, teams, recent results) replica was lost to a machine crash. We restored from the master and the replica is once again running normally. We are getting very close to having a replica of the back end science database. The initial data load is nearly complete. We will turn on replication either over the weekend or early next week.

Over in science development we are getting the splitter ready to handle the radar blanking signal that will be embedded in all new data once Arecibo comes back on line later this month.

-- Jeff

Profile Blurf
Volunteer tester
Send message
Joined: 2 Sep 06
Posts: 7578
Credit: 6,973,562
RAC: 3,121
United States
Message 654782 - Posted: 5 Oct 2007, 22:35:07 UTC

Thank you for the update Jeff.
____________


Profile petrusbroder
Send message
Joined: 2 Dec 01
Posts: 9
Credit: 30,048,082
RAC: 1,095
Sweden
Message 654791 - Posted: 5 Oct 2007, 22:59:50 UTC - in response to Message 654763.

Thanks for the info, I'll post a link @ TeAm AnandTech - there have been a few ;) questions ...
____________

Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15993
Credit: 690,597
RAC: 0
United States
Message 654827 - Posted: 6 Oct 2007, 0:29:06 UTC


Thanks Immensely for the Update Jeff . . . Nice Work Berkeley . . .

DJStarfox
Send message
Joined: 23 May 01
Posts: 1045
Credit: 560,066
RAC: 497
United States
Message 654882 - Posted: 6 Oct 2007, 2:25:06 UTC - in response to Message 654763.

I think that damn military radar has been the source of the -9 errors and noise in a lot of the WU in the last 6 months or so.

Profile Jim-R.
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 7 Feb 06
Posts: 1475
Credit: 46,565
RAC: 0
United States
Message 654941 - Posted: 6 Oct 2007, 4:12:26 UTC - in response to Message 654882.

I think that damn military radar has been the source of the -9 errors and noise in a lot of the WU in the last 6 months or so.

It is in *all* of the multibeam data. As I recall, the linefeed array was lower down on the structure so the radar signal was blocked by a nearby hill. The multibeam array is positioned higher so it receives the signal. The splitters have been configured to remove most of it, but some still gets through. Right now it's been a tradeoff between the amount of radar signal in the work units and the amount of good data that is not split due to the possibility of radar contamination. When they get the new setup working they can remove the radar signal and *only* the radar signal.
____________
Jim

Join GQFCrunchers
Problems?
Check the Boinc Wiki
Or NEW Enhanced FAQ

Profile JLDun
Volunteer tester
Avatar
Send message
Joined: 21 Apr 06
Posts: 307
Credit: 51,623
RAC: 0
United States
Message 655042 - Posted: 6 Oct 2007, 10:04:19 UTC - in response to Message 654941.

I think that damn military radar has been the source of the -9 errors and noise in a lot of the WU in the last 6 months or so.

It is in *all* of the multibeam data. As I recall, the linefeed array was lower down on the structure so the radar signal was blocked by a nearby hill. The multibeam array is positioned higher so it receives the signal. The splitters have been configured to remove most of it, but some still gets through. Right now it's been a tradeoff between the amount of radar signal in the work units and the amount of good data that is not split due to the possibility of radar contamination. When they get the new setup working they can remove the radar signal and *only* the radar signal.

Thanks for this explanation.


When they get the new setup working they can remove the radar signal and *only* the radar signal.

I know I've seen an explanation before, but how can SETI 'be sure' they've removed the "d*** military radar" from the WU's? Without going through every second of WU data before it's sent?
____________

Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8633
Credit: 51,543,643
RAC: 48,299
United Kingdom
Message 655067 - Posted: 6 Oct 2007, 10:27:03 UTC - in response to Message 655042.

I know I've seen an explanation before, but how can SETI 'be sure' they've removed the "d*** military radar" from the WU's? Without going through every second of WU data before it's sent?

There's a Scientific Newsletter article about it.

Scrat
Send message
Joined: 12 Jul 00
Posts: 2
Credit: 139,558
RAC: 0
Austria
Message 655268 - Posted: 6 Oct 2007, 17:45:42 UTC

If you have pending problems with garbage on the fibre cables you could check the mass points on the system.

One problem could be that the shielding of the fibre cables is directly connected on both ends of the cable. Its strongly recommended to connect ONLY one point directly with the mass point and the other over a spezial surge protector.

Another problem could be that the mass points of the server array is to long. If you connect too many servers over a larger distance with the same mass point the equalizing current of the mass wire could also cause serious problems on the sending and receiving modules. (You know these problems if you ever connected your hifi with the television....)

These two problems can occour and cause such problems. Mostly this problems are not taken care of at the construction of the servers, cause of the increased costs of building.

I already had such problems..... maybe i could help.....


Excuse me for the english.... its a long time since last using it ;)

Profile Clyde C. Phillips, III
Send message
Joined: 2 Aug 00
Posts: 1851
Credit: 5,955,047
RAC: 0
United States
Message 655284 - Posted: 6 Oct 2007, 18:29:14 UTC

I'm suggesting making an occulting screen probably something like that of a drive-in movie theatre but that just may be too expensive and on someone elses property. Also the probability of hurricanes and tornadoes. Maybe a mesh, rather than a light-opaque structure would be just as effective.
____________

Profile Odan
Send message
Joined: 8 May 03
Posts: 91
Credit: 15,331,177
RAC: 0
United Kingdom
Message 655312 - Posted: 6 Oct 2007, 19:13:41 UTC - in response to Message 655268.

If you have pending problems with garbage on the fibre cables you could check the mass points on the system .....


Scrat, the fibres are _optical_ fibres so electrical problems such as grounding (mass?) do not have any effect.

Excuse me for the english.... its a long time since last using it ;)

Your English is fine :~}
____________

Profile Jim-R.
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 7 Feb 06
Posts: 1475
Credit: 46,565
RAC: 0
United States
Message 655315 - Posted: 6 Oct 2007, 19:15:32 UTC - in response to Message 655284.

I'm suggesting making an occulting screen probably something like that of a drive-in movie theatre but that just may be too expensive and on someone elses property. Also the probability of hurricanes and tornadoes. Maybe a mesh, rather than a light-opaque structure would be just as effective.

Hmm, good suggestion. It wouldn't have to be very big, just big enough to block the radar signal from the receiving antennas. Unless there's reflections from some other structure. It would have to be made of something opaque to the freq of the radar. Maybe the sort of radar absorbing material they use on the stealth and other planes. Absorbing the signal would be preferable to reflecting it for the radar op's sake! hehe Unless it were reflected at an angle so as not to bounce straight back at them!
____________
Jim

Join GQFCrunchers
Problems?
Check the Boinc Wiki
Or NEW Enhanced FAQ

Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15993
Credit: 690,597
RAC: 0
United States
Message 655352 - Posted: 6 Oct 2007, 20:15:17 UTC - in response to Message 655315.

I'm suggesting making an occulting screen probably something like that of a drive-in movie theatre but that just may be too expensive and on someone elses property. Also the probability of hurricanes and tornadoes. Maybe a mesh, rather than a light-opaque structure would be just as effective.

Hmm, good suggestion. It wouldn't have to be very big, just big enough to block the radar signal from the receiving antennas. Unless there's reflections from some other structure. It would have to be made of something opaque to the freq of the radar. Maybe the sort of radar absorbing material they use on the stealth and other planes. Absorbing the signal would be preferable to reflecting it for the radar op's sake! hehe Unless it were reflected at an angle so as not to bounce straight back at them!



The First Invisibility Shield . . . Technology mighten be 'incorporated into' said 'meshin' . . . hmmm?

Profile Andy Lee Robinson
Avatar
Send message
Joined: 8 Dec 05
Posts: 615
Credit: 42,667,664
RAC: 26,625
Hungary
Message 655391 - Posted: 6 Oct 2007, 21:18:14 UTC - in response to Message 655315.
Last modified: 6 Oct 2007, 21:19:53 UTC

Absorbing the signal would be preferable to reflecting it for the radar op's sake! hehe Unless it were reflected at an angle so as not to bounce straight back at them!


Well I'd send it back to them amplified a few billion times... but that's me!

Problem with any amount of terrestrial rfi, it massively outweighs those we are looking for. Blocking rfi is difficult because of diffraction and reradiation, and the pylons will also resonate and reradiate all kinds of frequencies and pick up much more than just the radar. A huge multiwalled Faraday cage might help.

Damned inconvenient to have one within line-of-sight-and-a-bit-more near Arecibo. The power of these things can be 10s of kW so even when not facing the antenna, leakage could still be considerable.

Those responsible for the radar should also be responsible for it not interfering with their neighbours, and a radar absorber/attenuator/blocker or switcher offer would be more effective closer to the source.

Saving grace is that the signal can be neutralized somewhat, or skipped over but there can still be annoying traces.

Ideally would need to build a really big mountain around the dish and turn it
into a large tube, or move to the far side of the Moon and recruit a few convenient craters.

Scrat
Send message
Joined: 12 Jul 00
Posts: 2
Credit: 139,558
RAC: 0
Austria
Message 655481 - Posted: 7 Oct 2007, 0:03:39 UTC - in response to Message 655371.



Sorry, but you are confusing electrical signal cables with fiber optic cables. The grounding advice you gave is well applied to electrical cables, but fiber optics are immune to RFI and other electical interference, and do not require shielding which is earthed (grounded).


:) I know that fibre cables normally have no shielding, but sometimes the rodent protection of the cables is connected to the sending and the receiving modul, or to the server rack. This can cause such problems, not because of electrical interference in the cable itself, but in the sending or receiving modul.


I just wanted to mention it, cause i had such problems twice (with fibre optic) and it took me 2 months to locate the problem.

Profile Gary CharpentierProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 12729
Credit: 7,257,521
RAC: 17,252
United States
Message 655484 - Posted: 7 Oct 2007, 0:07:23 UTC - in response to Message 655315.
Last modified: 7 Oct 2007, 0:16:03 UTC

Don't worry about sending it back to them. Their software will just remove it as ground clutter.

Gary

I'm suggesting making an occulting screen probably something like that of a drive-in movie theatre but that just may be too expensive and on someone elses property. Also the probability of hurricanes and tornadoes. Maybe a mesh, rather than a light-opaque structure would be just as effective.

Hmm, good suggestion. It wouldn't have to be very big, just big enough to block the radar signal from the receiving antennas. Unless there's reflections from some other structure. It would have to be made of something opaque to the freq of the radar. Maybe the sort of radar absorbing material they use on the stealth and other planes. Absorbing the signal would be preferable to reflecting it for the radar op's sake! hehe Unless it were reflected at an angle so as not to bounce straight back at them!


____________

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5864
Credit: 60,549,787
RAC: 47,666
Australia
Message 655492 - Posted: 7 Oct 2007, 0:15:19 UTC - in response to Message 655489.

I suppose if the 2 interconnected pieces of equipment were not on the same ground bus there could be some differential between them thus inducing some current flow from one to the other.

All too common.
Earth loops can be an nightmare to track down & resolve.

____________
Grant
Darwin NT.

1 · 2 · Next

Message boards : Technical News : upload issues and other news (Oct 5, 2007)

Copyright © 2014 University of California