Currently as I write this we're recovering from the weekly outage (during which we take care of database backups and other sundry server details). It may take a while...

This past Friday we overloaded our science database trying to create a new index. A database engine restart solved the problem, but not after choking the whole local network. As mentioned in many posts past, we're strangely sensitive to heavy network bandwidth (I think due to linux's imperfect handling of NFS dropouts), and such periods cause random unexpected events. This time, for example, the bottleneck from the primary science database server ultimately caused the BOINC/mysql replica server to disconnect from the master. So the replica fell behind all weekend. Sigh. Instead of actually letting it catch up we're just re-mirroring it from the master as we just backed it up this morning.

Meanwhile, we're out of space again on the workunit server, and with no fast/easy way to add space. Eric's playing with the splitter mix to reduce the number of Astropulse workunits being generated (they are much larger than SETI@home workunits). Maybe that will help, but not immediately. This is what's mostly causing our headaches today as we can't create enough work to keep up with demand.

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

. . . Thanks - as usual Matt - for your Updates - iT is certainly appreciated Sir!

> Eric - Best of Luck w/ the splitter mix - looks like another Servers required for the Astropulse Units eh . . .

< Accolades to Each of You @ Berkeley

---

BOINC Wiki . . .

Science Status Page . . .

Meanwhile, we're out of space again on the workunit server, and with no fast/easy way to add space. Eric's playing with the splitter mix to reduce the number of Astropulse workunits being generated (they are much larger than SETI@home workunits). Maybe that will help, but not immediately. This is what's mostly causing our headaches today as we can't create enough work to keep up with demand.

- Matt

Maybe if Astropulse paid more, there'd be more demand for its WUs... I'm about to finish up an AP WU sent to me 3 weeks ago, but it's worth less when it was issued to me than now... This will not thrill my new wingman who is claiming more credit than me. In the economy this is call deflation :-( WUs should be worth more over time rather than less... There is no investment value if the claim doesn't increase over time with new wingmen!

How can the same WU be worth more credit just because it had to be reissued?

Maybe all users should cancel their AP WUs so the WUs will increase in value!

---

Meanwhile, we're out of space again on the workunit server, and with no fast/easy way to add space. Eric's playing with the splitter mix to reduce the number of Astropulse workunits being generated (they are much larger than SETI@home workunits). Maybe that will help, but not immediately. This is what's mostly causing our headaches today as we can't create enough work to keep up with demand.

- Matt

Maybe if Astropulse paid more, there'd be more demand for its WUs...

I don't think that's what Matt meant, JD. I think he meant that they can't create enough work in general (both SETI & AstroPulse) to keep up with demand. There were a couple reports over in Q&A of users being unable to get more SETI work. The limiting factor is that both SETI and AP are using the same disk (or set of disks) and the drives are filling up fast making it impossible to create new work.

This time, for example, the bottleneck from the primary science database server ultimately caused the BOINC/mysql replica server to disconnect from the master. So the replica fell behind all weekend.

I'm not sure if anyone has mentioned this.... But did you try installing separate NIC cards in the database and replica servers? You could either 1) bundle them to make one faster NIC, or 2) dedicate the secondary NICs as DB replica traffic only (using private subnet).

Meanwhile, we're out of space again on the workunit server, and with no fast/easy way to add space. Eric's playing with the splitter mix to reduce the number of Astropulse workunits being generated (they are much larger than SETI@home workunits). Maybe that will help, but not immediately. This is what's mostly causing our headaches today as we can't create enough work to keep up with demand.

I think this is the wrong way around.

1 AP result file is about the size of 22 MB result files.
But to process 1 AP result takes longer then 22 MB results.

To reduce the need for diskspace you need the get more AP results to the clients not less.

---

... we overloaded our science database trying to create a new index. A database engine restart solved the problem, but not after choking the whole local network. As mentioned in many posts past, we're strangely sensitive to heavy network bandwidth (I think due to linux's imperfect handling of NFS dropouts), and such periods cause random unexpected events. ... server to disconnect from the master...

Is the real overload due to packet fragmentation, or lost packets due to congestion on one of your switches or server NICs?... NFS doesn't handle packet loss very well. Eventually, things time out but not before descending into a slow death...

Excessive fragmentation due to an MTU=1500? Can you use jumbo packets for that network? Or at least increase the MTU for a subnet of machines?

Too small kernel buffers for the network packet assembly?

Are the switches getting their routing tables poisoned from noise traffic from elsewhere?

Note: Optimizing NFS Performance

5.3. Overflow of Fragmented Packets

Using an rsize or wsize larger than your network's MTU (often set to 1500, in many networks) will cause IP packet fragmentation...

There's a few good comments in there.


Hope that helps,

Regards,
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

Meanwhile, we're out of space again on the workunit server, and with no fast/easy way to add space. Eric's playing with the splitter mix to reduce the number of Astropulse workunits being generated (they are much larger than SETI@home workunits). Maybe that will help, but not immediately. This is what's mostly causing our headaches today as we can't create enough work to keep up with demand.

I think this is the wrong way around.

1 AP result file is about the size of 22 MB result files.
But to process 1 AP result takes longer then 22 MB results.

To reduce the need for diskspace you need the get more AP results to the clients not less.

Your typical result file for MB is 23-35Kb, not Mb, your typical AP result is about 45 Kb... but a MB WU is 350Kb, and an AP WU is 8Mb - and they have to store the WU, so that if someone misses deadline, they can send it out again.

MB = Multi-Beam
AP = Astro Pulse

---

.

Hello, from Albany, CA!...

Meanwhile, we're out of space again on the workunit server, and with no fast/easy way to add space. Eric's playing with the splitter mix to reduce the number of Astropulse workunits being generated (they are much larger than SETI@home workunits). Maybe that will help, but not immediately. This is what's mostly causing our headaches today as we can't create enough work to keep up with demand.

I think this is the wrong way around.

1 AP result file is about the size of 22 MB result files.
But to process 1 AP result takes longer then 22 MB results.

To reduce the need for diskspace you need the get more AP results to the clients not less.

Your typical result file for MB is 23-35Kb, not Mb, your typical AP result is about 45 Kb... but a MB WU is 350Kb, and an AP WU is 8Mb - and they have to store the WU, so that if someone misses deadline, they can send it out again.

MB = Multi-Beam
AP = Astro Pulse

I may have not be clear enough. I was talking about the 350 Kb and 8 Mb files.
8 Mb is about 22 * 350 Kb.

So my point still stands that it is beter to send out more AP.

---

Looks like more erratic behavior thanks to the bottlenecks du jour - the lack of the workunit storage and general NFS malaise. This will all clear up sooner or later. The NFS problems will always be there until we get about 50% more servers and shut everything down for a week to reorganize all the services and storage in such a manner that each system is only doing one thing. Until then server A is mounting server B for reason X, and server C is mounting server A for reason Y, etc. Anyway, don't see that getting better any time soon.

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Are you talking about having more actual servers or you just need more storage space? How much would something like that cost?

---

Are you talking about having more actual servers or you just need more storage space? How much would something like that cost?

I would expect both, and then also the time needed to plan and set it all up...

A short term help-fix could be to just install additional Gbit NICs in whichever servers and then wire direct point-point ethernet connections for exclusive use for the NFS between each pair of servers. Set fixed IPs and set the routing tables and viola! (You can also set a jumbo MTU=9660 to avoid all fragmentation. BIG performance boost for 8k NFS read/writes!)


Good luck,

Regards,
Martin

(You don't even need cross-over cables, the Gbit NICs will correctly autonegotiate for a normal patch lead.)

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

A short term help-fix could be to just install additional Gbit NICs in whichever servers and then wire direct point-point ethernet connections for exclusive use for the NFS between each pair of servers. Set fixed IPs and set the routing tables and viola! (You can also set a jumbo MTU=9660 to avoid all fragmentation. BIG performance boost for 8k NFS read/writes!)

You could even expand this idea by throwing in a small Gb switch and a few more NICS and slowly migrate all NFS links to connect over this separate network. Sort of a distributed NAS, with bonusses for offloading the existing network which handles the client traffic and slightly reducing the overall server load by reducing the serverbased network handling (less fragmentation and less resends).

A short term help-fix could be to just install additional Gbit NICs in whichever servers and then wire direct point-point ethernet connections for exclusive use for the NFS between each pair of servers. Set fixed IPs and set the routing tables and viola! (You can also set a jumbo MTU=9660 to avoid all fragmentation. BIG performance boost for 8k NFS read/writes!)

You could even expand this idea by throwing in a small Gb switch and a few more NICS and slowly migrate all NFS links to connect over this separate network. Sort of a distributed NAS, with bonusses for offloading the existing network which handles the client traffic and slightly reducing the overall server load by reducing the serverbased network handling (less fragmentation and less resends).

Nope. Poor idea. That gives little advantage over their existing Gbit switch. Note that the internet traffic on that switch maxes out taking about 8% capacity for bandwidth. However, it might be running out of routing/persistency table space...

The whole idea of a point-to-point link (NO intermediate switch) is to guarantee no packet loss due to switch overload or due to congestion on any one NIC.

The more immediate question is whether there are any spare IO slots for additional NICs... And whether Matt is game for more cuts and bruises in giving it a try...

Regards,
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

A short term help-fix could be to just install additional Gbit NICs in whichever servers and then wire direct point-point ethernet connections for exclusive use for the NFS between each pair of servers. Set fixed IPs and set the routing tables and viola! (You can also set a jumbo MTU=9660 to avoid all fragmentation. BIG performance boost for 8k NFS read/writes!)

The NICs would help, but the greater issue is that each server is doing multiple things, and hosting all kinds of files to all other servers. Basically every machine is mounting each other's disks, so if one system goes down all kinds of random hell breaks loose. Things like informix esql libraries, data sets, upload directories, download directories, home accounts, pieces of web sites...

Of course we'd like to clean this up, but that would require a lot more servers doing one specific thing (which means more servers *and* space *and* time to set all this up *and* move things safely to their new locations *and* make sure services haven't broken in the meantime).

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Note that the internet traffic on that switch maxes out taking about 8% capacity for bandwidth.

OK, this low load makes my suggestion a bad one.

... *and* ... *and* ... *and* ... *and* ...

Sounds like money (also as in 'time = ...') is more of an issue than anything else. That's science and the shoestring budgets that so often seem to come with it I guess...