Busy Bytes (Jul 06 2009)

There is no "prize" at the end of a fictitious line here, so if there is down time, people should understand that, take a deep breath and wait for it to clear. Being #1 cruncher gets you what, bragging rights on a message board!? Matt is in the most awful position here by having to obey orders, even if he disagrees, or has better ideas that are falling on def ears. It would be great for another 80k, but with today's financial rout, that will be just as hard as have a problem free seti@home day. BTW, I'll be in the top 50 of my class tomorrow, first milestone!
____________

Re: Maxed out Bandwidth.

This seems to cause the most anger amongst crunchers because the uploads are blocked, and when that happens some hosts run out of work and can't download any more WU's. Users then increase cache size to try to hold enough work to tide them over, which just makes the problem worse.

Why not restrict bandwidth of the (just the) download servers to 80-85Mb/s?

This would obviously lengthen the time of max DL bandwidth, but leave enough bandwidth for the uploads to get through. This in turn would:-
1). Reduce storage requirements for 'In progress' work
2). Allow crunchers to get new work because they can upload completed work. (reduce frustrations)
3). Reduce need for very large cache.

Theoretically everybody would get some (enough) work, and large caches would slowly fill over time, faster as demand reduced.

Is there a flaw in my logic?

Are there any "spare" servers, down the hill or elsewhere on/off campus, belonging to the Space Sciences Lab that could be used as a download mirror for data distribution to clients, to relieve pressure on the normal servers?
Nothing complicated - this would be only for downloads of a large reservoir of pre-split WUs at busy times. All processed data would be returned by clients to the normal servers up the hill. It could be one of the student's jobs to swap in a new drive of pre-split data every few days.
____________

I was just about to suggest using mirrors. Looks like Einstein@Home use them.

The mirror server(s) could be anywhere, there's 900mbit of unused capacity in that gigabit line to upload splitted wu's to mirror server(s). Of course the wu's can be splitted at the mirror. The current Seti servers could do the other jobs and maybe some of the download traffic.
____________

It's not possible to make it wireless?

An other idea.. (maybe)

I have built the "heart" of the statistics of our team. I realize my "problems" are of a much smaller magnitude compared to Berkeleys, but I do think there are similarities.

One MySql table, the user history, has been trouble for a while now.. probably due to it's size of almost 8 Gb.. (like: data corruption => complete restore from backup, sql queries getting stuck => detection of inactivity, restarting history generation with detection as to "were was I")..

To fix this I first tried an other engine (innoDB instead of MyISAM), no joy (table got much larger and processing got much slower).

Next we'll try to upgrade to a newer MySql engine.. but I fear that will not (completely) fix the problem.

Now I am thinking of splitting the data in several identical tables containing:
(arg, [ code ] doesn't preserve multiple spaces)

I think a picture really makes it clear what the effect of the recent problems have been on the grid. SETI's growth has effectively stopped.

How about sending more of the AP 5.05 and less Seti@home 6.03. More time out in the field processing with less contacts to the sever for more work. Days worth work at a time instead of hours.

Sure the initial change might be a little rough, but I think it will smooth out once you get enough "larger" workunits out in the field.

Once production of more larger work units is up to par, then re-examine the workunit compression idea. See if it would now be worth it due to the amount of AP units has been increased.

Is it possible to produce workunits for an offsite mirror while the system is down for weekely Tues. Maint.??? Stock up while things are cleaned up? This may reduce the startup stress on the servers when it all comes back online. Let the offsite mirror take the load and give the main server time to build up some workunits.


Just some things I wanted to throw out there for people to chew on.

- W
____________

How about sending more of the AP 5.05 and less Seti@home 6.03. More time out in the field processing with less contacts to the sever for more work. Days worth work at a time instead of hours.

Sure the initial change might be a little rough, but I think it will smooth out once you get enough "larger" workunits out in the field.

Matt,
If this is really a science/research project, then there are only two solutions to the bandwidth problem.

1. Get more funding
2. Deploy the NTPCkr

I HIGHLY favor these two over introducing some non-productive or artificial latency. From these two goals, other ideas become possible.

I would really like to install a server-side instance of BOINC on my machine at home, but I need to make time and upgrade drive space. It would really help me understand how this all works, and it sounds like something cool to get into.

Having read these forums for a while I appreciate your pain.

I am not a techie, but a money and marketing guy.

A few thoughts:

Define the probem, not just the symptoms (scientific)

Define the best solution (scientific)

Define the resources needed (scientific, poitical, monetary)

Create a liasion team to assist in the non - scientific issues.

Delegate the tasks to responsible people (of which there are thousands trying to help you)

Implement - Evaluate - Adjust - do it again.

I am sure you realize you are on the bleeding edge of technology You will eventually write the book (if you are not writing now) on distributed processing. I would think that several commercial organizations would be more than happy to review your work for a mere donation of money or equipment. You have ten+ years of experience going where no one has gone before (pardon the pun)

You and the guys at Berkley are making tremendous progress in computer science as well as Astronomy.

Like any other not for profit (not non-profit) organization, you have grown to the pont where you need additional infrastructure in terms on people skills to allow you to focus on your goals.

We are fully behind you! Take the step. Dare to be great, don't settle for good enough.

Go get em, tiger!

Greg
____________
Greg in Phoenix