It's interesting that 30 units/s creation rate isn't enough, at least during a shorty storm.

During a shorty storm 55/s is the minimum needed to meet demand.
The storm is over, but still the splitters aren't able to crank out enough work. For a while they were doing about 30/s (barely enough when there's a lot of VLARs in the mix). Now they've dropped down to less than 20.

Someone in the lab needs to take a look at what is going on- the splitters used to be able sustain 70/s no problems at all, now they can't even reach it as a peak.

Dunno...for 10-20 minutes, they were down to 10-15/second. You can see the resulting dip in the cricket graphs. I was kinda hoping they would add another one of them fancy new pfb splitters during today's outage, seeing as how the AP splitters have been far outrunning them as of late.
Last check, they were getting close to 30 again.

Might be something with the new pfb splitting thingy, although I think it was Richard that expressed the opinion he thought they might be a little faster than the old splitters.....hmmmmmm.
Might depend on the work they are splitting.

---

"Time is simply the mechanism that keeps everything from happening all at once."

Might be something with the new pfb splitting thingy, although I think it was Richard that expressed the opinion he thought they might be a little faster than the old splitters.....hmmmmmm.
Might depend on the work they are splitting.

I saw that mentioned, but so far they've proved to be about half the speed of the previous ones. 3/4 speed if they're really flying.
They're definately borked.

---

Grant
Darwin NT

Might be something with the new pfb splitting thingy, although I think it was Richard that expressed the opinion he thought they might be a little faster than the old splitters.....hmmmmmm.
Might depend on the work they are splitting.

I saw that mentioned, but so far they've proved to be about half the speed of the previous ones. 3/4 speed if they're really flying.
They're definately borked.

Well, I got to get some sleeps now with the kitties.

I suspect they shall get up and paw the keyboard once in a while to keep things in order whilst I sleep. And shall keep the lights on for any wayward WUs that need to find a good home.

All WUs are welcome to the kitty crunching farm.

Meow, and good night.

---

"Time is simply the mechanism that keeps everything from happening all at once."

It's interesting that 30 units/s creation rate isn't enough, at least during a shorty storm.

During a shorty storm 55/s is the minimum needed to meet demand.
The storm is over, but still the splitters aren't able to crank out enough work. For a while they were doing about 30/s (barely enough when there's a lot of VLARs in the mix). Now they've dropped down to less than 20.

Someone in the lab needs to take a look at what is going on- the splitters used to be able sustain 70/s no problems at all, now they can't even reach it as a peak.


EDIT- at least the shorty storm was over for a while. The work my systems were able to get after the outage while i was at work didn't have many shorties in it, but one of the systems was just able to get some more work (still nowhere near enough...) & it was almost all shorties.

The reason I was talking about a shorty storm was I had drained my queues to update BOINC and move from r390 to r1817 (my 3 cores pretty well match my low end GPU in performance so they were fairly close to running out at the same time) and when I turned off NNT I only got shorties for my CPU and the few GPU units I got were actually a mix but leaned toward shorties as well.

Edit: and once again without fail, the lack of GPU units get resolved as I'm writing about it. Woo Hoo. I finally have full queues again.

---

"Life is just nature's way of keeping meat fresh." - The Doctor

Splitters still not keeping up.

---

Grant
Darwin NT

Splitters still not keeping up.

The kitties shall just have to sniff and scratch and claw for all the WUs they can find for the crunchers today whilst I am at work.

They are usually pretty good at sniffing out Seti work.
Cache down about 330 from maximum right now across all 9 rigs, so they'll have to work at it.

But, it would be great if da boyz in da lab could commission one more of those neat pfb splitter jobs on the servers!

---

"Time is simply the mechanism that keeps everything from happening all at once."

Just within the last hour have I finally been able to top off my allocation on all three rigs. Looks like the shortie storm is at least temporarily over.

---

For some reason, I do have not been able to get any new units for most of the day, completely out of units to crunch as well, tried the usual kick to try and get things going but nothing :(

---

For some reason, I do have not been able to get any new units for most of the day, completely out of units to crunch as well, tried the usual kick to try and get things going but nothing :(

And, as is usually the case, 25 minutes after you posted, one of your rigs got work.

---

Splitters still unable to build a ready-to-send buffer.

---

Grant
Darwin NT

On 8 April 2013, as we came back up after the move to the Colocation Facility, Matt wrote this (emphasis mine):

Jeff and I predicted based on previous demand that we'd see, once things settled down, a bandwidth usage average of 150Mbits/second (as long as both multibeam and astropulse workunits were available). And in fact this is what we're seeing, though we are still tuning some throttle mechanisms to make sure we don't go much higher than that.

Why not go higher? At least three reasons for now. First, we don't really have the data or the ability to split workunits faster than that. Second, we eventually hope to move off Hurricane and get on the campus network (and wantonly grabbing all the bits we can for no clear scientific reason wouldn't be setting a good example that we are in control of our needs/traffic). Third, and perhaps most importantly, it seems that our result storage server can't handle much higher a load. Yes, that seems to be our big bottleneck at this point - the ability of that server to write results to disk much faster than current demand. We expected as much. We'll look into improving the disk i/o on that system soon.

I wonder if this is one of the throttles he's mentioned, to slow down the MB/pfb splitters so we don't overwhelm the result storage server and/or run out of data....

---

Donald
Infernal Optimist / Submariner, retired

On 8 April 2013, as we came back up after the move to the Colocation Facility, Matt wrote this (emphasis mine):

Jeff and I predicted based on previous demand that we'd see, once things settled down, a bandwidth usage average of 150Mbits/second (as long as both multibeam and astropulse workunits were available). And in fact this is what we're seeing, though we are still tuning some throttle mechanisms to make sure we don't go much higher than that.

Why not go higher? At least three reasons for now. First, we don't really have the data or the ability to split workunits faster than that. Second, we eventually hope to move off Hurricane and get on the campus network (and wantonly grabbing all the bits we can for no clear scientific reason wouldn't be setting a good example that we are in control of our needs/traffic). Third, and perhaps most importantly, it seems that our result storage server can't handle much higher a load. Yes, that seems to be our big bottleneck at this point - the ability of that server to write results to disk much faster than current demand. We expected as much. We'll look into improving the disk i/o on that system soon.

I wonder if this is one of the throttles he's mentioned, to slow down the MB/pfb splitters so we don't overwhelm the result storage server and/or run out of data....

Possibly, although the problem continues even when there aren't tonnes of shorties going through the system. Prior to the move & the change of splitter type it wasn't an issue.

---

Grant
Darwin NT

The blue line on the cricket graph is certainly interesting for the last 18 or 19 hours.

---

David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.

The blue line on the cricket graph is certainly interesting for the last 18 or 19 hours.

More datasets being transferred from the lab to the colo in preparation for splitting, I suspect.

God bless the new bandwidth, eh?

---

"Time is simply the mechanism that keeps everything from happening all at once."

The blue line on the cricket graph is certainly interesting for the last 18 or 19 hours.

More datasets being transferred from the lab to the colo in preparation for splitting, I suspect.

I hope so, with only about 300 MBs ready to send.

God bless the new bandwidth, eh?

Yup.

---

David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.

The blue line on the cricket graph is certainly interesting for the last 18 or 19 hours.

More datasets being transferred from the lab to the colo in preparation for splitting, I suspect.

I hope so, with only about 300 MBs ready to send.

That's a problem with the splitting rate, not the amount of data waiting to be split. There is plenty of data awaiting splitting right now.

We need another pfb splitter online.

Unless the powers that be are deliberately holding back to spare the bandwidth or database a bit.

---

"Time is simply the mechanism that keeps everything from happening all at once."

The blue line on the cricket graph is certainly interesting for the last 18 or 19 hours.

More datasets being transferred from the lab to the colo in preparation for splitting, I suspect.

I hope so, with only about 300 MBs ready to send.

That's a problem with the splitting rate, not the amount of data waiting to be split. There is plenty of data awaiting splitting right now.

We need another pfb splitter online.

Unless the powers that be are deliberately holding back to spare the bandwidth or database a bit.

With the comments about not wanting to be too big a data hog, the limits on total units, the comment that we are going through the data faster than they are collecting it, I would think the conclusion is they are holding back on purpose.

Face it, there are more crunchers than there is Seti data. This is a good thing. Now how do we convince more of them to say join the 84+ cents a month club so that ntpckr development can continue?

---

The blue line on the cricket graph is certainly interesting for the last 18 or 19 hours.

More datasets being transferred from the lab to the colo in preparation for splitting, I suspect.

I hope so, with only about 300 MBs ready to send.

That's a problem with the splitting rate, not the amount of data waiting to be split. There is plenty of data awaiting splitting right now.

We need another pfb splitter online.

Unless the powers that be are deliberately holding back to spare the bandwidth or database a bit.

If they are holding back data, then they need to let us know this explicitly. It's a lot harder to get work units now compared to last week, but I'm making the assumption that this is temporary and we will return to a somwhat normal condition in a little while. If this is not going to be the case, I just want to know.

---

If they are holding back data, then they need to let us know this explicitly. It's a lot harder to get work units now compared to last week, but I'm making the assumption that this is temporary and we will return to a somwhat normal condition in a little while. If this is not going to be the case, I just want to know.

You have been with this project a long while so surely you realise by now that "project communication" is at best sparse. Someone, probably Matt, will eventually let us know what has been going on. I suggest just waiting a while, I am sure all will be revealed when the team have time.

---

If they are holding back data, then they need to let us know this explicitly. It's a lot harder to get work units now compared to last week, but I'm making the assumption that this is temporary and we will return to a somwhat normal condition in a little while. If this is not going to be the case, I just want to know.

You have been with this project a long while so surely you realise by now that "project communication" is at best sparse. Someone, probably Matt, will eventually let us know what has been going on. I suggest just waiting a while, I am sure all will be revealed when the team have time.

This post from Matt seems pretty explicit to me ...

On 8 April 2013, as we came back up after the move to the Colocation Facility, Matt wrote this (emphasis mine):

Jeff and I predicted based on previous demand that we'd see, once things settled down, a bandwidth usage average of 150Mbits/second (as long as both multibeam and astropulse workunits were available). And in fact this is what we're seeing, though we are still tuning some throttle mechanisms to make sure we don't go much higher than that.

Why not go higher? At least three reasons for now. First, we don't really have the data or the ability to split workunits faster than that. Second, we eventually hope to move off Hurricane and get on the campus network (and wantonly grabbing all the bits we can for no clear scientific reason wouldn't be setting a good example that we are in control of our needs/traffic). Third, and perhaps most importantly, it seems that our result storage server can't handle much higher a load. Yes, that seems to be our big bottleneck at this point - the ability of that server to write results to disk much faster than current demand. We expected as much. We'll look into improving the disk i/o on that system soon.

---

Donald
Infernal Optimist / Submariner, retired