My last post discussed optimizing scoring - changing the weights of the 3 terms in the multiplet score function to make birdie multiplets (which are surrogates for ET) score higher relative to non-birdie multiplets.

My first attempt at this - to my surprise - produced negative weights. An example is shown here:

Example 1

Let me explain this page. First it shows the parameters of the optimization. "bary = True" means we're looking at barycentric multiplets. "min_weight = -1" means the weights can be negative.

The "Before" section shows the number of multiplets and their median scores (we use only the 10000 top-scoring non-birdie multiplets). "Found" says how many birdie multiplets had scores ranking among the top 1000 non-birdie multiplets.

In the graph, each point is a multiplet. The X coordinate is its original score, and the Y coordinate is its core after optimizing the weightings. Project the points onto the X axis to see the relationship of birdie to non-birdie with original scores; project onto the Y axis for weighted scores.

Here we're looking at the barycentric case. With optimized score, ALL 1369 of the birdie multiplets are found. That's good, right? Well, actually not. The optimized weight of signal_factor is -1.03. Apparently non-birdies have more of this than birdies, so giving it a negative weight penalizes non-birdies. We're "finding" the birdie multiplets by making good non-birdies look bad.

So allowing negative weights gives us nonsense results. We can disallow negative weights by adding a penalty term to the objective function: namely, if a weight is negative we add a multiple of its square. This gives us (in the barycentric case):

Example 2

This is a big improvement: the number of found birdies went from 339 to 776. The general slope of the graph is positive, but I don't like the fact that the non-birdie scores have been sort of flattened out.

What if we make the minimum weight something bigger, like .3?

Example 3

The #found is down to 463.

The objective function uses the median of the top 1000 non-birdie multiplets. From the point of "finding" birdies what matters is the 1000th-best score, so what if we use that instead? Adding this to Example 2 we get

Example 4

which, curiously, is not an improvement.

One more idea - since we're really trying to maximize the number of found birdies, which not subtract some multiple (say 10) of this from the object function?

Example 5

That finds more birdies, but in spite of the penalty one of the weight is negative, and that shows in the negative slope of the non-birdies. Let's increase the min weight to .2:

Example 6

Hmm. Now we're back down again.

---

Wait a minute, something's wrong here! The optimization seems to want to push the weights to either plus or minus infinity.

Well, d'oh!! For a given score factor, if the average value for birdies is greater (worse) than for non-birdies, then of course the optimization will want to give it infinitely negative weight. This is indeed the case for signal factor; see the histograms.

So this approach - at least as I've formulated it - is garbage. Maybe there's a better approach to improving the score function, but after thinking about it for a few minutes, nothing occurs to me. So I'm going to put this project on the back burner for now.

In case you're curious, here are the analogous examples for non-barycentric multiplets:

Example 1
Example 2
Example 3
Example 4
Example 5
Example 6