how to build a good test bed?

From memory, population size was about 20. It was something between a gradient descent and a genetic algorithm, by moving from the stronger members away from the weaker members, plus some random component. Remember, I had already extracted all of the features etc, and saved them just before inserting into the Kd-Tree, so the only thing I needed at evaluation time was:

1. read data from file
2. add points to the tree
3. KNN/KDE
4. count inliers vs outliers -> give a score

Then at the end multiply the evolved weights with the code weights, recompile, and collect a new set of data; repeat until happy.

Just to clarify, maybe not in the classic way, but your algo is more a elitist than a mutator, is that it?

I'm doing exactly that, just ran a generation of pop. size 30 on top of 155 battles from top bots of the Rumble and it took me about 15 minutes. I'll debug what is taking this much time later. Thanks for your help :)

Make sure you are only simulating aiming on waves that you actually fired.

So are you using something similar to PBIL instead of traditional GAs?

Cool, I have to put my hands on some ML texts :)

I think the closest would be something between gradient descent and stochastic learning.

So stochastic gradient descent?

Still not quite, because it uses a population like GA does, and used linear combinations between the population to estimate gradient similarly to how gradient descent would. Honestly, there were probably better/faster algorithms that would have worked better out-the-box, but this worked fine, it just took a bit longer.

Well, this combination sounds great, and it is more like how I'm tuning weights by hand than traditional GAs. And this way it should work way better than hand, as it's running way more battles with way more population.

And it's way faster (and also with less deviation) with recorded battles. The only problem is overfitting the recorded battles, but that should be solved well with many tune–rerecord iterations.

Anyway, I'm still wondering about — will it forget the previous tune–rerecord iterations to overfit new iterations? Anyway, since it sounds more like metric learning, it won't surprise me if this one is different. Did you experiment rerunning the old battles after tuning for newer ones to see that?