Density Calculations

My Anti-Surfer gun uses 4 data sets. They are k=3 of last {125 | 400 | 1500 | 4000} data points. So the least-decaying one has ~5 rounds of data, and many of the selected data points will overlap.

8% vs 12% of raw hit rate over a few battles doesn't could still be just fluctuation. If it's a normalized hit rate that might be significant... I generally run a few hundred to a few thousand battles to test a change, for what it's worth.

Its normalized calculated by DrussGT. I just did 15 battles, it seems unlikely due to random fluctuation. Is your anti-surfer gun a composite of multiple other guns? How does it work?

Well, I consider it one gun. :-) But instead of a single KNN search, it does 4 of them and combines the results.

You should be able to hazard a guess at the margin of error by looking at the distribution of those 15 values. Isn't it 1.96 * standard deviation = 95% confidence interval? Edit: Er, then divided by sqrt(num battles)? I don't know, but it's not hard to find. :-)

Does it use different trees for the different searches? Isn't that assuming a normal distribution? The mean and stdev for k = 1 are 12.12 and .34, for k = 100 (at max) its 9.26 and .36. Seems significant. Haven't tried intermediate values, k = 2 seemed to decrease performance, but I didn't do a careful study.