Cache effects on benchmark

Fragment of a discussion from Talk:K-NN algorithm benchmark
← Thread:Talk:K-NN algorithm benchmark/Cache effects on benchmark/reply

Jump to navigation Jump to search

You do not have permission to edit this page, for the following reasons:

The action you have requested is limited to users in the group: Users.
You must confirm your email address before editing pages. Please set and validate your email address through your user preferences.

You can view and copy the source of this page.

Maybe using a reference bot would make benchmarking more meaninful?

Pick one bot, put each tree inside it, one at a time, and run it against a 1v1 test bed.

Return to Thread:Talk:K-NN algorithm benchmark/Cache effects on benchmark/reply.

For the best comparison, absolutely. However, it might be difficult to set up the battles so that every one is the same, particularly if the trees are non-deterministic due to things like points being equal. Also, it adds a lot of overhead which would make testing very slow.

Skilgannon (talk)‎

It would make testing slow, but the overhead is what will make benchmarks meaningful. Remove the overhead and you also remove cache thrashing.

Running a test bed much like we run challenges would do. Instead of making every battle the same, run multiple random battles and measure average run time.

MN (talk)‎

What I worry is that many different trees would have to be tested in exactly the same way. Unlike scores, times are different on different computers. Perhaps if we put them all in parallel in the same robot, then see how much time they take?

Skilgannon (talk)‎

Retrieved from "http://robowiki.net/wiki/Thread:Talk:K-NN_algorithm_benchmark/Cache_effects_on_benchmark/reply#Cache_effects_on_benchmark_3560"