how to build a good test bed?

Fragment of a discussion from Talk:ScalarBot/Version History
← Thread:Talk:ScalarBot/Version History/how to build a good test bed?/reply (3)

Jump to navigation Jump to search

Until the pairing is complete, APS is not a good indicator. I always go to the details of my bot and then select an older version to compare with. In that case only the bots that both versions have fought, are taken into account. It indeed seems that the last 10% of the pairings involve the best opponents, GrubbmThree held around 58 APS till approx 1000 pairings, then fell down to 57.2. Note that even with 3000-5000 battles, there are still a lot of bots you have only have one fight against, so a few bad battles do have influence.

As for testbed, I used to have around 20 bots in my testbed (50 seasons): 5 top-50 bots, 5 'white-whales', 5 between place 100-300 en a few specific ones to check whether something was broke (f.e. bbo.RamboT must score less than 0.5%)

GrubbmGait (talk)‎

You do not have permission to edit this page, for the following reasons:

The action you have requested is limited to users in the group: Users.
You must confirm your email address before editing pages. Please set and validate your email address through your user preferences.

You can view and copy the source of this page.

Thanks for figuring out that! I thought the rumble is stabilized very fast (common APS diff of ~300 pairings are already useful), but it turned out not. 

Maybe I should build a test bed with more varieties, e.g. bots from all over the rumble with different kind of strategies.

Return to Thread:Talk:ScalarBot/Version History/how to build a good test bed?/reply (4).

Retrieved from "http://robowiki.net/wiki/Thread:Talk:ScalarBot/Version_History/how_to_build_a_good_test_bed%3F/reply_(3)#how_to_build_a_good_test_bed.3F_5793"