WaveSim

Background

To facilitate a new wave of gun research, I created a system that lets me test a gun's classification algorithm against pre-gathered wave data. This can take the place of running a zillion Robocode battles.

Benefits:

• By removing all the overhead of running actual battles, I can run a gun benchmark much faster -- like minutes instead of hours.
• Tests can be run against data from real battles instead of a TC scenario, which should be more relevant to real battle performance.
• Every benchmark runs against the exact same data, resulting in direct comparisons of test results without worrying about randomness.
• I can see hit percentages (normalized or not) or energy ratios (spent vs gained vs damage), which I think are a more direct measurement of a gun's accuracy than TC results.

Testing classification algorithms against raw data sets is nothing new or special, but it's about time someone set up a solid system for doing it with Robocode!

(Note that a good classification algorithm is not the only important part of a gun. But I feel that Diamond and most other top bots have the rest of their gun code polished to a point where classification is the thing we work on 99% of the time.)

Components

Data-gathering bot

A modified TripHammer outputs every gun wave it gathers to a comma delimited (CSV) file. Each battle gets a new filename and the bot appends to it at the end of each round.

You could, of course, modify your own bot to output wave data to a CSV file instead. Just make sure you update WaveRecord to understand your format.

Battle simulation

Each wave logged during a battle gets a unique ID, and each record contains the last wave ID that was collected before this wave was fired. To run a battle, we:

• Reset the classifier.
• Loop through the wave data:
  • Find the next firing wave.
  • Read each unread wave up to the "last seen" wave for this firing wave, feeding each wave to the classifier to train it.
  • Ask the classifier to produce a firing angle for this firing wave.
  • Check if this firing angle would have hit and update stats (hit percentages, energy gained/lost).

This should simulate precisely how this wave data would be learned and used in a real battle.

Classifiers

The Classifier interface implements a targeting algorithm. It's pretty simple. The main methods are:

• initalize() - Set up data structures, clear any existing data.
• feed(WaveRecord wr) - Train (learn from) a wave.
• classify(WaveRecord wr) - Attempt to classify (produce a firing angle) for this wave.

These two methods can specify alternate schemes for training:

• boolean refeed() - Instead of feeding all unread waves, feed all waves from the beginning of the battle. Some algorithms (though perhaps not suited to actually using in Robocode) require the entire data set; this saves you from needing to handle that in your Classifier.
• boolean feedOnce() - Feed the entire data set once, up front, before classifying the firing waves. Useful for testing algorithms that take a long time to train, since it's much faster than refeed.

Downloads

WaveSim

• WaveSim 1.1 - Source, .class files, and Javadoc to simulate battles using raw wave data.

If you put this on your build path, you should be able to:

• Set WaveRunner.BASEDIR to point to your data files directory.
• Implement the Classifier interface, which is your targeting algorithm. Methods for initialization/reset, training a wave, and classifying a wave. (See the Javadoc for reference.)
• Start simulating battles!

The doc directory in the JAR describes the only three classes you need to know: WaveRunner, WaveRecord, and the Classifier interface. Of course, you'll need to gather or download some data first (see below).

There is one sample data file in the JAR at voidious/test/wavesim/jam.mini.Raiko 0.43_1_wavedata.txt. Unzip the JAR and try: java -cp ~/robocode/libs/robocode.jar:. voidious.test.wavesim.WaveRunner -- this will simulate a battle vs Raiko using the Head-On Targeting classifier and the KNN classifier from Diamond's main gun. (Diamond's gun is what needs the Robocode JAR.) I use the main(...) method in WaveRunner as a sandbox for setting up whatever tests I want to run.

TripHammer RES

These two versions of TripHammer will output comma delimited data files containing all the waves collected during battles. They are in the format used by WaveRunner. Make sure to up your disk quota in robocode.properties. (robocode.robot.filesystem.quota=2000000000 will give you 2 gigs, enough for a few hundred battles or ~50 seasons of TCRM.)

• TripHammer RES1.6 - Diamond's movement and main gun.
• TripHammer RES1.6T - Diamond's main gun in TC mode.

Blog

1/25/2011

Been revisiting this, adding stuff that other people would probably need before it's of any use to them, like normal MEA and Wall Distance attributes. Almost ready for a new release. I also plan to add an option (on by default) to use random movement instead of Diamond's wave surfing, for faster data collection, and I'll probably zip up some gigs of sample data for download.

Of course, I ended up getting derailed using it to tune Diamond's gun first. =P

Just cleaned up this page quite a bit too.

3/17/2010

Posted first public version of my WaveSim utility. Have fun. =)

Of course I quickly realized that I overlooked something. The WaveRecord class also needs to be documented. And this uses precise MEA and wall distance based on precise MEA. I should add options for traditional MEA and orbital wall distance.

3/12/2010 pm

Super busy with work, so I can't focus on this much, but I've got the basics working now. For 100 seasons of TCRM:

TripHammer KNN classifier:
Took 1262.0 seconds
Hit percentage: 11.813935415920957
Energy ratio: 0.983354371037666

HOT classifier:
Took 341.0 seconds
Hit percentage: 8.739709205199102
Energy ratio: 0.7279814945288104

HOT score seems pretty high, but it gets 5.9% vs DT, which seems right. At avg distance of 500, you'd get about 5% against a perfectly flat movement.

Best of all is < 1 second per battle for Diamond's main gun. A real TCRM is 15-20 seconds per battle for this gun. Still have some obvious little optimizations to add and there are probably some bigger ones once I focus on it.

3/12/2010 am

I'll post all this soon (except the 4 gigs of data), if anyone's interested in tinkering with it. I may also post info about some of the clustering schemes I've tried. An example of one of the really intensive clustering algorithms that I find interesting is QT clustering. A couple of my previous experiments were inspired by that one.