WaveSim

Background

To facilitate a new wave of gun research, I created a system that lets me test a gun's classification algorithm against pre-gathered wave data. This can take the place of running a zillion Robocode battles.

Benefits:

• By removing all the overhead of running actual battles, I can run a gun benchmark much faster -- like minutes instead of hours.
• Tests can be run against data from real battles instead of a TC scenario, which should be more relevant to real battle performance.
• Every benchmark runs against the exact same data, resulting in direct comparisons of test results without worrying about randomness.
• I can see hit percentages (normalized or not) or energy ratios (spent vs gained vs damage), which I think are a more direct measurement of a gun's accuracy than TC results.

Testing classification algorithms against raw data sets is nothing new or special, but it's about time someone set up a solid system for doing it with Robocode!

(Note that a good classification algorithm is not the only important part of a gun. But I feel that Diamond and most other top bots have the rest of their gun code polished to a point where classification is the thing we work on 99% of the time.)

Components

Data-gathering bot

A modified TripHammer outputs every gun wave it gathers to a gzipped, comma delimited (CSV) file. Each battle gets a new filename and TripHammer writes to it at the end of each round.

You could, of course, modify your own bot to output wave data to a file instead. Just make sure you update WaveRecord to understand your format.

Battle simulation

Each wave logged during a battle gets a unique ID, and each record contains the last wave ID that was collected before this wave was fired. To run a battle, we:

• Reset the classifier.
• Loop through the wave data:
  • Find the next firing wave.
  • Read each (unread) wave that was collected before this firing wave was aimed and feed it to the classifier for training.
  • Ask the classifier to produce a firing angle for this firing wave.
  • Check if this firing angle would have hit and update stats (hit percentages, energy gained/lost).

This should simulate precisely how this wave data would be learned and used in a real battle.

Classifiers

The Classifier interface implements a targeting algorithm. It's pretty simple. The main methods are:

• initalize() - Set up data structures, clear any existing data.
• feed(WaveRecord wr) - Train (learn from) a wave.
• classify(WaveRecord wr) - Attempt to classify (produce a firing angle) for this wave.

These two methods can specify alternate schemes for training:

• boolean refeed() - Instead of feeding all unread waves, feed all waves from the beginning of the battle. Some algorithms (though perhaps not suited to actually using in Robocode) require the entire data set; this saves you from needing to handle that in your Classifier.
• boolean feedOnce() - Feed the entire data set once, up front, before classifying the firing waves. Useful for testing algorithms that take a long time to train, since it's much faster than refeed.

Downloads

WaveSim

• WaveSim 1.2.0 - Source, .class files, and Javadoc to simulate battles using raw wave data.

If you put this on your build path, you should be able to:

• Set WaveRunner.BASEDIR to point to your data files directory.
• Implement the Classifier interface, which is your targeting algorithm. Methods for initialization/reset, training a wave, and classifying a wave. (See the Javadoc for reference.)
• Start simulating battles!
• (You may need robocode.jar on your build path, as well, if any of your gun code needs it.)

The doc directory in the JAR describes the only three classes you need to know: WaveRunner, WaveRecord, and the Classifier interface. Of course, you'll need to gather or download some data first (see below).

There is one sample data file in the JAR at voidious/test/wavesim/jam.mini.Raiko 0.43_1_wavedata.csv.gzip. Unzip the JAR and try: java -cp ~/robocode/libs/robocode.jar:. voidious.test.wavesim.WaveRunner -- this will simulate a battle vs Raiko using the Head-On Targeting classifier and the KNN classifier from Diamond's main gun. (Diamond's gun is what needs the Robocode JAR.) I use the main(...) method in WaveRunner as a sandbox for setting up whatever tests I want to run.

TripHammer R

These versions of TripHammer will output gzipped, comma delimited data files containing all the waves collected during battles. They are in the format used by WaveRunner. Make sure to up your disk quota in robocode.properties. (robocode.robot.filesystem.quota=2000000000 will give you 2 gigs, enough for a few hundred battles or 150+ seasons of TCRM.)

• TripHammer R1.2.0 - Diamond's movement and main gun.
• TripHammer R1.2.0RM - Random movement (ripped from Jen) and Diamond's main gun.
• TripHammer R1.2.0T - Diamond's main gun in TC mode.

The RM version will execute a lot faster while still giving data from real battle scenarios. Your best bet may be to plug in your own movement, since that's really the data "demographic" you'd want to optimize your gun for. You might also consider updating the bullet power selection to match your own.

Data sets

• dibed20_6seasons.zip (1.1 GB) - 6 seasons of R1.2.0 vs a field of 234 mid-range bots (roughly ranks 100 - 500).

Blog

1/26/2011

Updated everything to 1.2.0. Primary changes:

• Includes traditional MEA and GuessFactors. (Still has precise MEA versions too.)
• Includes orbital Wall Distance, as used by most bots.
• Data files are compressed.
• WaveRunner.BASEDIR is no longer final (doh!), so you can actually use the JAR correctly.
• Option for Random Movement in TripHammer R for faster data collection.

1/25/2011

Been revisiting this, adding stuff that other people would probably need before it's of any use to them, like normal MEA and Wall Distance attributes. Almost ready for a new release. I also plan to add an option (on by default) to use random movement instead of Diamond's wave surfing, for faster data collection, and I'll probably zip up some gigs of sample data for download.

Of course, I ended up getting derailed using it to tune Diamond's gun first. =P

Just cleaned up this page quite a bit too.

3/17/2010

Posted first public version of my WaveSim utility. Have fun. =)

Of course I quickly realized that I overlooked something. The WaveRecord class also needs to be documented. And this uses precise MEA and wall distance based on precise MEA. I should add options for traditional MEA and orbital wall distance.

3/12/2010 pm

Super busy with work, so I can't focus on this much, but I've got the basics working now. For 100 seasons of TCRM:

TripHammer KNN classifier:
Took 1262.0 seconds
Hit percentage: 11.813935415920957
Energy ratio: 0.983354371037666

HOT classifier:
Took 341.0 seconds
Hit percentage: 8.739709205199102
Energy ratio: 0.7279814945288104

HOT score seems pretty high, but it gets 5.9% vs DT, which seems right. At avg distance of 500, you'd get about 5% against a perfectly flat movement.

Best of all is < 1 second per battle for Diamond's main gun. A real TCRM is 15-20 seconds per battle for this gun. Still have some obvious little optimizations to add and there are probably some bigger ones once I focus on it.

3/12/2010 am

I'll post all this soon (except the 4 gigs of data), if anyone's interested in tinkering with it. I may also post info about some of the clustering schemes I've tried. An example of one of the really intensive clustering algorithms that I find interesting is QT clustering. A couple of my previous experiments were inspired by that one.