Difference between revisions of "WaveSim"

From Robowiki
Jump to navigation Jump to search
(misc updates, add "blog")
(→‎Downloads: update links for JAR and data collecting bots, remove broken links to data sets.)
 
(17 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
{{tocright}}
 
== Background ==
 
== Background ==
  
To facilitate a new wave of gun research, I'm setting up a system that will let me easily test a gun's classification against pre-gathered wave data. This would take the place of running a zillion Robocode battles.
+
To facilitate a new wave of gun research, I created a system to benchmark a gun's accuracy using pre-gathered battle data. This can take the place of running a zillion Robocode battles.  
  
Benefits:
+
Basically, we just record all the stuff guns use to learn and to aim - input attributes like velocity, wall distance, acceleration, and outputs like GuessFactor and firing angle - then run that stuff directly through our targeting algorithms. Testing classification algorithms against raw data sets is nothing new or special, but it's about time someone set up a solid system for doing it with Robocode!
* By removing all the overhead of running actual battles, I think this will allow a given gun test to run an order of magnitude faster, a huge boon to the pursuit of wacky targeting techniques.
 
* It will make it more feasible to experiment with sophisticated algorithms that are an order of magnitude slower than what we usually use, as some clustering algorithms are.
 
* I could record data from actual battles instead of being limited to TCs, since every classification run would be against the same exact data, anyway.
 
* I can see hit percentages (normalized or not) or energy ratios (spent vs gained vs damage), which I think are a more direct measurement of a gun's accuracy than TC results.
 
  
Testing classification algorithms against raw data sets is nothing new, but it's high time I actually tried it for Robocode gun research. =) Of course, this will only work against non-adaptive movements. (Some of this process is similar to my old [[User:Voidious/Segmentation Research|Segmentation Research]].)
+
=== Benefits ===
 +
 
 +
* By removing all the overhead of running actual battles, you can run a gun benchmark much faster -- like minutes instead of hours.
 +
* Tests can be run against data from real battles instead of a [[Targeting Challenge|TC]] scenario, which should be more relevant to real battle performance.
 +
* Every benchmark runs against the exact same data, resulting in direct comparisons of test results without worrying about randomness.
 +
* You can see hit percentages or energy ratios (spent vs gained vs damage), which I think are a more direct measurement of a gun's accuracy than TC results.
 +
* The huge speed increase and the ease of reconfiguring guns between tests really lends itself to more complex development options, like genetic algorithms and other long-running parameter tweaking.
 +
 
 +
=== Disclaimer ===
 +
 
 +
Note that a good classification algorithm is not the only important part of a gun. But many top bots have the rest of their gun code polished to a point where classification is the thing we work on most of the time.
  
 
== Components ==
 
== Components ==
 
=== Data-gathering bot ===
 
=== Data-gathering bot ===
  
First step is collecting some data. I modified my [[TripHammer]] code to output every wave it gathers in a .csv type format. 100 seasons of [[Targeting Challenge RM|TCRM]] is about 4 gigs.
+
A modified [[TripHammer]] outputs data for every tick and gun [[Waves|wave]] it gathers to a gzipped binary file. Each battle gets a new filename and TripHammer writes to it throughout the battle.
  
This part's done, though some partial write issues need to be worked out. (Right now just ignoring lines that aren't written correctly and presuming a little duplicated data will not harm results much.)
+
You could, of course, modify your own bot to output wave data to a file of the same or similar format. Just take a look at the <code>TickRecord</code> class in WaveSim for guidance on the format, or look at/borrow code from TripHammer's <code>TripWave.writeTickRecord()</code> method.
  
 
=== Battle simulation ===
 
=== Battle simulation ===
  
I have utility classes to load wave data, step through it as a real battle would, feed waves to a classifier for learning, classify each firing wave / check for hits, and record overall hit rates, energy, and damage. This is basically simulating a battle from the perspective of a wave-based gun.
+
WaveSim records the raw attribute data for each tick and fires a wave for that tick. When the wave is collected, it records another type of record with the hitting angle from the wave. By reading these in the same order that they occurred, a targeting algorithm will learn from ticks and/or waves and be asked to choose firing angles in the same fashion it would be in a real Robocode battle.
  
This is done, though it could use some bells and whistles.
+
=== Classifiers ===
  
=== Classifiers ===
+
The <code>Classifier</code> interfaces implement a targeting algorithm. It's pretty simple. The main methods shared by all classifiers are:
 +
 
 +
* <code>initalize()</code> - Set up data structures, clear any existing data.
 +
* <code>classify(TickRecord)</code> - Attempt to classify (produce a firing angle for) this wave.
 +
 
 +
There are three Classifier subinterfaces - one of these is what you'll probably be working with:
 +
* <code>PreciseWaveClassifier</code> - Uses [[Waves/Precise Intersection|precise intersection]] and processes waves once they have completely passed the enemy bot. Includes the method: <code>feed(TickRecord, PreciseWaveEndRecord)</code>
 +
* <code>TraditionalWaveClassifier</code> - Records only the bearing to the center of the enemy bot once a wave has passed the center of the bot. Includes the method: <code>feed(TickRecord, TraditionalWaveEndRecord)</code>
 +
* <code>TickClassifier</code> - Fed each tick as it happens and ignores waves. This is for non-Wave-based targeting algorithms like [[Linear Targeting]] or [[Pattern Matching]], and may also be useful for [[Play It Forward]] firing angle projections. Includes the method: <code>feed(TickRecord)</code>
 +
 
 +
== Downloads ==
 +
 
 +
=== WaveSim ===
 +
 
 +
* '''[https://www.dropbox.com/s/y7rg4erazbfzooa/wavesim_2.1.0.jar?dl=1 WaveSim 2.1.0]''' - Source, .class files, Javadoc, and one sample data file.
 +
 
 +
==== Basic usage ====
 +
If you put this on your build path, you should be able to:
 +
* Set <code>WaveRunner.BASEDIR</code> to point to your data files directory.
 +
* Implement the <code>PreciseWaveClassifier</code>, <code>TraditionalWaveClassifier</code>, and/or <code>TickClassifier</code> interface to create your targeting algorithm. Methods for initialization/reset, training a tick or [[Waves|wave]], and classifying a wave. (See the Javadoc for reference.)
 +
* Start simulating battles!
 +
* You may need robocode.jar on your build path if any of your gun code needs it.
 +
 
 +
==== In the JAR ====
 +
The doc directory in the JAR describes many of the classes, but the main ones you need to know are <code>WaveRunner</code>, <code>TickRecord</code>, and the <code>Classifier</code> interfaces. Of course, you'll need to gather or download some data first (see below).
 +
 
 +
There is one sample data file in the JAR at <code>voidious/wavesim/cx.mini.Cigaret 1.31_1_wavedata.bin.gz</code>. Unzip the JAR and try: <code>java -cp ~/robocode/libs/robocode.jar:. voidious.wavesim.WaveRunner</code> -- this will simulate a battle vs [[Cigaret]] using [[Head-On Targeting]], [[Linear Targeting]], [[Circular Targeting]], and [[Dynamic Clustering|KNN]] classifier from Diamond's main gun. (Diamond's gun is what needs the Robocode JAR.)
 +
 
 +
=== TripHammer R ===
 +
 
 +
These versions of TripHammer will output gzipped binary data files containing all the wave data collected during battles. They are in the format used by <code>WaveRunner</code>. Make sure to up your disk quota in robocode.properties. (<code>robocode.robot.filesystem.quota=2000000000</code> will give you 2 gigs, enough for a couple thousand battles or so.)
 +
* '''[https://www.dropbox.com/s/lwhgp6r4d6t97v6/voidious.TripHammer_R2.1.0.jar?dl=1 TripHammer R2.1.0]''' - Diamond's movement and main gun.
 +
* '''[https://www.dropbox.com/s/2mvwwceiln6uxpo/voidious.TripHammer_R2.1.0RM.jar?dl=1 TripHammer R2.1.0RM]''' - Random movement (ripped from [[Jen]]) and Diamond's main gun.
 +
* '''[https://www.dropbox.com/s/mfqsx10wki5oqo0/voidious.TripHammer_R2.1.0T.jar?dl=1 TripHammer R2.1.0T]''' - Diamond's main gun in TC mode.
 +
 
 +
The RM version will execute a lot faster while still giving data from real battle scenarios. Your best bet may be to plug in your own movement, since that's really the data "demographic" you'd want to optimize your gun for. You might also consider updating the bullet power selection to match your own.
 +
 
 +
=== Data sets (offline) ===
  
I have a simple interface implemented by each classifier - initialize, feed it a wave, classify a wave. I've decided not to muddy up my main code base by making my guns work interchangeably with my classifier test code, but porting between them is pretty simple. Ported Diamond's main gun in a few minutes.
+
* '''dibed25_2.1.0.zip''' (2.1 GB) - 4 seasons of R2.1.0 vs a field of 493 bots across the whole range of the RoboRumble General 1v1 population.
 +
* '''dibed41 (25 seasons)''': 25 seasons vs 400 bots from the top half of RoboRumble General 1v1, split into 5-season chunks (2.3 GB each).
  
 
== Blog ==
 
== Blog ==
 +
 +
==== 6/23/2011 ====
 +
 +
Updated to 2.1.0. Now supports non-Wave targeting algorithms!
 +
 +
* Added TickClassifier, which feeds attribute data for each tick as it happens, instead of only after a wave breaks. This allows for better implementation of non-Wave-based targeting algorithms, like [[Circular Targeting]] and [[Pattern Matching]].
 +
* Added absolute heading attribute.
 +
* Now includes [[Linear Targeting]] and [[Circular Targeting]] classifiers for reference. These are good examples of non-Wave classifiers.
 +
* Fixed a bug with gun heat. Gun heat and virtuality are for the tick after the rest of the attributes - ie, they are zero when a real bullet could be fired based on this information.
 +
* Removed "refeed" and "feedOnce" methods from Classifier interface.
 +
* Lots of cleanup of code and Javadoc.
 +
 +
==== 4/13/2011 ====
 +
 +
Updated everything to 2.0.0. Big thanks to [[User:Rednaxela|Rednaxela]] for making a huge code contribution in this version! Primary changes:
 +
 +
* Multithreading! Make full use of those fancy multicore processors. Also loads data from disk in a separate thread.
 +
* New file format that outputs binary files and lays the foundation for supporting non-wave gun stuff, like [[Pattern Matching]] and [[Play It Forward]]. Should be recording all the data needed for those right now, but will need to update the APIs a bit more to make it work nicely.
 +
* Support for [[Waves/Precise Intersection|precise intersection]].
 +
* A fair bit of code restructuring and added/updated documentation.
 +
 +
==== 1/26/2011 ====
 +
 +
Updated everything to 1.2.0. Primary changes:
 +
 +
* Includes traditional [[Maximum Escape Angle|MEA]] and [[GuessFactors]]. (Still has precise MEA versions too.)
 +
* Includes orbital [[Wall Distance]], as used by most bots.
 +
* Data files are compressed.
 +
* WaveRunner.BASEDIR is no longer final (doh!), so you can actually use the JAR correctly.
 +
* Option for [[Random Movement]] in TripHammer R for faster data collection.
 +
 +
==== 1/25/2011 ====
 +
 +
Been revisiting this, adding stuff that other people would probably need before it's of any use to them, like normal [[Maximum Escape Angle|MEA]] and [[Wall Distance]] attributes. Almost ready for a new release. I also plan to add an option (on by default) to use random movement instead of Diamond's wave surfing, for faster data collection, and I'll probably zip up some gigs of sample data for download.
 +
 +
Of course, I ended up getting derailed using it to tune [[Diamond]]'s gun first. =P
 +
 +
Just cleaned up this page quite a bit too.
 +
 +
==== 3/17/2010 ====
 +
 +
Posted first public version of my WaveSim utility. Have fun. =)
 +
 +
Of course I quickly realized that I overlooked something. The WaveRecord class also needs to be documented. And this uses [[Maximum Escape Angle/Precise|precise MEA]] and wall distance based on precise MEA. I should add options for traditional MEA and orbital wall distance.
  
 
==== 3/12/2010 pm ====
 
==== 3/12/2010 pm ====
Line 53: Line 142:
 
I'll post all this soon (except the 4 gigs of data), if anyone's interested in tinkering with it. I may also post info about some of the clustering schemes I've tried. An example of one of the really intensive clustering algorithms that I find interesting is [http://en.wikipedia.org/wiki/Cluster_analysis#QT_clustering_algorithm QT clustering]. A couple of my previous experiments were inspired by that one.
 
I'll post all this soon (except the 4 gigs of data), if anyone's interested in tinkering with it. I may also post info about some of the clustering schemes I've tried. An example of one of the really intensive clustering algorithms that I find interesting is [http://en.wikipedia.org/wiki/Cluster_analysis#QT_clustering_algorithm QT clustering]. A couple of my previous experiments were inspired by that one.
  
[[Category:Utilities]]
+
[[Category:Utilities|WaveSim]]
 +
[[Category:Targeting|WaveSim]]

Latest revision as of 18:28, 4 February 2017

Background

To facilitate a new wave of gun research, I created a system to benchmark a gun's accuracy using pre-gathered battle data. This can take the place of running a zillion Robocode battles.

Basically, we just record all the stuff guns use to learn and to aim - input attributes like velocity, wall distance, acceleration, and outputs like GuessFactor and firing angle - then run that stuff directly through our targeting algorithms. Testing classification algorithms against raw data sets is nothing new or special, but it's about time someone set up a solid system for doing it with Robocode!

Benefits

  • By removing all the overhead of running actual battles, you can run a gun benchmark much faster -- like minutes instead of hours.
  • Tests can be run against data from real battles instead of a TC scenario, which should be more relevant to real battle performance.
  • Every benchmark runs against the exact same data, resulting in direct comparisons of test results without worrying about randomness.
  • You can see hit percentages or energy ratios (spent vs gained vs damage), which I think are a more direct measurement of a gun's accuracy than TC results.
  • The huge speed increase and the ease of reconfiguring guns between tests really lends itself to more complex development options, like genetic algorithms and other long-running parameter tweaking.

Disclaimer

Note that a good classification algorithm is not the only important part of a gun. But many top bots have the rest of their gun code polished to a point where classification is the thing we work on most of the time.

Components

Data-gathering bot

A modified TripHammer outputs data for every tick and gun wave it gathers to a gzipped binary file. Each battle gets a new filename and TripHammer writes to it throughout the battle.

You could, of course, modify your own bot to output wave data to a file of the same or similar format. Just take a look at the TickRecord class in WaveSim for guidance on the format, or look at/borrow code from TripHammer's TripWave.writeTickRecord() method.

Battle simulation

WaveSim records the raw attribute data for each tick and fires a wave for that tick. When the wave is collected, it records another type of record with the hitting angle from the wave. By reading these in the same order that they occurred, a targeting algorithm will learn from ticks and/or waves and be asked to choose firing angles in the same fashion it would be in a real Robocode battle.

Classifiers

The Classifier interfaces implement a targeting algorithm. It's pretty simple. The main methods shared by all classifiers are:

  • initalize() - Set up data structures, clear any existing data.
  • classify(TickRecord) - Attempt to classify (produce a firing angle for) this wave.

There are three Classifier subinterfaces - one of these is what you'll probably be working with:

  • PreciseWaveClassifier - Uses precise intersection and processes waves once they have completely passed the enemy bot. Includes the method: feed(TickRecord, PreciseWaveEndRecord)
  • TraditionalWaveClassifier - Records only the bearing to the center of the enemy bot once a wave has passed the center of the bot. Includes the method: feed(TickRecord, TraditionalWaveEndRecord)
  • TickClassifier - Fed each tick as it happens and ignores waves. This is for non-Wave-based targeting algorithms like Linear Targeting or Pattern Matching, and may also be useful for Play It Forward firing angle projections. Includes the method: feed(TickRecord)

Downloads

WaveSim

  • WaveSim 2.1.0 - Source, .class files, Javadoc, and one sample data file.

Basic usage

If you put this on your build path, you should be able to:

  • Set WaveRunner.BASEDIR to point to your data files directory.
  • Implement the PreciseWaveClassifier, TraditionalWaveClassifier, and/or TickClassifier interface to create your targeting algorithm. Methods for initialization/reset, training a tick or wave, and classifying a wave. (See the Javadoc for reference.)
  • Start simulating battles!
  • You may need robocode.jar on your build path if any of your gun code needs it.

In the JAR

The doc directory in the JAR describes many of the classes, but the main ones you need to know are WaveRunner, TickRecord, and the Classifier interfaces. Of course, you'll need to gather or download some data first (see below).

There is one sample data file in the JAR at voidious/wavesim/cx.mini.Cigaret 1.31_1_wavedata.bin.gz. Unzip the JAR and try: java -cp ~/robocode/libs/robocode.jar:. voidious.wavesim.WaveRunner -- this will simulate a battle vs Cigaret using Head-On Targeting, Linear Targeting, Circular Targeting, and KNN classifier from Diamond's main gun. (Diamond's gun is what needs the Robocode JAR.)

TripHammer R

These versions of TripHammer will output gzipped binary data files containing all the wave data collected during battles. They are in the format used by WaveRunner. Make sure to up your disk quota in robocode.properties. (robocode.robot.filesystem.quota=2000000000 will give you 2 gigs, enough for a couple thousand battles or so.)

The RM version will execute a lot faster while still giving data from real battle scenarios. Your best bet may be to plug in your own movement, since that's really the data "demographic" you'd want to optimize your gun for. You might also consider updating the bullet power selection to match your own.

Data sets (offline)

  • dibed25_2.1.0.zip (2.1 GB) - 4 seasons of R2.1.0 vs a field of 493 bots across the whole range of the RoboRumble General 1v1 population.
  • dibed41 (25 seasons): 25 seasons vs 400 bots from the top half of RoboRumble General 1v1, split into 5-season chunks (2.3 GB each).

Blog

6/23/2011

Updated to 2.1.0. Now supports non-Wave targeting algorithms!

  • Added TickClassifier, which feeds attribute data for each tick as it happens, instead of only after a wave breaks. This allows for better implementation of non-Wave-based targeting algorithms, like Circular Targeting and Pattern Matching.
  • Added absolute heading attribute.
  • Now includes Linear Targeting and Circular Targeting classifiers for reference. These are good examples of non-Wave classifiers.
  • Fixed a bug with gun heat. Gun heat and virtuality are for the tick after the rest of the attributes - ie, they are zero when a real bullet could be fired based on this information.
  • Removed "refeed" and "feedOnce" methods from Classifier interface.
  • Lots of cleanup of code and Javadoc.

4/13/2011

Updated everything to 2.0.0. Big thanks to Rednaxela for making a huge code contribution in this version! Primary changes:

  • Multithreading! Make full use of those fancy multicore processors. Also loads data from disk in a separate thread.
  • New file format that outputs binary files and lays the foundation for supporting non-wave gun stuff, like Pattern Matching and Play It Forward. Should be recording all the data needed for those right now, but will need to update the APIs a bit more to make it work nicely.
  • Support for precise intersection.
  • A fair bit of code restructuring and added/updated documentation.

1/26/2011

Updated everything to 1.2.0. Primary changes:

  • Includes traditional MEA and GuessFactors. (Still has precise MEA versions too.)
  • Includes orbital Wall Distance, as used by most bots.
  • Data files are compressed.
  • WaveRunner.BASEDIR is no longer final (doh!), so you can actually use the JAR correctly.
  • Option for Random Movement in TripHammer R for faster data collection.

1/25/2011

Been revisiting this, adding stuff that other people would probably need before it's of any use to them, like normal MEA and Wall Distance attributes. Almost ready for a new release. I also plan to add an option (on by default) to use random movement instead of Diamond's wave surfing, for faster data collection, and I'll probably zip up some gigs of sample data for download.

Of course, I ended up getting derailed using it to tune Diamond's gun first. =P

Just cleaned up this page quite a bit too.

3/17/2010

Posted first public version of my WaveSim utility. Have fun. =)

Of course I quickly realized that I overlooked something. The WaveRecord class also needs to be documented. And this uses precise MEA and wall distance based on precise MEA. I should add options for traditional MEA and orbital wall distance.

3/12/2010 pm

Super busy with work, so I can't focus on this much, but I've got the basics working now. For 100 seasons of TCRM:

TripHammer KNN classifier:
Took 1262.0 seconds
Hit percentage: 11.813935415920957
Energy ratio: 0.983354371037666

HOT classifier:
Took 341.0 seconds
Hit percentage: 8.739709205199102
Energy ratio: 0.7279814945288104

HOT score seems pretty high, but it gets 5.9% vs DT, which seems right. At avg distance of 500, you'd get about 5% against a perfectly flat movement.

Best of all is < 1 second per battle for Diamond's main gun. A real TCRM is 15-20 seconds per battle for this gun. Still have some obvious little optimizations to add and there are probably some bigger ones once I focus on it.

3/12/2010 am

I'll post all this soon (except the 4 gigs of data), if anyone's interested in tinkering with it. I may also post info about some of the clustering schemes I've tried. An example of one of the really intensive clustering algorithms that I find interesting is QT clustering. A couple of my previous experiments were inspired by that one.