Talk:RoboResearch

SVN Update Notes

If you update from SVN, be ready to change your .cfg files. The example "run.cfg" is updated - it's still easy. Perhaps easier than they were! --Simonton 04:25, 16 September 2008 (UTC)

HSQLDB No More

Beware of abrupt shutdowns of the database server. I just lost all my scores between Simonton/PFResearch 0073 and 0083, apparently because I closed Eclipse which simply killed the database process, instead of shutting it down nicely. I certainly thought it would handle such things better than that, and I have done it many times before with no consequence, but apparently my timing was bad this time. Does anyone have a good recommendation for a more reliable database that can be run without firing off a separate process, if desired? --Simonton 07:51, 18 September 2008 (UTC)

It looks like the Apache Derby implementation distributed with Java 6 would be the natural choice! Expect that change to come in the near future, for fear of losing more of my data otherwise! --Simonton 15:04, 18 September 2008 (UTC)

It happened again. This time I only lost some data for 0088. I wasn't going to be upset if I lost alot more, but it just confirms this is a switch that has to be made. For now I'm making a copy of the database directory before I kill the database server. Sending the sever the shutdown command via SQL would work, but that's too much effort. They should make it accept a clean shutdown command from the terminal in which you launch the server, that would be handy. --Simonton 02:21, 23 September 2008 (UTC)

Buggy when running from GUI

Hey Simonton if you're around. I found a bit of a odd bug when trying to run the TCRM here. Under the GUI it always fails with:

Thread 1: Unrecognized output from robocode, "Aborting battle, could not find robot: apv.AspidMovement 1.0".  Killing battle.

whereas in the CLI it works just fine. Not sure why all this is but if I find a fix I'll tell let you know. --Rednaxela 07:10, 21 December 2008 (UTC)

Hmm, okay, I discovered that the problem disappaered if I cleared the working_dirs directory, and renamed all the bot jars to match the version number they have in the properties file. It seems that RoboResearch doesn't like jars that are differently named than the default, and some such non-default names are included in the zip files for challenge bots. Anyways I have it working now, and maybe this info posted here might help someone else. --Rednaxela 19:06, 21 December 2008 (UTC)

Ran into this - thanks for the info --Miked0801 02:13, 22 May 2009 (UTC)

Melee

Has anyone used RoboResearch to run melee battles? Looking through the code, it seemed like the part that runs battles would support running a melee battle, but I didn't see any way to configure this (from a .rrc file or otherwise). My melee Test Bed process is currently pathetic and verging on useless... I'd love some RoboResearch support to help me out. --Voidious 15:06, 18 May 2009 (UTC)

I'll take that as a no on anyone using it for Melee? I'm going to try and get it going, then. I'm already like a fish out of water in the Melee arena, a powerful testing tool would help me big time. --Voidious 15:41, 19 May 2009 (UTC)

Sweet, got Melee working (thanks to a lot of the guts already supporting it), though it's not incredibly elegant and the output isn't quite right yet. Good enough to be my new Melee testing utility, though. =) I'll post it or contribute it back to Simonton if/when I clean it up a bit. --Voidious 02:04, 20 May 2009 (UTC)

Got a much more polished version of the Melee support working, doing it in a way that seems mostly elegant in the RoboResearch code. My previous hack was ugly and just didn't always work right. Most of the internals actually already supported Melee, so it wasn't that hard once I figured out the overall structure of the code.

• If you have "Melee" in your challenge name, it uses Melee mode. This probably should be done another way, but this makes sense, works fine, and it's easy to make it something else if we want.
• If it's Melee, all the bots in a category are put in the battle at once. Multiple categories gives you multiple battles per season.
• 10 bots per battle max, a limitation in the existing code / database structure. Also, it uses 1000x1000 for battles with > 2 bots, that was already in there too.
• Scoring is displayed just like it would be in the MeleeRumble: one battle yields 9 separate scores, with the category score being the average of those, and overall score being average of categories.

What I don't have done:

• When the CLI is showing what battle it's running, it just says "<challenger> vs <first reference bot>".
• I haven't worked with the GUI at all because I am using the CLI. The only change I needed to make to the CLI (besides the previous cosmetic change) was for the JAR copying: before, it was copying the first bot from each battle, while obviously I need to copy all bots from each battle.

I'll share this and/or share it back to the SourceForge project shortly. I feel so much less lost in the Melee world being able to run some serious benchmarks. =)

Please do - Infinity and DustBunny would love some number tweaking help to unseat Kawigi's monster Lib. The GUI stuff is my current preference, but beggar's can't be choosers :) --Miked0801 16:26, 28 May 2009 (UTC)

--Voidious 03:12, 28 May 2009 (UTC)

Alright, just made a few tiny cosmetic changes to the CLI and the GUI (the "<your bot> vs <reference bot>" stuff). Viewing results for 45 bots (my current melee test bed) in the GUI isn't pretty, but you can do the "Copy as wiki" into a text editor if you need it more compact. Not that it's that pretty as pure text, either, but at least it's not a 2500 pixel wide window.

Anyway, here it is: void_roboresearch_melee_01.zip. That's the same base stuff you'd get from the SVN checkout plus the binaries compiled for 1.5. You can just overwrite your 'bin' dir with the one from this .zip if you already have RoboResearch setup. (Hopefully Simonton doesn't mind me distributing it like this, but I saw no such mentions in the source and I'm going to get this stuff back to him sometime...) Let me know if you hit any problems - enjoy. =)

--Voidious 01:08, 29 May 2009 (UTC)

Grabbed and in use - thanks! --Miked0801 10:23, 29 May 2009 (UTC)

Getting started instructions

Has anyone here got this to work at all?!? It'd sure help with testing... --Miked0801 23:36, 18 May 2009 (UTC)

You mean like to work for 1v1 / not Melee? Yeah, I've been using it. What kind of problems are you having? I tried long ago and hit issues which I attributed to the Mac JVM. Since I've been Robocoding again and using SoyLatte Java, it's worked fine, but it's also a different version of Robocode, so it could be that too. --Voidious 02:35, 19 May 2009 (UTC)

Mainly, I'm not a java hack. There are no step by step instructions nor a downloadable distribution so I didn't even know where to start. Being able to run batch battles locally would be awesome. --Miked0801 05:04, 19 May 2009 (UTC)

I bumbled through a lot of stuff, myself, but I did get it working. And it's great once it's working. =) I've posted a quick and dirty write-up of the process below. It's for Mac / Unix, while for Windows you'd use ; instead of : for Java classpath stuff. But I also zipped and uploaded my roboresearch dir (minus a few dozen of my dev versions) here, if that helps. Indeed we should update the page with some polished instructions, but I just wanted to get something up before bed here...

Thanks for the files, I bumbled around and actually installed the sun java compiler, then decided to just use your files. Everything seems to run just fine, though the CLI output is ugly. I'll dork around with the GUI next. --Miked0801 01:05, 22 May 2009 (UTC)

And I have fallen in love with the GUI version of this. This will allow me to test locally instead of testing distributed over the internet. Wee! --Miked0801 02:13, 22 May 2009 (UTC)

to install from scratch:
svn checkout https://roboresearch.svn.sourceforge.net/svnroot/roboresearch
cd ./roboresearch/trunk
mkdir bin
javac -cp src:hsqldb.jar -d bin src/roboResearch/CLI.java
install a copy of Robocode to roboresearch/trunk/robocode_install

put your bots in robocode_bots

run battles with something like:
java -cp bin:hsqldb.jar roboResearch.CLI myBot.cfg

for multi-threaded, setup server with database_server.sh, then 
do something like:
java -cp bin:hsqldb.jar roboResearch.CLI -S -t 2 myBot.cfg

docs/getting_started_running.txt has some info, 
but some is outdated (boo)

--Voidious 05:28, 19 May 2009 (UTC)

Oops, I nearly forgot, one more important detail: I did have to actually edit the source to fix one thing. Some Robocode message had its wording changed, and RoboResearch was aborting if it didn't recognize the message. (I should find that and fire it back to Simonton.) So anyway, you should definitely grab my .zip and use that or compile from that source. --Voidious 05:33, 19 May 2009 (UTC)

• Well, or instead of the CLI thing, you can use the newish GUI version. I find it's rather nice, and if you want to do multi-threaded it can manage all the threads and has no need to seperately start the database server etc. I find it rather nice to be able to compare versions directly in the GUI version as the battles run... ;) --Rednaxela 07:21, 19 May 2009 (UTC)

Again, put a distribution or something together with instructions. I'm a C/C++/Asm programmer, not a Java dude - as you'd know if you look at my very C like code :) --Miked0801 16:10, 19 May 2009 (UTC)

I finally put this up on GitHub, I know a lot of us have moved over to RoboRunner/Jogger, but it doesn't quite feel as well polished as RoboResearch. Also this way we can all put our hacks in one place.

RoboResearch

On that note I have made a new build. It has a new and shiny Challenge Creator function now, right in the program. I still don't have all the new features RoboRunner added to the rrc format yet. In fact a spec should probably be decided one at some point. But I digress.

It still has some bugs I have to work out. I know a lot were reported here, but I somewhat lost track of which are fixed and which are not.

Any chance you feel like elaborating on what you like about RoboResearch over RoboRunner? It would take a heck of a lot to convince me to waste 25% of my CPU time on JVM restarts again. From the command line, I honestly feel like RoboRunner is superior in every way. I haven't used the RoboResearch GUI or RoboJogger much though.

On that I agree. RoboRunner is definitely better on the command line. However I tend to live in the GUI world, and I feel it's GUI is quite a bit ahead of RoboJogger.

Though I now consider I could have just made my own front end for RoboRunner. Since that would probably be much easier then trying to hook it into RoboResearch.

Eh... I'm sure plenty of the RoboResearch UI code could be applicable to a RoboRunner frontend.

Just to chime in on the UI comparison. I recently came back to Robocode, and having been using the RoboResearch GUI in the past, I decided to try RoboRunner w/ RoboJogger. While I like RoboRunner's performance, I prefer the RoboResearch GUI over RoboJogger... but I think I'll end up going with RoboRunner CLI now, as I don't normally live in the GUI world anyway.

Sorta off-topic, but some food for thought...

I took what I think is a somewhat novel approach to the GUI for the automation API in BerryBots. As background, you're writing a program that is run by BerryBots, like a bot - as opposed to Robocode, where you're writing a stand-alone program that embeds the Robocode engine. A big reason for this is so you can write automation code in the same language as the bots. But it also lets me do some of the heavy-lifting needed by most automation code, like multi-threading and user inputs.

So the Game Runner API provides the GUI/input functionality. You get a RunnerForm object when your Game Runner executes. You set what inputs you want, you decide when to show the form to the user, and after that you can fetch the values. The engine dynamically creates the appropriate GUI, shows it to the user, blocks until they hit OK/Cancel, and then the Game Runner resumes execution. You can see an example in simpletourney, which lets you run a tournament. I could also present a fully text-driven form, like if/when I write an ncurses UI.

I'd like to also provide the flip-side, so you could easily write a stand-alone program that uses BerryBots. That's actually not far off from the binary in use for the web UI. The script hit by the web UI is a stand-alone program that runs a BerryBots match from the command line and processes the results.

Lastly, it might also be cool to write a Game Runner-ish API for Robocode, maybe on top of RoboRunner. Something like running a Twin Duel tournament couldn't really be specified as a challenge file, but it would benefit from the multi-threading and some of the score details stuff that RoboRunner already has. Note that simpletourney offers the same tourney functionality as Twin Duel/Tourney Runner, but it's less code, everything's configurable at run-time via a GUI, and matches run multi-threaded.

RoboResearch is not starting JVMs in debug mode. It passes "-Ddebug" as parameter, but it should be "-Ddebug=true".

So that's just an argument to the JVM, right? Custom JVM args is near the top of my list for RoboRunner, so I'm working on it now. But there are some things like "-nodisplay" that are arguments to Robocode itself, which wouldn't work because I don't launch Robocode (just use the control API from my own program). So I'd have to add arguments myself if I wanted to let you configure it to actually display battles, for instance.

Nevermind, did some testing. =)

The bug is in RoboResearch, not RoboRunner. Currently I'm using a custom BattleRunner class which passes the correct argument. It's makes a 2% APS difference in some challenges when using all cores.

I'll try RoboRunner someday.

Yeah, I know, it just reminded me of the related issue in RoboRunner. It's funny with RoboResearch, I feel like everyone has their own set of hacks in their local instance. =)

Wow, 2% difference in debug mode? That's crazy.

I'm strongly considering writing up a new Robocode battle runner to replace my RoboResearch usage. I think I could even have something workable over the weekend. Here's my short list of design goals:

• Easy to setup / configure!
• Use Robocode control API instead of running a fresh Robocode engine / JVM from the command line for each battle.
• Support existing .rrc files.
• Support any battle configuration (battle size, Melee, Twin Duel, etc).

Anyone have any thoughts / requests? I think I would go command line only to start, but maybe if I really get something good I'll consider a GUI.

Now that I can crank out 1,000 battles per hour, the thought of also restarting the JVM 1,000 times per hour kind of makes me cringe. =)

You can take for base Distributed_Robocode

I really should take a look at what you have there, I've never tried it. Though I think I'm looking for something a bit simpler and lightweight. Does the client do multiple threads with multiple Robocode engines? Or do you just have to configure one client per thread, like with RoboRumble clients?

I just whipped up the core battle runner, which takes paths to multiple Robocode install directories, then runs a collection of battles across the threads using the control API. Though that's pretty much the easy part... =)

It uses one thread per server.

Yes, but in my case, i have more CPU power at my work, so support of remote servers was critical for me:) Latter today, i will publish soures on github.

Done

Yeah, support for remote clients is a killer feature in that situation!

Great feature list. Support of current .rrc files is a good idea. You'd need some sort of config file to parse anyway, and it's a nice, easy to read config format anyway.

Remote servers would be fun for me too. I've got my main home desktop, and an identical machine running as our living room media center.. so both of them could run battles for me. *evil grin*

I said it over in my talk page, Voidious, but I'd love to help contribute to this with code and testing. As a teacher my free time will contract somewhat during the school year.

My recent experience with RR has me interested in helping to make something a bit easier to get off the ground.

Hmm. It seems that Robocode itself is not thread safe for running multiple instances? I have multiple RobocodeEngine's going in separate threads, each with its own directory, and I'm still hitting some weird concurrency issues. Bummer! I wonder if I can do something with ClassLoaders to deal with it? I've never really played with them before, and not sure if it would kill performance or not.

Oh, yes i also faced with this problem and also think about class loaders but in result came to current architecture

Yep, the engine is not thread safe.

Creating ClassLoaders should be faster than creating separate JVMs, but don't know by how much.

Another approach would be staying with separate JVMs, but not restarting them before each battle. Exchanging messages between JVMs and using the API inside a loop instead, much like how the RoboRumble client works.

Creating ClassLoaders once and reusing them on many battles is also another approach. I remember trying this approach in a custom multi-threaded RoboRumble client a long time ago. Didn't remember exactly what went wrong. Maybe the need of a custom JAR loading mechanism.

Yeah, having each engine in its own process and doing IPC seemed like a huge ordeal that I wanted to avoid, but now I'm thinking it might not be so bad, and it would be easier and safer than fussing with classloaders. I brushed up on some RMI stuff last night, but even that requires some setup to get the RMI registry going and stuff, which I don't really want to force on people. I think just starting the separate processes with Runtime.exec and communicating with them over STDIN and STDOUT may be the simplest solution all around.

Cool, that wasn't so hard. =) I have this pretty much working now, including reading .rrc files, with option additional stuff like multiple bots per line for Melee battles and custom battle field sizes, and simple/fast/human readable results output to .properties files instead of a database. Still a few things left to do for usability, which is a high priority for me, but I think I can have something posted soon.

~29 mins for one season of a 500-bot tested, vs ~40 mins with RoboResearch. Yum. =)

Niice!

I've almost got deBroglie's score on the testbed you made for me back to the level of rev108 after my hitrate normalization fixes. It's been a great boon to have a working batch battle runner!

Looking forward to testing this work of yours out!

Sounds awesome. When there get to be a lot of robots in the install directory RoboResearch gets really slow starting each battle... try running a single season against the entire rumble and you'll see what I'm talking about.

I also see big slowdowns once the database gets too big, say 15k battles, which is a pretty frequent occurrence for me these days. It's pretty obvious when the first 6 battles fire right up at startup vs trickling in with half a second between them.

I'm not sure I'll have time to get it polished and posted tonight, but if not, definitely within the next day or two. And I'll post it on GitHub as public domain. It's not even that much code, if anyone wants to learn it to fork it or add a GUI or whatever.

There will still be a bunch of bells and whistles left on the to-do list as of the first release, but for general APS bot benchmarking it's in great shape. I'm using it for my own battles now and it ran overnight with no issues. I think I have the setup process pretty smoothed out, too (albeit Unix only, using a bash script).

Voidious, i like yours solution and think, that 3 tools for batch battles execution is too much. But i still need in execution of battles on remote machines, so i think it will be cool to combine some way of our solutions. Rigth now i have no inspiration for work on it, but if you make good interface for yours tool, i can add my client-server solution in future. What do you think? We can contact via email or skype to discuss interface.

Well, since my new tool is more of a replacement for RoboResearch, that only leaves us with 2, and I don't think 2 is too many. =) If you're only using one machine, then a fast, powerful tool that's easy to use for multi-threaded benchmarks is great. If you have multiple machines, no matter what you do it's going to require some setup, so having a tool for that is great too. My multi-process BattleRunner is pretty simple, though, so it shouldn't be hard to integrate with that if we want to use that in the client part of your system.

Problem in my solution is that you need run several servers on same machine to utilize cpu completly and i wish to run only one server, which will run several engines. Also even 2 is too much. It means that every feature (ui, hit rate colculation, melee support etc.) must be doubled and i think, that better for community to concentrate all resources on one tool. I will review your tool, when you publish it and contact you, if detect troubles in integration or impruvement ideas.

Is Chase leading the support for RoboResearch development now? Just let me know if you would like extra help. I have plenty of experience in Java and Swing.

I might end up redoing a bit of the GUI when I have the chance. But I don't even have time to really work on my robots at the moment, let alone on RoboResearch. :)

On that note I also want to switch it over to use SQLite instead of what it currently uses.

Doing both of these is not hugely difficult, the main problem will be transitioning the current robocode control and challenge definitions over, etc.

With my latest versions of XanderCat 9.0 and 9.0.1, they seem to perform just fine with 1.7.3.0, but I am now getting really wacky scores in RoboResearch. There are no exceptions happening, but the scores are coming up a good 20% to 30% lower than the scores with 1.7.3.0 (e.g. score against Rapture.Rapture is coming up at about 65% in RoboResearch, but a consistent 90% in 1.7.3.0 test runs and in the RoboRumble). I checked to ensure my robot is not throwing any exceptions. Any thoughts on what might be causing this? It is preventing me from using RoboResearch for testing.

It might be due to me setting it to debug mode (it keeps robots from skipping turns, so if enemies were skipping, they are not in RR).

I don't think so. I might have to spend some time digging into the code more. A lot of the docs and pages discussing RoboResearch seem to be outdated. For example, I keep seeing references to having a copy of Robocode installed in the robocode_install directory, but from what I can see, there doesn't seem to be any robocode_install directory. I assumed it just used robocode.jar from the lib directory. Quite confusing. So I don't even know what version of Robocode RoboResearch is using.

AFAIK you're supposed to install a copy of Robocode into robocode_install. But the original RoboResearch also came with a lib/robocode.jar that it used, which I only just recently realized it uses instead of the one in robocode_install/libs. I'm not sure what the problem is, but I'd install Robocode 1.7.3.x to robocode_install, start it up and recalculate the CPU constant, and copy that robocode.jar into roboresearch/lib. See if you still see a score discrepancy.

RoboResearch docs could definitely use a lot of work, but hopefully with Chase taking charge to start updating it again recently, we'll resolve some of the usability issues and keep it better maintained.

I dug into the code a little to figure it out. I didn't realize that RoboResearch just runs Robocode normally using "-nodisplay". That made it easy to debug, because I could just comment out the line that added "-nodisplay" and watch the battle as it ran. The version of Robocode in the lib directory that it is using is 1.6.1.2. For whatever reason, it appears as though the onRoundEnded() event never gets called, which is messing up some of the round data I keep track of. Ultimately, this lead to one of my multi-mode scenarios getting incorrectly activated much more often than it should have been, resulting in low scores. Now I need to either develop a work-around or try to set up RoboResearch to use 1.7.3.0.

Ah, cool. I wondered if it might be some robot method added in 1.7.x, which I think onRoundEnded is. You should be able to copy robocode.jar from robocode/libs to roboresearch/lib to get whatever Robocode version you want.

I modified my RoboResearch to use robocode_install/libs/robocode.jar, so whichever Robocode version I install there is used. That and the other patches I'm using are available here, and described/linked a bit further up this talk page. Recompile with javac -cp src:hsqldb.jar -d bin src/roboResearch/CLI.java (or GUI.java).

Oh yeah, there may be some other minor incompatibilities using 1.7.x. I fixed some issues with the command line version of RoboResearch in my patch, and Chase may have fixed some with the GUI in his version (if there were any).