new Robocode batch battle runner

Hmm. It seems that Robocode itself is not thread safe for running multiple instances? I have multiple RobocodeEngine's going in separate threads, each with its own directory, and I'm still hitting some weird concurrency issues. Bummer! I wonder if I can do something with ClassLoaders to deal with it? I've never really played with them before, and not sure if it would kill performance or not.

Oh, yes i also faced with this problem and also think about class loaders but in result came to current architecture

Yep, the engine is not thread safe.

Creating ClassLoaders should be faster than creating separate JVMs, but don't know by how much.

Another approach would be staying with separate JVMs, but not restarting them before each battle. Exchanging messages between JVMs and using the API inside a loop instead, much like how the RoboRumble client works.

Creating ClassLoaders once and reusing them on many battles is also another approach. I remember trying this approach in a custom multi-threaded RoboRumble client a long time ago. Didn't remember exactly what went wrong. Maybe the need of a custom JAR loading mechanism.

Yeah, having each engine in its own process and doing IPC seemed like a huge ordeal that I wanted to avoid, but now I'm thinking it might not be so bad, and it would be easier and safer than fussing with classloaders. I brushed up on some RMI stuff last night, but even that requires some setup to get the RMI registry going and stuff, which I don't really want to force on people. I think just starting the separate processes with Runtime.exec and communicating with them over STDIN and STDOUT may be the simplest solution all around.

~29 mins for one season of a 500-bot tested, vs ~40 mins with RoboResearch. Yum. =)

Niice!

I've almost got deBroglie's score on the testbed you made for me back to the level of rev108 after my hitrate normalization fixes. It's been a great boon to have a working batch battle runner!

Looking forward to testing this work of yours out!

Sounds awesome. When there get to be a lot of robots in the install directory RoboResearch gets really slow starting each battle... try running a single season against the entire rumble and you'll see what I'm talking about.

I also see big slowdowns once the database gets too big, say 15k battles, which is a pretty frequent occurrence for me these days. It's pretty obvious when the first 6 battles fire right up at startup vs trickling in with half a second between them.

I'm not sure I'll have time to get it polished and posted tonight, but if not, definitely within the next day or two. And I'll post it on GitHub as public domain. It's not even that much code, if anyone wants to learn it to fork it or add a GUI or whatever.

There will still be a bunch of bells and whistles left on the to-do list as of the first release, but for general APS bot benchmarking it's in great shape. I'm using it for my own battles now and it ran overnight with no issues. I think I have the setup process pretty smoothed out, too (albeit Unix only, using a bash script).

Voidious, i like yours solution and think, that 3 tools for batch battles execution is too much. But i still need in execution of battles on remote machines, so i think it will be cool to combine some way of our solutions. Rigth now i have no inspiration for work on it, but if you make good interface for yours tool, i can add my client-server solution in future. What do you think? We can contact via email or skype to discuss interface.

Well, since my new tool is more of a replacement for RoboResearch, that only leaves us with 2, and I don't think 2 is too many. =) If you're only using one machine, then a fast, powerful tool that's easy to use for multi-threaded benchmarks is great. If you have multiple machines, no matter what you do it's going to require some setup, so having a tool for that is great too. My multi-process BattleRunner is pretty simple, though, so it shouldn't be hard to integrate with that if we want to use that in the client part of your system.

Problem in my solution is that you need run several servers on same machine to utilize cpu completly and i wish to run only one server, which will run several engines. Also even 2 is too much. It means that every feature (ui, hit rate colculation, melee support etc.) must be doubled and i think, that better for community to concentrate all resources on one tool. I will review your tool, when you publish it and contact you, if detect troubles in integration or impruvement ideas.