BedMaker
The highlighted comment was created in this revision.
Glad to see someone else enthusiastic about BedMaker. =) I also dropped Darkcanuck an email a few days ago. I'm not sure what's up - he's usually pretty responsive. Maybe a summer vacation?
Anyway, while it's not a good permanent solution, let me know if you want me to run some configurations for you and post the RoboRumble challenge files, to get you started. For instance, my current test bed is like: "400 bots that Diamond scores 60-92 against" (~top half of the rumble, bedmaker.pl -n 400 -r roborumble -ref voidious.Diamond -min 60 -max 92), and I think my next one is going to be "500 bots that Diamond scores 55 - 99 against" (ie, large random sampling of most of the rumble, excluding DrussGT and bots I score almost perfectly against).
Thanks!
I'm sure it's vacation and summer stuff. No biggie. I'm really looking forward to making some test beds so that I can try out some fixes and tweaks, give them an evening/night run on my computer and see how they fare without bothering the Rumble with them.
It's been confounding to me how often I'll make a change, test and see improvement against 8/10 bots, and see that alteration tank in the Rumble! Not to mention the time taken in manual testing. Much more fun to just code, and see the results in the morning locally.
I've been trying to get up near CassiusClay, YersiniaPestis et. al. in Rumble score before adding on more advanced stuff like Bullet Shadows and Gun Heat Waves (both of which I've got my code in a state to pretty easily tack on, thankfully!) It's been a heck of a struggle though, and I'm fighting for every 0.1 APS it seems. Good ol' law of diminishing returns.
If you wouldn't mind generating a challenge file for me, I'd be appreciative! How about....
"150 bots that deBroglie rev0108 scores 57-95 against"
That should get me a decent size test bed of bots that I'd like to see myself do much better against. There hasn't been too much churn version-to-version above the 95 percent line.. and there's been lots of churn below ~57, most of which will take more advanced work to really beat anyway.
Feel free to tell me if you think that's a poor choice for an 'Am I helping or hurting this bot with this change' type of test bed, or if it should be bigger or smaller.
Sure, here you go - I made a second with the same parameters in case you hit errors with any of the bots, or they're super slow, and you want to swap some out by hand: [1] [2] One nice thing bedmaker does is copy just those bots into roboresearch/robocode_bots, so I guess you'll have to just copy your whole rumble/robots dir in there unless you want to write a script or something. =)
That seems like a good cross-section to test general rumble performance. I hate wasting CPU cycles re-running battles with almost no variance, like bots I score 99+ against, but at the same time you have to be careful not to specialize too much on the bots you have lots of room for improvement against. I certainly haven't mastered the art of choosing test beds, but big test beds across a large section of the rumble seem pretty safe. They'll at least show you pretty quickly when you've really screwed up. =)
Lately I've been running ~2000 battles, which seems accurate to within 0.05 or 0.1. That's made easier now that I can do that in 2-2.5 hours on my new machine, though - previously it was taking me 2+ hours for one season of a 250-bot test bed!
So you'll have to feel it out for yourself as far as your bot and what changes you're making (and how patient you are =)). Like with some changes, I just want to know it's not totally screwing things up before testing it in the rumble. While tweaking parameters little by little, I have to run a lot more battles to know what's going on.
Excellent! After culling a few bots that were throwing BufferedInputStream errors (seems like it was bots that like to output to the external Robocode terminal instead of their own System.out) It's cranking away.
Is there a convenient way to view RoboResearch output for test beds this big other than copying to wiki? That results window is awfully crowded. :)
Thanks so much for the help, Voidious!
No problem! I don't know, my test beds are usually either so big that I only check the overall score, or so small (like my 5 worst matchups) that I can just remember the order. =) RoboResearch also has a GUI which probably has nicer output, but IIRC it puts all the scores on one line, which might be a few thousand pixels wide with 150 bots.
Oh, "results window", maybe you are using the GUI? I use the command line ("CLI"), which I like, but it seems most people do prefer the GUI.
Okay. Hacked together a "cli.sh" using "gui.sh" as a template.
Problem I have now is how do I pass the challenger's name to the command line? That space between the bot name and the bot version number keeps getting me...
./cli.sh -S -C challenges/debroglie1.rrc -c tjk.deBroglie rev0108 -r 35 -s 10
fouls up on the bot name. Using the jar file doesn't seem to work either.
EDIT: Nevermind. Got it. ;)
You should be able to just put it in quotes. I have a di.sh with a commands like: java -cp bin:hsqldb.jar roboResearch.CLI -S -t 6 -c "voidious.Diamond 1.7.56.x1" di1.cfg
Quotes didn't work because quotes get a bit mangled when passing from one bash script to another. I just went ahead and called java directly. Now I have a terminal window running the RoboResearch server, and three windows each chugging away on bots. (rev108 and two changes to surf wave timing and MEA predict timing. Probably veeeery small changes there. Oh well.)
It's going though. Woot!
Is it bad to run two of these simultaneously out of the same directory. The CLI is needing lots of babysitting because of stuff like
Thead 1: Unrecognized output from robocode, "Got an error with alex.Diabolo5: java.lang.ClassNotFoundException: alex.Diabolo5". Killing battle.
followed with lots of java.io.IOException stuff.
I'm not sure, I always run one challenge at a time with the max threads I'm willing to use ("-t 6" for 6 threads). That error does sound like they might be stepping on each other's toes.
Oddly enough, using the -t tag for more threads causes problems for me...
(Running using server)
Result for robot named "tjk.deBroglie rev0115 (2)".
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1949)
at roboResearch.engine.Result.<init>(Result.java:31)
at roboResearch.engine.BattleResults.load(BattleResults.java:42)
at roboResearch.engine.BattleResults.load(BattleResults.java:24)
at roboResearch.engine.BattleRunner.run(BattleRunner.java:107)
at roboResearch.CLI.run(CLI.java:94)
at roboResearch.CLI.<init>(CLI.java:69)
at roboResearch.CLI.main(CLI.java:38)
Hmm, weird. Well, I don't know if this is the issue, but the instructions say when you use -t, you should also start the SQL DB separately and pass -S. The command to start the DB is:
java -Xmx1024M -cp hsqldb.jar org.hsqldb.Server -database.0 file:roboresearch -dbname.0 roboresearch
So I do that, then run with "-S -t 6" and it works. That "tjk.deBroglie rev0115 (2)" almost seems like you have two of that bot/version in a battle against itself, though - is that the case? Not sure if I've tried that or if it would cause issues.
Yeah, I've got the -S flag in there and have the server running in a separate terminal. Here's the whole thing:
tom@gecko ~/RoboResearch $ java -Xmx512M -cp bin:hsqldb.jar roboResearch.CLI -S -t 3 -C challenges/debroglie1.rrc -c "tjk.deBroglie rev0116" -r 35 -s 14
Challenge Specifications:
Challenge: deBroglierev108 150-57-95
Bot: tjk.deBroglie rev0116
Alias: null
Rounds: 35
Threads: 3
Type: null
Seasons: 14
(Running using server)
Result for robot named "tjk.deBroglie rev0116 (2)".
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1949)
at roboResearch.engine.Result.<init>(Result.java:31)
at roboResearch.engine.BattleResults.load(BattleResults.java:42)
at roboResearch.engine.BattleResults.load(BattleResults.java:24)
at roboResearch.engine.BattleRunner.run(BattleRunner.java:107)
at roboResearch.CLI.run(CLI.java:94)
at roboResearch.CLI.<init>(CLI.java:69)
at roboResearch.CLI.main(CLI.java:38)
Strange. I've been using the prepackaged one linked at the wiki here. Maybe I'll get the latest one via SVN and work around the build instructions that refer to nonexistent files.
And single threaded works, or the GUI? That's so odd. I know the code base a little and it seems like that path should execute the same across all of them. Or did you figure it out?
Single threaded works great.. same command line with the "-t 3" omitted.
Maybe having to do with the fact that I'm using OpenJDK? Do you use the Sun/Oracle JRE/JDK ?
I'm also using OpenJDK, on Ubuntu 12.04. (And previously the Apple JVM on a Mac.) The line that's failing is trying to parse the bot name from a line that, I think, would look like "1: tkiesel.deBroglie rev0116". If you want to hack on the RoboResearch source, you may be able to get a better picture. I would wonder if it's something about the Robocode version, except that it's working in those other configurations...
It looks like you have an extra space in the bot name, the " (2)". Could that be the issue?
Feel free to inspect the actual command I entered; I pasted it up above.
java -Xmx512M -cp bin:hsqldb.jar roboResearch.CLI -S -t 3 -C challenges/debroglie1.rrc -c "tjk.deBroglie rev0116" -r 35 -s 14
has problems while
java -Xmx512M -cp bin:hsqldb.jar roboResearch.CLI -S -C challenges/debroglie1.rrc -c "tjk.deBroglie rev0116" -r 35 -s 14
does not.
What's odd is that the lines that declare what config will be running (before the actual battles start) seem to indicate that the bot name was read correctly. Again, as I pasted above:
Challenge: deBroglierev108 150-57-95
Bot: tjk.deBroglie rev0116
Alias: null
Rounds: 35
Threads: 3
Type: null
Seasons: 14
But then, a moment later, RoboResearch has " (2)" appended to the end of the bot name. Very odd.
I think the (2) is appended to the bot name because it was a battle with two of the same bot. But I don't understand how RoboResearch could have gotten confused enough for that to happen. I think Skilgannon is right that it's confusing the parsing going on in BattleResults.load, though I can't exactly follow how (because I'm not too familiar with the format of the output it parses).
I'd really like to modify RoboResearch to use the Robocode control API, or write a new batch battle runner that does. RoboResearch runs a battle in a fresh Robocode instance each time from a .battle file and parses the results from the output. I recently tinkered with this - starting a new JVM each time is a lot more painful when you're running 1-round battles, like I was for my perceptual gun =). And I realized it requires a separate Robocode install for each thread, or you hit concurrency exceptions reading in JARs. RoboResearch makes use of a debug property that starts Robocode with a different "working directory" so each thread can load bots from a different directory, but there's really no way to do that for separate threads using the Robocode control API. (Not that it's a big deal to have a bunch of full Robocode installs, but it would make the change to RoboResearch quite a bit more complicated.) Also, the SQL db it uses slows down a lot once you have a lot of results in there, forcing me to manually clear it from time to time. I'm sure there's a better solution.
Sounds like a fun project! I'm no pro developer, just a physics guy who got into programming in college, but I'd love to contribute however I could to a project like that.
So I decided to go ahead and grab the latest tarball from the RoboResearch SourceForge site. A little rough with the documentation.. (TUI.java doesn't exist, and it never tells you to compile Server.java)
I've got the server up and running. The client is being reluctant though.
I got this:
Thread 1: Running tjk.deBroglie rev0117 vs. EBBU.Sim2 1.02
Thead 1: Unrecognized output from robocode, "jeremyreeder.Vincent: Got an error with this class: java.lang.IllegalStateException: Can't overwrite cause". Killing battle.
Thread 1: will be terminated
java.io.IOException: Stream Closed
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:236)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:282)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:324)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:176)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:153)
at java.io.BufferedReader.readLine(BufferedReader.java:316)
at java.io.BufferedReader.readLine(BufferedReader.java:379)
at simonton.utils.ReaderByEvents.run(ReaderByEvents.java:30)
Thead 1: battle was terminated
I saw on Talk:Scarlet that maybe it's an issue of Robocode version, so I copied over all of my 1.7.3.0 libs from the RoboResearch directory I've been using. No dice there, because now the error is:
Thead 0: Unrecognized output from robocode, "Loaded net.sf.robocode.api". Killing battle.
Problem running sample match, aborting
Dang. :(
