Talk:RoboResearch

SVN Update Notes

If you update from SVN, be ready to change your .cfg files. The example "run.cfg" is updated - it's still easy. Perhaps easier than they were! --Simonton 04:25, 16 September 2008 (UTC)

HSQLDB No More

Beware of abrupt shutdowns of the database server. I just lost all my scores between Simonton/PFResearch 0073 and 0083, apparently because I closed Eclipse which simply killed the database process, instead of shutting it down nicely. I certainly thought it would handle such things better than that, and I have done it many times before with no consequence, but apparently my timing was bad this time. Does anyone have a good recommendation for a more reliable database that can be run without firing off a separate process, if desired? --Simonton 07:51, 18 September 2008 (UTC)

It looks like the Apache Derby implementation distributed with Java 6 would be the natural choice! Expect that change to come in the near future, for fear of losing more of my data otherwise! --Simonton 15:04, 18 September 2008 (UTC)

It happened again. This time I only lost some data for 0088. I wasn't going to be upset if I lost alot more, but it just confirms this is a switch that has to be made. For now I'm making a copy of the database directory before I kill the database server. Sending the sever the shutdown command via SQL would work, but that's too much effort. They should make it accept a clean shutdown command from the terminal in which you launch the server, that would be handy. --Simonton 02:21, 23 September 2008 (UTC)

Buggy when running from GUI

Hey Simonton if you're around. I found a bit of a odd bug when trying to run the TCRM here. Under the GUI it always fails with:

Thread 1: Unrecognized output from robocode, "Aborting battle, could not find robot: apv.AspidMovement 1.0".  Killing battle.

whereas in the CLI it works just fine. Not sure why all this is but if I find a fix I'll tell let you know. --Rednaxela 07:10, 21 December 2008 (UTC)

Hmm, okay, I discovered that the problem disappaered if I cleared the working_dirs directory, and renamed all the bot jars to match the version number they have in the properties file. It seems that RoboResearch doesn't like jars that are differently named than the default, and some such non-default names are included in the zip files for challenge bots. Anyways I have it working now, and maybe this info posted here might help someone else. --Rednaxela 19:06, 21 December 2008 (UTC)

Ran into this - thanks for the info --Miked0801 02:13, 22 May 2009 (UTC)

Melee

Has anyone used RoboResearch to run melee battles? Looking through the code, it seemed like the part that runs battles would support running a melee battle, but I didn't see any way to configure this (from a .rrc file or otherwise). My melee Test Bed process is currently pathetic and verging on useless... I'd love some RoboResearch support to help me out. --Voidious 15:06, 18 May 2009 (UTC)

I'll take that as a no on anyone using it for Melee? I'm going to try and get it going, then. I'm already like a fish out of water in the Melee arena, a powerful testing tool would help me big time. --Voidious 15:41, 19 May 2009 (UTC)

Sweet, got Melee working (thanks to a lot of the guts already supporting it), though it's not incredibly elegant and the output isn't quite right yet. Good enough to be my new Melee testing utility, though. =) I'll post it or contribute it back to Simonton if/when I clean it up a bit. --Voidious 02:04, 20 May 2009 (UTC)

Got a much more polished version of the Melee support working, doing it in a way that seems mostly elegant in the RoboResearch code. My previous hack was ugly and just didn't always work right. Most of the internals actually already supported Melee, so it wasn't that hard once I figured out the overall structure of the code.

• If you have "Melee" in your challenge name, it uses Melee mode. This probably should be done another way, but this makes sense, works fine, and it's easy to make it something else if we want.
• If it's Melee, all the bots in a category are put in the battle at once. Multiple categories gives you multiple battles per season.
• 10 bots per battle max, a limitation in the existing code / database structure. Also, it uses 1000x1000 for battles with > 2 bots, that was already in there too.
• Scoring is displayed just like it would be in the MeleeRumble: one battle yields 9 separate scores, with the category score being the average of those, and overall score being average of categories.

What I don't have done:

• When the CLI is showing what battle it's running, it just says "<challenger> vs <first reference bot>".
• I haven't worked with the GUI at all because I am using the CLI. The only change I needed to make to the CLI (besides the previous cosmetic change) was for the JAR copying: before, it was copying the first bot from each battle, while obviously I need to copy all bots from each battle.

I'll share this and/or share it back to the SourceForge project shortly. I feel so much less lost in the Melee world being able to run some serious benchmarks. =)

Please do - Infinity and DustBunny would love some number tweaking help to unseat Kawigi's monster Lib. The GUI stuff is my current preference, but beggar's can't be choosers :) --Miked0801 16:26, 28 May 2009 (UTC)

--Voidious 03:12, 28 May 2009 (UTC)

Alright, just made a few tiny cosmetic changes to the CLI and the GUI (the "<your bot> vs <reference bot>" stuff). Viewing results for 45 bots (my current melee test bed) in the GUI isn't pretty, but you can do the "Copy as wiki" into a text editor if you need it more compact. Not that it's that pretty as pure text, either, but at least it's not a 2500 pixel wide window.

Anyway, here it is: void_roboresearch_melee_01.zip. That's the same base stuff you'd get from the SVN checkout plus the binaries compiled for 1.5. You can just overwrite your 'bin' dir with the one from this .zip if you already have RoboResearch setup. (Hopefully Simonton doesn't mind me distributing it like this, but I saw no such mentions in the source and I'm going to get this stuff back to him sometime...) Let me know if you hit any problems - enjoy. =)

--Voidious 01:08, 29 May 2009 (UTC)

Grabbed and in use - thanks! --Miked0801 10:23, 29 May 2009 (UTC)

Is this still the latest version? I'd like to merge this with my hacked-for-annoying-thread-stoppage version and do some melee testing =) --Darkcanuck 21:58, 5 September 2009 (UTC)

Yep, I haven't made any more changes since then. Let me know if you have any issues. --Voidious 22:35, 5 September 2009 (UTC)

Getting started instructions

Has anyone here got this to work at all?!? It'd sure help with testing... --Miked0801 23:36, 18 May 2009 (UTC)

You mean like to work for 1v1 / not Melee? Yeah, I've been using it. What kind of problems are you having? I tried long ago and hit issues which I attributed to the Mac JVM. Since I've been Robocoding again and using SoyLatte Java, it's worked fine, but it's also a different version of Robocode, so it could be that too. --Voidious 02:35, 19 May 2009 (UTC)

Mainly, I'm not a java hack. There are no step by step instructions nor a downloadable distribution so I didn't even know where to start. Being able to run batch battles locally would be awesome. --Miked0801 05:04, 19 May 2009 (UTC)

I bumbled through a lot of stuff, myself, but I did get it working. And it's great once it's working. =) I've posted a quick and dirty write-up of the process below. It's for Mac / Unix, while for Windows you'd use ; instead of : for Java classpath stuff. But I also zipped and uploaded my roboresearch dir (minus a few dozen of my dev versions) here, if that helps. Indeed we should update the page with some polished instructions, but I just wanted to get something up before bed here...

Thanks for the files, I bumbled around and actually installed the sun java compiler, then decided to just use your files. Everything seems to run just fine, though the CLI output is ugly. I'll dork around with the GUI next. --Miked0801 01:05, 22 May 2009 (UTC)

And I have fallen in love with the GUI version of this. This will allow me to test locally instead of testing distributed over the internet. Wee! --Miked0801 02:13, 22 May 2009 (UTC)

to install from scratch:
svn checkout https://roboresearch.svn.sourceforge.net/svnroot/roboresearch
cd ./roboresearch/trunk
mkdir bin
javac -cp src:hsqldb.jar -d bin src/roboResearch/CLI.java
install a copy of Robocode to roboresearch/trunk/robocode_install

put your bots in robocode_bots

run battles with something like:
java -cp bin:hsqldb.jar roboResearch.CLI myBot.cfg

for multi-threaded, setup server with database_server.sh, then 
do something like:
java -cp bin:hsqldb.jar roboResearch.CLI -S -t 2 myBot.cfg

docs/getting_started_running.txt has some info, 
but some is outdated (boo)

--Voidious 05:28, 19 May 2009 (UTC)

Oops, I nearly forgot, one more important detail: I did have to actually edit the source to fix one thing. Some Robocode message had its wording changed, and RoboResearch was aborting if it didn't recognize the message. (I should find that and fire it back to Simonton.) So anyway, you should definitely grab my .zip and use that or compile from that source. --Voidious 05:33, 19 May 2009 (UTC)

• Well, or instead of the CLI thing, you can use the newish GUI version. I find it's rather nice, and if you want to do multi-threaded it can manage all the threads and has no need to seperately start the database server etc. I find it rather nice to be able to compare versions directly in the GUI version as the battles run... ;) --Rednaxela 07:21, 19 May 2009 (UTC)

Again, put a distribution or something together with instructions. I'm a C/C++/Asm programmer, not a Java dude - as you'd know if you look at my very C like code :) --Miked0801 16:10, 19 May 2009 (UTC)

Hi! I'm also interested in RoboResearch but I don't get on well with it so far. :(

1. I downloaded voidious's zip and tried to run the run.bat. : [1]
2. Then I modified TUI to CLI in the batch file. : [2]

--HUNRobar 17:13, 13 June 2009 (UTC)

run.bat is out of date and only runs some of Simonton's tests anyway. Best way to get started is to use the GUI: "java -cp bin:hsqldb.jar roboResearch.GUI" (mac/linux) or "java -cp bin;hsqldb.jar roboResearch.GUI" (windows). Remember to start this from inside the roboresearch dir. --Darkcanuck 17:27, 13 June 2009 (UTC)

Argh, classNotFoundException again.. --HUNRobar 17:34, 13 June 2009 (UTC)

Just downloaded Voidious's zip and I see that everything's actually inside the "trunk" dir. So you'll need to cd into roboresearch/trunk before running the command I suggested. I have mine setup slightly differently but this should fix that exception... --Darkcanuck 21:08, 13 June 2009 (UTC)

Argh again... the same thing, just look at it: [3] --HUNRobar 16:26, 16 June 2009 (UTC)

Looks like that zip is missing GUI.class. You should be able to build it by doing "javac -cp src;hsqldb.jar -d bin src/roboResearch/GUI.java" from the trunk dir. --Darkcanuck 05:24, 17 June 2009 (UTC)

When running multi-threaded I didn't setup server with database_server.sh nor used the -S option as suggested in the instructions above, I just used the -t 2 option, is there any problem in doing so? --Navajo 21:54, 23 July 2009 (UTC)

Have you tried the GUI? No command line options and multi-threading is just a few clicks. I'm never going back... --Darkcanuck 04:34, 24 July 2009 (UTC)

I tried the GUI, but I like the CLI, to run multi-threaded instead of a few clicks I only need a few digits. I just wanted to know if there was any problem in just using -t 2 to run two threads instead of setup the server. I did it both ways and didn't notice any difference. --Navajo 13:59, 24 July 2009 (UTC)

Hmm, I have no idea. I thought it just wouldn't even let you do it. =) Maybe he updated the code at some point, but not the instructions? Also, just to be sure, do you have a dual core CPU or multiple CPUs? If not, multi-thread won't really do much for you, but it might make bots skip turns since they have less CPU time... --Voidious 15:55, 24 July 2009 (UTC)

I'm using the zip file you posted here, and it allows multi-threaded this way. I have a dual core CPU, so this multi thread is realy useful to me. --Navajo 16:17, 24 July 2009 (UTC)

I just trued running robo research on ubuntu 9.10 and for some misterious reason I'm unable to run more than 1 thread. Does anyone have any idea of how to solve this? --Navajo 23:45, 17 January 2010 (UTC)

What error are you getting or what is the symptom? Are you running the database by itself and using -S when you run RoboResearch? (I know someone, maybe you, said they didn't need to, but the instructions still say to do that for multiple threads.) I haven't personally tried it multi-threaded on Linux, but I can't imagine why it wouldn't work. --Voidious 01:58, 18 January 2010 (UTC)

I'm using only the GUI to run everything. What happens is that one thread runs normaly while the other gets stuck at starting forever --Navajo 03:49, 18 January 2010 (UTC)

What version of robocode is being used? Perhaps it has to do with that. I haven't used multi-threading much myself, but I recall it working in the past (I personally avoid using the multi-threading though for two reasons: 1) My laptop's CPU gets hotter than I'm comfortable with, and 2) I can't do other things on the system at the same time and be sure I'm not causing skipped turns) --Rednaxela 04:01, 18 January 2010 (UTC)

I've tried both 1.7.1.6 and 1.6.1.4, and none of them worked. I usually leave robo research running at night, so there is no problem with doing other things on the system, and I don't care much about the heat (what is the point of having a good computer if I can't use its resources...)--Navajo 22:15, 18 January 2010 (UTC)

Is there a roboresearch for dummies page ? when i type "javac -cp src;hsqldb.jar -d bin src/roboResearch/CLI.java" It tells me:

"javac is not reconised as an internal or external command..." -Jlm0924 20:37, 14 May 2010 (UTC)

Feature Request

One thing that is sorely missing is a standard deviation calcuation per battle (and/or overall) so that you can get a feel for just how accurate your results are. It would go a long ways towards telling someone just how many seasons they should run to get stable results. --Miked0801 20:56, 2 June 2009 (UTC)

I believe it shows something like that if you hover the mouse over a score for a moment in the GUI. ;) --Rednaxela 23:05, 2 June 2009 (UTC)

Is there a way to not have it bring a thread down on an exception? MosquitoPM cost me 1/2 a night of processing as it sabotaged all 3 of my threads :) --Miked0801 16:45, 16 June 2009 (UTC)

No, I think now. When I ran RoboResearch, I need to get up every 3 hours or so to check if they are running correctly. » Nat | Talk » 16:58, 16 June 2009 (UTC)

I hacked my version to handle a few more error messages so that it doesn't stop as often. I think the robocode console output has gotten more wordy than when Simonton first put this together. The relevant code is easy to spot in LineListener, near the bottom of src/roboResearch/engine/BattleRunner.java. I should add exceptions though, I'm constantly restarting my threads... --Darkcanuck 05:15, 17 June 2009 (UTC)

Personally... I really find it preferable to stop running a thread if there's an exception, because 1) if my bot is throwing an exception, I've got a bigger problem then benchmarks would help with, and 2) If a bot in the test bed is throwing an exception, then well, it's a bad test bed... --Rednaxela 06:31, 17 June 2009 (UTC)

Toys

I temporarily deleted the package toys (on the local copy), as it was giving me errors, will it still work? --Starrynte 02:06, 23 September 2009 (UTC)

Update for 1.7.*

Is there any known update or new version of this for the 1.7 line of robocode. — Chase-san 16:44, 14 July 2011 (UTC)

I've used it with multiple 1.7.x versions, the latest being 1.7.1.6. Are you hitting any issues? --Voidious 17:16, 14 July 2011 (UTC)

Yes, I am using the one pulled from the SVN. While I have been fixing the problems as I go. I just sorta want to use it. Is there a newer version I don't know about? — Chase-san 18:22, 14 July 2011 (UTC)

Nothing official. I ran my changes by Simonton at one point but we agreed they needed some polishing. But, of course, polishing goes on the back burner when what you have works and you're in the throes of addiction. =) I did post my binaries under the "Melee" section of this page, with my hacked Melee support and whatever else I had done: void_roboresearch_melee_01.zip. --Voidious 18:27, 14 July 2011 (UTC)

http://file.csdgn.org/image/category_selection.jpg

Ah, I have already fixed all the errors in mine. I even threw in category selection from the Add menu. I'll merge in your changes and any other convenience bits I can think of and upload a binary package. — Chase-san 18:55, 14 July 2011 (UTC)

I have modified your copy of roboresearch to have my category modification and to support 1.7.3 (which it didn't). You can get it here. I do not include a copy of robocode already installed, but a simple copy and paste solves that issue. — Chase-san 20:05, 14 July 2011 (UTC)

Some Suggestions

I've been using RoboResearch some lately, and have a few suggestions.

1. This one is just my own opinion, and others may see it differently. In the Results window, the "Total" is an average of the "Sub" group totals. I think this is misleading, and should be an average of all individual values. As it is now, I don't think it is intuitive, and you have to be very careful to balance each of the sub groups equally for the total to be meaningful. I know I can get around it by just eliminating all of my sub groups and running every robot under a single group so that I get a total that is of value to me, but I don't think I should have to do this.
2. The results and seasions windows should have the table in a JScrollPane.
3. Threads should have an option to set a number of retries when an exception occurs. Furthermore, if the number of retries is exceeded, it would be very nice if there was an option to just ignore the robot and continue on.
4. There needs to be an obvious way to reset challenges. For example, when I was having trouble with some robots throwing exceptions and pausing the threads (see previous item), the best thing to do was just remove the trouble robot from the challenge. However, when starting back up, even if the challenge is removed and readded, the old data for the challenge still gets picked up. It is very much not clear how to reset this to start fresh.
   1. Side note to this: I discovered that you can update the groups in the rrc file, remove the challenges from the GUI, then readd them. When you do this, it picks up the old battle data and puts it in the new grouping. This is nice, though I had to discover it by accident. And I still think it would be nice if there was an obvious way to clear out data so you don't have to keep changing the version numbers for every quick little bug/issue fix.
5. When there is only a single thread in shown in the GUI, and you try to remove it, it does not clear out the thread controls box.

-- Skotty 21:43, 18 August 2011 (UTC)

1. This isn't how challenges usually worked on the wiki, which what RoboResearch was designed with in mind. Just create that other challenge and open it in roboresearch after you have run the battles to get the other result.
2. Yes, it probably should. :)
3. Yes, again it probably should.
4. There was at one point a way to goto the additional add bot screen and erase a robot and delete all its values from the database as well, but that doesn't seem to work anymore (or yet?). It stores all battles, and just plugs them into whatever challenge you decide to add/run later.
5. This is expected behavior. Whats the point of having an empty thread box?

— Chase-san 23:30, 18 August 2011 (UTC)

Regarding "This is expected behavior. Whats the point of having an empty thread box?" -- If you look closer, you'll find that clicking the Remove button on the thread when it is the only thread in the box, the Remove button keeps it's depressed appearance and the controls on it no longer work. This happens because it's not actually there; the window needs a repaint. This can be demonstrated by then resizing the window, which causes a repaint, and the thread box disappears. -- Skotty 00:36, 19 August 2011 (UTC)

Actually the funny thing is, that was my original response, but I removed it after overthinking things and assuming it was just prevention of pointless. — Chase-san 03:29, 19 August 2011 (UTC)

Re: #3, I hacked my version to restart the thread after a bot exception (the result still gets thrown away). Probably not too hard to implement a retry either. For some reason, bots such as Phoenix crash occasionally on my system -- without this hack, I was constantly restarting threads. I keep an eye on the first challenge round to make sure my bot runs ok, just in case. --Darkcanuck 05:14, 19 August 2011 (UTC)

I suppose that wouldn't be too hard to do since I am running it from Eclipse. Pretty quick and easy to make a code change. It definitely needs it. I hate coming back to my PC 4 or so hours after starting a challendge to find that some bot I added to a challenge threw an exception and the thread(s) have been sitting idle for hours. I ran into this most recently with LemonDrop, which runs fine most of the time, but on infrequent occasion throws an exception (array index out of bounds, iirc).

New Version 1.2 (finally)

I have updated RoboResearch to version 1.2. It has been a long time since it has had much done on it.

Change Log:

1. Resolved numerous repaint errors in the GUI.
2. Now with configuration pane and working button. :)
   • Allows you to change the challenge name and number of seasons running.
   • Yes, it does save these changes between open/closing.
3. The result and season panes display a vertical scroll bar when needed.
4. Threads now automatically restart after 5 seconds when they fail.
5. Now runs robocode in Debug mode, to avoid skipping turns when you are doing other things as well as testing.
6. Now produces tabulated data when you copy from the results and seasons tables (instead of java references)
7. Updated for 1.7.X (awhile ago)

Known Issues:

1. The configuration pane doesn't resize correctly.

ToDo:

1. Add an option menu (or configuration file actually)
2. Fix the "Delete Bot" and "Delete Versions" buttons in the Version Browser (to delete all results for that bot/version)
3. Move the "Remove" button to the bottom of the button pane, away from the results button.
4. Ability to access the Version Browser from the main menu
5. Ability to automatically find open results browsers for the same challenge and add results to that when clicking results
   • This is a touch more involved then you may think.

— Chase-san 09:55, 25 August 2011 (UTC)

Cool - I've been thinking for a while someone needs to fork this and start updating it again. I don't think I agree with running in Debug mode by default - I don't really want skip turns masked in my tests, nor do I want my tests to run slower against bots that skip turns (I run a lot of tests against large fields of bots). I think Darkcanuck, I, and some other people might have some misc fixes that should be integrated, too, I'll look into my changes. --Voidious 12:34, 25 August 2011 (UTC)

It'll be the first thing I add to the configuration settings. (I felt this way too) — Chase-san 13:05, 25 August 2011 (UTC)

I currently work on distributed analog of RoboResearch. It's SOAP-based web service for battles execution and client. So we will can contribute out CPUs. It's now supports .rrc challenges format and will support wiki format for output results, so there're no big migration problems. I plan to release alpha version of it under open source licence in next few days. I need help, especially in UI part. So, may be, our community can switch development effort to this tool? --Jdev 12:54, 25 August 2011 (UTC)

Are you sure it wouldn't be easier to modify RoboResearch to run in a distributed manner? I would be better to consolidate our efforts if possible... --Skilgannon 13:12, 25 August 2011 (UTC)

Have you updated the parsing so that it handles 1.7.* style printing and results? --Skilgannon 13:12, 25 August 2011 (UTC)

Oh yeah, but I did that awhile ago. — Chase-san 13:16, 25 August 2011 (UTC)