View source for Talk:RoboJogger

I have been using RoboJogger for movement for about one week and it was working perfectly. However, when I tried testing my robot on TCRM and MC2K7 it didn't work. My robojogger's robocode is 1.7.3.0 but I don't think that's the case since it produced similar results to what I had on my robocode 1.9.3.2. It seems like RoboJogger only accepts PERCENTAGE_SCORE. Below there is my TCRM file. Thank you in advance.

Targeting Challenge RM
AVERAGE_BULLET_DAMAGE
35 rounds

Easy {
   apv.AspidMovement 1.0
   dummy.micro.Sparrow 2.5TC
   kawigi.mini.Fhqwhgads 1.1TC
   emp.Yngwie 1.0
   kawigi.sbf.FloodMini 1.4TC
}

Medium {
   abc.Tron 2.01
   wiki.etc.HTTC 1.0
   wiki.etc.RandomMovementBot 1.0
   davidalves.micro.DuelistMicro 2.0TC
   gh.GrubbmGrb 1.2.4TC
}

Hard {
   pe.SandboxDT 1.91
   cx.mini.Cigaret 1.31TC
   kc.Fortune 1.0
   simonton.micro.WeeklongObsession 1.5TC
   jam.micro.RaikoMicro 1.44TC
}

An addition: Same things happen when I use robocode 1.9.2.5

This question probably has a really simple answer but I have been trying for over a month and still, I couldn't test one robot on it.

Oh, I am so dumb. I gave the new Robocode version; now it works perfectly.

2 bugs so far that I have found:

1) A left over java process seems to hang around for each full execution of RoboRunner. This needs to be tracked down and eliminated (could it be the callback queue in RoboRunner?)

2) When RoboRunner throws an error (for example, due to a missing robot), RoboJogger does not realize that RoboRunner has died and the controls to "stop" RoboRunner do not function. This results in RoboJogger being stuck and requiring a force quit. This should be fixed so that either RoboJogger correctly detects the failed start of RoboRunner or at least is able to reset everything if the command is given to stop RoboRunner.

Are these bugs present in 0.9.6? I'm finding it quite annoying to have to keep restarting robojogger =)

I'll be looking at fixing these bugs soon. I kind of forgot they existed for awhile.

I'm currently on hold because my basement server died awhile ago, and it had my source repository on it (in addition to my data backups). I have mostly new hardware now, but I'm still waiting on a new hard drive for the OS, as I at first bought a refurbished one from Newegg and it was dead (last time I ever try to buy a refurbished part). Once the final replacement hard drive arrives, I will have a mostly updated system with an OS drive and 2 1TB data drives. The old system was a Pentium 4 with 1GB DDR1 RAM, so it was definitely due for an update. I will also be installing the latest version of Fedora, which also means I can easily use Java 7 on it and possibly finally set up a distributed Robocode node on it (if that project is still alive).

Back on topic, I'm glad to hear that someone is still using RoboJogger. Knowing that gives me encouragement to get back on it and make it better. I will also explore the possibility of moving the source to a public location, should anyone want to tinker with it.

Hi, since i am only user of drc, the project is more dead, than alive:) But if you install it and experience problems, then you can email me (see Contacts) or patch it yourself: github repo :)

Not sure if its a bug with RoboJogger, RoboRunner, or my robot. However I'm getting spurious results vs some bots.

For example vs Tron 2.01 in the TCRM challenge I'm getting an average of 87% percent when running 10 seasons manually, but in RoboJogger I'm getting an average of 61%.

I'm running Robojogger on the Mac, latest version of Robocode. I'm aware that Tron seems to be quite a slow bot, I was wondering if my bot was generating skipped turns for some reason as my bot is not too fast either. Notably in Robocode I am not generating any skipped turn events from what I can see. How do I access the Roborunner bot output? Is there a way or not?

Manually means running by hand in Robocode directly FYI incase it was unclear. :)

Oooh just saw there was a Robocode update that fixes some skipped turns, it literally came out today so I'm going to re-test with that and see what happens!

Tried the new Robocode version and I'm still getting the same result - running manually in Robocode gives a higher score by around 25% for my bot vs Tron compared to running in RoboJogger. :(

Are you sure you're using the same scoring mechanism? I know the TCs define score as TOTAL_BULLET_DAMAGE/ROUNDS.

Yes, the setup for the RoboRunner config file is using AVERAGE_BULLET_DAMAGE. Note that the results for all the other bots in the challenge look correct. It only appears to be vs Tron for some reason, which is why I postulated it was because Tron appears to be a slow bot which may cause skipped turns.

It might be because Tron starts firing - if it doesn't get its config file which puts it in TC mode copied correctly, for instance.

Yeah, I bet Tron is firing. Pretty dumb we don't just have a non-firing version in the TCRM downloads? Or do we? I remember doing that manually for a long time, anyway. Same with DT.

Bot output doesn't go anywhere in RoboRunner. Actually not exactly sure how to catch it but it would be a nice feature. You could log stuff to files though.

Try this: IBattleListener.onBattleMessage(BattleMessageEvent event)

Makes sense, but I checked the .data directory for tron for the 4 instances of RoboRunner that RoboJogger generates and all of them have a properties file with "challenger" set. :(

Oh, hmm. That sucks. 61 is a pretty unthinkably low score vs Tron, looking at the TCRM results. I guess it would be useful if RoboRunner/RoboJogger had a switch for displaying battles to help debug this kind of thing.

Can you try running battles manually from one of those Robocode instances?

Found the cause of the problem but im not sure why. Running Robocode from inside Eclipse means my robot doesn't get any Skipped Turn events, but running robocode from the robocode.sh file means my robot does get skipped turn events. That means two things:

1) There is a difference between running robocode from inside and outside eclipse 2) My robot is running dead slow (but mainly versus tron?!)

The second problem is something that I need to deal with, but the first is interesting. My command line arguments for running Robocode in Eclipse are "-Xmx512M -Dsun.io.useCanonCaches=false -Ddebug=true". Would this cause Robocode to ignore skipped turn events?

-Ddebug=true does disable skipped turns, I checked the engine source. It is made exactly for that, so you can pause and trace execution step-by-step.

Ahhhh cheers problem found. I guess I need to look at optimising my bot. Unfortunately I've found that I can greatly increase its score versus a lot of bots by doing a lot more work. Sigh :(

There needs to be a way for a third party (e.g. RoboJogger) to access battle results (on the fly results highly preferable). It looks like I could use ScoreLog to read results from the XML result file after it is created, but it would be preferable to have a way to add a listener that is notified each time a new battle result is available. For example, RoboRunner might allow third parties to add their own BattleResultHandler in addition to or in replacement of the one RoboRunner uses internally. Thoughts?

Yep, that makes sense. I think when you're listening to RoboRunner, you might want some higher level data too, like avg score and number of battles. So I think adding a new / similar listener to the RoboRunner class makes more sense than just letting you add more custom instances of the existing BattleResultHandler. The new interface method could take the raw scores and elapsed time, as now, plus whatever other summary data you want. At the end of RoboRunner's BattleResultHandler.processResults(), we call the higher level listener, if it's been set.

Does that sound about right?

Sounds good. Is this a change you would like to make or would you prefer I make the change and send it back to you for review?

I say just make the changes you need and go with it, and I can merge them into the main branch whenever. But I'm happy to look over stuff or even write up the interface if you need me to.

I still haven't gotten around to working on this. I suppose I would be happier if you added it such that it can be implemented in the manner you feel is best, but I will work on it once I run out of other tasks if you don't have time for it. I've actually gotten quite far into development relying only on what I can get from ScoreLog. Other than not having a nice way to track battles on the fly, the only thing ScoreLog doesn't provide that I wish it did is a way to retrieve or calculate the confidence values. I love the idea of computing confidence values, and am determined to include it somehow.

Sure, no problem, I'll take a look sometime today or tomorrow. It will be trivial to pass along the raw scores, some summary data, and the basic per-bot confidence interval stuff. Passing the groups and overall scores and confidence intervals might prompt some refactoring to do it cleanly, since right now they're kind of just calculated in-line before printing them, but it shouldn't be too tough.

I see in 0.9.5 notes: "Added a basic output parser for RoboRunner. Challenge run progress information in the main window is now updated on the fly..."

Does this mean you took care of this listener thing yourself since I never actually did it? Or are you still lacking some way to get updated with RoboRunner's results (and/or confidence estimates) on the fly?

Also, I'd like to merge back whatever changes you made to RoboRunner, at least to the GitHub repo. I guess I'll just wait and take a look at the source after 0.9.6.

The only information the output parser extracts is a count of battles completed. It is not able to determine the scores, nor does it parse out the confidence. The only thing it provides is on-the-fly information on how many battles have been run, in order to update the progress in running a challenge. It was a way to do a little more, but does not take the place of having a listener that actually provides score information.

The changes to RoboRunner so far have been very minimal. My focus has been almost exclusively on RoboJogger. The changes I have made to RoboRunner include only the following:

1. In BattleRunner, I made it so that getAllFutures(List<Future<String>>) could be interrupted in order to provide a way to stop RoboRunner. However, my particular implementation needs to be done in a better way than I did it. It sets a flag on InterruptedException that causes all remaining futures to be cancelled (it calls future.cancel(false) on all remaining futures). It needs to be improved such that it can safely call future.cancel(true) without potentially corrupting a score log, or have the futures be able to detect the stop request to shut down cleanly if running.
2. In ScoreLog, I added an ReentrantLock that is locked whenever loadScoreLog(...) or saveScoreLog(...) is called. Thus, only a single score log can be loaded or saved at a time. This was necessary as both RoboRunner and RoboJogger call the loadScoreLog(...) method and thus it needed some kind of synchronization. I know it would be safe to load and save different score logs at the same time, but I decided that there would not be enough concurrent loading and saving for it to be worth the added complexity.
3. In ScoreLog, I close the XMLEventReader and input stream after a ScoreLog is loaded. Not sure if this really matters or not in practice.

And that's it. Even though you may not have specifically designed it for being used by another program, RoboRunner is quite easy to interact with and it required very little modification to put my UI on top of it. Very nice.

I'm still losing chunks of score logs on rare occasions, and I have no idea why. I still need to add checks to ensure RoboRunner is cleanly shut down before exiting RoboJogger, but I've had data loss at a time when I was absolutely sure RoboRunner was finished running, so there is still a problem somewhere that needs to be solved. If I can track down that problem and take care of the other things noted above, I will be very close to having a non-beta-ish release.

I just made another change to RoboRunner that will be part of 0.9.7. Again, I kept it pretty minor, but that makes 4 minor things now, so I should probably put the source out there where you can get to it. The new change I made was to add isTerminated() methods to RoboRunner and BattleRunner. These new methods return whether or not the main thread pool and the callback pool are terminated (by checking isTerminated() on both thread pools and then feeding that info back up the hierarchy).

isTerminated() returns true on a thread pool when shutdown() has been called and all threads in the pool are done. This is important to RoboJogger in determining whether or not RoboRunner has finished shutting down before starting a new runner or allowing a user to exit the program.

Another alternative would be to add a shutdown method to RoboRunner that is blocking, but I didn't think that would work as well, so I didn't take that approach.

Updating the result windows on the fly doesn't seem like it would be particularly difficult. But I don't know the structure of your code at the moment, so I could be wrong.

Assuming a singular method that displays the windows given a result set.

a la "void showResultsWindow(resultSet)", could make it "ResultWindow showResultsWindow(resultSet)", keep it in a map "HashMap<ResultSet,ResultWindow>", and update the window based on updates in the set "ResultWindow window = map.get(resultSet); window.updateData(resultSet); window.markTableDirty();"

Mark table dirty would be this piece of thread safe code "RepaintManager.currentManager(table).markCompletelyDirty(table);"

Keeping the window in a map also allows you to just move that window to the front if someone tries to open the same result window twice. Though you make require a listener to determine if the window was closed, so you can remove it from the hashmap. Just hiding it could be problematic if someone keeps the application open for a long time and runs many different challenges.

It would be nice to have a season counter as well. That you can see without getting the wiki output.

Also I saw the logo in about, and my eyes bled. So here is a doodle.

http://i2.minus.com/iCUtdsJ56Qgpr.png

Also before I forget. You should have it do at least 3-4 battles per bot before 'smart' battles kick in. This is just in case you got two flukes in a row. Where as 3-4 in a row become considerably less likely. I only mention this because I just got two flukes in a row (a really high score at that).

It does sprinkle in random battles among the bots with lowest battles. I didn't want to do 3-4 for everyone up front because that largely nullifies the usefulness of smart battles if you're doing 500 bot test beds. But as it is, if you're running several thousand, everyone will get to 3 or 4 at least, I'm pretty sure.

Ah good to know.

Naturally, I found 2 bugs within an hour of releasing 0.9.5. Both will be fixed in the next version. First, I accidentally left a debug message in that gets written to the RoboRunner output window after every battle. That's already gone. Second, I discovered that running the Remove All function causes exceptions to start happening when trying to run RoboRunner after a Remove All. I still have to look into this, but it will get fixed in the next version. In the meantime, should this happen to anyone, the solution is just to rerun the Setup function.

Another bug that is still around, just FYI, is on rare occasions some data can be lost from one of the score logs. I am still not sure under what scenario this happens, but it did happen to me again recently. It didn't totally corrupt the score log, but it did lose some of the battles, and those battles had to be re-run for some of my challenges.

Another potential issue -- not really a bug -- is that RoboJogger can be slow to start up if you have a large number of challenge runs, because it recomputes completion information for every challenge run on startup, which can mean reading a lot of score logs. I was thinking for the next version I would store completion information separately (basically store it with the challenge runs instead of recomputing from the score logs) to make initial start up quicker.

Big bug due to a fat finger mistake in version 0.9.3. PERCENT_SCORE challenges will give you an Unsupported Challenge error when you try to add them. I'll get this fixed soon. It was due to a typo on my part, along with inadequate testing. To get around this bug, you can change your challenge file to be "PRECENT_SCORE" (note the spelling error) and then it will run. Or just wait for me to put out the next version, which could be tonight or several days from now depending on how busy the new baby in our family keeps me. Sorry for the error.

Okay folks. Help me out here. I didn't see any page on the wiki that details how all the challenge scoring types work. I'm basically just guessing on everything but normal scoring and bullet damage scoring. What I'm currently doing is best shown by just posting the class that currently handles scoring, and you all can let me know what needs to be changed. Thanks!

//TODO: Verify how each scoring function is supposed to work
public class ScoreFunctions {

	public static ScoreFunction PERCENT_SCORE = new ScoreFunction() {
		@Override
		public double getScore(RobotScore challenger, RobotScore opponent, int numRounds) {
			return challenger.score / (challenger.score + opponent.score);
		}
	};
	
	public static ScoreFunction SURVIVAL_FIRSTS = new ScoreFunction() {
		@Override
		public double getScore(RobotScore challenger, RobotScore opponent, int numRounds) {
			return challenger.survivalRounds / (double)numRounds;
		}		
	};
	
	public static ScoreFunction SURVIVAL_SCORE = new ScoreFunction() {
		@Override
		public double getScore(RobotScore challenger, RobotScore opponent, int numRounds) {
			return challenger.survivalScore / (challenger.survivalScore + opponent.survivalScore);
		}		
	};
	
	public static ScoreFunction BULLET_DAMAGE = new ScoreFunction() {
		@Override
		public double getScore(RobotScore challenger, RobotScore opponent, int numRounds) {
			return challenger.bulletDamage / (double)numRounds;
		}	
	};
	
	public static ScoreFunction MOVEMENT_CHALLENGE = new ScoreFunction() {
		@Override
		public double getScore(RobotScore challenger, RobotScore opponent, int numRounds) {
			return challenger.energyConserved / (double)numRounds; 
		}
	};
}

MOVEMENT_CHALLENGE is generally "100 - (bullet damage taken / total rounds)", or "return 100 - (opponent.bulletDamage / (double)numRounds)". Though if you can get the Energy Conserved, that might be approximately the same.

BULLET_DAMAGE is AVERAGE_BULLET_DAMAGE.

Otherwise I think it looks correct.

Also keep in mind that RoboRunner supports melee battles last I checked, so a single RobotScore opponent may not be sufficient unless you add up all the opponents data into that one entry. Even then, I am not sure if the math works out correctly, especially with my definition MOVEMENT_CHALLENGE.

I try and do a tcas/tcrm challenge, but I get a Challenge not supported. Unsupported scoring type: AVERAGE_BULLET_DAMAGE.

Doesn't robojogger support this scoring method? Or does it have a different challenge file syntax? If so, what is it?

Unfortunately, I need to go back to roboresearch.

EDIT: After some research, I found it was "BULLET_DAMAGE", please make it support "AVERAGE_BULLET_DAMAGE" as well, if only as an alias. So that we can just copy and paste roboresearch challenge files. :)

It seems to fill in the results with 100, despite the actual scoring in the roborunner output being something else. The results in the results window do not update during the running (not even the erroneous scores). The confidence scores were always 0.0, just as the scoring was always 100.0.

Even after a full three seasons, the scores did not correct themselves. The correct results are in RoboRunner output of course.

Hehe, while I wish I could believe I made a gun which produced such scores, I didn't.

The scores not updating during running is normal. I'm waiting for the next release of RoboRunner before I implement that. As for the messed up scores, I would guess this has something to do with an error in what I'm doing with the scoring type. All of my testing so far has been with PERCENT_SCORE. I'll start testing with other scoring types and fix whatever problems I find.

Would anyone like me to add an option to send RoboRunner output to a text file?

Okay, so I looked into what is happening with the scores under the BULLET_DAMAGE scoring type. What RoboJogger is doing is taking the score for the scoring type for the challenger and dividing it by the sum of the score for the scoring type for both the challenger and opponent. I think this seems right, but I don't use scoring modes other than PERCENT_SCORE very often, so I'm not entirely sure.

Why this comes up with a bad number is because when the scores are loaded from the ScoreLog (ScoreLog is part of RoboRunner), the bullet damage score (which the BULLET_DAMAGE score type relies on) is always 0 for the opponent. This might be a bug in RoboRunner -- either the scores not getting saved correctly or loaded correctly by the ScoreLog. I also noticed that the energy conserved values were also 0 for both the challenger and opponent, so may be a related bug there too.

TC scoring is bullet_damage / number_of_rounds. Which produces an output between 0 and 100 (not 0.0 and 1.0).

The your_score / (your_score + enemy_score) is a percent index. It can't be used with bullet damage where one of the robots does not fire (which is what happens in a TC). The reference robot (the one moving) will never have any bullet damage.

I guess I assumed all scores were just percent scores based on different scoring metrics. I need to figure out where I can find information on exactly how each challenge type is scored. I'm not following your explanation on TC scoring, as I don't see how a good robot that doesn't run itself out of energy with misses would ever score anything other than 100. If the opponent doesn't fire (or hit walls), the challenger would score 100 on every round.

Okay, reading a targeting challenge page more closely, given that the challenger is only supposed to fire power 3 bullets, I guess the intent is that on some rounds the challenger will run out of energy such that scores will vary. While I like the idea of not having the opponent firing back as an extra variable, having to alter your challenger's gun to only fire power 3 bullets means this is not a full test of the challenger's gun; it's just a test of the challenger's gun's aim with power 3 bullets, leaving out distance and power controls.

I also have to wonder why the movement and targeting challenges are not just inverses of each other.

Well, that could happen, except you are only allowed to pass 3.0 as the fire power. Which means that no current robot is able to get 100 bullet damage all of the time.

Well, an enemy who doesn't fire has no chance at regaining any energy. So you can only at the absolute most do 100 damage to them in a single round.

Of course if the enemy damages itself by hitting the wall or your robot. Then your score will not be 100, since you did not do 100 damage, even if you kill it. This also happens if you don't win the turn. If your robot disables itself from firing, then only the damage you did that round gets counted.

As for damage. Well you do more damage then power you put into a bullet. The algorithm is this. Taken from Rules.java

double damage = 4 * bulletPower;

if (bulletPower > 1) {
	damage += 2 * (bulletPower - 1);
}

So you can get 100% without hitting every shot. You do 16 damage for every 3 power bullet you fire. For every 1 power bullet you fire (say after you shot and missed 33 times), is 4. So you only have to hit 7 times to kill the enemy. Though it is often not that simple.

Post any bugs you find in 0.9.1 here. Note: If you have 0.9 and want to keep 0.9 data and challenges, just make sure you save and move robojogger.dat and all the files in the data directory. You will probably want to move all the robots from the bots directory too. Note that on first install, there is no bots nor data directory; you can add them manually or just start and stop RoboJogger once to let RoboJogger create them.

I had one strange thing happen so far. Once, when I was starting RoboRunner, the CPU hung at 50% and the battles never started. I stopped then restarted, and it ran fine after that. So be on the lookout for that. I'll be exploring to try to find out what might have caused it.

Had issues with locking up when RoboRunner is started again. I'll be running a lot more small challenges in development to track down the problem and fix it. Not sure what is going on, but it only seems to happen when RoboRunner starts a new challenge.

I had to update my java 1.6_11 to the newest 1.6_37 version to get RoboJogger running. I just mention it, if someone else has trouble to start the jar.

I haven't done anything so far, but i plan to give it a look over the next days.

Hm... that's on Mac, yes? I bet the Apple Java Extensions changed somewhere between 1.6_11 and 1.6_37. I'll at least see if I can figure out exactly what version of Java this changed with on the Mac so I can put it in the notes.

In other news, I found and fixed another bug, which will be fixed in the next release. RoboJogger can fail to load a Robot properly if the robot jar file contains multiple .properties files. Already fixed in my source.

FYI -- I'm adding a new little feature to the next release. For each challenge run, you will be able to add a note/description of what the challenge run is for. This is something I will definitely use and I hope some of you might find it useful as well. It will be an extra column on the main window, but I will probably add a way to show/hide various columns.

Great idea! I always have a scratch pad open with a short description and overall score for each dev version.

I did some more testing since I threw up links to version 0.9. I noticed that if you let RoboRunner process all challenges, once it is complete, the buttons and menu item to start RoboRunner and do things like Setup do not re-enable. This was an issue with the way I was doing locking, and I've already fixed it in my source. I'll put up a version 0.9.1 in a few days where that will be fixed.

I also noticed that if you have a challenge where the robots have nicknames (like MC2K7), error messages show up in the log for the nicknames. This doesn't affect usability any, but it shouldn't happen, so I'll get that fixed too.

Another minor issue I noticed is that after multiple seasons, the totals RoboRunner shows and the totals RoboJogger show can be a smidge off. I would guess this is due to some rounding error somewhere. I'll investigate it.

For the look and feel for Windows, I chose Nimbus, because I think it's pretty cool. But I think Chase is right, I should default to the system look and feel. I will probably change that; however, I will probably also add a dialog for changing the look and feel as a preference (I already have a class available that does that, so it's trivial to add).

Totals not showing up in the middle of the first season is normal. A total isn't really valid (imo) until at least one battle has been run against each robot in the group or challenge. However, if totals don't show up after a full season completes, that is a problem. And not a problem I have witnessed. I'll keep an eye on this, but so far I haven't seen this problem. If you continue to see this problem and I don't, you may need to post or send a copy of your challenge file for me to test with.

Chase had another good point about removing unnecessary stuff from the robocode_template directory. It is highly unlikely someone will download this and need or want a full copy of Robocode with it. I'll trim it down to the bare minimum.

Finally, I know it's a little unnerving not having some kind of progress indication when RoboRunner is running. Note that in the Tools menu there is an option to Show RoboRunner Output. This will show what RoboRunner would normally output to the console (but with each line timestamped), though it is limited to I think the last 300 printed lines. It is the only way to see on-the-fly results right now and have a good indication of progress.

Please post any other bugs you come across. Thanks!

FYI -- I have not tested any melee at all yet, so there is a higher chance of bugs for that.

My original comment was eaten by the reply box closing when I scrolled to hit save.

So in short, I did notice the output, and did use it and that is how I know that there were no results, mid season or after it finished. One robot challenge with percent scoring.

As for progress you could just parse the output of roborunner and redisplay it in the UI in some way. But this may require another thread.

I thought about parsing the RoboRunner output, but that is not a very clean way of interfacing with RoboRunner, and Voidious already seemed willing to provide an update in the future that will provide a more robust interface for getting on-the-fly results. So I am waiting on that (@Voidious -- let me know if you want me to help on this; I have time now that I'm mostly done with RoboJogger 0.9). In the meantime, if you close all result windows for a challenge and reopen the results, you should see full results. RoboJogger reloads everything using the RoboRunner ScoreLog when the results window is opened. I'll do some more testing to see if I can cause a scenario where results are missing.

Found another problem. After "stopping" RoboRunner, the RoboRunner threads appear to continue to use CPU. Not sure why yet. But it definitely needs to be looked into.

Solved the last bug already. I didn't realize that when shutdown() is called on the thread pool, it will still finish executing any queued tasks. I just had to make a minor update to my modified version of RoboRunner to also cancel all remaining Future's after being interrupted.

A question for anyone who cares to chime in. Tonight I created most of a build script for RoboJogger. In the past I have used tools like Launch4J and IzPack to make executables and installers for Java applications for Windows. I could do this for RoboJogger, if anyone prefers. In addition to just making the source available, would you prefer: 1) A zipped archive where the main class is in a jar (unzip and run with javaw -jar robojogger.jar), or 2) A zipped archive where the main class is in an exe (unzip and run robojogger.exe), or 3) An installer that is a jar (run installer with javaw -jar robojogger-installer.jar), or 4) An installer that is an exe (just run robojogger-installer.exe). For 3) and 4), you could also indicate whether you think the main class should be a jar or exe, if that matters to you. Or I could provide it several different ways. So if you care one way or another, let me know.

Hi Mate. I'm on a mac here and i would prefer a jar in all cases. It also has to be max java 1.6 to be usable for me.

I can set up a Mac build as well. I've done that before. It will even be somewhat Macish, if you will, as I try to follow the Mac application styling guide for Macs by using the Mac menu bar and doing things like reversing the ok/cancel buttons on dialogs (I have support for that kind of stuff built into my code). For a Mac version, I can either have a zip filled with jars, or I can also create a .dmg file if preferred.

Do whatever the easiest is for you. I'm fine with .jar or .dmg. If i'm going to use it, i will probable make an .app out of it anyway. Not sure what you mean with 'somewhat Macish' :) - do you mean you have programmed it this way or just using the -Xdock flags?

In you are asking specifics, being somewhat Mac-ish to me means setting setting property "apple.laf.useScreenMenuBar" to "true" to use the screen menu bar instead of a menu bar in the Java app, setting system property "com.apple.mrj.application.apple.menu.about.name" to set the application name, using the "com.apple.eawt" classes for setting up Exit, About, and Preferences menu items, and for ok/cancel style dialogs, making the ok button appear to the right of cancel button rather than the other way around.

If you are feeling particularly ambitious, you could use launch4j to make a windows exe to launch it (or wrap it). It doesn't change anything for me (I can run it by double clicking the jar). But others might find it useful.

I think I recall a recent version of launch4j also supporting making MacOSX executables too.

Congrats on the first release! :-) I'll be sure to test it out soon on my systems and let you know how it goes.

I think it makes sense for you to just include RoboRunner in your downloads like this. It makes the setup so much easier, and savvy users could still drop in the latest RoboRunner JAR if they want (after next version). I'll try to incorporate your interrupt changes and the new results listener soon, sorry to drag my feet on that.

No problem. Instead of trying to handle InterruptedExceptions, you might just consider adding a volatile cancel flag that gets checked before you run each battle. That way, another thread can set the cancel flag to true when it wants RoboRunner to stop, and RoboRunner can shutdown more cleanly the next time it checks the cancel flag.

Also, to avoid possible contention over a score log, you might add some way to lock a score log (maybe add a ReentrantLock to control it that both RoboRunner and external threads can access). That way RoboJogger (or anything anyone else might write) can lock a score log when it reads it, unlock it when it's done, without worrying about stepping on RoboRunner trying to write to the score log at the same time.

Just some thoughts I had....

Okay, first of all. Good work! Unlike roborunner by itself. Robojogger actually seems to work. On the other hand, the results don't seem to work. Mid-season or end of running. It just never shows up as completed. Not sure what the problem is here.

It doesn't feel considerably faster then RoboResearch. But I haven't tested them head to head or anything. It might be because RoboResearch shows the progress on the UI itself (I understand how this might not be possible in RoboJogger, at least on a per turn basis).

---

Other Notes:

I notice it uses a different look and feel. Usually people expect programs to use their system look at feel. You can achieve this in java by using UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName()); This just goes to making the program feel more 'comfortable' to people who use it.

If you are releasing robocode with it, you could 'cut down' the robocode version included to reduce the size of the zip. It doesn't need a compiler, javadoc, rumble, templates, sample bots, etc. You also seem to have multiple copies of robocode in there as well. One in template, one in robocode_jars.

On the other hand, you may want to include a few default challenge files. Like say the ones RoboResearch has. This will help if someone doesn't have RoboResearch already, and/or doesn't know how to create a challenge file.

One small comment about the results dialog. I may be abnormal, but I frequently have huge test beds, like 250 bots, or 60 different sets of 9-bot melee battles. Obviously there's no nice way to display 500 columns of scores, but just making sure not to do something ridiculous (like a 5000px wide window, which I think RoboResearch's UI does) would be nice. :-)

It will be in a scroll pane in a window that has a max size limit on it. Beyond that, do you think there is a better way to show results when there is a huge number of bots?

Not really, that seems good. At that point you're probably just interested in overall score. But on that note, if "Total" was in some fixed place instead of requiring me to scroll way to the right, that would be nice. :-)

Noted. Check out the updated screenshot I posted. I will probably have the preferred size set to something like 800 pixels wide for the center scroll pane. In the screenshot it is set to a somewhat small 400 pixels wide just to make testing easier.

Sorry I missed replying to this, but the updated screenshot looks perfect!

Something else I'm working on is providing a way to interrupt RoboRunner in the middle of a challenge. I'm not sure in what ways that could potentially mess up RoboRunner yet, but I did have to make a couple of changes to make this work:

First, in order to stop RoboRunner completely (and not just the current battle), I had to make an InterruptedException result in the bypass of all queued battles:

In BattleRunner:

  private void getAllFutures(List<Future<String>> futures) {
    for (Future<String> future : futures) {
      try {
        future.get();
      } catch (InterruptedException e) {
        e.printStackTrace();
        return;
      } catch (ExecutionException e) {
        e.printStackTrace();
      }
    }
  }

There might be some additional modification, but for now, I just added a return statement if an InterruptedException occurs (will probably also get rid of the printStackTrace call). This prevents calling get() on all remaining Futures. While I think this was an unexpected condition in RoboRunner, in RoboJogger an InterruptedException is now an expected result whenever a stop command is issued for RoboRunner. A remaining question is, what, if anything, will be broken as a result of this?

Another change I made to ScoreLog, such that trying to access battle results for a "botList" that does not exist will not cause a NullPointerException:

In ScoreLog:

  public List<BattleScore> getBattleScores(String botList) {
    List<BattleScore> scores = _scores.get(botList);
    return (scores == null)? null : ImmutableList.copyOf(_scores.get(botList));
  }

I provided the null check on scores. I was kind of surprised that ImmutableList didn't do that by design. The most likely scenario where this happens is related to my other change -- if a challenge is interrupted before battles have been run against all opponents, when I later access results from the ScoreLog, I am not aware of missing results until the getBattleScores method returns null. I suppose I could have also just added a try/catch in my own code for NullPointerException without having to change RoboRunner, but I felt doing so was not the better way of handling it.

These changes are not finalized. I'm just writing about them for the sake of discussion.

I just noticed that for getting battle results, there is a method hasBotList(String) method that I could call before trying to get battle scores. This would prevent the NPE without modifying RoboRunner. Given this, I could see arguing either way about whether getBattleScores should throw NPE or return null for a botList that does not exist.

I don't have a strong opinion about NPE vs returning null - I think the hasBotList is what made me feel ok with leaving the other one NPE-ing, but that doesn't mean it has to. Seems silly to insist you call 2 methods instead of 1.

I'll have to think about the interruption stuff. Are you using the same RoboRunner instance after interrupting and trying to use it again? Certainly that would give me pause and I'd want to look over RoboRunner and BattleRunner to see what internal state might be confused by this. If not, my only worry would be if the interruption came during a file write to the score log. Maybe in the file writing, we need to catch InterruptedException, close the file stream in the catch, and rethrow? I'm not really sure. Maybe Java is already smart enough not to corrupt a file stream when being interrupted? The code you have here makes sense and doesn't raise any red flags besides that.

No -- I create a new RoboRunner instance for each challenge started. If RoboRunner is interrupted in the middle of a challenge, when RoboRunner is restarted, a new RoboRunner instance is created. Good point on the potential for RoboRunner to be writing to the score log when interrupted; I do need to take a closer look at that.

Instead of dealing with interrupted exceptions, robo runner could just provide a cancel flag that gets checked before each get().

@Voidious -- I'm not sure what your plan for confidence is, but I eagerly went ahead and developed my own confidence calculator. I was looking over your code for calculating confidence and was having trouble following it, so I instead went to my wife's Principles of Biostatistics book and read the chapter on Confidence Intervals. For the sake of simplicity, I will stick with 95% confidence intervals, as that is what you used in your code (that's where the 1.96 comes from) and it seems reasonable. The confidence interval for a single robot turns out to be pretty simple to calculate (in special-character-challenged terms, it is x +- 1.96 * s / sqrt(n), where x is the mean, s is the standard deviation, and n is the sample size). Where it gets more complicated is in calculating the confidence interval for groups and the overall total score.

Lets talk groups first. What I did for a group was to take the first score for each opponent, average them all, and that becomes data point 1. Then take the second score for each opponent, average them, and that becomes data point 2. I determine how many data points to use by calculating the average number of battles for an opponent in the group, rounded. This means some data points for opponents with more scores end up getting thrown away, and some data points for opponents with fewer scores don't have enough scores. For the latter, I use as many extra randomly generated scores as I need where the random score falls within the confidence interval of scores for that particular robot. Once I have all of the data points, I then use the original means for calculating a confidence interval on the collected data points.

Now for the overall total. If there is only 1 group (or no groups, depending on how you look at it), then there is nothing more to do -- use the values calculated for the 1 group. But if there are multiple groups, then what? We should probably respect that the overall total is an average of the group totals. This would end up being just like calculating the group confidence intervals, only treating the groups like the robots.

Did that make sense? How is this different from what what you have done in RoboRunner?

Heh, well, what I did is a little complicated, but I think it's about the best you can do for a set of bots that each have their own distributions. Basically I run 1,000 or whatever random simulations of the overall score, based on the averages / standard deviations of each individual bot's score distribution. Then I can take those "overall score" samples, supposedly generated from the same distribution as the real scores, and use them as additional samples to calculate the confidence interval of the overall score. It's a fairly basic Monte Carlo method.

I see there was a discussion about it on the RoboRunner page. I should probably go read that. Never heard of the Monte Carlo method, so I'll look into it.

I'd heard the term, but it was totally Skilgannon that knew enough to suggest it. Once I looked into it, though, it was pretty simple.

But I also wanted to mention, I was planning to pass some object with all the confidence interval info you might need about the current battle in the new listener. I figured that was among the things you'd want in the application output, since it's among the things I print in the console version. But of course you're free to use whatever you like. :-)

I'll use it if it's there. I use the ScoreLog to show data from past battles, and wasn't sure if confidence information would also be available from the ScoreLog after your updates. If not, I can keep using my own confidence calculator for past data.

Hmm. Well first off, I am pretty sure you should make sure you are using the [t-distribution], not the normal distribution. Using that, I would generate a confidence interval for each individual bot. I am nearly certain that there is a way to generate a confidence interval from the mean of several other intervals. I can't remember off the top of my head but I vaguely recall it being something like the square root of the sum of the squares of the standard errors (not standard deviations since the sample size is presumably fairly small). I'll tell you if I can find it.

http://www.hilemansblog.com/?tag=root-sum-of-squares and https://www.westgard.com/lesson35.htm#6

I didn't read through them carefully (kind of busy with school), but skimming through them quickly, it appears that the square root of the sum of the variances of the individual distributions is correct.

I think that's correct if all of them have the same number of samples. However, with the cool new 'variance minimizer' pairings selection algorithm that isn't necessarily guaranteed. Although you may be right - could you see if your Monte Carlo gives the same results as a root-sum-of-squares, Voidious?

I'm finally at the point where I am trying to actually launch RoboRunner. Currently running into an error I will have to debug. Posting part of the stack trace here in case anyone wants to comment.

Copying missing bots... 0 JAR copies done!
Initializing engine: robocodes\z1... Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException
	at robowiki.runner.BattleRunner.initEngine(BattleRunner.java:66)
	at robowiki.runner.BattleRunner.<init>(BattleRunner.java:42)
	at robowiki.runner.RoboRunner.<init>(RoboRunner.java:172)
	at org.xandercat.roborunner.runner.RoboRunnerService.startRunner(RoboRunnerService.java:44)
	at org.xandercat.roborunner.runner.action.LaunchRoboRunnerAction.actionPerformed(LaunchRoboRunnerAction.java:46)
	at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)

And the chunk of relevant code from RoboRunner:

      System.out.print("Initializing engine: " + enginePath + "... ");
      ProcessBuilder builder = new ProcessBuilder(command);
      builder.redirectErrorStream(true);
      Process battleProcess = builder.start();
      BufferedReader reader = new BufferedReader(
          new InputStreamReader(battleProcess.getInputStream()));
      String processOutput;
      do {
        processOutput = reader.readLine();
      } while (!processOutput.equals(BattleProcess.READY_SIGNAL));
      System.out.println("done!");
      _processQueue.add(battleProcess);

Presumably, the input stream never provided the BattleProcess.READY_SIGNAL. I'll have to do some digging to figure out why. I'm not entirely clear on what the RoboRunner requirements are, but at the moment I am running it under Java 6 with Robocode 1.7.3.0.

FYI -- line 66 is the while part of the do/while loop.

I'll take a deeper look later when I'm home. At a glance, it seems like processOutput is coming up null - maybe the condition should be "processOutput != null && ...". What command are you using to launch this?

Looks like the problem was I didn't have one of the needed Robocode jars in the classpath. Thanks for including source in the RoboRunner jar; that made debugging easier.

Fixing the classpath fixed the problem I was having. Also, I am running RoboRunner via new RoboRunner(...) and then calling the runBattles() method. I need to dig a little deeper to determine how best to extract the battle results; at the moment it is just letting RoboRunner barf them on System.out. :-)

One thing is for sure -- it runs oodles faster than RoboResearch. I'm definitely switching. I suppose it may have been possible to branch RoboResearch to run battles a la RoboRunner, but I'm having fun building a new UI, so I'm continuing on.

Cool, good to hear! I don't think I kept any real test results of speed vs RoboResearch, but I think it was in the range of 20% less time for my bot / system. The smart battles stuff helps too, but it's hard to measure.

Similarly, I had long wanted to update RoboResearch to use the control API instead of launching external Java processes. When I started digging into it, it just looked easier / better / more fun to rewrite from scratch.