Talk:RoboJogger

A question for anyone who cares to chime in. Tonight I created most of a build script for RoboJogger. In the past I have used tools like Launch4J and IzPack to make executables and installers for Java applications for Windows. I could do this for RoboJogger, if anyone prefers. In addition to just making the source available, would you prefer: 1) A zipped archive where the main class is in a jar (unzip and run with javaw -jar robojogger.jar), or 2) A zipped archive where the main class is in an exe (unzip and run robojogger.exe), or 3) An installer that is a jar (run installer with javaw -jar robojogger-installer.jar), or 4) An installer that is an exe (just run robojogger-installer.exe). For 3) and 4), you could also indicate whether you think the main class should be a jar or exe, if that matters to you. Or I could provide it several different ways. So if you care one way or another, let me know.

Hi Mate. I'm on a mac here and i would prefer a jar in all cases. It also has to be max java 1.6 to be usable for me.

I can set up a Mac build as well. I've done that before. It will even be somewhat Macish, if you will, as I try to follow the Mac application styling guide for Macs by using the Mac menu bar and doing things like reversing the ok/cancel buttons on dialogs (I have support for that kind of stuff built into my code). For a Mac version, I can either have a zip filled with jars, or I can also create a .dmg file if preferred.

Do whatever the easiest is for you. I'm fine with .jar or .dmg. If i'm going to use it, i will probable make an .app out of it anyway. Not sure what you mean with 'somewhat Macish' :) - do you mean you have programmed it this way or just using the -Xdock flags?

In you are asking specifics, being somewhat Mac-ish to me means setting setting property "apple.laf.useScreenMenuBar" to "true" to use the screen menu bar instead of a menu bar in the Java app, setting system property "com.apple.mrj.application.apple.menu.about.name" to set the application name, using the "com.apple.eawt" classes for setting up Exit, About, and Preferences menu items, and for ok/cancel style dialogs, making the ok button appear to the right of cancel button rather than the other way around.

If you are feeling particularly ambitious, you could use launch4j to make a windows exe to launch it (or wrap it). It doesn't change anything for me (I can run it by double clicking the jar). But others might find it useful.

I think I recall a recent version of launch4j also supporting making MacOSX executables too.

Congrats on the first release! :-) I'll be sure to test it out soon on my systems and let you know how it goes.

I think it makes sense for you to just include RoboRunner in your downloads like this. It makes the setup so much easier, and savvy users could still drop in the latest RoboRunner JAR if they want (after next version). I'll try to incorporate your interrupt changes and the new results listener soon, sorry to drag my feet on that.

No problem. Instead of trying to handle InterruptedExceptions, you might just consider adding a volatile cancel flag that gets checked before you run each battle. That way, another thread can set the cancel flag to true when it wants RoboRunner to stop, and RoboRunner can shutdown more cleanly the next time it checks the cancel flag.

Also, to avoid possible contention over a score log, you might add some way to lock a score log (maybe add a ReentrantLock to control it that both RoboRunner and external threads can access). That way RoboJogger (or anything anyone else might write) can lock a score log when it reads it, unlock it when it's done, without worrying about stepping on RoboRunner trying to write to the score log at the same time.

Just some thoughts I had....

Okay, first of all. Good work! Unlike roborunner by itself. Robojogger actually seems to work. On the other hand, the results don't seem to work. Mid-season or end of running. It just never shows up as completed. Not sure what the problem is here.

It doesn't feel considerably faster then RoboResearch. But I haven't tested them head to head or anything. It might be because RoboResearch shows the progress on the UI itself (I understand how this might not be possible in RoboJogger, at least on a per turn basis).

---

Other Notes:

I notice it uses a different look and feel. Usually people expect programs to use their system look at feel. You can achieve this in java by using UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName()); This just goes to making the program feel more 'comfortable' to people who use it.

If you are releasing robocode with it, you could 'cut down' the robocode version included to reduce the size of the zip. It doesn't need a compiler, javadoc, rumble, templates, sample bots, etc. You also seem to have multiple copies of robocode in there as well. One in template, one in robocode_jars.

On the other hand, you may want to include a few default challenge files. Like say the ones RoboResearch has. This will help if someone doesn't have RoboResearch already, and/or doesn't know how to create a challenge file.

One small comment about the results dialog. I may be abnormal, but I frequently have huge test beds, like 250 bots, or 60 different sets of 9-bot melee battles. Obviously there's no nice way to display 500 columns of scores, but just making sure not to do something ridiculous (like a 5000px wide window, which I think RoboResearch's UI does) would be nice. :-)

It will be in a scroll pane in a window that has a max size limit on it. Beyond that, do you think there is a better way to show results when there is a huge number of bots?

Not really, that seems good. At that point you're probably just interested in overall score. But on that note, if "Total" was in some fixed place instead of requiring me to scroll way to the right, that would be nice. :-)

Noted. Check out the updated screenshot I posted. I will probably have the preferred size set to something like 800 pixels wide for the center scroll pane. In the screenshot it is set to a somewhat small 400 pixels wide just to make testing easier.

Sorry I missed replying to this, but the updated screenshot looks perfect!

Something else I'm working on is providing a way to interrupt RoboRunner in the middle of a challenge. I'm not sure in what ways that could potentially mess up RoboRunner yet, but I did have to make a couple of changes to make this work:

First, in order to stop RoboRunner completely (and not just the current battle), I had to make an InterruptedException result in the bypass of all queued battles:

In BattleRunner:

  private void getAllFutures(List<Future<String>> futures) {
    for (Future<String> future : futures) {
      try {
        future.get();
      } catch (InterruptedException e) {
        e.printStackTrace();
        return;
      } catch (ExecutionException e) {
        e.printStackTrace();
      }
    }
  }

There might be some additional modification, but for now, I just added a return statement if an InterruptedException occurs (will probably also get rid of the printStackTrace call). This prevents calling get() on all remaining Futures. While I think this was an unexpected condition in RoboRunner, in RoboJogger an InterruptedException is now an expected result whenever a stop command is issued for RoboRunner. A remaining question is, what, if anything, will be broken as a result of this?

Another change I made to ScoreLog, such that trying to access battle results for a "botList" that does not exist will not cause a NullPointerException:

In ScoreLog:

  public List<BattleScore> getBattleScores(String botList) {
    List<BattleScore> scores = _scores.get(botList);
    return (scores == null)? null : ImmutableList.copyOf(_scores.get(botList));
  }

I provided the null check on scores. I was kind of surprised that ImmutableList didn't do that by design. The most likely scenario where this happens is related to my other change -- if a challenge is interrupted before battles have been run against all opponents, when I later access results from the ScoreLog, I am not aware of missing results until the getBattleScores method returns null. I suppose I could have also just added a try/catch in my own code for NullPointerException without having to change RoboRunner, but I felt doing so was not the better way of handling it.

These changes are not finalized. I'm just writing about them for the sake of discussion.

I just noticed that for getting battle results, there is a method hasBotList(String) method that I could call before trying to get battle scores. This would prevent the NPE without modifying RoboRunner. Given this, I could see arguing either way about whether getBattleScores should throw NPE or return null for a botList that does not exist.

I don't have a strong opinion about NPE vs returning null - I think the hasBotList is what made me feel ok with leaving the other one NPE-ing, but that doesn't mean it has to. Seems silly to insist you call 2 methods instead of 1.

I'll have to think about the interruption stuff. Are you using the same RoboRunner instance after interrupting and trying to use it again? Certainly that would give me pause and I'd want to look over RoboRunner and BattleRunner to see what internal state might be confused by this. If not, my only worry would be if the interruption came during a file write to the score log. Maybe in the file writing, we need to catch InterruptedException, close the file stream in the catch, and rethrow? I'm not really sure. Maybe Java is already smart enough not to corrupt a file stream when being interrupted? The code you have here makes sense and doesn't raise any red flags besides that.

No -- I create a new RoboRunner instance for each challenge started. If RoboRunner is interrupted in the middle of a challenge, when RoboRunner is restarted, a new RoboRunner instance is created. Good point on the potential for RoboRunner to be writing to the score log when interrupted; I do need to take a closer look at that.

Instead of dealing with interrupted exceptions, robo runner could just provide a cancel flag that gets checked before each get().

@Voidious -- I'm not sure what your plan for confidence is, but I eagerly went ahead and developed my own confidence calculator. I was looking over your code for calculating confidence and was having trouble following it, so I instead went to my wife's Principles of Biostatistics book and read the chapter on Confidence Intervals. For the sake of simplicity, I will stick with 95% confidence intervals, as that is what you used in your code (that's where the 1.96 comes from) and it seems reasonable. The confidence interval for a single robot turns out to be pretty simple to calculate (in special-character-challenged terms, it is x +- 1.96 * s / sqrt(n), where x is the mean, s is the standard deviation, and n is the sample size). Where it gets more complicated is in calculating the confidence interval for groups and the overall total score.

Lets talk groups first. What I did for a group was to take the first score for each opponent, average them all, and that becomes data point 1. Then take the second score for each opponent, average them, and that becomes data point 2. I determine how many data points to use by calculating the average number of battles for an opponent in the group, rounded. This means some data points for opponents with more scores end up getting thrown away, and some data points for opponents with fewer scores don't have enough scores. For the latter, I use as many extra randomly generated scores as I need where the random score falls within the confidence interval of scores for that particular robot. Once I have all of the data points, I then use the original means for calculating a confidence interval on the collected data points.

Now for the overall total. If there is only 1 group (or no groups, depending on how you look at it), then there is nothing more to do -- use the values calculated for the 1 group. But if there are multiple groups, then what? We should probably respect that the overall total is an average of the group totals. This would end up being just like calculating the group confidence intervals, only treating the groups like the robots.

Did that make sense? How is this different from what what you have done in RoboRunner?

Heh, well, what I did is a little complicated, but I think it's about the best you can do for a set of bots that each have their own distributions. Basically I run 1,000 or whatever random simulations of the overall score, based on the averages / standard deviations of each individual bot's score distribution. Then I can take those "overall score" samples, supposedly generated from the same distribution as the real scores, and use them as additional samples to calculate the confidence interval of the overall score. It's a fairly basic Monte Carlo method.

I see there was a discussion about it on the RoboRunner page. I should probably go read that. Never heard of the Monte Carlo method, so I'll look into it.

I'd heard the term, but it was totally Skilgannon that knew enough to suggest it. Once I looked into it, though, it was pretty simple.

But I also wanted to mention, I was planning to pass some object with all the confidence interval info you might need about the current battle in the new listener. I figured that was among the things you'd want in the application output, since it's among the things I print in the console version. But of course you're free to use whatever you like. :-)

I'll use it if it's there. I use the ScoreLog to show data from past battles, and wasn't sure if confidence information would also be available from the ScoreLog after your updates. If not, I can keep using my own confidence calculator for past data.

Hmm. Well first off, I am pretty sure you should make sure you are using the [t-distribution], not the normal distribution. Using that, I would generate a confidence interval for each individual bot. I am nearly certain that there is a way to generate a confidence interval from the mean of several other intervals. I can't remember off the top of my head but I vaguely recall it being something like the square root of the sum of the squares of the standard errors (not standard deviations since the sample size is presumably fairly small). I'll tell you if I can find it.

http://www.hilemansblog.com/?tag=root-sum-of-squares and https://www.westgard.com/lesson35.htm#6

I didn't read through them carefully (kind of busy with school), but skimming through them quickly, it appears that the square root of the sum of the variances of the individual distributions is correct.

I think that's correct if all of them have the same number of samples. However, with the cool new 'variance minimizer' pairings selection algorithm that isn't necessarily guaranteed. Although you may be right - could you see if your Monte Carlo gives the same results as a root-sum-of-squares, Voidious?

I'm finally at the point where I am trying to actually launch RoboRunner. Currently running into an error I will have to debug. Posting part of the stack trace here in case anyone wants to comment.

Copying missing bots... 0 JAR copies done!
Initializing engine: robocodes\z1... Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException
	at robowiki.runner.BattleRunner.initEngine(BattleRunner.java:66)
	at robowiki.runner.BattleRunner.<init>(BattleRunner.java:42)
	at robowiki.runner.RoboRunner.<init>(RoboRunner.java:172)
	at org.xandercat.roborunner.runner.RoboRunnerService.startRunner(RoboRunnerService.java:44)
	at org.xandercat.roborunner.runner.action.LaunchRoboRunnerAction.actionPerformed(LaunchRoboRunnerAction.java:46)
	at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)

And the chunk of relevant code from RoboRunner:

      System.out.print("Initializing engine: " + enginePath + "... ");
      ProcessBuilder builder = new ProcessBuilder(command);
      builder.redirectErrorStream(true);
      Process battleProcess = builder.start();
      BufferedReader reader = new BufferedReader(
          new InputStreamReader(battleProcess.getInputStream()));
      String processOutput;
      do {
        processOutput = reader.readLine();
      } while (!processOutput.equals(BattleProcess.READY_SIGNAL));
      System.out.println("done!");
      _processQueue.add(battleProcess);

Presumably, the input stream never provided the BattleProcess.READY_SIGNAL. I'll have to do some digging to figure out why. I'm not entirely clear on what the RoboRunner requirements are, but at the moment I am running it under Java 6 with Robocode 1.7.3.0.

FYI -- line 66 is the while part of the do/while loop.

I'll take a deeper look later when I'm home. At a glance, it seems like processOutput is coming up null - maybe the condition should be "processOutput != null && ...". What command are you using to launch this?

Looks like the problem was I didn't have one of the needed Robocode jars in the classpath. Thanks for including source in the RoboRunner jar; that made debugging easier.

Fixing the classpath fixed the problem I was having. Also, I am running RoboRunner via new RoboRunner(...) and then calling the runBattles() method. I need to dig a little deeper to determine how best to extract the battle results; at the moment it is just letting RoboRunner barf them on System.out. :-)

One thing is for sure -- it runs oodles faster than RoboResearch. I'm definitely switching. I suppose it may have been possible to branch RoboResearch to run battles a la RoboRunner, but I'm having fun building a new UI, so I'm continuing on.

Cool, good to hear! I don't think I kept any real test results of speed vs RoboResearch, but I think it was in the range of 20% less time for my bot / system. The smart battles stuff helps too, but it's hard to measure.

Similarly, I had long wanted to update RoboResearch to use the control API instead of launching external Java processes. When I started digging into it, it just looked easier / better / more fun to rewrite from scratch.