View source for Talk:RoboJogger
- [View source↑]
- [History↑]
Contents
Thread title | Replies | Last modified |
---|---|---|
Calculating Confidence | 7 | 05:49, 13 December 2012 |
Problem Running RoboRunner | 6 | 23:16, 5 December 2012 |
![]() First page |
![]() Previous page |
![]() Next page |
![]() Last page |
You do not have permission to edit this page, for the following reasons:
You can view and copy the source of this page.
Heh, well, what I did is a little complicated, but I think it's about the best you can do for a set of bots that each have their own distributions. Basically I run 1,000 or whatever random simulations of the overall score, based on the averages / standard deviations of each individual bot's score distribution. Then I can take those "overall score" samples, supposedly generated from the same distribution as the real scores, and use them as additional samples to calculate the confidence interval of the overall score. It's a fairly basic Monte Carlo method.
I see there was a discussion about it on the RoboRunner page. I should probably go read that. Never heard of the Monte Carlo method, so I'll look into it.
I'd heard the term, but it was totally Skilgannon that knew enough to suggest it. Once I looked into it, though, it was pretty simple.
But I also wanted to mention, I was planning to pass some object with all the confidence interval info you might need about the current battle in the new listener. I figured that was among the things you'd want in the application output, since it's among the things I print in the console version. But of course you're free to use whatever you like. :-)
I'll use it if it's there. I use the ScoreLog to show data from past battles, and wasn't sure if confidence information would also be available from the ScoreLog after your updates. If not, I can keep using my own confidence calculator for past data.
Hmm. Well first off, I am pretty sure you should make sure you are using the [t-distribution], not the normal distribution. Using that, I would generate a confidence interval for each individual bot. I am nearly certain that there is a way to generate a confidence interval from the mean of several other intervals. I can't remember off the top of my head but I vaguely recall it being something like the square root of the sum of the squares of the standard errors (not standard deviations since the sample size is presumably fairly small). I'll tell you if I can find it.
http://www.hilemansblog.com/?tag=root-sum-of-squares and https://www.westgard.com/lesson35.htm#6
I didn't read through them carefully (kind of busy with school), but skimming through them quickly, it appears that the square root of the sum of the variances of the individual distributions is correct.
I think that's correct if all of them have the same number of samples. However, with the cool new 'variance minimizer' pairings selection algorithm that isn't necessarily guaranteed. Although you may be right - could you see if your Monte Carlo gives the same results as a root-sum-of-squares, Voidious?
I'm finally at the point where I am trying to actually launch RoboRunner. Currently running into an error I will have to debug. Posting part of the stack trace here in case anyone wants to comment.
Copying missing bots... 0 JAR copies done! Initializing engine: robocodes\z1... Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException at robowiki.runner.BattleRunner.initEngine(BattleRunner.java:66) at robowiki.runner.BattleRunner.<init>(BattleRunner.java:42) at robowiki.runner.RoboRunner.<init>(RoboRunner.java:172) at org.xandercat.roborunner.runner.RoboRunnerService.startRunner(RoboRunnerService.java:44) at org.xandercat.roborunner.runner.action.LaunchRoboRunnerAction.actionPerformed(LaunchRoboRunnerAction.java:46) at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
And the chunk of relevant code from RoboRunner:
System.out.print("Initializing engine: " + enginePath + "... "); ProcessBuilder builder = new ProcessBuilder(command); builder.redirectErrorStream(true); Process battleProcess = builder.start(); BufferedReader reader = new BufferedReader( new InputStreamReader(battleProcess.getInputStream())); String processOutput; do { processOutput = reader.readLine(); } while (!processOutput.equals(BattleProcess.READY_SIGNAL)); System.out.println("done!"); _processQueue.add(battleProcess);
Presumably, the input stream never provided the BattleProcess.READY_SIGNAL. I'll have to do some digging to figure out why. I'm not entirely clear on what the RoboRunner requirements are, but at the moment I am running it under Java 6 with Robocode 1.7.3.0.
I'll take a deeper look later when I'm home. At a glance, it seems like processOutput is coming up null - maybe the condition should be "processOutput != null && ...". What command are you using to launch this?
Looks like the problem was I didn't have one of the needed Robocode jars in the classpath. Thanks for including source in the RoboRunner jar; that made debugging easier.
Fixing the classpath fixed the problem I was having. Also, I am running RoboRunner via new RoboRunner(...) and then calling the runBattles() method. I need to dig a little deeper to determine how best to extract the battle results; at the moment it is just letting RoboRunner barf them on System.out. :-)
One thing is for sure -- it runs oodles faster than RoboResearch. I'm definitely switching. I suppose it may have been possible to branch RoboResearch to run battles a la RoboRunner, but I'm having fun building a new UI, so I'm continuing on.
Cool, good to hear! I don't think I kept any real test results of speed vs RoboResearch, but I think it was in the range of 20% less time for my bot / system. The smart battles stuff helps too, but it's hard to measure.
Similarly, I had long wanted to update RoboResearch to use the control API instead of launching external Java processes. When I started digging into it, it just looked easier / better / more fun to rewrite from scratch.
![]() First page |
![]() Previous page |
![]() Next page |
![]() Last page |