Talk:RoboRunner

Quoted from APS:

The server calculates APS for each bot by:

1. taking the average percentage score of all battles against each opponent separately to get an APS for each pairing,
2. then averaging all pairing scores to obtain the final average.

Formally, it's

mean(challenger_score / total_score)

for each pairing.

LiteRumble seems to follow this algorithm.

However, RoboRunner is using:

sum(challenger_score) / sum(total_score)

instead for each pairing.

This causes scores calculated by RoboRunner to be different from LiteRumble.

Here is a Python script that calculates APS correctly. Use this script when you want to align with LiteRumble.

import os
import gzip
import xml.etree.ElementTree as ET

from statistics import mean
from collections import defaultdict


roborunner_dir = os.path.expanduser('~/roborunner')  # replace with your roborunner directory


def get_aps_dict(bot):
  scores = ET.parse(gzip.open(f'{roborunner_dir}/data/{bot}.xml.gz', 'r')).getroot()

  aps_raw = defaultdict(list)

  for bot_list in scores:
    for battle in bot_list:
      battle_scores = {}
      total_score = 0

      for robot_score in battle.findall('robot_score'):
        name = robot_score.find('name').text
        score = int(robot_score.find('score').text)
        battle_scores[name] = score
        total_score += score

      aps = 100 * battle_scores[bot] / total_score

      for name, _ in battle_scores.items():
        if name != bot:
          aps_raw[name].append(aps)

  return dict((name, mean(aps)) for name, aps in aps_raw.items())

Update: The APS fix is now included in the newest release of my fork. A PR is also made to Voidious’s version.

Did this fix also apply to melee score calculation? I've been having difficulty getting my offline results with roborunner to align with literumble for the nano melee rumble.

Yes, the definition of APS applies to all of the rumbles.

I think I've solved this now, it looks like the problem was with the way I was generating battles.

I wrote a custom GUI for RoboRunner, it directly interfaces with a modified RoboRunner. It is in early early alpha. The feature set is nowhere near complete. Please let me know of any bugs.

Robots are not automatically copied to robocode directories. Nothing is saved. Paths cannot (at the moment) be altered. Results do no update automatically, they need to be closed and reopened to update them (I plan to fix this). Thread count cannot be changed after starting (for now, changing this would be involved).

https://github.com/Chase-san/RoboRunner-GUI/releases

I guess I should mention its advantages over RoboJogger, the other GUI for RoboRunner, also over RoboRunner itself.

It has a queue that runs challenges in whatever order they happen to be in the queue. This means once you start it, you can run multiple challenges without robocode needing to restart, and you can also reorder the queue after starting the battle threads.

It is better to consider it's limitations compared to RoboResearch. Aside from the first post, you cannot start and stop threads individually. I may support this eventually, but at the moment it would require rewriting RoboRunner's BattleRunner class. Which also means once you start the threads, you cannot alter the number of threads running.

Congrats on the first release! I'll try to have a look tonight or tomorrow.

Do we need any setup instructions? Or are the roborunner.properties and empty Robocode installs already in the zip?

I am working on automating the runner creation and robot jar copying process today. As well as adding an options dialog to configure the paths and jvm arguments.

The GUI won't use the properties as the standard roborunner. But the current version doesn't use any at all.

But for the current version, you need to create the robocodes/r[0-9]+ directories yourself (or use the script you wrote). You can pick the thread count in the program itself (but only up to the number of runners you created). Make sure to copy the robots into robocodes/r[0-9]+/robots directories as well.

v0.9.2-Alpha

Adding options (that save), automatic robocode runner creation (based on threads), and automatic robot jar copying.

I'm curious what kind of speed improvements people have seen, if anyone keeps track. Today, in the midst of developing my own UI for RoboRunner, I decided to do a test run to compare RoboRunner vs RoboResearch. I ran MC2K7 Fast Learning challenge against XanderCat 11.6. All data was cleared from the robots directory in both tests. 2 Threads in both cases. RoboResearch ran in 5:30. RoboRunner ran in 3:50. That's about a 30% speed improvement -- pretty dramatic. With dirty data directories where there is a lot of past data, I bet the speed difference would be an order of magnitude more significant. RoboResearch can get pretty slow starting up battles when you haven't cleared out the data for awhile. Results will vary based on the data directories and what kind of robots are being run, but regardless, I think anyone using RoboRunner will see a noticeable speed boost. Very nice.

RoboResearch has an annoying problem when bots print something in the console. The clients crash and the threads become stuck in paused mode. You have to recreate the clients manually. So, RoboResearch is a lot more than 30% slower.

I actually hopefully just fixed the problem about the threads pausing. I still use RoboResearch since getting RoboJogger/RoboRunner to cooperate is difficult at times.

If possible I wouldn't mind hammering RoboRunner into RoboResearch, since aside from a few bugs it works rather well.

How should we be computing the number of completed seasons with smart battles?

When scores are loaded from the ScoreLog, the bullet damage score (which the BULLET_DAMAGE score type relies on) is always 0 for the opponent. Is this a bug in RoboRunner -- either the scores not getting saved correctly or loaded correctly by the ScoreLog? I also noticed that the energy conserved values were also 0 for both the challenger and opponent, so may be a related bug there too.

Okay, this wasn't a bug. It was because it was an opponent who wasn't firing bullets.

Please provide a simple UI for this. All attempts I have made to run this on windows have met with extreme aversion and then failure.

I got it setup, installed, and finally detecting the correct classes (multiple reasons why this was failing). But then it constantly complained I was not defining the robot to run, the challenge or so forth. I was do all of those, in the exact same format presented in the help examples. I made sure. In the meantime I have switched back to RoboResearch, which works.

It doesn't have to be a comprehensive UI. A simple program that runs the console command for (to remove human error) would be perfectly acceptable. Probably two file browse boxes and a number spinner for seasons.

That way we know its a problem with the program if it doesn't work.

I could probably build a Java UI to launch it. Skotty

I'm now working on a Java user interface for RoboRunner that I call the RoboJogger UI for RoboRunner (or just RoboJogger). I have past experience writing Java Swing applications and also prefer having a UI rather than just a command line tool, so this is a good project for me. I don't have nearly as much free time as I would like, so it may be a few weeks before I have it up and running, but I can keep anyone who is interested up to date on my progress. Skotty

Would be very cool to see a nice UI, please do keep us up to date. :-) I think the main thing I'd want to make sure a UI could do is to queue up multiple runs at a time, which is easy to overlook and comes for free (with shell scripts) with the command line version.

When I was considering a UI, my main idea was to make it a web interface. In part because that seems simple, portable, and like something I know how to do, and also because it would offer remote monitoring and access for free. I know I like to check on/alter long running tests while I'm out sometimes. But others may not be as OCD as me and prefer a more native UI.

A WebUI doesn't actually seem like that bad of idea actually.

Just peeked at RoboRunner distribution package a while ago. It would be nice if initial setup was made inside java code instead of a shell script. So Windows users don't have to port the scripts.

Argh, sorry about that. When I wrote the setup, it made a lot of sense to use a shell script instead of spending 10x the time to write it in Java, since I wasn't even sure anyone else would ever use it. But at this point Windows setup support is probably the most glaring omission. A batch file might work, but I don't have Windows to test on so I guess I should go with Java.

I have Windows, but wasn´t willing to port de scripts. :P

Laziness combined with a working RoboResearch setup makes everything harder.

Does RoboRunner support team battles (1200x1200 battlefield)? Or does it support custom battlefield sizes?

Trying RoboResearch I noticed battlefield sizes are hard-coded at 800x600 or 1000x1000, and there is no place to configure battlefield sizes in .rrc files.

Yep, both work fine. To modify battle field size, just add width and height on their own lines after the "num rounds" line in the challenge file. And if your team JARs are in the bots/ dir, just specify them like you would a bot in the challenge file. So e.g.:

My Teams Test Bed
PERCENT_SCORE
35
1200
1200

abc.ShadowTeam 3.83
gimp.GimpTeam 0.1

Ouch, I may have spoken too soon. I'm seeing an exception when I try to run a team battle right now. I'll try and see if I can figure that out, I don't think I needed to do anything special originally to support teams.

Ok, figured it out. Turns out Robocode is a bit confused between getRobotNameAndVersion() vs getTeamLeaderName() in results of a team battle. So I need to use getTeamLeaderName() instead (and for non-teams they return the same). I'll post a fix now.

Ok, posted 1.2.3 with a fix.