Talk:RoboRunner

Quoted from APS:

The server calculates APS for each bot by:

1. taking the average percentage score of all battles against each opponent separately to get an APS for each pairing,
2. then averaging all pairing scores to obtain the final average.

Formally, it's

mean(challenger_score / total_score)

for each pairing.

LiteRumble seems to follow this algorithm.

However, RoboRunner is using:

sum(challenger_score) / sum(total_score)

instead for each pairing.

This causes scores calculated by RoboRunner to be different from LiteRumble.

Here is a Python script that calculates APS correctly. Use this script when you want to align with LiteRumble.

import os
import gzip
import xml.etree.ElementTree as ET

from statistics import mean
from collections import defaultdict


roborunner_dir = os.path.expanduser('~/roborunner')  # replace with your roborunner directory


def get_aps_dict(bot):
  scores = ET.parse(gzip.open(f'{roborunner_dir}/data/{bot}.xml.gz', 'r')).getroot()

  aps_raw = defaultdict(list)

  for bot_list in scores:
    for battle in bot_list:
      battle_scores = {}
      total_score = 0

      for robot_score in battle.findall('robot_score'):
        name = robot_score.find('name').text
        score = int(robot_score.find('score').text)
        battle_scores[name] = score
        total_score += score

      aps = 100 * battle_scores[bot] / total_score

      for name, _ in battle_scores.items():
        if name != bot:
          aps_raw[name].append(aps)

  return dict((name, mean(aps)) for name, aps in aps_raw.items())

Update: The APS fix is now included in the newest release of my fork. A PR is also made to Voidious’s version.

Did this fix also apply to melee score calculation? I've been having difficulty getting my offline results with roborunner to align with literumble for the nano melee rumble.

Yes, the definition of APS applies to all of the rumbles.

I think I've solved this now, it looks like the problem was with the way I was generating battles.

I wrote a custom GUI for RoboRunner, it directly interfaces with a modified RoboRunner. It is in early early alpha. The feature set is nowhere near complete. Please let me know of any bugs.

Robots are not automatically copied to robocode directories. Nothing is saved. Paths cannot (at the moment) be altered. Results do no update automatically, they need to be closed and reopened to update them (I plan to fix this). Thread count cannot be changed after starting (for now, changing this would be involved).

https://github.com/Chase-san/RoboRunner-GUI/releases

I guess I should mention its advantages over RoboJogger, the other GUI for RoboRunner, also over RoboRunner itself.

It has a queue that runs challenges in whatever order they happen to be in the queue. This means once you start it, you can run multiple challenges without robocode needing to restart, and you can also reorder the queue after starting the battle threads.

It is better to consider it's limitations compared to RoboResearch. Aside from the first post, you cannot start and stop threads individually. I may support this eventually, but at the moment it would require rewriting RoboRunner's BattleRunner class. Which also means once you start the threads, you cannot alter the number of threads running.

Congrats on the first release! I'll try to have a look tonight or tomorrow.

Do we need any setup instructions? Or are the roborunner.properties and empty Robocode installs already in the zip?

I am working on automating the runner creation and robot jar copying process today. As well as adding an options dialog to configure the paths and jvm arguments.

The GUI won't use the properties as the standard roborunner. But the current version doesn't use any at all.

But for the current version, you need to create the robocodes/r[0-9]+ directories yourself (or use the script you wrote). You can pick the thread count in the program itself (but only up to the number of runners you created). Make sure to copy the robots into robocodes/r[0-9]+/robots directories as well.

v0.9.2-Alpha

Adding options (that save), automatic robocode runner creation (based on threads), and automatic robot jar copying.

I'm curious what kind of speed improvements people have seen, if anyone keeps track. Today, in the midst of developing my own UI for RoboRunner, I decided to do a test run to compare RoboRunner vs RoboResearch. I ran MC2K7 Fast Learning challenge against XanderCat 11.6. All data was cleared from the robots directory in both tests. 2 Threads in both cases. RoboResearch ran in 5:30. RoboRunner ran in 3:50. That's about a 30% speed improvement -- pretty dramatic. With dirty data directories where there is a lot of past data, I bet the speed difference would be an order of magnitude more significant. RoboResearch can get pretty slow starting up battles when you haven't cleared out the data for awhile. Results will vary based on the data directories and what kind of robots are being run, but regardless, I think anyone using RoboRunner will see a noticeable speed boost. Very nice.

RoboResearch has an annoying problem when bots print something in the console. The clients crash and the threads become stuck in paused mode. You have to recreate the clients manually. So, RoboResearch is a lot more than 30% slower.

I actually hopefully just fixed the problem about the threads pausing. I still use RoboResearch since getting RoboJogger/RoboRunner to cooperate is difficult at times.

If possible I wouldn't mind hammering RoboRunner into RoboResearch, since aside from a few bugs it works rather well.