Upgrade client version

Jump to navigation Jump to search
Revision as of 4 October 2018 at 00:45.
The highlighted comment was edited in this revision. [diff]

Upgrade client version

Hi all

I've been thinking of upgrading the accepted client version. What are your thoughts on the matter? Have any of you tested extensively with the latest releases locally? Are there any changes in scores (or any changes that might affect scores) that we should do more extensive testing of first?

    Skilgannon (talk)19:21, 5 July 2018

    Finally 1.9.3.3 released, with RoboRumble battle duplicating fix, and with shorter participants list check period from 1.9.3.1 as well. It’s time to do a test and discuss to upgrade.

    Question, what kind of tests should be done in local? Should we run a literumble locally and see whether it yields the same score as the main rumble? (which gives the best result but takes monthes). Or just using roborunner with a few robots (both 1v1 and melee certainly), and see their score is unaffected.

      Xor (talk)01:08, 11 September 2018

      Just roborunner and a few bots should be fine. At some point it would be good to reset the meleerumble as well, since old bots get biased downwards by excessive battles against new (good) bots.

        Skilgannon (talk)06:02, 11 September 2018
        Edited by author.
        Last edit: 00:28, 4 October 2018

        I've been always thinking about the pairing systems of meleerumble.

        Once every combination of 10 bots had a run, the score is unbiased, which takes N = n! / (10! * (n - 10)!) ≈ 10^19 battles in current settings (n is the total of participants).

        However we should get approximate score with feasible battles via monte carlo method. In current settings, ~10000 battles already gives a somewhat stable score (for the new participant).

        Let each bot gets m battles, randomly selected from all (N / n) 10-bot combinations containing that bot, then the probability of meeting another specific bot in a battle is (N / n / (n - 1)) / (N / n) = 1 / (n - 1).

        Assume that when a new bot is released, every battle contains that bot, then the probability of meeting that bot is 1 instead of 1 / (n - 1), which is highly biased.

        To fix this, we have two options — mutate our current pairing systems to get unbiased score online, or to reset the entire meleerumble periodically.

        Since the score of new bots are unbiased, all we need to do for an unbiased score is to ignore (n - 2) / (n - 1) biased battles randomly when calculating the score of an old bot. However this approach takes much more battles.

        A more practical way is is, when bot A is added, for each battle, select another bot B randomly, and run melee battles containing those two bots as usual. A battle containing A, B and other 8 bots should yield 45 pairings, but only those matching (A, *) or (B, *) is taken into account. This produces 17 parings.

        This scheme does not affect the pairings of the new bot itself at all, which is already unbiased; And for an old bot, the probability of being chosen as B is 1 / (n - 1), therefore the probability of a battle with A present being taken into account for old bots is 1 / (n - 1), the same as the unbiased one.

          Xor (talk)09:29, 11 September 2018

          Your second approach seems good, it will need patches on the client side so that if a priority pairing needs to happen it only uploads the battles which contain one of the priority bots. This filter will work fine for the 1v1 rumble as well.

          Why don't we wait to update the client until this can be fixed too?

            Skilgannon (talk)20:50, 11 September 2018

            The only reason for upgrading earlier is that those two bugs are very annoying when running rumble client 7/24.

            And it generally takes monthes for fnl to release a new version in normal cycle ;(

            Anyway, since those two fixes are unrelated to robocode engine, one should expect no performance difference if we cherry pick the fix back to 1.9.2.5.

            Maybe we could release a special version of 1.9.2.5 for those running rumble 7/24, before the melee pairing fix is released.

            Or, we may just ask fnl whether he could please release 1.9.3.4 soon after the meleerumble patch is applied.

              Xor (talk)01:51, 12 September 2018

              I patched robocode 1.9.2.5's roborumble by cherry-picking roborumble client fixes from 1.9.3.2 and 1.9.3.3

              The patched roborumble.jar can be downloaded from:

               https://drive.google.com/uc?export=download&id=12lu8H4SlpBxOFRF5L8_koU5j7Cla5BLj
              

              This should check participants list more often preventing retired bots from being battled and submitted nullifying smart battles. And on unstable network it won't upload 80000 duplicate battles for a single pairing any more. (See pulsar.PulsarNano 0.2.4 and ad.last.Bottom 1.0)

              I suspect we should separate roborumble version and robocode version (since they are weakly dependent) and get roborumble bug fixes quicker than robocode, since roborumble bugs are generally more serious, and roborumble fixes has nothing to do with the rest of robocode users.

              Currently fixes in roborumble takes almost a year to be actually deployed.

                Xor (talk)04:57, 20 September 2018

                I've enabled 1.9.3.3 for uploads. Let's try it for a while and see how it goes. If we notice anything strange, well, the rumble probably needs a wipe some time soon anyway, and definitely melee.

                  Skilgannon (talk)16:18, 3 October 2018

                  I'll upgrade all my clients later and upload the same bot of mine with different version number to see the impact on score...

                  The melee part is already discussed, but why the 1v1 rumble needs a wipe as well? And how long it should take to reload the rumble? Or should we run the current rumble as a backup in parallel until the reload completes? (so that new players can still submit their bots)

                    Xor (talk)16:40, 3 October 2018

                    Hopefully we don't need to wipe 1v1 rumble. I am more just concerned with bad configurations, running with Java10 etc that have accumulated.

                    Have you managed to make a patch for the melee priority battles?

                      Skilgannon (talk)16:45, 3 October 2018

                      The rumble clients load version number dynamically, maybe we should tweak this to append Java version after version number as well to prevent running and uploading with Java 10 incidentally.

                      And since we have rolling average, the accumulated effect of bad configurations should be fixed automatically (after a lot of battles).

                      A patch for melee priority battle is harder than I thought, as the data structure (currently, very bad, string) needs a redesign for the extra 1 bit of information. And almost every line of code needs a rewrite as it depends on the implementation details of the (not extensible) data structure.

                        Xor (talk)17:05, 3 October 2018
                         

                        Done! After some refactoring, the rumble client should upload only pairings containing the prioritized bot, or containing the predetermined random bot when running prioritized battles.

                        The only flaw is that when the rumble server returns prioritized pairings, etc. A and B, if B is not evenly distributed, then B will get biased battles (meet more A). Is literumble using this feature currently? How not evenly distributed is bot B returned in this case?

                          Xor (talk)01:44, 4 October 2018