View source for Talk:LiteRumble

From Robowiki
Jump to navigation Jump to search

Contents

Thread titleRepliesLast modified
Roborumble Bot Details Gives Server Error715:19, 21 November 2013
more informative landing page122:25, 19 November 2013
Better Priority Battles818:29, 18 November 2013
LiteRumble preconfigured client - Error (404)222:15, 24 October 2013
Clarity & Other suggestions415:49, 29 June 2013
Gigarumble with missing pairings022:31, 26 June 2013
Rumble questions and issues ...521:56, 16 June 2013
Score Distribution614:37, 2 June 2013
Queue full521:36, 31 May 2013
Bad uploads316:01, 31 May 2013
KNN PBI912:01, 30 May 2013
Backlog011:33, 20 May 2013
Put_Your_Name_Here113:44, 21 April 2013
Rerun of Pairings1915:03, 6 April 2013
Bot pairing vs itself511:08, 6 April 2013
LiteRumble Statistics621:48, 5 April 2013
Awesome job ...321:36, 1 April 2013
Assymetric pairings1317:44, 30 March 2013
Ranking column in pairings page108:29, 30 March 2013
quota reached714:10, 29 March 2013
First page
First page
Last page
Last page

Roborumble Bot Details Gives Server Error

I've been getting Server Error whenever trying to see details or do a comparison with game=roborumble. It works fine for other game types.

Skotty (talk)23:38, 18 November 2013

I think roborumble just do it more often. I notice the correlation with stats upload time. At least right around when my client uploads data it gives this error. May be CPU cycles taken for stats recalculation.

Beaming (talk)02:11, 19 November 2013
 

I've just fixed a bug where removing a bot would corrupt the rankings list (it was still storing it in an old format - gah!) until another battle was uploaded and processed, which saved it in the new format. This should fix the problem.

Skilgannon (talk)09:16, 19 November 2013

One more thing I noticed, once I retire a bot. I see them both new and old one in rating for quite a while. Is it related to the fixed bug?

Beaming (talk)14:52, 19 November 2013

That's a longstanding-ish normal thing. IIRC, to keep clients from fighting over removing/readding bots, I think the server doesn't remove bots until a certain amount of time since the last upload for that bot.

Rednaxela (talk)16:33, 19 November 2013

LiteRumble doesn't do that, the moment a client requests a bot to be removed it removes it. However, it keeps all the pairing data against it for 365 days in case it is re-added so that battles don't have to be re-run.

I'm guessing the delay was due to a client taking a while to re-download the participants page.

Skilgannon (talk)22:23, 19 November 2013
 
 

Thanks, Skilgannon. It's all good now.

Skotty (talk)16:42, 19 November 2013

I actually don't think that's what the issue was, I've fixed another bug in the priority battles generation which will give more weight to bots that are missing more pairings. I've also got a lot more debugging so I can see what is happening if this pops up again.

Skilgannon (talk)15:19, 21 November 2013
 
 
 

more informative landing page

It would be nice to have some more links on the LiteRumble landing page to general info about Robocode and RoboRumble. Even just robocode.sourceforge.net and RoboRumble wiki page would help a lot imo. A few times I've found myself wanting to mention the RoboRumble to an outsider (like just now) and I sometimes have to provide multiple links, or if I'm just providing one, I use robowiki.net/?RoboRumble. I'd rather provide literumble.appspot.com, but it basically assumes familiarity with Robocode / RoboRumble.

Voidious (talk)20:32, 18 November 2013

Good idea. I'll see if I can add something over the next few days, perhaps a link to the RoboWiki RoboRumble page and to the robocode.sf.net project homepage.

Skilgannon (talk)22:25, 19 November 2013
 

Better Priority Battles

A while ago I made a change with the priority battles to do a global search for bots that didn't have full pairings, instead of a descent towards lowest by following the lowest bot of the current processed pairing. It really helped with making sure that all pairings filled out ASAP. I've now added something similar to battle count, so only bots with a battle count within 10% of the lowest will get priority battles once pairings are full. It is already making a difference in directing priority battles to more recently added bots.

Next to add is a "lowest APS against enemy" column to rating details.

Skilgannon (talk)10:34, 12 September 2013

Good to hear, thanks Skilgannon. I actually thought it had already been doing that for some time.

Chase12:10, 12 September 2013

It did have priority battles based on battle count, but it was a 'gradient descent' method, so it gives a priority battle to the bot in the uploaded pairing with less battles, and eventually it will 'descend' to the bot with the least battles. The new method goes directly to the bots with lowest battles: if the currently uploaded bot is one of the 'priority bots' it intelligently selects which pairing to give as priority weighted towards pairings with lower battle counts, but the new behaviour is if the currently uploaded bot isn't one of the priority bots, and now instead of giving a priority battle to the currently uploaded bot with less battles it gives a random pairing to one of the 'priority bots'.

Skilgannon (talk)12:24, 12 September 2013
 

Cool, nice work!

Voidious (talk)17:12, 12 September 2013
 

I've noticed that there are currently 4 bots in the roborumble which seem sort of stuck at around 950 pairings. I'm running my client, and it appears to just be running random pairings; it's not running pairings for any of those 4 bots. I do have a number of bots that my client won't run due to the "major.minor" version issue, but it's only about 30 bots, and there are will over 100 pairings missing from each of the 4 bots without full pairings, so that doesn't quite explain it. Might be worth looking into.

Skotty (talk)14:39, 18 November 2013

I'm not quite sure what was happening, I think it might have been due to changing code without changing version number. But I've re-deployed the code with a new version number and it seems to be fixed now, those bots are getting priority battles again.

Skilgannon (talk)17:43, 18 November 2013

Do you have a list of what 4 bots those were? I'm wondering because robocode-archive.strangeautomata.com doesn't automatically update when the same version number already exists.

Rednaxela (talk)17:53, 18 November 2013

eem.EvBot v4.4.5, xander.cat.XanderCat 12.7, zezinho.QuerMePegarKKKK 1.0, EH.nano.NightBird M. EvBot was the most recent release. My client was running quite a few battles for EvBot initially (when it only had about 400 pairings) but when it got up to the 900's, suddenly it was just running random pairings of all bots despite the 4 being short roughtly 150 pairings each. It appears as though all 4 of those bots are now getting priority again, as they have each picked up at least 50 pairings since this morning.

Skotty (talk)18:02, 18 November 2013
 

Appengine code, not bot code :-)

Skilgannon (talk)18:29, 18 November 2013
 
 
 
 

LiteRumble preconfigured client - Error (404)

I´d like to use the preconfigured client explained here(http://robowiki.net/wiki/LiteRumble) , but it gives me a 404 Error while trying to download this. (https://dl.dropboxusercontent.com/u/4066735/literumble-template.zip).For sure , I could use the BitBucket page (https://bitbucket.org/jkflying/literumble/src) ,but I would appreciate it very much to use this comfortable feature.

MAESchortens (talk)21:34, 24 October 2013

Thank you for telling me, the link is fixed.

Skilgannon (talk)21:46, 24 October 2013

Thank you, now it is working fine.

MAESchortens (talk)22:15, 24 October 2013
 
 

Clarity & Other suggestions

Robocode is a game for teaching people how to program as well as a great way for experienced programmers to test their knowledge and skills. The wiki is pretty nice and user friendly. But both the old and the new rumble pages are entirely unfriendly. It would be good to:

  • On the landing page, describe what the bot classes are, or at least link to the wiki explaining the bot classes & types of rumble.
  • On the rankings explain what the columns are at the top of the page:
    - WTF is : APS, PWIN, ANPP, Vote, Survival etc? Its not exactly what I would call noob friendly.

Any other suggestions for improving friendliness of the rumble pages? In the same manner as bot authors can set up their flag, how about allowing them to also set up a link to their bots robowiki page? Then when you click on bot details in the rankings, the bot's page has a "Bot Details on WIki" link. Might be neat.

I know its more work for you chaps to implement, this is a friendly suggestion list. I think you are doing a great job of it at the moment! :)

Wolfman12:54, 5 April 2013

Explaining the different scoring systems is something I've been meaning to do for a while, so absolutely. I was actually thinking of doing it as mouse-over text on the rankings page, although maybe a separate page would be better? As for explaining bot classes, I'd rather keep the server entirely free of any sort of class-specific data, and leave everything up to the client configuration.

As for the back-to-wiki links in the bot details, how about a link that searches for the bot name on the wiki? That would minimise the amount of admin, and would just involve adding a bit of HTML to the BotDetails page.

Skilgannon13:45, 5 April 2013
 

I think it would be nice to have one or more new columns on the Bot details table which would tell whether a certain bot "Voted" for the bot currently being viewed, and whether the bot currently being viewed "Voted" for a certain bot. It would also be nice to see whom else a certain bot voted for, if the vote was split multiple ways.

Thanks

Sheldor (talk)13:59, 29 June 2013

You can infer that from the NPP score. If you got 100NPP against them, then they voted for you, if they got 100NPP against you then you voted for them. NPP isn't symmetric so you their 100NPP doesn't necessarily align with your lowest NPP, but you can just check your lowest APS score, and it will tell you who you voted for (multiple if there were ties).

Skilgannon (talk)14:08, 29 June 2013

Thanks.

Sheldor (talk)15:49, 29 June 2013
 
 
 

Gigarumble with missing pairings

I uploaded about 10k battles and gigarumble still has missing pairings.

I re-added one competitor which already had many battles, and missing pairings seems to be all against it.

MN (talk)22:31, 26 June 2013

Rumble questions and issues ...

Hi mates

I try to bring my robocode environment back to work and would have some questions and maybe some issues.

Are there some changes in the rumble client for 1.8.1.0, I should know about? Beside the increased upload speed, I mean? Because melee battles are not canceled if one/more bots couldn't loaded. Shame on me - I'm still on java 1.6 and a couple of bots are 1.7 written. I try to exclude all 1.7 bots right now but not sure if I get them all. In 1vs1 I get an 'could not load' error for 'apv.TheBrainPi_0.5fix.jar' which looks perfectly fine, link and package wise, so far :(.

Something different: I have to admit that I do not understand how to use the new scoring columns. I know what they do (maybe not fully), but don't get how to interpret the values. For example NPP. If I have, lets say, 88% against an particular bot - where do I have to look and what should I do to increase this number (or better what is wrong and then find a way to make it better)? Is there a way to spot bugs/uncertainties from the scoring columns? I remember looking at the battle table against each robot quite often to see if I loose one or more rounds here and there which brought me to a couple of bugs/mistakes I made.

Wompi (talk)11:53, 16 June 2013

I'm not sure what's up with the 1.8.1.0 issues, what version of the JVM are you using? OpenJDK or Oracle?

As for the scoring columns, I added a link on the Literumble homepage which explains how the scores are calculated. NPP is your APS score normalised against the min and max score against this bot, so if you got 88% then you are close the the maximum score achieved, 0 would mean you got the lowest score and 100 the highest score. So somebody gets 0 NPP and somebody gets 100 NPP against every bot.

When looking for bugs I normally look at the lowest KNNPBI score, or compare to previous versions to see the biggest drop in score. However, in melee this is harder because the score in each battle obviously also depends on who else was in the battle at the time. I'm thinking of adding a 'variance' column, which should help point to bots which cause bugs to happen only some of the time, but I'm very busy right now so if I do that it will only be in late August (I'm in the final writing stage for my MSc!)

Skilgannon (talk)14:13, 16 June 2013

Yes I saw the score description page and it tells very good where the numbers come from - but unfortunately (for me) not enough how to interpret or use it. I think I have to get used to it with a little more observations how the numbers change. My guess for the NPP was, if I get 88% there are bots who perform better then I (score wise) against this bot. But how can I find the bots that are better than me? KNNPBI is great to spot the bots that I have trouble in general with but I don't see how I could use the numbers to see that I loose lets say 1-2 rounds (rounds not battles) every now and then which normally tells me that I have a bug for very specific situations (trapped in the corner at round start for example). Is the K value calculated for general or for every class new? Yes you are right in melee you get a wider spread in score for each bot because you can't tell what other bots are on the field and it would be nice to see the range or something.

But by all means take your time and concentrate on your MSc - good luck with it! I donated a little bit to help you with the maintenance costs and hope you can use it.

Take care

Edit: argh - forget about the NPP question I found it right after I wrote this. I just have to look at the bots score table and see who is better than me. Well I feel a little stupid right now :)

Wompi (talk)18:35, 16 June 2013

Thank you for the donation =) Each rumble is completely independent, only the client knows that minirumble is related to nanorumble, microrumble and roborumble more than say, gigarumble. So the K value is dependant on the rumble that the scores are from, for example if you are looking at Yatagan scores in the nanorumble then K will be calculated from the nanorumble size, if you look at the Yatagan scores in the roborumble K will be larger.

I'm not sure how to do exactly what you are asking without storing every single pairing, which gets far too complicated (and expensive) with the design I have now. Checking the Survival will tell you how many battles you win/lose, but again that is only an average. I am thinking of keeping a Variance score for both APS and Survival (this will be easy to calculate incrementally just like I do the APS for a pairing), and from the variance I can also calculate Standard Deviation and Confidence Interval as I render the page.

About the NPP question, yes you have it, it is a problem that requires the entire score matrix to answer, so you need to check the scores on the other bot's page =)

Thanks for the good wishes for the MSc. Right now it just feels like lots of hard, boring work!

Skilgannon (talk)21:56, 16 June 2013
 
 

Are you sure it's a change of behavior about the Java 7 bots? I would expect the battle to still run and those bots to get zero scores. I've removed some bots from the participants list for requiring Java 7 and asked people to update with Java 6 compatibility - we shouldn't be requiring Java 7 yet. A more prominent notice about that somewhere might be nice. (Excluding them is also fine of course.)

I remember TheBrainPi could get into a broken state with his save data. I think that's part of what we "fixed" but maybe it's still possible? Could you delete him from the .data dir, or was this on a fresh install already?

Voidious (talk)16:43, 16 June 2013

I'm not sure if it is a change, but I remember I had this before and the missing bots where just replaced with another bot from the list. Hmm not sure if all bots get zero score. I just saw the battle starts and takes his time to finish and the upload starts like normal (I will check this). And you get a message on battle start 'bot ... could not loaded'. There are quite a lot bots written with java 7 so my guess was it is standard now.

Yes it's a new install from the scratch with all bots loaded (empty robots directory). I had to fix some broken links in the participants files. So - yep fresh install.

I'm on Oracle 1.6 for all my systems.

Wompi (talk)18:11, 16 June 2013
 
 

Score Distribution

Can someone describe how to read the fancy new Score Distribution graph? What is the X axis? What is the Y axis? What do the different colored dots represent?

Skotty (talk)03:31, 13 May 2013

There is a caption under the diagram describing X and Y. Mouse over the image to see what the colors mean.

Basically X, opponent strength, Y is how good your robot is against that opponent.

Chase06:06, 13 May 2013
 

Basically, the X axis is the score that each particular component got in the rumble, while the Y is the score you got against them. Right now I have red = Opponent APS vs Pairing APS, green = Opponent Survival vs Pairing Survival and blue = Opponent APS vs.(KNNPBI+50). Each pixel colored in represents at least one pairing with the score at that location. Both axes go from 0 at the origin to 100 at the top and right edges of the picture.

I'm thinking of changing green to Opponent APS vs Pairing Survival, just so that the X axis is always Opponent APS. Any thoughts?

Skilgannon (talk)09:01, 13 May 2013

Can I ask a question over here? If I look at my or at any "bot detail page" (like http://literumble.appspot.com/BotDetails?game=roborumble&name=mae.Mae1%201.1 ) I see many abbreviations ,for example APS,NPP and KNNPBI.Is there a page in this wiki or somewhere else where these words are explained ,because my English is not so good that I could deduce the meaning myself. Thank you very much.

MAESchortens (talk)22:36, 1 June 2013

Some of them are listed at LiteRumble.
The important one is APS, as that is the primary ranking. APS is average percentage score.
For each opponent your percent score is 100% * <your score> / (<your score> + <opponent score>). APS is this value averaged over all the battles/opponents for your bot, so it is important not just to win, but to win by the largest margin possible.
PWIN is percentage of wins.

Nz.jdc (talk)02:43, 2 June 2013
 

I have added a page on the Literumble here (and a link to it from the main page) which provides better explanations of what the different scores are.

If you have any questions, or think that they need to be clarified, just ask.

Skilgannon (talk)13:50, 2 June 2013

Thank you very much.

MAESchortens (talk)14:37, 2 June 2013
 
 
 
 

Queue full

When the server queue is full, LiteRumble is returning an OK, making the client discard the uploaded battle.

IMHO it would be better to return an ERROR instead, telling the client to keep the battle and retry the upload later.

MN (talk)21:40, 30 May 2013

I wouldn't mind doing that if the client would do a delay for 10 seconds or something before trying again after an error. Right now it just retries the battles that failed each iteration (along with the new ones) and this quickly leads to all clients just trying to upload at full speed the whole time, which will put too much load on the server.

If we wanted to change the client protocol so that the client had a delay when this happened, I wouldn't have a problem. However, another issue is that the priority battles get delayed then, which means that a) the bots that need priority battles end up getting too many once the queue is run and b) new bots take a long time to enter the rumble. So, optimal client behaviour would be if the server returns OK. QUEUE FULL. then the client should wait 10 - 30 seconds then retry uploading the same pairing to the server.

PS. the new rumble structure lends itself well to bulk upload strategies. If you want to write a bulk upload protocol I would be happy to look it over and implement it on the server.

Skilgannon (talk)09:17, 31 May 2013

I´m writing a custom client right now (slowly writing it from scratch in the last months). Yesterday I managed to make it fully functional, although it still needs polishing (make hardcoded behaviour more configurable). I´ll make it available here after it becomes more stable.

I can add bulk upload, but it will break compatibility with the current protocol. Unless both clients and servers support 2 protocols at the same time.

Features I managed to include in this custom client so far:

- Full compatibility with the current protocol. (I hope that underscore bug was the last one)

- Multiprocess/multithread support.

- Parallel downloading of JARs. (currently hardcoded at 15 simultaneous downloads)

- Processing battles in parallel while uploading results in a separate process. (currently hardcoded at 1 simultaneous upload)

- Abusing the Java 5 concurrent API to keep the code readable in the presence of parallelism.

- Upload throttling in case of errors (currently hardcoded at 10 seconds delay after each error). It is possible to throttle uploads in the absence of errors too, although I wasn´t planning to do that.

- Smarter handling of priority battles. One big pairing matrix handles priority battles, new competitors and competitors with low battle count, all at the same time. And it is independent from iterations (which I eliminated).

- Communication between processes through the network, allowing clients spread accross a LAN. (currently hardcoded at "localhost" address and 1099 port only)

- Automatic copying of JARs between clients. If a single "server" process has all JARs, no client needs to download from internet.

- Logging support. No more System.out.println. You can configure how messages appear in the console (or in a file), adding for example, time and severity.

- Internally, battle parameters are all dynamic. Parameters like number of competitors, inactivity time, gun heat cooldown, codesize classes, hideRobotNames, are all concentrated in a configuration class. The idea is to put them all in configuration files and make divisions like twin duel, team melee or anything else fully supported.

MN (talk)18:23, 31 May 2013

Neat! Sounds like a lot of work. What's the setup like for multi-process battles? And is it the same mechanism locally vs clients across a network?

Voidious (talk)18:59, 31 May 2013

There is a "server" process and multiple "worker" processes. You start the server process by calling server.cmd. And start each worker process by calling worker.cmd. Each one runs in a separate JVM and needs its own Robocode installation. This way each process runs in a separate window and you can see what each is doing.

All communication to LiteRumble is done by the server process alone. Server and workers communicate through RMI.

Server process is currently using the same configuration file of the official client. Worker processes are currently 100% hardcoded, but server address/port and robocode home could be configured.

Server process downloads participants list, ratings, download JARs (in a separate "jar" thread-pool), calculate codesize, remove old participants and generate a local participants list. They run in a "download" single-thread pool (except jars). Participants list and battle count are sent to a "battle generation" thread-pool, which is single-threaded.

Worker processes connect to the server and requests a battle, which is generated on-the-fly by the server in the "battle generation" thread pool. Then the worker runs the battle and sends the result back to the server. Worker processes are mono-threaded (except for threads internal to Robocode).

Server receives the result, splits it in codesize classes and sends them to an "upload" thread pool, which is currently single-threaded.

In the "upload" thread pool, results are uploaded to LiteRumble. Battle count and priority battles are downloaded and sent to the "battle generation" thread pool. If workers flood the "upload" thread pool with results, upload requests are kept in a queue, and are uploaded one at a time.

In the battle generator, participants list, battle count and priority battles are grouped and used to generate a smart battle whenever a worker requests. All battle generation logic is kept in a single class, in a single thread, making it easy to customize.

The result is you see battles going non-stop on workers, and uploads going almost non-stop in the server process, one at a time. Makes a huge difference in melee.

MN (talk)21:00, 31 May 2013
 

Yes, it is the same mechanism locally and across a network.

The overhead from RMI could be avoided locally, but it is so low I didn´t bother.

MN (talk)21:36, 31 May 2013
 
 
 
 

Bad uploads

While testing a custom client I screwed up while uploading results to LiteRumble. Now there are a few bots with wrong names on the server (underlines instead of spaces):

bvh.mini.Wodan_0.50

taqho.taqbot_1.0

eat.HumblePieLite_1.0

ntw.Sighup_1.5

jcs.Megatron_1.2

Polkwane.Intensive_1.0

Legend.Biogon_1.5

gwah.GerryBotMkII_1.5.1

cw.megas.Silhouette_1.1

el.JumpShoot_0.2

ags.Midboss_1q.fast

Polkwane.Piyane_0.7b

bjl.LoneDragon_0.5

pez.mini.VertiLeach_0.4.0

vft.Valkyrie_1.0

japs.Sjonniebot_0.9.1

pmc.SniperBot_1.0

supersample.SuperTracker_1.0

pez.micro.Aristocles_0.3.7

timmit.nano.TimDog_0.33

oog.nano.Caligula_1.15

And the API doesn't let me delete them. It translates underlines to spaces on delete requests making these names inaccessible.

MN (talk)06:29, 31 May 2013

I've changed the API so that it doesn't filter like that in the delete, but does in the upload. I'm not sure if they'll be automatically removed now, or if the client can't handle the underscores.

Skilgannon (talk)09:07, 31 May 2013

The client couldn't do it, so I quickly scripted something to remove them from robo/mini/micro/nano. It should be fine now.

Skilgannon (talk)09:52, 31 May 2013

The client can´t automatically distinguish names with underscores and with spaces when removing old participants. It´s because the rating list downloaded from the rumble server uses underscores, and the list downloaded from the wiki uses spaces.

...and remove requests use underscores ...and upload requests use spaces ...and priority battles responses use underscores.

MN (talk)15:59, 31 May 2013
 
 
 

Wow, I'm really excited about this! Finally I have a better idea where to focus my benchmarks. =) [1]

What k are you using? And is DrussGT getting k/2 because there's nobody above him, or still getting k, but all below him?

Voidious21:19, 12 June 2012

(Well, I mean, I'll know where to focus once you've run about a million more battles... ;))

Voidious21:38, 12 June 2012
 

Sorry I missed this...

Right now I'm using k=sqrt(opponents), and it just chops it off. So the top bot gets k/2, second bot gets (k/2)+1 etc.

Skilgannon16:46, 8 July 2012
 

I just discovered that my numpy conversion had broken the KNNPBI completely, so I've fixed that and re-run all of the rumbles. Now that it is using numpy, it should give nice symmetrical results (although KNNPBI isn't really symmetrical, by design, but now they are consistent).

Also, looking at the code, k=sqrt(len(bots))/2

Skilgannon20:42, 9 April 2013
 

Hi, the KNNPBI doesnt work for DeepThought (http://literumble.appspot.com/BotDetails?game=roborumble&name=cb.DeepThought%201.0)... any idea why?

Cb (talk)11:33, 28 May 2013

I'm not sure, I'll look into it this evening. How long has DeepThought been in the rumble?

Skilgannon (talk)12:07, 28 May 2013

Since 15:13, 24 May 2013.

Sheldor (talk)14:20, 28 May 2013
 

OK, I've made some changes in the way the batch rankings (KNNPBI, ANPP, Vote) are calculated. Notably, the memcache is set using different sized batches and while I was messing about I made Vote account for ties (so it should be more stable). I'm not sure if this will fix the scores for DeepThought since I'm not really sure why they aren't being saved, I'll check back in a few hours to see if it worked. I know DeepThought is definitely getting the scores calculated because the ANPP and Vote show up in the main rumble score, so clearly it isn't being saved somehow.

Skilgannon (talk)14:15, 29 May 2013

It seems that worked! DeepThought now has KNNPBI and NPP scores. If you want to improve your score the quickest, concentrate on these bots =)

Skilgannon (talk)17:59, 29 May 2013

Thanks!

Cb (talk)12:01, 30 May 2013
 
 
 
 

I noticed the backlog was ~22hours, which seems to me a bit excessive =) I've added a check to the upload so that if there is a backlog of more than 2 hours it discards new uploads until the backlog drops again. This 'check' is refreshed twice an hour so that it doesn't thrash too much, and also minimises the overhead of checking the backlog (which takes a second or so).

Skilgannon (talk)11:33, 20 May 2013

Put_Your_Name_Here

I see that there is an actual listing for "Put_Your_Name_Here", any chance we could get that translated to "Anonymous"?

Chase21:55, 20 April 2013

Done

Skilgannon13:44, 21 April 2013
 

Rerun of Pairings

I see quite a few robots that haven't changed are re-running pairings today as if they had new versions. Any idea why that is?

Skotty15:23, 30 March 2013

Looking into it myself. From what I can tell a bunch of battles didn't load into the Batch Rankings, so it assumed that they didn't exist and pulled them from the participants scores list. They've been slowly added back by clients over the last few hours.

I've removed the section of code that removed the battles from the Batch Rankings, but that is just putting a bandaid over the problem. I'll have to look deeper to see what caused them not to load in the first place.

Skilgannon15:30, 30 March 2013
 

Looks like half of General 1v1 has incomplete pairings now. Should I put my clients into overdrive to fix it, or am I making it worse by running clients because of some bug?

Voidious17:45, 31 March 2013
 

Running a bunch of clients isn't going to make it any worse, from what I can tell it was a once-off problem to do with the backend instance being unable to load data. I've removed the mechanism it used to remove the bots, but I'm still not sure (and may never know) why it happened.

I've also just identified a bottleneck/threadlock which will severely limit the ability to upload from multiple clients at once without increasing upload latency to where it will spawn new server instances and cause my quota to be hit again, but I have a fix for that which I'll implement and test tomorrow. The load right now seems pretty healthy though, I see in the logs uploads from you, MN and Wompi, thanks guys. I'll let you know when you can unleash the full power of your machine(s) =).

Skilgannon20:54, 31 March 2013
 

When a competitor is removed from the rumble, is it´s data also removed?

MN20:17, 1 April 2013
 

It's data isn't removed, so you can still see it in the BotDetails, but the pairings info in the other competitors which points to it is removed. Otherwise over many versions the access to other bots will get slower and slower due to increased serialising costs.

Skilgannon20:21, 1 April 2013
 

Keeping pairing data for a while can help protect the database against faults in clients removing competitors from the rumble, only to be re-added again some time later.

MN20:38, 1 April 2013
 

That sounds reasonable, yes. Perhaps adding a 30 day error window, so only if the last battle was more than 30 days ago the pairing data in the 'alive' bot gets purged. Until then it is just marked as 'removed'. I think this purging and checking will have to happen in the backend, because the frontends are fully loaded right now with your and Voidious's uploads.

Skilgannon20:48, 1 April 2013
 

That is exactly what I had in mind.

MN20:53, 1 April 2013
 

The number of bots with not full pairings has gone up - we were under 400 yesterday and back up to 471 now. I noticed an over quota message from last night, was there another loss of data?

Let me know if I should dial back my clients or if there's anything else I can do.

Voidious16:53, 2 April 2013
 

The source of the problem has to be tracked down or the rumble will never stabilize.

I guessed it was the excludes feature from the clients erasing pairing data in the server. But looks like it is something else.

MN17:08, 2 April 2013
 

I suggest tons of logging, tracing all requests from clients.

MN17:10, 2 April 2013
 

Sorry guys, I was trying to see if I could use the marshal module to do my serialisation instead of cPickle because my local testing showed it is about 50% faster, but it corrupted a few bots from each pairings dict so I quickly changed it back. I'm not sure why it had these issues since I tested locally on the dev server and it worked fine, but anyway it is fixed now, and was a completely different issue to what happened before.

It did hit the quota last night, so perhaps tone down the clients a little. There's a threshold below which it is cheap to run, but as the load increases I start leaving the free quota for the instances as well (not just database writes), which gets expensive much more quickly.

Skilgannon17:58, 2 April 2013
 

Took my clients from 4 down to 2.

Can you protect against us overloading your server? Both to avoid hitting quota, and to avoid someone DDoSing your bank account :-), it seems like it would be good to have some throttling or something in place.

Refilling the pairings is going to take a while. Is it possible to tune things (for now) to support a higher client load, to prioritize overall throughput over losing pairings here and there, while consuming less quota?

Voidious19:02, 2 April 2013
 

I've been trying to think of a good way to do that, but the 'recommended way' using Task Queues (which I can then limit to 3-4 queries a second instead of the 5-6 I was getting yesterday) will break any reasonable way of having priority battles.

Also, there is no way to programatically retrieve the current quota usage stats, which means I can't do any auto-throttling.

I can tune it not to do database writes unless a bot has x or more pending battles, which is how I did it previously when on free tier, but the majority of the time is actually taken up with (de)serialising the pairings data, which is why I was trying to shoehorn in marshal. I'll add a min pending battles limit, and you can turn those clients back on, we'll see what happens. Of course, it will probably only hit quota tomorrow night if it's an issue, since today has been pretty slow.

Skilgannon20:12, 2 April 2013
 

I´m thinking in building a custom client which groups results from all local clients and uploads them in a single thread, so the server needs only a single instance per user to receive data.

Combined with multi-threading, clients can keep running battles in parallel while a single thread uploads everything, making it faster than the current client, while at the same time consuming less server resources.

MN20:14, 2 April 2013
 

That would be great. I'm not sure how you'd do priority battles though, would you have a local queue which would be filled, and you just take from there? I guess I could sort-of do this with task queues, but it wouldn't be very pretty.

Skilgannon20:22, 2 April 2013
 

Priority battles are downloaded by the uploader thread after each pairing is uploaded. They would be sent to a queue, which would be consumed by the clients.

Battle results would be sent to another queue, which would be consumed by the uploader thread.

That is the basic idea. You can add some logic inside the queues to make them smarter, like dealing with duplicated battles, excessive amount of data, or lack of data and fallback to random battles.

MN20:35, 2 April 2013
 

I've essentially implemented what you've said here but on the server side using a Task Queue, the only thing we lost was on-the-fly updated battle numbers, but those aren't really being used now that we have priority battles. Also, priority battles are delayed by up to 100 pairings per rumble, but this new design should mean that stuff sticks around in local memory longer than before.

Once I add contributor stats I'll also add information about the current amount of queue backlog, so people can decide whether or not to run a client.

If you check your clients you can see that the uploads are going much quicker, and it tells you it is adding it to a queue instead =)

Skilgannon10:03, 4 April 2013
 

Yay, back to full pairings in General 1v1!

Voidious15:03, 6 April 2013
 

Bot pairing vs itself

Seems like there's a bug in the TwinDuel rankings: [1]

DuoLedByDroid and TwintelligenceTeam each have battles vs themselves and it's counting as an extra pairing. (I noticed because the archived rankings code determines that rankings are stable by everyone having the same number of pairings.)

Voidious16:23, 26 March 2013

Same for PanzerGeneral in minimelee: [1]

Voidious16:24, 26 March 2013
 

OK, I modified the code that checks that pairings are still in the rumble to also check against the bot name. They should be fixed next time they get a result upload.

Skilgannon16:41, 26 March 2013
 

Sweet, already got those fixed with my clients. LiteRumble's looking pretty awesome!

Voidious17:06, 26 March 2013
 

Dunno if this is the same bug or a different one, but wompi.Kowari 1.4 and ag.Gir 0.99 show 989 pairings in the main rankings, but 988 in the bot details. (Should have 988, there's 989 bots.)

Voidious01:27, 6 April 2013
 

This was something different, but thanks for catching that! Fixed.

Skilgannon11:08, 6 April 2013
 

LiteRumble Statistics

If you refresh your LiteRumble home page, you'll see a link at the bottom which takes you to the all-new LiteRumble Statistics page. This page is updated once an hour, on the hour, and contains the latest contributor information as well as how big the queue is and the expected processing backlog based on how many were processed in the last minute.

I also improved the OverQuotaError handling so that from now on your rumble client will see a regular server response and won't go crazy with the uploading errors. Also, from now on, when the queue is full it doesn't give an OverQuotaError, but instead waits half a second and sends a nice message back to the client saying that the upload was discarded.

Enjoy!

Skilgannon08:02, 5 April 2013

Is this upload queue a task queue from app engine?

MN15:57, 5 April 2013
 

Yes. Because of this it doesn't have pure FIFO behaviour, sometimes new items get run before old ones, but for the most part it is FIFO.

Skilgannon16:01, 5 April 2013
 

Is the upload queue size the amount of battles which still needs to be processed?

Which means while the queue size is greater than zero, we can stop uploading battles and the server still has work to do?

MN17:17, 5 April 2013
 

Exactly. The amount of time it is estimated the work will take is the Queue Delay, but this is only based on the average speed in the last minute so it fluctuates a lot.

Skilgannon17:40, 5 April 2013
 

Did the queue replace the older cache? No more evicted battles?

Also, I know there is a quota for task queues, of about 100,000 messages/day.

MN18:04, 5 April 2013
 

You do not have permission to edit this page, for the following reasons:

  • The action you have requested is limited to users in the group: Users.
  • You must confirm your email address before editing pages. Please set and validate your email address through your user preferences.

You can view and copy the source of this page.

Return to Thread:Talk:LiteRumble/LiteRumble Statistics/reply (6).

 

Awesome job ...

Hi mate. I just wanted to thank you for all the effort you put in keeping LiteRumble up and running - it is very much appreciated by me. Awesome job. Of course thanks too Voidious, Sheldor and all the others who spend her time to keep RoboCode alive. Right now I have not much time to help out - I barely could write a couple of bot code even on easter weekend :( . But let me know if I could drop some money or something else to show my respect for your work.

take care

Wompi20:03, 1 April 2013

Thanks. At some point I might ask for some paypal donations to cover costs, but not yet =)

Skilgannon20:18, 1 April 2013
 

Yeah man, LiteRumble is great! It's pretty remarkable Darkcanuck's server could just disappear one day and we have a viable alternative, with rankings already up to date, to switch to on the spot. And you've been lightning quick in adding polish and fixing stuff since it became the canonical rumble server. Great work, and I too am ready to pitch in for costs whenever you say.

Voidious20:27, 1 April 2013
 

Wompi,
Thanks for the mention. Though I doubt that my contributions to the RoboWiki have done much for Robocode itself.

I look forward to seeing how Kowari turns out.


Skilgannon,
Thanks for saving the RoboRumble. :)
Great job!


Voidious,
I agree. The LiteRumble's awesome!

Sheldor21:36, 1 April 2013
 

Assymetric pairings

I noticed pairings between 2 competitors are being scored differently. A vs B score is different than B vs A score.

LiteRumble could stabilize twice as fast if a given battle upload is updated for both competitors involved.

MN22:05, 22 March 2013

Certainly looks that way. At this moment, XanderCat 12.6 has run against all other opponents, but if I go look at some of the opponents who do not have full pairings, ones that XanderCat 12.6 has run against, XanderCat 12.6 doesn't show up.

Skotty22:44, 22 March 2013
 

Also, a number of values are coming up 0. Will those eventually update when pairings complete, or are not all scores currently tallied (either intentionally or due to a bug)? I know there was talk of dropping ANPP, so maybe that is 0 on purpose, but Vote for XanderCat 12.6 also comes up 0, even though XanderCat 12.6 shows up against PolishedRuby, who XanderCat gets top score against.

Skotty23:01, 22 March 2013

Those complex rankings are coming up 0 because they didn't have some battles run against them after the batch rankings was run, and got evicted from cache before their new data was written to disk. I think I'll do a disk dump at the end of each batch processing job, see if that sorts this out.

A lot of these problems are showing up because I put a bit too much 'eventually' into the 'eventually consistent' model.

Skilgannon08:01, 23 March 2013
 

Some of the complex rankings are updated in batch at intervals.

Voidious23:07, 22 March 2013
 

They are both updated at the same time, but for a long time I had very aggressive caching on to keep within the free tier. Because App Engine Memcache can be cleared without warning, this meant that often one bot would lose a battle before it could be written to disk, but not the other. Ironically, this is one of those situations where a higher client load on the server means that less data gets lost, since the bot was more likely to hit the 'write threshold' before being evicted from the cache. Now that I have more writes to play with, I've lowered the write caching thresholds a lot, and we should eventually see everything settling down.

Skilgannon07:05, 23 March 2013
 

Is the cache independent for each rumble? Or one big cache for everything?

MN14:07, 23 March 2013
 

One big cache for everything. I used to have a queue for each rumble that would fill up, and each time a bot had over X battles it would be written to the database. Of course, when the nanorumble only gets 1/10th or less the battles of the main rumble, the bots in it are more likely to be evicted, resulting in incomplete and asymmetric pairings. So I've just rewritten that section to keep just two queues (only split between 1v1 and melee), and bots are written as a batch as soon as the queue is a certain size. That way, even bots getting less battles will still be written because they are all pushing each other through, so it should stabilize better. Unfortunately I couldn't do this before because I was too limited with writes.

Skilgannon14:16, 23 March 2013
 

What is the queue size?

MN21:51, 29 March 2013
 

30. Bots with more than 1 unsaved pairing only count once, though.

Skilgannon08:20, 30 March 2013
 

Is the queue size per competitor instead of per pairing then?

MN14:48, 30 March 2013
 

I keep a dictionary of {bot.Name:unsavedBattles} and once there are more than 30 entries it gets written.

Skilgannon14:51, 30 March 2013
 

Let me explain that better: Each time a pairing is uploaded, the two bots are added to the dictionary with unsavedBattles of 1. If they already existed in the dictionary, their unsavedBattles count is incremented. Once the dictionary size is bigger than 30 (size = number of entries, not sum of unsavedBattles), all bots listed in the dictionary are written to database.

Skilgannon14:56, 30 March 2013
 

Dictionary size reaches 30 when at least 30 different bots had pairings uploaded.

I configured my clients to run/upload 30 battles per iteration. With meleerumble still having 3 battles per iteration. If all battles have at least one different participant, the server flushes the queue at least once per iteration, with higher chance of it happening exactly after the last upload leaving an empty queue.

It is an attempt to minimize the amount of evicted battles.

MN17:42, 30 March 2013
 

Ranking column in pairings page

I miss the ELO rating column in the pairings page. When you viewed all pairings a bot has, ELO rating column allowed you to sort all pairings by opponent's strength.

There is no column which allows you to do that anymore. A sortable column with APS rank or league APS would be nice.

MN22:52, 29 March 2013

Interesting, that was never something I used. I'll look into it.

Skilgannon08:29, 30 March 2013
 

quota reached

ERROR WRITING DATA!! QUOTA REACHED?

Oh no! I guess I'll stop my GigaRumble client. =) I've only been one running one slow client on my laptop, and with 30 slow bots it's pretty low volume - I think ~20 minutes per 25-battle iteration, at least.

Voidious19:04, 4 August 2012

Don't worry, I don't think it's you. I have 8 clients running at the moment in my lab, and it seems that they suddenly started producing a whole lot more battles over the last few hours (I'm not sure why). It's quite interesting, on the requests-per-second graph I can see when the latest Diamond was released quite clearly =) Check it out:

Literumble-request-per-second-chart.png

Again, I have no idea why that sudden burst of data came in. Considering it's about triple what I normally get, it would correspond with ~24 clients running regular rumble battles. Very strange. And no wonder I'm out of quota...

Skilgannon19:24, 4 August 2012
 

Hmm. If you have battles per bot 2000, and then they stopped running DrussGT / Diamond battles, I could easily see the battle rate tripling when switching to minibots or random battles.

Voidious19:33, 4 August 2012
 

It shouldn't switch to random battles though, because I've got priority battles going the whole time and the 1.7.3.2+ clients ignore the BATTLESPERBOT if there are priority battles waiting. I think I've identified a bug in the priority battle selection algorithm though, it is giving higher priority to pairings with more battles, rather than less (although that only starts happening once all the pairings are complete, at least).

Also, it hasn't been waiting on Diamond and DrussGT to fill out their battles like Darkcanuck's rumble, because until just now *everybody* was below 2k battles.

Skilgannon19:49, 4 August 2012
 

Ruh roh, my client's hitting this again.

OK. mrm.MightyMoose .2 vs logiblocs.Fire 1.0 received in 365ms
ERROR WRITING DATA!! QUOTA REACHED?
Voidious23:23, 28 March 2013
 

That makes no sense, it is well below hitting anything. There was also an issue with priority battles not having any options because everybody had full pairings, which was some errors, as well as malformed requests which were coming from my university connection, I think they had something to do with the SEACOM cable currently being down.

I've added some logging, so next time it happens I can see exactly what the problem is.

Skilgannon08:26, 29 March 2013

OK, fixed. It was having trouble with a mixture between numpy floats and native Python floats when writing them to the database.

Skilgannon09:17, 29 March 2013
 

Ah, so this was just some other error with the same message? That's a relief. :-) Turned my clients back on.

Voidious14:10, 29 March 2013
 
 
First page
First page
Last page
Last page