Talk:Darkcanuck/RRServer/Roadmap

Why Re-Create Battles?

I think "delete all my battles & re-load all battles from ABC's server" should definitely be on the list of next steps. It might even encourage my contribution to world peace. --Simonton 16:46, 26 September 2008 (UTC)

I didn't realize that all battles were available on the old server architecture. For some reason, I thought it only kept summaries, but if I can import all that data, that would be great! I won't be deleting any battles from the new server though -- almost all of the data submitted has also been relayed to ABC's server, so his server should be a superset of the two. For an import I would just pull in all of ABC's data (plus the Elo rankings, it would be nice not to have to recalculate those) and there's a duplicate checker which would throw away duplicated results (bot, challenger, timestamp + user IP determine uniqueness). But if you want to contribute to both, just upload your results to mine and they'll also be relayed to ABC's. And keep your "remove old" and "ratings" URLs pointed at ABC's server if you want to flesh out the pairings over there, my clients will keep the pairings here in line. -- Darkcanuck 17:01, 26 September 2008 (UTC)

The old server never deletes battles, it keeps a (huge) log of all of them. That's actually what led to the first file corruption problems, disk space usage can grow a lot when you leave it running for years. Keep that in mind when you design your battles table, it can easily grow into the millions of rows. BTW, in my server I did a backup/reset of that log somewhere around 100 battles per bot, you should download the battles_* files form the backup dir and append the ones in the main dir. --ABC 17:22, 26 September 2008 (UTC)

Cool. I had a look at those files, it should be straightforward to create an import script, the format is simple. But weren't there some issues with bad results early on? Did you remove all of them by hand, or do I need to filter them out? Also need to figure out how to keep the two systems relatively in sync. Duplicate checker or not, if all that data gets imported and we want to import some updates later on, it might crush the server to keep having to re-import a 40MB battle log.

Storage space is something I've kept in mind. I've tried to keep the battle results limited to fixed-width binary fields and have some more space-saving plans in the works. Should be much more compact in a database than in plain text files. But I'll probably periodically archive retired bots' results to keep the table lean & mean. -- Darkcanuck 18:24, 27 September 2008 (UTC)

I removed all of the bad results (and some good ones in the process), those files are clean. The battle timestamp field should be enough for avoiding a full re-import. We should also discuss what to do about the future, do we merge efforts or keep 2 separate servers running? --ABC 18:53, 27 September 2008 (UTC)

I'll give my perspective as somewhat of a bystander: I think it would be really cool to have a RR server both with some of the features Darkcanuck has been working on at the same time as some of the features ABC has been working on (real neat graphs!). That said, I'd prefer to sitck with having ABC's separate server until this new codebase has proven itself stable. I'd also be tempted to say that I would try to help with a merged effort, however currently I don't think I have time to devote to that, due to a combination of coursework and addiction to working on my next bot. --Rednaxela 19:20, 27 September 2008 (UTC)

If I had known you were going to enhance the old server code when you set up the temporary server, I might not have started this. :) But I think merging the two efforts is better in the long-term, it doesn't make sense to have different rumble servers all over the place. In the meantime, you could try relaying your results over to my server -- if we both check the referrer field (I'm sending "http://darkcanuck.net/rumble") we can avoid endless relay loops.

My new problem, much to my chagrin, is that the relay wasn't working 100%. Which means I have results that you don't. It's fixed now (and slower, yay) and the relay confirmation is now also sent to the client. I've disabled relaying for melee because it's far too slow and I don't think you're set up for melee? -- Darkcanuck 19:32, 27 September 2008 (UTC)

I'm guessing that each time your server gets a result, it uploads it to ABC's server and then only once it gets a confirmation from there it sends the confirmation back to the client? How about going multi-threaded, and for each result start a new thread that uploads the result to ABC and then terminates if it is successful? You can send confirmation to the client immediately and the client can immediately upload the next result. This should be significantly quicker. While we're on the topic of multi-threaded, I always wondered why the client didn't do this, ie. simply upload the result after every battle in a separate thread. Another option would be to zip the results into one file and upload them all together at the end of each iteration. --Skilgannon 22:09, 27 September 2008 (UTC)

Speeeed

You just made a comment about this, but let me reiterate it. Submitting results is really slow. I just did some timings: it took 7:42 to run 30 battles, and then 2:28 to upload the results. That's alomst 25% of the time uploading results. The solution is in the use of threads, but really it would be best for those threads to be on the client side, to avoid overwhelming the server. However, nobody has been working on the client, so that's why I bring it up here. Any thoughts, ideas? --Simonton 22:28, 27 September 2008 (UTC)