Talk:RoboResearch/Development

From Robowiki
< Talk:RoboResearch
Revision as of 00:49, 2 October 2008 by Rednaxela (talk | contribs) (Wait.... better statistical soundness now!)
Jump to navigation Jump to search

Showing Results

I have already talked with some robocoders about how they would like to see results in the RoboResearch GUI. So far the best design I've come up with (merging ideas from Synapse, Rednaxela, Voidious and Chase-san) would open a separate window for each challenge+challenger. It would have a table that shows live-feed results like the summary on wiki pages. That window could be expanded to show a second table underneath the first, with the scores for each season. The top table could also be expanded to include a row for each version of the challenger in the database. Whichever version is selected in the top table is the one whose seasons would be displayed in the lower table. Can I get any feedback or new ideas? --Simonton 05:35, 21 September 2008 (UTC)

Well, apparently everyone either liked the idea or doesn't care enough to offer alternative input. The above idea has been implemented with a slight difference: the seasons are displayed in a separate window, at a button push. This means they do not change with as you select different version - instead you can open the seasons for each version in its own window. --Simonton 15:29, 1 October 2008 (UTC)

I would like to add some statistics to the results windows. Firstly I would like to highlight the top score among versions in the results table (or make it italic or underlined, or whatever). But to make that more useful I'd like to calculate the standard deviation of each score, then highlight any that are "close enough" to be tied for the top score. Then I'd like to do the same thing but only within those scores you select. I do it for my own benefit, but I hope candy like this makes you want to update RoboResearch and reap the benefits of my labor :). At some point I'll make a zip file and put it on sourceforge for everyone, so you don't have to deal with SVN. --Simonton 15:38, 1 October 2008 (UTC)

I'd be in the former group, I didn't see anything I didn't like =) Thanks for all this work you're putting into it. If I update will I lose all the season's I've already run? I'm not that familiar with the database system you're using... -- Skilgannon 16:47, 1 October 2008 (UTC)

You will not loose any data by updating. However I have lost data when closing down my database process before, so I recommend periodic backups of the database directory (and due to the way it works, I recommend not deleting a backup until you're sure a more recent one actually has all your data). If you had any scripts that you modified from those in SVN, they may get updated or deleted (run.bat and run.cfg come to mind). Note that SVN is not as up-to-date as the features listed on the wiki; I'm currently without internet at home, so I only get to commit on the weekends. While I'm posting again let me ask, does anyone have any pointers about how to calculate which scores are "close enough to be tied". I'm sure it's not hard, but I might as well ask for pointers instead of doing the statistics research myself :). --Simonton 17:26, 1 October 2008 (UTC)

  • The scores are "close enough to be tied" if result1 +- margin_of_error1 and result2 +- margin_of_error2 overlap. The more they overlap the closer they are to tied, tied% = overlap/(2*min(margin_of_error1, margin_of_error2))*100 -- Skilgannon 20:06, 1 October 2008 (UTC)
  • Makes sense, but how does one calculate "margin_of_error"? Some percentage of a standard deviation? If so, what precentage? And what about long-learning challenges where there really isn't enough data to get a good stardard deviation (e.g. at one season s.d. is zero)? --Simonton 20:34, 1 October 2008 (UTC)
    • Once there are enough seasons run, I believe a number proportional to the standard deviations would make sense. If we assume the scores are roughly in a normal distribution, then once there is enough data for accurate calculation of standard deviation it's not hard to state criteria like "how wide is the region in which 90% of scores are expected to fall", in which case it would be 1.64485 standard deviations (see the table of confidence intervals here). One very important factor however, is that as the number of seasons increases, the "margin of error" of the AVERAGE should decrease proportionally, therefore I believe the most statistically sound thing to use for a "margin_of_error" would be, something like "1.64485 standard deviations, divided by the number of seasons", or if we want a more stringent confidence interval of 95% or something, then 1.95996 instead of 1.64485, but either way, something of that form would be most statistically sound I believe. --Rednaxela 22:27, 1 October 2008 (UTC)
    • Update: Actually I'll revise that. If this is correct, we should divide by the square root of the number of seasons, rather than the number of seasons. That will give error in the average score to whatever confidence interval we specify. --Rednaxela 23:49, 1 October 2008 (UTC)

Nice work on this lately! I just got Subclipse working again here and downloaded the latest version, however am having one problem: Trying to start the gui causes the following error: 

Exception in thread "main" java.sql.SQLException: socket creation error
	at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
	at org.hsqldb.jdbc.jdbcConnection.<init>(Unknown Source)
	at org.hsqldb.jdbcDriver.getConnection(Unknown Source)
	at org.hsqldb.jdbcDriver.connect(Unknown Source)
	at java.sql.DriverManager.getConnection(DriverManager.java:620)
	at java.sql.DriverManager.getConnection(DriverManager.java:200)
	at roboResearch.engine.Database.<init>(Database.java:65)
	at roboResearch.GUI.<init>(GUI.java:44)
	at roboResearch.GUI.main(GUI.java:25)

--Rednaxela 18:22, 1 October 2008 (UTC)

Now how is it you compliment the latest work when you can't even use it? ;) You need to run the database in a separate process (until I update SVN this weekend). I think that's your problem. Instructions for that are in ... umm ... I think it gives you the command to run if you try executing TUI with no command line args ... or invalid args ... or something like that. --Simonton 18:33, 1 October 2008 (UTC)

Ah nice, it work now. By the way, one issue I just noticed, is it doesn't like if very much when you browser for a bot outside of that robocode_bots directory ;) --Rednaxela 18:41, 1 October 2008 (UTC)

Networking Proxies

I know how to use raw TCP/IP streams to write the proxies required to implement a networked RoboResearch, but I'm not sure that's the "best" solution. I know how to use Spring to make remote method calls, but it does not allow callbacks for things like listeners. Does anybody know of some other technology that would be appropriate to support the kind of remote communication necessary? I think I've heard RMI can do callbacks, so that might be one possibility. JMS might be another. I'm just not familiar enough with any of these to know their strengths and limitations. Something that supports file transfer for uploading/downloading bots would be ideal. Any thoughts/recommendations/input? --Simonton 16:38, 25 September 2008 (UTC)

Retrieved from "http://robowiki.net/w/index.php?title=Talk:RoboResearch/Development&oldid=3186"