Debugging and tuning AIs

[yieldthought]

I've had a lot of sucess clustering the debug logs of the runs and using this to track down bugs, unintended and suboptimal behaviour in a lua AI I'm writing at the moment:

http://yieldthought.com/post/9572288205 ... -better-ai

If anyone is interesting in experimenting with this I'd happily spend some time looking for a good way to make it available. Compiling up the ML libraries and their dependencies wasn't exactly straightforward but it would be easy enough to expose as a service that takes your game logs and returns a nice SVG file or something.

[mattsc]

Hi yieldthought,

That sounds really nice and very interesting and, man, can I relate to a lot of the things you write in your introductory paragraphs! I will definitely keep this in mind and might come back to you about it at some time. Right now, I don't think it would be worth it for you to spend the time and effort on making it available for general use (not for me, I mean). First of all, I am taking a semi-break from AI work for the most part and the little bit of time I spend on Wesnoth in general is on my campaign. Second, my biggest problem at the moment is not finding out what my AI does wrong. I know that, I just have absolutely no idea what to do about it. So in that sense, I'd actually be more interested in what your AI does rather than how you analyze it.

As a side comment, I often find that even when my AI wins against the default AI, it is making some horrible mistakes and would, in fact, do worse against a human opponent than the default. It's just that the default AI doesn't do very well profiting from the enemy's mistakes (I've seen my AI come back and win from being down in total hitpoints by a factor 3), while a human opponent would just kill you even after the first serious mistake. Often, the bigger mistakes actually happen in games which the AI wins (that's probably just a selection effect, since it wins so many more than it loses). Is there any way you could investigate that sort of thing with your method?

Anyways, I really like this and I will come back to it at some point (to think in more detail about if/how I might be able use this for the AI work I want to do). I'm just not very active on the AI front right now (and unfortunately I don't know of anybody else who is). Are you planning to keep working on Wesnoth AIs? I'm sure I'll be drawn back to them sooner or later myself...

Thanks and keep up the good work,
mattsc

[iceiceice]

Hi yieldthought,

Nice post! This is quite thoughtful, and reminds me of some discussion on the dev list of how to more easily find opportunities to improve AIs.

The visualization tool seems quite nice and it sounds like your system could be easily extended to include other stats from games as well. I think I would have to use it to fully appreciate it but at the same time I can see why its nifty.

I have a question though, did you also try just a simple vector distance before trying the normalized compression distance? I think I would be curious if the results are tangibly different. At least off hand, I think I would expect that might be slightly better, since presumably I want a number 2 to be more similar to 3 than to 10, regardless of what the compression algorithm does with it (and it's not arithmetic in any sense if I understand correctly? so it probably can have large reactions to small numerical changes). And similarly I guess I would want a vector like <7 8 10> to be more similar to a vector <8 9 11> than to a vector <8 10 7>? I'm not sure, this might not matter very much.

Cheers,
iceiceice

[yieldthought]

Hi mattsc,

I had such fun playing the default AI as a beginner, gradually learning how to explot its weaknesses until I could finally triumph! I hope to build up a set of entirely different AIs with their own unique set of crippling flaws to complement it, rather than to create the master AI that provides a strong challenge for an experienced player. I haven't had much time lately but will certainly look to contribute these back once they're a bit more neatly rounded off!

Thanks for making this fun possible
Mark

[yieldthought]

Hi iceiceice,

No, I didn't try something simple like L2 vector distances, I would expect slightly better results using that, too. The nice thing about NCD is that you can run it on *anything* and get some kind of clustering out, so I had the right tools for it lying around as "let's quickly NCD that" is one of my standard actions The sparklines are included in the text used for NCD calculations and make up a good proportion of the compressible length of the text so although 9 and 10 are more similar numerically than under compression, two identical unicode characters for the sparkline are even more so, which may mitigate the effect somewhat. I was originally going to strip the numbers altogether and just NCD the sparklines but the results were good enough that I didn't need to.

This morning a question on the MachineLearning subreddit prompted me to try NCD without the sparklines and the resulting clustering is still very similar... I was pretty surprised by that result!

Thanks,
Mark