Before I get started on beta testing, let me tell you something about myself: I am a QA manager at Ascaron, where I handle the organization of beta tests; I am also the head of customer support, and responsible for conceptional matters such as the design of the highscore ladder in Sacred 2 (more on that in another blog). I also used to work for THQ and SWING! Entertainment as a producer, among other things.
But now let’s talk about beta testing Sacred 2. What is the purpose of beta testing? Most people will say: finding errors/bugs. That is undoubtedly an important reason. External people who, unlike us, do not already know the game inside out, often approach the game in their own unique way and may thus encounter bugs that our QA company could not detect despite all their in-depth probing; often enough, they could not even have imagined that such bugs might exist – and (luckily) beta testers then demonstrate that it is perfectly possible.
Of course, a large number of beta testers will increase the potential for finding more bugs. However, there is no linear increase of the number of bugs found with an increasing number of beta testers; it actually declines: even a (theoretically) unlimited number of beta testers would only be able to find a limited number of bugs. This fact naturally limits the number of beta testers that makes sense in practice – as does the necessity of properly managing and supporting all those testers.
But finding bugs is not the only purpose of beta testing. Another point that is becoming increasingly important is balancing. A game should always be fun, it must never be too hard or too easy (which would leave you either frustrated or bored), and players should be eager to find out what happens next.
Well, as developers, we know our own game mechanics, and we have certain concepts as to the level of difficulty within the game, and how it should be played. But are they really the same as the concepts of players "out there"? Internally, we can only verify this to a certain degree; beyond that, we need more sophisticated evaluations, and the main source for those is beta testing. Does everybody really think that combat skills are too strong? If so, why? More opinions make for clearer tendencies and results. This is another fundamental aspect of beta testing, possibly even the most important.
Compatibility testing is also essential. Of course, we work with specialized companies who separately conduct such tests, but beta testers offer one special advantage: they own "home-grown" PC systems, i.e. end user systems where all kinds of software have previously been installed – some sensible, some not, and some defective or outdated. Such constellations may lead to all kinds of problems during gameplay, and the game itself is often not to blame. But such findings allow us to figure out suitable countermeasures early on, and if we cannot completely prevent such issues, at least we will be able to point them out in the FAQ, and customer support will be aware of the problem even before the game is released.
Besides general feedback, ideas and suggestions from beta testers are another case in point – a fairly important one, I think. It is amazing how often, during previous beta tests, testers came up with some really good suggestions that we were able to implement.
A really good suggestion is one that makes the game even more fun to play, but can be implemented in next to no time. Usually, this kind of idea is obvious, and we have already implemented it well before the beta testing phase. But not necessarily… I am always especially happy when a beta tester comes up with a great idea which then makes its way into the game. But beta testers shouldn’t get their hopes up too much: 99% of all ideas are never implemented – usually not because they aren’t any good, but simply because we lack the time to act upon them. For time is a rare commodity, and the beta testing phase comes at the very end of the development process; at this point, it is really too late for any major changes. But these suggestions might at least make it into an add-on or a sequel at some point in the future. (Just to nip possible discussions in the bud: absolutely nothing has been decided along those lines!)
Now you know why we rely on beta testing: for balancing, bug identification, compatibility tests, general feedback and ideas/suggestions, as well as – to a limited extent – for stress tests, i.e. server hardware and software capacity testing.
But what exactly happens during beta testing? What do I have to do to participate?
I am going to answer these questions in part two.
Last Updated: July 7, 2008