Bungie explains how a recent Destiny 2 bug was caused, discovered and fixed

Contrary to popular belief, video games aren’t fixed by yelling at the nearest developer to push the magic “make game work good again” button whenever a problem arises. If my time with Media Molecule’s Dreams has taught me anything, it’s that every video game is built with a series of delicate strands whose tensile strength can snap at any given moment and snowball out of control into an avalanche that wipes away a whole lot of work in the process.

Bungie had just such an issue this week with Destiny 2, a game that oftentimes runs like a dream and then drops the mother of all bugs on players who are all too keen to make good use of that exploit to power up their Guardians. Remember the 2014 loot cave incident from the first game? Good times. This week’s latest bug was a bit more serious though, as players began losing extremely valuable currency and rare items.

A fix was of course on the cards for Bungie’s emergency response team, one that the studio actually took time to talk about and explain how things got out of control in the first place. The relevant bits are copied below from Bungie’s site, but the TLDR is that a few small problems created a big dilemma, whose fixing resulted in another problem surfacing which somehow resurrected the first bug that was responsible for the mess in the first place. It’s kind of like the Laurel and Hardy of programming:

Several months ago, players reported that quest log sorting wasn’t working properly, and we wanted to fix that. The team investigated and found that the clean-up process was resetting the timestamp on a subset of quests, which was breaking chronological sorting. We decided to fix this by disabling the timestamp-resetting behavior for quests. That fix was conceptually reasonable but, through subtle side effects, it ended up disabling too much of the clean-up process.

The net result was that the game calculated the wrong cap quantity for stacked items (such as currencies and materials), which caused items above the cap to be lost. We knew this code was critical and, per our typical process, we had two domain experts provide code reviews for the change – but sadly, we didn’t spot the bug.

A few days later, our internal test teams caught this issue. However, we incorrectly concluded that it was caused by a tooling failure with debug workflows we use for testing, and not an actual bug within the game. Having dodged all our diligence, the issue went live in 2.7.1.

And that’s why Bungie dropped a server reset on Destiny 2 for its hundreds of thousands of players on console and PC. And the tens of people on Google Stadia:

Fast forward again to today, February 11th, when we rolled out the 2.7.1.1 update coinciding with the launch of Crimson Days. After launch, some of the WorldServers once again crashed on startup because of a high volume of servers starting simultaneously. Once again we manually restarted those servers and thought everything was fine. We were wrong.

Unbeknownst to us, this crash resulted in those WorldServers not applying the previous character data corruption fix. This meant that a small percentage of WorldServers were running the old code and the bug that was corrupting character data.

I don’t think anyone is naive enough to believe that game development is easy stuff, but it’s often fascinating to read just how complex it really is when you’re given a tour of the trenches. Destiny is no stranger to these game-breaking bugs, and in all likelihood, something along these lines will probably happen again in the future. When it does and after it’s fixed, it’ll be fascinating to hear how Bungie found a solution yet again for their cosmic sandbox.

Last Updated: February 13, 2020