Every real software developer I know knows the importance of testing their software in a production environment because there’s always that slight chance that something may be different from your development environment. This is why in enterprise software you have a development environment, a staging environment, and a production environment. The development environment is as you might expect, is where things are tried out. However staging is where things get different, this is where your environment should replicate as closely as possible your production environment. Although it’s not always entirely possible to completely reproduce it, the closer the better. And production is your live environment.

Normally for LandlordMax, when we’re developing we run it directly from our IDE (Integrated Development Environment), which is our development environment. If everything works here, it should work in the production environment. You always want to test it in staging though, just in case, which we do. Also, to remove the chances of errors, what you also want to do is automate the process of taking your development application and “productizing it”. That is to say, you want to create something called a build script which will automatically create everything for you, removing any chances of human errors in creating the final product.

Once that’s done, at least for us here at LandlordMax, you have an installer created. You then try the installer. You then run the software to make sure everything is still working the same (hopefully someone else will do this to avoid any biases while testing). It’s a pretty simple process.

However with version 3.11 of LandlordMax we re-learned the valuable lesson of just important it is to test and re-test the software after it’s been productized. Yes we normally run it through it’s gamut of tests, but for some reason we missed one simple test case with version 3.11 that where we just a few minutes ago released version 3.11a tonight to correct. The bug, which worked in our development environment, didn’t work in staging, or production. Unfortunately it wasn’t caught until just recently by two of our customers, which I personally thank you for letting us know so quickly. The good news is that because they contacted us right away, within 1 day we have a new version with this fix available!

So what happened between the two environments, shouldn’t they be the same? Yes they should be the same, there should absolutely be no difference. However this was not the case with version 3.11. But before I get into the details, let me step back a minute and explain what happens in our automated script (which by the way was not at fault for those of you thinking ahead).

Our build script consists of several steps. First it gets the latest code from our version control, compiles it, and creates the appropriate objects. Once this is done, it grabs all the appropriate resources (for example images, etc.) and puts them into their appropriate objects. After that, it takes these objects and “Obfuscates” them (if you’re interested there is a famous contest held each year to see who can create the most obfuscated code called IOCCC). This is important for us as a company because it takes the compiled objects (intellectual property) and makes it extremely hard to reverse engineer. Nothing is impossible, but I can tell you from looking at the code after the obfuscater has run through it that I can barely understand it myself! Anyways, once this is done, it takes all the objects, and runs another script which then creates the installer (LLMaxSetup.exe) which you download and install. That’s our build script. Not too complicated, but not too easy either (I’m glossing over some of the details).

What happened with version 3.11 is that we updated the software that obfuscates LandlordMax. We’ve done this many times over the last 4 years, but this time for some reason it changed the way it reads the configuration file. Not significantly, but enough that it caused this issue. Without being too technical, in our code we overwrite a few libraries we use (for example we changed how the report printout calculates the report totals by adding a clause for “NSF” checks, and so on). Now when LandlordMax tries to call this library, our code is intercepted beforehand and is used rather than just going directly to the library’s code. Nothing major, this is called “overwriting” in programming. However, one key feature in overwriting code is that you need to use the exact same names!

Well to our dismay, the configuration in our obfuscater no longer accepted the wild card we used in the past. It overwrote the names of these classes, and therefore because the names weren’t the same, our code didn’t intercept the library’s code. And because of this, the software didn’t know how to handle “NSF” or other little tweaks we added! Hence all the totals were $0.00 on the printout.

So between our development environment and our staging environment something had changed. Something that wasn’t suppose to change, had never changed in the past, but suddenly now did. This is why it’s important to thoroughly test your final production product. We did, but we somehow still missed this issue and for this I apologize to all our customers. We have therefore just released a new version (version 3.11a) that you can download and install to resolve this issue right now.

PS: As an aside, I’d just like to say this was a fairly difficult bug to track down. This is a bug that worked in every development environment but not in our staging or production environment, regardless of the settings and configurations! It also illustrates even more the importance of properly testing in every environment!

Discuss (3)