Costliest software bugs of 2017 & how to avoid them...

Posted by Mark Tennant on Friday 2 February 2018

The worldwide demand for online interaction doesn’t show any sign of slowing down.  Internet users, websites and applications continue to grow as more and more of us work, play and manage our lives through a digital interface.

This is great news for digital agencies – their customers are constantly looking for new ways to interact, while providing a digital experience that sets them apart from their competition.


Pushing technology boundaries can bring a daunting challenge when it comes to delivering a high-quality experience.  With the growth of devices, operating systems and browsers combined with the increased complexity of customer requirements, it’s no wonder that many agencies are only able to provide QA for a small fraction of what their services will be running on in the real world.

This leaves applications and websites with bugs that potentially are not discovered until they are in the hands of users…. risking brand reputation damage for the digital agency and their customer, plus potential lost revenue if users are unable to purchase or complete their online journeys.


Top 10 Bug Count

At BugFinders we pride ourselves on being the world leading provider of scaled on-demand Digital, UX and Security Testing. 

Our clients ranging from the largest enterprise brands to small start-ups all have the same desire to give a high quality online experience.

In 2017, through our global community of professional testers, we collected a vast amount of rich, valuable data from thousands of bugs identified for hundreds of our clients.

Each time a bug is found, we verify its true impact to our client.  This data-set provides us with a deep insight into the nature and impact of the bug, enabling us to put a potential value on the likely negative customer experience.

For this report, we picked a cross section of functional tests from various sized projects…from large ecommerce sites to small brochure-ware sites. 

When we report to customers the bugs we find, we categorise the issues into 3 key areas of severity, illustrated in the table below.

table 2c.png


The Top 10 Costliest Bugs 2017

So, to the top ten…!

The table below provides a short description of the bugs we found.  In many cases, we identified similar bugs across several websites and applications from different customers.  The impact cost is calculated based on a combination of two things:

  1. BugFinders has collected data from clients over the last few years, specifically how fixing of issues has improved conversions. From this data we have been able to identify the impact of specific issues categorised in our platform.
  2. This specific data was then surfaced when we worked with the client to understand their traffic on web and mobile platforms, combined with conversion rates and average basket size and fail data.

We cross reference the two datasets and can conclude the impact of the issues.  For singular issues such as these, we also verify directly with the client. (The data has been rounded). 

table 2_3.png

The impact of fixing these bugs as early as possible can make a massive difference to development costs…A study from Systems Sciences Institute at IBM shows that leaving a bug unfixed until production can cost 100 times more to fix than if it was discovered and fixed at the design stage.

table 1c.png

The cost of doing nothing can be astronomical.  In a presentation from UX guru Craig Sullivan, he highlighted some of his real-world cases showing resulting loss of revenue due to bugs remaining unfixed...

Table 3a.png 

What you can do to avoid this?

All of the issues identified in this report have been missed through in-effective testing, either functional or usability, so the big take-away from this report is that #qualityiscrucial and therefore it is vital that new software development is tested thoroughly, both from a functional perspective, but also from a user perspective.

Why was the functional testing not done on these sites?

Quite often, companies will ‘test’ their website.  However, today’s consumer-led economy means that it is very difficult to balance the rapid development and push for quick releases against the need for quality without testing holding up the project for multiple months.

For this reason, many teams find that they need to compromise, and this normally comes down to ‘test more’ vs ‘release on-time’.  Release normally wins this battle as it is the tangible.

So, release trumps testing, meaning that insufficient testing is done.  However, even with sufficient time, any regular sized test team would struggle to deliver the coverage required on the plethora of today’s devices and browsers.


Why is user testing not done on these sites?

It may have been, but it is unlikely it was iterated and revisited through the process, and in many cases companies do more than a cursory check with friends/family etc, rather than detailed analysis.

It’s really missing a trick not listening to customers of the product.

So here are our top 5 tips you can do to avoid a problem like these: (in no particular priority order)

  1. Use a localised team

If you are going to launch a website outside the UK, use a test team in that country to test it.  Combine this with usability testing and you give yourself the best chance to success.  Local people will quickly feedback on issues that you don’t see – these can be big or small. 

  1. Cover lots of devices

Test on a wide variety of devices to ensure that the users of the site will be able to use it correctly.  80% of traffic comes from 152+ devices. That’s a large number to test.  You need to think about a minimum of 30 mobile devices combined with 7 browsers variations as required.  12 people should be enough to cover that – bearing in mind that 1 person can cover a maximum of 3 devices before they become ‘browser blind’.

  1. Be diligent – script

It’s easy to jump right in and start testing.  Many issues are not picked up by this method though.  Your team should have detailed and structured scripts that cover the relevant scenarios in a repeatable way.  This will ensure that all home-delivery options, quantity fields etc are checked to avoid embarrassing and costly issues later.

  1. Don’t forget the user

It’s all too easy to do a bit of user testing at the end of the development cycle, when the eventual user of your site should be put front-and-centre.  Completing early usability testing with real customers is an excellent way of doing this.  The customers should also be consulted through the process to ensure that the final product meets their needs. 

Usability labs are a great way to get customers views.  However, bear in mind that they are synthetic situations with people watching them.  Look for options where customers can complete their test from home or in their own relaxed setting.

  1. Allow enough time

The final and probably most important point.  If you rush your testing, you are likely to miss things.  Despite how painful it is, you need to give your team time to cover everything properly.  If the business disagrees, then build a business case to increase the team size, or look at outsourcing options, with a focus on speed.

These are the basics but they will put your launch in a good place.


How can BugFinders help?

At BugFinders, we work closely with our clients’ development and project teams, finding bugs early in the dev cycle.

The scale of our community; 55,000 professional testers across 149 countries means that we can achieve a huge amount of testing in a very short space of time…. typically, the equivalent of months’ worth of testing in just a day or more.

Added to that, with the large number of testers on projects, we achieve a huge amount of device, operating system and browser coverage.

We report bugs via our client portal.  Bugs are verified to ensure they can be replicated and are not duplicated. Step-by-step instructions on how to reproduce the bug, along with screenshots and videos of the bug in action are also available….and if you want to export to another bug tracking environment such as Jira, then that option is available too.


In summary…

By bringing in the scale of the BugFinders’ community at the start of your digital projects, you can be assured that your website and application will be thoroughly tested for the highest to the lowest impact bugs, across all the devices, browsers and operating systems your customers will be using in the real world.  

Avoiding the nastier impact bugs head on will save a lot of problems later and protect the revenue and brand reputation of you and your customers.

New call-to-action

Topics: Featured