Dolby Laboratories Amplifies Regression Testing

Dolby Laboratories, a pioneer in advancing the science of sight and sound, would not be synonymous with delivering the best entertainment experience without a strict adherence to quality assurance. If you’re enjoying the latest blockbuster in a cinema theater, watching a binge-worthy show on a television, or even listening to a conference call at home or on your smartphone, Dolby likely had a hand in the technology behind it to make it look and sound great. Dolby technologies have made it the leader in creating entertainment experiences that reach billions of people worldwide through partnerships with storytellers, businesses, and consumers.

In 2016, Dolby’s Enterprise Applications area sought an equally stringent level of quality assurance for its business process testing when inconsistent testing coverage and a lack of testing standardization revealed risk coverage gaps and lengthy integration cycles that had the potential to introduce risk to mission-critical processes, including its customer-facing Dolby Reporting Portal, which its licensees use to report royalties.

“In 2016, we put forward our vision and mission statements to include continuous quality assurance by developing, documenting, and maintaining a comprehensive change management program to include test automation,” says Mamatha Mitr, Senior IT Manager of Dolby’s Enterprise Applications. Mitr heads up the strategy for Testing Center of Excellence (TCoE), a byproduct of Dolby’s efforts to introduce standards to its testing landscape.

A focus area for Dolby was to provide end-to-end coverage for what Dolby refers to as its financially significant (FINSIG) applications, to include several SAP ERP Financials applications (for order-to-cash, sales and distribution, and procure-to-pay processes) as well as SAP SuccessFactors Employee Central. These are all key test cases for Dolby’s Sarbanes-Oxley audit compliance.

“When we sought and selected a tool for automated testing, we wanted it to lend itself to the different technologies in our landscape, which includes SAP software,” Mitr says. “We have a large SAP footprint in terms of our FINSIG applications.”

The Dolby Theatre has hosted the Academy Awards ceremonies since 2002

Gap Compression with a Risk-Based Approach

One of Dolby’s requirements was that an automated testing tool follow a risk-based approach, which, in addition to diminishing risk through early and end-to-end testing automation, also estimates the potential damage that can occur based on requirement levels for the affected software.

With a risk-based approach as a key requirement, and based in part on a Gartner assessment, Dolby opted to partner with Tricentis. The Tricentis Tosca risk-based platform helps to optimize test cases with its Linear Expansion methodology that reduces the number of test cases while optimizing risk coverage. The software enables customers to assess an aggregated risk coverage from its automated testing requirements.

“The reason Dolby selected this solution was because it is completely risk-based,” Mitr explains. “Its two key attributes are frequency and damage class. Frequency is a measure of how often the associated item (requirement, user story, etc.) is used and the damage class is the potential impact to the business — the risk or damage that could result from the failure of the associated item.”

In June 2016, Dolby began a pilot program by selecting test cases that spanned various applications and workstreams. The scripts for customer-facing applications such as Dolby Reporting Portal were in the mix. After the 90-day pilot program, Mitr and her team categorized the benefits into two buckets, tangible and intangible, to determine return on investment (ROI). Metrics for the former included hours saved per testing cycle, number of reported defects resulting from testing automation, and the percentage of automation coverage versus the coverage during manual test cycles. “That was an important metric,” says Mitr. “When testing was manual, our business system analysts weren’t running all of the tests because of limited time and resources. They only ran the portion that touched the code, and that prevented us from having end-to-end test coverage.”

This scenario echoes the traditional challenges facing an enterprise that tries to mirror a risk-based approach with limited resources; the workaround to mimic risk-based automation is either to test exhaustively or to test at random. By prioritizing test coverage based on frequency and potential damage, the Tricentis Tosca Risk Optimization tool assigns an absolute weight in its Linear Expansion algorithm to continuously ensure that testing activities are aligned with business risk objectives.

Mitr says that running test case scripts for customer-facing applications during the pilot led to a consistent drop in the number of incidents, measured by a reduction in tickets, through three monthly release cycles. The growing TCoE team, which started with a handful of full-time employees, also proved ROI with significant savings in hours per testing cycle.

As for the intangible benefits resulting from the pilot program, Mitr says that Dolby quickly developed a standardized approach for all of its testing deliverables. “We came up with a robust maintenance process,” she says. “Subsequent to doing automation for a particular workstream, we were able to document all of that for ongoing maintenance. One of the greatest intangible benefits was when we surveyed everyone who worked with us in terms of automation or technical tasks, including functional consultants and system managers, we heard consistent feedback that they now had peace of mind with automated testing.”

Dolby has over 140 Dolby Cinema sites open around the world

“Surround” Coverage

In September 2016, Dolby began a deep dive into Tricentis Tosca. Before expanding its regression test case library, however, the company wanted to establish a firm footing with a regression test bed feathered with documentation for the test cases run during the pilot. With the Dolby Reporting Portal, for example, the TCoE team wanted to verify that all the test steps were covered. This was an important task, according to Mitr, because the test team did not necessarily have the functional experience to be well versed in the application.

“First and foremost, the TCoE team got a functional walkthrough of the technology and the application,” says Mitr. “We had a requirements-gathering session to ensure we understood the documentation. From there, we started a sprint planning exercise coupling it with a detailed charter and a scope deck.”

With the planning exercise, the team matched its business risk objectives with the Tricentis Tosca Risk Optimization tool based on frequency and damage class. A regression test bed and robust documentation set allowed Dolby to then branch out into more of an agile methodology. “Once we went through a few release cycles, we measured ourselves to make sure that we met our success metrics, and then we moved on to the next workstream,” Mitr says. “We had a varied methodology — agile, waterfall, and hybrid — and we catered our planning accordingly to match the methodology of the workstream.”

Crystal-Clear Workstreams

By September 2018 — the close of Dolby’s fiscal year — Mitr says the goal is to have automated regression testing for at least 50% of its FINSIG applications. As of April, the TCoE team had already reached the 40% mark. “I estimate that we’ll exceed our target and be at roughly 65% by the end of the fiscal year,” she says.

According to Mitr, automated testing has allowed the TCoE team to catch defects prior to impacting a workflow, and the number of incidents — meaning the number of tickets issued for an incomplete regression testing cycle — is drastically falling.  “That number is now close to zero for many of the workstreams, and the number of hours saved per testing cycle is significant,” she says. “The precise number of hours saved depends on the workstream, but it’s more than 75% of the time saved for business system analysts and other functional leads.”

Another key success metric was the volume of end-to-end coverage. Prior to implementing Tricentis Tosca, limited resources meant that Dolby relied mostly on spot testing, only allocating the resources for complete regression testing on large-scale, mission-critical projects.

“Now, we can run the entire regression test suite immaterial of where the change is, so the amount of coverage we get is close to 100%,” says Mitr. “Running test cases that are frequently used and are critical to our business is very important and it would have been close to impossible for us to get anywhere near 100% without test automation software such as Tricentis Tosca.”