Dolby Laboratories Amplifies Regression Testing

Dolby Laboratories Amplifies Regression Testing

Sound Pioneer Finds “Peace of Mind” with Risk-Based Approach

Published: 14/May/2018

Reading time: 8 mins

In 6 months, Dolby reduced their regression test cycle times by 75% and production defects dropped close to zero.

 

Dolby Laboratories, a pioneer in advancing the science of sight and sound, would not be synonymous with delivering the best entertainment experience without a strict adherence to quality assurance. If you’re enjoying the latest blockbuster in a cinema theater, watching a binge-worthy show on a television, or even listening to a conference call at home or on your smartphone, Dolby likely had a hand in the technology behind it to make it look and sound great. Dolby technologies have made it the leader in creating entertainment experiences that reach billions of people worldwide through partnerships with storytellers, businesses, and consumers.

In 2016, Dolby’s Enterprise Applications area sought an equally stringent level of quality assurance for its business process testing when inconsistent testing coverage and a lack of testing standardization revealed risk coverage gaps and lengthy integration cycles that had the potential to introduce risk to mission-critical processes, including its customer-facing Dolby Reporting Portal, which its licensees use to report royalties.

“In 2016, we put forward our vision and mission statements to include continuous quality assurance by developing, documenting, and maintaining a comprehensive change management program to include test automation,” says Mamatha Mitr, Senior IT Manager of Dolby’s Enterprise Applications. Mitr heads up the strategy for Testing Center of Excellence (TCoE), a byproduct of Dolby’s efforts to introduce standards to its testing landscape.

Mamatha Mitr

Mamatha Mitr

Senior IT Manager,

Enterprise Applications,

Dolby Laboratories

One of the greatest intangible benefits was when we surveyed everyone who worked with us in terms of automation or technical tasks, we heard consistent feedback that they now had peace of mind with automated testing.

— Mamatha Mitr, Senior IT Manager, Enterprise Applications, Dolby Laboratories

A focus area for Dolby was to provide end-to-end coverage for what Dolby refers to as its financially significant (FINSIG) applications, to include several SAP ERP Financials applications (for order-to-cash, sales and distribution, and procure-to-pay processes) as well as SAP SuccessFactors Employee Central. These are all key test cases for Dolby’s Sarbanes-Oxley audit compliance.

“When we sought and selected a tool for automated testing, we wanted it to lend itself to the different technologies in our landscape, which includes SAP software,” Mitr says. “We have a large SAP footprint in terms of our FINSIG applications.”

Dolby Theatre

The Dolby Theatre has hosted the Academy Awards ceremonies since 2002

Gap Compression with a Risk-Based Approach

One of Dolby’s requirements was that an automated testing tool follow a risk-based approach, which, in addition to diminishing risk through early and end-to-end testing automation, also estimates the potential damage that can occur based on requirement levels for the affected software.

With a risk-based approach as a key requirement, and based in part on a Gartner assessment, Dolby opted to partner with Tricentis. The Tricentis Tosca risk-based platform helps to optimize test cases with its Linear Expansion methodology that reduces the number of test cases while optimizing risk coverage. The software enables customers to assess an aggregated risk coverage from its automated testing requirements.

“The reason Dolby selected this solution was because it is completely risk-based,” Mitr explains. “Its two key attributes are frequency and damage class. Frequency is a measure of how often the associated item (requirement, user story, etc.) is used and the damage class is the potential impact to the business — the risk or damage that could result from the failure of the associated item.”

In June 2016, Dolby began a pilot program by selecting test cases that spanned various applications and workstreams. The scripts for customer-facing applications such as Dolby Reporting Portal were in the mix. After the 90-day pilot program, Mitr and her team categorized the benefits into two buckets, tangible and intangible, to determine return on investment (ROI). Metrics for the former included hours saved per testing cycle, number of reported defects resulting from testing automation, and the percentage of automation coverage versus the coverage during manual test cycles. “That was an important metric,” says Mitr. “When testing was manual, our business system analysts weren’t running all of the tests because of limited time and resources. They only ran the portion that touched the code, and that prevented us from having end-to-end test coverage.”

This scenario echoes the traditional challenges facing an enterprise that tries to mirror a risk-based approach with limited resources; the workaround to mimic risk-based automation is either to test exhaustively or to test at random. By prioritizing test coverage based on frequency and potential damage, the Tricentis Tosca Risk Optimization tool assigns an absolute weight in its Linear Expansion algorithm to continuously ensure that testing activities are aligned with business risk objectives.

Mitr says that running test case scripts for customer-facing applications during the pilot led to a consistent drop in the number of incidents, measured by a reduction in tickets, through three monthly release cycles. The growing TCoE team, which started with a handful of full-time employees, also proved ROI with significant savings in hours per testing cycle.

As for the intangible benefits resulting from the pilot program, Mitr says that Dolby quickly developed a standardized approach for all of its testing deliverables. “We came up with a robust maintenance process,” she says. “Subsequent to doing automation for a particular workstream, we were able to document all of that for ongoing maintenance. One of the greatest intangible benefits was when we surveyed everyone who worked with us in terms of automation or technical tasks, including functional consultants and system managers, we heard consistent feedback that they now had peace of mind with automated testing.”

Dolby Cinema

Dolby has over 140 Dolby Cinema sites open around the world

“Surround” Coverage

In September 2016, Dolby began a deep dive into Tricentis Tosca. Before expanding its regression test case library, however, the company wanted to establish a firm footing with a regression test bed feathered with documentation for the test cases run during the pilot. With the Dolby Reporting Portal, for example, the TCoE team wanted to verify that all the test steps were covered. This was an important task, according to Mitr, because the test team did not necessarily have the functional experience to be well versed in the application.

“First and foremost, the TCoE team got a functional walkthrough of the technology and the application,” says Mitr. “We had a requirements-gathering session to ensure we understood the documentation. From there, we started a sprint planning exercise coupling it with a detailed charter and a scope deck.”

With the planning exercise, the team matched its business risk objectives with the Tricentis Tosca Risk Optimization tool based on frequency and damage class. A regression test bed and robust documentation set allowed Dolby to then branch out into more of an agile methodology. “Once we went through a few release cycles, we measured ourselves to make sure that we met our success metrics, and then we moved on to the next workstream,” Mitr says. “We had a varied methodology — agile, waterfall, and hybrid — and we catered our planning accordingly to match the methodology of the workstream.”

 

Dolby Laboratories

 

Dolby Laboratories

Headquarters: San Francisco, California

Industry: Entertainment and communications (audio, video, and voice technology)

Employees: 2,122 (September 2017)

Revenue: $1.081 billion (2017)

Company details:

  • Founded by Ray Dolby (1933-2013) in 1965
  • Brand anthem: On a quest for the spectacular
  • The Dolby Theatre has hosted the Academy Awards ceremonies since 2002; Ray Dolby received a star on the Hollywood Walk of Fame directly in front of the Dolby Theatre
  • Dolby Laboratories has received 10 Academy Awards and 14 Emmy Awards
  • Dolby has over 140 Dolby Cinema sites open around the world
  • Netflix, Amazon, Disney, Tencent, Apple, Warner Bros., Samsung, Sony, and more have chosen Dolby Vision HDR and Dolby Atmos immersive audio to deliver entertainment experiences

SAP solutions: SAP ERP Financials, SAP SuccessFactors Employee Central, SAP Supply Chain Management, SAP Fiori, SAP Solution Manager, SAP Concur solutions, SAP governance, risk, and compliance solutions, SAP Analytics solutions, and SAP Business Warehouse powered by SAP HANA

Third-party solution: Tricentis Tosca

Crystal-Clear Workstreams

By September 2018 — the close of Dolby’s fiscal year — Mitr says the goal is to have automated regression testing for at least 50% of its FINSIG applications. As of April, the TCoE team had already reached the 40% mark. “I estimate that we’ll exceed our target and be at roughly 65% by the end of the fiscal year,” she says.

According to Mitr, automated testing has allowed the TCoE team to catch defects prior to impacting a workflow, and the number of incidents — meaning the number of tickets issued for an incomplete regression testing cycle — is drastically falling.  “That number is now close to zero for many of the workstreams, and the number of hours saved per testing cycle is significant,” she says. “The precise number of hours saved depends on the workstream, but it’s more than 75% of the time saved for business system analysts and other functional leads.”

Another key success metric was the volume of end-to-end coverage. Prior to implementing Tricentis Tosca, limited resources meant that Dolby relied mostly on spot testing, only allocating the resources for complete regression testing on large-scale, mission-critical projects.

“Now, we can run the entire regression test suite immaterial of where the change is, so the amount of coverage we get is close to 100%,” says Mitr. “Running test cases that are frequently used and are critical to our business is very important and it would have been close to impossible for us to get anywhere near 100% without test automation software such as Tricentis Tosca.”

 

How Tricentis Helped Dolby Laboratories Reduce Its SAP Testing Burden

 

How Tricentis Helped Dolby Laboratories Reduce Its SAP Testing Burden

With an industry-leading platform for continuous testing, Tricentis is recognized for re-inventing software testing for DevOps. Through risk-based testing, scriptless end-to-end test automation, and extensive SAP technology support, Tricentis breaks through the barriers experienced with conventional SAP software testing methods. The Tricentis Tosca solution helps cut SAP regression testing from weeks to minutes, supports the full testing of end-to-end SAP business processes, and helps identify SAP test cases with the highest risk coverage.

When Dolby Laboratories was looking for an automated testing solution for its end-to-end SAP business processes, it selected Tricentis because of the core risk-based approach of the Tricentis Tosca Continuous Testing platform. Tricentis has proved the right choice for Dolby, with comprehensive, end-to-end testing of its SAP business processes. Instead of spot testing, Dolby now gets to run the entire regression test suite with Tricentis Tosca — helping the company get close to 100% coverage.

In its recent “Critical Capabilities for Software Test Automation 2018” report, Gartner reviewed 10 different software testing solutions — giving Tricentis Tosca the highest score for enterprise end-to-end testing. Tricentis Tosca features mature and robust SAP test automation capabilities with fully automated accelerator packages designed to help you fast track your SAP test automation projects. For more information, visit https://www.tricentis.com/gartner-sap.

Find other insightful content from Tricentis on their Vendor Showcase

Tricentis logo

 

 


More Resources