Robotic Test Automation: Transforming Regression Testing in SAP

Jim Dugger, Senior Technology Evangelist, Basis Techologies, discusses robotic test automation with Matt Shea of SAPinsider. Below is a transcript of the conversation.

Matt Shea, SAPinsider: Hello, and welcome to the SAPinsider podcast. I’m excited to be joined today by Jim Dugger of Basis Technologies to discuss robotic test automation and how it can transform regression testing in SAP. Jim currently serves as a Senior Technology Evangelist for Basis Technologies in the Americans and Asia-Pacific regions. Jim has nearly 25 years of experience in the software industry, including roles in the development, product management, product marketing, and sales at start-ups and large companies. Jim’s interests in software include business process, systems management, and software development topics. Jim has been a regular speaker and technology evangelist at industry conferences including CA World, IBM InterConnect, SAP Tech Ed, SAP SAPPHIRE, and other major events. Welcome, Jim.

Jim Dugger, Basis Technologies: Good morning, and thanks for having me on.

Matt: Would you be able to describe the current state of SAP testing?

Jim: Sure, this is probably not going to surprise a lot of our listeners. When we got into looking at the SAP testing market and what customers were doing we discovered that they were broadly manual. One of the things that we’ve been privy to with our DevOps practice and our DevOps solutions is understanding a very broad range of enterprise customers, how they use SAP, how they manage SAP. Obviously, we’re at the center of that because of the DevOps practice and when we started looking at the testing problem, here are billion-dollar organizations where literally everything that’s done to ensure SAP works is entirely manual; logging in manually, running manual tests, doing manual screen compares.

It’s not good in terms of effectiveness and efficiency. There’s a couple of other problems with it as well. But most of our customers admitted to the fact that they simply have more to test than they can get done. So even though they are applying manual labor to this problem and doing it in a traditional way, they’re also having to select down what work they gets done. So, it’s this exercise of managing risk, where you hope you test the things that have a probability of changing or causing an issue for the business and you don’t test the things that you hope won’t be an issue. This is obviously not ideal for a lot of businesses but it’s a reality they face. That’s the current state of SAP testing. 

Matt: So what’s changing in SAP customers that drives the need to change?

Jim: I think this can be described in one word: Business. All organizations are subject to, or at least hopefully have, great growth and acceleration. With that of course comes the pace of change and the need to be able to bring innovation to the business and the ability to support customers in unique and different ways. Whatever that business does, there’s no doubt competition in their market and they need to be able to keep pace with what the market demands in terms of capability and delivery for their customers. The other aspect of this that drives change is often regulation. So you have ordinary law and change in Congress, or other bodies of states that require businesses to stay compliant and change the way they operate or do things in order to simply stay around. So these are all things that we see every day in businesses that require change to the way that they use SAP because after all SAP supports their business. They’re all drivers of what’s creating this constant need to be delivering new software, new capability, and of course test that and make it work.

Over time what’s happened in these companies… I like to think of it as a cost iceberg, right? You’ve got a little bit above the water and a whole lot below the water. Well that little bit above the water is the change. If we look at what a customer does in any given iteration or any given update of how they use SAP, they are bringing new functionality, they are bringing change, they are bringing new capabilities. But that change is relatively modest in size compared to that huge body of what was already in place for the organization and what they depend on day in and day out.

Taking this from a test perspective, you have a little bit of new functionality that you need to test but you have a giant body of regression which is essentially all the features that have ever existed in that capability, in that application in SAP that the business uses every single day and will continue to depend on even after that new update is delivered. This combination of constant change, acceleration of change, but also this huge exposure in terms of risk and existing functionality that needs to be validated every time you’re doing an update, well that creates a lot of cost, and a lot of opportunity for something to go wrong for the business; a lot of risk, and businesses must be able to effectively address that huge body of testing. So that’s definitely an exposure that when we went and looked at this problem for our customers they all admitted to – they may not tell somebody in public about it but they certainly are well aware that it’s a big problem and effectively and comprehensively testing everything that the business depends on each time that there’s a change is an exposure that almost every customer faces.

Matt: Automation isn’t new, so what’s new about automation?

Jim: Automation came around specifically because obviously manually testing doesn’t give you the coverage you need and it’s not fast and efficient. So the idea was, “Hey, let’s create a solution that will allow us to programmatically execute the things that normal people do in the daily course of business.” And because it’s at computer time instead of human time, we’ll be able to get this great body of testing accomplished because it will execute so much faster than doing it manually. And that’s true, with one caveat – if you can create all the automation.

So essentially what’s happened in practice is we’ve taken manual testing and instead of having a whole bunch of people that know how to do the business process go practice that as a test we’ve hired a whole bunch of technical people that have to go then learn the business process to create essentially a testing robot or this automation script that then performs what the human user would have done and then you can finally run it. Of course, scripts break and there’s test data problems and there’s all these other issues that come along with regular automation. Effectively what the industry has done is just change who does the work. It’s still the same amount of work, it’s just who’s doing it? Now it’s a highly technical, highly skilled individual as opposed to the regular daily business user that knew how to do the business process doing it.

I don’t think we’ve improved anything obviously with regular automation.

What has changed is industry is now looking at the automation problem saying, “Hey we only got it half right.” We did the execution part of it OK, but we forget to look at automating all the upfront work so everything that happened with automation is we just moved who did the work as I was describing. So what if we could automate the work itself? What if we could look at the whole lifecycle of what’s required in testing and solve for the whole problem. The idea would be a robot that could learn by simply observing what the human user does. That’s important not only at the individual level but of course comprehensively at the enterprise level. Then that robot could understand how to produce a test that would be accurate and true to the way that the business really depends on SAP without intervention of humans, and without intervention of highly skilled and highly specialized QA specialists that write scripts and things like that. Basically, it automates all that work.

With all of that work then automated, you go execute the automated test itself in the QA environment after the changes that are being proposed for production have been brought in. And we look at solving this whole problem comprehensively.

When we looked at this market this is what we knew had to happen. We literally had to produce a solution that would not only automate testing, but automate the creation of the tests themselves. You had to solve the whole thing at once. So that’s really what’s new about automation is industries realized that the first attempt at this only solves for half of the problem, and now we had to go back to the drawing board and say, “All right, how are we going to completely eliminate the need of this very highly skilled, this very difficult, very time-consuming work of creating automation and do it all for the customer instead, so that’s the big change.

Matt: Why had this not been done before?

Jim: I think there are a number of reasons we’ve not seen an approach like this before. The first one is probably historically related to automation itself. I mean the idea was that all we needed to do was automate the playback of the test, if you will. As we’ve talked about already, that wasn’t the whole part of the story. We didn’t really innovate anything by moving who did the work and what kind of work they did. That was the first part of it, is just historical.

The second part of it is, we’re now getting a much more comprehensive understanding of how people depend on systems and how systems are used. Especially for SAP; here’s an environment that not only has traditional desktop human users performing business process, but it’s also heavily integrated and heavily used as a system of record and the back-end for transactional processing, meaning that web front-end and other applications are using the services that SAP provides in a programmatic way through API calls and those kinds of integrations and not just as a standard user interface application that needs to be automated.

Here again we have a much bigger story that’s been traditionally accepted by testing; now we have to understand not only how do human users depend on the application, but how does the whole enterprise and every part of the enterprise depend on SAP and how business gets done in it?

The shift had to happen technologically instead of thinking about, “Well let’s just automate a robot at the desktop” now we have to automate a robot that exists fundamentally in SAP itself. It has to be able to support all the API calls. It has to be able to support all the interactions. It has to be able to support every user interaction. It has to be able to do everything that the enterprise does when they touch SAP. That’s a tough problem. We’re here to innovate and industry is constantly improving and getting better, and this is a great example of where industry has now looked at the testing problem and said, “it’s got to be much more comprehensive than it’s been traditionally”

There is a new minimum standard that if you want to be successful testing SAP and ensuring that your business is going to stay durable and reliable, you have to do it this better way because you’re not getting the coverage you need with traditional automation, it’s certainly too expensive to use and this kind of approach, this robotic testing automation approach, where we’re automating both front and back-side to the testing problem and looking at not only this client side or the end user side but also the server side of SAP testing, well this large comprehensive story is what needs to be done now in 2017-2018 versus where we were say even a few years ago. So that’s why in my opinion why it hasn’t been done before. 

Matt: Basis had a big announcement at TechEd in this space with Testimony. What is unique about Testimony?

Jim: Well, no shortage of hints in our conversation so far obviously. Testimony was the culmination of our study into this problem of automated testing and setting out to solve the problems that we had observed in industry with being successful at testing SAP comprehensively.

Testimony has a couple of unique properties. It’s first property is that it does not require any scripting, any test case creation, or any test data management. Those are completely unique features in the test automation space because every other automation solution up until Testimony required the end users to do exactly that in order to be successful with automation. Instead what we do is instrument SAP itself and that gives us the ability to understand exactly how the business uses and depends on SAP, not only at the end user level but at the API and server-side level as well. Then we can take that observation that we’re able to make in a customer’s production environment and turn that into a set of test cases that will automatically ensure that any change that they’re bringing into production will not introduce a regression or the elimination of features that they’re using today that they depend on. We can do that automatically for the customer without all the traditional overhead of test case automation, the things that you have to do using traditional automation solutions.

It was obviously a big announcement in terms of the change in how regression testing can be performed in SAP, and a nice complement to our other solutions – we’ve been in the DevOps space for a very long time so we’re very good at helping customers bring higher quality code through the development process in SAP effectively.

Testing becomes part of that natural iteration cycle and we were looking for a way to complement our existing DevOps solutions with this additional testing capability to help our customers fully automate, become fully agile, be able to develop in SAP as effectively and with the same kinds of techniques and methodology that they’re used to using over on the JAVA or the C++ or C # side. And bringing that whole story into the SAP fold. Quite a few things there, but we think that we’ve got something very unique. The response was overwhelming at TechEd. Personally, I was three deep at the booth every minute that I was there, so super excitement in industry about it and we’re excited to see how much we can do for our customers with it.

Matt: How does Testimony help SAP users move toward agile methodology?

Jim: We were just hinting at that in the last question; it’s actually a very complicated question to ask about SAP in general. Those that have some background in development in SAP will know that the basic waterfall model that SAP uses is rather rigid in the design of the application itself. So when you build code in SAP, you do it in a shared collaborative environment that doesn’t have integrated version control, it doesn’t have branching and those kinds of things that we’re used to maybe over on the C# or JAVA side of the house. It makes it difficult to adopt a team, agile, iterative kind of approach that we’ve seen in traditional software development.

There’s a whole other discussion we should have around DevOps and agile that we can have with our ActiveControl solution, but your question was about Testimony. Testing plays one small part of the overall agile story and that is “How do I, when I am doing a high rate of iteration, ensure that the changes I am making have not introduced regressions into the code base? Have not caused a new failure that I didn’t plan on because of that change? And how do I streamline the effort of making sure that every change I make gets fully tested, doesn’t break existing business practice, that it’s done in a highly iterative fashion?”

We designed Testimony specifically to be able to integrate in that agile environment and to be able to take part as a step in that methodology, in that iteration, and be able to provide a full regression test for every single part of the business even as customers are changing only small portions of it – back to that iceberg idea. So, it’s highly flexible, automatically updates its own regression library based on current business practice that it observes in production, doing all the things that are necessary to integrate with CI and CD in an SAP environment. Those are the things that we had in mind when we built Testimony, to support our customers that are adopting that kind of methodology in their daily practice. 

Matt: So what’s next for Basis?

Jim: A very good question. Obviously continue to support development of Testimony. We’re very excited about the product and continued investment in our DevOps solutions. One of the things that was clear to us at TechEd was just how much customers appreciated our approach to ensuring that agile methodology and DevOps technologies could be brought to SAP and could work very effectively with SAP, and combined SAP and external development projects. So we’re definitely going to continue investing there and we’re going to continue getting feedback from our customers on our tooling and our solutions and adding the features and capabilities that they need as we look forward.

Matt: Great. Thank you for your time today Jim.

Jim: Thank you very much.