Posted Date: 3/3/2013
The Matrix of Mobile Testing
By Darren Madonick, Mobile Testing Evangelist, Keynote
For desktop-based testing it’s a no-brainer: use object-based scripting to maximize reuse across platforms/browsers. In today’s mobile world it really isn’t that simple. There are many different platforms, OS versions, form factors and carrier/manufacturer customizations. Multiply that by mobile web, native app, or some hybrid in-between and you’ve got yourself a healthy testing matrix — a daunting task for even the most skilled automation engineer.
In order to tackle this problem, an automation engineer cannot simply look at it from a “one size fits all” perspective to create a set of objects and re-use them across all combinations of platforms. For example, there are fundamental differences in how an app behaves on iOS and Android, even with something as basic as a back button has its quirks. Although these fundamental differences can be grouped together as a step or action, they are unique enough to not be able to simply share an object between the two OSes.
In some cases with mobile testing, you may be able to get to the object-level, however this usually requires that you instrument your app or test on an emulator. While this fulfills a piece of your testing matrix, you will probably need to seek a couple tools to get this done across all platforms. In other cases, the content you are testing might be HTML-based and you can test by WebKit profiling. Again, part of your testing matrix is fulfilled, however you aren’t quite there.
This may be enough to satisfy a short-term goal, but at some point you need to be testing on real mobile devices. In order to truly automate on mobile, your mobile testing “utility belt” needs to be designed in such a way that allows for testing by object when possible, element when possible, and also be able to quickly fall back on text or image verification in order to satisfy all areas of your testing matrix and assure the highest quality of your mobile product.
Having the flexibility to be able to choose how to get the testing done is paramount since as an automation engineer, you very rarely have a say in how a particular app or mobile web site is developed. The job requires you to sometimes understand functionality without necessarily being privy to the construction, and there is always a tight timeline to achieve results. The right tool for the job is a tool that takes all of this into consideration, and provides a platform to consolidate all of these different types of testing approaches.
Start With This Question
The first step is to determine the type of app you are testing. Is it fully native, fully web or somewhere in between?
- If it’s fully native, you may be able to get to some objects on a per platform basis, but you will probably be falling back on text and image based verification, especially if you are trying to go cross-platform.
- If it’s fully web, a lot of testing can be done up front in a WebKit profiler. When it comes to real devices, element-based testing can be done cross-platform if you want to instrument, or you can fall back on text and image verification.
- If it’s somewhere in between, you’ll need to mix and match.
The second step is to find which pieces or steps of your test cases are reusable between each other, and can accept parameterization to fulfill the task. For instance, automating the selection of an item or link on your main screen of your app or landing page: maximize reuse by engineering a parameter to accept different values, and reuse it across each test case. Although you may need to individually determine what type of verification you will use to achieve this on a per platform or device level, you will save time in the long run when you write additional test cases.
The third step is to then group those pieces or steps together by device screens or pages. This way, as you write the test cases you have an organizational structure that is easy to identify by where you are within the app or site and where you need to navigate to next.
Following these steps will provide a structure that can be grown to accommodate new features within an app or new sections within a mobile web site. As mobile devices become easier to automate against, this structure can easily adapt to emerging technologies that allow for greater reuse across platforms.