Real testers use real mobile devices
If you’re serious about mobile testing, you need to use real devices and here’s why.
Behind every good app is a good mobile strategy that accounts for the diversity of devices in the hands of users. The mobile user experience differs from device to device, with different operating system brands, OS versions, carrier networks and devices creating an endless possibility of use-case issues.
In a study done by Perfecto 63% of testers admitted that they could not test across all the different devices and OS versions. And that is just the tip of the iceberg when it comes to the challenges associated with testing mobile apps. Mobile experiences go way beyond atomic app functionality. An app also needs to respond quickly and give the customer what they are looking for. This requires rigorous testing of the app and supporting systems.
The trouble with simulators and emulators
When you use a simulator or an emulator, your computer’s screen resolution is set and you can’t rotate it easily (unless you are amazing at typing sideways). What happens if you “zoom out” to see the entire device leaving the question, “could I actually click that little object with a finger?”
Simulators also don’t accurately test CPU speed. When you use a simulator, you are using the CPU of the device you are running the test on. Most notebook computers run Intel processors, while mobile devices run Exynos chips with 5 to 8 cores or A6 to A9X chips with multiple cores, they are just different.
It is not reasonable to think that the speed and processing of an application in a simulator or emulator will be the same as the real device. If you have insomnia, try googling the speed differences and you will find hours of commentary on the subject.
Also, QA needs to see more than just how the application looks from a graphical model. They need to see how the application looks on different devices and how it responds to things like changing the screen orientation. Depending on screen sizes things may move around and shift and they can become unreadable or difficult to navigate.
Lastly, unlike a Web UI there is a lot of actual business logic that runs on the device and that specific functionality needs to be tested on the actual device. This creates the challenge; tests need to be run across the most common devices that your targeted users have. Each time we move across different platforms the screen resolution and aspect changes, and behind the scenes CPU speeds are going to be different. Something running on the latest Samsung or an iPhone 6 is going to run much faster than it did on an iPhone 4S.
Achieving testing efficiency on real devices
The decision to test on real devices creates new and unique challenges. For example most QA/test teams like to run tests at night. How do you efficiently do that with real devices? What happens if someone shuts down the workstation the device is attached to, the power cord isn’t attached correctly, or someone else is using the device?
If you want to ensure the highest level of test automation, test devices need to be in a well-managed lab, ideally one that has a large variety of devices and can be shared by teams regardless of location. With a proper lab in place, tests can then be automated, run across multiple devices at the same time and continuous integration can then be achieved.
But building and maintaining an in-house lab of this caliber is difficult and can put additional stress on limited resources. It’s typically not feasible to go to your boss and say “I need 42 iPhones.” Even if you do get the phones, how are you going to manage and access them for testing? The better way to do this is to have the devices hosted in a secure cloud with 24×7 monitoring and support. This way they are always available for testing and easily shared across team members.
Enter cloud-based mobile device labs
A cloud-based lab not only frees up QA to focus on testing but also enables parallel development. Productivity is improved because users no longer have to wait for devices but instead can schedule tests from the shared pool of devices or quickly procure another one when they need it.
With the CA Mobile Cloud there is day-zero support for new devices and iOS and Android upgrades. Devices are hosted in a secure data center and available 24 hours a day. OCR is available. Scalability and reliability are improved by the ability to run tests across more devices simultaneously.
Testing on real devices is just the beginning
In the real world, there are bad network conditions, background apps running, low-battery concerns, call and text message interruptions and shared physical resources such as cameras. Testing using real-world requirements and integration of third-party services like GPS add additional layers of complexity.
With that in mind, CA Mobile Cloud offers customers Wind Tunnel , which optimizes testing for end-user conditions by defining and personifying end-user profiles and by enabling testing across common scenarios such as degraded network conditions, conflicting apps and call interruptions.
Wind Tunnel takes testing an application on a real device to the next level, letting you run your application with the same constraints and conditions it would experience “in the wild.” For example, you can specify 10 other apps to be running and working in the background competing for resources. This enables you to test the overall performance of your application the same way an end user would be experiencing it.
For more on this topic consider checking out the webcast I’m presenting on June 21 “Get Real with Your Mobile Testing Strategy”. During that call, I’ll also talk more about how to simplify back-end validations and how to segment tests based on specific logic running on the device vs. tests that need to be run to test the APIs, back-end systems and ESBs.