Mobile Testing

Mobile Testing

Facilitated by Morris Nye (@mossnz) from Push Pay (https://pushpay.com/nz … I think)

Morris led most of the discussion as most of the other attendees were a few steps back from where he is. The focus was largely on Mobile Test Automation but Mobile Testing was also covered.

He talked about the tooling he’s used for Automation - he chose Calabash (in Ruby) over Appium basically as Calabash was seen as more mature when he was making the choice. He uses Cucumber as the test runner.

He avoids emulators as they have considerably different behaviour from real devices The specific examples he gave were

• Garbage collection on emulators being better than real devices, masking memory leaks
• Emulators sometimes being too fast, hiding performance issues

Remote devices are used only for basic testing outside of the core device set - they are hard to debug and network / sensors etc… cannot be tested realistically. For both Remote and Emulated devices - mocked location services can mask bugs by coming back unrealistically quickly with location info. Morris has found interesting crashes due to lags in real location services

Advice on Mobile Auto using Calabash:

• Test Devices should not have passcodes, so they can be woken programmatically
• If an app has to ask for permission, you cannot set this programmatically
• Android default keyboard can cause Calabash to lose focus on the App under test (works OK with different keyboard apps)

Choosing devices is down to usage stats. They review on a quarterly basis, updating their device set so they have the top usage devices in-house, with devices used less than that (but above a minimum threshold) tested using remote device services.

He does a lot of manual, exploratory testing in addition to his auto checks

Someone else (I believe they were from Fairfax) uses Robolectric (http://robolectric.org/) as a framework to do logic layer tests for Android apps. Apparently quite a mature / stable / fast framework.

Morris mentioned some security issues discovered with Mobile, Android in particular. Security tokens are stored in a file on the system which root can see … not exactly secure!

App has been instrumented with a crash reporter to get quality data for bugs. For iOS they use Crashlytics (http://try.crashlytics.com/) for Android they use Raygun (http://raygun.io/raygun-providers/android)

Someone else used Spoon by Square (https://github.com/square/spoon) working with Robotium for distributed screenshots.

General agreement that Windows phones would be an increasing section of market as they’re competitively priced and easier to work with than Android.

Test Flight (http://testflightapp.com/) used to push app builds on iOS, otherwise have to go through the full device rego process for each device.