Android and iOS UI Testing with Calabash

Testing is an essential part of any mobile app development process. Whether you are automating such tests or not, no sane developer considers their work to be done unless they have tested their app.

A well-tested app usually goes through multiple steps of testing: Unit testing, integration testing, acceptance testing, and so on. As your app grows, the importance of testing increases and automation in testing becomes a necessity.

While other platforms, such as the web, have advanced significantly in terms of testing mechanisms and frameworks, the mobile realm is not lagging behind. In this article, you will learn how you can use Calabash to automate the UI for your Android and iOS apps using plain English instructions and make acceptance testing them as painless as possible.

What Is UI Testing All About?

If you have been testing your apps manually, you are probably wasting a big chunk of your time performing the same tasks over and over again. You make some changes to the code, build the app, run it in a device or an emulator, and fiddle with the app to figure out if it works as expected.

By automating UI testing, you can perform those same manual steps automatically. If your app is of a decent size, this can save a lot of your time and also save your app from being riddled with embarrassing bugs, especially the regression ones.

“That sounds awesome,” you say, but how do you do it for your Android or iOS app?

UI Testing Frameworks for Android and iOS

If you read the official documentation for Android and iOS, they suggest you write and run UI tests in their official IDEs. For Android, it’s Android Studio, and for iOS, it’s Xcode.

The official documentation goes as far as to recommend specific frameworks for testing. The official Android documentation covers some topics about Espresso, the Android UI testing framework. Similarly, Apple suggests using the XCTest framework.

And if you are going to work seriously on UI tests, you may be following these suggestions, which makes sense since Espresso is maintained by Google and is a part of Android Support Repository. It is very likely that Espresso will support all the new features that Google will introduce for Android in the future. You could say the same about XCTest framework for iOS.

However, it is worth keeping in mind that despite the numerous benefits of automated testing, many developers simply don’t write them at all.

Every developer that is aware of test automations, deep inside, knows it is a great idea. But, when it comes to sitting down and writing these tests, many developers start questioning if it is worth their time, because “touch the button” manually turns out to be a much faster operation than writing a code that will “touch this button” automatically. Sometimes, clients and managers, eagerly waiting to try the app, don’t help either.

Many developers, at that point, decide that it is better to continue working on new features of the application rather than write automated UI tests for the existing ones.

When the application grows, it becomes more and more time consuming to “touch these buttons” manually every time you’re updating the application.

But what if there was a framework that made UI testing easier, and didn’t give you any excuse to not write UI tests for your apps?

Meet Calabash.

Calabash: Automated Acceptance Test for Mobile Apps

About a year ago, I started searching for a testing framework that will be easy to use for the people who are not software developers. And, that is when I found Calabash.

This open source testing framework, developed and maintained by the Xamarin team, works for both Android and iOS. It lets you write and execute automated acceptance tests for mobile applications.

Acceptance tests are generally what comes after system tests that determine if your app satisfies the business requirements. Given that it operates on the UI level, this works well as our choice of UI testing automation framework.

Calabash can interact with your app like Espresso or XCTest does. However, what makes Calabash an excellent choice here is its support for Cucumber.

Cucumber is a tool that can run automated tests written in plain English (if you want you can adjust it to use any other plain language). So to write automated tests on Cucumber, the tester doesn’t need to know Java, Objective-C, or any other programming language.

What Makes Calabash Tick?

Calabash framework consists of libraries that can interact with the Android and iOS apps. It can be run on real devices. So it can do things that the tester is doing manually.

There are two different projects on GitHub that make Calabash possible:

• calabash-android - for Android

• calabash-ios - for iOS

Calabash can work with any Ruby-based test framework. In this article, we will cover Cucumber - the most popular and convenient way of writing tests for Calabash.

Before continuing, if you want to try Calabash as you follow the rest of the article, be sure you have Ruby installed on your machine. You can find detailed instructions of installation here.

Next, install Calabash for your favorite platform by following the GitHub links above.

Writing Your First Test on Calabash

Writing tests on Calabash is quite easy. Let’s see how a simple test for an iOS app looks:

Feature: User Login

    Scenario: Unsuccessful user login
        Given the app has launched
        Then I wait for the "Login" button to appear
        When I enter "tstuser" into the "Username" field
        And I enter "qwerty" into the "Password" field
        And I touch "Login"
        Then I should see "Username you entered is incorrect"

    Scenario: Successful user login
        Given the app has launched
        Then I wait for the "Login" button to appear
        When I enter "testeruser" into the "Username" field
        And I enter "qwerty" into the "Password" field
        And I touch "Login"
        Then I should see "Hey testeruser!"

Here, an app is being tested with incorrect username and password, and then is being tested with correct username and password. The test expects the app to fail login for the first scenario, but succeed in the second one.

You can create as many scenarios as need, and all you need to do that is break down the steps/instructions into simple English sentences. Just like you would write a story!

Anyone who knows about behavior-driven development (BDD) will already find themselves familiar with this.

How Does Calabash Work?

To see what goes on behind the steps that tester is using, you can open the project on GitHub and check the following file:

calabash-cucumber/features/step_definitions/calabash_steps.rb

Let’s see a definition of the following step:

When I enter "testeruser" into the "Username" field

Then /^I enter "([^\"]*)" into the "([^\"]*)" field$/ do |text_to_type, field_name|
  touch("textField marked: '#{field_name}'")
  wait_for_keyboard
  keyboard_enter_text text_to_type
  sleep(STEP_PAUSE)
end

This small snippet of Ruby code looks for a specific field, touches it, waits for keyboard to appear, types the text from the text_to_type variable, and waits for a bit before switching to the next step.

The first word of the step can be “Given,” “When,” “Then,” “And,” or “But.” It does not matter what keyword you will use. You can use any of them to make the story clearer.

How to Add Custom Steps

If you need a step that is not implemented in Calabash yet, you can write it by yourself. The syntax is exactly the same as it is in already predefined steps.

For example, if a tester needs to access the input field by placeholder, instead of field name:

Then /^I enter "([^\"]*)" into the field with placeholder "([^\"]*)"$/ do |text_to_type, placeholder|
	touch("textField placeholder:'#{placeholder}'")
	wait_for_keyboard()
	keyboard_enter_text text_to_type
	sleep(STEP_PAUSE)
end

This step definition is much the same as it was the previous one, but you’re accessing the field by placeholder instead of the field name. Given how your app looks, this may make things even easier for the tester.

And it’s easy for the developer, too. The developer is implementing the step once, and then the tester is using it whenever they need it. Moreover, you don’t need to know a lot of Ruby to implement your own custom steps.

You can find the Ruby functions you can use, here:

http://www.rubydoc.info/gems/calabash-cucumber/Calabash/Cucumber

Xamarin Test Cloud

There is one more challenge when testing mobile applications. You should test them on as many devices as possible, because there are so many devices and so many OS versions.

This is where Xamarin Test Cloud helps a lot. There are about 2,000 real devices in the cloud and the good news is that it supports Calabash tests.

The same Calabash tests that helped you save time by saving you from doing repetitive work can now be used to test your application on many real devices.

Start Writing UI Tests

Whether Calabash is the testing solution your app needs, with the advantages it brings, it leaves no room for excuses when it comes to writing automated UI tests for your mobile apps. Calabash may fall short if your app heavily relies on certain device features (e.g., the camera), but it still makes writing tests for a majority of the apps a much easier feat.