A Splash of EarlGrey: UI Testing the Toptal Talent App

One of the most important things you can do as a tester to make your work more efficient and fast is to automate the app you are testing. Relying solely on manual tests is not feasible since you would need to run the full set of tests every day, sometimes multiple times a day, testing every change pushed to the app code.

This article will describe our team’s journey to identifying Google’s EarlGrey 1.0 as the tool that worked best for us in the context of automating the iOS Toptal Talent app. The fact that we are using it does not mean EarlGrey is the best testing tool for everyone - it just happens to be the one that suited our needs.

Why We Transitioned to EarlGrey

Over the years, our team has built different mobile apps on both iOS and Android. In the beginning, we considered using a cross-platform UI testing tool that would allow us to write a single set of tests and execute them on different mobile operating systems. First, we went with Appium, the most popular open-source option available.

But as time went by, Appium limitations became more and more obvious. In our case, Appium’s two main drawbacks were:

• The framework’s questionable stability caused many test flakes.
• The comparatively slow update process hampered our work.

To mitigate the first Appium shortcoming, we wrote all sorts of code tweaks and hacks to make the tests more stable. However, there was nothing we could do to address the second. Every time a new version of iOS or Android was released, Appium took a long time to catch up. And very often, because of having many bugs, the initial update was unusable. As a result, we were often forced to keep executing our tests on an older platform version or completely turn them off until a working Appium update was made available.

This approach was far from ideal, and because of these issues, along with additional ones that we won’t cover in detail, we decided to look for alternatives. The top criteria for a new testing tool were increased stability and faster updates. After some investigation, we decided to use native testing tools for each platform.

So, we transitioned to Espresso for the Android project and to EarlGrey 1.0 for iOS development. In hindsight, we can now say that this was a good decision. The time “lost” due to the need to write and maintain two different sets of tests, one for each platform, was more than made up by not needing to investigate so many flaky tests and not having any downtime on version updates.

Local Project Structure

You will need to include the framework in the same Xcode project as the app you are developing. So we created a folder in the root directory to host the UI tests. Creating the EarlGrey.swift file is mandatory when installing the testing framework and its contents are predefined.

EarlGreyBase is the parent class for all test classes. It contains the general setUp and tearDown methods, extended from XCTestCase. In setUp, we load up the stubs that will be generally used by most of the tests (more on stubbing later) and we also set some configuration flags that we’ve noticed increase the stability of the tests:

// Turn off EarlGrey's network requests tracking since we don't use it and it can block tests execution

GREYConfiguration.sharedInstance().setValue([".*"], forConfigKey: kGREYConfigKeyURLBlacklistRegex)
GREYConfiguration.sharedInstance().setValue(false, forConfigKey: kGREYConfigKeyAnalyticsEnabled)

We use the Page Object design pattern - each screen in the app has a corresponding class where all UI elements and their possible interactions are defined. This class is called a “page.” The test methods are grouped by features residing in separate files and classes from the pages.

To give you a better idea of how everything is displayed, this is what the Login and Forgot Password screens look like in our app and how they are represented by page objects.

Later in the article, we will present the code contents of the Login page object.

Custom Utility Methods

The way EarlGrey synchronizes the test actions with the app is not always perfect. For example, it might try to click on a button that is not yet loaded in the UI hierarchy, causing a test to fail. To avoid this issue, we created custom methods to wait until elements appear in the desired state before we interact with them.

Here are a few examples:

static func asyncWaitForVisibility(on element: GREYInteraction) {
     // By default, EarlGrey blocks test execution while
     // the app is animating or doing anything in the background.           
     //https://github.com/google/EarlGrey/blob/master/docs/api.md#synchronization
     GREYConfiguration.sharedInstance().setValue(false, forConfigKey: kGREYConfigKeySynchronizationEnabled)
     element.assert(grey_sufficientlyVisible())
     GREYConfiguration.sharedInstance().setValue(true, forConfigKey: kGREYConfigKeySynchronizationEnabled)
}


static func waitElementVisibility(for element: GREYInteraction, timeout: Double = 15.0) -> Bool {
        GREYCondition(name: "Wait for element to appear", block: {
            var error: NSError?
            element.assert(grey_notNil(), error: &error)
            return error == nil
        }).wait(withTimeout: timeout, pollInterval: 0.5)
        if !elementVisible(element) {
            XCTFail("Element didn't appear")
        }
        return true
}

One other thing that EarlGrey is not doing on its own is scrolling the screen until the desired element becomes visible. Here is how we can do that:

static func elementVisible(_ element: GREYInteraction) -> Bool {
	var error: NSError?
	element.assert(grey_notVisible(), error: &error)
	if error != nil {
		return true
	} else {
		return false
	}
}

static func scrollUntilElementVisible(_ scrollDirection: GREYDirection, _ speed: String, _ searchedElement: GREYInteraction, _ actionElement: GREYInteraction) -> Bool {
        var swipes = 0
        while !elementVisible(searchedElement) && swipes < 10 {
            if speed == "slow" { 	
            actionElement.perform(grey_swipeSlowInDirection(scrollDirection))
            } else {             
            actionElement.perform(grey_swipeFastInDirection(scrollDirection))
            }
            swipes += 1
        }
        if swipes >= 10 {
            return false
        } else {
            return true
        }
}

Other utility methods missing from EarlGrey’s API that we identified are counting elements and reading text values. The code for these utilities is available on GitHub: here and here.

Stubbing API Calls

To make sure we avoid false test results caused by back-end server issues, we use the OHHTTPStubs library to mock server calls. The documentation on their homepage is pretty straightforward, but we will present how we stub responses in our app, which uses GraphQL API.

class StubsHelper {
	static let testURL = URL(string: "https://[our backend server]")!
	static func setupOHTTPStub(for request: StubbedRequest, delayed: Bool = false) {
		stub(condition: isHost(testURL.host!) && hasJsonBody(request.bodyDict())) { _ in
			let fix = appFixture(forRequest: request)
			if delayed {
				return fix.requestTime(0.1, responseTime: 7.0)
			} else {
				return fix
			}
		}
	}
	static let stubbedEmail = "fixture@email.com"
	static let stubbedPassword = "password"
	enum StubbedRequest {
		case login
		func bodyDict() -> [String: Any] {
			switch self {
				case .login:
					return EmailPasswordSignInMutation(
						email: stubbedTalentLogin, password: stubbedTalentPassword
						).makeBodyIdentifier()
			}
		}
		func statusCode() -> Int32 {
			return 200
		}
		func jsonFileName() -> String {
			let fileName: String
			switch self {
				case .login:
					fileName = "login"
			}
			return "\(fileName).json"
		}
	}
	private extension GraphQLOperation {
		func makeBodyIdentifier() -> [String: Any] {
			let body: GraphQLMap = [
				"query": queryDocument,
				"variables": variables,
				"operationName": operationName
			]
        // Normalize values like enums here, otherwise body comparison will fail
        guard let normalizedBody = body.jsonValue as? [String: Any] else {
        	fatalError()
        }
        return normalizedBody
    }
}

Loading the stub is performed by calling the setupOHTTPStub method:

StubsHelper.setupOHTTPStub(for: .login)

Putting Everything Together

This section will demonstrate how we use all the principles described above to write an actual end-to-end login test.

import EarlGrey

final class LoginPage {

    func login() -> HomePage {
        fillLoginForm()
        loginButton().perform(grey_tap())
        return HomePage()
    }

    func fillLoginForm() {  
	ElementsHelper.waitElementVisibility(emailField()) 
    	emailField().perform(grey_replaceText(StubsHelper.stubbedTalentLogin))
        passwordField().perform(grey_tap())
        passwordField().perform(grey_replaceText(StubsHelper.stubbedTalentPassword))
    }

    func clearAllInputs() {
        if ElementsHelper.elementVisible(passwordField()) {
            passwordField().perform(grey_tap())
            passwordField().perform(grey_replaceText(""))
        }
        emailField().perform(grey_tap())
        emailField().perform(grey_replaceText(""))
    }
}

private extension LoginPage {
    func emailField(file: StaticString = #file, line: UInt = #line) -> GREYInteraction {
        return EarlGrey.selectElement(with: grey_accessibilityLabel("Email"), file: file, line: line)
    }

    func passwordField(file: StaticString = #file, line: UInt = #line) -> GREYInteraction {
        return EarlGrey.selectElement(
            with: grey_allOf([
                    grey_accessibilityLabel("Password"),
                    grey_sufficientlyVisible(),
                    grey_userInteractionEnabled()
                ]),
            file: file, line: line
        )
    }

    func loginButton(file: StaticString = #file, line: UInt = #line) -> GREYInteraction {
        return EarlGrey.selectElement(with: grey_accessibilityID("login_button"), file: file, line: line)
    }
}


class BBucketTests: EarlGreyBase {
    func testLogin() {
        StubsHelper.setupOHTTPStub(for: .login)
        LoginPage().clearAllInputs()
        let homePage = LoginPage().login()
        GREYAssertTrue(
            homePage.assertVisible(),
            reason: "Home screen not displayed after successful login"
        )
    }
}

Running Tests in CI

We use Jenkins as our continuous integration system, and we run the UI tests for each commit in every pull request.

We use fastlane scan to execute the tests in CI and generate reports. It’s useful to have screenshots attached to these reports for failed tests. Unfortunately, scan doesn’t provide this functionality, so we had to custom-make it.

In the tearDown() function, we detect if the test failed and save a screenshot of the iOS simulator if it did.

import EarlGrey
import XCTest
import UIScreenCapture

override func tearDown() {
        if testRun!.failureCount > 0 {
            // name is a property of the XCTest instance
            // https://developer.apple.com/documentation/xctest/xctest/1500990-name
            takeScreenshotAndSave(as: name)
        }
        super.tearDown()
}

func takeScreenshotAndSave(as testCaseName: String) {
        let imageData = UIScreenCapture.takeSnapshotGetJPEG()
        let paths = NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true)
        let filePath = "\(paths[0])/\(testCaseName).jpg"

        do {
            try imageData?.write(to: URL.init(fileURLWithPath: filePath))
        } catch {
            XCTFail("Screenshot not written.")
        }
}

The screenshots are saved in the Simulator folder, and you will need to fetch them from there in order to attach them as build artifacts. We use Rake to manage our CI scripts. This is how we gather the test artifacts:

def gather_test_artifacts(booted_sim_id, destination_folder)
  app_container_on_sim = `xcrun simctl get_app_container #{booted_sim_id} [your bundle id] data`.strip
  FileUtils.cp_r "#{app_container_on_sim}/Documents", destination_folder
end

Key Takeaways

If you are looking for a fast and reliable way to automate your iOS tests, look no further than EarlGrey. It is developed and maintained by Google (need I say more?), and in many respects, it is superior to other tools available today.

You will need to tinker a bit with the framework to prepare utility methods to promote test stability. To do this, you can start with our examples of custom utility methods.

We recommend testing on stubbed data to make sure your tests won’t fail because the back-end server doesn’t have all the test data you would expect it to have. Use OHHTTPStubs or a similar local web server to get the job done.

When running your tests in CI, make sure to provide screenshots for the failed cases to make debugging easier.

You may be wondering why we did not migrate to EarlGrey 2.0 yet, and here’s a quick explanation. The new version was released last year and it promises some enhancements over v1.0. Unfrotunately, when we adopted EarlGrey, v2.0 was not particularly stable. Therefore we didn’t transition to v2.0 just yet. However, our team is eagerly awaiting a bug fix for the new version so we can migrate our infrastructure in the future.

Online Resources

EarlGrey’s Getting Started guide on the GitHub homepage is the place you want to start from if you’re considering the testing framework for your project. There, you will find an easy-to-use installation guide, the tool’s API documentation, and a handy cheat sheet listing all the framework’s methods in a manner that is straightforward to use while writing your tests.

For additional information on writing automated tests for iOS, you can also check out one of our previous blog posts.