Firebase Test Lab for Android reduced American Express app test costs by 50%
Introduction
The AMEX mobile app, with a million Android downloads, is used by Card Members to track spending and rewards, find offers, pay bills, and enjoy app-only features. To facilitate frequent updates, American Express built a device lab to test code changes in their app across all platforms before release.
Challenge
With dozens of devices running various Android releases, the lab was tough to scale. Adding new tests decreased productivity because of the device lab's limited throughput, and when new team members joined, the increased number of code changes added even more load.
Maintaining the device lab was time-consuming and costly, diverting the attention of engineers who otherwise would have been focused on enhancements to drive additional value for Card Members. The devices commonly had issues that required manual intervention, often forcing reruns of the entire test suite. Problematic devices were regularly removed from the device lab, and replacing them often created down-time while the team awaited the arrival of new devices.
Solution
During the product's alpha period, the frequency of updates to the AMEX app, commitment to high-quality releases, and willingness of engineers to collaborate provided an ideal environment for the Firebase Test Lab team. American Express, in turn, provided valuable feedback that helped to shape the released version of the product. Firebase Test Lab integrated with American Express' complex suite of hundreds of instrumentation tests. Even while Test Lab was in early development, American Express was able to switch all of their continuous integration testing to Test Lab and successfully retire much of their device lab. This enabled the engineers maintaining the devices to shift their focus to shipping new features to Card Members while also writing and running tests to ensure high product quality.
In addition, American Express was able to scale the number of tests without worrying about slowing down test-suite execution and scaling up a homegrown device lab. This includes handling surges in the number of tests as new team members join and generating additional code changes. American Express now runs over 1,000 instrumentation tests on every pull request, which equals hundreds of thousands of tests per public release. Running so many concurrent tests dramatically increases their ability to prevent new and recurring bugs, all thanks to Firebase Test Lab's ability to massively parallelize testing.
DISCLAIMER: Results may vary.
Results
30% more tests in half the time
2X+ faster execution of full test suite, with no slowdown when adding tests
50% lower cost vs. internal device lab
Firebase Test Lab enables American Express to execute our entire extensive Android test suite for every pull request. Firebase takes care of scaling the test infrastructure, so our engineers can focus on the code, driving improvements to the app and corresponding changes to our test suite."
- Michael Bailey, VP and Principal Engineer, American Express