Challenges of a Physical Device Lab

At Beta Breakers, we’ve chosen to do all of our mobile device testing on physical devices rather than using emulators and simulators. In the early days of mobile testing, this was the only way to get reliable results, and even with today’s advances in virtualization, the best way to really get a feel for a touch-based UI/UX is to physically experience it. While we stand firmly by our reasons for choosing to use physical hardware over virtualization, maintaining a large (and ever-growing) inventory of smartphones and tablets is not without its challenges.

How do we decide which devices to maintain?

This is (relatively) straightforward—on a roughly quarterly basis, we conduct research on popular mobile devices using multiple sources, and then select a number of common devices to purchase and integrate into our mobile device testing labs. This really applies mostly to the ever-diverse Android ecosystem; Apple product releases are more routine and incremental, so we can usually stagger our device acquisition to meet whatever our volume needs of the moment are.

How do we manage OTA OS updates?  

There short answer is, carefully. As new OSs are released, we sit down internally to review our current inventory of devices and decide which ones should be updated and which should remain at their current OS version. Broadly, we endeavor to have reasonably comprehensive inventories of devices running the last two versions of iOS (current, -1) and three versions of Android (current, -1, -2). Android versions tend to remain relevant longer than their iOS counterparts, and carriers tend to move versions forward more slowly than Apple does, so we keep a wider range of Android versions available as a result. We do keep a small inventory of devices running older OSs for both platforms, but we intentionally keep these numbers small.

Of course, all this planning often goes down the drain when someone taps through an update scheduler too quickly, but overall we have a pretty good track record even when the inevitable handling errors occur.

How do we decide which devices get cellular service?

It’s expensive enough acquiring and maintaining a diverse lab of devices (for reference, we presently have over 130 combined iOS and Android devices, plus a handful of others) without adding in cellular plans, but we do maintain a few, split across the major carriers. For the most part, we change devices on the plans when they come up for renewal, since there are often financial incentives attached to certain devices that we would be acquiring anyway. While we can always move the physical SIMs and have the carrier add the new device to the plan, this often ends up being a more expensive way of doing business, so we prefer to limit this to when a device is lost or broken (it happens, sometimes). Most of our devices (including those with active cell plans) use Wi-Fi connections; we only switch on cellular data for specific tests that require it.

What about physical storage and usage?

Lots of small shelves, and lots of screen wipes. Shared devices are a serious haven for cold germs, so everything gets its screen cleaned. We also maintain both physical and virtual device sign out systems, so that we know which devices are in use on particular projects or are currently off site or unavailable for general use. We also spend a fair amount of time gently reminding folks (such as, cough, myself) to sign their hardware out and back in.

There’s a lot more to managing a large inventory of mobile test hardware, but these are some of the points that we constantly have to address. Let us know if you would like to leverage our mobile device inventory for one of your projects, or contact us if you have any questions about our mobile app testing services.