Here is a quick rundown on Android and CTS, Google’s Compatibility Test Suite.
(Disclaimer: while I’ve tried to write this blog post to the best of my knowledge there may be some mistakes as I’m not too familiar with the Android source. Please let me know of any problems!)
What is the CTS?
Google’s Compatibility Test Suite is used to make sure that devices conform to the standards that Google has set for Android. In basic terms, it tests the overall device and makes sure the policies set by Google are met by the manufacturer.
So what are these policies? Well, it could be things like checking for encryption on new devices that require encryption, or making sure that APIs are conformant to the protocol outlined the documentation.
These CTS tests are not very important to consumers, because they’re usually used during the development and design of the device. Manufacturers use these tests to make sure that your device will work like any other Android device.
Google’s CTS is one of the prerequisites for bundling the Google Mobile Services (GMS), which includes things like Google Play and Google Play Services. In particular, Play Services is pretty important because it provides the required libraries for other apps, such as the Google Maps component for Uber, and so on.
How do I check if my device is certified?
If you bought your device from a proper manufacturer, such as Samsung, LG, Sony, etc., then you can count on your phone being certified. Most if not all major manufacturers have to pass CTS, or risk losing access to GMS.
However, if you bought a cheap Android tablet from eBay or AliExpress and it’s from a Chinese brand, and you can’t use the Play Store without sideloading, you can suspect the device of not passing the CTS.
A good way to check is by opening up the Play Store, going into Settings, and making sure it says the device is certified at the bottom of the Settings page.
I bought my device from a known-good manufacturer, but it says uncertified. What gives?
Rooting or installing custom ROMs can cause the device to become uncertified. Usually, doing either of those involves unlocking the bootloader or modifying the system partition, which will cause the device to lose the certification status.
In most cases, this certification can be restored by flashing stock firmware and re-locking the bootloader, such as on Google Pixel devices. Some custom ROMs have ways to cheat the certification status and fool GMS into thinking your device is certified (although this cloaking mechanism gets blocked from time to time). However, if your device has hardware attestation, then this mechanism may become useless in the future, so be warned.
On Samsung phones, modifying the phone in any way will trip KNOX, but flashing stock firmware will still return the certification status. It won’t bring back KNOX though, so things like Samsung Pay or Secure Folders won’t work.
What won’t work on uncertified devices?
Well, it depends. Uncertified devices usually do not pass SafetyNet, which is what developers use to make sure the device is in a verifiable state. Without it, things that handle extremely sensitive data, like banking apps, won’t work.
App developers may also choose to filter and exclude devices (more reasons on why below) that do not pass the CTS. Those devices will not be permitted to download these apps from the Play Store. For example, Netflix won’t even show up in the search results if you use an uncertified device.
Apps can also query the certification status and prevent certain features from working. The most recent example is Google Messages, which is showing signs of blocking out uncertified devices.
Also, many apps that rely on behavior from certified devices will break, as detailed below.
What’s so bad about uncertified devices?
Remember how I said that the CTS tests for the overall behavior of the device? Well, devices that failed the CTS will behave in unpredictable ways.
For example, let’s say I’m an app developer and I want to query how many cameras there are on the device. I would use the API to query the list of cameras on a given device. On a device that passes CTS, I can confidently expect the correct number of cameras to be returned.
But let’s say one greedy device manufacturer decided to lie about the number of cameras on their phone, for example. They could modify the source and instruct the API to return more camera units than what is available on the hardware side.
This becomes a problem, because my app assumes that the core Android APIs will not lie to me and behave predictably. And when the behavior becomes unpredictable, the app will become unpredictable, and may even crash. If the app tries to access, say, camera ID 4 when there are only two cameras, and the API to open the camera hardware with the given ID hasn’t been patched to handle the fake cameras, then my app will receive an exception.
Remember, this exception will probably never, ever happen on proper devices that have passed CTS. So it’s most likely that app developers like me won’t handle this exception. So the app will crash, and the user will blame the app developers, when in reality it’s the manufacturer that screwed you over. Do keep this in mind if certain apps crash on uncertified hardware!
This is a real problem, and even affects devices that pass CTS because manufacturers often drastically change the behavior away from the standard set in the original source. Just look at dontkillmyapp.com. If CTS-certified manufacturers make app developers’ lives harder, just think about how annoying uncertified hardware can be.
For the user, having an uncertified device is bad precisely because it is uncertified. For example, you don’t know if the manufacturer properly followed all security guidelines and wrote sensible SELinux (a sort of security mechanism on Android and Linux) rules to protect you. And because devices these days contain a lot of our personal information, housing our stuff on these devices is a really bad idea.
In short, Google’s CTS is good because it makes sure that manufacturers all follow the standards set by Google, and allows app developers to not worry about manufacturers tweaking the source in unexpected ways.
As a consumer, it is important to choose CTS-certified devices to make sure that apps will properly work and your data is kept secure.