— android, testing, grpc — 9 min read
There are currently ~2.5 billion Android devices — consisting of ~1,300 discrete brands and ~24,000 unique device models. I'm exploring tapping into this latent resource pool to make automated testing affordable & unlock a more comprehensive configuration coverage.
The goal of DART (Distributed Android Remote Testing) is to enable any Android device to run automated tests — without ADB or complex setups/maintenance. A commercial product could pay people per hour of device use, allowing anyone to utilize idle device time. Each device is subsumed to provide a distributed testbed.
DART is orthogonal to crowdsourced testing — the latter requires active human participation.
DART seems to be the best solution to utilize device idle time — the best way to verify that an app works on "Samsung S9" is by executing it on "Samsung S9". Contrast this to other procedures where phones are usually a suboptimal option: server hosting, remote computation, etc.
In the demo below, I ran a few test cases on Mozilla Focus Privacy Browser without cables/ADB.
The following problems motivated the development of DART.
The top device farms barely cover 1% of the total number of Android device models. Nondistributed device farms favor depth instead of breadth — maintenance cost increases as more diverse devices are added.
Testing on flagship devices alone doesn't always suffice.
Anecdote: I once spent an entire day debugging an ANR that occurs to a small fraction of our users. I discovered it only affects phones purchased in specific regions. I never was able to reproduce the issue on the same phone bought in the USA.
The Android version distribution exacerbates this problem.
Headspin aims to alleviate some of these issues. They claim to have a global infrastructure of more than 22,000 SIM-enabled (similar) devices in 150 locations. A big plus as traditional services (like Firebase Testlabs) can't handle geographical concerns. At Headspin's core is the DeviceFarmer STF open-source library. But like its predecessors, Headspin only supports a limited set of device models.
Real device test farms have an initial cost and require periodic maintenance, hence why they are more expensive than virtual device farms.
Running 10 hours of test daily on real devices costs $1000/month (assuming 20 working days of equal levels of productivity).
The key 🔑: Uber averts enormous maintenance costs by not owning all the cars on its global network. Similarly, with a distributed network of test devices, extensive device coverage is attainable with minimal maintenance costs. For one or two co-located devices, maintenance isn't fastidious.
From DeviceFramer STF FAQ: "Aside from a single early failure we had within only a few months, all of our devices were doing fine for about two years. However, having reached the 2-3 year mark, several devices have started to experience visibly expanded batteries [...] In our experience, the system runs just fine most of the time, and any issues are mostly USB-related. You'll usually have to do something about once a week."
Organizations can decide what type of tests are ideal for DART and if the network is made-up of single nodes or clusters of nodes. The simple taxonomy of android testing below should be helpful. [1]
The network can be internal to an organization. An example: X
devices stored in HQ, each employee can remotely contribute their test device into the system.
It can also be external to an organization. Imagine paying people $1.5/hour
of device time. Assuming a device runs test overnight (thanks to timezone differences), the estimated monthly earning is $240
($1.5 * 8 hours * 20 working days
).
The median income in some countries is ~$400/month.
Below is a demo of the proof-of-concept.
Currently, a simple dashboard renders the test summary.
There are two perspectives here:
The end-user is concerned about malignant binaries & privacy.
Security checks will guard devices from malicious binaries. The system will be closed — only verified publishers allowed. GooglePlay is leveraged to enforce package name and signature correspondence.
A sandboxed test environment guarantees that private data is inaccessible to apps under test.
The publisher cares about test fraud and product leaks.
Proof of work disincentivizes test fraud. The server can use device logs, screenshots, videos with UUID, and past executions to validate submitted work results. As this isn't watertight, it is best combined with other strategies: user verifications, device attestation, randomization, and device throughput throttling.
Preventing product leaks is hard — if it runs on a phone, any sufficiently motivated (& knowledgeable) person can figure out a way to access objects. Three good-enough solutions are:
a. Running headless tests.
b. Using a private network of vetted devices.
c. Adopting only for binaries at the later end of the release pipeline.
It is possible to obscure the application under test using a simple overlay. The video below shows the execution of the test cases of the Google IO 2019 app. On the right, the app interactions are invisible. Depending on the strategy, concealing actions do not affect screenshots or videos.
With framework UI component wrappers, it is possible to borrow some ideas from Layoutlib to offer a full headless solution. Layoutlib
is a custom version of the android View framework designed to run inside Eclipse. The goal of the library is to provide layout rendering in Eclipse that are very close to their rendering on devices.
Tests need to run in an isolated environment. A plugin framework like VirtualAPK can be built to offer a layer of isolation between apps and the OS. This works using the DexClassLoader
and some reflection hacks to modify framework components. With this approach, apps can be seamlessly loaded & unloaded without any user interaction.
The GooglePlay dynamic feature delivery system uses some of these reflection techniques to support legacy devices (pre Android 7.0) 🙈. For instance,
addAssetPath
is called via reflection to make resources not bundled in an APK available for use.
This plugin approach adds cruft that is nonexistent on a regular device. An alternative is using the Android work profile. Work profile creates a separate, self-contained profile on Android devices that isolates corporate data from personal apps and data. [3]
Work profiles also provide always-on VPN configuration, control of runtime permissions, extra security, etc. Work profiles are a great fit, only deviating slightly from the operating mode used by the average user.
Android allows apps signed with the same key to run in the same process, if the apps so request, so that the system treats them as a single application. [3]
The system should eliminate most causes of flakiness that results from running tests on real peoples' devices — I have run over 5,000 tests so far without any issues 🚀. In an external network, it would be naive to assume perfection as there are so many variables introduced. As the project evolves, these issues would be identified and mitigated.
A few researchers manually examined 423 projects featuring the Espresso automated GUI testing tool. They derived a set of 27 different causes of modifications and grouped them into nine macro-categories. Category percentages were then computed based on the frequency of modification causes. [2]
There are four primary directories in the Github repo:
i. Apollo
: Android project for the client (worker).
ii. WorkServer
: Kotlin project for the gRPC work server.
iii. WorkSpecs
: Protobuf definitions shared by both the server & Android client.
iv. Dashboard
: React project for the web dashboard.
All projects are still a work in progress and lack adequate documentation.
As shown above, multiple components must work in tandem to run tests on a device. The sequence diagram below shows the component interactions at a high-level.
The diagram shows the finite system states and their respective transitions.
The next few sections cover individual system components. For the sake of brevity, I exclude codes that are irrelevant to the topic at hand. For each component, I only focus on some key ideas. You can always explore the Github repo for the full source code.
The framework's UiAutomation
can't be used when tests aren't run traditionally. Fortunately, the functionalities of UiAutomation
can be replicated using the platform Accessibility APIs
and some other functionally equivalent alternatives. As evident from the documentation, UiAutomation
can be viewed as an AccessiblityService
with extras.
1/**2 * Class for interacting with the device's UI by simulation user actions and3 * introspection of the screen content. It relies on the platform accessibility4 * APIs to introspect the screen and to perform some actions on the remote view5 * tree. It also allows injecting of arbitrary raw input events simulating user6 * interaction with keyboards and touch devices. One can think of a UiAutomation7 * as a special type of {@link android.accessibilityservice.AccessibilityService}8 * which does not provide hooks for the service life cycle and exposes other9 * APIs that are useful for UI test automation.10 * <p>11 * The APIs exposed by this class are low-level to maximize flexibility when12 * developing UI test automation tools and libraries. Generally, a UiAutomation13 * client should be using a higher-level library or implement high-level functions.14 * For example, performing a tap on the screen requires construction and injecting15 * of a touch down and up events which have to be delivered to the system by a16 * call to {@link #injectInputEvent(InputEvent, boolean)}.17 * </p>18 * <p>19 * The APIs exposed by this class operate across applications enabling a client20 * to write tests that cover use cases spanning over multiple applications. For21 * example, going to the settings application to change a setting and then22 * interacting with another application whose behavior depends on that setting.23 * </p>24 */25public final class UiAutomation {26}
The alternative UiAutomation
(internal) server can be implemented like this:
1class UiAutomationServer(2 private val screenRotator: ScreenRotator,3 private val screenViewer: ScreenViewer,4 private val appPermissioner: AppPermissioner5): AutomationServer.Stub() {67 var lastAccessibilityEvent: AccessibilityEvent? = null8 var accessibilityEventListener: OnAccessibilityEventListener? = null910 fun onAccessibilityEvent(event: AccessibilityEvent) {11 lastAccessibilityEvent = event1213 val uiEvent = addUiEvent(event)14 accessibilityEventListener?.onAccessibilityEvent(uiEvent)15 }1617 override fun setOnAccessibilityEventListener(listener: OnAccessibilityEventListener?) {18 accessibilityEventListener = listener19 }2021 override fun findFocus(focus: Int) =22 runWithOriginalIdentity { AutomationService.INSTANCE?.findFocus(focus) }2324 override fun performGlobalAction(action: Int) =25 runWithOriginalIdentity { AutomationService.INSTANCE?.performGlobalAction(action) ?: false }2627 override fun getLastEvent() = lastAccessibilityEvent2829 override fun setRotation(rotation: Int) =30 runWithOriginalIdentity { screenRotator.setRotation(rotation) }3132 override fun unfreezeCurrentRotation() =33 runWithOriginalIdentity { screenRotator.unfreezeCurrentRotation() }3435 override fun freezeCurrentRotation() =36 runWithOriginalIdentity { screenRotator.freezeCurrentRotation() }373839 override fun restoreInitialRotation() =40 runWithOriginalIdentity { screenRotator.restoreInitialSettings() }4142 override fun getWindows(): List<AccessibilityWindowInfo> =43 runWithOriginalIdentity { AutomationService.INSTANCE?.windows ?: emptyList() }4445 override fun findAccessibilityNodeInfosByText(nodeInfo: UiNodeInfo, text: String): List<UiNodeInfo> {46 return try {47 nodeInfo.accessibilityNodeInfo.findAccessibilityNodeInfosByText(text).map {48 addUiNodeInfo(it)49 }50 } catch (e: Exception) {51 Timber.e(e)52 emptyList()53 }54 }5556 override fun findAccessibilityNodeInfosByViewId(nodeInfo: UiNodeInfo, viewId: String): List<UiNodeInfo> {57 return try {58 nodeInfo.accessibilityNodeInfo.findAccessibilityNodeInfosByViewId(viewId).map {59 addUiNodeInfo(it)60 }61 } catch (e: Exception) {62 Timber.e(e)63 emptyList()64 }65 }6667 override fun performNodeAction(nodeInfo: UiNodeInfo, action: Int): Boolean {68 return nodeInfo.accessibilityNodeInfo.performAction(action)69 }7071 override fun getUiEventSource(event: UiEvent): UiNodeInfo? {72 return event.accessibilityEvent.source?.let { addUiNodeInfo(it) }73 }7475 override fun getRootInActiveWindow() =76 runWithOriginalIdentity { AutomationService.INSTANCE?.rootInActiveWindow }7778 override fun getServiceInfo() =79 runWithOriginalIdentity { AutomationService.INSTANCE?.serviceInfo }8081 override fun setServiceInfo(serviceInfo: AccessibilityServiceInfo): Boolean {82 return runWithOriginalIdentity {83 AutomationService.INSTANCE?.let {84 // TODO: Security on ServiceInfo from clients. This maybe should be validated/restricted.85 it.serviceInfo = serviceInfo86 return@runWithOriginalIdentity true87 }8889 return@runWithOriginalIdentity false90 }91 }9293 override fun takeScreenshot(): Bitmap? {94 return try {95 screenViewer.capture()96 } catch (e: Exception) {97 Timber.e(e, "Failed to take screenshot")98 null99 }100 }101}102103inline fun <R> runWithOriginalIdentity(action: () -> R): R {104 Binder.clearCallingIdentity()105 val identity = Binder.clearCallingIdentity()106 try {107 return action()108 } finally {109 Binder.restoreCallingIdentity(identity)110 }111}
Not all functionality is directly available using the platform Accessibility APIs
. For instance, DevicePolicyManager
is used to grant runtime permissions & MediaProjector
for device-wide screenshots.
1class AppPermissioner(2 private val appContext: Context,3 private val devicePolicyManager: DevicePolicyManager4 private val adminComponent: ComponentName5) {67 @TargetApi(Build.VERSION_CODES.M)8 fun grantRuntimePermissionAsUser(packageName: String, permission: String, userHandle: UserHandle) {9 devicePolicyManager.setPermissionGrantState(10 adminComponent,11 packageName,12 permission,13 PERMISSION_GRANT_STATE_GRANTED14 )15 }1617 @TargetApi(Build.VERSION_CODES.M)18 fun revokeRuntimePermissionAsUser(packageName: String, permission: String, userHandle: UserHandle) {19 devicePolicyManager.setPermissionGrantState(20 adminComponent,21 packageName,22 permission,23 PERMISSION_GRANT_STATE_DEFAULT24 )25 }26}
1private const val DISPLAY_NAME_SCREENSHOT = "ScreenyScreenshotDisplay"2// Max images is set to 2 to allow [ImageReader.acquireLatestImage] to do its thing3private const val MAX_IMAGES = 24private const val SCREENSHOT_TIMEOUT_MILLIS = 10_000L5private const val LOG_TAG = "Screenshot"67internal class Screenshot(8 private val mediaProjection: MediaProjection,9 private val backgroundHandler: Handler10) {1112 fun capture(windowManager: WindowManager): Bitmap? {13 Log.i(LOG_TAG, "Capturing Screenshot!")14 val countDownLatch = CountDownLatch(1)15 val displayProps = windowManager.getDefaultDisplayProps()16 val imageReader = ImageReader.newInstance(displayProps.width, displayProps.height, PixelFormat.RGBA_8888, MAX_IMAGES)17 var bitmap: Bitmap? = null1819 val virtualDisplay = mediaProjection.createVirtualDisplay(20 DISPLAY_NAME_SCREENSHOT,21 displayProps,22 imageReader.surface,23 backgroundHandler24 ) {25 countDownLatch.countDown()26 }2728 imageReader.setOnImageAvailableListener({29 imageReader.setOnImageAvailableListener(null, null)30 val image = imageReader.acquireLatestImage()3132 image.close()33 34 try {35 bitmap = image.getBitmap(displayProps)36 Log.i(LOG_TAG, "Screenshot captured!!")37 } catch (e: Exception) {38 Log.e(LOG_TAG, "Failed to get bitmap from image", e)39 } finally {40 image.closeCatching { Log.e(LOG_TAG, "Error closing Image", it) }41 imageReader.closeCatching { Log.e(LOG_TAG, "Error closing ImageReader", it) }42 virtualDisplay.surface = null43 countDownLatch.countDown()44 }45 }, backgroundHandler)4647 try {48 countDownLatch.await(SCREENSHOT_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)49 } finally {50 virtualDisplay.release()51 }5253 return bitmap54 }55}5657@WorkerThread58private fun Image.getBitmap(displayProps: DisplayProperties): Bitmap {59 val buffer: ByteBuffer = planes.first().buffer60 val pixelStride = planes.first().pixelStride61 val rowStride = planes.first().rowStride62 val rowPadding = rowStride - pixelStride * displayProps.width6364 val bitmap = Bitmap.createBitmap(65 displayProps.width + (rowPadding.toFloat() / pixelStride.toFloat()).toInt(),66 displayProps.height,67 Bitmap.Config.ARGB_888868 )6970 bitmap.copyPixelsFromBuffer(buffer)71 return bitmap72}
A custom class with the exact set of functions is used as a drop-in replacement for UiAutomation
. Client test code either has to use this custom class directly or use a Gradle plugin that changes the import
using bytecode manipulation.
The AccessibilityService runs in a dedicated process to isolate it from the work runner app. Also, we use a proxy service to communicate with the AccessibilityService because onBind is final.
Once the AppUnderTest.apk and Test.apk has been installed, the TestServer
calls the Context.startInstrumentation
function to run the test cases.
1/**2 * Start executing an {@link android.app.Instrumentation} class. The given3 * Instrumentation component will be run by killing its target application4 * (if currently running), starting the target process, instantiating the5 * instrumentation component, and then letting it drive the application.6 *7 * <p>This function is not synchronous -- it returns as soon as the8 * instrumentation has started and while it is running.9 *10 * <p>Instrumentation is normally only allowed to run against a package11 * that is either unsigned or signed with a signature that the12 * the instrumentation package is also signed with (ensuring the target13 * trusts the instrumentation).14 *15 * @param className Name of the Instrumentation component to be run.16 * @param profileFile Optional path to write profiling data as the17 * instrumentation runs, or null for no profiling.18 * @param arguments Additional optional arguments to pass to the19 * instrumentation, or null.20 *21 * @return {@code true} if the instrumentation was successfully started,22 * else {@code false} if it could not be found.23 */24 public abstract boolean startInstrumentation(@NonNull ComponentName className,25 @Nullable String profileFile, @Nullable Bundle arguments);
For the arguments parameter, a Bundle
is populated with configuration details from the server.
1class TestConfigs(private val arguments: Bundle) {23 fun isObscureWindowEnabled() = arguments.getBoolean(ARG_OBSCURE_WINDOW_ENABLED)45 fun shouldRetrieveTestFiles() = arguments.getBoolean(ARG_RETRIEVE_TEST_FILES)67 fun shouldRetrieveAppFiles() = arguments.getBoolean(ARG_RETRIEVE_APP_FILES)89 fun getTestsCount() = arguments.getInt(ARG_TESTS_COUNT)1011 fun getProfilerSampleFrequency() = arguments.getInt(ARG_PROFILER_SAMPLE_FREQUENCY)1213 fun isClearDataEnabled() = arguments.getBoolean(ARG_CLEAR_DATA)1415 fun isAutoScreenShotEnabled() = arguments.getBoolean(ARG_AUTO_SCREEN_SHOT_ENABLED)1617 fun getAutoScreenShotFps() = arguments.getInt(ARG_AUTO_SCREEN_SHOT_FPS)1819 fun getScreenShotQuality() = arguments.getInt(ARG_AUTO_SCREEN_QUALITY)20}
TestObserver.aidl
is defined for TestClient
<> TestServer
communication.
1interface TestCallback {23 void onTestRunStarted(in TestDescription description);45 void onTestRunFinished(in TestResult result);67 void onTestStarted(8 in TestDescription description,9 String logFileName,10 String profilerFileName,11 String autoScreenShotNamePrefix12 );1314 void onTestFinished(in TestDescription description);1516 void onTestFailure(in TestFailure failure);1718 void onTestAssumptionFailure(in TestFailure failure);1920 void onTestIgnored(in TestDescription description);2122 void onProcessCrashed(in TestDescription failure, String stackTrace);2324 void onClientConnected(Finisher finisher);2526 void onInterrupted(int reasonId);2728 void sendString(String message);29}
Logs and other artifacts need to be "streamed" to the TestServer
. This data needs to be stored in the server's private directory but the Android security model doesn't allow foreign apps to write directly into an app's private directories. This limitation can be circumvented using a ContentProvider
and FileDescriptors
.
1object RemoteStorageConstants {23 const val PREFIX_CONTENT = "content://"4 const val AUTHORITY = "com.fluentbuild.apollo.runtime.remotestorage"5 const val BASE_URI = "${PREFIX_CONTENT}${AUTHORITY}/"67 const val MODE_READ = "r"8 const val MODE_WRITE = "w"9 const val MODE_APPEND = "wa"10}
1private const val REMOTE_STORAGE_DIR = "stash"23class RemoteStorageProvider: ContentProvider() {45 override fun openFile(uri: Uri, mode: String): ParcelFileDescriptor? {6 val child = uri.toString().substringAfter(RemoteStorageConstants.BASE_URI)7 if(child.isBlank()) return null89 return try {10 File(getDir(context!!), child).run {11 Timber.i("Opening file: %s with mode: %s", absolutePath, mode)12 parentFile?.mkdirs()13 openParcelFileDescriptor(mode)14 }15 } catch (e: Exception) {16 null17 }18 }1920 companion object {2122 fun getDir(context: Context) = File(context.filesDir, REMOTE_STORAGE_DIR).apply { mkdirs() }23 }24}
The TestClient
is included in the Test.apk and controls test execution. It is part of the "com.fluentbuild.apollo:client:version"
artifact.
1private const val LOG_TAG = "TestClient"2private const val RUNNER_PACKAGE = "com.fluentbuild.apollo"3private const val RUNNER_SERVICE = "com.fluentbuild.apollo.RunnerService"4private const val SERVICE_CONNECTION_TIMEOUT_MILLIS = 10_000L56/**7 * For the current instrumentation to communicate information back to the RuntimeService.8 *9 */10class TestClient(11 private val instrumentation: Instrumentation,12 private val clientFinalizer: ClientFinalizer,13 private val logWrapper: LogWrapper14): WorkInterruptCallback {1516 private val connectionLatch = CountDownLatch(1)1718 @Volatile19 private lateinit var testCallback: TestCallback2021 private val serviceConnection = object : ServiceConnection {2223 override fun onServiceConnected(className: ComponentName, service: IBinder) {24 logWrapper.i(LOG_TAG, "TestClient connected to runner service")25 testCallback = TestCallback.Stub.asInterface(service)26 connectionLatch.countDown()27 }2829 override fun onServiceDisconnected(className: ComponentName) {30 logWrapper.e(LOG_TAG, "TestClient is disconnected from runner service")31 instrumentation.finishInstrumentation(Activity.RESULT_CANCELED)32 }33 }3435 // Called on the test thread36 fun connect() {37 logWrapper.i(LOG_TAG, "Connecting to runner service")3839 val intent = Intent()40 intent.setClassName(RUNNER_PACKAGE, RUNNER_SERVICE)4142 instrumentation.context.requireServiceBind(intent, serviceConnection)4344 if(!connectionLatch.await(SERVICE_CONNECTION_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {45 unbindService()46 throw TimeoutException("Couldn't connect to runner service")47 }4849 onClientConnected()50 }5152 private fun onClientConnected() {53 try {54 testCallback.onClientConnected(object : Finisher.Stub() {5556 override fun finish(resultCode: Int) {57 logWrapper.i(LOG_TAG, "Instrumentation finish requested")58 clientFinalizer.finalize()59 instrumentation.finishInstrumentation(resultCode)60 }61 })62 } catch (e: RemoteException) {63 handleRemoteFailure("Unable to notify runner service of connection!", e)64 }65 }6667 @MainThread68 fun disconnect() {69 unbindService()70 }7172 // Just simple test callbacks below73}
TestClient
works in tandem with an instance of Instrumentation
. A custom instance is required to be able to intercept Instrumentation
callbacks. In the "com.fluentbuild.apollo:client:version"
artifact, an implementation that subclasses AndroidJUnitRunner
is provided. A reflection hack is used to register a org.junit.runner.notification.RunListener
in AndroidJUnitRunner
.
TestClient
works in tandem with an instance of Instrumentation
. A custom instance is required to be able to intercept Instrumentation
callbacks. In the "com.fluentbuild.apollo:client:version"
artifact, an implementation that subclasses AndroidJUnitRunner
is provided. In this implementation, I used a reflection hack to hook into the InstrumentationResultPrinter
. Passing in the TestObserver
, which captures test states and eventually calls into the real printer.
1private fun init(testConfigs: TestConfigs) {2 val printerField = getPrinterField()3 val printer = printerField.get(runner) as InstrumentationResultPrinter4 initializer.init(testConfigs, printer)5 printerField.set(runner, initializer.getTestObserver())6}78private fun getPrinterField(): Field {9 return AndroidJUnitRunner::class.java.getDeclaredField("instrumentationResultPrinter")10 .apply { isAccessible = true }11}
For publishers that don't use AndroidJunitRunner
, they can still use the client artifact by calling into the relevant functions.
UiAutomation
can be used to request runtime permission. An extra layer similar to the one in the androidx.test.runner.permission
package can be created to simplify this flow.
1/**2 * Requests a runtime permission on devices running Android M (API 23) and above.3 *4 * This class is usually used to grant runtime permissions to avoid the permission dialog from5 * showing up and blocking the App's Ui. This is especially helpful for Ui-Testing to avoid loosing6 * control over your application under test.7 *8 * The requested permissions will be granted for all test methods in the test class. Use [addPermissions] to add a permission to the permission list. To request all9 * permissions use the [requestPermissions] method.10 *11 */12interface PermissionGranter {1314 /**15 * Adds a permission to the list of permissions which will be requested when [.requestPermissions] is called.16 *17 * Precondition: This method does nothing when called on an API level lower than [Build.VERSION_CODES.M].18 *19 * @param permissions a list of Android runtime permissions.20 */21 fun addPermissions(vararg permissions: String)2223 /**24 * Request all permissions previously added using [.addPermissions]25 *26 * Precondition: This method does nothing when called on an API level lower than [ ][Build.VERSION_CODES.M].27 */28 fun requestPermissions()29}
The components of the AppUnderTest are closely monitored to power some telemetry functionalities. For each type of component, we create an implementation of the Monitor
interface.
1abstract class Monitor<CallbackT> {23 protected val callbacks = mutableListOf<WeakReference<CallbackT>>()45 internal fun registerCallback(callback: CallbackT) {6 if(callbacks.none { it.get() == callback }) {7 callbacks += WeakReference(callback)8 }9 }1011 internal fun unregisterCallback(callback: CallbackT) {12 callbacks.removeAll { it.get() == callback }13 }14}
AppMonitor
monitors the Application
lifecycle.
1class AppMonitor: Monitor<AppMonitor.Callback>() {23 private var appRef: WeakReference<Application>? = null45 private fun signalLifecycleChange(application: Application, stage: ApplicationStats.Stage) {6 callbacks.forEach { it.get()?.onStageChanged(application, stage) }7 }89 fun onCallApplicationOnCreate(app: Application, action: () -> Unit) {10 appRef = WeakReference(app)11 signalLifecycleChange(app, ApplicationStats.Stage.PRE_ON_CREATE)12 action()13 signalLifecycleChange(app, ApplicationStats.Stage.CREATED)14 }1516 fun getActiveApp(): Application? = appRef?.get()1718 interface Callback {19 fun onStageChanged(app: Application, stage: ApplicationStats.Stage)20 }21}
ActivityMonitor
monitors the lifecycle of all activities in the AppUnderTest. The action function allows the monitor control when the caller (the Instrumentation) can pass the event downstream.
1class ActivityMonitor: Monitor<ActivityMonitor.Callback>() {23 private val activeActivities = WeakHashMap<Activity, ActivityStats.Stage>()45 private fun signalLifecycleChange(activity: Activity, stage: ActivityStats.Stage) {6 activeActivities[activity] = stage7 callbacks.forEach {8 it.get()?.onStageChanged(activity, stage)9 }10 }1112 fun onCallActivityOnDestroy(activity: Activity, action: () -> Unit) {13 signalLifecycleChange(activity, ActivityStats.Stage.DESTROYED)14 action()15 activeActivities.remove(activity)16 }1718 fun onCallActivityOnRestart(activity: Activity, action: () -> Unit) {19 action()20 signalLifecycleChange(activity, ActivityStats.Stage.RESTARTED)21 }2223 fun onCallActivityOnCreate(activity: Activity, bundle: Bundle?, action: () -> Unit) {24 signalLifecycleChange(activity, ActivityStats.Stage.PRE_ON_CREATE)25 action()26 signalLifecycleChange(activity, ActivityStats.Stage.CREATED)27 }2829 fun onCallActivityOnCreate(30 activity: Activity,31 bundle: Bundle?,32 persistentState: PersistableBundle,33 action: () -> Unit34 ) {35 signalLifecycleChange(activity, ActivityStats.Stage.PRE_ON_CREATE)36 action()37 signalLifecycleChange(activity, ActivityStats.Stage.CREATED)38 }3940 fun onCallActivityOnStart(activity: Activity, action: () -> Unit) {41 action()42 signalLifecycleChange(activity, ActivityStats.Stage.STARTED)43 }4445 fun onCallActivityOnStop(activity: Activity, action: () -> Unit) {46 action()47 signalLifecycleChange(activity, ActivityStats.Stage.STOPPED)48 }4950 fun onCallActivityOnResume(activity: Activity, action: () -> Unit) {51 action()52 signalLifecycleChange(activity, ActivityStats.Stage.RESUMED)53 }5455 fun onCallActivityOnPause(activity: Activity, action: () -> Unit) {56 action()57 signalLifecycleChange(activity, ActivityStats.Stage.PAUSED)58 }59 60 fun getActiveActivities(): Set<Activity> {61 return activeActivities.keys62 }6364 interface Callback {65 fun onStageChanged(activity: Activity, stage: ActivityStats.Stage)66 }67}
Only Parcelable
classes can be used in AIDL IPC calls. Because of this constraint, the following model classes were created to pass notifications from JUnit to the server.
1data class TestDescription(2 val className: String,3 val methodName: String?,4 val displayName: String5): Parcelable67data class TestFailure(8 val description: TestDescription,9 val trace: String10): Parcelable1112data class TestResult(13 val runtimeMillis: Long,14 val ignoreCount: Int,15 val failures: List<TestFailure>16): Parcelable
The jUnit models will need to be mapped to the model classes defined above.
1private const val MAX_TRACE_SIZE = 64 * 102423internal fun Description.createTestModel(): TestDescription {4 return TestDescription(className, methodName, displayName)5}67internal fun Failure.createTestModel(): TestFailure {8 var stackTrace = trace910 if (stackTrace.length > MAX_TRACE_SIZE) {11 // Since we report failures back to the runtime via a binder IPC, we need to make sure that12 // we don't exceed the Binder transaction limit - which is 1MB per process.13 Log.w(LOG_TAG, "Stack trace too long, trimmed to first $MAX_TRACE_SIZE characters.")14 stackTrace = trace.substring(0, MAX_TRACE_SIZE) + "\n"15 }1617 return TestFailure(description.createTestModel(), stackTrace)18}1920internal fun Result.createTestModel(): TestResult {21 return TestResult(runTime, ignoreCount, failures.map { it.createTestModel() })22}
The TestObserver
is an instance of org.junit.runner.notification.RunListener
, and is notified of events that occur during a test run. These events are passed to the TestClient
which then passes it to the TestServer
.
1internal class TestObserver(2 private val testClient: TestClient,3 private val wrappedPrinter: InstrumentationResultPrinter,4 private val clientFinalizer: ClientFinalizer,5 private val collatorsManager: CollatorsManager6): InstrumentationResultPrinter() {78 private var startedCount = 09 private var lastStartedTest: Description? = null1011 override fun testRunStarted(description: Description) {12 testClient.testRunStarted(description)13 wrappedPrinter.testRunStarted(description)14 }1516 override fun testStarted(description: Description) {17 lastStartedTest = description18 startedCount++19 testClient.testStarted(description, collatorsManager.getInfo())20 wrappedPrinter.testStarted(description)21 }2223 override fun testAssumptionFailure(failure: Failure) {24 testClient.testAssumptionFailure(failure)25 restartMeasurement()26 wrappedPrinter.testAssumptionFailure(failure)27 }2829 override fun testRunFinished(result: Result) {30 testClient.testRunFinished(result)31 wrappedPrinter.testRunFinished(result)32 }3334 override fun sendString(msg: String) {35 testClient.sendString(msg)36 wrappedPrinter.sendString(msg)37 }3839 override fun instrumentationRunFinished(40 summaryWriter: PrintStream,41 resultBundle: Bundle,42 junitResults: Result43 ) {44 clientFinalizer.finalize()45 wrappedPrinter.instrumentationRunFinished(summaryWriter, resultBundle, junitResults)46 }4748 override fun testFailure(failure: Failure) {49 testClient.testFailure(failure)50 restartMeasurement()51 wrappedPrinter.testFailure(failure)52 }5354 override fun testFinished(description: Description) {55 testClient.testFinished(description)56 restartMeasurement()57 wrappedPrinter.testFinished(description)58 }5960 override fun testIgnored(description: Description) {61 testClient.testIgnored(description)62 restartMeasurement()63 wrappedPrinter.testIgnored(description)64 }6566 override fun reportProcessCrash(throwable: Throwable) {67 testClient.processCrashed(Failure(lastStartedTest, throwable))68 restartMeasurement()69 wrappedPrinter.reportProcessCrash(throwable)70 }7172 private fun restartMeasurement() {73 collatorsManager.restart()74 }75}
OverlayView
is a simple custom fullscreen opaque view that is used to obscure Activities when requested. The overlay is attached to the window immediately after an Activity is created. I haven't noticed any interferences yet between the overlay and test UI interactions.
1internal class WindowOverlay(activity: Activity) {23 private val overlayView = OverlayView(activity)45 init {6 activity.window.addContentView(overlayView.rootView, getWindowParams())7 }89 fun updateLabel(labelText: String) {10 overlayView.updateLabel(labelText)11 }1213 fun getRoot() = overlayView.rootView1415 private fun getWindowParams(): WindowManager.LayoutParams {16 val type = if(AndroidVersion.isAtLeastOreo()) {17 WindowManager.LayoutParams.TYPE_APPLICATION_OVERLAY18 } else {19 @Suppress("DEPRECATION")20 WindowManager.LayoutParams.TYPE_PHONE21 }2223 val formats = WindowManager.LayoutParams.FLAG_NOT_TOUCHABLE or24 WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE or25 WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON or26 WindowManager.LayoutParams.FLAG_FULLSCREEN2728 return WindowManager.LayoutParams(29 WindowManager.LayoutParams.MATCH_PARENT,30 WindowManager.LayoutParams.MATCH_PARENT,31 type,32 formats,33 PixelFormat.TRANSPARENT34 )35 }36}
Any external interactions should immediately stop test execution to prevent interference. The NavigationInteractionObserver
detects when a user presses a button.
1private const val SYSTEM_DIALOG_REASON_KEY = "reason"2private const val SYSTEM_DIALOG_REASON_GLOBAL_ACTIONS = "globalactions"3private const val SYSTEM_DIALOG_REASON_RECENT_APPS = "recentapps"4private const val SYSTEM_DIALOG_REASON_HOME_KEY = "homekey"56internal class NavigationInteractionObserver(private val callback: Callback): BroadcastReceiver() {78 override fun onReceive(context: Context, intent: Intent) {9 val reason = intent.getStringExtra(SYSTEM_DIALOG_REASON_KEY)1011 if (reason == SYSTEM_DIALOG_REASON_HOME_KEY) {12 callback.onHomePressed()13 } else if (reason == SYSTEM_DIALOG_REASON_RECENT_APPS) {14 callback.onRecentAppsPressed()15 }16 }1718 fun start(context: Context) {19 context.registerReceiver(this, IntentFilter(Intent.ACTION_CLOSE_SYSTEM_DIALOGS))20 }2122 fun stop(context: Context) {23 context.unregisterReceiver(this)24 }2526 interface Callback {2728 fun onHomePressed()2930 fun onRecentAppsPressed()31 }32}
The WindowInteractionObserver
detects when a user taps on the screen.
1internal class WindowInteractionObserver(private val callback: Callback): ActivityMonitor.Callback {23 override fun onStageChanged(activity: Activity, stage: ActivityStats.Stage) {4 if(stage == ActivityStats.Stage.CREATED) {5 activity.window.callback = object: WindowCallbackWrapper(activity.window.callback) {67 override fun dispatchTouchEvent(event: MotionEvent): Boolean {8 evaluateUserMotion(event)9 return super.dispatchTouchEvent(event)10 }1112 override fun dispatchKeyEvent(event: KeyEvent): Boolean {13 evaluateUserKeyInput(event)14 return super.dispatchKeyEvent(event)15 }16 }17 }18 }1920 interface Callback {2122 fun onWindowTouchedByUser()2324 fun onKeyPressedByUser()25 }26}
The client can utilize the server's ContentProvider to save files in the server's private directory.
1class FileDescriptorProvider(private val context: Context) {23 @Throws(Exception::class)4 internal fun getReadableDescriptor(fileName: String) = getDescriptor(fileName, MODE_READ)56 @Throws(Exception::class)7 internal fun getWritableDescriptor(fileName: String) = getDescriptor(fileName, MODE_WRITE)89 @Throws(Exception::class)10 internal fun getAppendableDescriptor(fileName: String) = getDescriptor(fileName, MODE_APPEND)1112 @Throws(Exception::class)13 private fun getDescriptor(fileName: String, mode: String): ParcelFileDescriptor =14 context.contentResolver.openFileDescriptor(getFileUri(fileName), mode)!!.apply { checkError() }1516 private fun getFileUri(filePath: String) = Uri.parse("${BASE_URI}${filePath}")17}
After running all tests or when the AppUnderTest crashes, all miscellaneous artifacts are moved to the TestServer
.
1class ArtifactsCopier(2 private val targetContext: Context,3 private val instrumentationContext: Context,4 private val testConfigs: TestConfigs,5) {67 fun copy() {8 if (testConfigs.shouldRetrieveTestFiles()) {9 copyDir(destinationDir = "test_files", sourceDir = instrumentationContext.filesDir)10 }1112 if (testConfigs.shouldRetrieveAppFiles()) {13 copyDir(destinationDir = "app_files", sourceDir = targetContext.filesDir)14 }15 }16}
1internal class ClearDataTask(private val targetContext: Context): Task {23 override fun run() {4 targetContext.externalMediaDirs.forEach { it?.deleteRecursively() }5 targetContext.noBackupFilesDir.deleteRecursively()67 targetContext.cacheDir.deleteRecursively()8 targetContext.codeCacheDir.deleteRecursively()9 targetContext.externalCacheDir?.deleteRecursively()1011 targetContext.filesDir.deleteRecursively()12 if(AndroidVersion.isAtLeastNougat()) {13 targetContext.dataDir.deleteRecursively()14 } else {15 targetContext.filesDir.parentFile?.deleteRecursively()16 }17 }18}
When an error occurs transmitting test state, or when the server connection is broken, the client automatically cancels all executions. The server can also cancel the client using the Finisher
interface.
1interface Finisher {2 void finish(int resultCode);3}
When the connection to the client is severed, the server uses the CancelSignalObserver
in a last-ditch effort to terminate the client. The CancelSignalObserver
is a BroadcastReceiver
that is tied to the lifecycle of the test process.
1class CancelSignalObserver(private val instrumentation: Instrumentation): BroadcastReceiver() {23 override fun onReceive(context: Context, intent: Intent) {4 if(intent.action == ACTION_CANCEL_SIGNAL) {5 instrumentation.finishInstrumentation(Activity.RESULT_CANCELED)6 }7 }89 fun start() {10 instrumentation.targetContext.registerReceiver(11 this,12 IntentFilter(ACTION_CANCEL_SIGNAL)13 )14 }1516 fun stop() {17 instrumentation.targetContext.unregisterReceiver(this)18 }19}
While tests are running, performance & device health data is periodically collated and stored. Protocol buffers are used for data serialization.
1message MemoryStats {23 /** The proportional set size for dalvik heap. (Doesn't include other Dalvik overhead.) */4 int32 appDalvikPss = 1;56 /** The private dirty pages used by dalvik heap. */7 int32 appDalvikPrivateDirty = 2;89 /** The shared dirty pages used by dalvik heap. */10 int32 appDalvikSharedDirty = 3;1112 /** The proportional set size for the native heap. */13 int32 appNativePss = 4;1415 /** The private dirty pages used by the native heap. */16 int32 appNativePrivateDirty = 5;1718 /** The shared dirty pages used by the native heap. */19 int32 appNativeSharedDirty = 6;2021 /** The proportional set size for everything else. */22 int32 appOtherPss = 7;2324 /** The private dirty pages used by everything else. */25 int32 appOtherPrivateDirty = 8;2627 /** The shared dirty pages used by everything else. */28 int32 appOtherSharedDirty = 9;2930 /**31 * The total memory accessible by the kernel. This is basically the32 * RAM size of the device, not including below-kernel fixed allocations33 * like DMA buffers, RAM for the baseband CPU, etc.34 */35 int64 systemTotalSizeBytes = 10;3637 /**38 * The available memory on the system. This number should not39 * be considered absolute: due to the nature of the kernel, a significant40 * portion of this memory is actually in use and needed for the overall41 * system to run well.42 */43 int64 systemAvailableSizeBytes = 11;4445 /**46 * The threshold of {@link #availMem} at which we consider memory to be47 * low and start killing background services and other non-extraneous48 * processes.49 */50 int64 systemThresholdSizeBytes = 12;5152 int32 relativeTime = 13;53}
1message NetworkStats {23 /**4 * Return number of packets received by the given UID since device boot.5 * Counts packets across all network interfaces, and always increases6 * monotonically since device boot. Statistics are measured at the network7 * layer, so they include both TCP and UDP usage.8 * <p>9 * Before {@link android.os.Build.VERSION_CODES#JELLY_BEAN_MR2}, this may return10 * {@link #UNSUPPORTED} on devices where statistics aren't available.11 * <p>12 * Starting in {@link android.os.Build.VERSION_CODES#N} this will only13 * report traffic statistics for the calling UID. It will return14 * {@link #UNSUPPORTED} for all other UIDs for privacy reasons. To access15 * historical network statistics belonging to other UIDs, use16 * {@link NetworkStatsManager}.17 *18 * @see android.os.Process#myUid()19 * @see android.content.pm.ApplicationInfo#uid20 */21 int64 rxPackets = 1;2223 /**24 * Return number of packets transmitted by the given UID since device boot.25 * Counts packets across all network interfaces, and always increases26 * monotonically since device boot. Statistics are measured at the network27 * layer, so they include both TCP and UDP usage.28 * <p>29 * Before {@link android.os.Build.VERSION_CODES#JELLY_BEAN_MR2}, this may return30 * {@link #UNSUPPORTED} on devices where statistics aren't available.31 * <p>32 * Starting in {@link android.os.Build.VERSION_CODES#N} this will only33 * report traffic statistics for the calling UID. It will return34 * {@link #UNSUPPORTED} for all other UIDs for privacy reasons. To access35 * historical network statistics belonging to other UIDs, use36 * {@link NetworkStatsManager}.37 *38 * @see android.os.Process#myUid()39 * @see android.content.pm.ApplicationInfo#uid40 */41 int64 txPackets = 2;4243 /**44 * Return number of bytes transmitted by the given UID since device boot.45 * Counts packets across all network interfaces, and always increases46 * monotonically since device boot. Statistics are measured at the network47 * layer, so they include both TCP and UDP usage.48 * <p>49 * Before {@link android.os.Build.VERSION_CODES#JELLY_BEAN_MR2}, this may50 * return {@link #UNSUPPORTED} on devices where statistics aren't available.51 * <p>52 * Starting in {@link android.os.Build.VERSION_CODES#N} this will only53 * report traffic statistics for the calling UID. It will return54 * {@link #UNSUPPORTED} for all other UIDs for privacy reasons. To access55 * historical network statistics belonging to other UIDs, use56 * {@link NetworkStatsManager}.57 *58 * @see android.os.Process#myUid()59 * @see android.content.pm.ApplicationInfo#uid60 */61 int64 txBytes = 3;6263 /**64 * Return number of bytes received by the given UID since device boot.65 * Counts packets across all network interfaces, and always increases66 * monotonically since device boot. Statistics are measured at the network67 * layer, so they include both TCP and UDP usage.68 * <p>69 * Before {@link android.os.Build.VERSION_CODES#JELLY_BEAN_MR2}, this may return70 * {@link #UNSUPPORTED} on devices where statistics aren't available.71 * <p>72 * Starting in {@link android.os.Build.VERSION_CODES#N} this will only73 * report traffic statistics for the calling UID. It will return74 * {@link #UNSUPPORTED} for all other UIDs for privacy reasons. To access75 * historical network statistics belonging to other UIDs, use76 * {@link NetworkStatsManager}.77 *78 * @see android.os.Process#myUid()79 * @see android.content.pm.ApplicationInfo#uid80 */81 int64 rxBytes = 4;8283 int32 relativeTime = 5;84}
1message ResourceUsageStats {23 int32 audioCount = 1;4 int64 audioTimeMillis = 2;5 int32 videoCount = 3;6 int64 videoTimeMillis = 4;7 int32 vibratorCount = 5;8 int64 vibratorTimeMillis = 6;9 int32 gpsSensorCount = 7;10 int64 gpsSensorTimeMillis = 8;11 int32 bluetoothCount = 9;12 int64 bluetoothTimeMillis = 10;13 int32 cameraCount = 11;14 int64 cameraTimeMillis = 12;15 int32 flashlightCount = 13;16 int64 flashlightTimeMillis = 14;17 int32 wifiScanCount = 15;18 int64 wifiScanTimeMillis = 16;19 int32 mobileRadioActiveCount = 17;20 int64 mobileRadioActiveTimeMillis = 18;2122 int64 wifiMultiCastMillis = 19;23 int64 bluetoothRxBytes = 20;24 int64 bluetoothTxBytes = 21;25 int64 bluetoothRxPackets = 22;26 int64 bluetoothTxPackets = 23;2728 /**29 * Key for a measurement of number of millseconds the wifi controller was30 * idle but turned on on behalf of this uid.31 */32 int64 wifiIdleMillis = 31;3334 /**35 * Key for a measurement of number of millseconds the bluetooth controller was36 * idle but turned on on behalf of this uid.37 */38 int64 bluetoothIdleMillis = 33;3940 /**41 * Key for a measurement of number of millseconds the mobile radio controller was42 * idle but turned on on behalf of this uid.43 */44 int64 mobileIdleMillis = 35;4546 /**47 * Key for a measurement of the estimated number of mA*ms used by this uid48 * for wifi, that is to say the number of milliseconds of wifi activity49 * times the mA current during that period.50 */51 int64 wifiPowerMams = 32;5253 /**54 * Key for a measurement of the estimated number of mA*ms used by this uid55 * for bluetooth, that is to say the number of milliseconds of activity56 * times the mA current during that period.57 */58 int64 bluetoothPowerMams = 34;5960 /**61 * Key for a measurement of the estimated number of mA*ms used by this uid62 * for mobile data, that is to say the number of milliseconds of activity63 * times the mA current during that period.64 */65 int64 mobilePowerMams = 36;6667 /**68 * Key for a measurement of number of millseconds the wifi controller was69 * active on behalf of this uid.70 */71 int64 wifiRunningMs = 37;7273 /**74 * Key for a measurement of number of millseconds that this uid held a full wifi lock.75 */76 int64 wifiFullLockMs = 38;7778 map<string, Timer> jobs = 24;79 map<string, Timer> sensors = 25;80 map<string, Timer> syncs = 26;81 map<string, Timer> wakeLocksDraw = 27;82 map<string, Timer> wakeLocksFull = 28;83 map<string, Timer> wakeLocksPartial = 29;84 map<string, Timer> wakeLocksWindow = 30;8586 int32 relativeTime = 39;8788 message Timer {89 int32 count = 1;90 int64 timeMillis = 2;91 }92}
1message StorageStats {23 Info internalStorage = 1;45 repeated Info externalStorage = 2;67 message Info {8 int64 totalSizeBytes = 1;9 int64 availableSizeBytes = 2;10 }11}
1message ThreadStats {23 repeated ThreadInfo threadsInfo = 2;45 int32 relativeTime = 3;67 message ThreadInfo {89 int64 id = 1;1011 string name = 2;1213 int32 priority = 3;1415 bool isInterrupted = 4;1617 bool isAlive = 5;1819 bool isDaemon = 6;2021 State state = 7;22 }2324 /**25 * A thread state. A thread can be in one of the following states:26 * <ul>27 * <li>{@link #NEW}<br>28 * A thread that has not yet started is in this state.29 * </li>30 * <li>{@link #RUNNABLE}<br>31 * A thread executing in the Java virtual machine is in this state.32 * </li>33 * <li>{@link #BLOCKED}<br>34 * A thread that is blocked waiting for a monitor lock35 * is in this state.36 * </li>37 * <li>{@link #WAITING}<br>38 * A thread that is waiting indefinitely for another thread to39 * perform a particular action is in this state.40 * </li>41 * <li>{@link #TIMED_WAITING}<br>42 * A thread that is waiting for another thread to perform an action43 * for up to a specified waiting time is in this state.44 * </li>45 * <li>{@link #TERMINATED}<br>46 * A thread that has exited is in this state.47 * </li>48 * </ul>49 *50 * <p>51 * A thread can be in only one state at a given point in time.52 * These states are virtual machine states which do not reflect53 * any operating system thread states.54 *55 * @since 1.556 * @see #getState57 */58 enum State {5960 /**61 * Thread state for a thread which has not yet started.62 */63 NEW = 0;6465 /**66 * Thread state for a runnable thread. A thread in the runnable67 * state is executing in the Java virtual machine but it may68 * be waiting for other resources from the operating system69 * such as processor.70 */71 RUNNABLE = 1;7273 /**74 * Thread state for a thread blocked waiting for a monitor lock.75 * A thread in the blocked state is waiting for a monitor lock76 * to enter a synchronized block/method or77 * reenter a synchronized block/method after calling78 * {@link Object#wait() Object.wait}.79 */80 BLOCKED = 2;8182 /**83 * Thread state for a waiting thread.84 * A thread is in the waiting state due to calling one of the85 * following methods:86 * <ul>87 * <li>{@link Object#wait() Object.wait} with no timeout</li>88 * <li>{@link #join() Thread.join} with no timeout</li>89 * <li>{@link LockSupport#park() LockSupport.park}</li>90 * </ul>91 *92 * <p>A thread in the waiting state is waiting for another thread to93 * perform a particular action.94 *95 * For example, a thread that has called <tt>Object.wait()</tt>96 * on an object is waiting for another thread to call97 * <tt>Object.notify()</tt> or <tt>Object.notifyAll()</tt> on98 * that object. A thread that has called <tt>Thread.join()</tt>99 * is waiting for a specified thread to terminate.100 */101 WAITING = 3;102103 /**104 * Thread state for a waiting thread with a specified waiting time.105 * A thread is in the timed waiting state due to calling one of106 * the following methods with a specified positive waiting time:107 * <ul>108 * <li>{@link #sleep Thread.sleep}</li>109 * <li>{@link Object#wait(long) Object.wait} with timeout</li>110 * <li>{@link #join(long) Thread.join} with timeout</li>111 * <li>{@link LockSupport#parkNanos LockSupport.parkNanos}</li>112 * <li>{@link LockSupport#parkUntil LockSupport.parkUntil}</li>113 * </ul>114 */115 TIMED_WAITING = 4;116117 /**118 * Thread state for a terminated thread.119 * The thread has completed execution.120 */121 TERMINATED = 5;122 }123}
1message BinderStats {23 int32 deathObjectCount = 1;45 int32 localObjectCount = 2;67 int32 proxyObjectCount = 3;89 int32 receivedTransactions = 4;1011 int32 sentTransactions = 5;1213 int32 relativeTime = 6;14}
1message FileIoStats {23 /* characters read4 * The number of bytes which this task has caused to be5 * read from storage. This is simply the sum of bytes6 * which this process passed to read(2) and similar system7 * calls. It includes things such as terminal I/O and is8 * unaffected by whether or not actual physical disk I/O9 * was required (the read might have been satisfied from10 * pagecache)11 */12 int64 charsReadBytes = 1;1314 /* characters written15 * The number of bytes which this task has caused, or16 * shall cause to be written to disk. Similar caveats17 * apply here as with rchar.18 */19 int64 charsWriteBytes = 2;2021 /* read syscalls22 * Attempt to count the number of read I/O operations—that23 * is, system calls such as read(2) and pread(2)24 */25 int64 numSysReadCalls = 3;2627 /* write syscalls28 * Attempt to count the number of write I/O operations—29 * that is, system calls such as write(2) and pwrite(2).30 */31 int64 numSysWriteCalls = 4;3233 /* bytes read34 * Attempt to count the number of bytes which this process35 * really did cause to be fetched from the storage layer.36 * This is accurate for block-backed filesystems.37 */38 int64 readBytes = 5;3940 /* bytes written41 * Attempt to count the number of bytes which this process42 * caused to be sent to the storage layer.43 */44 int64 writeBytes = 6;4546 /*47 The big inaccuracy here is truncate. If a process48 * writes 1MB to a file and then deletes the file, it will49 * in fact perform no writeout. But it will have been50 * accounted as having caused 1MB of write. In other51 * words: this field represents the number of bytes which52 * this process caused to not happen, by truncating page‐53 * cache. A task can cause "negative" I/O too. If this54 * task truncates some dirty pagecache, some I/O which55 * another task has been accounted for (in its56 * write_bytes) will not be happening.57 */58 int64 cancelledWriteBytes = 7;5960 int32 relativeTime = 8;61}
1message FrameStats {23 string activityName = 1;45 int64 animationDuration = 2;67 int64 commandIssueDuration = 3;89 int64 drawDuration = 4;1011 bool firstDrawFrame = 5;1213 int64 inputHandlingDuration = 6;1415 int64 layoutMeasureDuration = 7;1617 int64 swapBuffersDuration = 8;1819 int64 syncDuration = 9;2021 int64 totalDuration = 10;2223 int64 unknownDelayDuration = 11;2425 int64 intendedVSyncTimestamp = 12;2627 int64 vSyncTimestamp = 13;2829 int32 relativeTime = 14;30}
1message GcStats {23 /** The number of garbage collection runs. */4 int32 runCount = 1;56 /** The total duration of garbage collection runs in ms. */7 int32 runTotalDuration = 2;89 /** The total number of bytes that the application allocated. */10 int64 totalBytesAllocated = 3;1112 /** The total number of bytes that garbage collection reclaimed. */13 int64 totalBytesFreed = 4;1415 /** The number of blocking garbage collection runs. */16 int32 blockingRunCount = 5;1718 /** The total duration of blocking garbage collection runs in ms. */19 int32 blockingRunTotalDuration = 6;2021 int32 relativeTime = 9;22}
1message UnixProcessStats {23 /**4 *5 * One of the following characters, indicating process state:6 *7 *8 * * 'R' Running9 * * 'S' Sleeping in an interruptible wait10 * * 'D' Waiting in uninterruptible disk sleep11 * * 'Z' Zombie12 * * 'T' Stopped (on a signal) or (before Linux 2.6.33) trace stopped13 * * 't' Tracing stop (Linux 2.6.33 onward)14 * * 'W' Paging (only before Linux 2.6.0)15 * * 'X' Dead (from Linux 2.6.0 onward)16 * * 'x' Dead (Linux 2.6.33 to 3.13 only)17 * * 'K' Wakekill (Linux 2.6.33 to 3.13 only)18 * * 'W' Waking (Linux 2.6.33 to 3.13 only)19 * * 'P' Parked (Linux 3.9 to 3.13 only)20 *21 */22 string state = 1;2324 /**25 * The number of minor faults the process has made which have not required loading a memory26 * page from disk.27 */28 int64 numMinorFaults = 2;2930 /**31 * The number of minor faults that the process's waited-for children have made.32 */33 int64 numChildMinorFaults = 3;3435 /**36 * The number of major faults the process has made which have required loading a memory page37 * from disk.38 */39 int64 numMajorFaults = 4;4041 /**42 * The number of major faults that the process's waited-for children have made.43 */44 int64 numChildMajorFaults = 5;4546 /**47 * Amount of time that this process has been scheduled in user mode, measured in clock ticks48 * (divide by sysconf(_SC_CLK_TCK)). This includes guest time, guest_time (time spent running49 * a virtual CPU, see below), so that applications that are not aware of the guest time field50 * do not lose that time from their calculations.51 */52 int64 userTime = 6;5354 /**55 * Amount of time that this process has been scheduled in kernel mode, measured in clock ticks56 * (divide by sysconf(_SC_CLK_TCK)).57 */58 int64 systemTime = 7;5960 /**61 * Amount of time that this process's waited-for children have been scheduled in user mode,62 * measured in clock ticks (divide by sysconf(_SC_CLK_TCK)). (See also times(2).) This63 * includes guest time, cguest_time (time spent running a virtual CPU, see below).64 */65 int64 childUserTime = 8;6667 /**68 * Amount of time that this process's waited-for children have been scheduled in kernel mode,69 * measured in clock ticks (divide by sysconf(_SC_CLK_TCK)).70 */71 int64 childSystemTime = 9;7273 /**74 * Virtual memory size in bytes.75 */76 int64 virtualMemorySize = 10;7778 /**79 * Resident Set Size: number of pages the process has in real memory. This is just the pages80 * which count toward text, data, or stack space. This does not include pages which have not81 * been demand-loaded in, or which are swapped out.82 */83 int64 rss = 11;8485 /**86 * (since Linux 2.2.8)87 * CPU number last executed on.88 */89 int32 lastCpuExecutedNumber = 12;9091 /**92 * (since Linux 2.6.18)93 * Aggregated block I/O delays, measured in clock ticks (centiseconds).94 */95 int64 aggregatedBlockIoDelaysInTicks = 13;9697 int32 relativeTime = 14;98}
The logcat
process is used to capture logs. The logcat
input stream is piped to a file on the TestServer's remote storage. While logcat can directly write to a file, it can't write to the TestServer's private directory.
1private const val CMD_CLEAR_LOGCAT_BUFFERS = "logcat -b all -c"2private const val CMD_START_LOGCAT = "logcat -b all -v threadtime,epoch,printable --dividers"34class LogTunnel {56 fun start(sink: File) {7 Thread {8 getRuntime().exec(CMD_CLEAR_LOGCAT_BUFFERS)9 val logcat = getRuntime().exec(CMD_START_LOGCAT)1011 val sinkFileDescriptor = fileProvider.getAppendableFileDescriptor(sinkFileName)12 logcat.inputStream.pipe(sinkFileDescriptor)13 }.apply {14 name = "LoggerThread"15 start()16 }17 }18}
This approach might be problematic if there is ever a need for external logs. Starting from Jellybean, the logs from external apps can't be read.
Three strategies are employed to take a screenshot of the app:
The best strategy is picked and used during runtime.
1class ScreenShotterFactory(2 private val screenshotHandler: Handler3) {45 fun createOrderedBestShotters(): Set<ScreenShotter> {6 val shotters = mutableSetOf<ScreenShotter>()78 if(AndroidVersion.isAtLeastOreo()) {9 shotters += WindowScreenShotter(screenshotHandler)10 }1112 shotters += ReflectionScreenShotter(screenshotHandler)13 shotters += RootViewScreenShotter(screenshotHandler)14 return shotters15 }1617 fun createBestShotter() = createOrderedBestShotters().first()18}
The WorkRunner
sits at the heart of DART. Once the user clicks the start button, the Worker
notifies the gRPC server that it is now available for work. The worker includes its current hardware and software states in the request. This information allows the gRPC server to match a worker to the right job at the right time. For instance, a worker won't receive work if it's hot.
The work object contains details about the work to run and what configurations to use.
1message Work {2 /**3 * Unique key of the given work.4 */5 string key = 1;6 string packageName = 18;7 string testPackageName = 19;8 /**9 * Type of the test being performed.10 */11 TestType type = 2;12 /**13 * The APK under test.14 */15 RemoteFile payload = 3;16 /**17 * The max time this test execution can run before it is cancelled (default: 15m).18 * It does not include any time necessary to prepare and clean up the target device.19 * The timeout unit is seconds. The maximum possible testing time is 1800 seconds.20 */21 int32 timeout = 4;22 /**23 * The locale is a two-letter (ISO 639-1) or three-letter (ISO 639-3) representation of the language.24 */25 string locale = 5;26 /**27 * The default orientation of the device.28 */29 ScreenOrientation orientation = 6;3031 /**32 * TODO: Update the comment33 * A comma-separated, key=value map of environment variables and their desired values. The environment variables are mirrored as extra options to the am instrument -e KEY1 VALUE1 … command and passed to your test runner (typically AndroidJUnitRunner). Examples:34 Enable code coverage and provide a directory to store the coverage results when using Android Test Orchestrator (--use-orchestrator):3536 --environment-variables clearPackageData=true,coverage=true,coverageFilePath=/sdcard/37 Enable code coverage and provide a file path to store the coverage results when not using Android Test Orchestrator (--no-use-orchestrator):3839 --environment-variables coverage=true,coverageFile=/sdcard/coverage.ec40 */41 map<string, string> environmentVariables = 8;4243 /**44 * Specifies the number of times a test execution should be reattempted if one or more of its test cases fail for any reason. An execution that initially fails but succeeds on any reattempt is reported as FLAKY.45 The maximum number of reruns allowed is 10. (Default: 1, which implies one rerun.)46 */47 int32 numRetriesPerDevice = 9;48 int32 numTestRetries = 27;4950 /**51 * Monitor and record performance metrics: CPU, memory, and network usage. Enabled by default.52 */53 bool isPerformanceMonitoringEnabled = 10;54 stats.SampleFrequency sampleFrequency = 24;5556 /**57 * Enable video recording during the test. Enabled by default.58 */59 bool isVideoRecordingEnabled = 11;60 /**61 * The fully-qualified Java class name of the instrumentation test runner (default: the last name extracted from the APK manifest).62 */63 string testRunnerClassName = 12;64 /**65 * TODO: Update the comment and think of implementation66 * A list of one or more test target filters to apply (default: run all test targets).67 * Each target filter must be fully qualified with the package name, class name, or test annotation desired.68 * Any test filter supported by am instrument -e … is supported. See https://developer.android.com/reference/android/support/test/runner/AndroidJUnitRunner for more information.69 * Examples:70 --test-targets "package com.my.package.name"71 --test-targets "notPackage com.package.to.skip"72 --test-targets "class com.foo.ClassName"73 --test-targets "notClass com.foo.ClassName#testMethodToSkip"74 --test-targets "annotation com.foo.AnnotationToRun"75 --test-targets "size large notAnnotation com.foo.AnnotationToSkip"76 */77 repeated string testTargets = 13;78 repeated tests.AtomicTest tests = 14;7980 /**81 * Whether each test runs in its own Instrumentation instance with the Android Test Orchestrator (default: Orchestrator is not used, same as specifying --no-use-orchestrator).82 * Orchestrator is only compatible with AndroidJUnitRunner v1.0 or higher.83 * See https://developer.android.com/training/testing/junit-runner.html#using-android-test-orchestrator for more information about Android Test Orchestrator.84 */85 bool isIsolated = 15;86 bool shouldClearPackageData = 16;87 bool obscureScreen = 17;88 bool takeWindowAutoShots = 20;89 bool retrieveAppFiles = 21;90 bool retrieveTestFiles = 22;91 bool useSystemProfiler = 23;9293 int32 autoScreenShotFps = 25;94 int32 autoScreenShotQuality = 26;95}9697message RemoteFile {98 string url = 1;99 int64 sizeBytes = 2;100 int64 lastModified = 3;101}102103enum TestType {104 INSTRUMENTATION = 0;105}106107enum ScreenOrientation {108 PORTRAIT = 0;109 LANDSCAPE = 1;110}
The gRPC server is still in early development. Feel free to check out the Github repo.
Hopefully, I was able to pique your interest in this topic. Please drop a comment if you have any questions.