Android Test Station: Running CTS for com.android.cts.priv.ctsshim

YouTube video

Introduction

In this article, we will discuss how to run the Android Compatibility Test Suite (CTS) using the Android Test Station. The CTS is an extensive set of tests that ensures compatibility between the Android operating system and device implementations. We will go through the step-by-step process of running the CTS and explore various customization options available.

Step 1: Finding and Running a CTS Version

To start running the CTS with the Android Test Station, you need to find the desired CTS version from the bundled CTS configs. These configs can be found on the test suites page after installing the tool. Once you have selected a CTS version, simply click on the “Run Task” button to initiate the test run.

Step 2: Customizing Test Run Parameters

After clicking the “Run Task” button, you will be taken to a step-by-step UI where you can customize various test run parameters.

  1. Step 2.1: CTS Command Line Customization

In the first step, you will see the default CTS command line, which runs the full plan. However, you have the option to customize the command line to run specific tasks, cases, or modules. You can make modifications to the plan name and options based on your requirements.

  1. Step 2.2: Adjusting Maximum Retry Number

By expanding the advanced settings, you can also modify the maximum retry number. This allows you to control the number of retries for failed test cases.

  1. Step 2.3: Setting Labels for the Test Run

To make it easier to find the test run later on, you can set labels for your run. For example, if you are running a test on a Pixel 2 XL device, you can label it with the model name.

Step 3: Choosing the Device

In the second step, you can choose the device on which you want to run the test suite. The Android Test Station supports running the tests on one or more devices simultaneously. You don’t need to worry about sharding options as the tool will automatically set them for you.

Step 4: Device Actions and Test Resources

In the third step, you have the option to choose device actions to be executed before the first run and subsequent retries. For optimal results, it is recommended to add reboot and CTS device setup actions. These actions help minimize interference between retries.

Additionally, you can customize the download location of the CTS package in the last step. The default value is set to download the latest release version from Google’s server. However, you can override it to use a pre-downloaded CTS package on your machine.

Starting the Test Run

Once all the parameters are set, simply click the “Start Test Run” button to begin the test run. You will be redirected to a test run detail page where you can monitor the status of the test run. Live logs are also available for real-time updates.

Multiple Test Runs and Parallel Execution

If you have multiple devices, you can run multiple test runs in parallel. The Android Test Station allows you to manage and monitor all current and previous test runs from the test runs page. By clicking the “View” button, you can access the detail page of each test run.

Checking Test Results

Once a test run is completed, you can check the result summary from the test results tab. By clicking the “View Output Files” button, you can browse through the test result files. This allows you to review the test results and identify any failed test cases.

Rerunning Failed Test Cases

If you encounter failed test cases, it is recommended to rerun them to ensure they are real failures. To schedule a rerun, click on the “Rerun” button. This will take you back to the step-by-step UI used to schedule the initial run. The only difference is that the “Previous Task Frame” field will be pre-populated with the ID of the first test run.

During the rerun, you have the option to customize the retry command by adding additional options such as --module or --exclude filter. You can also choose to keep the current device or select a different one. Make sure to select a device with the same build fingerprint as the first test run.

Customizing Test Resources

In the case of test resources, you can override the download URL of the Android CTS zip file. This allows you to use a local file if needed. By clicking on the “Local File” tab, you can upload the desired file to the Test Station.

Exporting Test Results

After a test run is completed, you may need to export the results for further analysis or to retry on a different device connected to another workstation. You can also upload the results to the Android Partner Academy (APA) for build approval.

To export the results, simply click on the “Export Result” button and download the results. This will provide you with the necessary files for further analysis or submission to APA.

Rerunning CTS Results

In some cases, you may want to rerun a CTS result from a manual execution or from another Test Station server. To do this, schedule a new test run with the test results ZIP file. Select the “Remote” option as the previous test run type and upload the test results ZIP file. The rest of the configuration remains the same as for a normal test run.

Conclusion

Running the Android Compatibility Test Suite (CTS) with the Android Test Station is a simple and customizable process. By following the step-by-step instructions outlined in this article, you can ensure device compatibility and identify any failed test cases. Use the Test Station’s features to customize test runs, rerun failed test cases, and export results for further analysis or submission to the Android Partner Academy (APA).