Get started with Game Loop tests

It can be hard to automate game testing when gaming apps are built on different UI frameworks. Game Loop tests allow you to integrate your native tests with Test Lab and easily run them on devices you select. A Game Loop test runs your test through your gaming app while simulating the actions of a real player. This guide shows you how to run a Game Loop test, then view and manage your test results in the Firebase console.

Depending on your game engine, you can implement tests with single or multiple loops. A loop is a full or partial run-through of your test on your gaming app. Game loops can be used to:

  • Run a level of your game the same way an end user would play it. You can either script the input of the user, let the user be idle, or replace the user with an AI if it makes sense in your game (e.g., say you have a race car gaming app and already have an AI implemented. You can easily put an AI driver in charge of the user's input).
  • Run your game at the highest quality setting to see if devices support it.
  • Run a technical test (compile multiple shaders, execute them, check that the output is as expected, etc).

You can run a Game Loop test on a single test device, a set of test devices, or on Test Lab. However, we don't recommend running Game Loop tests on virtual devices because they have lower graphics frame rates than physical devices.

Before you begin

To implement a test, you must first configure your app for Game Loop tests.

  1. In your app manifest, add a new intent filter to your activity:

    <activity android:name=".MyActivity">
   <intent-filter>
       <action android:name="com.google.intent.action.TEST_LOOP"/>
       <category android:name="android.intent.category.DEFAULT"/>
       <data android:mimeType="application/javascript"/>
   </intent-filter>
   <intent-filter>
      ... (other intent filters here)
   </intent-filter>
</activity>

    This allows Test Lab to launch your game by triggering it with a specific intent.

  2. In your code (we recommend inside the onCreate method declaration), add the following:

    Kotlin+KTX 

    val launchIntent = intent
if (launchIntent.action == "com.google.intent.action.TEST_LOOP") {
    val scenario = launchIntent.getIntExtra("scenario", 0)
    // Code to handle your game loop here
}
    MainActivity.kt

    Java 

    Intent launchIntent = getIntent();
if(launchIntent.getAction().equals("com.google.intent.action.TEST_LOOP")) {
    int scenario = launchIntent.getIntExtra("scenario", 0);
    // Code to handle your game loop here
}
    MainActivity.java

    This allows your activity to check the intent that launches it. You can also add this code later if you prefer (e.g., after initially loading your game engine).

  3. Recommended: At the end of the test, add:

    Kotlin+KTX 

    yourActivity.finish()
    MainActivity.kt

    Java 

    yourActivity.finish();
    MainActivity.java

    This closes your app when the Game Loop test is complete. The test relies on your app's UI framework to start the next loop, and closing your app tells it that the test is finished.

Create and run a Game Loop test

After you configure your app for Game Loop tests, you can immediately create a test and run it in your gaming app. You can choose to run a test in Test Lab using either the Firebase console or the gcloud command line interface (CLI), or on a local device using the Test Loop Manager.

Run on a local device

Test Lab's Test Loop Manager is an open source app that helps you integrate Game Loop tests and run them on your local devices. It also allows your Quality Assurance team to run the same game loops on their devices.

To run a test on a local device using the Test Loop Manager:

  1. Download the Test Loop Manager on a phone or tablet and install it by running: 
    adb install testloopmanager.apk
  2. On your device, open the Test Loop Apps app on your phone or tablet. The app displays a list of apps on your device that can be run with game loops. If you don't see your gaming app here, make sure your intent filter matches the one described in the first step of the Before you begin section.
  3. Select your gaming app, then select the number of loops you want to run. Note: At this step, you can choose to run a subset of loops instead of just one loop. For more information on running multiple loops at once, see Optional features.
  4. Click Run test. Your test starts running immediately.

Run in Test Lab

You can run a Game Loop test in Test Lab using either the Firebase console or the gcloud CLI. Before you begin, if you haven't already, open the Firebase console and create a project.

Use the Firebase console

  1. In the Firebase console, click Test Lab from the left panel.
  2. Click Run Your First Test (or Run a Test if your project has previously run a test).
  3. Select Game Loop as the test type, and then click Continue.
  4. Click Browse, and then browse to your app's .apk file. Note: At this step, you can choose to run a subset of loops instead of just one loop. For more information on running multiple loops at once, see Optional features.
  5. Click Continue.
  6. Select the physical devices to use to test your app.
  7. Click Start Tests.

For more information on getting started with the Firebase console, see Start testing with the Firebase console.

Use the gcloud command-line (CLI)

  1. If you haven't already, download and install the Google Cloud SDK

  2. Sign in to the gcloud CLI using your Google Account:

    gcloud auth login

  3. Set your Firebase project in gcloud, where PROJECT_ID is the ID of your Firebase project:

    gcloud config set project PROJECT_ID

  4. Run your first test:

    gcloud firebase test android run \
 --type=game-loop --app=<var>path-to-apk</var> \
 --device model=herolte,version=23

For more information on getting started with the gcloud CLI, see Start testing from the gcloud command line.

Optional features

Test Lab offers several optional features that let you further customize your tests, including the ability to write output data, support for multiple game loops, and labels for related loops.

Write output data

Your Game Loop test can write output to a file specified in the launchIntent.getData() method. After you run a test, you can access this output data in the Test Lab section of the Firebase console (see Game Loop test output file example).

Test Lab follows best practices for sharing a file between apps described in Sharing a File. In your activity’s onCreate() method, where your intent is located, you can check your data output file by running following code:

Kotlin+KTX 

val launchIntent = intent
val logFile = launchIntent.data
logFile?.let {
    Log.i(TAG, "Log file ${it.encodedPath}")
    // ...
}
MainActivity.kt

Java 

Intent launchIntent = getIntent();
Uri logFile = launchIntent.getData();
if (logFile != null) {
    Log.i(TAG, "Log file " + logFile.getEncodedPath());
    // ...
}
MainActivity.java

If you want to write to the file from the C++ side of your game app, you can pass in the file descriptor instead of the file path:

Kotlin+KTX 

val launchIntent = intent
val logFile = launchIntent.data
var fd = -1
logFile?.let {
    Log.i(TAG, "Log file ${it.encodedPath}")
    fd = try {
        contentResolver
            .openAssetFileDescriptor(logFile, "w")!!
            .parcelFileDescriptor
            .fd
    } catch (e: FileNotFoundException) {
        e.printStackTrace()
        -1
    } catch (e: NullPointerException) {
        e.printStackTrace()
        -1
    }
}

// C++ code invoked here.
// native_function(fd);
MainActivity.kt

Java 

Intent launchIntent = getIntent();
Uri logFile = launchIntent.getData();
int fd = -1;
if (logFile != null) {
    Log.i(TAG, "Log file " + logFile.getEncodedPath());
    try {
        fd = getContentResolver()
                .openAssetFileDescriptor(logFile, "w")
                .getParcelFileDescriptor()
                .getFd();
    } catch (FileNotFoundException e) {
        e.printStackTrace();
        fd = -1;
    } catch (NullPointerException e) {
        e.printStackTrace();
        fd = -1;
    }
}

// C++ code invoked here.
// native_function(fd);
MainActivity.java

C++

#include <unistd.h>
JNIEXPORT void JNICALL
Java_my_package_name_MyActivity_native_function(JNIEnv *env, jclass type, jint log_file_descriptor) {
// The file descriptor needs to be duplicated.
int my_file_descriptor = dup(log_file_descriptor);
}

Output file example

You can use output data files (formatted like the example below) to display game loop test results in the Test Lab section of the Firebase console. Areas shown as /.../ can contain any custom fields that you need, as long as they don't conflict with the names of other fields used in this file:

{
  "name": "test name",
  "start_timestamp": 0, // Timestamp of the test start (in us).
                           Can be absolute or relative
  "driver_info": "...",
  "frame_stats": [
    {
      "timestamp": 1200000, // Timestamp at which this section was written
                               It contains value regarding the period
                               start_timestamp(0) -> this timestamp (1200000 us)
      "avg_frame_time": 15320, // Average time to render a frame in ns
      "nb_swap": 52, // Number of frame rendered
      "threads": [
        {
          "name": "physics",
          "Avg_time": 8030 // Average time spent in this thread per frame in us
        },
        {
          "name": "AI",
          "Avg_time": 2030 // Average time spent in this thread per frame in us
        }
      ],
      /.../ // Any custom field you want (vertices display on the screen, nb units …)
    },
    {
      // Next frame data here, same format as above
    }
  ],
  "loading_stats": [
    {
      "name": "assets_level_1",
      "total_time": 7850, // in us
      /.../
    },
    {
      "name": "victory_screen",
      "total_time": 554, // in us
      /.../
    }

  ],
  /.../, // You can add custom fields here
}

Multiple game loops

You might find it useful to run multiple game loops in your app. A loop is a complete run-through of your game app from beginning to end. For example, if you have multiple levels in your game, you might want to have one game loop to launch each level instead of having one loop that iterates through all of them. That way, if your app crashes on level 32, you can directly launch that game loop to reproduce the crash and test bug fixes.

To enable your app to run multiple loops at once:

  • If you're running a test with the Test Loop Manager:

    1. Add the following line to your app's manifest, inside the <application> element: 

      <meta-data
  android:name="com.google.test.loops"
  android:value="5" />

      This launch intent contains the target loop as an integer parameter. In the android:value field, you can specify an integer from 1 to 1024 (the maximum number of loops allowed for a single test). Note that loops are indexed starting from 1, not 0.

    2. In the Test Loop Manager app, a selection screen appears that allows you to select which loop(s) you want to run. If you select multiple loops, each loop is launched in sequence after the preceding loop completes.

  • If you're running a test with the Firebase console, enter a list or a range of loop numbers in the Scenarios field.

  • If you're running a test with the gcloud CLI, specify a list of loop numbers by using the --scenario-numbers flag. For example, --scenario-numbers=1,3,5 runs loops 1, 3, and 5.

  • If you're writing C++ and want to change the behavior of your loop, pass the following extra to your native C++ code:

    Kotlin+KTX 

    val launchIntent = intent
val scenario = launchIntent.getIntExtra("scenario", 0)
    MainActivity.kt

    Java 

    Intent launchIntent = getIntent();
int scenario = launchIntent.getIntExtra("scenario", 0);
    MainActivity.java

    You can now change the behavior of your loop based on the resulting int value.

Label game loops

When you label your game loops with one or more scenario labels, you and your QA team can easily launch a set of related game loops (e.g., "all compatibility game loops") and test them in a single matrix. You can create your own labels or use the predefined labels offered by Test Lab:

  • com.google.test.loops.player_experience: For loops used to reproduce a real user's experience when playing the game. The goal of testing with these loops is to find issues that a real user would face while playing the game.
  • com.google.test.loops.gpu_compatibility: For loops used to test GPU-related issues. The goal of testing with these loops is to execute GPU code that might not run properly run in production, to expose issues with hardware and drivers.
  • com.google.test.loops.compatibility: For loops used to test a broad range of compatibility issues, including I/O issues and OpenSSL issues.
  • com.google.test.loops.performance: For loops used to test the performance of the device. For example, a game might run at the most complex graphics settings to see how a new device behaves.

To enable your app to run loops with the same label:

  • If you're running a test with the Test Loop Manager:

    1. In your app's manifest, add the following meta-data line and replace LABEL_NAME with a label of your choice:

      <meta-data
 android:name="com.google.test.loops.LABEL_NAME"
 android:value="1,3-5" />

      In the android:value field, you can specify a range or a set of integers from 1 to 1024 (the maximum number of loops allowed for a single test) that represent the loops you want to label. Note that loops are indexed starting from 1, not 0. For example, android:value="1,3-5" applies LABEL_NAME to loops 1, 3, 4, and 5.

    2. In the Test Loop Manager app, enter one or more labels in the Labels field.

  • If you're running a test with the Firebase console, enter one or more labels in the Labels field.

  • If you're running a test with the gcloud CLI, specify one or more scenario labels by using the --scenario-labels flag (e.g., --scenario-labels=performance,gpu).

App licensing support

Test Lab supports apps that use the App Licensing service offered by Google Play. To successfully check licensing when testing your app with Test Lab, you must publish your app to the production channel in the Play store. To test your app in the alpha or beta channel using Test Lab, remove the licensing check before uploading your app to Test Lab.

Known issues

Game Loop tests in Test Lab have the following known issues:

  • Some crashes do not support backtraces. For example, some release builds may suppress the output of the debuggerd process using prctl(PR_SET_DUMPABLE, 0). To learn more, see debuggerd.
  • API Level 19 is not currently supported due to file permission errors.