image recognition based test automation

I did a few automation scripts (selendroid/Appium) for native and hybrid apps. With quite a few game apps UI elements can’t be identified, detected and controlled using standard automation commands due to the ways the respective game engines display graphics. I looked out for a way to write automation scripts for these kind of apps nevertheless and found an approach using image recognition.

test automation example using image recognition

      • selendroid (running with Java/Eclipse/Windows 10)
      • JUnit 4.12
      • OpenCV 3.4.1 & AKAZE
      • Java image recognition classes for testing from bitbar
      • free game demo of AR-K (point & click adventure)
      • Yuntab K107 & Huawei Ascend Y300 as test devices

In the test I used image recognition primarily as flow control. The positions were taken from a reference device and are automatically adjusted by the script to match the resolution/viewport of the connected device. Since with AR-K gameplay objects rarely move, detecting the position of such an object is unnecessary but would slow down the script. The loading times on the other hand depend on the used device so a test script won’t work without some kind of flow control.

The test script runs the first scenes of the game including basic gameplay controls (dialogue, take objects, use object from inventory, scroll location).

test script structure

The script is structured in different “layers”. The top “layer” is only a list of simple gameplay/flow control commands. The actual image recognition is controlled in lower “layers”. Packages & classes are arranged accordingly.


package SelendroidTest

      • Class TestdroidImageRecognition provides test project independent gameplay functions to be called by simple commands.
      • Class AndroidSample provides test project dependent gameplay functions to be called by simple commands and contains the list of simple commands that controls the gameplay

package imagerecognition

      • Class ImageRecognition handles the image recognition process (assembles the needed parameters, does basic checks of the results).
      • Class AkazeImageFinder class controls the single steps of the image recognition process (finds both homographies, runs the AKAZE match, returns the results).
      • (from bitbar but slightly modified)

package objects

      • Classes that define basic settings of the image recognition process and location parameters of found images.
      • (from bitbar)

videos of devices controlled by the script

Yuntab K107:

Huawei Ascend Y300 (barely able to run the game):

script related downloads

AR-K demo IR based automation test script (using packages from bitbar)

used libraries, executables and external jar’s