image recognition based test automation
I did a few automation scripts (selendroid/Appium) for native and hybrid apps. With quite a few game apps UI elements can’t be identified, detected and controlled using standard automation commands due to the ways the respective game engines display graphics. I looked out for a way to write automation scripts for these kind of apps nevertheless and found an approach using image recognition.
test automation example using image recognition
In the test I used image recognition primarily as flow control. The positions were taken from a reference device and are automatically adjusted by the script to match the resolution/viewport of the connected device. Since with AR-K gameplay objects rarely move, detecting the position of such an object is unnecessary but would slow down the script. The loading times on the other hand depend on the used device so a test script won’t work without some kind of flow control.
The test script runs the first scenes of the game including basic gameplay controls (dialogue, take objects, use object from inventory, scroll location).
test script structure
The script is structured in different “layers”. The top “layer” is only a list of simple gameplay/flow control commands. The actual image recognition is controlled in lower “layers”. Packages & classes are arranged accordingly.
- Class ImageRecognition handles the image recognition process (assembles the needed parameters, does basic checks of the results).
- Class AkazeImageFinder class controls the single steps of the image recognition process (finds both homographies, runs the AKAZE match, returns the results).
- (from bitbar but slightly modified)
videos of devices controlled by the script
Huawei Ascend Y300 (barely able to run the game):
script related downloads
AR-K demo IR based automation test script (using packages from bitbar)
used libraries, executables and external jar’s
- OpenCV 3.4.1
- AKAZE executable
- screenshot2 executable (compiled with alternate adb path detection)