{"id":16,"date":"2018-04-02T19:53:45","date_gmt":"2018-04-02T19:53:45","guid":{"rendered":"http:\/\/bakhtosh.bplaced.net\/?page_id=16"},"modified":"2018-04-07T07:11:18","modified_gmt":"2018-04-07T07:11:18","slug":"test-automation","status":"publish","type":"page","link":"http:\/\/bakhtosh.bplaced.net\/?page_id=16","title":{"rendered":""},"content":{"rendered":"<h3>image recognition based test automation<\/h3>\n<p>I did a few automation scripts (selendroid\/Appium) for native and hybrid apps. With quite a few game apps UI elements can&#8217;t be identified, detected and controlled using standard automation commands due to the ways the respective game engines display graphics. I looked out for a way to write automation scripts for these kind of apps nevertheless and found an approach using image recognition.<\/p>\n<p><strong>test automation example using image recognition<\/strong><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>selendroid (running with Java\/Eclipse\/Windows 10)<\/li>\n<li>JUnit 4.12<\/li>\n<li>OpenCV 3.4.1 &amp; AKAZE<\/li>\n<li>Java image recognition classes for testing from <a href=\"https:\/\/bitbar.com\/using-opencv-and-akaze-for-mobile-app-and-game-testing\/amp\/\">bitbar<\/a><\/li>\n<li>free game demo of <a href=\"https:\/\/play.google.com\/store\/apps\/details?id=com.GatoSalvaje.ARKIEng\">AR-K <\/a>(point &amp; click adventure)<\/li>\n<li>Yuntab K107 &amp; Huawei Ascend Y300 as test devices<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>In the test I used image recognition primarily as flow control. The positions were taken from a reference device and are automatically adjusted by the script to match the resolution\/viewport of the connected device. Since with AR-K gameplay objects rarely move, detecting the position of such an object is unnecessary but would slow down the script. The loading times on the other hand depend on the used device so a test script won\u2019t work without some kind of flow control.<\/p>\n<p>The test script runs the first scenes of the game including basic gameplay controls (dialogue, take objects, use object from inventory, scroll location).<\/p>\n<h3>test script structure<\/h3>\n<p>The script is structured in different &#8220;layers&#8221;. The top &#8220;layer&#8221; is only a list of simple gameplay\/flow control commands. The actual image recognition is controlled in lower &#8220;layers&#8221;. Packages &amp; classes are arranged accordingly.<\/p>\n<p><strong><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-101\" src=\"http:\/\/bakhtosh.bplaced.net\/wp-content\/uploads\/2018\/04\/Structurea.png\" alt=\"\" width=\"878\" height=\"460\" srcset=\"http:\/\/bakhtosh.bplaced.net\/wp-content\/uploads\/2018\/04\/Structurea.png 878w, http:\/\/bakhtosh.bplaced.net\/wp-content\/uploads\/2018\/04\/Structurea-300x157.png 300w, http:\/\/bakhtosh.bplaced.net\/wp-content\/uploads\/2018\/04\/Structurea-768x402.png 768w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><br \/>\npackage SelendroidTest<\/strong><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Class<a href=\"http:\/\/bakhtosh.bplaced.net\/TestAutomation\/TestdroidImageRecognition.java.html\"><em> TestdroidImageRecognition<\/em><\/a>\u00a0provides test project independent gameplay functions to be called by\u00a0simple commands.<\/li>\n<li>Class <a href=\"http:\/\/bakhtosh.bplaced.net\/TestAutomation\/AndroidSample.java.html\"><em>AndroidSample<\/em><\/a>\u00a0provides test project dependent gameplay functions to be called by\u00a0simple\u00a0commands and contains the list of simple commands that controls the gameplay<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><strong>package imagerecognition<\/strong><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Class<a href=\"http:\/\/bakhtosh.bplaced.net\/TestAutomation\/ImageRecognition.java.html\"><em> ImageRecognition<\/em><\/a>\u00a0handles the image recognition process (assembles the needed parameters, does basic checks of the results).<\/li>\n<li>Class <a href=\"http:\/\/bakhtosh.bplaced.net\/TestAutomation\/AkazeImageFinder.java.html\"><em>AkazeImageFinder<\/em><\/a> class controls the single steps of the image recognition process (finds both homographies, runs the AKAZE match, returns the results).<\/li>\n<li>(from <a href=\"https:\/\/bitbar.com\/using-opencv-and-akaze-for-mobile-app-and-game-testing\/amp\/\">bitbar<\/a> but slightly modified)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><strong>package objects<\/strong><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li><a href=\"https:\/\/github.com\/bitbar\/opencv-library\/tree\/master\/library\/src\/main\/java\/objects\"><em>Classes<\/em><\/a> that define basic settings of the image recognition process and location parameters of found images.<\/li>\n<li>(from <a href=\"https:\/\/bitbar.com\/using-opencv-and-akaze-for-mobile-app-and-game-testing\/amp\/\">bitbar<\/a>)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h3>videos of devices controlled by the script<\/h3>\n<p><strong><em>Yuntab K107:<\/em><\/strong><\/p>\n<p><iframe loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/rgJCfPwPWss\" width=\"840\" height=\"473\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<p><em><strong>Huawei Ascend Y300<\/strong> <\/em>(barely able to run the game):<\/p>\n<p><iframe loading=\"lazy\" src=\"https:\/\/www.youtube.com\/embed\/Df5wrhCjL3s\" width=\"840\" height=\"473\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<h3>script related downloads<\/h3>\n<p><strong>AR-K demo IR based automation test script<\/strong>\u00a0(using packages from <a href=\"https:\/\/bitbar.com\/using-opencv-and-akaze-for-mobile-app-and-game-testing\/amp\/\">bitbar<\/a>)<\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li><a href=\"http:\/\/bakhtosh.bplaced.net\/TestAutomation\/ARK-selendroid-Test_source-and-queryimages.zip\">sources and queryimages<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><strong>used libraries, executables and external jar\u2019s<\/strong><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li><a href=\"https:\/\/mvnrepository.com\/artifact\/io.selendroid\/selendroid-client\/0.17.0\">selendroid-client-0.17.0.jar<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/selendroid\/selendroid\/releases\">selendroid-standalone-0.17.0-with-dependencies.jar<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/SeleniumHQ\/selenium\/releases\">selenium-java-3.7.1.zip<\/a><\/li>\n<li><a href=\"https:\/\/www.opencv.org\/releases.html\">OpenCV 3.4.1<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/bitbar\/opencv-library\/tree\/master\/example\/lib\/win\/akaze\">AKAZE executable<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/JetBrains\/adt-tools-base\/blob\/master\/misc\/screenshot2\/src\/main\/java\/com\/android\/screenshot\/Screenshot.java\">screenshot2 executable\u00a0<\/a>(compiled with alternate adb path detection)<\/li>\n<li><a href=\"https:\/\/github.com\/junit-team\/junit4\/releases\">junit-4.12.jar<\/a><\/li>\n<li><a href=\"https:\/\/mvnrepository.com\/artifact\/org.hamcrest\/hamcrest-core\/1.3\">hamcrest-core-1.3.jar<\/a><\/li>\n<li><a href=\"https:\/\/www.apache.org\/dist\/commons\/io\/binaries\/\">commons-io-2.6.jar<\/a><\/li>\n<li><a href=\"https:\/\/mvnrepository.com\/artifact\/org.json\/json\/20140107\">json-20140107.jar<\/a><\/li>\n<li><a href=\"https:\/\/www.slf4j.org\/download.html\">slf4j-api-1.7.25.jar<\/a><\/li>\n<li><a href=\"https:\/\/www.slf4j.org\/download.html\">slf4j-simple-1.7.25.jar<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>image recognition based test automation I did a few automation scripts (selendroid\/Appium) for native and hybrid apps. With quite a few game apps UI elements can&#8217;t be identified, detected and controlled using standard automation commands due to the ways the respective game engines display graphics. I looked out for a way to write automation scripts &hellip; <a href=\"http:\/\/bakhtosh.bplaced.net\/?page_id=16\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-16","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"http:\/\/bakhtosh.bplaced.net\/index.php?rest_route=\/wp\/v2\/pages\/16","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/bakhtosh.bplaced.net\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"http:\/\/bakhtosh.bplaced.net\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"http:\/\/bakhtosh.bplaced.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/bakhtosh.bplaced.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=16"}],"version-history":[{"count":11,"href":"http:\/\/bakhtosh.bplaced.net\/index.php?rest_route=\/wp\/v2\/pages\/16\/revisions"}],"predecessor-version":[{"id":133,"href":"http:\/\/bakhtosh.bplaced.net\/index.php?rest_route=\/wp\/v2\/pages\/16\/revisions\/133"}],"wp:attachment":[{"href":"http:\/\/bakhtosh.bplaced.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=16"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}