Finding Images in Sikuli X with Java: A Step-by-Step Guide

Finding Images in Sikuli X with Java: A Step-by-Step Guide

21 May 2024 Stephan Petzl Leave a comment Tech-Help

When working with image recognition in Sikuli X using Java, it is common to encounter issues such as the software failing to find the specified image on the screen. This guide will help you troubleshoot and successfully implement image recognition in your project.

Understanding the Problem

Consider a scenario where you are trying to find an image on the screen using Sikuli X in a Java project. You might have written code similar to the following:


    Screen s = new Screen();
    Pattern test = new Pattern("/Users/yourname/Desktop/image.jpg");
    try {
        s.find(test);
    } catch (FindFailed e) {
        System.out.println("Image not found");
    }
    

Despite having the correct screenshot, you may encounter an error message indicating that the image cannot be found. Let’s explore how to resolve this.

Solution: Verifying Images

One effective method to verify images involves taking a screenshot and comparing it with the expected image. Here’s a comprehensive approach to achieve this:

Step-by-Step Implementation


    @Test
    public void verifyImages() throws IOException {
        // Take a screenshot of the screen
        File screen = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
        
        // Capture the image of the targeted element
        List imageList = driver.findElementsByXPath("//*[@class='android.widget.ImageView' and @index='0']");
        WebElement image = imageList.get(1);
        Point point = image.getLocation();
        
        // Get element dimensions
        int width = image.getSize().getWidth();
        int height = image.getSize().getHeight();
        
        // Crop the screenshot to the dimensions of the element
        BufferedImage img = ImageIO.read(screen);
        BufferedImage dest = img.getSubimage(point.getX(), point.getY(), width, height);
        ImageIO.write(dest, "png", screen);
        File file = new File("Menu.png");
        FileUtils.copyFile(screen, file);
        
        // Verify the images
        verifyImage("Menu.png", "ExpectedImage.png");
    }
    
    public void verifyImage(String image1, String image2) throws IOException {
        File fileInput = new File(image1);
        File fileOutPut = new File(image2);
        
        BufferedImage bufileInput = ImageIO.read(fileInput);
        DataBuffer dafileInput = bufileInput.getData().getDataBuffer();
        int sizefileInput = dafileInput.getSize();
        
        BufferedImage bufileOutPut = ImageIO.read(fileOutPut);
        DataBuffer dafileOutPut = bufileOutPut.getData().getDataBuffer();
        int sizefileOutPut = dafileOutPut.getSize();
        Boolean matchFlag = true;
        
        if (sizefileInput == sizefileOutPut) {
            for (int j = 0; j < sizefileInput; j++) {
                if (dafileInput.getElem(j) != dafileOutPut.getElem(j)) {
                    matchFlag = false;
                    break;
                }
            }
        } else {
            matchFlag = false;
        }
        
        Assert.assertTrue(matchFlag, "Images are not the same");
    }
    

This method ensures that the images are correctly captured and compared, which can resolve the issue of not finding the image on the screen.

Additional Considerations

Ensure that the image formats are consistent across your project. For example, if the expected image is a PNG file, the screenshot should also be saved as a PNG file.

For further reading, you can check out our detailed guides on Virtual Test Devices and Test Exception Handling.

Enhancing Your Testing with Repeato

If you find that tools like Appium are slow and tests are unstable, consider using Repeato. Repeato is a no-code test automation tool for iOS and Android that allows you to create, run, and maintain automated tests for your apps efficiently. It utilizes computer vision and AI, making the process of editing and running tests significantly faster.

With Repeato, you can ensure that your tests are not only faster to create but also run blazingly fast, providing a more reliable and efficient testing experience.

For more information, visit our documentation or download the tool to get started.

Like this article? there’s more where that came from!