Table of contents

Introduction

Hello and welcome.

My name is Michal, and I’m a Quality Engineer in Red Hat. I’ve been in Red Hat for over 7 years now and my responsibility is to write, maintain and improve an automation stack for GNOME Applications on RHEL while also having a responsibility for specific components and their automation suites, like gnome-control-center, gnome-terminal, gnome-shell, evolution and more.

In this article I will show and explain how our team (DeskopQE) uses Accessibility to automate and test GNOME Applications.

I will explain in great detail what we are using, how are we using it, how to do basic setup and execution of a testing suite. After reading this article you will have enough information to successfully start gnome-terminal automation suite on Fedora 38 with Wayland.

Please keep in mind that nobody is perfect, and we are no exception. If you see anything that we are doing wrong, let us know. We love to learn and welcome any feedback that would improve our automation suites.

Automation stack for GNOME Applications

First, let’s go over individual parts of our automation stack that we use.

What do we use? Automation API - dogtail

We use Assistive Technology - Service Provider Interface (AT-SPI) which is a set of interfaces that allow access technologies, such as screen readers, to programmatically determine what is being displayed on the screen and simulate keyboard and mouse events. It can be also used for automated testing.

To do this, we utilize Python module GObject introspection that holds Python bindings and support for GTK toolkit and GNOME applications, namely Atspi module.

  from gi.repository import Atspi

We are using the dogtail project as an API for all we do. The reality of things is that the dogtail is a wrapper over Atspi and pyatspi2 (which is a wrapper over Atspi itself). There are some parts that are implemented only in dogtail project for ease of use, but if we take out dogtail and pyatspi2, everything can work only with Atspi module with some modifications.

    ┌──────────────────────────────────┐
    │         ┌──────────────────────┐ │
    │         │          ┌────────┐  │ │
    │ dogtail │ pyatspi2 │ AT-SPI │  │ │
    │         │          └────────┘  │ │
    │         └──────────────────────┘ │
    └──────────────────────────────────┘

Below you can see a simple example of how we interact with applications, provided you have a running session, the accessibility toolkit is enabled, and you are using Xorg. To make this example fully functional with Wayland, you will need information from the next section. This example will open overview, start gnome-terminal and execute a command:

#!/usr/bin/python3
from dogtail.tree import root
from dogtail.rawinput import typeText, pressKey
from time import sleep

# First open the application.
pressKey("Super") # Open overview.
sleep(1) # Give overview a little time to show.

typeText("Terminal") # Search application.
pressKey("Enter") # Confirm by Enter.

sleep(1) # Give the application a little time to start.

# Load application root to variable.
app = root.application("gnome-terminal-server") # Save root object.

# Search the application tree for objects.
app.child("Terminal", "terminal").click(3) # Right click in the middle of the terminal.

sleep(1) # Give application a little time to open the menu.

# Find the item Show Menubar, that is showing on the screen.
show_menubar = app.findChild(
    lambda x: x.name == "Show Menubar"
    and x.roleName == "check menu item"
    and x.showing
)
# If the Show Menubar is not checked, click it.
if not show_menubar.checked:
    show_menubar.click()
else:
    # Close the menu if the Menubar is already checked.
    pressKey("Esc")

sleep(1) # Give the application a little time to close the menu.

app.child("File", "menu").click() # Find File menu and click.
app.child("New Tab", "menu").click() # Find New Tab menu and click.
app.findChild(lambda x: "1." in x.name).click() # Confirm the profile.

sleep(1) # Give the terminal a little time to open the New Tab.

# Execute the command.
typeText("echo Hello World") # Type the command to terminal.
pressKey("Enter") # Confirm the command.

With these basic queries we can do quite a lot.

How are we dealing with automation on Wayland? The gnome-ponytail-daemon

While the example provided above will work for Xorg. For Wayland there is an extra step we need to do in order to successfully navigate the application via correct coordinates.

With Wayland, there is no translation of window coordinates to desktop coordinates. Meaning that if we have a button in the top left corner of the application and the application is in the middle of the screen, the click itself will be translated to the top left corner of the screen and not into the application window. Therefore, missing the click and failing the step.

Fortunately we have a solution in the form of a gnome-ponytail-daemon. This project was written by Olivier Fourdan @ofourdan and was based on gnome-remote-desktop written by Jonas Adahl @jadahl. I would also like to mention a new developer that recently joined our team José Expósito @jexposit who already contributed to the gnome-ponytail-daemon project. Changes are communicated to us after which we can verify the function with our suites. Since we depend on gnome-ponytail-daemon for translation of coordinates, any issue will show quickly.

With GNOME on Wayland we have screencasts and remote desktop APIs that can be used for controlling the keyboard and cursor. Wayland does not expose desktop coordinates and the usual process will return window coordinates of the various application widgets, which is where the RecordWindow method from screencast can be used, as it will translate global coordinates into surface relative coordinates.

To record any given window, there is a need to identify such window, for this we use window-list API. The Introspect D-BUS API in mutter provides a way to list all toplevel windows.

Functions from this project that connect a window and do actions in it for automation are integrated in dogtail project to be used seamlessly without any required user setup. So in effect, everything between Xorg and Wayland on our automation side is exactly the same and functions are handled differently on the dogtail side.

There are of course some shortcomings. In rare cases we need to test a feature that dogtail does not have a function for, the user needs to manually change window connection, but such cases are rare enough that examples for these cases fall out of the scope of this article.

If there would be a need for a very specific test case, the dogtail and ponytail can be used in unusual ways (read: hacked around). With enough knowledge about the system and how these APIs work, a lot of things can be done even if they are not provided as methods and functions. We have years of experience in this. Accessibility is not always in the state we would like it to be in, but we can work around issues most of the time.

I will show how to build and use ponytail in the GNOME Terminal Full Project Example section.

The ponytail project repository is located here gnome-ponytail-daemon

You will notice that in the project is an example how to use gnome-ponytail-daemon for automation without a dogtail API. The ponytail API is used by dogtail when required.

Giving the API a structure to be used in automation - behave

So now we have explained what APIs we use at the base level. Now we need some structure to use this and have the code base scalable. We use behave and its file structure as our automation structure.

While the behave structure is very simple:

    features
    ├── environment.py
    ├── main.feature
    └── steps
        └── steps.py

We need some extra parts for our use cases and improvement of readability:

    gnome-terminal # automation project - component name
    ├── features # behave main directory
    |   ├── environment.py # behave environment file
    |   ├── scenarios # folder to have our scenarios in
    |   |    ├── general.feature # specific scenarios
    |   |    └── example.feature # specific scenarios
    |   └── steps # behave steps directory containing implementation of scenarios, can have multiple files
    |       └── steps.py # steps implementation
    ├── behave.ini # behave configuration file which we use to adjust our formatter behave module
    ├── mapper.yaml # DesktopQE specific file for our CI
    └── runtest.sh # main script that does some basic setup, scenario start and reporting result to our CI

The feature files have following contents:

# -- FILE: features/scenarios/example.feature
Feature: Showing off behave

  @showing_off_behave
  Scenario: Run a simple test
    Given we have behave installed
    When we implement 5 tests
    Then behave will test them for us!

The steps.py file contents are as follows:

# -- FILE: features/steps/steps.py
from behave import given, when, then, step

@given('we have behave installed')
def step_impl(context):
    pass

@when('we implement {number:d} tests')
def step_impl(context, number):  # -- NOTE: number is converted into integer
    assert number > 1 or number == 0
    context.tests_count = number

@then('behave will test them for us!')
def step_impl(context):
    assert context.failed is False
    assert context.tests_count >= 0

@step('Dummy Step')
def dummy_step(context):
    pass

To better visualize the structure of behave's .feature files. Single Feature file can contain multiple Scenarios and each Scenario contains Steps that are implemented in steps.py file that is located in steps directory:

┌───────────────────────────────────────────────────────────────────────────────────┐
│         ┌────────────────────────────────┐ ┌────────────────────────────────┐     │
│         │          ┌──────┐ ┌──────┐     │ │          ┌──────┐ ┌──────┐     │     │
│ Feature │ Scenario │ Step │ │ Step │ ... │ │ Scenario │ Step │ │ Step │ ... │ ... │
│         │          └──────┘ └──────┘     │ │          └──────┘ └──────┘     │     │
│         └────────────────────────────────┘ └────────────────────────────────┘     │
└───────────────────────────────────────────────────────────────────────────────────┘

Now we can run behave:

$ behave
Feature: Showing off behave # features/example.feature:2

  Scenario: Run a simple test          # features/example.feature:5
    Given we have behave installed     # features/steps/example_steps.py:4
    When we implement 5 tests          # features/steps/example_steps.py:8
    Then behave will test them for us! # features/steps/example_steps.py:13

1 feature passed, 0 failed, 0 skipped
1 scenario passed, 0 failed, 0 skipped
3 steps passed, 0 failed, 0 skipped, 0 undefined

The behave run was a success, now we can use behave to run all of our test cases with ease.

For all of our test cases, we are using tags to differentiate between different scenarios.

So while $ behave will start every single scenario defined in feature files, our best practice is to start test cases one by one and separate them into their own result pages, which I will get into later.

We are running behave like this $ behave -kt showing_off_behave. The -k (note that in development version of behave v1.2.7.dev# this is available as --no-skipped, in the current pypi version 1.2.6 -k is still working) will skip all unexecuted tests, so they are not printed in summary and -t will match the tag in feature file and will start that one specific scenario. One tag can be used any number of times, so we can mark the whole scenario with one tag and start the execution that will run multiple tests.

For the example bellow, we can start specific test by $ behave -kt dummy_1 or run them both as $ behave -kt dummy. Of course if we have only those 2, the equivalent command is $ behave. If there are many more tests, we do not want to duplicate the tag too many times, therefore we can tag the entire feature file and start all scenarios in given feature file as $ behave -kt dummy_feature.

To use finer execution of a few tests from a larger set, you can execute the behave with a list of tags $ behave -k --tags="dummy_1,dummy_2".

Note that instead of Given, When, Then, And and But we can use asterisk * to prefix all steps in feature files. We use asterisk in most cases.

@dummy_feature
Feature: Dummy Feature

  @dummy
  @dummy_1
  Scenario: Dummy Scenario
    * Dummy Step

  @dummy
  @dummy_2
  Scenario: Dummy Scenario
    * Dummy Step

I have mentioned separating test scenarios executed to their own result pages. That is what the project behave-html-pretty-formatter is for.

The automation suite result page in form of behave-html-pretty-formatter project

The result of the behave run you saw above is given to the console in a pretty format, which is the Standard colourised pretty formatter. The behave has quite a lot of formatters to use. These formatters are built-in and can be chosen from to get the resulted data in a lot of formats ready to be used for various purposes.

Unfortunately, none of them are really useful to the extent of what we need in terms of reporting the result and debugging if something goes wrong.

New formatter can be added as a module to the behave. We provide the formatter as a Python module available from pypi behave-html-pretty-formatter

All that remains is to connect the module to the behave so that behave can use the new formatter. And this is done in the behave.ini file you saw in our project structure. Once the behave.ini file has the configuration of the behave-html-pretty-formatter, it will be seen by behave and can now be used when running the behave.

This is the behave.ini file you will see in the gnome-terminal automation example:

# -- FILE: behave.ini
# Define ALIAS for PrettyHTMLFormatter.
[behave.formatters]
html-pretty = behave_html_pretty_formatter:PrettyHTMLFormatter

# Optional configuration of PrettyHTMLFormatter
# also possible to use "behave ... -D behave.formatter.html-pretty.{setting}={value}".
[behave.userdata]
behave.formatter.html-pretty.title_string = GNOME Terminal Test Suite
# Example usecase, print {before/after}_scenarios as steps with attached data.
behave.formatter.html-pretty.pseudo_steps = false
# Structure of the result html page readable(pretty) or condensed.
behave.formatter.html-pretty.pretty_output = true
# The '%' must be escaped in ini format.
behave.formatter.html-pretty.date_format = %%d-%%m-%%Y %%H:%%M:%%S (%%s)
# Defines if the summary is expanded upon start.
behave.formatter.html-pretty.show_summary = false
# Defines if the user is interested in what steps are not executed.
behave.formatter.html-pretty.show_unexecuted_steps = true
# Define what to collapse by default, possible values:
#  "auto" - show everything except embeds (default)
#  "all" - hide everything
#  comma separated list - specify subset of "scenario,embed,table,text"
#  "none" - show everything, even embeds
behave.formatter.html-pretty.collapse = auto
# Defines if the user wants to see previous attempts when using auto retry.
# Auto retry https://github.com/behave/behave/blob/main/behave/contrib/scenario_autoretry.py
behave.formatter.html-pretty.show_retry_attempts = true
# Override global summary visibility
#  "auto" - show global summary if more than one feature executed (default)
#  "true" - show global summary
#  "false" - hide global summary
behave.formatter.html-pretty.global_summary = auto

That run will be now executed as $ behave -kt dummy -f html-pretty -o test.html. This will change the formatter from the default pretty to the html-pretty that will take the data from behave, transform them and generate self-contained HTML page to the output file test.html. Our test result files are named after the test case that was executed but omitted here for simplicity.

We made the new formatter html-pretty quite recently (January 2023) to improve the older html formatter. The new html-pretty formatter is coded in a very different way and can be more easily enriched with new features. It is also, in our opinion, much cleaner and simpler, while allowing us to have more data available to us. Output of this project allows us to have a self-contained HTML page so that the test results are always in this single file with CSS and JS. The page contains a lot of information. It prints each step and provides data if something goes wrong.

Although the formatter supports quite a lot of use cases e.g. compression of data, clickable links, images, videos and text logs, the data has to be somehow generated and injected into the formatter. The how is going to be addressed in the next section.

You can see example pages in the Examples section below.

Now, we have our APIs, a project structure for our automation, and we also have a self-contained HTML page that will contain full result of the single test, multiple tests or even entire features.

The project page can be found here behave-html-pretty-formatter

We can start the automating at this point. Although for the purpose of a general automation that will be very hard, as you would start from scratch and I imagine you would like to have much more that what is presented. Which is where qecore comes in.

Filling all the gaps and providing useful tools with the qecore project

The qecore project is a library of tools and commonly used functions that are required throughout our entire automation stack.

The project page can be found here qecore.

I have started to develop qecore only a few years back, so this project is relatively new and is being continuously developed and improved with new features and is indispensable for our day-to-day use when working with GNOME Applications.

GNOME Terminal Full Project Example

Now that I have covered everything our automation solution needs, let’s get to the setup and execution of the GNOME Terminal test suite on Fedora 38 with Wayland.

Basic machine setup required before any action

There are a few preparation steps our solution requires.

Since we have test suites that can be destructive to the system and/or change the system configuration, we do not start the automation on our local machines. We have local Virtual Machines with given distributions ready, and we connect to them via ssh. If anything goes wrong during development of the suite, we would break only the VM which can be fixed or snapshoted back and in case of an unrecoverable issue, we will still be fine since the VM’s only purpose was running the suites and setting up a new one is trivial.

This setup is generally handled by our CI, which is why when trying this you will have to do this setup once by hand.

Installing, building and execution

Now we need to:

It is important that following commands are executed with sudo if the command says so. Reason for this is consistency and not having to deal with permissions. There is also no benefit installing it in any other way. The user and automation will have full control of the machine, so we can test everything we are required to.

Errors you can encounter

The main queries you will be using

# Most common queries.
# Returns <Atspi.Accessible object ..>
<Atspi.Accessible object ..>.child(name="..", roleName="..")
# Used for finer queries and returns <Atspi.Accessible object ..>
<Atspi.Accessible object ..>.findChild(lambda x: x.name == "" and x.size == (0, 5) and x.showing)
# Used for finer queries that returns list of results [<Atspi.Accessible object ..>, ..]
<Atspi.Accessible object ..>.findChildren(lambda x: x.roleName == "" and x.position[0] >= 0 and x.focused)

# All nodes have the base methods defined. Most notably:
In : app.child("File").click()
Clicking on [menu | File]
Mouse button 1 click at (46.0,83.5)
# Note the coordinates of the click. The dogtail will calculate a center of the node from its position and size.

# Some nodes have actions defined.
In : app.child("File").actions
Out: {'click': <dogtail.tree.Action at 0x7f97d44b0210>}
# Which you can use instead of mouse or keyboard events in some cases.
In : app.child("File").doActionNamed("click")
click on [menu | File]
Out: True

# When working with text fields the appropriate Atspi nodes have attribute .text
<Atspi.Accessible object ..>.text = "Write a text to the text field."
# Beware that sometimes you will need to click to the text field or the .text attribute will not update.

# When working with scroll panes you can use .value attribute to scroll in the page.
scroll_pane = <Atspi.Accessible object ..>.child(name="..", roleName="scroll pane")
# Look at the value.
scroll_pane.value
# Look at min and max value.
scroll_pane.minValue
scroll_pane.maxValue

# Scroll the window - use value between min and max.
scroll_pane.value = <int/float>
# To simply scroll to the bottom you can use.
scroll_pane.value = scroll_pane.maxValue

# Beware that this is very dependant on the Accessibility in the application.
# It is not uncommon to have the minValue and maxValue to be both 0.0,
# in which case you will not be able to scroll this way.

With these you can do quite a lot.

Let’s say we do not care about a single button, we want to see the entire tree. Again we have multiple options.

We can use the <Atspi.Accessible object ..>.dump() method on nodes. This will return very simple tree representation from the node we are in.

Or you can use the <Atspi.Accessible object ..>.tree() method, which returns a proper tree like representation of the application. Let’s see how it looks for gnome-terminal. Note that the format of the Atspi nodes is as follows [<name>-<roleName>-<description>]. Format could be better to prevent confusion in the dash '-' character but for now it serves its purpose well.

A little disclaimer. Use with care on large trees like gnome-shell, it will take a very long time, this feature was not written with efficiency in mind. For large trees the dump() method is faster.

In : app.tree()
[gnome-terminal-server-application-]
  └── [test@localhost-live:~-frame-]
      ├── [-panel-]
      │    ├── [-filler-]
      │    │    ├── [-separator-]
      │    │    └── [Close-push button-]
      │    ├── [-push button-]
      │    ├── [Menu-toggle button-]
      │    ├── [-filler-]
      │    │    ├── [test@localhost-live:~-label-]
      │    │    └── [-label-]
      │    └── [-filler-]
      │         ├── [-push button-]
      │         └── [Menu-toggle button-]
      ├── [-filler-]
      │    ├── [-menu bar-]
      │    │    ├── [File-menu-]
      │    │    │    ├── [New Tab-menu item-]
      │    │    │    ├── [New Window-menu item-]
      │    │    │    ├── [-separator-]
      │    │    │    ├── [Close Tab-menu item-]
      │    │    │    └── [Close Window-menu item-]
      │    │    ├── [Edit-menu-]
      │    │    │    ├── [Copy-menu item-]
      │    │    │    ├── [Copy as HTML-menu item-]
      │    │    │    ├── [Paste-menu item-]
      │    │    │    ├── [-separator-]
      │    │    │    ├── [Select All-menu item-]
      │    │    │    ├── [-separator-]
      │    │    │    └── [Preferences-menu item-]
      │    │    ├── [View-menu-]
      │    │    │    ├── [Show Menubar-check menu item-]
      │    │    │    ├── [Full Screen-check menu item-]
      │    │    │    ├── [-separator-]
      │    │    │    ├── [Zoom In-menu item-]
      │    │    │    ├── [Normal Size-menu item-]
      │    │    │    └── [Zoom Out-menu item-]
      │    │    ├── [Search-menu-]
      │    │    │    ├── [Find…-menu item-]
      │    │    │    ├── [Find Next-menu item-]
      │    │    │    ├── [Find Previous-menu item-]
      │    │    │    └── [Clear Highlight-menu item-]
      │    │    ├── [Terminal-menu-]
      │    │    │    ├── [Set Title…-menu item-]
      │    │    │    ├── [-separator-]
      │    │    │    ├── [Read-Only-check menu item-]
      │    │    │    ├── [-separator-]
      │    │    │    ├── [Reset-menu item-]
      │    │    │    ├── [Reset and Clear-menu item-]
      │    │    │    ├── [-separator-]
      │    │    │    ├── [1. 80×24-menu item-]
      │    │    │    ├── [2. 80×43-menu item-]
      │    │    │    ├── [3. 132×24-menu item-]
      │    │    │    └── [4. 132×43-menu item-]
      │    │    └── [Help-menu-]
      │    │         ├── [Contents-menu item-]
      │    │         └── [About-menu item-]
      │    └── [-page tab list-]
      │         └── [-page tab-]
      │              └── [-panel-]
      │                   └── [-filler-]
      │                        ├── [Terminal-terminal-test@localhost-live:~]
      │                        └── [-scroll bar-]
      └── [-panel-]
            └── [-panel-]
                └── [-filler-]
                      └── [-filler-]
                          ├── [-filler-]
                          │    └── [-filler-]
                          │         ├── [Zoom Out-push button-]
                          │         ├── [100%-push button-]
                          │         └── [Zoom In-push button-]
                          ├── [-filler-]
                          │    └── [-filler-]
                          │         ├── [New Window-push button-]
                          │         └── [Full Screen-push button-]
                          ├── [-filler-]
                          │    ├── [-separator-]
                          │    └── [-filler-]
                          │         ├── [Read-Only-check box-]
                          │         ├── [Set Title…-push button-]
                          │         ├── [-filler-]
                          │         │    └── [-filler-]
                          │         └── [Advanced-push button-]
                          └── [-filler-]
                                ├── [-separator-]
                                └── [-filler-]
                                    ├── [Preferences-push button-]
                                    ├── [Help-push button-]
                                    └── [About-push button-]

We also have a Sniff or AT-SPI Browser that you can use to search for nodes. That is an application that comes with dogtail installation.

Lets exit the ipython3 since there is a lock in effect and Sniff will not/should not start when you are working with it in command line.

Once you open the Sniff you can see the applications opened and you can browse their content. For better visual presentation click on Actions -> Highlight Items. With Xorg this shows you the red squares correctly around the selected nodes, with Wayland this will be shown incorrectly. That is one of the reason why going through the tree via interactive shell is better. Once you get used to it, searching for your desired node is a matter of seconds.

Explaining the rest of the gnome-terminal project

Examples

Comparison of OpenQA vs Accessibility

While OpenQA is a good product for what it does, it is in my opinion not usable for GNOME Desktop Automation and not in the sense that it cannot be used, it can very well be used with some degree of difficulty. I have seen it being used testing anaconda and that is a perfect use case. It would be my choice as well for parts of the system where Atspi cannot be used.

As I mentioned, we are using the image matching in some capacity as well. From my experience the image matching is very fragile, the tests are difficult to maintain, and it is quite unscalable for our purposes. The few tests that I have written were a pain, and I am the one who introduced this feature to our team, so I can imagine others were even more frustrated with it. We have it in a good enough state today but on any UI change we have to redo the needles.

There were instances where I could not differentiate between new and old needle, to my eye it looked exactly the same, but the new needle started passing the test. It is quite a long process to write even a single test.

On the other hand with Atspi, if we see a failed test, most of the time it is a bug or the UI label or placement has changed. On bugs the issue is clear and there is nothing to be changed on the automation side. With UI change we check Screenshot or Video to see what happened, and usually we can fix the test very quickly by changing order of clicks or rewriting a string. Once the suite is written, it can be easily expanded with new test cases and maintained through different version releases. There are some issues we encounter when writing the tests that cause instability and that has to be accounted for but once the suite is tried and tested it is very stable.

For me personally these tools are way too different to be compared, but there is no question about difficulty of using opencv and comparing two images and working with Python objects. Working with objects will always be more stable.

There are a few other advantages that need to be mentioned.

Usage with GTK4

While what I have described here will work for GTK4 Applications, you will soon find that there are extra steps required.

There is a question of shadows. GTK4 applications have large shadows and that shadow left upper corner is the base (0, 0) of the application. So when you attempt to click, the click will be very much in the wrong place.

To fix this you can use the update_coords() method from dogtail.rawinput to offset (x, y) of the final coordinate.

I still hope I will be able to remove this method in the future and figure out the size of the shadows dynamically based on the application, so that the user is not forced to make manually offset for all clicks, although my recent experiments and research is proving me wrong.

This offset, while working with GTK4, will now cause the GTK3 applications actions to be in the wrong place and there is no reason for a suite to not work with both.

We are currently trying to figure out, with some level of success, how to make any differences between GTK3 and GTK4 irrelevant from the user point of view. So hopefully in the future there will be no need to have any offsets.

Usage with Fedora 39 and newer.

Please beware that this article is for Fedora 38. While it will work on 39 too, there are some design changes that the qecore is not yet adapted for. Like the Activities label missing which qecore currently uses for closing gnome-shell overview.

This suite will work on Fedora 39 and newer as is, but you will see some tests failing.

Update 2024 July 16: Automation will now work with no issues on newer Fedoras with qecore==3.26.1.

Reason for this article

There is another reason for this article apart from showcasing our solution.

While I want to present our solution and I can say that what we have is good, it is not perfect by a long shot and requires a lot of knowledge to get the automation running as I have demonstrated in this article, it can also quite quickly stop working.

All this work that went into making our tool sets is very dependent on Accessibility working, everything depends on and is built around Atspi. If Accessibility went away or would be broken, we would not be able to do as much as we can today. Currently, we can automate most of the GNOME Applications including gnome-shell since from the point of view of Accessibility, the gnome-shell is just another application.

This article aims to show how easy it is to get the automation of GNOME Applications off the ground. We hope, by providing the full gnome-terminal component automation example for anyone, a lot of people will try it out. We would love if people found it useful and would attempt to contribute to upstream projects with automation tests.

The most desired outcome would be to have more eyes on Accessibility. To provide justification and motivation for development of Accessibility and its ability to be used for automation.

Future Plans/Aspirations -> GNOMEAutomation

To be honest, there is no need to do any of the following things that I am listing. Currently, it works good enough as is. But it can be better, it can work better, there is always a way to improve what we have.

Qecore

The qecore project is currently serving as a library of tools that also serves as a glue layer across different parts of our solution. Qecore does a lot of stuff on its own but is still designed around the main projects:

Aspirations is to have a single project that covers everything that quality engineers need -> GNOMEAutomation.

Accessibility

If accessibility goes away we are going to be “attached to another object by an inclined plane wrapped helically around an axis”.

There are some issues that come from Atspi already, we are able to identify them on our end and can notify developers, but there is an issue. If developer wants to see it reproduced they would need quite a lot of setup and even if the setup is running they see dogtail/qecore errors, we currently have no easy way how to provide reproducers in Atspi that would allow them to see the issue and not waste their time with our environment. The constraint here is time.

We have a few issues that we could report and help developers identify and also help to fix it. But the problem is that developers have already a lot of stuff on their hands, our issues are not a priority, understandably. We also do not really have time to make difficult reproducers and debugging issues that we can bypass. We have quite a lot of responsibilities and there is only a finite time. The time we would spend on making reproducers is quite higher that the time we spend on making a workaround. Workarounds are doable in matter of minutes, and we bypass the issue altogether.

So unless there is a blocker, we usually opt to not report issues and work around them.

Aspirations: If we introduce automation via accessibility to more people with a coherent project like GNOMEAutomation, that is not collection of projects spliced together to just work, we might have a wider audience and user base. That might help to attract talent to our teams that would help improve Accessibility and keep it in good state.

With a project like this we also might have a wider base for Fedora testing days. Some things already work on Fedora and are mostly usable. The issue is a long setup as you can see, that this project would hopefully solve. I would imagine that once a test day comes, we could have GitLab/GitHub repositories with our testing suites that I and others would contribute to, so that anyone can just come, download the project and run the tests. This would be quite rich source of data that user would not have to spend a lot of time on. Simply boot VM, run the suits, report results. There is of course a need for real HW testing as well, but that is not the issue we are trying to solve here. Currently, I try to participate in test days, but I am not able to fit it to my schedule every time, which is a shame.

The behave

Its file structure is the template of our project and all other things are going before or after the behave command line execution. Behave has limitations that we have to hack around sometimes to get our desired outcome. Those are rare, although we have a recent example.

We need to generate a log no matter what part fails, so that the end user can evaluate what went wrong. The problem starts when the very first function is called, before_all. Something can still go wrong, and we need to attach the data to the report. The problem is that behave, even if it was called as behave -f html-pretty, does not know about the formatter in the before_all function. So when we are dealing with an error in setup, we have no place to attach it to. We can bypass it by saving the error and evaluate any errors in the very next function before_scenario, end the run, and attach it to the report since behave now knows it has a formatter defined. This is solvable but quite inconvenient.

There are rare issues where the behave fails, and no logs are generated while we would love at least partial results. But since behave is the one who generates the output, if behave fails we have nothing to parse for our html-pretty output. Sometimes behave also captures errors, and we have to go for an adventure to see where the error was coming from since the error we get said nothing about the real issue. There are no blockers currently, just inconveniences that we would love not to deal with.

Aspiration is to not have behave dictate our structure, possibilities and output but having project that enables our wants/needs/requirements. But again, behave works perfectly fine in the majority of our cases - no reason to reimplement something that exists with small changes.

There is currently no proper reason to remove behave from our solution.

The dogtail API

This project serves as our API, it has its set of problems while it still works most the time.

Currently, there is no incentive to focus on reimplementation as what we now have works fine and most of our suites are written and are being adapted between package and system versions. There are situations where new feature or bug appears, and we include it in our suite, but that is using previously used functions, so there is nothing new from the testing point of view.

I am going to include personal experience when I first started working at Red Hat. I was assigned a responsibility of a project gnome-contacts. The automation was already written, and I was required to understand the code, learn how to use it, modify it and improve it. At first, I was just copying around what was already there, and I had no issues making it work. Until I encountered a strange artifact, function not working. It was the same as any other function used in the code. I had no idea why it was not working. I found dogtail source code and went through it and I saw nothing wrong. No one was able to help me as they did not see anything wrong on the dogtail side either. So I marked the test as broken or simply worked around it, I do not remember. It was quite a while before I realized the dogtail is not a full API but a wrapper and extension of others. So I started looking for the other libraries imported in dogtail, in pyatspi2 and finally Atspi. I found a documentation that I do not believe was up-to-date, but it was usable. I found the C source code. Furthermore, I went through it to verify missing documentation parts. I still do not know the exact source code and documentation, I might have been looking at the wrong place altogether. Most of the stuff I tried that I know are or are not working, is a result of experimentation. As I mentioned there is not much time, so trial and error was the chosen solution. I started making reproducer in Atspi, only to find I am able to reproduce the issue I had in the past quite a lot and always thought the dogtail was not working.

The point is that the issue that was in the code for years was not identified because no one knew where to look. I had no one to teach me where to look or to say that the problem might be somewhere else, because it was poorly documented, and I did not find any tutorials how to debug such issues.

Aspirations is to improve dogtail, but not in that project - an alternative to dogtail. Dogtail is implementing its own functions, it wraps over pyatpsi2 and Atspi, while pyatspi2 also wraps over Atspi, so debugging is a problem.

To make sure problems are in the Atspi library, I had no other option than to make lightweight clone of dogtail and not wrapping over anything other than Atspi. I did the bare minimum to make the API work and to verify some specific issue was having a source in Atspi. It was a success. I saw the same issues without any use of dogtail and pyatspi2.

The GNOMEAutomation would provide API just like dogtail, but wrapping only over Atspi so any issue that is found will have a single source. The best case scenario would be pointing developers to the API and have them see that we are using the Atspi functions, so it cannot be an issue we introduced. They will also get a reproducer directly to the Atspi and will not have to deal with our environment. In the worst case, installing GNOMEAutomation with a given project it was reproduced in, which would be start-able right away without any difficult setup. Documentation will be present from the start (I have a habit to have docstring and proper documentations everywhere) so no one will need to go on long searches to figure out what is wrong and where.

Update 2024 July 16: We have agreed to make required changes in dogtail project and rewrite is in progress https://gitlab.com/dogtail/dogtail/-/issues/29.

The gnome-ponytail-daemon

I would imagine there is a way how to include it in this project, so any issue can be tracked accordingly. This project was originally created to enable us to continue working on Wayland. It’s hard to imagine anyone but us using gnome-ponytail-daemon, but I could be proven wrong.

The behave-html-pretty-formatter

The formatter was required to be written in a way to fit in our current solution. It had to be done as a standalone project. Which on its own is just bare bones so that it also can be used by upstream (we were pleasantly surprised how many people are using it already). We are still generating the data with qecore.

Projects like GNOMEAutomation would include an output generator which behave-html-pretty-formatter would fulfil beautifully or with some minor changes. I always imagined that we could have a self-contained suite output wrapped over the entire stack, which is currently not feasible. That said, having it over the behave part is good enough.

An Aspiration is to have logs and reports over the entire automation stack.

Finally

Hopefully information contained here were useful to you.

We would love to know, how many people will actually attempt to execute GNOME Terminal test suite.

If you follow the setup here and succeed, please consider Staring the GNOMETerminalAutomation project on GitHub. It will help us in two ways. One is to see how many people got this far and two is making this project more visible.

If you would fail for any reason, please open an Issue on GitHub so that I can fix something I might have missed or to help you fix something I did not think about.

Thank you for reading.

Keywords

Accessibility, AT-SPI, a11y, GNOME, Fedora, Wayland, automation, suite, test, gnome-terminal, Atspi, pyatspi2, dogtail, gnome-ponytail-daemon, behave, qecore, behave-html-pretty-formatter

Sources