GIMP has a lot of intertwined menus, dialogs and tools. When we change a part of the interface, another may stop working and sometimes, the problem can be missed until a user reports it.
To find these problems early, a new GUI testing platform was implemented. A typical GUI test starts GIMP, clicks menus and buttons, and then checks the result. These tests run automatically each time we merge contributor changes.
Why we need them
GIMP already has unit tests. A unit test checks one small part of the code, for example how GIMP reads an XCF file. But our unit tests were not opening a real window. So it coult not tell us if a dialog opens, or if a menu item did its job.
Note that GIMP already had tests of GUI code, usually interacting directly with the core code, and sometimes even simulating input devices’ events (pointer movements, clicks, drags…). But nothing can beat interacting with the real interface. Furthermore such legacy tests directly hijacking core code come with their own problems, such as breaking all too often (and not because they detected a bug), all while not being able to detect whole categories of bugs.
UI tests check the program as a whole, in the same way as you use it.
What the tests do
Each test starts a real GIMP and does one small task. Then it reads the result from GIMP. If the result is not correct, the test fails and tells us what went wrong.
We have 13 tests at this time. They check that:
- GIMP starts and shows its main window.
- The Welcome dialog opens on the correct tab. It also stays hidden when you turn off “Show on Start”.
- File > New makes an image with the size you set.
- File > Export As writes a correct PNG file.
- Tools > Swap Colors swaps the foreground and background colors.
- You can type a new foreground color in the Colors dialog.
- You can add a layer and delete a layer.
- Undo and Redo work.
- Zoom In changes the zoom level.
- A click on a tool button in the toolbox selects that tool. The test then uses the Bucket Fill tool on the canvas.
- A change in Preferences is saved when you click OK.
- A menu action, Image > Flatten Image, does its job.
How they work
The tests do not need a screen. Each test starts a small, hidden desktop, runs GIMP in it, and removes it at the end. Because of this, the tests can run on our CI servers, where no screen is available.
To find a button or a menu, the tests do not look at the pixels on the screen. They use the accessibility information that GIMP gives to tools such as screen readers. This has one more good result: when we make GIMP easier to test, we also make it easier to use with a screen reader, hence improve the software accessibility.
The tests use dogtail, a Python tool for UI tests. To run GIMP on a server with no screen, dogtail needed a new feature. We contributed this feature upstream, to dogtail itself, so other projects can use it too.
At this time, the tests run only on Linux (both X11 and Wayland) since
dogtail does not support Windows or macOs.
Adding even these few simple tests considerably increases confidence that the GIMP build has worked, and also shows the test infrastructure is working and useful.
Credits
- Harsh Verma wrote the UI tests and the new dogtail feature.
- Jehan reviewed the UI tests.
- Michal Odehnal reviewed the new dogtail feature.
You can read the full change in merge request !2909.
How you can help
13 tests is a good start, but GIMP has many more dialogs and tools. If you want to help, you can write a test for the part of GIMP that you use the most.
We wrote a guide that shows how to write a new test and how to find the cause when a test fails: UI tests on developer.gimp.org.
We hope this work will help us find bugs earlier and keep GIMP working well for everyone.