Screen capturing in Wayland on Ubuntu 21.04

Ubuntu 21.04 has changed the graphics system from the Xorg server to the Wayland server. At first glance, this seems to work well – I had not even recognized this until I wanted to do a screen recording using SimpleScreenRecorder:

When I tried to create a screenshot of the above dialog using Shutter, the whole desktop was filled with this funny pattern:

So, it seems that especially screenshot and sreen recording applications still have their issues with the Wayland server. Before going back to Xorg as suggested by SimpleScreenSaver, I tried some alternative applications which reportedly already work with Wayland:

Kazam

Kazam is available in the universe repository. Hence installation with apt install kazam is straightforward. It looks less feature rich as SimpleScreenSaver though:

Unfortunately the application crashed several times while creating a screenshot from an area. Other options like “Fullscreen” did not work either, so I was not able to test it further.

Kooha

Kooha is relatively new, and there is no package available in the Ubuntu repository yet. So it needs to be built from source. meson and ninja are required for the build, and also a couple of build dependencies need to be satisfied, in particular gettext, libglib2.0-dev and appstream-util. Then, the application can be built using

$ git clone https://github.com/SeaDve/Kooha.git
$ cd Kooha
$ meson builddir --prefix=/usr/local
$ ninja -C builddir install

Unfortunately, running the application then terminates with an error which indicates that it requires Gtk 4.0 (Ubuntu is still on 3.38):

Traceback (most recent call last):
  File "/usr/local/bin/kooha", line 44, in <module>
    from kooha import main
  File "/usr/local/share/kooha/kooha/main.py", line 22, in <module>
    gi.require_version('Gtk', '4.0')
  File "/usr/lib/python3/dist-packages/gi/__init__.py", line 129, in require_version
    raise ValueError('Namespace %s not available for version %s' %
ValueError: Namespace Gtk not available for version 4.0

I decided not to track this down further.

OBS Studio

OBS Studio is a very feature rich application aimed especially towards screen recording and streaming. It is available in the universe repository, however this version (26.12) only records a black screen when using the Wayland server. Version 27.0 is available as Snap, and this seems to work well. After installing it with snap install obs-studio, I was able to create a screen recording and play it back with VLC.

However, the application crashes with a segmentation fault when starting it a second time. Removing the configuration files in the user’s snap directory solves this, but then requires to go through the complete setup process again when restarting the application.

Conclusion

When doing a lot of screen capturing and/or screen recording, it is probably best to stay on Xorg for the time being – this can simply be configured with a button in the lower right corner on the login screen, after selecting the login user:

This selection is persisted across logout/login.
With the Xorg server, both Shutter and SimpleScreenRecorder are working again. Also, OBS Studio does not segfault when relaunching.

Run time data structure of a Java Virtual Machine

The following diagram summarizes the internal structure of a Java Virtual Machine’s memory structure as specified in chapter “2.5. Run-Time Data Areas” in The Java® Virtual Machine Specification, Java SE 8 Edition. In this example, three classes A, B, and C have been loaded into the JVM, and for each of those classes, a varying number of instances have been created. Two threads have also been created.

The Frames on each Thread’s stack are associated with a particular method. Each time a method is invoked, a new Frame is created on the Stack. When the method returns, the frame is discarded from the stack. The Frame which is associated to the current method is the Current Frame. Each Frame has an Operand Stack which is used by the bytecode instructions to perform operations. Local variables are referenced by their index, starting at 0. The Runtime Constant Pool reference in each Frame is a reference to the Runtime Constant Pool of the class which contains the method which is currently executing.

Separating color channels in JavaFX

Suppose you have an Image and want to extract the color channels for red, green and blue into three new images. A straight forward way would be to create a new empty image with the same size as the source image, iterate the source image pixel by pixel, convert the pixel color of each pixel and set the converted color for each pixel in the destination image. In JavaFX, this is possible through the PixelReader and PixelWriter interfaces:

public Image getBlueChannel(Image sourceImage) {
    PixelReader pr = sourceImage.getPixelReader();
    int width = (int) sourceImage.getWidth();
    int height = (int) sourceImage.getHeight();

    WritableImage result = new WritableImage(width, height);
    PixelWriter pw = result.getPixelWriter();
    for (int x = 0;  x < width;  x++) {
        for (int y = 0; y < height;  y++) {
            Color col = pr.getColor(x, y);
            pw.setColor(x, y, new Color(0, 0, col.getBlue(), 1.0));    
        }
    }

    return result;
}

This example creates a new WritableImage with the same size as the source image, and then iterates the source image pixel by pixel and sets the corresponding pixel in the writable image, using the blue component of the source pixel only.

While this works well (and shows how to access individual pixels of an Image, which can be useful in various situations) it is definitely not a good approach, especially due to performance reasons. It might work well enough for a smaller image (like 200×100 pixels) but for larger images it will most likely get too slow.

A much better way is to use the JavaFX Blend effect which takes two input sources and combines them into an output. The two input sources are called topSource and bottomSource, and they can be combined by a couple of operations which define how each individual pixel of the two input sources are combined to create the result pixel. When the effect is applied to a node in a scene graph, the node itself is taken as top source and/or bottom source if either of them has not been set explicitly. Thus, if the top source is explicitly set for the Blend effect and then the effect is applied to a node, the result is the topSource combined with the node itself. The following example creates a ColorInput where all pixels are set to blue (rgb(0, 0, 255)) as the top source, and an ImageView node is used as the bottom source. The two sources are combined by multiplying the color values of each pixel to generate the resulting pixel:

public Image getBlueChannel(Image sourceImage) {
    ColorInput colorInput = new ColorInput();
    colorInput.setPaint(Color.Blue);
    colorInput.setX(0);
    colorInput.setY(0);
    colorInput.setWidth(sourceImage.getWidth());
    colorInput.setHeight(sourceImage.getHeight());

    Blend blend = new Blend();
    blend.setMode(BlendMode.MULTIPLY);
    blend.setTopInput(mask);

    Node img = new ImageView(sourceImage);
    img.setEffect(blend);

    SnapshotParameters params = new SnapshotParameters();
    Image result = img.snapshot(params, null);

    return result;
}

Why does the example use the MULTIPLY operation for each pixel? In the earlier days, a 24 bit RGB pixel was made up of three eight-bit-values, one for each color channel:

   Red         Green        Blue
xxxx xxxx    xxxx xxxx    xxxx xxxx

So, to get one of the color components only (say, blue) it sufficient to create a bit mask where only the bits from the blue channel were set to 1. By AND-combining the color with the mask, the blue channel was then extracted:

       Red         Green        Blue
    xxxx xxxx    xxxx xxxx    xxxx xxxx
AND 0000 0000    0000 0000    1111 1111
----------------------------------------
  = 0000 0000    0000 0000    xxxx xxxx

In JavaFX, the color channel values are floting point values between 0.0 and 1.0. In order to keep the values for one color channel and set the values for all other color channels to 0.0, we can multiply the color with (0.0, 0.0, 1.0):

    Red   Green  Blue
    n.n    n.n    n.n
X   0.0    0.0    1.0
----------------------
=   0.0    0.0    n.n

https://github.com/afester/JavaFXsamples/tree/master/src/afester/javafx/examples/colorchannels contains a complete example in the ColorChannelsExample.java file, which extracts the red, green and blue color channels from an image:

The lower blue image uses the initial example where each individual pixel is processed through a PixelReader and PixelWriter.

Alternative approach

If we only want to display the color channel of the image, and if there is no need to further process the images which contain a specific color channel, we do not need to generate real Image instances for each color. We can simply use different ImageView nodes which all use the same source image, and apply the Blend effect for red, green and blue to the ImageView nodes:

    ...
    // Create Blend effect for red channel
    ColorInput colorInput = new ColorInput();
    colorInput.setPaint(Color.RED);
    colorInput.setX(0);
    colorInput.setY(0);
    colorInput.setWidth(sampleImage.getWidth());
    colorInput.setHeight(sampleImage.getHeight());
    Blend redBlend = new Blend();
    blend.setMode(BlendMode.MULTIPLY);
    blend.setTopInput(mask);

    // Create Blend effect for green channel
    Blend greenBlend = ...

    // Create Blend effect for blue channel
    Blend blueBlend = ...

    ImageView sourceView = new ImageView(sampleImage);

    ImageView redView =   new ImageView(sampleImage);
    redView.setEffect(redBlend);

    ImageView greenView =   new ImageView(sampleImage);
    greenView.setEffect(greenBlend);

    ImageView blueView =   new ImageView(sampleImage);
    blueView.setEffect(blueBlend);
    ...

The result is the same as above, but we did not have to create all the Image instances:

The complete example is available at https://github.com/afester/JavaFXsamples/tree/master/src/afester/javafx/examples/colorchannels in the ColorChannelsExample2.java file.

9-patch scaling in JavaFX

While playing around with Borders in JavaFX, I came across the term “9-patch scaling technique”, and I wondered what that actually means.#

First of all, “9-patch” means that a rectangular area is divided into 9 areas (patches) of possibly different sizes as shown in the following image:

Each area has either a fixed size (the corner areas 1, 3, 7, 9), or it can be resized vertically (4,6) or horizontally only (2,8), or it can be resized both vertially and horizontally (the central area, 5).

When using “9-patch scaling”, each area gets a particular fill color, pattern or even image assigned. If a pattern or image is used, it is either stretched or tiled (repeated) in those areas which are resizable, so that the pattern or image fills the whole area.

Normally, the central area contains some particular content, and the outer areas are used to define a border around that content. In JavaFX, the rectangular area itself is a Region or one of its sub classes, and its Border can be set with the Region.setBorder(Border value) method.

Border is essentially a list of strokes and images which are drawn one above the other, in the order they are stored in the list. There are some special conditions to consider especially when using both Strokes and Images with the same Border. The remaining article will focus on using images, hence I will not go into the details of these constraints. See https://docs.oracle.com/javase/8/javafx/api/javafx/scene/layout/Border.html for more information.

Image based borders can be created either in code through the BorderImage class, or they can be defined in CSS. The following diagram shows the BorderImage parameters, their corresponding style sheet property and how the given values are used to specify the border:

The image itself is defined through the Image parameter or through the -fx-border-image-source CSS property. The slices parameter or the -fx-border-image-slice property defines the slices on the source image. The filled parameter specified whether the central slice should also be used when rendering the border – in CSS, this is part of the -fx-border-image-slice property by adding a value of fill.

The remaining parameters define how the border is constructed – the widths parameter or the -fx-border-image-width CSS property (shown in red above) defines the width for each of the four border edges. Note that its also possible to specify one value only (the BorderWidths constructor has a single argument variant) – in that case the value applies to all four edges. The insets parameter or the -fx-border-image-insets property (shown in blue above) defines the distance from the Region’s bounding box to the border. Finally, the repeatX/repeatY parameters or the -fx-border-image-repeat property defines how the slices in the border are filled if the corresponding image slice is too small – essentially, the image can be repeated (tiled) or scaled (stretched).

As an example, lets assume that we have a simple Button which we would like to surround with a border:

The button is created in Java code like this:

Region r = new Button("Hello World");
r.setId("hello");

To add the border, we need to create an image (lets call it border.png) similar to the following one:

How exactly the image looks like and how large it is is of secondary interest – we could even discard the central white rectangle, since we define the area to use from that image through the slice property. It is just important that the image can be divided into nine areas when applying the sizes given in the slice parameter:

#hello {
  -fx-border-image-source: url(border.png);
  -fx-border-image-slice:  6px;
  -fx-border-image-insets: 0px;
  -fx-border-image-repeat: repeat;
  -fx-border-image-width:  6px;
}

Some more advanced examples are shown at https://www.w3.org/TR/css3-background/#border-images. As the next step, we might want to try to add an additional image like a status indicator to the border. Lets assume that we have the following image trafficlight.png which we would like to add to the upper right corner:

In theory, this is possible since we can define a list of images and/or strokes when specifying the border. In CSS, this is done by specifying the parameters for each image separated by commas:

#hello {
  -fx-border-image-source: url(border4.png), url(trafficlight.png);
  -fx-border-image-slice:  6px             , 16px 46px 0px 0px;
  -fx-border-image-insets: 0px             , 0px;
  -fx-border-image-repeat: repeat          , no-repeat;
  -fx-border-image-width:  6px             , 16px 46px 0px 0px;
}

In order to see the whole image unscaled, we set the top and right slice to the image’s size, and we also specify the corresponding border widths with the same size. This results in the following image:

As you see, the right border now has the width of 46 pixels as specified – depending on the use case, this might work well but probably the expectation was to achieve something like this:

We could try to put the traffic light into the top slice instead of the top right slice, but as said above the slices are always fillled, either by stretching or by repeating the corresponding image slice (the CSS value no-repeat is actually “stretch”):

#hello {
 -fx-border-image-source: url(border4.png), url(trafficlight.png);
 -fx-border-image-slice:  6px             , 16px 0px 0px 0px;
 -fx-border-image-insets: 0px             , 0px;
 -fx-border-image-repeat: repeat          , no-repeat;
 -fx-border-image-width:  6px             , 16px 0px 0px 0px;
}

Again, depending on the use case, this might be usable, and if it is assured that the central area always has the same width (which often is not the case) one could use a larger source image which fills the whole border in order to avoid scaling/stretching. However, a better solution in this case is to use a layout component like BorderPane and not use borders for such use cases.

A complete sample application which demonstrates the results shown above is available at https://github.com/afester/JavaFXsamples/tree/master/src/afester/javafx/examples/border. Simply clone the https://github.com/afester/JavaFXsamples repository, import the eclipse project into Eclipse and launch the afester.javafx.examples.border.BorderExample class.