ELAINE – Evolving Image Enhancement Pipelines

You can interact with ELAINE at the following link:

http://elaine.dei.uc.pt/

Image enhancement is an image processing procedure in which the original information of the image is improved. It alters an image in several different ways, for instance, by highlighting a specific feature to ease post-processing analyses by a human or machine. In this work, we show our approach to image enhancement for digital real-estate-marketing. The aesthetic quality of the images for real-estate marketing is critical since it is the only input clients have once browsing for options. Thus, improving and ensuring the aesthetic quality of the images is crucial for marketing success. The problem is that each set of images, even for the same real-estate item, is often taken under diverse conditions making it hard to find one solution that fits all. State of the art image enhancement pipelines applies a set of filters that solve specific issues, so it is still hard to generalise to all types of issues encountered. With this in mind, we have created a Genetic Programming (GP) engine — ELAINE (EvoLutionAry Image eNhancEment) — for the evolution of image enhancement pipelines based on pre-defined image filters. The presented work is a step towards an Automatic Image Enhancement framework.

ELAINE – Evolving Image Enhancement Pipelines

EvoLutionAry Image eNhancEment (ELAINE) is an evolutionary framework based on Genetic Programming (GP) that evolves image filters’ pipelines for image enhancement. It is a conventional GP approach where the function set comprises image filters, conditional operators, ephemeral constants and the image as terminal.

The filters are implementations of image operations that focus on five main aspects of image enhancement approaches contrast adjustment, brightness adjustment, colour balance, noise removal and edge enhancement (also referred to as sharpening). They are based on image operations on the image from the literature, such as Contrast Stretching (CS); histogram equalisation (HE); Contrast Limited Adaptive Histogram Equalization (CLAHE); Gamma Correction (CB); Non-local Means Denoising (NLMD); Unsharp Masking (UM) and; Simplest Color Balance (SCB).

**Figure 1**
Example of an ELAINE’s Individual.

For conditionals, we introduced an if-then-else function that, depending on the boolean value of a condition, allowing the same program solution to behave differently according to the input characteristics. Since one of the primitives is the whole image, we required values that could be used for comparison to make conditionals. To make this possible, conditional functions were introduced, a group of five primitive functions that extract relevant features of the image. The features are image-related features that capture characteristics of the perceived quality of the image: gaussian noise estimation, contrast estimation, average saturation based on average the value of the HSV’s saturation channel, brightness from pixel intensity in the HSB colourspace, and sharpness by calculating variance after applying a Laplacian filter. The functions were modified to expect an image as input and an ephemeral constant, with the latter serving as a threshold for the conditional functions.

Results on Real Estate Images

The approach is instantiated in real estate marketing to improve images from different types of real estate under diverse conditions, which requires a modular approach. We deployed EvoImaghen with 7 filters from the literature related to image enhancement and Image Quality Assessment (IQA) tools to assess the results.

We made use of 4 distinct no-reference IQA tools, where no-reference means that the evaluation does not depend on a target or reference image to evaluate the quality of an input image. We resorted to the following tools: PhotoILike (PHIL) is an IQA service with closed source, third-party, black-box software that receives an image and returns a value from 1 to 10, where 1 means the worst quality and 10 the best quality, based on the images aesthetic but also on multiple features considered relevant for real-estate marketing; Blind Image Spatial Quality Evaluator (BRISQUE) used in image enhancement contexts, based on a collection of 36 features per image and; Neural Image Assessment (NIMA) a tool based on a deep Convolutional Neural Network, trained with different datasets for predicting both technical and aesthetic scores of images, thus yielding two different IQA metrics.

**Figure 2**
Average and standard deviation values of the Original images from the test dataset using 4 no-reference metrics. Manual and Evolutionary values are the differences from the values obtained by the Original set of images to the ones obtained by each setup. All the metrics values range from 1 to 10 where 1 means the lowest quality and 10 highest quality. For the differences, a positive number represents an improvement and conversely, a negative number represents a degradation compared with the values of the unaltered original images.

We evaluated the original images using the IQA metrics for the experiments and applied a Manually defined pipeline (Manual) based on filters of the literature and by applying the best individual found using ELAINE. In the context of real estate imaging, we asserted that the image enhancement should aim towards aesthetically pleasing images. Thus, we explore that characteristic by resorting to the aesthetic classifier NIMA to evaluate the individuals. After evolving pipeline solutions, we tested the best solution in a subset of test images compared with the initial results and manual pipeline as the baseline. We show that the system can create filter application pipelines that improve the image quality in all the metrics chosen for image quality assessment.

The technical IQA tool metric was the only metric that suffered a negative effect. We argue that the images tend to be altered much to the point that some parts can become more stylised than the original.

**Figure 3**
Distribution of scores of the original images (blue) and the scores of the images after applying the best individual found when using ImagEnh (orange) according to different Image Quality Evaluation metrics.

The overall results suggest that it is possible to attain suitable pipelines that visually enhance the image according to image quality assessment metrics. The evolved pipelines show improvements across the validation metrics, showing that it is possible to create image enhancement pipelines automatically.

**Figure 4**
The Original image (left), the output image processed by the Manual pipeline with metrics from the literature (middle) and the output image best-evolved solution (right).

Non-Photorealistic Rendering

During the experiments, some of the created pipelines create non-photorealistic rendering effects in a moment of computational serendipity. In such cases, the pipeline outputs have their characteristics such as colour, edges, and contrast visually exaggerated, which transformed the content and created stylized versions of the images. Nevertheless, even some of these outputs maximized all IQA metrics used.

We further analysed the different evolved non-photorealistic solutions, showing the potential of applying the evolved pipelines in other types of images for artistic purposes. Below we can observe the application of the same set of 12 evolved pipelines that generate a non-photorealistic effect on:

1 – images from the real-estate problem dataset

2 – photographs extracted from the web

3 – artworks

4 – abstract wallpapers

This showcases the effect and potential of enhancing or creating new stylized images with the pipelines for different image types.

For more information check our publications on the topic.

Publications

J. Correia, L. Vieira, N. Rodriguez-Fernandez, J. Romero, and P. Machado, “Evolving Image Enhancement Pipelines,” in Artificial Intelligence in Music, Sound, Art and Design, Cham, 2021, pp. 82-97.
- Bibtex
- |
- PDF
J. Correia, D. Lopes, L. Vieira, N. Rodriguez-Fernandez, A. Carballal, J. Romero, and P. Machado, “Experiments in Evolutionary Image Enhancement with ELAINE,” Genetic Programming and Evolvable Machines, vol. 23, iss. 4, pp. 557-579, 2022.
- Bibtex
- |
- Link
- |
- PDF

Author

João Nuno Correia

Leonardo Vieira

Nereida Rodriguez-Fernandez

Juan Romero

Penousal Machado

Daniel Lopes

Adrian Carballal