ESSYS* Sharing #UC
ESSYS* Sharing #UC is an audiovisual installation that presents a multimodal environment representing the content shared online, on Twitter, about the city and the university of Coimbra.
ESSYS*: Sharing #UC installation. Photo by © Jorge das Neves / CAPC
ESSYS* Sharing #UC presents a hybrid nature, being accessible both in a physical art gallery and online (accessible at http://sharing-uc.dei.uc.pt). In both installation media, one may see a set of generated typographic poster designs, displaying tweets related to Coimbra, accompanied by an ever-changing ambient sound. The audience feedstock the audiovisual generation of the installation, influencing indirectly (through their tweets) the creation and the characteristics of the exhibited multimedia artefacts.
The installation is a mirror to an audiovisual environment, created by a computational creativity system that throughout the detection and textual recognition, classifies the content based on the eight basic and prototypal emotions: joy, anticipation, trust, fear, anger, disgust, surprise, and sadness. The detected emotions are feedstock to the generation of audiovisual artefacts. The sonic compositions travel melodic and harmonically in a musical universe that breathes these emotional atmospheres, emerging from the most provocative contents in the form of probabilities of sound parameters. On the other hand, the typographic compositions translate the emotional charge of contents into visual variables, inspired by the production processes of letterpress.
This artwork was commissioned by the Institute of Interdisciplinary Research of the University of Coimbra (UC), in the context of the celebration of the 730th anniversary of UC. The installation was designed for the Francisco Tropa Museu art gallery of Círculo de Artes Plásticas de Coimbra (CAPC) in Praça das Cortes, Coimbra, Portugal. This space presents a participatory and easily accessible nature, being developed simultaneously as an urban sculpture and an always-open exhibition space. More recently, in May 2022, a new physical setup version of the installation by invitation of the Faculty of Science and Technology of UC in the commemoration of its 250 years.
Photography of a new configuration of. #ESSYS* Sharing #UC in the commemoration of 250 years of Faculty of Science and Technology of UC (May 2022), during one keynote speaker of Fernado Boavida. Photography by Tiago Martins. © 2022.
The present audiovisual artefacts are designed by an autonomous computational creativity approach that comprises three modules: (I) the Input Preprocessing module, which employs a Machine Learning approach to emotionally classify the tweets; (II) the Typesetter module, which generates and renders the poster designs employing a greedy strategy based on a set of predefined rules and several OpenType Variable Fonts; and (III) the Emotion Sonification System (ESSYS) module, which generates the ambient audio based on a rule-based system.
Input Preprocessing
The Input Preprocessing module employs a neural NLP approach to predict the emotional charge of a tweet poster about Coimbra. We focused the system on recognising the eight emotions of Plutchik’s theoretical model: anger; anticipation; disgust; fear; joy; sadness; surprise; and trust. These data inform the outputs generated by the other modules, both visuals and audio.
Typesetter
The Typesetter module generates the poster designs informed by the gathered emotional context, through a generative styling method (which defines the visual features of posters) and a greedy typesetting method (which composes the text). The posters’ background, chromatic and typographic styles are selected using a stochastic method informed by the emotional recognition data. There are available four background styles: (i) solid; (ii) diagonally halved; (iii) solid divided; and (iv) gradient. In these background styles, the employed colours are chosen based on the emotions recognised in the text. The poster’s typeface is chosen based on the predominant emotion. The text alignment and text box alignment are randomly determined. When no emotion is recognised in the tweet (i.e. neutral), the text is composed in black over a white background. The following figure illustrates the correlation between these recognised emotional data and the background, chromatic and typographic styles of generated poster designs.
Demonstration video of the generation of poster designs, through the created typesetter module using a greedy strategy employing on a set of predefined rules and several OpenType Variable Fonts.
ESSYS
The ESSYS module generates MIDI sonic compositions that reflect the predominant emotion on the posters presented on installation. This sonic composition is generated by a rule-based system where the musical parameters are dynamically chosen according to predefined probabilities, which depend on the emotion being portrayed.
Resources
ESSYS* Sharing #UC (online version)
Design Configuration File (colour and typeface configuration rules of typesetter)
Installation Documentation (some generated posters, photos, and screenshots)
Poster Designs generated during the evaluation stage
Poster Background variations caption
On News