ESSYS* Sharing #UC

ESSYS* Sharing #UC is an audiovisual installation that presents a multimodal environment representing the content shared online, on Twitter, about the city and the university of Coimbra.

ESSYS* Sharing #UC presents a hybrid nature, being accessible both in a physical art gallery and online (accessible at http://sharing-uc.dei.uc.pt). In both installation media, one may see a set of generated typographic poster designs, displaying tweets related to Coimbra, accompanied by an ever-changing ambient sound. The audience feedstock the audiovisual generation of the installation, influencing indirectly (through their tweets) the creation and the characteristics of the exhibited multimedia artefacts.

The installation is a mirror to an audiovisual environment, created by a computational creativity system that throughout the detection and textual recognition, classifies the content based on the eight basic and prototypal emotions: joy, anticipation, trust, fear, anger, disgust, surprise, and sadness. The detected emotions are feedstock to the generation of audiovisual artefacts. The sonic compositions travel melodic and harmonically in a musical universe that breathes these emotional atmospheres, emerging from the most provocative contents in the form of probabilities of sound parameters. On the other hand, the typographic compositions translate the emotional charge of contents into visual variables, inspired by the production processes of letterpress.

This artwork was commissioned by the Institute of Interdisciplinary Research of the University of Coimbra (UC), in the context of the celebration of the 730th anniversary of UC. The installation was designed for the Francisco Tropa Museu art gallery of Círculo de Artes Plásticas de Coimbra (CAPC) in Praça das Cortes, Coimbra, Portugal. This space presents a participatory and easily accessible nature, being developed simultaneously as an urban sculpture and an always-open exhibition space. More recently, in May 2022, a new physical setup version of the installation by invitation of the Faculty of Science and Technology of UC in the commemoration of its 250 years.

**Figure 2**
Photography of a new configuration of. #ESSYS* Sharing #UC in the commemoration of 250 years of Faculty of Science and Technology of UC (May 2022), during one keynote speaker of Fernado Boavida. Photography by Tiago Martins. © 2022.

The present audiovisual artefacts are designed by an autonomous computational creativity approach that comprises three modules: (I) the Input Preprocessing module, which employs a Machine Learning approach to emotionally classify the tweets; (II) the Typesetter module, which generates and renders the poster designs employing a greedy strategy based on a set of predefined rules and several OpenType Variable Fonts; and (III) the Emotion Sonification System (ESSYS) module, which generates the ambient audio based on a rule-based system.

Demonstration video of the generation of poster designs, through the created typesetter module using a greedy strategy employing on a set of predefined rules and several OpenType Variable Fonts.

Figure 3

ESSSY* is a joint venture that explores the multimodal production of audiovisual artefacts employing the Poster Factory’s systems and the ESSYS system. This artwork is produced with the support of Iniciativa de Apoios Financeiros para Promoção da Cultura Científica da Universidade de Coimbra (in the scope of the celebration of 730 years of UC) and CAPC. We like to thanks to the following institutions and people: João Bicker, F. Amílcar Cardoso, (Rui) Buga Lopes, Penousal Machado, Serviço de Informática e Comunicações do DEI/FCTUC, Círculo de Artes Plásticas de Coimbra, Jorge das Neves, Tiago Martins, Pedro Silva, Centro de Informática e Sistemas da Universidade de Coimbra e Departamento de Engenharia Informática da Universidade de Coimbra.