Using Images Schemas in the Visual Representation of Concepts

Two types of categorisation processes can be said to take place in concept formation: perceptual (what objects look like) and conceptual (e.g. affordances). Yet, systems that produce visual representations for concepts mostly focus on perceptual features (e.g. shapes, colours, etc). We propose an approach to include affordance-related features in such systems, based on the idea that they can be modelled using image schemas – learned spatio-temporal relations that can be seen as conceptual building blocks (e.g. CONTAINMENT, SUPPORT, etc).

 
 

Implementation Strategy

 

The proposed approach uses the following 4-step pipeline:

  1. Identification of the concept to be visually represented, which is normally introduced by the user (e.g. house or elevator in Fig. 1);
  2. Identification of image schemas by employing techniques to extract image schematic structures for concepts or spacial descriptions from text (e.g. CONTAINMENT for house in Fig. 1);
  3. Gathering input visual representations, by using a dataset of visual representations and corresponding semantic information, allowing to match concepts with visual representations (e.g. building and person for the house example in Fig. 1);
  4. Production of visual representations: using the gathered visual representations (e.g. building and person) in combination with the identified image schema(s) (e.g. CONTAINMENT) to generate visual representations of the concept (e.g. house).

 

Figure 1

Identification of image schemas for house and elevator, using examples retrieved from Oxford Dictionaries and WordNet.

 
 

Illustrative example

 

Icons often seen in signage system show how image schemas are used in icon design. One example is the icon for the concept house. The concept house can be represented using only perceptual features (e.g. the icon shown in step 1 only represents the roof and the walls of a house, see Fig. 1). However, it can also use the affordance of serving as a shelter. In this sense, it is important to mention that the roof shape may also be seen as affordance-indicating and not purely perceptual. By considering the affordance of serving as a shelter, one may relate it to the CONTAINMENT image schema – identified in the example descriptions (“human habitation” or “living quarters”). The CONTAINMENT schema implies a container entity and a contained entity, which can be respectively linked to “building” or “dwelling”, and “human”, based on the descriptions provided in Fig. 1. This can result in a person sign placed inside a building (see step 2 of house example in Fig. 3). Other examples are given in the published paper [1].

 
 

Implementation Issues

 

The proposed approach has several implementation issues of considerable complexity that need to be addressed. The main issues are:

 

  1. the identification of the image schema (e.g. for SOURCE_PATH_GOAL in Fig. 3 vs PART-WHOLE in Fig. 4 for the concept love);
  2. the visual representation of the image schema (i.e. how to visually used the image schema);
  3. the choice of the adequate entities (using an old person or death to represent the GOAL for life in Fig. 2);

    4.  

    Figure 2

    Two representations of life using different entities for GOAL (old person or death), based on the metaphor life as a journey (PATH_SOURCE_GOAL image schema).

     
     
     

  4. the meaning variation triggered by using different examples (Figs. 4);

    5.  

    Figure 3

    Two representations for love based on the metaphor love as a journey (PATH_SOURCE_GOAL image schema), using different examples.

     
     
     

    Figure 4

    Representation of love using the metaphor love as unity (PART-WHOLE image schema).

     
     
     

  5. the image schema activation: how to correctly activate the intended image schema (e.g. activation of CONTAINMENT using visual representations for boat and person in Fig. 5);
  6. suitable combination of elements in the blending process.

 

Figure 5

Issues in the activation of the CONTAINMENT image schema for the concept boat

 

The approach and current issues are described in detail in [1].
 
 

Publications

 

  • J. M. Cunha, P. Martins, and P. Machado, “Using Image Schemas in the Visual Representation of Concepts,” in Joint Proceedings of the Workshops C3GI: The 7th International Workshop on Computational Creativity, Concept Invention, and General Intelligence ISD4: The 4th Image Schema Day, and SCORE: From Image Schemas to Cognitive Robotics, Bozen-Bolzano, Italy, December 13-15, 2018., 2018.
    • Bibtex
      • @inproceedings{cunha2018schema,
      Author = {Jo{\~{a}}o Miguel Cunha and Pedro Martins and Penousal Machado},
      Booktitle = {Joint Proceedings of the Workshops {C3GI:} The 7th International Workshop on Computational Creativity, Concept Invention, and General Intelligence {ISD4:} The 4th Image Schema Day, and {SCORE:} From Image Schemas to Cognitive Robotics, Bozen-Bolzano, Italy, December 13-15, 2018.},
      Publisher = {CEUR},
      Title = {Using Image Schemas in the Visual Representation of Concepts},
      Year = {2018}}
    • |
    • PDF

Author

João Miguel Cunha

Pedro Martins