MET:Semiotic Domains and Non-Textual Technologies

From UBC Wiki

This page was originally authored by Barrie Carter and Duncan Knight (2008).
It was updated and edited by Rachel Graf (2014).


Introduction

Semiotics is the study of symbols and signs and their use or interpretation. In a broader sense, semiotics includes the study of signs and sign processes, indication, designation, likeness, analogy, metaphor, symbolism, signification as well as communication. It is often depicted as a complex, tripartite relationship between a sign, the signifier and the signified. The sign, according to Ferdinand de Saussure, is an independent and “arbitrary” result of the connection made by the interpreter between the signifier (the word or sound) and the signified (the meaning or concept). Thus, a dog would be the sign that was produced by the interpreter, who saw the word on a page (signifier) and thought of a mental picture of a Collie (signified). Others might have interpreted the word to be another breed or even to have a negative connotation of a machine that did not work. Charles Sanders Peirce referred to this relationship as between a representamen (sign), object (signifier) and interpretant (signified).

The following are three different ways of representing the tripartite relationship between the sign, the signifier and the signified:


Simplified Model

File:Blue Semiotic Triangle.jpg

Peirce’s Model

File:Peirce Triangle.gif

Complex Model

File:Yojo Semiotic Triangle.jpg


Three Types of Signs

While Charles Sanders Pierce (1994) has classified 74 different types of signs, they can be narrowed down to three key types: icons, symbols and index. Icons are signs that resemble closely what they signify. For example, many road signs show a silhouette of a car which is easy to recognize. Symbols have meaning because it has been assigned. They simply mean what they mean even though there is no natural relationship between the sign and the object it refers to. For example, road signs in the shape of a yellow triangle could signify that the driver is to yield even though there is no direct relationship between yellow triangles and the action of yielding. Indexes can be described as a learned association. For example, a thermometer is an index for temperature.

File:Calculator symbol.gif




Pedagogical Implications

Semiotics is a difficult subject to categorize given that it depends wholly on an individual’s interpretation: Some have argued that it is its own discipline, while others deem it to be a form of linguistics. Furthermore, there are branches of semiotics, notably: syntactics (the focus on the relationships between signs the structure of the language), semantics (the connections between signs and their meanings) and pragmatics (the impacts that signs have on their targets) (Hylinka, 1989).

Regardless of the complexity and lack of universally defined methodology or theory, there are serious implications for teachers:

  • Students are faced with a plethora of media and need to be able to navigate within these environments, as well as decode them, in order to obtain knowledge.
  • They need semiotic awareness in order to be able to join affinity groups as well as understand the symbols and contexts present (Gee, 2003, p. 27).
  • Students must know the codes necessary within a specific domain in order to be active and critical learners (Gee, 2003, pp. 45-50).
  • Students need to learn the skills of "abduction" (inferences about meanings of signs if the context is foreign) in order to be able to work effectively in specific semiotic domains, such as Math or Science (Yeh and Nason, 2004, pp. 3-4).
  • The skills and knowledge acquired by a learner in one domain can be applied or modified in other arenas without being bound by rituals (Gee, 2003, pp. 44-45).


Semiotics extends far beyond the realm of print and can be applied to art work, images in advertising, mathematics and many other media forms. The audience is an important component in how things are understood given that there need to be agreed upon conventions. The driving force of how meaning is constructed and understood is the invention, utilization, as well as the manipulation of signs and symbols within any communication system. Furthermore, semiotic domains are a shared set of modalities that communicate meaning to a community of people. Here, the community of people is children who also use signs and symbols to convey meaning, as well as to demonstrate understanding. Combining constructivist and experiential activities with semiotic principles provides deeper, significant meanings for students (Semetsky, 2007). Consequently, technologies, such as a paint programme or even a video game (Gee, 2003), enable students to become fluent with these media and increase their membership in various semiotic domains.

Visual Literacy

Visual literacy can be defined as the ability to interpret messages as well as generate images for communicating ideas and concepts (Bleed, 2004). The enGauge Report (2003) on 21st century learning skills lists visual literacy as one of the key skills students should have for the future. They believe that students who are visually literate can:

  • Understand basic elements of visual design, technique and media
  • Be aware of emotional, psychological, and cognitive influences on perceptions of visuals
  • Apply knowledge of visuals in electronic media
  • Be effective visual communicators

It is becoming increasingly important for today's learners to be visually literate. Including more semiotic domains and non-textual technologies in the classroom is one way to increase visual literacy skills.

Non-Textual Technologies

The following list of technologies are far from exhaustive: They illustrate how technology can use the knowledge of semiotics in non-print domains to engage students, to help them construct their own meanings and develop the meta-thinking necessary to develop a deep understanding of the domain involved.


Paint Programmes

File:Paint Programme Semiotics.jpg

Electronic paint is a form of media that allows children to create art using digital paint. Here, children use a stylus-driven tablet PC, electronically painting and drawing images for representation. The authors have come to discover that electronic, digital, interactive devices like the tablet PC help children’s development in semiotic understanding. After all, by way of interaction between the children and the adults, the children understood computer-based metaphoric language and painting language having to do with colour, shape, light, and movement. As well, the authors believe that electronic paint, if used well by teachers, may help remove some of the challenges associated with art lessons (Matthews & Seow, 2007, pp. 251-262).


File:Semiotics Paint Example.jpg

Art (and other forms of visual media) greatly influences the frames of reference of children. Here, art and cultural teachers need to reposition their use of artefacts, images, and performances into a context for new discourses. That is, today’s children use iconographic elements, for example, as part of their experiential understanding. As such, in order to find relevancy for today’s art education, interrelationships between children’s frames of reference and visual experiences need to be founded upon the development of new strategies between viewers (i.e. children), artefacts, and educators (Fulkova & Tipton, 2008, pp. 27-40). Overall, educational strategies need to allow for multiple forms of visual media in order to provide children with learning opportunities to explore self-expression.


Hypermedia and Multi-modal Texts (Print and Pictures)

File:Any key.jpg

Electronic literacy is another form of media that incorporates both text and pictures. Although electronic literacy is not in the form of print, it still has the same value and importance as printed text and picture books. Indeed, the only difference is that this form of media is electronic. Today’s children are exposed to new media (e.g. websites) that have them organize and interpret text and images differently from the traditional printed text. Although new demands are placed upon children to transfer literacy skills from print to hypertext, children are able to do so without much difficulty (Mackey & McClay, 2000, pp. 191-200).

File:Signs in the classroom.png
Using signs and symbols in the classroom is a way to get students' attention quickly and to provide a broader use of visual learning techniques.



Interactive whiteboard signs were created by Ross Morrison McGill (2006) who is widely known in the UK for his Twitter handle @TeacherToolkit. He conducted a study to see if using semiotics can improve teaching and learning in the elementary classroom. He found that using the signs and symbols for classroom management and and activities were useful to teachers and students because it involved a visual cue and reduced the amount of repetitive instructions given by the teacher.


Virtual Reality (VR) Technologies

The field of Mathematics presents its own challenging set of signs and domains. Lemke (1991) states that there are two forms of semiotics at play in Mathematics: typological (symbols, chemical icons, spoken words etc.) and topological (meaning is developed by play with variables, such as size, shape, colour, pitch or coordinate relationships). 3-D Geometry has a similar semiotic framework to Peirce’s model: Communication (signs) is developed from the interaction between objects and spatial skills (Yeh and Nason, 2004). Programmes, such as VR Math 2.0, Logo and Geometer’s Sketchpad, enable students to complete tasks by learning and consolidating the signs they have learned. Students develop their topological understanding by exploring the results of changing angles or trying new coordinates, as well as their typological knowledge through discussion (Yeh and Nason, 2004).


Another sub-domain of the virtual reality arena is video gaming. Video gaming, in turn, has a number of sub-sections, such as first person shooter or role playing games. Gee (2003, pp. 45-46) advocates that video games are valuable tools for enhancing childrens’ learning in four ways:

  • Students acquire new experiences and are exposed to new worlds
  • Gamers become part of new affinity groups with shared languages and conventions (often the result of talking about games online)
  • Players learn to apply knowledge to later stages of games or other different games
  • Learners think about the environment in which they are working and develop an understanding of how games can be contrived or designed to make people think about certain situations

Consequently, video game playing is a positive endeavour that enables students to expand their ability to work in various semiotic domains. Gee’s arguments are cogent and interesting to read, but some critics argue that the inherent violence of many video games negate their educational benefit. Studies have shown that some games result in increased aggression and antisocial behaviour (Anderson, 2004). This does not necessarily contradict Gee’s assertion that video games result in acceptance within certain “affinity groups” given that they may be included in one group (online gaming forums for example), but would find it difficult to cope in other domains.


Critical Issues

The process of learning occurs by way of the development of cognitive systems, and parts of those systems are not only a child’s different forms of knowledge and cognitive abilities, but also other people, things and signs. Cognitive systems, in turn, are semiotic systems, for they are dependent on signs and representations as intermediaries (Hoffman, 2007, pp. 185). As such, the semiotic domain of visual media is applicable, for this type of media includes, but is not limited to: signs, art, symbols, images, and pictures, and the interrelationship between child and adult (e.g. student and teacher). Technology must allow students to create their own environments in order to develop a rich set of signs, which have been the results of the interaction between signifiers and their interpretation. Consequently, it is important that the development and use of technology serve to further the inclusion of students within a wider array of semiotic domains, while still being user-friendly.

Design Factors

Plass, Homer, and Hayward (2009) have conducted research on the design factors which influence the educational effectiveness of animations and simulations. They conclude that the following design factors are key when designing animations and simulations for use in the classroom:

  • Dynamic visualizations that are interactive in nature
  • Cueing, also referred to as signaling, is used to direct the learner's attention to the important aspects of the learning material
  • Key information is represented in iconic (pictorial) form rather than only in symbolic (textual) form
  • Colour is used to highlight specific features

Visual attention is an important aspect of semiotic domains. Neuroscience research has shown that objects compete for representation and processing even at a neuronal level, and more research is needed to better understand how bottom-up and top-down processes can be used to guide learners’ attention to the essential parts of the visual information.

Bottom-up processes make use of the salient features of an object which direct the learner's attention. Examples include high contrast and visual uniqueness. Top-down processes are dependent on the learner's schema or his or her knowledge, background information, goals and expectations. Top-down processes have been found to influence what the learner perceives or takes away from an image. This means that two learners with different goals observing the same material or process can have different perceptual experiences (Plass et al. 2009).

Social Semiotics

Social semiotics focuses on sign systems in particular contexts or situations. Every community is different, therefore the signs used by one community may be different from those used by another. When designing educational tools using semiotics, it is important to look at cultural context when it comes to symbols, colours and signs. Yeh and Nason (2008) have studied the importance of multiple knowledge representations in mathematics education. Representations of mathematical concepts have been classified differently in different contexts. They found that when multiple semiotic resources were used, the mathematical ideas or concepts could be better learned. In their study, participants had to mix paints and create certain colours according to the colour wheel. They then had to apply what they new about mixing certain amounts of paint to learning ratio and fractions. In a pre-interview with the participants, Yeh and Nason noted that some participants were involved in the vocational training workshops where they were mixing paints. Some participants were able to identify slight differences in colours while others were not able to.


File:6 different, but same old guy.jpg


This image shows the same man in six different poses showing different emotions and actions. There are many different forms of communication beyond the spoken or written word and semiotics are a powerful tool in the creation of non-textual technologies.




















Stop Motion Video

Edusemiotics Stop Motion by Dana Stanley

References

  • Anderson, C.A. (2004). An update on the effects of violent video games. Journal of Adolescence, 27, 113-122.
  • Gee, James Paul (2003). What Video Games have to Teach Us About Learning and Literacy. New York: Palgrave MacMillan.
  • Hylinka, Denis (1989). Applying Semiotic Theory to Educational Technology. Dallas, TX: Annual Meeting of the Association for Educational Communications and Technology. (ERIC Document Reproduction Service No. ED308820).
  • Lemke, J. L. (December, 1993). Multiplying Meaning: Literacy in a Multimedia World. Charleston, SC: Annual Meeting of the National Reading Conference. (ERIC Document Reproduction Service No. ED365940).
  • Yeh, Andy J. and Nason, Rodney A. (2004). Towards a Semiotic Framework for Using Technology in Mathematics Education: The Case of Learning 3D Geometry. In Proceedings International Conference on Computers in Education, Melbourne, Australia. Retrieved February 24, 2008, from http://vrmath.yeh.id.au/publications/icce2004_semioticsYeh.pdf.
  • Yeh, A. and Nason, Rodney, A. (2008). Mixing colours: An ICT tool based on a semiotic framework for mathematical meaning-makng about ratio and fractions. In: Proceedings of the: 31st Annual Conference of the Mathematics Education Research Group of Australasia: Navigating currents and charting directions, June 28 - July 01,2008, St Lucia, Brisbane, Queensland. Retrieved from: http://eprints.qut.edu.au/29374/1/29374.pdf


External Links

Many of the links to project sites, resources and other sources of interest were found in the readings and studies cited in the references section:

  • Semiotics Wikipedia's Explanation of Semiotic Theory.
  • The Impressionist An online paint program that allows students to explore Impressionism by making a painted representation of a photograph.
  • Queeky Paint A great online draw program that helps the drawer explore the process of creating artwork.
  • Queeky Paint Tutorials You can view over 50,000 artists' work as well as find out tips and tricks for using Queeky.
  • Scratch Another excellent website for young artists where you can create stories, games and animations.
  • VRMath 2.0 A site that uses 3D imaging technology and Web 2.0 ideas to teach geometry concepts. You are able to explore math in 3D virtual worlds.
  • Treehugger An interesting site that explores semiotics (particularly in advertising) and the environment.
  • Working Examples A place for researchers, designers and educators to connect. It also serves as a new form of academic publishing, one that accommodates non-textual material such as images, videos and diagrams to explain how educational technologies are used.
  • Interactives. Draw three-dimensional figures on two-dimensional isometric dot paper. Try holding the cubes in different orientations so you can see the differences between 2D and 3D
Retrieved from "https://wiki.ubc.ca/index.php?title=MET:Semiotic_Domains_and_Non-Textual_Technologies&oldid=687771"