One current goal of multimedia research is to make multimedia information pervasively accessible and useable. An essential question to achieve that goal is under which circumstances a particular medium or a mix of media will serve a particular communication need more effectively than others. This question becomes in particular relevant for research in social media and mobile media, as both provide the environments that establish contexts allowing, for the first time, to observe social behavior on a large scale and in real time.
One promising approach to interpreting this data, and as a result facilitate the building of tools that are needed to support communication between humans as well as between humans and their living environment more effectively, is founded in understanding the semantics of various media within a computationally informed and systematic study of media production and reception. The purpose of this tutorial is to provide an understanding of the role and applicability of semiotics to facilitate the modelling of media semantics for various contexts. The tutorial addresses a number of specific issues:
- Basic communication and semiotic theory
- The applicability of media semiotics to capturing, representing, processing, managing, and personalizing media.
Audience / Format / Requirements / Schedule
Tutorial lengths: half day on Monday the 25th of October
Level: Introductory to Intermediate.
This tutorial is designed for researchers and practitioners who would like to learn how to understand the semantics of various media, how to describe them, and how to make use of such descriptions in the whole value chain of media creation, management, distribution, and delivery, with the help of semiotics. While the tutorial is focused on media semantics viewed from a semiotic point of view, however given media semantics’ fundamental import to multimedia systems, the tutorial should also be of interest to people working in: Social Media, Mobile Media, Desktop Video Editing, Multimedia on the Web, Digital Asset Management, Media-Rich Homes / Digital Entertainment, Practical Digital Libraries, Multimedia User Interface Design, Multimedia in Collaboration, Multimedia Information Retrieval. The tutorial is set up in the form of lecture units, where hands-on exercises are included to assure audience participation. The exercises are introductory and fun, yet provide the direct experience needed to understand and apply semiotic concepts.
The preliminary schedule is as follows:
8:30 – 8:40 Welcome, Introductions, and Overview (10 minutes)
Provide overview of tutorial goals, schedule, and ground rules
8:50 – 9:50 Fundamentals of Semiotics (60 minutes)
Explain the basic concepts of communication theory and semiotics as foundations for media semantics
Communication (the creation of meaning)
The sign
Code systems
Syntagms and paradigms
Denotation, connotation and modality
Rhetorical tropes
Encoding, decoding and articulation
9:50 – 10:05 Break (15 minutes)
10:05 – 11:05 Semiotic Properties in Visual Media (60 minutes)
Introduce the main semantic concepts of static and temporal visuals (space, people, objects, composition and how these can be computationally described and exploited (semantics of time and motion, montage, argumentation structures, meta-data systems)
11:05 – 11:15 Break (10 minutes)
11:15 – 12:15 Semiotics-inspired Media systems (60 minutes)
Describe a few additional applications using described concepts to provide a better understanding of how media semiotics help to restructure, represent, re-sequence, or repurpose media in order to establish stimulating experiences in the user. The applications address social media applications that make use of Flickr and Twitter data, mobile applications that address experiential aspects, and a number of applications that facilitate or support creativity.
Organizers/Presenters
Frank Nack, Intelligent Systems Lab Amsterdam (ISLA), Institute for Informatics, University of Amsterdam Science Park 107, 1098 XG Amsterdam, The Netherlands
Frank is tenure assistant professor at the Intelligent Systems Lab Amsterdam (ISLA) of the Informatics Institute of the University of Amsterdam (UvA). He obtained his Ph.D. with a thesis on “The Application of Video Semantics and Theme Representation for Automated Film Editing,” at Lancaster University, UK. The main thrust of his research is on representation, retrieval and reuse of media in hypermedia systems, context and process aware media knowledge spaces, representation and adaptation of experiences, hypermedia systems that enhance human communication and creativity, interactive storytelling, computational applications of media theory & semiotics, AI and film (semantics, semiotics, perception) and automated video editing, and computational humour theory. He has published more than 100 papers on these topics.
He was a member of the MPEG-7 standardization group where he served as the editor of the Context and Objectives Document and the Requirements Document, and chaired the MPEG-7 DDL development group. He organized workshops, tutorials and panels at major conferences, such as ACM Multimedia, ACM Hypertext, ACM CHI, IEEE Conference on Multimedia and Expo (ICME), International Conference on Entertainment Computing (ICEC), or ICIDs (International Conference on Interactive Digital Storytelling). He has been serving on program committees (member or chair) at the major conferences in the field of multimedia: ACM MM, ACM Hypertext, Multimedia Modelling (MMM), ICME.
He is on the editorial board of IEEE Multimedia, where he edits the Media Beat column and also serves as associated editor in chief. He also serves on the board of IEEE Transactions on Computational Intelligence and AI in Games.
T04 – The Role of Media Semiotics to Facilitate Media Understanding
One current goal of multimedia research is to make multimedia information pervasively accessible and useable. An essential question to achieve that goal is under which circumstances a particular medium or a mix of media will serve a particular communication need more effectively than others. This question becomes in particular relevant for research in social media and mobile media, as both provide the environments that establish contexts allowing, for the first time, to observe social behavior on a large scale and in real time.
One promising approach to interpreting this data, and as a result facilitate the building of tools that are needed to support communication between humans as well as between humans and their living environment more effectively, is founded in understanding the semantics of various media within a computationally informed and systematic study of media production and reception. The purpose of this tutorial is to provide an understanding of the role and applicability of semiotics to facilitate the modelling of media semantics for various contexts. The tutorial addresses a number of specific issues:
Audience / Format / Requirements / Schedule
Tutorial lengths: half day on Monday the 25th of October
Level: Introductory to Intermediate.
This tutorial is designed for researchers and practitioners who would like to learn how to understand the semantics of various media, how to describe them, and how to make use of such descriptions in the whole value chain of media creation, management, distribution, and delivery, with the help of semiotics. While the tutorial is focused on media semantics viewed from a semiotic point of view, however given media semantics’ fundamental import to multimedia systems, the tutorial should also be of interest to people working in: Social Media, Mobile Media, Desktop Video Editing, Multimedia on the Web, Digital Asset Management, Media-Rich Homes / Digital Entertainment, Practical Digital Libraries, Multimedia User Interface Design, Multimedia in Collaboration, Multimedia Information Retrieval. The tutorial is set up in the form of lecture units, where hands-on exercises are included to assure audience participation. The exercises are introductory and fun, yet provide the direct experience needed to understand and apply semiotic concepts.
The preliminary schedule is as follows:
8:30 – 8:40 Welcome, Introductions, and Overview (10 minutes)
Provide overview of tutorial goals, schedule, and ground rules
8:50 – 9:50 Fundamentals of Semiotics (60 minutes)
Explain the basic concepts of communication theory and semiotics as foundations for media semantics
Communication (the creation of meaning)
The sign
Code systems
Syntagms and paradigms
Denotation, connotation and modality
Rhetorical tropes
Encoding, decoding and articulation
9:50 – 10:05 Break (15 minutes)
10:05 – 11:05 Semiotic Properties in Visual Media (60 minutes)
Introduce the main semantic concepts of static and temporal visuals (space, people, objects, composition and how these can be computationally described and exploited (semantics of time and motion, montage, argumentation structures, meta-data systems)
11:05 – 11:15 Break (10 minutes)
11:15 – 12:15 Semiotics-inspired Media systems (60 minutes)
Describe a few additional applications using described concepts to provide a better understanding of how media semiotics help to restructure, represent, re-sequence, or repurpose media in order to establish stimulating experiences in the user. The applications address social media applications that make use of Flickr and Twitter data, mobile applications that address experiential aspects, and a number of applications that facilitate or support creativity.
Organizers/Presenters
Frank Nack, Intelligent Systems Lab Amsterdam (ISLA), Institute for Informatics, University of Amsterdam Science Park 107, 1098 XG Amsterdam, The Netherlands
He was a member of the MPEG-7 standardization group where he served as the editor of the Context and Objectives Document and the Requirements Document, and chaired the MPEG-7 DDL development group. He organized workshops, tutorials and panels at major conferences, such as ACM Multimedia, ACM Hypertext, ACM CHI, IEEE Conference on Multimedia and Expo (ICME), International Conference on Entertainment Computing (ICEC), or ICIDs (International Conference on Interactive Digital Storytelling). He has been serving on program committees (member or chair) at the major conferences in the field of multimedia: ACM MM, ACM Hypertext, Multimedia Modelling (MMM), ICME.
He is on the editorial board of IEEE Multimedia, where he edits the Media Beat column and also serves as associated editor in chief. He also serves on the board of IEEE Transactions on Computational Intelligence and AI in Games.