A Survey on Multimedia Ontologies for a Semantic Annotation of Cinematographic Resources for the Web of Data

Regarding various spheres of interest and values, the Web provides a variety of information sources. The quantity and attractiveness of resources available on the Web have grown over time [1]. The techniques provided by the Semantic Web enable reasoning over these resources. Multimedia resources including videos, audios, films, and images are increasingly common in contemporary Web contents. A prominent indicator of the rise of such resources in daily activities of the current society is the deluge of multimedia resources on the Internet and in some offline platforms. Both experts and amateurs create and consume these multimedia contents.

Cinematographic works are a subcategory of multimedia resources that include all on-screen visual components, such as framing, composition, camera angles, film selection, depth of focus, color, exposure, and filtering. A majority of film creators and viewers use the Internet to learn more about cinematic topics, and nearly 24% of films are web-driven [1]. Access to such pertinent content in a timely manner is therefore a major challenge [2].

For data access and retrieval, semantic indexation of video content has evolved to be a critical issue. Contrary to textual assets, media search relies on methods that either have onerous specifications for feature comparison (such as color or texture), or on associated descriptors, which involve choosing elements of a frame or video and translating them into text or concepts from a predefined vocabulary [3]. However, a sizable number of concepts in a taxonomical framework, concepts, roles, and rules are also required for the content representation of Multimedia Ontologies [4], [5].

There are various aspects of sociocultural concepts in the world of cinema. Examples of concepts that can be utilized for indexing such resources include actor names, generics, and places used in films. As in “Jet Lee” (Asian), “Siriki” or “Souke” (African), “Jack Bauer,” “Mr. Bean,” etc., an actor's name has a social identity (country or town). A generic in a film is typically a familiar melody fragment; other generics include the channels on which films are shown, the audiences they are aimed at, etc. The indexation system of the web of data is improved by the modeling of such ideas, e.g., removing a list of films shown at a particular tourist destination or at a popular location. From this perspective, it is necessary to have a shared vocabulary among ontologies addressing cinematographic content and associated semantic structures. The theories of cultural heritage modeling can be applied in this direction.

In this research, the authors examined multimedia ontologies that are currently in use to describe cinematographic works. The remainder of this article is structured as follows: Section 2 introduces the semantic web and multimedia resources; Section 3 examines existing multimedia ontologies in relation to cinematographic works and film production; Section 4 provides a brief discussion on these multimedia ontologies and their shortcomings, with conclusive open perspectives.

Data access, retrieval, and actual application now depend heavily on semantic indexation of images and videos [6]. This section examines the evolution of the Web and how it has affected knowledge representation in general for multimedia resources and specifically for cinematographic resources [7].

Multimedia data has a very complex formalism as it contains a variety of things, including videos, texts, sounds, photos, 3D models, and more [8]. There are numerous multimedia applications in the fields of distance learning, digital libraries, health (telemedicine, medical image databases), entertainment (databases of on-demand video, interactive TV), business (video conferencing, e-commerce), and cultural heritage. The MPEG-7 format has a wide range of uses, including home entertainment systems, broadcast media choices, digital libraries for multimedia, and multimedia editing. This format makes it possible for users to access enormous multimedia archives and multimedia catalogues in order to find content that can be purchased (Table 1). The MPEG-7 standard is compatible with others that have shown success and acceptance in both the traditional media and new media industries. The semantic web has improved the understanding of knowledge representation. It enables computers to comprehend web material and provide a clear meaning in relation to the desired application [9]. By providing high quality video metadata, the semantic web can be searched for information more effectively.

All on-screen visual components are included in cinematographic works. An initial plot, idea, or commission is followed by screenplay, casting, shooting, recording sound during preproduction, editing, and screening the finished product in front of an audience, which may lead to a film release and exhibition. The cinematographic production is both an aesthetic and commercial endeavor. From its conception until its consumption, it follows a production chain. Along with the publishing, music, fashion, design, and other cultural and creative sectors, the cinema business fits more widely into these categories. In terms of operation and financial success, the cultural and creative industries have changed significantly during the previous fifteen years [10]. This is a result of technological development, as well as changes related to dissemination and consumption behaviors, which are also related to technological improvements. Around the world, many technologies and cinematic techniques are used during the making of films in a variety of economic, social, and political circumstances. Tcheuyap [11] presented the two key themes dictating the cultural identity of African Film products, African Films and Films from Africa.

In general, the film industry is strongly anchored in the process of cultural globalization. The American and French domination in the production and distribution sector is a reality and has an influence on other cultures around the world. A recent study carried by the CNC [12] shows that, there is a considerable qualitative and quantitative increase in film production in almost all the contexts (see Figure 1), revealing more of cultural identifies. An information vectors prescriptive analysis of movie production, enlightens more about the fact that 24% of movie viewers are influenced by the content available on the web (see Figure 2). We believe that a more specific annotation process (considering infrastructure, language, activities, communities, etc.) may be a key factor of the integration of multimedia content on the web of data.

Numerous studies on the annotation, indexation, and retrieval of multimedia information on the Web have produced positive findings recently [13]. The majority of current methods rely on the semantic web to allow computers to interpret and comprehend metadata effectively. We are able to find some really intriguing multimedia ontologies that are provided in the following after searching via web repositories and available literature.

NINSUNA Ontology, which Deursen et al. [14] introduced, supports common media formats on the web and acts a metadata-driven system for media modification and delivery (MP4, WebM, Ogg, MPEG-2 TS). It is made up of classes and properties that can be used to retrieve information about media fragments and create a multimedia description apart from the metadata of the original multimedia format. Despite being primarily concerned with format description and redefinition, it offers less insight into the media's substance.

Multimedia ontology manage (MOM), which Bertini et al. [15] modeled as a complete system, enables the creation of multimedia ontologies, supports automatic annotation and the production of extended text (and audio) commentaries of video sequences, and realizes complex queries by reasoning on the ontology [16]. The majority of the time, it is used to automatically caption video scenes and occasionally to create sports commentary.

For effective semantic annotation, Thomas and Syama [17] suggested an ontology-based machine learning system that employs SIFT and HoG characteristics of images from the selected frames and trains them with SVM classifiers.

For the study of China Central Television (CCTV) data, CCTV Ontology was created based on the standard event ontology paradigm for multimedia applications [18]. CCTV Ontology aims to exploit semantics for the identification of potential criminal activity and its perpetrators in a video fragment. It formalizes the fundamental ideas, characteristics, and connections between ideas in the forensic domain [19]. It can be applied to security concerns like searching for information (person, face, and car detection, as well as throwing, kicking, and running actions) in videos of riots and violent conflicts.

An MPEG-7-based multimedia ontology called COMM is comprised of multimedia patterns based on deep ontological foundations, which are built on the fundamental ontology DOLCE and reliable ontology engineering principles [20].

Another Multimedia ontology proposed by Saathoff and Scherp [21] and Gangemi and Presutti [22] is Multimedia Metadata Ontology (M3O). Because of its extensive axiomatization, it is well known as a pattern-oriented ontology design based on the fundamental ontology DOLCE [23]. It identifies five fundamental patterns necessary to convey metadata for multimedia content using Description Logics for formalism. It supports both the annotation with low-level features taken from the multimedia content as well as the representation of high-level semantic annotation with background information. M3O has been included into the Semantic MM4U framework [24] for experimental purposes, and it has assisted in enhancing the Yahoo and Google search engines. It achieved great success in a number of areas, including customized sports news, context-aware travel guides, and the creation and semantic augmentation of individual photo albums.

A full OWL defined ontology framework called Dynamic Pictorially Enriched Ontologies (DPEO) was created by Ballan et al. [25] for the semantic annotation of soccer video clips. To increase the effectiveness of automatic semantic annotation, it employs the Semantic Web Rule Language (SWRL) to reason on both concepts and their instance values.

To construct a wider collection of semantic ideas for reasoning about multimedia information, Naphade et al. [26] presented Large-Scale Concept Ontology for Multimedia. It has over 834 visual concepts and is specifically made to present news footage.

Media resource 1.0, a basic vocabulary ontology created by the W3C Media Annotations Working Group, is presented by Lee et al. [27]. It seeks to provide an interoperable set of reusable information by standardizing the mapping of popular media formats. It supports six different multimedia file types and roughly 18 different multimedia metadata formats, including Dublin Core, MPEG-7, IPTC, and OGG (3GP, FLV, QuickTime, MP4, OGG, WebM).

In order to simplify the process of expressive video annotation, Valkanas et al. [28] created POLYSEMA, a tool that combines semantic web ontologies and MPEG-7 metadata. The gaming industry frequently takes use of this ontology.

The ReDeFer project’s Rhizomik ontology [29] was created using higher-level multimedia characteristics for the music concepts. This ontology resulted from the conversion of the XML Schema from the MPEG-7 standard into an OWL ontology in accordance with the guidelines provided.

Kanzaki is an OWL DL ontology that, by itself, provides the vocabulary used to identify musical works, events, instruments, and performers as well as the semantic relationships between them [30]. The FOAF ontology is specialized in its ideas.

Additionally, we came across VidOnto [31], an ontology that can be used to comprehend video content and enhance the quality of structured annotations of ideas and predicates, but it is limited in its ability to convey the semantic knowledge of the video content. It is appropriate for reasoning-based high-level scene interpretation of videos and for querying video descriptions. Although the rationale is quite intriguing, it does not define the socio-culturally relevant reasoning variables and pays little attention to audiovisual events like awards. This ontology does not have a strong representation of concepts connected to the cultural identity of the video clip.

The large annotation ontology that defines film semantic annotation, a lexicon, and ideas was proposed by Li et al. [32] based on a film video benchmark data set. This model is capable of annotating films and video scenes, according to a HowNet experiment.

The Department of Informatics at the University of Zurich developed the Film Ontology in 2009, an OWL-specified ontology that Amancio Bouza started. The ontology attempts to provide controlled vocabulary to semantically characterize film-related ideas. This ontology includes a hierarchy of ideas that are specifically related to the cinematic realm. Looking at this ontology from several angles, it appears to be among the finest for the category of cinematic works. Although context-aware ideas have the potential to enrich the semantic level of the concepts.

Teyeb et al. [33] presented a semantic annotation of French film events and imagery. The results of this experiment, which was run on ten documents that contained images of film works, were highly promising. This ontology's weakness stems from the absence of a clear connection between it and concepts from the outside world.

A well-known project called M3 Multimedia was started by the Spanish research team Buscamedia [34]. It intends to develop a semantic search engine for media resources, specifically for tools for media production and delivery that are audiovisual. A wide range of comprehensive multimedia ideas and languages are included in this ontology.

We are able to create comparative research that identified five criteria for evaluating these multimedia ontologies that are related to cinema thanks to some elements of the NEON approach [35] and Lemos et al.'s analysis on multimedia annotation on the Web of Data. Therefore, we took into account the following factors: the type of features used in the annotation process, such as High-Level Features (HLF) or Low-Level Features (LLF), the target application field, the semantic level, the degree of interoperability with other reputable annotators, and the index value for consistency [30] (see Table 1).

There are some really fascinating ontologies in the multimedia field. To disclose the evolutionary trend from 2001 to 2019, Suarez-Figueroa et al. [30] proposed to classify multimedia ontologies into four subclasses: Images and shape, object, visual objects, and music ontologies (Figure 3). We observed that the majority of these studies focused more on multimedia ontologies related to musical arts and object annotation and less on cinematographic works.

The accessibility of these kinds of materials on the Semantic Web might yet be improved, according to organizations representing the film industry, by many additional concepts that allow for more information to be available about a cinematographic production. As seen, the annotation already demonstrates that the majority of the suggested multimedia's ontologies consider the semantic background while constructing knowledge [36]. Thus, researching the sociocultural aspects of films seems to hold promise for the web of Data. The notions and characteristics unique to cinematographic works started to emerge soon afterwards (Figure 3). The absence of widely used vocabularies that have been used for multimedia metadata annotation presents certain difficulties when dealing with multimedia annotation on the Semantic Web; as a result, some features of distinct vocabularies may have the same name but different meanings. The IBM 2002 TREC team considered a semi-automatic method for building multimedia ontologies from multimedia collections in these suggested ontologies [37], which combined tools for content-based retrieval and tools for building ontologies from text. The results of an application to a sizable collection of educational films from the 1940s to the 1960s are intriguing, but the semantic description of films and films is still very poor. The Multimedia Ontologies discussed in Section 3 above do not include many aspects of film production, such as the geospatial location of the film or the various roles played by each person during the production process. A significant study conducted [38] demonstrates how linguistic diversity is increasingly influencing cinematic productions. In the case of Morocco, which has 79% of its films in Arabic, 17% in French, and 4% in English, the UNESCO Statistic branch [39] demonstrates that over 50% of films made have a main language. On the other hand, there is no dominant language in Indian cinema. India has a more diverse film industry thanks to the 16.1% of their films that are produced in Hindi, 15.3% in Telougou, 14.7% in Tampul, 11% in Kannada, and 10% in Bengale. The research on multimedia ontologies does not cover all of these limitations.

The Semantic Web could provide more functionality for the cinematographic domain when it comes to accessing or retrieving information. The target audience, the language used in the film, the events or awards to which it has already been nominated, the context of the film's production (which varies from continent to continent), the relationship between actor names and socio-cultural origins, and subtitling, for example, can further enrich the semantic annotation of film products (see use-case illustration in Figure 4). For instance, names (such as Jet Li, Siriki, and Arnold Schwarzenegger) or locations (such as the Tour Eiffel or landmarks) that may disclose an actor's ancestry are not yet taken into account in ontological annotations on films.

Multimedia annotation has gained importance since it attempts to give the contents of films various semantic and visual aspects. It is very challenging to analyze and retrieve content-based cinematographic works, drawing on the semantics attached to the backbone multimedia ontology, as it needs to consider the semantics of utilized vocabularies a particular category of multimedia resources.

In this paper, we attempted to examine the semantic properties that current multimedia ontologies took into consideration. A comparative study was carried out with the aim of evaluating the vocabulary taken into account in relation to the annotation of filmmaking or cinematographic works. Films frequently show a degree of cultural identity, which may also be annotated. Since these film products are connected to the actual world, and because previous studies presented films in a content-based perspective, another intriguing objective may be to depict the cultural semantics of film notions. By suggesting vocabulary to expand the Linked Open Vocabularies and hence improve information retrieval of this content on the Web of Data, we could come closer to capturing these socio-cultural truths.

Building content-based semantics to address socio-cultural knowledge found in film products and proposing vocabulary to address them on the LOV may be of interest. Another intriguing angle is to model film theory in order to illustrate the process of film making.