Modelling the structure of a multimodal artefact

Tutkimustuotos: OpinnäyteVäitöskirjaMonografia

Kuvaus

This dissertation studied the structure of multimodal artefacts, or how language,
image and other semiotic modes combine and interact in documents. This places
the study within the emerging field of multimodal research, which uses linguistic
methods to study the interaction of multiple semiotic modes.

Despite the growing amount of multimodal research, the structure of multimodal
artefacts has not received the attention it warrants. Previous studies have
been either very detailed or exceedingly abstract, leaving a significant gap between data and theory, which this dissertation attempted to bridge. To do so, the dissertation adopted a data-driven approach to multimodal analysis, addressing the structure of multimodal artefacts, the factors that shape the artefact structure, and the role of structure in the recognition and interpretation of the artefacts.

The data consisted of tourist brochures produced by the city of Helsinki between
1967 and 2008, which allowed a longitudinal perspective to their multimodal
structure. A total of 58 double-pages were annotated for their content, visual appearance, layout and rhetorical organisation, and compiled into an XML-based
multimodal corpus. To study the corpus, the dissertation developed visualisation
methods that combined information from multiple analytical layers of the corpus
to represent the multimodal structures in the data.

The study revealed the functional motivation behind the structure of the tourist
brochures, identifying patterns in their hierarchical and rhetorical organisation,
which were used to fulfil specific communicative tasks. The configuration of these
patterns, in turn, signalled how the brochure was to be interpreted. The results
also showed that after the year 1985, which marked the introduction of desktop
publishing software, the organising principles of the tourist brochures have shifted towards a more fragmented and non-linear structure.
Alkuperäiskielienglanti
JulkaisupaikkaHelsinki
Kustantaja
Painoksen ISBN978-952-10-9426-2
Sähköinen ISBN978-952-10-9427-9
TilaJulkaistu - 2013
OKM-julkaisutyyppiG4 Tohtorinväitöskirja (monografia)

Tieteenalat

  • 6121 Kielitieteet

Lainaa tätä

Hiippala, T. (2013). Modelling the structure of a multimodal artefact. Helsinki: University of Helsinki.
Hiippala, Tuomo. / Modelling the structure of a multimodal artefact. Helsinki : University of Helsinki, 2013. 247 Sivumäärä
@phdthesis{afaf7c81aea44b5f9bafa5498757f508,
title = "Modelling the structure of a multimodal artefact",
abstract = "This dissertation studied the structure of multimodal artefacts, or how language,image and other semiotic modes combine and interact in documents. This placesthe study within the emerging field of multimodal research, which uses linguisticmethods to study the interaction of multiple semiotic modes.Despite the growing amount of multimodal research, the structure of multimodalartefacts has not received the attention it warrants. Previous studies havebeen either very detailed or exceedingly abstract, leaving a significant gap between data and theory, which this dissertation attempted to bridge. To do so, the dissertation adopted a data-driven approach to multimodal analysis, addressing the structure of multimodal artefacts, the factors that shape the artefact structure, and the role of structure in the recognition and interpretation of the artefacts.The data consisted of tourist brochures produced by the city of Helsinki between1967 and 2008, which allowed a longitudinal perspective to their multimodalstructure. A total of 58 double-pages were annotated for their content, visual appearance, layout and rhetorical organisation, and compiled into an XML-basedmultimodal corpus. To study the corpus, the dissertation developed visualisationmethods that combined information from multiple analytical layers of the corpusto represent the multimodal structures in the data.The study revealed the functional motivation behind the structure of the touristbrochures, identifying patterns in their hierarchical and rhetorical organisation,which were used to fulfil specific communicative tasks. The configuration of thesepatterns, in turn, signalled how the brochure was to be interpreted. The resultsalso showed that after the year 1985, which marked the introduction of desktoppublishing software, the organising principles of the tourist brochures have shifted towards a more fragmented and non-linear structure.",
keywords = "6121 Languages, multimodality, corpora, tourism, visual communication, print media",
author = "Tuomo Hiippala",
year = "2013",
language = "English",
isbn = "978-952-10-9426-2",
publisher = "University of Helsinki",
address = "Finland",

}

Modelling the structure of a multimodal artefact. / Hiippala, Tuomo.

Helsinki : University of Helsinki, 2013. 247 s.

Tutkimustuotos: OpinnäyteVäitöskirjaMonografia

TY - THES

T1 - Modelling the structure of a multimodal artefact

AU - Hiippala, Tuomo

PY - 2013

Y1 - 2013

N2 - This dissertation studied the structure of multimodal artefacts, or how language,image and other semiotic modes combine and interact in documents. This placesthe study within the emerging field of multimodal research, which uses linguisticmethods to study the interaction of multiple semiotic modes.Despite the growing amount of multimodal research, the structure of multimodalartefacts has not received the attention it warrants. Previous studies havebeen either very detailed or exceedingly abstract, leaving a significant gap between data and theory, which this dissertation attempted to bridge. To do so, the dissertation adopted a data-driven approach to multimodal analysis, addressing the structure of multimodal artefacts, the factors that shape the artefact structure, and the role of structure in the recognition and interpretation of the artefacts.The data consisted of tourist brochures produced by the city of Helsinki between1967 and 2008, which allowed a longitudinal perspective to their multimodalstructure. A total of 58 double-pages were annotated for their content, visual appearance, layout and rhetorical organisation, and compiled into an XML-basedmultimodal corpus. To study the corpus, the dissertation developed visualisationmethods that combined information from multiple analytical layers of the corpusto represent the multimodal structures in the data.The study revealed the functional motivation behind the structure of the touristbrochures, identifying patterns in their hierarchical and rhetorical organisation,which were used to fulfil specific communicative tasks. The configuration of thesepatterns, in turn, signalled how the brochure was to be interpreted. The resultsalso showed that after the year 1985, which marked the introduction of desktoppublishing software, the organising principles of the tourist brochures have shifted towards a more fragmented and non-linear structure.

AB - This dissertation studied the structure of multimodal artefacts, or how language,image and other semiotic modes combine and interact in documents. This placesthe study within the emerging field of multimodal research, which uses linguisticmethods to study the interaction of multiple semiotic modes.Despite the growing amount of multimodal research, the structure of multimodalartefacts has not received the attention it warrants. Previous studies havebeen either very detailed or exceedingly abstract, leaving a significant gap between data and theory, which this dissertation attempted to bridge. To do so, the dissertation adopted a data-driven approach to multimodal analysis, addressing the structure of multimodal artefacts, the factors that shape the artefact structure, and the role of structure in the recognition and interpretation of the artefacts.The data consisted of tourist brochures produced by the city of Helsinki between1967 and 2008, which allowed a longitudinal perspective to their multimodalstructure. A total of 58 double-pages were annotated for their content, visual appearance, layout and rhetorical organisation, and compiled into an XML-basedmultimodal corpus. To study the corpus, the dissertation developed visualisationmethods that combined information from multiple analytical layers of the corpusto represent the multimodal structures in the data.The study revealed the functional motivation behind the structure of the touristbrochures, identifying patterns in their hierarchical and rhetorical organisation,which were used to fulfil specific communicative tasks. The configuration of thesepatterns, in turn, signalled how the brochure was to be interpreted. The resultsalso showed that after the year 1985, which marked the introduction of desktoppublishing software, the organising principles of the tourist brochures have shifted towards a more fragmented and non-linear structure.

KW - 6121 Languages

KW - multimodality

KW - corpora

KW - tourism

KW - visual communication

KW - print media

M3 - Doctoral Thesis

SN - 978-952-10-9426-2

PB - University of Helsinki

CY - Helsinki

ER -

Hiippala T. Modelling the structure of a multimodal artefact. Helsinki: University of Helsinki, 2013. 247 s.