The effects of temporal synchrony and visual material on audio-visual integration

Research output: Chapter in Book/Report/Conference proceedingConference contributionProfessional

Abstract

Associating letters with speech sounds is essential when learning to read. Recent studies on audio-visual integration suggested that 1) the integration of letter and speech sound takes place automatically and 2) that temporal synchrony seems to be critical for the integration. In the current study, we wished to determine the neural networks associated with an automatic integration of written and heard syllables in literate adults. The mismatch negativity (MMN) was recorded in a new multi-feature paradigm whereby the integration of several speech sounds with letters can be addressed within one experimental session. Subjects were presented with auditory stimuli (consonant-vowel syllables) together with visual stimuli, which differed between conditions. The auditory stimuli included vowel or consonant changes, or changes in syllable intensity, frequency, or vowel length. In the experimental condition the visual stimuli were written syllables, whereas in the control condition, they were scrambled images of the written syllables. The visual stimuli were presented in half of the blocks synchronously with the auditory stimuli and in the other half with a time delay. During the stimulation, the subjects pressed a button to designated visual targets. Changes in auditory stimuli elicited a MMN/N2 response, in all conditions. The topography of this response was different in the stimulus blocks including syllables than in those including scrambled images. Furthermore, topographic differences were observed between the conditions with and without the time delay. The results indicated that different neural networks are activated when speech stimuli are presented together with linguistic than with non-linguistic visual stimuli and that this effect seems to be influenced by temporal synchrony.
Original languageEnglish
Title of host publicationFrontiers Human Neuroscience Conference Abstract : MMN 09 Fifth Conference on Mismatch Negativity (MMN) and its Clinical and Scientific Applications
Number of pages1
Place of PublicationEpalinges
PublisherFrontiers Media S.A.
Publication date26 Mar 2009
DOIs
Publication statusPublished - 26 Mar 2009
MoE publication typeD3 Professional conference proceedings
EventConference on Mismatch Negativity (MMN) and its Clinical and Scientific Applications (MMN 09) - Budapest, Hungary
Duration: 4 Apr 20097 Apr 2009
Conference number: 5th

Fields of Science

  • 515 Psychology

Cite this

Mittag, M., Takegata, R., & Kujala, T. (2009). The effects of temporal synchrony and visual material on audio-visual integration. In Frontiers Human Neuroscience Conference Abstract: MMN 09 Fifth Conference on Mismatch Negativity (MMN) and its Clinical and Scientific Applications Epalinges: Frontiers Media S.A.. https://doi.org/10.3389/conf.neuro.09.2009.05.145
Mittag, Maria ; Takegata, Rika ; Kujala, Teija. / The effects of temporal synchrony and visual material on audio-visual integration. Frontiers Human Neuroscience Conference Abstract: MMN 09 Fifth Conference on Mismatch Negativity (MMN) and its Clinical and Scientific Applications. Epalinges : Frontiers Media S.A., 2009.
@inproceedings{ce349243ee834499baf379b03f47df25,
title = "The effects of temporal synchrony and visual material on audio-visual integration",
abstract = "Associating letters with speech sounds is essential when learning to read. Recent studies on audio-visual integration suggested that 1) the integration of letter and speech sound takes place automatically and 2) that temporal synchrony seems to be critical for the integration. In the current study, we wished to determine the neural networks associated with an automatic integration of written and heard syllables in literate adults. The mismatch negativity (MMN) was recorded in a new multi-feature paradigm whereby the integration of several speech sounds with letters can be addressed within one experimental session. Subjects were presented with auditory stimuli (consonant-vowel syllables) together with visual stimuli, which differed between conditions. The auditory stimuli included vowel or consonant changes, or changes in syllable intensity, frequency, or vowel length. In the experimental condition the visual stimuli were written syllables, whereas in the control condition, they were scrambled images of the written syllables. The visual stimuli were presented in half of the blocks synchronously with the auditory stimuli and in the other half with a time delay. During the stimulation, the subjects pressed a button to designated visual targets. Changes in auditory stimuli elicited a MMN/N2 response, in all conditions. The topography of this response was different in the stimulus blocks including syllables than in those including scrambled images. Furthermore, topographic differences were observed between the conditions with and without the time delay. The results indicated that different neural networks are activated when speech stimuli are presented together with linguistic than with non-linguistic visual stimuli and that this effect seems to be influenced by temporal synchrony.",
keywords = "515 Psychology",
author = "Maria Mittag and Rika Takegata and Teija Kujala",
note = "Volume: Proceeding volume:",
year = "2009",
month = "3",
day = "26",
doi = "10.3389/conf.neuro.09.2009.05.145",
language = "English",
booktitle = "Frontiers Human Neuroscience Conference Abstract",
publisher = "Frontiers Media S.A.",
address = "Switzerland",

}

Mittag, M, Takegata, R & Kujala, T 2009, The effects of temporal synchrony and visual material on audio-visual integration. in Frontiers Human Neuroscience Conference Abstract: MMN 09 Fifth Conference on Mismatch Negativity (MMN) and its Clinical and Scientific Applications. Frontiers Media S.A., Epalinges, Conference on Mismatch Negativity (MMN) and its Clinical and Scientific Applications (MMN 09), Budapest, Hungary, 04/04/2009. https://doi.org/10.3389/conf.neuro.09.2009.05.145

The effects of temporal synchrony and visual material on audio-visual integration. / Mittag, Maria; Takegata, Rika; Kujala, Teija.

Frontiers Human Neuroscience Conference Abstract: MMN 09 Fifth Conference on Mismatch Negativity (MMN) and its Clinical and Scientific Applications. Epalinges : Frontiers Media S.A., 2009.

Research output: Chapter in Book/Report/Conference proceedingConference contributionProfessional

TY - GEN

T1 - The effects of temporal synchrony and visual material on audio-visual integration

AU - Mittag, Maria

AU - Takegata, Rika

AU - Kujala, Teija

N1 - Volume: Proceeding volume:

PY - 2009/3/26

Y1 - 2009/3/26

N2 - Associating letters with speech sounds is essential when learning to read. Recent studies on audio-visual integration suggested that 1) the integration of letter and speech sound takes place automatically and 2) that temporal synchrony seems to be critical for the integration. In the current study, we wished to determine the neural networks associated with an automatic integration of written and heard syllables in literate adults. The mismatch negativity (MMN) was recorded in a new multi-feature paradigm whereby the integration of several speech sounds with letters can be addressed within one experimental session. Subjects were presented with auditory stimuli (consonant-vowel syllables) together with visual stimuli, which differed between conditions. The auditory stimuli included vowel or consonant changes, or changes in syllable intensity, frequency, or vowel length. In the experimental condition the visual stimuli were written syllables, whereas in the control condition, they were scrambled images of the written syllables. The visual stimuli were presented in half of the blocks synchronously with the auditory stimuli and in the other half with a time delay. During the stimulation, the subjects pressed a button to designated visual targets. Changes in auditory stimuli elicited a MMN/N2 response, in all conditions. The topography of this response was different in the stimulus blocks including syllables than in those including scrambled images. Furthermore, topographic differences were observed between the conditions with and without the time delay. The results indicated that different neural networks are activated when speech stimuli are presented together with linguistic than with non-linguistic visual stimuli and that this effect seems to be influenced by temporal synchrony.

AB - Associating letters with speech sounds is essential when learning to read. Recent studies on audio-visual integration suggested that 1) the integration of letter and speech sound takes place automatically and 2) that temporal synchrony seems to be critical for the integration. In the current study, we wished to determine the neural networks associated with an automatic integration of written and heard syllables in literate adults. The mismatch negativity (MMN) was recorded in a new multi-feature paradigm whereby the integration of several speech sounds with letters can be addressed within one experimental session. Subjects were presented with auditory stimuli (consonant-vowel syllables) together with visual stimuli, which differed between conditions. The auditory stimuli included vowel or consonant changes, or changes in syllable intensity, frequency, or vowel length. In the experimental condition the visual stimuli were written syllables, whereas in the control condition, they were scrambled images of the written syllables. The visual stimuli were presented in half of the blocks synchronously with the auditory stimuli and in the other half with a time delay. During the stimulation, the subjects pressed a button to designated visual targets. Changes in auditory stimuli elicited a MMN/N2 response, in all conditions. The topography of this response was different in the stimulus blocks including syllables than in those including scrambled images. Furthermore, topographic differences were observed between the conditions with and without the time delay. The results indicated that different neural networks are activated when speech stimuli are presented together with linguistic than with non-linguistic visual stimuli and that this effect seems to be influenced by temporal synchrony.

KW - 515 Psychology

U2 - 10.3389/conf.neuro.09.2009.05.145

DO - 10.3389/conf.neuro.09.2009.05.145

M3 - Conference contribution

BT - Frontiers Human Neuroscience Conference Abstract

PB - Frontiers Media S.A.

CY - Epalinges

ER -

Mittag M, Takegata R, Kujala T. The effects of temporal synchrony and visual material on audio-visual integration. In Frontiers Human Neuroscience Conference Abstract: MMN 09 Fifth Conference on Mismatch Negativity (MMN) and its Clinical and Scientific Applications. Epalinges: Frontiers Media S.A. 2009 https://doi.org/10.3389/conf.neuro.09.2009.05.145