Xiaoning Chen
Abstract
From the socio-semiotic perspective, this paper intends to examine the impact of technology on multimodal design of digital picture books. The study analyzes 26 digital picture books completed by pre-service teachers in a Literacy Methods course, and identifies three key categories, which are static multimodal semiotic resources, dynamic multimodal semiotic resources, and usability features, that reshape the elements of written language, visual image, design, and the overall reading experience of digital books. This study has research and pedagogical implications. The results of the study contribute to the limited understanding of how digital texts emerge as a new form of visual literacy and how technology has a significant impact on the multimodal design and meaning making process. The study also appeals to educators as it has a close connection to the Common Core State Standards (2011), which places a strong emphasis on students’ abilities to recognize, interpret, and evaluate the information represented through diverse media and formats. Effective literacy strategies to support children’s interpretations of multimodal digital texts are discussed to support the construction of a dynamic and active reader’s role in navigating “new textual landscapes” (Carrington, 2005).
Introduction
JJ is a five-year-old kindergartener. At school, he uses iPads and the SMART board for his daily learning. At home, he completes his homework on the school-subscribed websites so his progress is documented and monitored electronically by his teachers. At leisure time, he turns on the TV to watch his favorite show on the PBS Kids channel. He also enjoys snapping a picture of himself doing something silly using his parents’ smart phone, creating a funny Chatterbox (an iPad app) using pictures and recorded voice, or video chatting with his grandparents and sending them lots of stickers and messages to express his love.
The above scenario is a typical day for JJ and many children growing up in today’s digital world. Imagine how much time they have spent interacting with different digital devices at such a young age! As a result, it is imperative to recognize the dramatically changing landscape of children’s reading and early literacy development (Meyers, Zaminpaima, & Frederico, 2014). “Language arts education can no longer ignore the way that our social, cultural, and economic worlds now require facility with texts and practices involving the full range of representational modes” (Seigel, 2006, p. 65). Digital texts, emerging as a new genre of visual literacy, provide one of the many choices that we can bring into classrooms to enrich our conversations with children about how multiple representations are integrated to create meaning.
There are many different types of digital books. However, one common element is that they are using technology to transform the traditional print-based books. The purpose of this study is to explore the impact of technology on multimodal designs of digital picture books created by pre-service teachers. The outcome is twofold. On one hand, it is important to recognize the newly emerged multimodal semiotic resources brought forward by pre-service teachers, who are considered “digital natives” (Prensky, 2001). On the other hand, the multimodal design embedded in the digital picture books calls for the development of pedagogical strategies in order to prepare young children’s exploration, interpretation, and creation of the new form of visual literacy.
Significance of the Study
Even though digital texts have been around for over a decade, there are limited studies that take an in-depth look at the dynamic semiotic resources presented in digital texts. Studies have detailed how to read and analyze contemporary picture books in print (e.g., Hassett & Curwood, 2009; Lewis, 2001; Serafini, 2012; Serafini & Clausen, 2012). A number of studies have been focused on how young readers interact with digital technologies (e.g., Chiong, Ree, Takeuchi, & Erickson, 2012; Gasparini, 2012; Gutnick, Robb, Takeuchi, & Kotler, 2010; Vaala & Takeuchi, 2012), or how to evaluate book apps (Bird, 2011; Meyers et al., 2014). However, a theoretical framework to understand and analyze how technology impacts multimodal design in digital picture books has been lacking. The intention of this study is to contribute to the development of such a multimodal framework. Moreover, this study is significant in the field of teacher education. Through the examination of these digital picture books created by pre-service teachers who are going to be the main taskforce that moves technology to the center of school curriculum, it is important to find out what future teachers know and how prepared they are to engage in conversations with children about effectively using multimodal resources in designing digital books.
Theoretical Framework
The multimodal socio-semiotic theory and related research provides a theoretical framework for this study (Halliday, 1978; Kress, 2003; Kress & van Leeuwen, 1996, 2001). This theoretical perspective is relevant and powerful as it focuses on describing and analyzing socially organized resources that people use to make meaning (Bezemer, Diamantopoulou, Jewitt, Kress, & Mavers, 2012).
A mode is defined by Bezemer and Kress (2008) as a “socially and culturally shaped resource for meaning making” (p. 171). Drawing is one of the modes that children often utilize to express meaning. A drawing of a heart, for instance, is a socially and culturally recognized sign to express the meaning of love. In today’s fast changing world, meaning is typically communicated and represented through the integration of multiple semiotic resources such as writing texts, visual images, screens, three-dimensional forms, audios, and videos (Bezemer, 2012). The multimodal socio-semiotic theory challenges the traditional view of prioritizing the linguistic mode and shifts our attention to understand how the meaning is created through the integration of all modes (Bezemer et al., 2012; Hassett & Curwood, 2009).
The production of meaning is influenced by materials and social contexts (Halliday, 1978; Halliday & Hason, 1985). In a writing activity, for example, the product is dictated by the accessible materials (paper and pencil, or keyboard and mouse) and how the activity is defined within the context (the social purpose). Similarly, modes have different resources and affordances (Bezemer & Kress, 2008; Bezemer, 2012). Skaar (2009) explains the distinct affordances of modes (Kress, 2003) as the creation of a sign that is “always a result of an interplay between what we intend to express and what we succeed in expressing with the resources at our disposal” (p. 38). In other words, mode affordance refers to people making a norm-based selection of a most appropriate mode for a socially, culturally, and historically defined context (Bezemer, 2012)
In the context of multimedia and technology, a digital text is “a multimodal design where meaning is created through the interplay between different modes” (Skaar, 2009, p. 37). The creators of digital texts have access to multiple semiotic resources enhanced by technology; however, they are also bound by social and cultural norms about whether meaning is communicated exactly as intended and reasonably for the readers or viewers. Therefore, informed by the multimodal socio-semiotic perspective, how meaning is created through the interplay of text (language) and non-print representations in a digital text is the central focus of this study.
Methods
This is a qualitative inquiry into the impact of technology on multimodal designs of digital picture books. The data for the study were 26 completed digital picture books created by pre-service teachers who were taking a literacy methods course in a public four-year college located in the Northeastern part of the United States. The pre-service teachers were predominantly middle class Caucasian females enrolled in the elementary education program. They were at the junior standing at the time of the study.
The instructor-researcher introduced the project of creating digital picture books as a course assignment. The procedure included multiple steps. First, the students were asked to examine several picture books authored by Ann Morris. Ann Morris’s books were chosen as the mentor texts because of the use of realistic photos as illustrations, simplistic language, descriptive structure, and the multicultural theme. The titles included: Bread, Bread, Bread; Houses and Homes; Tools; On the Go; Families; Hats, Hats, Hats; and Shoes, Shoes, Shoes. Further, the instructor modeled technology considerations in regard to supporting struggling readers and English Language Learners (ELLs). One example of previously made digital picture books was shared to establish the expectation for the project. The students worked in small groups of three or four to brainstorm ideas for creating their own digital picture books first by choosing an age group for the audience and considering the availability of digital photos they were able to obtain from the campus surroundings. Next, they worked together to snap digital photos, draft a digital picture book, and plan several lesson ideas while providing accommodations for a diverse classroom with multiple English Language Learners and struggling readers. Finally, they presented their digital books in class and reflected on their authoring experience.
Two class sessions (each lasting about two hours and twenty minutes) were spent on the project. Students were able to use any software to create their digital book. Occasionally, they were allowed to use a few non-copyrighted images from the internet as needed, due to the content of their book. Many groups spent extra time outside of class to complete their project. The software used for the project included: Microsoft PowerPoint, iMovie (Mac), Movie Maker (PC), and Prezi. The majority of the targeted audience was primary grade (K-2) students.
The data collection occurred across two semesters (Spring and Fall, 2011), with a total of 26 completed digital books created by 94 pre-service teachers (Spring semester with 41 and Fall semester with 53 students).
While examining the digital picture books, the researcher developed two questions to guide the study: 1) What multimodal resources did pre-service teachers use to design digital picture books? and 2) How did the multimodal design impact the meaning making process of the digital picture books?
Data analysis
The researcher analyzed the digital texts by surveying the semiotic resources used to create meaning, followed by an analysis of the impact of such resources on the multimodal design and meaning as a whole. Serafini (2012) discussed three major semiotic resources in designing and analyzing print picture books: written language, visual images, and design elements. In their analysis of book apps, Meyers et al. (2014) adapted Lewis’s (2001) ecology of picture books and created an expanded ecology that includes textual features, multimodal features, and usability features. Adapting key elements from Serafini (2012) and Meyers et al. (2014), the researcher intended to use the emerging categories from data to create an analysis framework.
Data were categorized and coded to generate emerging themes (Merriam, 1988). Specifically, a list of semiotic resources was developed after thoroughly reading through the 26 digital books several times. The initially coded semiotic resources included: fonts, colors, sizes, slant, weights, animations, video, audio, and hyperlinks, all of which had a direct impact on the presentation of written language, visual images, and overall design. Noticing that some of these resources were often referenced in some contemporary children’s books in print while others were more unique to digital texts, two major categories, namely static multimodal resources and dynamic multimodal resources, were developed. Furthermore, the usability features, as discussed in Meyers et al. (2014), were also noted as significant for designing digital texts. The data analysis framework is shown in Figure 1.
Results
Multimodal Resources
Data analysis identified an interwoven display of a variety of semiotic resources from both static and dynamic categories in the digital texts (see Table 1). These semiotic resources include both written text (e.g., colors, fonts, sizes, weights, slant, positioning, and text animations), visual images (e.g., positioning, sizes, framing, backgrounds, and image animations), audio (sound and narration), video, and hyperlinks (e.g., internal and external links on text and image).
| Semiotic Resources | Static Resources | Dynamic Resources |
|---|---|---|
| Written Text: Typographical elements | Written text that doesn’t change form (e.g., fonts, colors, weights, sizes, slant, and positioning) | Written text with animations (e.g., colors, fonts, weights, sizes, directionality, appearances, and special effects) |
| Visual Image | Visual images that don’t change form (e.g., sizes, patterns, lines, shapes, colors, texture, composition, positioning, and backgrounds) | Visual images with animations (e.g., sizes, frames, appearances, and animated backgrounds) |
| Audio | N/A | Sound clips or narrations of the text; audio on click or automatic play |
| Video | N/A | Video clips |
| Hyperlink: Hypertext/Hyperimage | N/A | Internal (links to other parts in the digital book); external (e.g., links to web resources such as videos, songs, dictionaries, or games) |
| Design Elements | Text and image appearing at the same time (T/I); page turning | Text and image appearing in different sequences (e.g., T-I; I-T; T/I); text and image grouped to present partially or as a whole; text and image merging; and page turning (e.g., transition in between pages) |
Table 1: Semiotic Resources
Static Multimodal Resources
The study found that the digital books created by pre-service teachers contained many of the static multimodal elements used in print-based picture books. For example, when examining the written text, the researcher identified noticeable typographical elements. Typography, being one of the semiotic resources, is defined as “the art and technique of arranging type to make language visible” (Serafini & Calusen, 2012, p. 4). The typographical elements presented in the digital books included weights, colors, sizes, slant, framing, formality, and flourishes. With regards to visual and design elements (Kress & Van Leeuwen, 1996), patterns, lines, shapes, colors, and texture were noted in the multimodal texts.
For example, Figures 2, 3, and 4 were pages from a digital book titled Art. In Figure 2, two different fonts were chosen to correlate with the “serious” or “funny” mood evoked by different art forms. In Figure 3, the creative use of boundaries (the lines that mark the blue area on the top right corner of the page) demonstrated a differentiation between art that is inside versus art that is outside.
In Figure 4, the color and texture of the words “clay” and “metal” were purposefully chosen to highlight the materials from which sculptures can be made. Figure 5 was taken from a digital book titled Let’s Fall into Fall! In this example, color was used meaningfully to show how leaves change as the season of Fall is approaching.
The static multimodal elements were found to be used commonly in the digital books. Many of these static multimodal elements were analyzed in contemporary picture books (Hassett & Curwood, 2009; Serafini, 2011, 2012; Serafini & Clausen, 2012).
Dynamic Multimodal Resources
Most significantly, the digital picture books utilized many dynamic multimodal elements that did not exist in traditional print-based picture books. These multimodal elements included animations, hyperlinks (hypertext/hyperimage), audio (sound clips or narrations), and video. These new multimodal elements enabled the authors to create design elements that were unique to the digital picture books. For example, the digital books grouped text and image, designed the sequential order of how and when text and image would appear/disappear on the page, merged visual and written text, used many multimodal backgrounds, and cross-referenced a variety of resources through hyperlinking them together.
In Figure 6, for instance, the materials on the page were sequenced in multiple layers. The first line of the text (except the word “hang”) appeared together with the two images. The word “hang” appeared afterwards, slanted in a way as if it were a piece of art that was hanging on the wall. The second line of the text on the bottom appeared next with the exception of the word “TALL.” This word was set in all capital letters and animated to grow from the bottom of the page to stand up much taller than the rest of the text.
Figure 7 illustrates how the text and image could be merged to create meaning, and text could be grouped with images to sequentially depict parts of a page. The first view of the page showed two incomplete phrases: “Some signs are…” [on the left], and “some signs are…” [on the right]. With another click on the mouse, the word “outside” and the visual image of a sun and a tree were merged and grouped together with the image on the left side. On the third click, the word “inside” framed in an image of a house appeared together with the inside sign on the right.
Figures 8 and 9 are samples from the digital book titled Recycling for All: Big and Small. The cues “Click on me for a song/fun video” on the pages sent an active invitation for readers to explore more resources related to the topic. The hyperlinks were hidden behind the images to provide a “magic-click” experience.
Internal hyperlinks embedded behind the star to take readers to the definition of the key vocabulary.
Figures 10 and 11 are screenshots from the digital book titled Hair, Hair, Hair. They demonstrate how different dynamic semiotic resources such as text animations (e.g., twisting and curling), animated backgrounds (e.g., with waving water), and hyperlinks worked together in action.
Even though this paper presents static and dynamic semiotic resources in separate categories, it by no means suggests that one category is better than or superior to the other. Moreover, it is the intertwined design using both static and dynamic semiotic resources that makes the whole presentation unique in digital picture books. With the use of technology, both static and dynamic semiotic resources can be employed to enhance authors’ semiotic repertoire to express meaning.
Usability Features
With regard to the usability features in the digital picture books, Table 2 is a comparison of print-based to digital books that highlights the advantages as well as the challenges a reader encounters while using digital texts.
| Usability Features | Print (represented by Ann Morris’s books) | Digital |
|---|---|---|
| Access | Physical copy | Computer or tablet with Internet connection |
| Interface | Page | Screen |
| Sequence | Lineal & sequential with temporal sequence | Spatial sequence (in particular arrangements) |
| Navigation | Page turning | On click or automatically play once open |
| Safety & Privacy | Should not be a major concern as the content is limited inside the covers of the book | Could be a major concern as the content is above and beyond what’s inside the file, for instance, all the hyperlinks connected to other websites (e.g., Google and Youtube) |
| Stability | Permanent | Not stable; links may become outdated or invalid |
| Customizability | Generally rather limited | Can be easily customized |
Digital texts can be easily accessed through file sharing. Multiple copies can be duplicated without investing in the traditional material resources. In addition, the content of the digital books can be customized rather conveniently. However, a more costly investment of a computer/SMART device and Internet connection must be guaranteed. While readers navigate through digital texts, the interface is on the screen rather than on the page. Unlike print books, digital texts usually show a nonlinear, spatial sequence (Kress, 2003) through the design elements such as transitions and hyperlinks. While these features create an exciting reading experience, it presents challenges for readers to stay focused on the main content and not be distracted by other non-content related features (Chiong et al., 2012). Safety and privacy may become a major concern (Meyers et al., 2014) as the digital nature extends beyond the boundaries of the digital file. At the same time, clicking on a non-active hyperlink can be a frustrating experience for readers. The concern about the stability of content in digital materials is valid and in a sense, directly relates to the evaluation of digital text quality.
Impact of Multimodal Design on Meaning Making Process
By examining all the formats and varieties of the 26 digital books, the researcher found that there were four major aspects in the multimodal design that had an impact on the meaning making process. These aspects include: interactivity, choices, access to additional resources, and varied effective use of semiotic resources.
Interactivity
Even though some print books contain the feature of interactivity by posing questions throughout the reading, digital books, with the multilayered presentation of different semiotic resources, can serve as a facilitator to encourage readers’ self-check and thereby enhance comprehension.
Figures 12 and 13 are screenshots from the digital book titled Signs, Signs, Everywhere Signs. In these two figures, the background highlighting a road in the middle was consistently used throughout the book. What’s more interesting was the multilayered presentation of content. The book had a fictional figure named “Sammy” and it was through Sammy that the functions of various road signs were introduced through interactive questions. In Figure 12, a question about the job for the stop sign was raised.
In Figure 13, the answer was provided before a follow-up question was asked to encourage readers to self-check for comprehension.
Choices
Choice is by far the most powerful factor affecting readers and their meaning-making process. Due to the spatial sequence embedded in the digital texts, each reader’s reading experience differs based on his or her interest and choices.
For example, Figure 14 is the first content page after the title page in the digital book Power of Communication. The presence of multiple images and the simple text “Pick a picture!” marked in red is a call to explore the book in any way a reader prefers. Which picture seems most interesting? Which one shows a relatable experience to the reader? Ultimately, the authors design a digital book but it is the reader who decides how she or he reads it and in what order.
Access to Additional Resources
While some print books usually provide additional resources at the end of the book as paratextual features (Serafini, 2012), the digital books in this study showed that additional resources can be linked to the content anywhere in the book. The content and format of additional resources are up to the authors to design. Yet the readers have the final say in whether to pursue the information and how in-depth they want to go.
Figures 8 and 9, as presented earlier, show examples of additional outside resources digital pictures books could provide for readers to interact with during reading. Figures 11 (previously shown) and 15, on the contrary, are examples of internal links and resources embedded in the digital book titled Hair, Hair, Hair. After all the information was shown on the page, the word “wavy” changed to a bright orange color as a highlight, and the yellow star on the top right hand corner was an implicit hyperlink that took readers to the glossary page where the word was defined. Readers could use the other hyperlink shown on the bottom of Figure 15 to get back to the page in which they left off.
Effective Use of Multimodal Resources
From a micro and macro perspective, the researcher examined whether the use of multimodal design supported the meaning on the page as well as the overall meaning of the book.
Figure 16 is from the digital book titled Let’s Fall into Fall. The purposeful use of a wider image of night and narrower image of day correlated well with the appearance of the words “longer” and “shorter” in text. This was deemed an effective use of multimodal resources in text features and visual images to support local and overall meaning of the book. There was another semiotic resource of a sound clip that was shown in the symbol of a microphone on the right lower part of the page. By clicking on the symbol, a reader hears the sound of a cricket chirping. It seemed that the audio resource was a nice addition to the multimodal design, yet readers may be left wondering how the cricket sound related to the lengths of day and night and the season of fall. This was an example of ineffective use of multimodal resources.
Furthermore, some of the semiotic resources, such as hyperlinks, were not explicitly specified for readers. On one hand, reading digital books created a discovery process, in which readers could find new and exciting features along the way. On the other hand, however, the intended meaning and efforts to make meaning comprehensible may have been neglected or lost due to the way the reader chose to approach the book without explicit instruction.
In addition, one of the challenges the researcher faced was that some multimodal semiotic resources malfunction in the transferring process. For instance, some digital books came with narration but when they were played on a different device, the narration did not work. Another challenge was that the researcher could not determine how some of the additional resources supported the meaning of the book because those hyperlinks were no longer valid.
Discussion & Implications
This research project, through a socio-semiotic perspective, examined semiotic resources that were utilized in the 26 digital picture books created by pre-service teachers in a Literacy Methods course. The results identified semiotic resources and usability features that are unique to the multimodal design of digital texts. The semiotic resources included typographical features such as fonts, sizes, colors, and weights, visual images, animations, hyperlinks, audio, and video. The usability features of digital texts are found to be rather different than those of print books. Through clicking on a mouse or touchscreen, readers navigate through digital books on a computer or a SMART device screen. Information is likely to be presented via a spatial sequence, and hyperlinked to a variety of resources inside and outside the boundaries of the book. Due to the digital nature, the resources may be rather unstable. In addition, readers may unintentionally stumble upon some developmentally inappropriate content via the links on the outside resources, creating threats to safety and privacy to young readers. However, the digital books are highly engaging and customizable, gaining an unarguable place in today’s classrooms full of curious and diverse readers.
The variety of semiotic resources, both static and dynamic, together with the usability features, have a significant impact on the way a reader approaches and interacts with a digital book. First of all, readers’ roles need to be refined from a passive follower to an active meaning co-constructor (Serafini, 2012). With a digital text, readers embark on a journey full of new findings. For instance, they may come across hyperlinks clearly marked and discover some hidden ones. Readers have different choices as to how they want to read and interpret a book. They interact and construct meaning from “the use of multimodal tools in socially situated practice” (Hassett & Curtwood, 2009, p. 281).
Secondly, readers are also taking the role of decision makers as they are confronted with many choices along the journey. As Hassett and Curwood (2009) suggest, “Rather than having simple, static images paired with standardized alphabetic print, multimodal texts take on dynamically interactive elements, as readers (not authors) choose where to look and how to engage with certain aspects of the text” (p. 271). This new role is as exciting as it is challenging. It is exciting because readers can engage in many sensory experiences as they see the text and visual elements, hear sounds, and watch videos to explore the meaning. On the other hand, however, it is challenging to decide whether they should click on a certain button, for example, as it may provide supplementary information to enhance comprehension or distract or even confuse readers as to why they are brought to a page in which information is remotely connected to the central idea of the digital book. A small scale study conducted by Chiong et al. (2012) compared parent-child co-reading on print, basic (an electronic version of the print book), and enhanced ebooks (an electronic book with many advanced digital features). The findings indicate that enhanced ebooks prompted more non-content related interactions, thus they were less effective than the print and basic e-books in supporting content-related vocabulary and overall language development. As a result, Chiong et al. (2012) caution designers of ebooks to avoid adding features that are not directly related to the story and enhance customizability features. Similar patterns were noticed in the digital picture books in this study. At times, clicking through a book becomes a tedious process as there are many bells and whistles embedded in the book that do not enhance the meaning. Consequently, those advanced digital features can become a distraction for young children.
Third and last, with a strong emphasis in Common Core State Standards (2011) on students’ abilities to recognize, interpret, and evaluate the information represented through diverse media and formats, educators need to explicitly teach effective digital literacy strategies to support children’s navigation and interpretation of multimodal digital texts. Even though many children in today’s classroom were born and grew up with technology, it becomes necessary to teach basic computer literacy skills such as how to navigate on screen and how to interact with digital books. For example, it is important to help students develop an understanding of the functions of different parts of the mouse. Moreover, there are many shapes a cursor can take (see sample cursor shapes in Figure 17) to communicate with the reader what s/he can do (e.g., a hand shaped cursor indicates a hyperlink to click). Not being able to read and interpret those symbols can create frustration for the reader. Interestingly enough, however, due to the fact that many children are familiar with the touchscreen features of SMART phones and iPads they assume that everything is touchable. This may be the direction that technology is moving toward, but it is not the current reality. Computers operated by mouse or a touchpad may co-exist with touchscreen devices for a period of time.
Furthermore, research has shown evidence that internet and hypertext reading involves different strategies than traditional print reading (Afflerbach & Cho, 2010). Thus, digital literacy strategies that examine the use of multimodal resources and usability features in digital texts should be modeled and practiced consistently across subject areas.
Some sample questions for discussion include:
- In what order are text and image presented? For what purpose?
- What do static and dynamic semiotic resources do to support/hinder my understanding of the meaning/topic?
- Regarding hyperlinks:
Before clicking: How do I know if there’s a hyperlink? With what can the link help me? What purpose does it serve?
While on linked web resources: What information do I look for? Is the link providing safe information? Should I pursue it or not?
After clicking: Do I know how to get back to where I left off in the book? Does the additional information on the linked resources support my understanding of the book/topic?
Additionally, Liu (2013) provides a solid framework that consists of meta-interpretive, perceptual, analytical, and sociocultural strategies to interpret visual images in multimodal texts.
Instructional activities such as repeated reading, creating book reviews, and making one’s own digital books are beneficial in supporting students’ interpretation, evaluation, and creation of diverse multimodal texts. After initial reading to become familiar with the content in the digital text, teachers can guide students in subsequent readings to explore, for instance, presentational sequence of text and visual, page layout, effectiveness of using multimodal resources, and ease of navigation/usage. Such conversations will model and give ownership to students in terms of how to use the digital-related literacy strategies. As a result, students can be more confident in participating in activities evaluating multimodal texts (e.g., writing book reviews) and in creating their own digital texts.
Limitations and Future Research
There are some limitations of this study. The data were limited to the finished product of digital pictures books. However, a closer look at the process of creating these digital books and perspectives from their creators—pre-service teachers—could provide new insights on how and why multimodal resources were chosen in the socio-semiotic framework. Further, the study was limited in providing authentic audience for pre-service teachers to use as context for their creation of digital books—having young children as the audience and users of these books could provide interesting information. Future research in getting feedback and responses from the intended audience will provide a reflective exercise to ponder the effectiveness of certain semiotic resources and usability features.
Conclusion
With the advances of technology, students in the 21st century process and create meaning in digital formats on a daily basis. An in-depth look at the digital picture books created by the pre-service teachers sheds lights on how technology opens new ways for the interplay of different semiotic resources and usability features in designing multimodal digital texts. From the analysis of the finished products, the authors of these digital picture books were quite successful in tapping into socially and culturally recognized multimodal resources to communicate meaning, even though there were some multimodal design elements left to be desired. This finding is promising as these authors are becoming the backbone of the teacher educators who work with children to develop equivalent digital literacy skills and strategies to interpret, evaluate, and negotiate effective multimodal designs in the digital world.
References
Afflerbach, P., & Cho, B. Y. (2010). Determining and describing reading strategies: Internet and traditional forms of reading. In H.S. Waters & W. Schneider (Eds.), Metacognition, Strategy Use, and Instruction (pp. 201-225). New York, NY: Guilford.
Bezemer, J. (2012). What is multimodality? Retrieved from https://mode.ioe.ac.uk/2012/02/16/what-is-multimodality/
Bezemer, J., & Kress, G. (2008). Writing in multimodal texts: A social semiotic account of designs for learning. Written Communication, 25(2), 166-195. doi:10.1177/0741088307313177
Bezemer, J., Diamantopoulou, S., Jewitt, C., Kress, G., & Mavers, D. (2012). Using a social semiotic approach to multimodality: Researching learning in schools, museums and hospitals. National Center for Research Methods Working Paper, 1-14. Retrieved from http://eprints.ncrm.ac.uk/2258/4/NCRM_working_paper_0112.pdf
Bird, E. (2011). Planet App: Kids’ book apps are everywhere. But are they any good? School Library Journal, 57(1), 26-31. Retrieved from: http://www.slj.com/2011/01/reviews/apps/planet-app-kids-book-apps-are-everywhere-but-are-they-any-good/#_
Carrington, V. (2005). New textual landscapes, information and early literacy. In J. Marsh (Ed.), Popular Culture, New Media and Digital Literacy in Early Childhood. London and New York: RoutledgeFalmer.
Chiong, C., Ree, J., Takeuchi, L., & Erickson, I. (2012). Print books vs. e-books. The Joan Ganz Cooney Center, 1-5. Retrieved from http://www.joanganzcooneycenter.org/wp-content/uploads/2012/07/jgcc_ebooks_quickreport.pdf
Common Core State Standards Initiatives. (2011). The Standards: English Language Arts Standards. Retrieved Feb. 26, 2012 from http://www.corestandards.org/the-standards/english-language-arts-standards
Computers Basic 1: Mouse and keyboard. (n. d.). Resources from Providence Public Library. Retrieved from http://www.provlib.org/sites/default/files/Computer Basics 1 Mouse and Keyboard.pdf
Gasparini, A. A. (2012). Touch, learn, play: What children do with an iPad in the classroom (Master’s thesis). Retrieved from https://www.duo.uio.no/bitstream/handle/10852/9015/Gasparini.pdf?sequence=2&isAllowed=y
Gutnick, A. L., Robb, M., & Takeuchi, L., Kotler, J. (2010). Always connected: The new digital media habits of young children. The Joan Ganz Cooney Center at Sesame Workshop, 3. Retrieved from http://www.joanganzcooneycenter.org/wp-content/uploads/2011/03/jgcc_alwaysconnected.pdf
Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and meaning. Baltimore, MD: University Park Press.
Halliday, M. A. K., & Hasan, R. (1985). Language, context and text: Aspects of language in a social semiotic perspective. Burwood, VIC: Deakin University Press.
Hassett, D. D., & Curtwood, J. S. (2009). Theories and practices of multimodal Education: The instructional dynamics of picture books and primary classrooms. The Reading Teacher, 63(4), 270-282.
Kress, G., & van Leeuwen, T. (1996). Reading images: The grammar of visual design. London: Routledge.
Kress, G., & van Leeuwen, T. (2001). Multimodal discourses: The modes and media of contemporary communication. London: Arnold.
Kress, G. (2003). Literacy in the new media age. London: Routledge.
Lewis, D. (2001). Reading contemporary picturebooks: Picturing text. New York: Routledge Falmer.
Liu, J. (2013). Visual images interpretive strategies in multimodal texts. Journal of Language Teaching and Research, 4(6), 1259-1263.
Merriam, S. B. (1988). Qualitative research and case study applications in education. San Francisco: Jossey-Bass Publishers.
Meyers, E. M., Zaminpaima, E. & Frederico, A. (2014). The future of children’s texts: Evaluating book apps as multimodal reading experiences. In iConference 2014 Proceedings (pp. 916-920). doi:10.9776/14312.
Prensky, M. (2001). Digital natives, digital immigrants. On the Horizon, 9(5), 1-6.
Serafini, F., & Clausen, J. (2012). Typography as semiotic resources. Journal of Visual Literacy, 31(2), 1-16.
Serafini, F. (2012). Design elements of picturebooks: Interpreting visual images and design elements of contemporary picturebooks. The Connecticut Reading Association Journal, 1(1), 3-9.
Siego, M. (2006). Rereading the signs: Multimodal transformations in the field of literacy education. Language Arts, 84(1), 65-77.
Skaar, H. (2009). In defense of writing: A social semiotic perspective on digital media, literacy and learning. Literacy, 43(1), 36-42.
Vaala, S., & Takeuchi, L. (2012). Co-reading with children on iPads: Parents’ perceptions and practices. The Joan Ganz Cooney Center at Sesame Workshop. Retrieved from http://www.joanganzcooneycenter.org/wp-content/uploads/2012/11/jgcc_ereader_parentsurvey_quickreport.pdf
Children’s Literature Cited
Morris, Ann. (1993). Bread, Bread, Bread. Illustrated by Ken Heyman. New York: HarperCollins.
Morris, A. (1993). Hats, Hats, Hats. Illustrated by Ken Heyman. New York: HarperCollins.
Morris, A. (1994). On the Go. Illustrated by Ken Heyman. New York: HarperCollins.
Morris, A. (1995). Houses and Homes. Illustrated by Ken Heyman. New York: HarperCollins.
Morris, A. (1998). Shoes, Shoes, Shoes. New York: HarperCollins.
Morris, A. (1998). Tools. Illustrated by Ken Heyman. New York: HarperCollins.
Morris, A. (2000). Families. New York: HarperCollins.
—
IVLA Selected Readings, 2014-15, first published on visualliteracytoday.org.