New Ways of Thinking about Literacy and Learning in Electronic Environments

Gang of 13 @ NRC

2005-11-10T12:06:00.000-05:00

I am looking forward to the gathering of the Gang of 13 at NRC on December 1 (7 AM in the Presidential Suite thanks to Don Leu.) I am also keenly aware that if this meeting is to be productive, we will need to come into it with goals and a plan for achieving them. The purpose of this memo is to begin a conversation about goals and the steps we can take to achieve them.

Please take some time to think about the issues and proposals described below and respond, either by email or on the weblog. I think it is important that we begin this conversation before we get to Miami. If we don't, I don't think we'll be able to accomplish what needs to be done when we meet. Please don't think that I'm assuming the proposals below are THE plan. If these proposals do the trick for us that is great, BUT I think we should be prepared to consider any well defined alternative. I'm not prepared to abandon a specific plan for vague ideas but I very much would like to see different ideas slug it out so we can see which ones hold up.

My sense from last year and our online conversation is that we have one major overarching goal - to develop proposals for and carry our collaborative research related to technology and literacy. I believe that recently developed literacy technologies provide new opportunities and tools to support "distance" collaboration and I'd like to see if we can't begin to develop a model for doing this kind of work.

I would like to suggest that we set out to achieve this larger goal by setting three more specific sub-goals:

1. Develop two proposals for symposia at NRC 06 focusing on theory and methodology in online literacy study,
2. Initiate empirical research projects that will serve as a basis for proposals submitted for presentation at NRC 07, and
3. Submit a proposal for a formal study group at NRC 07 that will build on what we have started by extending a more open invitation to NRC colleagues.

NRC 06 Proposals
I propose that we develop and submit a double symposium proposal related to our interests in online literacy.

Symposium 1 will focus on research methodology in studies of online literacy with papers addressing research design, data collection, and analysis.
Topics that might be included are verbal protocols/think alouds, developmental issues (i.e., expert/novice) in online literacy, real-time data collection, visualization techniques, etc. The goal of this session would be to help our colleagues better understand how to go about studying literacy online.

Symposium 2 will focus on theoretical frameworks within which studies of online literacy can be grounded. Theoretical frameworks we might include are Kintsch's Construction-Integration Model, Cognitive Flexibility Theory, Transactional theory, Universal design for learning, and other frameworks that can help us better conceptualize and design our studies in online literacy.
I propose that we select individuals to lead proposal development for each of these two symposia when we meet on Thursday morning. In preparation, I suggest that, as individuals, we decide where we want to focus our efforts and identify topics we would be interested in presenting within one or both of the symposia.

Collaborative Research
Developing symposia proposals provides an initial context for collaborative work but neither of the two symposia are intended to present the results of empirical research. Rather, the symposia proposals will define the theoretical and methodological points of departure for our empirical work. I believe our work in developing the proposals will begin to frame up the structures within which we can begin to carry out collaborative inquiry and I hope that we will take more concrete steps to initiate empirical collaboration soon after the NRC 06 proposals are submitted.

Ideally, I'd like us to arrive at NRC 06 with a well developed framework for conceptualizing and carrying out online literacy studies AND have collected enough data by that time so that we can say something about how these ideas really play out in practice. A subsequent goal then is to submit a proposal to present empirical results of our collaborative projects at NRC 07.
In order to do this, I think we need to think about what specifically each of us wants to contribute to a larger project. For example, I have developed technologies for delivering experimental materials, collecting research data (both real-time and post-reading) through a web browser, and displaying navigational data using graphs. Subjects in my recent studies login to a web page to access reading materials. The code I have developed tracks use of the materials during reading (sequence of pages read and time/page) and delivers and records both objective (true/false, multiple choice) and open-ended (essay) post-reading assessment measures. When the experimental session is complete the data is saved to a server database. Data analysis includes both traditional quantitative approaches and visualization methods that allow patterns of navigation to be displayed for both individuals and groups.

These techniques for collecting, analyzing and displaying data have worked well for me but my use has been limited to questions that, not surprisingly, are tied to my specific interests (reader stance, transactional theory, navigation in hypertext.) I am interested in collaborative projects that will allow me to build on what I have developed thus far in terms of the technology while also broadening the scope of questions and issues that these techniques can address. Moreover, these technologies (and others such as email, blogs, etc.) support forms of collaboration that eliminate many of the barriers that arise when researchers are widely scattered. Researchers who assume responsibility for data collection don't need to fuss with experimental materials for example as they are accessible from any computer on the web. Likewise, those who assume responsibility for data analysis have immediate access to the data at any time as they can be provided access to the server where the data is stored.

I propose that, if time allows, we discuss possible research projects but that we defer selecting research group leaders/coordinators until the NRC 06 proposals have been submitted. At that time, I think we will have a better sense about what we are interested in doing and the various skill sets and resources people bring to the table.

NRC 07 Study Group
I propose that we think about the NRC 07 study group as our first face-to-face meeting to discuss results from our collaborative efforts, plan for presentations, publications, and further collaborative inquiry. I envision a series of meetings addressing specific problems and issues we are dealing with in our distance collaboration. I also view the study group as an opportunity to broaden our base of interest within NRC, providing room for new people and perspectives either within ongoing research projects or in new efforts.

I propose that we select an individual to lead proposal development for an NRC 07 study group that focuses on online literacy studies.

John

2005-10-25T12:48:00.000-05:00

I’ve had an email from Don Leu asking if our group would like to use the Presidential Suite on Thursday morning for our NRC gathering and be treated to breakfast. He also suggested we broaden our group a bit by inviting other NRC members who have interests in technology and literacy.

I told Don I was delighted at his generosity and would present his invitation to the group. As I thought about Don’s offer I wondered about how the size of our group would influence how we operate. If we formalize our status and open the group up it may be that we end up moving in directions different from those that originally motivated us. Furthermore, organizing a research collaborative involving a group of 20 is a different task than dealing with a dozen.

Perhaps it will help to more clearly define what it is we want to do so we can better assess how to get there. 20 people seems like a lot for even a double symposium but perhaps an alternative format that includes both panels and papers could structure a session that involves that large a group.

How big a group do we want to be?
Do you see the size of the group as relevant to our goals?
Shall we assume a more open stance and see where that leads us?

What do you think?

I’ll post this note both by email and on the Blogger site (http://www.literacyandlearning.blogspot.com/). You can respond to all by email or on Blogger (http://www.blogger.com - you'll have to log in to post at Blogger however). If you’d like to respond anonymously, send me your response privately and I'll post it.

John

NRC 05 Meeting

2005-10-20T09:54:00.000-05:00

It looks like 7:00 AM on Thursday (12/1) morning will work. The Indigo Restaurant in the main lobby area also opens at 7 AM so we shouldn't have any trouble getting seats for our group. I'll arrive a few minutes early and see if we can push together some tables in a farther corner of the restaurant.

I propose that our goal for the Thursday morning breakfast be to decide where we want to be in December 2006, how we intend to get there, and what we hope to accomplish before the end of NRC 05 as a first step.

Here are two proposals to get the conversation started:

1. I propose that we develop and submit a (double?) symposium proposal related to our interests in online literacy.

One possibility is to focus on research methodology with papers addressing research design, data collection, and analysis. Topics that might be included are verbal protocols/think alouds, developmental issues (i.e., expert/novice) in online literacy, real-time data electronic collection, visualization techniques, etc. The goal of this session would be to help our colleagues better understand how to go about studying literacy online.

Another possibility that I find attractive focuses on defining larger theoretical frameworks within which studies of online literacy can be grounded. I think we need to put studies simply comparing print to electronic text behind us and get serious about rooting online literacy in well defined theoretical frameworks. The goal of this session would be to begin to articulate a broad but empirically useful framework within which we can talk about research, practice, and pedagogy in online reading environments.

If we're feeling ambitious, we can consider a double symposium that addresses both.

2. I propose that we develop and submit a study group proposal so that we can build on the momentum we establish this year with an even more ambitious agenda at NRC 06.

Perhaps we can discuss our general objectives at the Thursday morning meeting and divide up into groups to work on the objectives we select.

John

Online Reading

2005-02-22T05:34:00.000-05:00

Hi,

Thank you for inviting me to this Blog. Currently, I'm working with Colin Harrison in finding a way to guide / assess online reading. I'm from pure computer science background and just recently aware of online reading / reading materials after joining the online reading research group. Your discussions will be very useful for me.

Many thanks,

Nas.

My Research Interests--Searching

2005-02-18T19:39:00.000-05:00

My research interests are focused on looking at the literacy skills kids need to search for and access information on the Internet. I'm beginning a pilot study this semester that will look at several different components of this issue. There are two main parts to this project.

This first piece is looking at instructional models for teaching students skills for searching on the Internet. I have two treatment and one control group at each of three grade levels (6, 7 & 8). One group will receive instruction through a webquest-type instructional component (title of blog entry is linked to my blog that documents development and design aspects of this instructional unit). This group will be in a computer lab setting with two adults available to scaffold but no instruction will be delivered by the teachers. The second group will receive direct instruction in a regular classroom using one computer with a projector to model searching skills. The third group will be a control group and receive no instruction. Students will be randomly selected from each condition for assessment of searching skills.

The second component will look at the role of prior knowledge in students' abilities to search for information. Students in grades 6, 7 & 8 will be randomly selected to participate in a series of 3 search tasks. The first task will be centered on a topic in which students have high prior knowledge. The second search task will be a topic in which the students have little or no prior knowledge. The third task will be a student selected topic of high interest.

I have a couple survey instruments in the works as well. One is an Internet Usage Survey that will be administered to a sample of middle school students. The other is an attitude survey that will be administered to middle school teachers to uncover beliefs about teaching reading on the Internet.

Laurie

Interests and agendas

2005-02-17T13:42:00.000-05:00

I think this resource will be a useful tool for exploring ideas and planning for work to be completed at NRC 2005 in Miami. Thank you to Laurie for setting this up for us!

I'd like to suggest that we exercise our blogging skills by writing about the kind of research we are most interested in pursuing with respect to technology and literacy. Part of what I think we want to do is identify colleagues with whom we might collabaorate and I suspect an important part of that process will be learning more about the kinds of research folks are doing and what we'd like to be doing over the next few years.

I am interested in how readers move around inside electronic reading environments (by selecting links). Since starting this line of work in the later 90s, I have focused primarily on quantitative approaches, defining specific measures and assesing the influence of various factors on these measures. I am, however, also quite interested in the interpretation of visual displays of reader navigation, an approach that seems to me to be more qualitative than quantitative (although I have linked numerical measures to these visual displays as well.) I ground my work with hypertext in transactional theory.

I expect to continue working on quantitative questions related to the structure of hypertext and reader navigation within it, but I also hope to begin to define more "up close" qualitative approaches that focus on individual readers as they read online text. I am also very interested in reading theory generally and the more specific issue of how existing theory does or does not address the new possibilities for literacy practice online.

I look forward to learning more about the interests and agendas of others in this group!

Structuring access to web-based learning and literacy

2005-02-15T10:00:00.000-05:00

I agree with Julie's comments about the importance of structuring access to
web-based learning and literacy. I think that one of the most significant
results from recent work is that an interface can really change what readers
do and how they view a literacy task. Tools don't just add resources, they
can alter a task by changing how learners/readers perceive it.

John

PS - If anyone receiving these postings does NOT wish to participate, please
let me know and we'll pull your name from the list. We'd like to avoid
spamming those of you who'd prefer to opt out.

An "off-the-record" study group for idea development

2005-02-15T09:43:00.000-05:00

John,
You've done a wonderful job in outlining a proposal and crafting ideas about issues to be considered and a daily breakdown for how we might begin to consider them!
I do like your suggestion of an "off-the-record" study group since we are in the beginning stages of "idea development" and it may become too difficult, as you say, "to manage a genuinely open invitation" until we further clarify some of the issues.

That being said, I'll join in as participant whether it's formal, informal, or even if we hang out on a sidewalk somewhere with a Margarita in hand! ;-) (hmmm...maybe not a bad idea!)

About the topics, I'd like to suggest that somewhere in there we address how our developing theories and research methods can inform the practical development of some type of web-based learning scaffold. I keep going back to that idea we were envisioning at your session of some electronic environment that guides readers/learners in identifying their purpose, background knowledge, text structure, strategy selection, etc and then provides them with open-ended tools/supports/features that enable each learner to customize them according to his/her needs. Just a toolbar or two and a frame with certain levels of features that incorporates each of our perspectives could be so powerful, I think, in seeing how technology can be used to inform, support, track and modify strategies readers use as they interact with and respond to Internet text. (and these same tools/features can simultaneously be used to track, analyze and report data collected within these environments). The teacher in me continues to harbor just below the surface of the researcher in me :-)

Thanks for pulling this together so quickly. What do others think?
:-) Julie

NRC Study Group?

2005-02-15T08:56:00.000-05:00

Here is a first draft of a study group proposal. If you have suggestions or
would like to be included as a participant, please let me know. I will be
happy to add as many names as I can to this proposal as I suspect this will
have an impact on whether or not the study group is approved.

One other matter I'd like to suggest we consider is whether we should
consider establishing an "off-the-record" study group rather than
formalizing our collaboration in a proposal. Although I like the openness of
the study group approach, I'm not sure we're ready to manage a genuinely
open invitation. Every time I have gone to study groups in the morning
there have been empty tables. What if we just meet in the study group hall
following the general outline of the proposal below without formalizing our
status as a study group?

Any thoughts on the subject?

John

The study group proposal follows:

Importance of the topic
Although traditional print technologies are likely to be with us for many
years to come, electronic print media are assuming major roles in the way we
access and use information (Leu, Kinzer, Coiro, & Cammack, 2004). There is
ample evidence, however, that the transition from traditional print to
electronic media has cognitive consequences and that simply making
information electronically available will not assure it is effectively or
efficiently used (Wenger & Payne, 1996; McEneaney, 1998; Conklin, 1987;
Egan, Remde, Gomez, Landauer, Eberhardt, & Lochbaum, 1989).

Issues to be considered
Three main issues will be addressed by the study group: 1) Theory and
research design for studying online reading, 2) Research methodology and
real-time data collection, and 3) Technology tools to support collaborative
research. These issues will be discussed in this order across three days of
the conference. On the fourth day, the study group will define working
procedures that will govern one or more collaborative research projects in
online literacy. Participants in the research will determine individual
responsibilities, outline procedures to support communication and exchange
of data, and define timelines for work. The research that is completed will
be submitted for presentation at the 2006 annual meeting of the National
Reading Conference.

Evidence of interest
This study group arises out of a session at NRC 2004 that focused on
research issues and strategies in studying online reading. Discussion at
the session and subsequent online interaction indicates there is a core of
researchers with interests in defining and exploring research methods and
strategies that are specific to online reading. Moreover, we believe a focus
on defining and applying specific research methodologies makes our work
complimentary to the broader goals of the Technology Committee and the
Tech-saviness, Multiliteracies, and Technology-mediated Literacy Experiences
Study Group.

Structure and organization
Discussion on each day will begin with a specific topic (identified below)
that is discussed by the study group as a whole. The last 15 minutes of
days 1-3, however, will be devoted specifically to forming research groups
and exploring research proposals. On Day 4, the entire study group session
will be devoted to defining specific research proposals that will be carried
out by the virtual research groups that coalesce over days 1-3.

Day 1 (Wednesday): Theory for Research in Online Reading
The focus of discussion on this day will be on identifying theoretical
frameworks that support integrative views of literacy within online reading
environments. There will be a special emphasis on frameworks suitable for
research agendas and objectives that rely on collaborative work.

Day 2 (Thursday): Methodological and Research Design Issues
The focus of discussion on Day 2 will be on exploring methodological and
research design issues in studying online literacy. How can qualitative and
quantitative methodologies be applied to frame our understanding of broader
issues and/or address specific research questions? Discussion on day 2 will
also consider the role of real-time data collection available in online
reading environments.

Day 3 (Friday): Technology tools for managing virtual collaboration
Discussion on day 3 will focus on the communication and data exchange tools
that research groups will use. Email, discussion boards, weblogs, and
web-based collaborative authoring environments will be considered
specifically as research tools and a plan will be developed to apply
specific technologies in support of one or more research projects.

Day 4 (Saturday): Defining responsibilities and timelines
Research groups that have formed over days 1-3 will meet in small groups to
define research activities, participant responsibilities, and timelines for
completing work. If there are multiple research groups, each research group
will present its proposal and one person from each study group will be
identified as the primary contact person.

Prior group activities
Participants initiated an electronic discussion by email immediately
following NRC 2004. Electronic interaction will be employed in addressing
issues well ahead of the annual meeting so that our productivity in the
short time we have to work in Miami can be maximized.

Study group goal
The goal of the study group is to initiate and support research initiatives
among study group members. More specifically, it is our goal to define one
or more collaborative research efforts that incorporates research
collaborators from multiple locations. Results of our work will be developed
as proposals for presentation at NRC 2006.

NRC 2005 "alternate session" proposal?

2005-02-14T19:10:00.000-05:00

Hi all,

Count me in. Am swamped but following the discussion. In addition:

Who's interested in joining me in writing up an NRC 2005 "alternate
session" proposal to explore the BASIC question in which (I think) we
are all interested:

What are the primary differences between reading print texts vs. reading
hypermedia texts? Same/different cueing systems? Same/diff
metacognitive strategies? Same/diff mediating factors (learner, text,
task)? Different paths? Expository vs. narrative genres? THEN, What
does this imply for pedagogical strategies?

AND/OR, What established theories of print reading can be "ported" over
to online reading and adapted so we can make sense of all this? John's
been exploring transactional theories, are there others we want to
investigate?

I'm open to ideas. Proposals are due March 1.

Best,
Maya

P.S. Can we request a longer session than is typical for NRC? The 2004
session to which Julie referred (McEneaney, Eagleton, & Dalton) was too
short for all the rich discussion that ensued. Is a study group the
only alternative?

Anyone up for instituting a virtual research group?

2005-02-13T16:41:00.000-05:00

The Pirolli & Fu paper is interesting and overlaps with quite
a few of the issues we talked about in San Antonio. As I
understand the model, the declarative memory component is
based on a latent semantic analysis network that draws both on
prior work (the Tipster corpus) and a web-specific augmentation.

Broadly, processing seems to operate in three stages:
1) Identify the current display state and the user action that
followed that state,
2) Update the display state (if the user action changes it),
generate a predicted action based on the new state (drawing on
the hybrid spreading activation-production system model), and
identify the user action that follows this state,
3) If the predicted action matches the user action, fire the
predicted action, If the predicted action does not match the
user action, fire the user action.

Although I think this is the kind of work we need to be doing,
I think that the paper overstates the power of the model. One
reason is that, while stage 3 records “misses” (and uses this
data in the subsequent chi square analysis), it treats
decisions on different pages as if they were independent of
one another, something that is clearly not the case. The
decision a reader makes on the first page viewed is almost
certainly related to the decision she makes on the second.
Moreover, setting up the model to always choose the correct
decision keeps the model's behavior on track by reseting
“overall” error in each cycle. I suspect this reduces overall
error quite a bit and inflates the model's goodnbess of fit.

A variation on this same problem seems to arise in the data
analysis. The chi-square presumes that observations are
independent, but the decision making that is explored in this
study is one that occurs across an extended period of time
involving multiple related decisions. It also isn’t clear to
me why the chi square was computed to assess whether the
predicted actions differed from random choice rather than the
choices actually made by readers in the study. I’m not at all
surprised the model is not random but that doesn’t seem to me
to be the point.

Two ideas particularly intrigued me in that they suggested
specific steps that could be taken without requiring the
complexities involved with integrating a latent semantic
analysis module. In the analyses we presented at NRC, we
informally categorized pages according to content and noted
that navigation varied quite a bit even when the content that
seemed to be driving the process was the same across subjects.
We didn’t really know what to make of it, but Pirolli and Fu’s
use of color suggests another way to qualitatively capture
what is going on. With some adjustment to the routines we use
to generate graphics we can color-code the nodes that appear
in path diagrams according to the content they present. This
won’t “explain” what is going on but I think it could give us
insights that we can follow up on. I suspect that color may
help us see content-related patterns and possibly even reveal
interactions between navigation patterns and content. Best of
all, we have all the data we need to generate new graphics and
see if they are useful.

Secondly, I’m intrigued with the notion that task difficulty
might be reflected in the “branchiness of the WBGs (first
paragraph in section 4)”. My work on navigation has
incorporated specific measures of “branchiness” (path stratum
and path compactness) and in some studies these measures seem
to be related to performance. I’d also like to follow up on
an issue that I don’t think is really acknowledged in this
study: how the nature of navigation and decision making
changes within a single extended reading episode. Pirolli & Fu
seem interested in contrasting within-site and across-site
(e.g., Figure 4) but I don’t think this comparison can be
based on the few pages before and after the transition from
one site to another. I think there is a need for a broader
view across the episode of reading within a site if we want to
understand the logic of the decision to leave that site. Data
we presented at another NRC session supports the hypothesis
that as readers navigate within a site, they develop expertise
that changes the time they spend on pages and the type of
navigational patterns they employ.

I would very much like to follow up on Julie’s general
invitation to explore these issues and suggest that we develop
a proposal for a study group to meet at NRC 2005. Perhaps if
we keep up the conversation online we can define a
collaborative approach to studying these issues that draws on
the varied interests and types of expertise we all bring to
the table. If we make progress before the conference, we can
focus on practical issues when we meet in Miami. If we don’t
make progress electronically, we can use our study group
sessions to resolve the issues that remain.

Anyone up for instituting a virtual research group?
Any thoughts on the idea of submitting a study group proposal?

John

New Thinking Related to NRC Presentation

2005-02-12T19:10:00.000-05:00

Hi all!

I hope everyone is surviving the beginning of the spring semester without keeping TOO busy!

I have been doing some reading and came across a very interesting article that reminded me of some of the conversations that emerged during and after the NRC presentation by John McEneaney, Bridget Dalton, and Maya Eagleton. I came across this paper and thought I'd send it along to the people I could remember were there (and a few that weren't but might be interested) just to share a new idea (or at least new to me) that might spark some more thinking along those lines.

Just to refresh your memories: After the presentations, we were collectively brainstorming ways that John's trace methodologies could potentially be used within a scaffolded environment (like Strategy Tutor) to simulate novice vs. expert interaction within online informational websites and thus, "with the click of a button" guide or support readers engaged in complex information searches on the Internet.

The title of the paper I just found is SNIF-ACT: A Model of Information Foraging on the World Wide Web -
available in pdf format at
http://www2.parc.com/istl/projects/uir/pubs/items/UIR-2003-02-Pirolli-UM-SNIFACT.pdf

This study (it's short - only about 13 pages) examines log files generated by WWW interaction while users are engaged in ecologically valid WWW reading and learning tasks. The data is represented in "web behavior graphs" that reminded me of John's work with graphical analyses of reader patterns for different purposes.

The trace methodologies here integrate data from something called Weblogger (which captures keystrokes, browser actions, etc), Eyetracker, and video recordings along with verbal protocol data to produce a "web protocol transcript". It sounds like Camtasia could combine weblogger and video (though not sure which platforms produce data that is most efficiently analyzed for the purposes we may have).

The results suggest that some of the ideas we were thinking about at NRC might actually be possible (though I don't think I have the technical brainpower that John does to actually make sense of the data! ;-)

Also, on the bottom of page 5, the authors refer to a database of tasks collected from a survey of over 2000 WWW users (Morisson, Pirolli & Card, 2001) - has anyone ever heard of this? I thought it might be a nice database to get hold of for ideas we could all use in our work with students reading on the Internet.

Well - I just thought I'd pass this information on in case it sparked any new ideas for any of you. It reminded me of the great NRC presentation and how the technologies seem to be quickly emerging to begin providing the scaffolding we'd like for online readers - it's just a matter of pulling information from different fields of work to inform the development of this support system. I'd love to hear anyone else's thoughts on the matter!

Hope all is well with everyone!
Take Care,
:-) Julie