Direct Instruction of Metacognition Benefits Adolescent


Download Direct Instruction of Metacognition Benefits Adolescent


Preview text

Journal of Educational Psychology 2015, Vol. 107, No. 4, 954 –970

© 2015 American Psychological Association 0022-0663/15/$12.00 http://dx.doi.org/10.1037/edu0000022

Direct Instruction of Metacognition Benefits Adolescent Science Learning, Transfer, and Motivation: An In Vivo Study

Cristina D. Zepeda and J. Elizabeth Richey
University of Pittsburgh

Paul Ronevich
Pittsburgh Science and Technology Academy

Timothy J. Nokes-Malach
University of Pittsburgh
Prior studies have not tested whether an instructional intervention aimed at improving metacognitive skills results in changes to student metacognition, motivation, learning, and future learning in the classroom. We examined whether a 6-hr intervention designed to teach the declarative and procedural components of planning, monitoring, and evaluation could increase students’ metacognition, motivation, learning, and preparation for future learning for middle school science. Forty-six eighth-grade students were randomly assigned to either a control group, which received extensive problem-solving practice, or an experimental group, which received more limited problem-solving practice along with metacognitive instruction and training. Results revealed that those who received the metacognitive instruction and training were less biased when making metacognitive judgments, p ϭ .03, d ϭ 0.65, endorsed higher levels of motivation after instruction (e.g., there was a large effect on task value, p ϭ .006, d ϭ 0.87), performed better on a conceptual physics test, p ϭ .03, d ϭ 0.64, and performed better on a novel self-guided learning activity, p ϭ .007, d ϭ 0.87. This study demonstrates that metacognitive instruction can lead to better self-regulated learning outcomes during adolescence, a period in which students’ academic achievement and motivation often decline.
Keywords: instruction, learning, metacognition, motivation, transfer
Supplemental materials: http://dx.doi.org/10.1037/edu0000022.supp

A student’s ability to adapt his or her problem-solving behaviors to different types of academic tasks and feedback is critical for successful learning and academic achievement. Educational psychologists have referred to this ability as self-regulated learning (SRL) and define it as a set of interrelated skills and motivations to control learning. Most theories of SRL hypothesize that both metacognitive skills (e.g., planning, monitoring, and evaluation) and student motivation (e.g., beliefs, goals, and dispositions) interact to determine learning outcomes (Boekaerts & Corno, 2005; Efklides, 2011; Winne, 1995; Zimmerman, 2001, 2011). However, relatively few studies have tested whether a metacognitive instructional intervention de-
This article was published Online First March 16, 2015. Cristina D. Zepeda and J. Elizabeth Richey, Department of Psychology, Learning Research and Development Center, University of Pittsburgh; Paul Ronevich, Pittsburgh Science and Technology Academy, Pittsburgh, Pennsylvania; Timothy J. Nokes-Malach, Department of Psychology, Learning Research and Development Center, University of Pittsburgh. This research was supported by Grant SBE 0836012 from the National Science Foundation to the Pittsburgh Science of Learning Center. No endorsement should be inferred. We thank research assistant Sarah Honsaker for her help in coding the data. Correspondence concerning this article should be addressed to Cristina D. Zepeda, Learning Research and Development Center, 3939 O’Hara Street, University of Pittsburgh, Pittsburgh, PA 15260. E-mail: [email protected] pitt.edu

signed to improve students’ metacognitive knowledge and skills can also improve student motivation more broadly. Moreover, we know of no work that has examined metacognitive, learning, transfer, and motivational outcomes together in a single study.
To address these issues, we conducted an in vivo classroom experiment with students randomly assigned to either a metacognitive instruction and training condition or a problem-solving practice condition. We examined whether students given the metacognitive instruction acquired knowledge and skills about metacognition and whether those skills improved their learning of the target instructional content (physics concepts and problem-solving procedures) as well as new material given later in the semester (experimental design, control of variables strategy). We also examined whether the intervention affected a wide range of motivational constructs specified in Zimmerman’s (2011) sociocognitive SRL theory. In the following sections, we review relevant literature that guided the development and theoretical framing of this study.
Metacognition and Motivation in SRL
We focus on the metacognitive skills of planning, monitoring, and evaluation that occur in Zimmerman’s (2000, 2011; Zimmerman & Campillo, 2003) SRL phases of forethought, performance, and selfreflection, respectively (see Figure 1). We define planning as identifying the goal of the problem, the critical features, and a set of strategies to move toward that goal; monitoring as keeping track of one’s current state and progress moving toward the goal; and evalu-

954

DIRECT INSTRUCTION OF METACOGNITION

955

Forethought Phase
Planning
Self-Motivation Beliefs Self-efficacy Task value
Goal orientation Need for cognition*

Performance Phase
Monitoring
Self-Control Effort regulation* Control of learning beliefs*

Self-Reflection Phase
Evaluating
Self-Judgment Theories of intelligence*

Figure 1. Phases and subprocesses of self-regulated learning (SRL). Asterisks denote constructs we have added to the model based on related literature. Adapted from Zimmerman and Campillo’s “Motivating selfregulated problem solvers,” in The psychology of problem solving (p. 239) edited by J. Davidson & R. J. Sternberg, 2003, New York: Cambridge University Press. Adapted with permission.

ating as assessing one’s solution and determining whether it satisfies the goal as well as reviewing which strategies worked best. Zimmerman’s model provides a theoretical framework for determining which motivational constructs might be affected by a metacognitive intervention targeting these skills, including aspects of self-motivation beliefs, self-control, and self-judgment. In Figure 1 we have emphasized the metacognitive skills hypothesized to occur during each SRL phase (in bold) and the hypothesized motivational constructs associated with each phase. In addition to Zimmerman’s self-motivation beliefs we have added motivational constructs hypothesized to be closely related to the later phases of the model. For example, students’ beliefs about effort regulation and control of learning are likely to be associated with their self-control while students’ theories of intelligence may affect their self-judgment.
Although a number of studies have explored possible relationships between metacognition and motivation, the majority of empirical investigations exploring those connections have used correlational and quasi-experimental designs (e.g., Ford, Smith, Weissbein, Gully, & Salas, 1998; Somuncuoglu & Yildirim, 1999). The experiments that have used metacognitive interventions to test the relationship typically assess only one or two motivational outcomes, making it difficult to determine whether there are wide-ranging changes in student motivation or just changes to a few constructs.
Our investigation addresses these issues in multiple ways. First, by conducting an experiment with a randomized control, we can test the causal effects of a metacognitive intervention on motivational outcomes. Second, because our intervention targets metacognitive skills used across all three phases of SRL, we expect it to have a broad impact on student motivation, as the hypothesized and cyclical relationships in Zimmerman’s SRL model would suggest. To assess the breadth of motivational changes produced by metacognitive training, we included a number of motivational constructs including selfefficacy (Pintrich, Smith, Garcia, & McKeachie, 1991), task value (Pintrich et al., 1991), achievement goal orientations (Elliot & Murayama, 2008), need for cognition (Cacioppo, Petty, Feinstein, & Jarvis, 1996), effort regulation and control of learning beliefs (Pintrich et al., 1991), and theories of intelligence (Dweck, 1999).

We briefly discuss the way (or ways) in which each motivational construct might be affected by the intervention, based on the metacognitive skill most closely associated with it. Some hypothesized effects have more prior empirical support whereas others are more exploratory. While this work contributes to the literature by testing the relationship between a metacognitive intervention and a number of specific motivational outcomes, we do not test the relationship between distinct components of metacognition and different motivational constructs. Zimmerman’s phases are thought to be cyclical, and therefore, it would be difficult to identify which metacognitive skill led to a specific motivational outcome. Future work exploring these relationships will be needed to test the underlying mechanisms at play.
Planning
Planning skills serve as domain-general knowledge that can be applied to solve new problems. Having knowledge of these skills should increase students’ self-efficacy, defined as confidence in their capabilities to solve such problems, because it suggests a set of strategies to apply when students might otherwise feel unsure about how to approach a new problem (Pajares, 2008; Zimmerman, Bonner, & Kovach, 1996). Students’ self-efficacy is thought to relate to their value of a particular task, defined as the degree to which they believe that the task is interesting, important, and useful, as past work has identified positive correlations between perceived confidence and task value (Jacobs, Lanza, Osgood, Eccles, & Wigfield, 2002). Some prior research also suggests that value is positively related to the use of constructive strategies, cognitive engagement, and mastery-approach goal orientation, defined as aiming to fully understand material (Meece, Blumenfeld, & Hoyle, 1988; Nolen, 1988). If students can more successfully solve new problems and come to value such tasks, this could lead them to a have a have a higher need for cognition, defined as a desire to experience more complex or challenging thinking (Cacioppo et al., 1996).
Pintrich (2000) argues that achievement goal orientations play an important role in self-regulated learning because learners’ goals serve as criteria by which they can evaluate and regulate their progress. Students with a mastery-approach goal orientation engage in competence-related activities to improve their understanding (Elliot & Murayama, 2008), and this type of goal has been related to successful self-regulated learning (Pintrich, 1999). Since metacognitive interventions teach students different skills to improve their understanding, students who learn about metacognition might be more likely to adopt and endorse mastery-approach goals to create consistency between their goals, knowledge, and behaviors.
Monitoring
If students are able to monitor their progress toward a goal, they may also be better equipped to make decisions about how to manage their available resources, including their effort. Being aware of their progress may also make them more willing to apply effort, which can be measured by the effort regulation scale (Pintrich et al., 1991). Zimmerman and Martinez Pons (1988) found that students’ self-regulated learning strategies were related to their efforts to learn. Relatedly, improving students’ ability to monitor their task performance might also make them more aware of their own control of learning, which can be measured by the control of learning beliefs scale (Pintrich et al., 1991).

956

ZEPEDA, RICHEY, RONEVICH, AND NOKES-MALACH

Evaluating
Part of the evaluation process involves making causal attributions for success or failure on a task. Learners with poor evaluation skills may be more likely to attribute success or failure to their personal characteristics (e.g., having ability in a certain domain or not), while learners with more accurate evaluation skills and awareness may be better equipped to identify learning behaviors that led to a given outcome. If students’ beliefs of intelligence are informed by their task evaluation skills, then improving those skills might promote endorsement of incremental theories of intelligence—the belief that intelligence is malleable and not fixed (Dweck, 1999). It is also possible that students might adopt an incremental view of intelligence after learning that evaluation skills can be changed with instruction and practice.
Metacognitive Interventions: Transfer and Future Learning Measures
Metacognitive skills have also been hypothesized to facilitate knowledge transfer and preparation for future learning (Schraw, Dunkle, Bendixen & Roedel, 1995; Schraw & Nietfeld, 1998; Veenman, Elshout & Meijer, 1997; Veenman & Verheij, 2001; Veenman, Wilhelm, & Beishuizen, 2004; Wolters & Pintrich, 1998). However, little empirical work has tested these hypotheses. Through the use of metacognitive skills, students can acquire both procedural (BerardiColetta, Buyer, Dominowski, & Rellinger, 1995) and declarative knowledge (Cross & Paris, 1988). The type of knowledge a learner acquires is important as it has implications for when and how that knowledge transfers (Nokes, 2009; Nokes-Malach & Mestre, 2013). Proceduralized knowledge (i.e., knowing how to do a task) typically facilitates near transfer to problems that have surface features and structures identical or similar to prior problems, whereas declarative knowledge (i.e., knowing descriptive information about a task) can support performance on far-transfer problems that have different surface features but similar structures (Nokes, 2009; Nokes-Malach & Mestre, 2013; Nokes & Ohlsson, 2005). Adopting Barnett and Ceci’s (2002) transfer framework, we define near transfer of content knowledge as the execution of prior procedures or the recall and application of prior concepts to familiar problem features and far transfer of content as the recall and application of prior concepts and principles to new problem features.
We also distinguish between two sources of knowledge that may support transfer from metacognitive interventions: the domainrelevant instruction such as physics content given over the course of the intervention and instruction about metacognitive skills themselves. These two sources of knowledge have different implications for transfer. Domain-relevant instruction should support transfer to the degree that the knowledge acquired is abstract and applicable to new problems or questions (Barnett & Ceci, 2002). Metacognitive instruction should support different types of transfer, including near, far, and preparation for future learning, through the application of domain-general metacognitive skills to new problems or learning opportunities. This suggests two important questions about the types of transfer supported by metacognitive interventions: what type of knowledge transfers and what is the source of the knowledge being transferred?
Many past metacognitive interventions have not assessed fartransfer outcomes, and few have taken a rigorous approach to

defining levels and types of transfer. However, some studies have shown promising results for both near and far transfer (e.g., Brand, Reimer, & Opwis, 2003; Palincsar & Brown, 1984; Lin & Lehman, 1999). Lin and Lehman (1999) found that students who received metacognitive prompts performed better than other prompt conditions and a control condition on near- and far-transfer assessments. In order to solve the far-transfer problems, the students needed to adapt their previous conceptual understanding to the new problem features.
In the current work, we employed multiple transfer assessments to distinguish between near transfer (structurally similar problems included at the end of each instructional packet) and far transfer (questions included on a conceptual test given after a delay). We also include assessments to differentiate between transfer of domain-relevant content covered in the instructional packets (the end-of-packet transfer problems and questions) and the transfer of domain-general metacognitive skills to new learning opportunities (a preparation for future learning, or PFL, measure). In contrast to the more classical conceptualization of transfer that focuses on whether or not one can use knowledge acquired from instruction to solve novel problems (Barnett & Ceci, 2002), the PFL measure focuses on whether the initial instruction affects what one learns from subsequent instruction and how that knowledge is then used to solve new problems (Bransford & Schwartz, 1999). In this study we examined whether students could utilize metacognitive skills in a self-guided learning activity on a novel topic two weeks after the intervention. We discuss the details of these transfer measures in the methods section below.
Present Study
Given this theoretical and empirical backdrop, we tested a self-guided metacognitive intervention that targeted planning, monitoring, and evaluating skills and highlighted how those skills support adaptive problem solving. Meta-analyses of metacognitive interventions (Dignath & Büttner, 2008; Dignath, Buettner, & Langfeldt, 2008; Hattie, Biggs, & Purdie, 1996) were particularly helpful in guiding our design and implementation of the intervention, including the decision to provide instruction on all three skills along with conditional, interactive process knowledge of how they work together. We view this as a critical component of the intervention because if students know the interrelations between planning, monitoring, and evaluating, then they can use that knowledge to effectively adapt their behavior. Without such knowledge, they might become confused about when or how to use the skills or just focus on one skill.
Hypotheses
This study had three central goals. The first goal was to create an easy-to-implement instructional intervention designed to increase students’ metacognitive knowledge and skills (i.e., awareness, accuracy, and use). The second was to examine whether the metacognitive intervention affects different motivational constructs hypothesized to relate to metacognition and self-regulated learning. The third was to test whether the intervention results in greater learning and transfer. Figure 2 represents our hypotheses as they relate to each goal.

DIRECT INSTRUCTION OF METACOGNITION

Providing Information & Practice of
Metacognitive Skills

A More Self-Regulated Learner Metacognitive Awareness

H1. Metacognition

Monitoring Accuracy

H2. Motivation
H3. Learning & Transfer

Metacognitive Use
Self-Efficacy, Task Value, Achievement Goals, Need for Cognition, Effort Regulation, Control of
Learning Beliefs, Theories of Intelligence
Near Transfer (end-of-packet transfer problems)
Far Transfer (conceptual test)
Prepration for Future Learning (novel learning activity)

957

Figure 2. Graphical representation of the hypotheses.

Method
Participants. Forty-nine students from two eighth-grade physics classes at the same urban, public middle school participated in the experiment (23 students in one class and 26 in the other). Three students were dropped from the study because they did not complete the majority of the assessments. Of the remaining 46 students (13 females, 33 males), 44% self-identified as White, 41% as African American, 13% as mixed race, and 2% as Latino(a)/Hispanic.
Design. We used a between-subjects, preposttest design with students randomly assigned to either the experimental (n ϭ 23) or control condition (n ϭ 23). There were no differences between the conditions across any of the demographic variables including students’ gender, ␹2(1, N ϭ 46) ϭ .11, p ϭ .74, or race, ␹2(3, N ϭ 46) ϭ 1.05, p ϭ .79. All students had the same teacher, who was blind to the condition assignment of the students. Students participated as a regular part of their classroom instruction and received participation points for completing the various activities. The teacher distributed all materials in packet form and made sure that students were not looking at each other’s packets, working together, or asking each other for help. The packets had a noninformative group label on the cover page so the teacher could distribute them without learning the condition assignments. We provided a script for the teacher to follow and met regularly with him to ensure treatment fidelity.
Due to an implementation error, one student from each condition completed instructional materials from the opposite condition for one of the eight learning packets. We decided to include these students in the data analysis as they received the majority of training materials for their conditions and contribute to increasing power, albeit with a weaker dosage of the intervention.1 Figure 3 shows an overview of the experiment design, materials, and procedure.
Intervention materials. We used puzzle problems for the first round of the intervention to avoid the potentially distracting knowledge demands that science content might have introduced. This enabled us to emphasize problem-solving and metacognitive skills over the problem content. The puzzles consisted of spatial

and verbal insight problems, riddles, rebus word problems, and simple math problems (see online Supplemental Materials for an example of each problem type). All students were given the same initial problem, followed by a hint, another opportunity to solve it, and the solution. At the end of each packet, all students were given a transfer problem with a similar structure to that of the initial problem. For example, the initial and transfer problems in the first packet were spatial insight problems that focused on the manipulation of shapes. By using the same problem type, we could see if training improved immediate performance on near-transfer problems. Following the transfer problem, students in both conditions received a packet quiz that assessed their declarative knowledge of the targeted metacognitive skill. The quiz consisted of an openended question and multiple-choice questions (see online Supplemental Materials).
Packets 1– 4: experimental. In the first packet, students studied an explanation of planning, reviewed worked examples of plans, responded to questions about their own planning activities, and created a plan to solve a new problem (see Table 1). In the second packet, students studied an explanation of monitoring, reviewed and analyzed fictional students’ attempts to solve problems, and responded to questions about their own monitoring activities (see Table 1). In the third packet, students studied an explanation of evaluating and responded to prompts to evaluate their solutions (see Table 1). In the fourth packet, students reviewed descriptions of planning, monitoring, and evaluating, read about how to integrate the three skills when problem solving, and responded to prompts and questions targeting all three skills (see Table 1). Unlike the other packets, Packet 4 concluded with two transfer problems instead of one. See the online Supplemental Materials for the detailed definitions presented in each of the four packets. On average the experimental condition completed 93% of the packet materials (SD ϭ .07).
1 The same general pattern of results was observed when we remove these students from the analyses. When we exclude the two students the only difference is that the effect of condition on the Force Concept Inventory (FCI) becomes marginal.

958

ZEPEDA, RICHEY, RONEVICH, AND NOKES-MALACH

Week(s)
1 Day 1
6 Day 2 Day 3
Day 4
13-15 Day 5

15-17

Day 6 Day 7 Day 8 Day 9

18-19

Day 10 Day 11 Day 12 Day 13
Day 14 Day 15

Day 16

Day 17 20
Day 18
22 Day 19

29 Day 20

30 Day 21

Control Condition

Experimental Condition

Pretests • Conceptual Knowledge of Physics: Force Concept Inventory (45 minutes; Hestenes et al., 1992)

• Metacognition: Metacognitive Awareness Inventory (20 minutes; Schraw & Dennison, 1994) • Task Value, Control of Learning Beliefs, Self-Efficacy and Effort Regulation: Motivated Strategies for Learning
Questionnaire (20 minutes; Pintrich et al., 1991) • Achievement Goals: Achievement Goal Questionnaire – Revised (Elliot & Murayama, 2008) • Beliefs about Intelligence: Theories of Intelligence (Dweck, 1999) – 20 minutes • Engagement in Thinking: Need for Cognition (20 minutes; Cacioppo et al., 1996)

Round One: Puzzles

• More puzzle problems • Received a worked example of the initial problem • No direct instruction or worked examples of
metacognition • Practiced solving problems without metacognitive
prompts • Received solutions to all problems
Problem Solving 1 (45 minutes)

• Fewer puzzle problems • Received a worked example of the initial problem • Direct instruction and worked examples of
metacognition • Practiced solving problems with metacognitive
prompts • Received solutions to all problems
Planning and Problem Solving 1 (45 minutes)

Problem Solving 2 (45 minutes)

Monitoring and Problem Solving 2 (45 minutes)

Problem Solving 3 (45 minutes)

Evaluating and Problem Solving 3 (45 minutes)

Problem Solving 4 (45 minutes)

Integration and Problem Solving 4 (45 minutes)

Round Two: Physics

• More physics problems • Received a worked example of the initial problem • No direct instruction or worked examples of
metacognition • Practiced solving problems without metacognitive
prompts • Received solutions to all problems
Problem Solving 5 (45 minutes)

• Fewer physics problems • Received a worked example of the initial problem • Direct instruction and worked examples of
metacognition • Practiced solving problems with metacognitive
prompts • Received solutions to all problems
Planning and Problem Solving 5 (45 minutes)

Problem Solving 6 (45 minutes)

Monitoring and Problem Solving 6 (45 minutes)

Problem Solving 7 (45 minutes)

Evaluating and Problem Solving 7 (45 minutes)

Problem Solving 8 (45 minutes)

Integration and Problem Solving 8 (45 minutes)

Posttests • Metacognition: Metacognitive Awareness Inventory (20 minutes; Schraw & Dennison, 1994) • Task Value, Control of Learning Beliefs, Self-Efficacy and Effort Regulation: Motivated Strategies for Learning
Questionnaire (20 minutes; Pintrich et al., 1991) • Achievement Goals: Achievement Goal Questionnaire – Revised (Elliot & Murayama, 2008) • Beliefs about Intelligence: Theories of Intelligence (Dweck, 1999) – 20 minutes • Engagement in Thinking: Need for Cognition (20 minutes; Cacioppo et al., 1996)

• PFL Task: Control of Variables Strategy Activity (45 minutes)

• Conceptual Knowledge of Physics and Metacognitive Monitoring: Force Concept Inventory (50 minutes; Hestenes et al., 1992) with Confidence Ratings Delayed Posttests
• Preparation for Future Learning Task – Retention Rate: Delayed Transfer Test of the Control of Variables Strategy Activity (25 minutes)
• Metacognition Reflection: Declarative Knowledge and Utility Defined (30 minutes)

Figure 3. Outline of the procedure by condition as indicated by the first row.

Packets 1– 4: control. The control materials also consisted of puzzle problems (see online Supplemental Materials for examples). The packets did not include any instruction on planning, monitoring, or evaluating, but instead instructed students that they could improve their general problem-solving skills by working through the packets. The initial problem for each packet was the same as in the experimental materials. Following the initial problem, packets were divided into sections of problems. At the end of each section, students were given solutions and encouraged to check their answers before continuing on to the next section of problem solving. Piloting work revealed that the problems within each packet differed in the amount of time it took to complete them; consequently, the first packet had seven problems, the second had 13, the third had 16, and the fourth had seven. We gave the control condition a sufficient amount of problems to ensure they did not finish before

the experimental condition. The control packets concluded with the same transfer problems and packet quizzes as the experimental packets. On average the control condition completed 87% of the packet materials (SD ϭ .12).
In round two of the intervention, we integrated the instruction from round one into a series of physics problems that were adapted from the students’ physics textbook (Hsu, 2005). Each packet focused on different physics concepts about which the students had previously received instruction. The first packet consisted of problems that required students to calculate the average speeds of two objects. The second packet contained conservation of momentum problems. The third packet contained problems that required students to apply Newton’s second law to calculate speed, acceleration, and distance for a single falling object. The fourth packet required students to apply Newton’s second law

DIRECT INSTRUCTION OF METACOGNITION

959

Table 1 Definition, Questions, and Practice Procedure for the Experimental Group

Packet Planning Monitoring
Evaluating

Definition Understanding the problem, identifying the
goal, and strategizing to create a plan.
Thinking about where one is on the path to solving the problem in order to monitor progress toward the goal.
Comparing the answer to the problem’s goal, and looking to see which strategies worked best to evaluate the solution.

Example Prompting Questions Does the problem make sense? Are there similar problems you have
encountered that can help you decide which strategies to use? Are you close to the solution? What step are you on? Are your reasons for taking each step moving you closer to the goal?
Does my solution reach the problem’s goal(s)?
Does my solution make sense?

Practice Procedure
Before starting to solve a problem, students wrote how they planned to solve the problem.
While problem solving, students stopped to check their progress, looking for any errors, and answered a progress check:
1. Do you know what step you are on? 2. Why are you on that step? 3. How close to the solution do you think you
are? 4. Is your approach working? 5. How do you know you are on the right path? 6. Are you going to start over with a different
strategy?
After students solved the problem, they checked their solution to make sure it made sense by answering an evaluation check:
1. What was the goal(s) of the problem? 2. How does your solution meet the goal(s)? 3. Is your solution correct? 4. Why?

to calculate weight, acceleration, and time for pairs of falling objects.
Similar to the first round, each of the packets began with an initial problem (see Table 2), followed by a hint with an additional opportunity to solve that problem, followed by a solution and explanation. The control group then received several sets of isomorphic physics problems with solutions given at the end of each set, while the experimental group received the same direct instruction and practice of metacognitive skills from round one. Again, students in the control group received sufficient problems to make sure they would not finish before the experimental condition. The first packet had five problems, the second had seven, the third had nine, and the fourth had four. For the experimental condition, the first packet emphasized planning, the second monitoring, the third evaluation, and the fourth integrated the three skills. The instructional materials were identical to the first round of the intervention, except that the planning packet for the experimental condition required students to create a plan before solving the initial problem. At the end of each packet, both conditions received a packet

quiz identical to the one given in the first round of the intervention to assess their declarative knowledge of the targeted metacognitive skill. On average the control condition completed 76% of the packet materials (SD ϭ .18) and the experimental condition completed 85% of the packet materials (SD ϭ .08). Although the control condition completed proportionally less of the packet materials, on average they solved four more problems per packet than the experimental condition.
Scoring of packets. The initial and transfer problems within the intervention packets were used to evaluate how well the students were able to transfer what they had learned in the packets to problems that required similar problem-solving procedures. To investigate performance on these near-transfer problems, we scored each transfer problem from the first round as either incorrect (0) or correct (1). We scored each transfer problem from the second round as either incorrect (0), partially correct if students had either the correct number or the correct unit (.5), or completely correct (1) if they had both the correct number and unit. Since the integration packet for each round had two transfer problems, we

Table 2 An Example of the Initial, Isomorphic, and Near-Transfer Problems for the First Packet, Round 2

Initial
Sam and Mitch wanted to know who would win if they raced against each other. They know that during a 6.2-mile race, Sam runs at a constant speed of 6 miles per hr. They also know that during a 10-kilometer race, Mitch finishes in 50 min. (1 kilometer ϭ .62 mile)
Who would win if they raced against each other? Explain how you arrived at your answer.

Isomorphic
Marcus can swim 40 laps in 50 min, while John can swim at a constant rate of 1.75 miles per hr. (32 laps ϭ 1 mile)
If Marcus and John race in a swim meet, who will win?

Transfer
Sarah rides her bicycle at 10 miles per hr for 3 hr. She stops to rest for 1 hr and then continues her ride. For the next hour, she rides at a speed of 20 miles per hr.
During this ride, what was the rider’s average speed for the 5-hr period?

960

ZEPEDA, RICHEY, RONEVICH, AND NOKES-MALACH

calculated accuracy on those packets by dividing students’ scores by two. Transfer problems from round two also had multiple questions, and each of these questions was scored for accuracy and then divided by the total by the number of questions per problem. After these calculations, the scores for each round were summed for a total of four points each (1 point for each packet) and divided by four to give a proportion of accuracy. The transfer scores ranged from 0 to 0.88 for both rounds.
Scoring of packet quizzes. For each of the eight packets, we scored the quiz to evaluate students’ declarative knowledge about the metacognitive skills. Within the quiz there were two types of questions, open-ended and multiple-choice. The open-ended questions asked students to state important steps to take while problem solving. We gave one point for every key term or phrase that matched or was synonymous to a metacognitive concept from the corresponding experimental packet (see Table 3). For example, if a student wrote the words “plan” and “create a goal” on the planning packet then she would receive two points. If a student used a synonymous phrase such as “think about how to approach the problem” and “figure out what the problem is asking for,” she would also receive two points. We calculated the number of points each participant received across all four packets for each round and divided by the total number of points possible (five for planning, four for monitoring, four for evaluating, and 13 for integrating). Two raters independently coded all the open-ended responses and reached adequate reliability (␬ Ͼ .70). The two raters discussed and resolved all disagreements. The open-ended scores ranged from 0 to 0.42 for the first round and 0 to 0.38 for the second.
There were also two types of multiple-choice questions. The first type had a single best answer and was scored as correct (1), partially correct (.5), or incorrect (0). The second type had multiple correct responses (checklist) and was scored as the proportion of correctly selected and correctly rejected alternatives out of the total number of alternatives (scores ranged between 0 and 1 per problem). To calculate the accuracy score per round, we divided the sums by the total number of points possible for that round (14).

The multiple-choice scores ranged from 0.16 to 0.77 for the first round and 0.20 to 0.71 for the second.
Pretests and Posttests
Conceptual knowledge test. Students’ conceptual understanding of Newtonian physics was measured with Hestenes, Wells, and Swackhamer’s (1992) Force Concept Inventory (FCI). The FCI consists of 29 multiple-choice questions, each describing an applied situation or scenario in which the student must qualitatively reason about the relevant physics concepts to determine the correct answer (␣ ϭ .69). Some of the concepts targeted by the FCI were covered in the packet problems, while others were not. This assessment provides a strong test of conceptual understanding as students had to distinguish the correct answer from four alternatives that capture common physics misconceptions. The FCI was given before and after the intervention, with both versions identical except that the posttest asked students to rate their confidence after each answer on a scale, ranging from one (not confident at all) to five (very confident). This confidence rating enabled us to examine whether the intervention improved students’ metacognitive accuracy in assessing their physics knowledge on a difficult conceptual test.
Scoring. The FCI was analyzed to evaluate whether the intervention affected how well students performed on a far-transfer task. We scored each question as either correct or incorrect, and then calculated the proportion of questions students answered correctly by dividing the sum of correct responses by the total number of items. On the pretest, scores ranged from 0.14 to 0.53 and on the posttest scores ranged from 0.18 to 0.86.
We also used the confidence ratings (␣ ϭ .92) and accuracy scores from the posttest to assess metacognitive monitoring using absolute accuracy and two forms of relative accuracy (for a distinction see Schraw, 2009, Table 4). Absolute accuracy (sometimes referred to as calibration) measures the difference between a student’s confidence judgment and the corresponding accuracy

Table 3 Scoring for the Open-Ended Questions

Instructional Packet Planning
Monitoring Evaluating Integrating

Phrases Awarded Points
1. Mentions the process of creating a plan 2. Mentions creating a goal, using goals, goal(s) 3. Mentions making sure they understand the problem/what is being
asked of them 4. Mentions using strategies or processes 5. Mentions the rules of the problem, the limitations of the problem
1. Mentions monitoring or staying on track 2. Specifies monitoring in terms of the solution, goal, or understanding 3. Mentions providing reasoning 4. Mentions understanding why you complete a step or the logic
behind the step
1. Mentions checking and comparing or assessing answers 2. Mentions checking to make sure your solution makes sense 3. Mentions that there are multiple solutions and/or strategies and you
have to figure out the best one 4. Mentions helping them in the future
All of the above

DIRECT INSTRUCTION OF METACOGNITION

961

Table 4 Formulas Used to Calculate Absolute Accuracy, Relative Accuracy: Gamma and Relative Accuracy: Discrimination

Type of Equation Absolute accuracy Relative accuracy: Gamma
Relative accuracy: Discrimination

Equation

͚1 N ͑ci Ϫ pi͒2 N iϭ1 NSϪND NSϩND where NS is the number of concordant pairs

and ND is the number of discordant pairs

͚ͫ ͚ ͬ 1 Nc

Ni

N iϭ1 ͑cicorrect͒ Ϫ iϭ1 ͑ciincorrect͒

score. This measure assesses metacognitive precision. A smaller deviation indicates greater accuracy. Gamma is a form of relative accuracy (sometimes referred to as resolution) and measures the agreement between a student’s confidence judgments and accuracy on each item in relationship to other items (Goodman & Kruskal, 1954). This measure assesses whether a student is consistent with her metacognitive judgments regardless of absolute accuracy. Discrimination is also a form of relative accuracy, but this measure assesses whether a student can differentiate between correct versus incorrect performance. Positive discrimination scores indicate that a student has higher confidence in correct versus incorrect items, whereas negative discrimination scores indicate that a student has higher confidence in incorrect versus correct items. Using multiple measures can provide greater insight into the monitoring process because different measures could be differentially affected by underlying mechanisms such as feedback and motivation (Schraw, Kuch, & Gutierrez, 2013). For instance, prior research has shown that absolute accuracy and relative accuracy are unrelated and that absolute accuracy tends to be sensitive to task difficulty and student ability whereas Gamma is not (Maki, Shields, Wheeler, & Zacchilli, 2005; Kelemen, Frost, & Weaver, 2000).
Metacognition. Students’ beliefs about metacognition were assessed with three components of the Metacognitive Awareness Inventory (MAI) including seven items on planning (“I set specific goals before I begin a task”), seven items on monitoring (“I ask myself questions about how well I am doing while I am learning something new”), and six on evaluating (“I ask myself how well I accomplish my goals once I’m finished”; Schraw & Dennison, 1994). Students rated how characteristic they thought each statement was of themselves on a five-point Likert scale from one (extremely uncharacteristic) to five (extremely characteristic).2 The MAI scores had high internal reliability at pretest (␣ ϭ .85) and posttest (␣ ϭ .92).
Self-efficacy, value, effort regulation, and control of learning beliefs. Students completed four components of the Motivated Strategies for Learning Questionnaire (MSLQ; Pintrich et al., 1991) including: (1) seven items for self-efficacy, for example, “I’m confident I can understand the basic concepts taught in this class,” (2) six items for task value, for example, “I think the material in this class is useful for me to learn,” (3) four items for strategies of effort regulation, for example, “I work hard to do well in this class even if I don’t like what we’re doing,” and (4) three items for control of learning beliefs, for example, “If I don’t

understand the class material, it is because I didn’t try hard enough.” Students rated how much they agreed or disagreed with each of the items on a seven-point Likert scale from one (strongly disagree) to seven (strongly agree). Scores from the measures of self-efficacy (␣ ϭ .87) and task value (␣ ϭ .78) had adequate reliability at the pretest whereas scores from the measures of effort-regulation (␣ ϭ .65) and control of learning beliefs (␣ ϭ .45) had low reliability. At posttest, scores from the self-efficacy (␣ ϭ .87), task value (␣ ϭ .82), and effort-regulation (␣ ϭ .71) measures had adequate reliability, but scores from the control of learning beliefs measure (␣ ϭ .63) had low reliability. Due to the low reliability of scores from the control of learning beliefs and effort regulation measures, we do not analyze these results further.
Goal orientations. Students’ achievement goal orientations were assessed using the Achievement Goal Questionnaire – Revised (AGQ-R; Elliot & Murayama, 2008). The questionnaire evaluates students’ endorsement of mastery-approach (“My aim is to completely master the material presented in class”), masteryavoidance (“My aim is to avoid learning less than I possibly could”), performance-approach (“My aim is to perform well relative to other students”), and performance-avoidance goals (“My aim is to avoid doing worse than other students”) with three items per orientation. Students rated how much they agreed or disagreed with each item on a seven-point Likert scale from one (strongly disagree) to seven (strongly agree). The scores from the masteryapproach, performance-approach, and performance-avoidance measures had high reliability at pretest (ranging from ␣ ϭ .89 to ␣ ϭ .91) and posttest (ranging from ␣ ϭ .81 to ␣ ϭ .90). Mastery-avoidance scores had low reliability at pretest (␣ ϭ .38) and high reliability at posttest (␣ ϭ .81), and it was therefore excluded from further analyses.
Theories of intelligence. We used the six-item Implicit Theories of Intelligence Scale for Children – Self-Form to assess students’ beliefs about intelligence (Dweck, 1999). This survey measures the degree to which students think intelligence is malleable (incremental) or fixed (entity). Students rated how much they agreed or disagreed with three entity theory and three incremental theory statements on a seven-point Likert scale from one
2 Although the original survey comprised several other metacognitive components, we assessed only planning, monitoring, and evaluation because our intervention explicitly targeted these three components.

962

ZEPEDA, RICHEY, RONEVICH, AND NOKES-MALACH

(strongly disagree) to seven (strongly agree; pretest: ␣ ϭ .94, posttest: ␣ ϭ .89). Entity questions were reverse scored to calculate a single score across the six items, with low values indicating greater endorsement of entity beliefs and higher values indicating greater endorsement of incremental beliefs, for example, “No matter who you are, you can change your intelligence a lot.”
Need for cognition. Students’ enjoyment of and desire to engage in cognitive activities that require large amounts of thinking and understanding were assessed by the 18-item Need for Cognition Scale, for example, “I like to have the responsibility of handling a situation that requires a lot of thinking” (NCS; Cacioppo et al., 1996). Students rated how characteristic or uncharacteristic each item was of themselves using a five-point Likert scale from one (extremely uncharacteristic) to five (extremely characteristic). The NCS scores had high reliability at pretest (␣ ϭ .90) and posttest (␣ ϭ .91).
Posttests and Delayed Posttests
Preparation for future learning—Control of variables strategy. We created a preparation for future learning activity based on Siler, Klahr, and Price’s (2012) “control of variables strategy” (CVS) instructional materials. These materials are designed to teach students how to create valid experiments, which is critical for reasoning about variables and confounds when designing experiments and interpreting results (Chen & Klahr, 1999). Although CVS is a cornerstone of experimental design, many students struggle to understand and use it in science class. For our purposes, this activity served as a preparation for future learning measure as the students could apply their newly acquired metacognitive knowledge and skills to learn from self-guided materials about CVS. Critically, the PFL activity differed from problems covered in the intervention and class in terms of both content and structure, which allowed us to gauge how the intervention prepared students to learn a novel concept and skill.
During the PFL activity, students completed pretest, learning, and posttest packets. The pretest asked students to design four valid experiments and evaluate whether an experiment was correctly designed. Following the pretest packet, students worked through a nine-page learning packet that explained how to create a valid experiment through an analysis of a well-designed and poorly designed pair of experiments. Students were given a series of questions that asked them to explain why the experiments were well or poorly designed, and they were shown how to correct the confounded experiment.
After the learning packet, the students completed a posttest packet with the same four design questions from the pretest, two new questions asking them to evaluate an experimental design, and three new questions asking them to generate an experiment testing whether a specific variable affected an outcome (see online Supplemental Materials). After each of the new questions, students were asked to explain their answers, with the option of saying they guessed after each of the “generate” questions.
Approximately 2 months after completing the PFL activity, students received a delayed transfer test on creating valid experiments. Similar to the posttest, the delayed transfer test contained two new questions asking students to evaluate an experimental design and two new questions asking them to generate an experiment testing a specific variable. Students were again prompted to

explain their answers, with the option of saying that they guessed after each of the “generate” questions.
Scoring. To investigate the effect of condition on performance on the PFL activity, we calculated the number of questions answered correctly for the pretest, learning, posttest and delayed transfer packets as well as the number of times they guessed during the activity. The five pretest items were coded as either correct or incorrect. Pretest scores (␣ ϭ .92) were summed for a maximum of five points and divided by five to calculate the proportion of accuracy, ranging from 0 to 1. We combined the scores on the learning packet (two questions), the posttest packet (nine questions), and the delayed transfer packet (four questions) to provide a simple measure of PFL and to have enough data points for parametric analyses. PFL scores (␣ ϭ .86) were summed for a maximum of 15 points and divided by 15 to give a proportion of accuracy, ranging from 0.13 to 1. Students also had six opportunities to say they guessed during the activity (two per phase during learning, posttest, and transfer phases). We calculated the number of times they reported guessing across the six problems and divided it by six to calculate the proportion of reported guessing, which ranged from 0 to 0.67.
Reflection survey. Students received an opportunity to reflect on their understanding and the utility of the training intervention a week after completing the delayed PFL transfer test. Both groups answered a multiple-choice question about important steps of problem solving (i.e., metacognitive skills) and a question about whether they found the problem-solving activities useful. The metacognitive skills question was scored dichotomously as either correct or incorrect and the usefulness of the activities was scored as useful (1) or not (0). They also answered questions using a Likert scale from one to five about the frequency (never, rarely, sometimes, a lot, all the time) with which they generally used metacognitive skills across different contexts as well as the frequency with which they used each skill during problem solving.
Procedure
This experiment took place over 30 weeks throughout the academic school year with 21 days of activities. The procedure is divided into three time frames: before the intervention (pretest), the intervention, and after the intervention (posttest). For an overview of the experimental procedure, see Figure 3. The teacher covered motion and forces in the first part of the year and had completed all relevant physics content before the intervention began.
Before the intervention. Twelve weeks before the intervention, students were given the FCI (Hestenes et al., 1992) to measure their conceptual knowledge in physics. Students were told to try their best and were given 45 min to complete the test. At the time the pretest was given, the teacher had covered some of the content on the FCI (e.g., basic motion and Newton’s First Law) but not all of it. This timing was selected so that students would have some context for understanding the questions but also to identify misconceptions from everyday experience. Once the FCI was completed, the teacher covered more of the material from the FCI (e.g., Newton’s Second and Third Laws). Nine weeks later students completed a series of self-report surveys designed to assess key aspects of their metacognition, learning strategies, and motivation during a 4-day period, taking approximately 20 min each

DIRECT INSTRUCTION OF METACOGNITION

963

day. To ensure that the students took the survey materials seriously, the teacher told them he would use the survey materials to better understand how they were learning to improve his teaching.
Intervention. After the pretests, the teacher administered the instructional intervention over the course of 4 weeks. The intervention consisted of eight learning packets for both the experimental and control conditions. Each week students completed one packet every Tuesday and Thursday. Students were given 45 min in class to work on each packet. The packets were self-guided but had a few subcomponents that were timed by the teacher (see online Supplemental Materials for packet timings). In total, students spent 6 hours on the intervention materials.
After the intervention. The week after the intervention, students were given the same set of self-report surveys and the PFL scientific reasoning task. Similar to the pretest procedure, students were given the surveys in the same order across 4 days, with 20 min each day to complete them. On the day after the last set of posttest surveys, students were given the PFL activity. This activity was administered in packet form and included pretest, learning, and posttest packets. Students were given 45 min total with 15-min blocks for each packet.
A week and a half later, students were given the FCI posttest. Students were given 50 min to complete the test in class. Seven weeks after that, students spent approximately 20 min completing a delayed transfer test on the PFL scientific reasoning task. A week later, students were given 20 min to complete a shortanswer and multiple-choice assessment of their use of metacognition including planning, monitoring, and evaluation.
Results
To test the effect of instructional condition on posttest performance, we conducted analyses of covariance (ANCOVAs) controlling for students’ pretest scores. This approach provides a test of instructional condition on metacognition, learning, transfer, and motivational outcomes while controlling for students’ initial scores on the construct of interest. We implemented this approach for all analyses with the exception of the packet quizzes, confidence ratings, and metacognitive reflections, as we did not have any pretest measures for these materials. Instead we used analyses of variance (ANOVAs) and chi-square tests, where appropriate. We set the alpha level at .05 and report effects for p values less than .05 (Keppel & Wickens, 2004). Assumptions underlying ANCOVAs and ANOVAs were tested. For cases in which the assumptions were not met, we note them below. For all effects, we report effect sizes (Cohen’s d or partial eta squared, ␩p2). We interpret effects as small when ␩p2 Ͻ 0.06 or d Ͻ 0.2, medium when 0.06 Ͻ ␩p2 Ͻ 0.14 or 0.3 Ͻ d Ͻ 0.8, and large when ␩p2 Ͼ 0.14 or d Ͼ 0.8 (see Cohen, 1988; Olejnik & Algina, 2000). See Table 5 for a summary of all the results with descriptives and effect sizes.
Performance on Learning Packets
Problem solving. To examine whether there was an effect of condition on near-transfer problems we used a 2 ϫ 2 mixed-design ANOVA with a within-subjects factor of round (1 vs. 2) and a between-subjects factor of condition (experimental vs. control). Analyses revealed a medium effect of condition, F(1, 44) ϭ 6.13, p ϭ .02, ␩p2 ϭ .12, with the experimental condition performing

better than the control condition across both rounds. There was no effect of round, F(1, 44) ϭ 0.43, p ϭ .52, ␩p2 ϭ .01, nor was there an interaction between round and condition, F(1, 44) ϭ 0.01, p ϭ .92, ␩p2 ϭ .00 (see Figure 4).
Packet quizzes. To investigate the effect of condition on the open-ended questions for rounds one and two, we conducted a 2 ϫ 2 mixed-design ANOVA with round of the intervention as a within-subjects factor and condition as a between-subjects factor. Analyses revealed a large effect of condition, F(1, 44) ϭ 11.48, p ϭ .001, ␩p2 ϭ .21, with the experimental condition receiving more points than the control condition on the open-ended questions across both rounds. There was no effect of round, F(1, 44) ϭ 0.001, p ϭ .97, ␩p2 ϭ .00, nor was there an interaction between round and condition, F(1, 44) ϭ 0.01, p ϭ .92, ␩p2 ϭ .00 (see Figure 4).
We also tested the effect of condition on the multiple-choice questions for rounds one and two, using a 2 ϫ 2 mixed-design ANOVA with a within-subjects factor of round and a betweensubjects factor of condition. Results revealed a large effect of condition, F(1, 44) ϭ 10.35, p ϭ .002, ␩p2 ϭ .19, with the experimental condition performing better than the control condition across both rounds. There was no effect of round, F(1, 44) ϭ 0.19, p ϭ .68, ␩p2 ϭ .00, nor was there an interaction between round and condition, F(1, 44) ϭ 0.83, p ϭ .37, ␩p2 ϭ .02 (see Figure 4). These results provide evidence that the experimental group acquired declarative knowledge of the metacognitive skills presented in the learning packets.
Self-Report Surveys
Metacognitive Awareness Inventory. The MAI measured students’ metacognitive awareness on a scale from one to five, with higher scores indicating higher levels of metacognitive awareness. An ANCOVA was conducted to examine the effect of condition on metacognitive awareness. Analyses revealed a large effect of the covariate, F(1, 43) ϭ 75.08, p Ͻ .001, ␩p2 ϭ .64, showing that students’ MAI pretest scores predicted their posttest scores. There was no effect of condition, F(1, 43) ϭ 0.83, p ϭ .37, ␩p2 ϭ .02.3 Contrary to our predictions, these results show that the metacognitive intervention did not enhance participants’ metacognitive awareness as measured by this survey compared to the control.
Task value. Task value measured how much students valued classroom tasks. Testing the effect of condition while controlling for prior task value, an ANCOVA revealed that there was a large effect of the covariate, F(1, 43) ϭ 48.98, p Ͻ .001, ␩p2 ϭ .53, which indicates that students’ pretest scores predicted their posttest scores. There was also a large effect of condition, F(1, 43) ϭ 8.40, p ϭ .006, ␩p2 ϭ .16, with the experimental condition placing a higher value on learning the class materials than the control group.
Self-efficacy. Self-efficacy measured how much students believed they were able to accomplish classroom tasks. An ANCOVA revealed that there was a large effect of the covariate, F(1, 43) ϭ 46.32, p Ͻ .001, ␩p2 ϭ .52, showing that students’ pretest scores predicted their posttest scores. There was also a large effect of condition, F(1, 43) ϭ 8.22, p ϭ .006, ␩p2 ϭ .16, with the experi-
3 When the planning, monitoring, and evaluation components of the MAI were analyzed separately there was no effect for any subcomponent.

Preparing to load PDF file. please wait...

0 of 0
100%
Direct Instruction of Metacognition Benefits Adolescent