PIRLS 2001 User Guide for the International Database

PIRLS 2001 User Guide
for the International Database
Edited by Eugenio J. Gonzalez Ann M. Kennedy

Contributors Jay Campbell Pierre Foy Eugenio J. Gonzalez Dirk Hastedt Ursula Itzlinger Marc Joncas Dana L. Kelly Barbara Malak Michael O. Martin Ina V.S. Mullis Marian Sainsbury Knut Schwippert
PIRLS PIRLS 2001 User Guide for the International Database
Edited by Eugenio J. Gonzalez Ann M. Kennedy

© 2003 International Association for the Evaluation of Educational Achievement (IEA)
PIRLS 2001 User Guide for the International Database / Edited by Eugenio J. Gonzalez and Ann M. Kennedy
Publisher: International Study Center Lynch School of Education, Boston College
Library of Congress Catalog Card Number: 2003109101
ISBN 1-889938-31-9

For more information about PIRLS contact:

PIRLS International Study Center Lynch School of Education Boston College Chestnut Hill, MA 02467 United States

tel:

+1-617-552-1600

fax:

+1-617-552-1203

e-mail: [email protected]

http://www.pirls.bc.edu

Boston College is an equal opportunity, affirmative action employer.
Printed and bound in the United States.

Table of Contents

1 Overview of User Guide
1.1 Overview of User Guide

1.2 Overview of PIRLS

1.3 Participating Countries

1.4 Student Population Assessed

1.5 Assessment Dates

1.6 Study Management and Organization

1.7

Overview of Assessment Framework

Exhibit 1.1

Percentages of Reading Assessment Devoted to Reading Purposes and Processes

1.8 PIRLS Reading Assessment
Assessment Design Passages Items and Scoring Guides Releasing Assessment Material to the Public
1.9 PIRLS Background Questionnaires
Student Questionnaire Learning to Read (Home) Survey Teacher Questionnaire School Questionnaire
1.10 Translation and Verification of Instruments
1.11 Data Collection
1.12 Scoring the Constructed-Response Items
1.13 Data Processing
1.14 IRT Scaling
1.15 Data Analysis and Reporting
1.16 The Trends in IEA’s Reading Literacy Study

1-2 1-3 1-4 1-4 1-5 1-5 1-6
1-7
1-7
1-7 1-8 1-8 1-8
1-8
1-9 1-9 1-9 1-9
1-10 1-10 1-11 1-11 1-11 1-12 1-13

PIRLS 2001 Database User Guide

iii

2

PIRLS Test Instruments and Booklet Design

2.1 Overview

2-2

2.2

The PIRLS Assessment Framework
Text Types Processes and Strategies Test Booklet Design Exhibit 2.1 Distribution of Literacy and Informational Blocks Across Booklets

2-2
2-3 2-4 2-5 2-6

2.3 Released Status for PIRLS Passages

2-6

3
3.1 3.2 3.3
3.4

PIRLS Background Questionnaires

Overview

Framework for the Questionnaires
Exhibit 3.1 Contexts for the Development of Reading Literacy

Exhibit 3.2

Factors within the Home, School, and National and Community Contexts Addressed by PIRLS

3-2 3-2
3-3
3-3

PIRLS Main Survey Questionnaires
Student Questionnaire Exhibit 3.3 Content of the PIRLS Student Questionnaire Learning-to-Read Survey (Home) Exhibit 3.4 Content of the PIRLS Learning-to-Read Survey (Home Questionnaire) Teacher Questionnaire Exhibit 3.5 Content of the PIRLS Teacher Questionnaire School Questionnaire Exhibit 3.6 Content of the PIRLS School Questionnaire

3-4
3-4 3-5 3-6 3-6 3-7 3-7 3-9 3-9

Reporting Questionnaire Data

Summary Indices from Background Data

Exhibit 3.7

Summary Indices from Background Data in the PIRLS 2001 International Report

Reporting Student Questionnaire Data Reporting Teacher Questionnaire Data Reporting Parents’ Questionnaire Data Reporting School Questionnaire Data

3-11
3-11
3-13 3-16 3-16 3-17 3-17

iv

PIRLS 2001 Database User Guide

4 PIRLS Survey Operations Procedures
4.1 Overview
4.2 Responsibilities of the National Research Coordinator
4.3 Within-School Sampling Procedures
Survey Tracking Forms Excluding Students from Testing Assigning Instruments to Students and Teachers
4.4 Within-School Assessment Activities
Arranging the Testing Sessions Distributing Materials Test Administration Timing of the Testing Sessions Activities Following the Test Administration
4.5 Scoring the Constructed-Response Items
Preparing Materials to Train the Scorers Documenting the Reliability of the Constructed-Response Scoring
4.6 Trends in IEA’s Reading Literacy Study
5 Sampling and Sampling Weights
5.1 Overview
5.2 PIRLS Target Population
Sampling from the Target Population
5.3 Sample Design
Units of Analysis and Sampling Units Sampling Precision and Sample Size Stratification
5.4 Trends in IEA’s Reading Literacy Study
Trends in IEA’s Reading Literacy Study Target Population Sample Design
5.5 Sampling Weights
Weight Variables Included in the Student Data Files Weight Variables Included in the Student-Teacher Linkage Files Weight Variables Included in the School Data Files

4-2 4-2 4-3
4-4 4-4 4-5
4-6
4-6 4-6 4-6 4-7 4-8
4-8
4-8 4-9
4-10
5-2 5-3
5-3
5-5
5-5 5-6 5-7
5-8
5-8 5-9
5-9
5-10 5-13 5-13

PIRLS 2001 Database User Guide

v

5.6 Other Sampling Variables Included in the Student and Student-Teacher Link Files
5.7 Using Sampling Variables in the Analysis

5-14 5-15

6
6.1 6.2 6.3
6.4

Scaling Methodology and Achievement Scores

Overview

PIRLS 2001 Scaling Methodology
Two- and Three-Parameter IRT Models for Dichotomous Items The IRT Model for Polytomous Items Proficiency Estimation Using Plausible Values Generating Proficiency Scores Working with Plausible Values

Student Achievement Scores
Achievement Scores in the Student Files Exhibit 6.1 Plausible Values Based on PIRLS 2001 Content Area

Exhibit 6.2 Plausible Values Based on Reading Literacy Study Content Area

Exhibit 6.3 Raw Scores Included in the PIRLS Data Files

Exhibit 6.4 Raw Scores Included in the Reading Literacy Study Data Files

Exhibit 6.5 Standardized Scores Included in the PIRLS Data Files

Exhibit 6.6

Standardized Scores Included in the Reading Literacy Study Data Files

Exhibit 6.7 National Rasch Scores Included in the PIRLS Data Files

Exhibit 6.8

National Rasch Scores Included in the Reading Literacy Study Data Files

International and National Benchmarks of Achievement
Exhibit 6.9 PIRLS International Benchmarks of Reading Achievement

Exhibit 6.10

PIRLS International Benchmarks of Reading for Literary Purposes Achievement

Exhibit 6.11

PIRLS International Benchmarks of Reading to Acquire and Use Information Achievement

Exhibit 6.12

Variables Indicating Achievement of International Benchmarks of Reading to Acquire Information Achievement

Exhibit 6.13

Variables Indicating Achievement of National Benchmarks of Reading to Acquire Information Achievement

6-2 6-3
6-3 6-4 6-6 6-8 6-9
6-11
6-11 6-12 6-12 6-13 6-13 6-14
6-14 6-15
6-15
6-16
6-16
6-17
6-17
6-17
6-19

vi

PIRLS 2001 Database User Guide

7 Estimating Sampling and Imputation Variance
7.1 Overview 7.2 Estimating Sampling Variance
Computing Sampling Variance Using the JRR Method Construction of Sampling Zones for Sampling Variance Estimation Computing the JRR Replicate Weights
7.3 Estimating Imputation Variance 7.4 Combining Sampling and Imputation Variance 7.5 Comparing Achievement with the International Mean 7.6 Gender Differences

8
8.1 8.2
8.3 8.4 8.5 8.6

Content and Format of Database Files
Overview
The Data Files
Data File and Variable Naming Conventions Exhibit 8.1 PIRLS and Reading Literacy Data Files Exhibit 8.2 Country Identification and Availability of Data Files Exhibit 8.3 Background Questionnaire Variable Location Conventions Coding Convention Types of Variables Included in the Data Files PIRLS Background Data Files PIRLS Student Achievement Data Files PIRLS Scoring Reliability Data Files PIRLS Student-Teacher Linkage Files Reading Literacy Student Background and Achievement Data Files
Data Almanacs
Exhibit 8.4 Data Almanac Files for PIRLS 2001 and Reading Literacy Study Background Item Data Almanacs Achievement Item Data Almanacs
Codebook Files
Program Files
Item Information Files

7-2 7-2
7-3 7-4 7-4
7-6 7-6 7-7 7-7
8-2 8-3
8-3 8-5 8-6 8-8 8-9 8-13 8-17 8-18 8-18 8-20 8-20
8-20
8-22 8-22 8-23
8-24 8-26 8-27

PIRLS 2001 Database User Guide

vii

9
9.1 9.2 9.3 9.4 9.5
9.6
9.7

Performing Analyses with the PIRLS Data Using SPSS

Overview

Contents of the CD

Creating SPSS System Files

Exhibit 9.1

Extract from SPSS Control Code for Creating a Student Background SPSS System File

9-3 9-4 9-6
9-7

Scoring the Items

Exhibit 9.2

Extracted Sections of SPSS Macro SCOREIT Used to Convert Cognitive Item Response Codes to Correctness-Score Levels

9-7
9-9

Basic Analyses with the PIRLS Data: Means, Percentages, Regression Coefficients, and Their JRR Standard Errors

Computing Sampling and Imputation Variance for Plausible Values Using SPSS (JACKPV.SPS)

Exhibit 9.3

SPSS Control Code and Extract of Output File for Using the Macro JACKPV.SPS

Computing Sampling Variance for Variables Other Than Plausible Values Using SPSS (JACKGEN.SPS)

Exhibit 9.4

SPSS Control Code and Extract of Output File for Using the Macro JACKGEN.SPS

Computing Regression Coefficients and Their JRR Standard Errors for Variables Other Than Plausible Values (JACKREG.SPS)

Exhibit 9.5

SPSS Control Code and Extract of Output File for Using the Macro JACKREG.SPS

Computing Regression Coefficients and Their JRR Standard Errors with Plausible Values (JACKREGP.SPS)

Exhibit 9.6

SPSS Control Code and Extract of Output File for Using the Macro JACKREGP.SPS

9-10
9-10 9-14 9-15 9-19 9-20 9-23 9-24 9-27

Conducting Analyses with the PIRLS 2001 International Database: Student Level

9-28

Example Analysis with Student-Level Variables Not Using Plausible Values 9-28

Exhibit 9.7

SPSS Control Statements for Performing Analyses with Student-Level Variables NOT Using Plausible Values (EXAMPLE1.SPS)

9-30

Exhibit 9.8

Extract of SPSS Computer Output for Performing Analyses with Student-Level Variables Not Using Plausible Values (EXAMPLE 1)

9-31

Example Analysis with Student-Level Variables Using Plausible Values

Exhibit 9.9

Sample Exhibit for Student-Level Analysis Involving Plausible Values Taken From the PIRLS 2001 International Report

9-31 9-32

Exhibit 9.10

SPSS Control Statements for Performing Analyses with Student-Level Variables Involving Plausible Values (EXAMPLE2.SPS)

9-33

Exhibit 9.11

Extract of SPSS Computer Output for Performing Analyses with Student-Level Variables Involving Plausible Values (EXAMPLE 2)

9-34

Performing Analyses with Teacher-Level Variables

Exhibit 9.12

Sample Exhibit for Teacher-Level Analysis Involving Plausible Values Taken From the PIRLS 2001 International Report

9-35
9-36

viii

PIRLS 2001 Database User Guide