Artificial Intelligent Based Video Analysis on the Teaching

International Journal of Information and Education Technology, Vol. 11, No. 3, March 2021

Artificial Intelligent Based Video Analysis on the Teaching Interaction Patterns in Classroom Environment
Kaiyue Lv, Zhong Sun, and Min Xu

 Abstract—Recently, the development of technology has
enriched the form of classroom interaction. Exploring the characteristics of current classroom teaching interaction forms can clarify the deficiencies of teaching interactions, thereby improving teaching. Based on the existing classroom teaching interactive coding system, this paper adopted ITIAS coding system, and took classroom with interactive whiteboard, interactive television or mobile terminals as research scene, selected 20 classroom videos of teaching cases in this environment as research objects. Computer vision, one of the artificial intelligent technologies was applied for video analysis from four aspects: the classroom teaching atmosphere, the teacher-student interaction, the student-student interaction, the interaction between human and technology. Through cluster analysis, three clusters of sample’s behavioral sequences were found. According to the analysis on the behavioral sequences and the behavioral transition diagram of each cluster, three classroom teaching interaction patterns were identified, including immediate interaction pattern, waiting interaction pattern and shallow interaction pattern.
Index Terms—Classroom interaction, artificial intelligent, interaction patterns, video analysis, lag sequential analysis.
I. INTRODUCTION
Interaction is the core of classroom teaching and the direct reflection of teaching effect [1]. Early studies of classroom interaction focused on observation and description of the functions and structure of teacher–student talk according to observation schemes [2]. Video as a tool can record the process of real classroom teaching [3]. And video analysis provides a new way for researchers to analyze and assess the classroom interaction. However, there has been a shift in classroom interaction assessment from an early focus on teacher talk to formative assessment of the interaction process [4]. The patterns of classroom interaction reflect its process and multifaceted construct [5].
With rapid development of information technology and education, the subject and form of interaction have gradually become richer, adding human-technology interaction. However, in technology-rich environment, what are the patterns of classroom interaction? And what are the characteristics of these patterns? These problems are rarely explored by scholars at present. In this context, based on summarizing the existing teaching behavior quantification
Manuscript received February 23, 2020; revised November 15, 2020. This research was partially funded by the National Science Foundation of China (Research on key technology of classroom teaching interactive analysis based on artificial intelligence, Grant Number: NSFC61977048).
The authors are with Capital Normal University, Beijing, China (corresponding author: Zhong Sun; e-mail: [email protected], [email protected], [email protected]).

tools, the paper adopted the ITIAS which is the abbreviation of Information Technology-based Interaction Analysis System, attempting to utilize ITIAS to record and analyze the interactive process of primary math classroom in the technology-rich environment. By using cluster analysis and lag sequential analysis to explore the teaching interaction patterns of technology-rich classroom environment and summarized their characteristics.
II. LITERATURE REVIEW
A. Video-Based Classroom Interaction Analysis
Early classroom interaction studies mainly focused on recording and describing teacher-student behavior and talk in the process of actual classroom teaching according to a certain observation framework. The S-T analysis method is to observe the actual teaching process and to quantify the distribution of teacher behaviors and student behaviors in classroom teaching according to the observation framework, so as to put forward suggestions for improving classroom teaching [6]. S-T analysis method only analyzes data from the two dimensions of teacher and student behavior, which reduces the ambiguity of behavior classification and facilitates practical application, but the data type is too simple to reflect the entire classroom teaching process. In the 1960s, Flanders proposed the Flanders Interaction Analysis System (FIAS), which is typical mature quantitative analysis tool for classroom teaching [7]. The analysis system can analyze different classroom teaching structures, models, and styles through a quantitative study of the recorded data. Although FIAS has its advantages, it pays too much attention to teachers' behavior in classroom teaching (7 categories), ignores student behavior (2 categories), and cannot reflect the role of technology in classroom teaching. The use of technology has grown up to an essential part of classroom teaching today, but human-technology interaction can’t be reflected in FIAS, which is treated as silence merely. Gu improved FIAS and proposed Information Technology-Based Interaction Analysis System (ITIAS) [8], which enriches students’ behavior and increases the interaction behaviors between human and technology.
In recent years, there has been a notable increase in the use of video as a tool for teacher reflection on classroom lessons and for scholars to analyze classroom interaction [9]. Researchers generally agree with that the use of video offers several benefits [10]. For example, it has the capacity to capture the richness and complexity of classroom dialogues in lesson activities without losing authenticity [11]. Besides, video can be played repeatedly and paused at any time giving researchers enough time to think and record the data of

doi: 10.18178/ijiet.2021.11.3.1500

126

International Journal of Information and Education Technology, Vol. 11, No. 3, March 2021

classroom teaching. Research on classroom interaction has gradually focused on the analysis of teaching videos according to observation frameworks. Gao et.al developed a visual learning analytics tool, the Classroom Discourse Analyzer (CDA), which offers one approach to employing visual learning analytics technologies to facilitate teacher analysis and learning about dialogic teaching [12]. Santagata and Angelici developed the Lesson Analysis Framework (LAF) for pre-service teachers to learn from videos of classroom teaching [13].
B. Classroom Teaching Interaction patterns
Formative evaluation of interaction processes is one of the shifting trends of classroom interaction [14]. Classroom interaction is a multifaceted construct, and teachers interact with their students in patterned ways [15]. Research focusing on classroom interactions suggests that teaching interaction patterns can affect student learning attitude and outcomes. For example, an authoritative teaching style is related to higher levels of students' academic and motivational outcomes [16]. When teachers use pro-social techniques, students are more likely to respond positively to teacher influence attempts; when teachers use negative techniques, students are more likely to resist teacher influence attempts [17]. With the integration of information technology and classroom teaching, the subject and form of classroom interaction have changed. Exploring the classroom teaching interaction patterns in a technology-rich environment is necessary for teaching assessment. And using the LSA (Lag Sequential Analysis) method to identify interaction patterns is feasible [18]. In this context, this study aimed to answer the following research questions: 1) What types of classroom interactions patterns can be
grouped according to behavioral sequence characteristics? 2) What are the characteristics of various types of interaction patterns?
III. METHOD
A. Data Source
In this study, 20 teaching cases were randomly selected as samples from 638 participating cases in a classroom teaching competition in China. All the lessons are grade 4 math lessons in elementary school, which use interactive whiteboards, interactive televisions or mobile terminals. Each submitted case includes four documents: the lecturer's information, the instructional design plan, a complete class teaching video, and teaching reflection. This research mainly analyzes classroom teaching videos.
B. Data Analysis
Currently the classical analysis approaches of classroom teaching include S-T analysis and Flanders Interaction Analysis System (FIAS). Comparing the advantages and disadvantages of these two analysis methods, this research adopts the classification and analysis method of classroom interaction behaviors proposed by Gu and Wang. Gu and Wang proposed ITIAS (Information Technology-based

Interaction Analysis System), which add student talk and technology categories to FIAS, as shown in Table I. Researchers adopt the ITIAS encoding system to encode the content of 20 instructional videos. In order to confirm the accuracy of the encoding, researchers used Premiere (a video editing software) to review and confirm the actual teaching records.

TABLE I: INFORMATION TECHNOLOGY-BASED INTERACTION ANALYSIS SYSTEM

Category
Teacher talk
Student talk Silence Technology

Code 1 2 3 4 5 6 7 8 9 10 11 12
13
14 15 16 17 18

Content Accept Praise or encourage Adopt ideas Ask open question Ask closed question Teaching Indicate Criticize Respond passively Respond actively Ask question actively Discuss A state of confusion that has no help in the teaching Think independently Complete task Teacher use technology Student use technology Technology for students

This research used SPSS 25.0 for cluster analysis of behavioral data and used GSEQ 5.1 for behavioral sequence analysis.

IV. RESULT
A. Video Analysis
Through encoding the samples of teaching videos, each class obtained the code sequences of interactive behaviors of the Given (the initial behavior) and Target (the next behavior). The code sequence of Given was combined with that of Target to form sequence pairs. Each sequence pair was composed of: (the code sequence of Given, the corresponding code sequence of Target). The following sequence pairs may be formed by recording according to the above “interactive behavior sampling record table”.

Fig. 1. Interactive behavior analysis matrix of Class 1.

127

International Journal of Information and Education Technology, Vol. 11, No. 3, March 2021

The corresponding behavior codes of Given and Target constituted rows and columns of the matrix diagram, respectively. This research counted the frequency of sequence pairs and filled it in the matrix to get Fig. 1 and Fig. 2 (only presenting the analysis matrixes of Class 1 and 2).

categorical variables. According to the hierarchical diagram and coefficients, the clustering results are classified into three categories, as shown in Fig. 3. Among the three categories, the first category has the largest proportion, reaching 75%, followed by the second category (20%) and the third category (5%).

Fig. 2. Interactive behavior analysis matrix of Class 2.
As shown in the Fig. 1, the total number of interactive behaviors in Class 1(interactive whiteboards class) was 261times. The interactive behavior of “Ask closed question-A state of confusion that has no help in the teaching” occurred most frequently in 44 times; followed by the frequency of 28 times for the occurrence of the behavior “Ask closed question-Respond passively”, 17 times for the behavior “Respond passively-Adopt idea”, and 15 times for the behaviors “Ask open question-Respond actively” and “Respond passively-Ask closed question” each. As shown in the Fig. 2, while in Class 2 (mobile terminals), the total
number of interactive behaviors was 277 times. The behavior “Respond passively -Teaching” occurred most frequently in 32 times; followed by the frequency of 24 times for the occurrence of the behaviors “Teaching- Respond passively”, 14 times for the behavior “Ask closed question-Respond passively”, 11 times for behavior “Teaching-Ask open question” and “Teaching-Ask closed question”.
The following findings could result from the integrated research results. First, of interactive behaviors in the 20 teaching videos, “Ask closed question-Respond passively” occurred most frequently. Second, teachers are the main initiators of classroom interaction, whose behaviors mainly include “Ask open/closed questions”, “Teaching” and “Indicate”. Those behaviors, like “accept”, “Praise or encourage”, “Adopt ideas”, “Teacher use technology” also frequently occurred. And the behavior “Criticize” happened occasionally. Third, students are primarily responders to classroom interactions including, whose behavior mainly include “respond passively/actively”, “think independently” and “Complete task”. Those behaviors, such as “discuss” and “Student use technology” also frequently took place. And the behaviors of “Ask question actively” and “A state of confusion that has no help in the teaching” occasionally. Fourth, the overall distribution of the matrix was relatively concentrated. For example, when the teacher had the behavior “Ask open/closed question”, students mostly “Respond actively/passively”.
B. Cluster Analysis
The study uses Hierarchical clustering to cluster 20 teaching cases samples with 306 behavior sequences as

Fig. 3. Hierarchical diagram of interaction patterns.
C. Lag Sequential Analysis The study further uses a lag sequence analysis method to test the behavior sequence of the three types of classroom occurrence probability reaching a significant level, in order to investigate the behavioral conversion characteristics of the three types of classroom interaction. The study finally obtained the behavior conversion diagrams of the three types cases, as shown in Fig. 4, Fig. 5 and Fig. 6.
Fig. 4. Sequence diagram of interactive behaviors of the first cases.
Fig. 5. Sequence diagram of interactive behaviors of the second cases.
Fig. 6. Sequence diagram of interactive behaviors of the third cases.

128

International Journal of Information and Education Technology, Vol. 11, No. 3, March 2021

Fig. 4 displays several noteworthy interaction behavioral sequences of the first type of classroom interaction. These sequences fall into two categories: one round of Q&A (question and answer) and two rounds of Q&A. The first sequence was 4/5→9→1/3→2→6, indicating that teachers initially asked an open question (4) or a closed questions (5), gave feedback after students answered the question passively (9), such as accepting (1) or adopting ideas (3), giving praise or encouragement (2), and then continued to teach (6). In the second interaction behavioral sequence, 4→10→3→2→6, indicating that teachers initially asked an open question (4), gave feedback after students answered the question actively (10), such as adopting ideas (3), giving praise or encouragement (2), and then continued to teach (6). The third interaction behavioral sequence, 4/5→9→8, indicated that teachers initially asked an open question (4) or a closed question (5), criticized the students (8) after they answered the question passively (9). These sequences belonged to one round of Q&A sequence.
In the two rounds of Q&A sequences, 4→9/10→3→5→9→1/3→2→6, indicating that teachers initially asked an open questions (4), adopted ideas (3) after students answered the questions passively (9) or actively (10), and then asked a closed question (5), accepted (1) or adopted ideas (3) and gave students praise or encouragement (3) after students provided answer, and then continued to teach (6). The other sequence,4→9/10→3→5→9→8, indicating that teachers initially asked an open questions (4), adopted ideas (3) after students answered the questions passively (9) or actively (10), and then asked a closed question (5), criticized the students (8) after they answered the question passively (9).
Fig. 5 displays several noteworthy interaction behavioral sequences of the second type of classroom interaction. These sequences also fall into two categories: one round of Q&A (question and answer) and two rounds of Q&A. Compared with the first kind of teaching cases, the interaction sequence formed by the second kind of teaching cases is relatively simple. For example, the sequence which is 4→9/10→3, indicated that teachers initially asked an open question (4), and then adopted ideas after students answered the question passively (9). The feedback type of teachers is relatively single, lacking praise and encouragement. In the second case, “Think independently” (14) and “A state of confusion that has no help in the teaching” (13) appear in the behavioral sequence, for example 4/5→14→7/13.
Fig. 6 displays several noteworthy interaction behavioral sequences of the third type of classroom interaction. Compared with the first two types of teaching cases, the third type of teaching cases does not form an interactive sequence that includes both open and closed questions. Teachers in the third type of teaching case also attach importance to the independent thinking of students. However, the state of confusion that has no help in the teaching also exists.
V. DISCUSSION
A. Immediate Interaction Patterns
Interaction is mutually created and shared between

students and teachers, with an emphasis on the role of shared emotions and feelings used by students and teachers to respond both affectively and effectively to each other. To communicate effectively with their students, teachers use affinity-seeking strategies and immediacy behaviors, are supportive and confirming [19]. In the first type of classroom interaction mode, teachers will give students feedback on their answers, such as acceptance or praise. However, teachers have also criticized students, which indicates that the classroom atmosphere is relatively serious. Teacher–student interaction revolves around the primary roles played by the teacher which are teacher as lecturer and teacher as discussion leader. In this first type of interaction mode, the teacher is the initiator of the discussion. Teachers ask questions first, then ask students to answer, and then give students feedback. It can be seen that the rhythm of teacher-student interaction is fast. In addition, this type of interaction pattern forms a question chain that is the teacher asking immediately after the student answers the previous question, which can deepen student's own thinking and sharing with others [20]. In summary, according to the classroom atmosphere and rhythm, the first type of interaction pattern is defined as the immediate interaction pattern.
B. Waiting Interaction Patterns
The second type of interaction pattern also pays more attention to the classroom atmosphere that is more active and relaxed. For example, teachers give their students feedback that is more acceptance and praise than criticism. Researchers have studied interaction patterns and identified that sentiments play a considerable role in students' engagement [21]. For example, positive sentiments may give rise to higher quality and more frequent interactions [22]. The same as the first kind of interaction, the teacher is also the initiator of the interaction, and also forms a chain of questions. But the difference is that teachers ask students to think before answering questions. Research suggests that sufficient waiting time facilitate student to think deeply [23]. It can be seen that the rhythm of the classroom is relatively mild. But at the same time, if the teacher does not control the thinking time, there will be confusion that has no help in the teaching. Based on the above characteristics, the second type of interaction pattern is defined as the waiting interaction pattern.
C. Shallow Interaction Patterns
In the third type of interaction pattern, the classroom atmosphere is also relaxed and pleasant, and the classroom rhythm is also relaxed. But unlike the first two patterns, the third type of interaction pattern does not form a question chain. Teachers will not follow up after a round of questions, which can be seen that teachers and students are only asking questions on the surface without forming deep interaction. Based on the above characteristics, this pattern is defined as the shallow interaction pattern. At the same time, the study found that in the classroom interaction patterns in a technology-rich environment, the technical elements were not significant. Teachers only use technology as a display tool rather than a communication tool or resource platform

129

International Journal of Information and Education Technology, Vol. 11, No. 3, March 2021

tool between teachers and students.
VI. CONCLUSION
This study took the classroom with interactive whiteboard, interactive television or mobile terminals as research scene, and utilized the ITIAS coding system to analyze the classroom videos of 20 teaching cases from four aspects. Three types of interaction patterns and their characteristics of the technology-rich environment are extracted, which are summarized as follows: 1) Immediate interaction patterns: serious classroom atmosphere, fast-paced class and deep interaction. 2) Waiting interaction patterns: relaxed classroom atmosphere, slow-paced class and deep interaction. 3) Shallow interaction patterns: relaxed classroom atmosphere, slow-paced class and shallow interaction. In addition, the technical elements in the three types of interaction patterns are not significant.
CONFLICT OF INTEREST
The authors declare no conflict of interest.
AUTHOR CONTRIBUTIONS
Zhong Sun conducted the research; Kaiyue Lv analyzed the data; Kaiyue Lv wrote the original draft; Zhong Sun and Min Xu reviewed and edited the final paper; all authors had approved the final version.
REFERENCES
[1] J. Shu, M. Zhi, and Q. Hu, “Research of the university teaching interaction behavior characteristics in the smart classroom,” International Journal of Information and Education Technology, vol. 8, 2018.
[2] J. M. Sinclair and R. M. Couthard, Towards an Analysis of Discourse: The English Used by Teachers and Pupils, Oxford University Press, London, 1975.
[3] G. Chen, S. N. Clarke, and L. B. Resnick, “Classroom discourse analyzer (CDA): A discourse analytic tool for teachers,” Technology, Instruction, Cognition & Learning, vol. 10, 2015.
[4] M. Antón, “Shifting trends in the assessment of classroom interaction,” The Handbook of Classroom Discourse and Interaction, vol. 115, 2015.
[5] T. E. Virtanen, G. S. Vaaland, and S. K Ertesvåg, “Associations between observed patterns of classroom interactions and teacher wellbeing in lower secondary school,” Teaching and Teacher Education, vol. 77, pp. 240-252, 2019.
[6] Y. Cheng, Q. Liu, F. Wang, and Y. Wang, “Research on the application of an improved, video-based S-T analysis method,” E-Education Research, pp. 90-96, 2016.
[7] N. Flanders, “Intent, action and feedback: A preparation for teaching,” Journal of Teacher Education, vol. 14, pp. 251-260, 1963.
[8] X. Gu and W. Wang, “New exploration of classroom analysis in support of teacher professional development,” China Educational Technology, pp. 18-21, 2004.
[9] S. Hennessy, T. Dragovic, and P. Warwick, “A research-informed, school-based professional development workshop programme to promote dialogic teaching with interactive technologies,” Professional Development in Education, vol.44, pp.145-168, 2018.
[10] G. Chen, “Visual learning analytics to support classroom discourse analysis for teacher professional learning and development,” The Routledge International Handbook of Research on Dialogic Education, 2019.
[11] C. Gaudin and S. Chaliès, “Video viewing in teacher education and professional development: A literature review,” Educational Research Review, vol. 16, pp. 41-67, 2015.

[12] G. Chen, “A visual learning analytics (VLA) approach to video-based teacher professional development: Impact on teachers’ beliefs, self-efficacy, and classroom talk practice,” Computers & Education, vol. 144, 2020.
[13] R. Santagata and G. Angelici, “Studying the impact of the lesson analysis framework on preservice teachers’ abilities to reflect on videos of classroom teaching,” Journal of Teacher Education, vol. 61, pp. 339-349, 2010.
[14] M. Antón, “Shifting trends in the assessment of classroom interaction,” The Handbook of Classroom Discourse and Interaction, vol. 115, 2015.
[15] S. K. Ertesvåg, “Measuring authoritative teaching,” Teaching and Teacher Education, vol. 27, pp. 51-61, 2011.
[16] B. V. Dever and S. A. Karabenick, “Is authoritative teaching beneficial for all students? A multi-level model of the effects of teaching style on interest and achievement,” School Psychology Quarterly, vol. 26, pp. 131-144, 2011.
[17] S. A. Myers, “Classroom student-teacher interaction,” The International Encyclopedia of Communication, 2008.
[18] C. Q. Huang, Z. M. Han, M. X. Li, M. S. Y. Jong, and C. C. Tsai, “Investigating students' interaction patterns and dynamic learning sentiments in online discussions,” Computers & Education, vol. 140, 2019.
[19] S. Myers, The International Encyclopedia of Communication, 2008. [20] N. M. Webb, M. L. Franke, M. Ing, J. Wong, C. H. Fernandez, N. Shin
and A. C. Turrou, “Engaging with others' mathematical ideas: Interrelationships among student participation, teachers' instructional practices, and learning,” International Journal of Educational Research, vol. 63, pp. 79-93, 2014. [21] S. A. Crossley, K. Kyle, and D. S. McNamara, “Sentiment analysis and social cognition engine (SEANCE): An automatic tool for sentiment, social cognition, and social-order analysis,” Behavior Research Methods, vol. 49, pp. 803-821, 2017. [22] R. Pekrun, T. Goetz, A. C. Frenzel, P. Barchfeld, and R. P. Perry, “Measuring emotions in students' learning and performance: The achievement emotions questionnaire (AEQ),” Contemporary Educational Psychology, vol. 36, pp. 36-48, 2011. [23] C. Zhang and L. Ning, “A study on the characteristics of different levels of questions in primary classrooms,” Curriculum, Teaching Material and Method, vol. 31, 2011.
Copyright © 2021 by the authors. This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).
Kaiyue Lv is a master student studying in the College of Informational Engineering at Capital Normal University, Beijing, China. Her current research interests are towards technology enhanced teacher professional development and learning analytics. She is currently working on a data analytics project related to the classroom teaching interaction supported by artificial intelligent technology.
Zhong Sun is a professor in the College of Informational Engineering at Capital Normal University, Beijing, China. Her research interests include mobile learning and technology enhanced teacher professional development. Prof. Zhong Sun is the corresponding author of this paper.
Min Xu is an associate professor with the College of Information Engineering, Capital Normal University, Beijing, China. She has authored more than 20 scientific papers in peer-reviewed journals and conferences. Her research interests include computer vision, pattern recognition, and machine learning.

130