Chris Roberts Thesis 3

Methods and Procedures

Participants    The participants in this study consist of two groups of Spanish 1 students in a suburban public middle school in northeast Ohio.  This school, consisting of 7th and 8th graders, has a population in the 900 range.  In that, 4.0% is black, non-Hispanic; 91.1% White; 21.7% economically disadvantaged; and 12.7% students with disabilities.  The two groups of Spanish 1 students are groups of 8th graders, aged 13-14.  The control group (traditional group) consists of 30 students.  The experimental group (input-based, TPRS) consists of 29 students.  The demographics of these two groups are as follows, in Table 1:

Table 1, demographics of each group Demographics Control group (traditional) Experimental group (TPRS) White, non-Hispanic 26 24 Black, non-Hispanic 2 2 Hispanic 1 2 Other 1 1 Students with disabilities 0 1 Gifted students 8 0 Heritage speakers 1 0

In the control group, there is one Hispanic female who is a heritage speaker of Spanish.  A heritage learner is defined as “a student who is exposed to a language other than English at home.  Heritage speakers can be categorized based on the prominence and development of the heritage language in the student’s daily life” (Annenberg).  The heritage speaker in the control group has full oral fluency and literacy in Spanish and Spanish is the primary language spoken at home.  In the experimental group, there are two Hispanic students but neither of them would be categorized as heritage speakers as English is the primary language spoken in their homes, and neither of them have oral fluency or written literacy in Spanish.  The control group consists of eight students who are in the gifted program at school while the experimental group does not consist of any gifted students.  The control group has no students with disabilities and the experimental group has one student who is on an IEP for a learning disability in reading and this student also has a 504 Plan for ADHD.

The control group was taught during the 2011-2012 academic year and the experimental group was taught during the 2012-2013 academic year.

Instruments

The instrument used to measure the achievement in both groups for the comparison study is the National Spanish Exam.  The National Spanish Exam is a standards-based assessment that measures both achievement and proficiency.  The achievement section of the exam assesses content standards through specifications for vocabulary and for grammar.  The proficiency section assesses students’ listening comprehension and reading comprehension.  The National Spanish Exam has high reliability coefficients so it is a reliable test.  This exam is given to my students each year to see how they perform compared to other students across the country.  Given the fact that I already administer this exam to my students, and the exam has a high reliability coefficient, I see it as the best assessment tool to use to compare these two groups of students being taught with different teaching methods.

Because of limited funding available, I normally administer the previous year’s National Spanish Exam, which is available for free.  Therefore, for the control group, I administered the 2011 National Spanish Exam.  Although the experimental group could have been given the 2012 National Spanish Exam, I decided to give them the 2011 exam as well to ensure that my results are free from as many extraneous variables as I could control; therefore, I found it to be the best choice to give these students the same exam as the control group.

Hypothesis

My hypothesis is that this study will align with the studies examined above and that the students in the comprehensible-input based, TPRS classroom will outperform the students in the output and grammar-based traditional classroom, demonstrating that comprehensible-input based methods are more effective than more output-based methods.

Variables and validity of the study

For this research study, the independent variable is the teaching method being used, which will have an effect on the student achievement, or test scores.

The independent variable is student achievement, the students test scores on the National Spanish Exam.  My hypothesis is that the independent variable, teaching method, will have an effect on the NSE test scores, the dependent variable.

Extraneous variables are factors other than the independent variable that may have an effect on the outcome of the study.  Extraneous variables can jeopardize the internal validity of a study.  In this study, there are three extraneous variables that may have an effect on the outcome of the study. One is the eight gifted students in the control group, who may have an effect on the mean test scores of the control group.  Another extraneous variable that may have an effect on the outcome of the study is that for the first quarter of the 2012-2013 year, approximately 9 weeks, there was a student teacher who was teaching the experimental group.  Having a less experienced teacher may have an effect on the outcome of the study.  One more extraneous variable that may affect the outcome of the study is the date when each group took the exam.  Due to time constraints on this study, the control group took the exam in May while the experimental group took the exam in March.  Having almost two months of extra instruction may give the control group an advantage.

One factor that may make this study challenging to carry out is the question of my credibility.  Being that my hypothesis is that TPRS is a more effective teaching method than the more output-drive traditional approach, it could be said that I am biased towards TPRS.  Given that I am the teacher for both groups, it could be said that there is potential for my bias to affect my teaching in the control group so that the experimental group performs better on the National Spanish Exam.  However, I taught the control group and administered the exam before I began teaching with the TPRS method.  When I taught the control group, I taught as well and as enthusiastically as I could with the method that I was using.

Summary

The purpose of this study is to research the effects of teaching with TPRS, a comprehensible input-based methodology, in comparison with more output-based communicative, traditional teaching.  Despite the studies demonstrating the effectiveness of input-based instruction in comparison with output-based based instruction, there are no studies comparing students in these types of classes using the National Spanish Exam, a nationally standardized assessment with a high reliability coefficient.

Procedures and Materials

Each group was in a Spanish 1 class for a full academic year, receiving instruction in a different teaching methodology.  Various types of activities, assessments and experiences were provided to each group.  The procedures and materials for each group were different.

Procedure and materials for control group.   Activities for this group were designed with communication in mind and were designed around the Ven Conmigo level one textbook.  The focus in this group for the academic year was to develop communicative competence and knowledge of grammar rules.  Communicative competence is defined, by Richards (2006), as knowing how to use language for various purposes and functions.  Classes typically followed the Presentation, Practice, Production approach.  This 3 P approach drove much of the classroom instruction, whether the focus was on vocabulary or on grammar.  Vocabulary would typically be presented to students either with translation, pictures, or both.  Grammar would be introduced both within context and as rules to remember.  New grammar structures would be presented by means of a conversation or short text, and I would also explain the grammar rules and have students take notes on the grammar rules being explained.  For example, if the focus was on verbs ending in –ar, a short text would be provided using various –ar verbs in context to show students the different forms within that context.  Students would also fill in the verb charts in their notebooks and I would explain how verb conjugations work in Spanish.  Oftentimes, many new grammatical structures also had a song to accompany it to help students remember the rules for proper use.  Then, students would practice new grammar structures or vocabulary “in a controlled context, through drills or substitution exercises” (Richards, 2006, p. 32).  After this practice, students would then practice the new vocabulary or grammatical structures through communicative activities with a partner, such as information gap activities.  The end goal for most lessons was to have students use the language within a context to practice communicating in L2.  This group also occasionally had projects to practice producing the target vocabulary structures.   For example, in a unit in which city vocabulary and prepositions are introduced, the students had a video project in which they went to a park, which is a miniature model of the city, and in Spanish they conducted a guided tour of the “city”, explaining what each building was, its location in relation to other buildings and locations in the mini-city, and what people typically do in those buildings, practicing some of the verb structures in the unit.  While this group did not necessarily have a weekly routine, the weekly schedule below reflects what one would probably see in a typical week in this class: • Mondays: Introduction of new vocabulary and/or grammar; using new vocabulary in context by asking questions to students using the vocabulary or reading a short text with the vocabulary; listen to a song using the new vocabulary; listen to audio recordings provided by the textbook to practice listening to the vocabulary and answer questions about the vocabulary • Tuesdays: Bell work practicing Monday’s vocabulary or grammar; practice using vocabulary or grammar through drills or activities from the textbook;  communicative partner activities to practice production of vocabulary • Wednesdays: Quiz based on Monday’s new vocabulary; Introduction of more vocabulary or grammar within the same unit; repeat Monday’s activities with new vocabulary or grammar • Thursdays: Practice Wednesday’s vocabulary and grammar; communicative activities • Fridays: More production activities; quiz or test based on entire week’s vocabulary

This group also read the novel Pobre Ana (Ray, 1999) during the second quarter of the school year.  After each chapter of this book, there would be a quiz on the content of each chapter.  On average, in this class, I spoke Spanish approximately 55-60% of the time and I spoke English approximately 40-45% of the time.

Procedure and materials for experimental group.  All activities for this group were designed according to the guidelines found in Ray and Seely (2012).  The focus in this group for the academic year was to provide oral and written comprehensible to students in order to aid language acquisition according to the Comprehension Hypothesis (Krashen, 1997).   Input was provided in the form of PQA, stories and reading activities.  Vocabulary and target structures were drawn from various sources including:  Blaine Ray’s Look I Can Talk level 1 curriculum, Michael Miller’s Charo y Lee level 1 Spanish TPRS curriculum, various modified legends from Spanish-speaking countries, various level 1 readers, and high-frequency words in Spanish taken from A Frequency Dictionary of Spanish: Core Vocabulary for Learners (Davies, 2006).  Over the course of the year, the level 1 “novels” or readers Tumba (Canion, 2012), Pobre Ana (Ray, 1999),  Patricia va a California (Ray, 2001) and Esperanza (Gaab, 2011) were read as a whole class.  With Tumba and Pobre Ana, there were occasional chapter quizzes to assess students’ comprehension of the chapters being read, however Patricia va a California and Esperanza typically did not have any quizzes or assessments associated with them as the primary focus with these readers was to read for enjoyment and to provide comprehensible input to students.  Individually, students read between 4 and 8 level 1 readers, depending on each reader’s speed.  Grammar was taught in context, as meaning rather than as rules to memorize.  Grammar explanations were in the form of “pop-up grammar” (Ray & Seely, 2012), brief explanations where grammar is taught as meaning, for example: the n in comen essentially means they.  Another example, in French, would be to contrast the e in il/elle aime with the nt in ils/ells aiment (third-person singular vs. third-person plural).  The teacher would write the two forms on the board so that the students can see the difference.  The teacher would point out that the pronunciation is the same and then point out the plural forms of the pronouns, “Class, what does the nt do?  What does just il/elle aime mean without the nt?” (Ray & Seely, 2012).

This group followed a weekly routine as follows: • Mondays: PQA of target structures • Tuesdays: class story • Wednesdays: Reading based on Tuesday’s story • Thursdays: Extended reading, or reading from Look I Can Talk, Charo y Lee, a cultural reading, reading of a novel, or reading a legend • Fridays: Preferred Activity Time

On Mondays, the class engaged in PQA, Personalized Questions and Answers.  Some teachers refer to this time as “Teacher Talk” (Miller, 2009).  PQA is an open time of chatting with students, using the target structures, where you ask personal questions, make jokes, tell anecdotes and have a good time while focusing on the target structures (Roberts, 2013).  I think of PQA as a “gossip” session where we just “make stuff up” about each other in the target language.  For example, if our target structure is “vive – s/he lives”, some of the PQA, or gossip, that would come out of that could include: • Josh lives in a purple igloo in the Bermuda Triangle • Tyler lives in a red box. (We then proceed to bring Tyler up and we have him stand behind a red box and he shoves a DVD out of the box, playing with the term “red box” to insinuate that he lives in the RedBox DVD machines) • Austin lives in a small cave in Cleveland with the Bogeyman While these sentences are silly, the goal is for students to be focusing on comprehensible, compelling messages in order to acquire the target structure “lives”.

Whether doing PQA, stories or a reading, I used the circling technique in TPRS.  The circling technique is a questioning technique used to get many repetitions of target vocabulary words.  By asking questions using the limited number of target structures, students would get a lot of comprehensible input and many repetitions of the target vocabulary, which allows them to focus on the meaning, the message, rather than on the language, allowing acquisition and internalization of vocabulary to occur.  The reason for focusing on getting repetitions of vocabulary is because research shows that repetition is very important.  Cambridge neuroscientist, Dr. Yury Shtyrov, round that after 160 repetitions of a new vocabulary word, the brain forms a whole new network of neurons specifically tasked with remembering that word (Adams, 2010).  In the book Brain Rules (Medina, 2008), developmental molecular biologist John Medina says that we have 30 seconds to repeat something before it is forgotten.  The circling technique was the preferred method to get the repetitions of vocabulary.  The way circling works is the teacher “circles” each part of a sentence, the subject, verb and complement.  The format used in circling consists of this pattern: ? Start with a positive statement ? Ask a yes/no question that gets a “YES” ? Ask either/or question ? Ask a yes/no question that gets a “NO” ? Restate the negative and the positive ? Ask a question word question ? Positive Statement

For example, if the target structure is “wants to eat”, a possible sentence could be “John wants to eat chocolate”.  In order to “circle” this sentence, the teacher would circle the subject, then the verb and then the complement.  Therefore, when circling the subject, the teacher’s script would look like this: ? Start with a positive statement › John wants to buy chocolate ? Ask a yes/no question that gets a “YES” › Does John want to buy chocolate? ? Ask either/or question › Does John or Sally want to buy chocolate? ? Ask a yes/no question that gets a “NO” › Does Hannah want to buy chocolate? ? Restate the negative and the positive › No, Hanna does not want to buy chocolate, John wants to buy chocolate ? Ask a question word question › Who wants to buy chocolate? ? Positive Statement › John wants to buy chocolate

This is then repeated with the verb (wants to eat) and with the complement (chocolate).

Reading followed the Read and Discuss format (Ray & Seely, 2012).  The procedures in the Read and Discuss technique are as follows: 1. Students are given a printed story, or novel 2. Teacher reads a sentence out lout in target language 3. Students chorally translate the sentence out loud a. The reason for this is so that the teacher knows that students understand, and Ray (2012) has found that students learn repeated words when they are translated that they don’t learn when the words are simply read in L2. 4. Paragraph by paragraph, the teacher discusses the reading in L2 a. Relating the situation, characters and plot to students’ lives b. Capitalizing on cultural information in story c. Discuss character development, choices and values d. Discuss details in the reading, or details that may not be in the reading (who, what, where, when) 5. Explain any grammar within the reading, so that grammar is tied to meaning, not to grammar rules

Along with the weekly routine, this class also followed a daily routine: • Sustained Silent Reading /Self-Selected Reading – 10 minutes • PQA, story, or reading, depending on the day of the week – 15 minutes • Brain Break – 2 minutes • More PQA, continue story or reading – 15 minutes • Quick quiz – 5 minutes

The SSR mentioned in the daily routine began in October and continued through the entire school year.  I have a classroom library full of various level 1 and level 2 Spanish readers.  For SSR, students would enter the classroom, choose one of the readers and read for 10 minutes.  After the 10 minute timer went off, students would return the books and we would begin either PQA, a story, or reading, depending on which day of the week it was.  A 2 minute break was typically given each day in order to allow students to have a brain break from all of the input in Spanish that they were receiving, and to also allow students to get up and stretch and talk amongst one another.  The focus of every class was to provide as much oral or written comprehensible input as possible.

On average, in this class, I spoke Spanish approximately 90% or more of the time.    Results

As mentioned before, the instrument used to measure the achievement in both groups for the comparison study was the National Spanish Exam.  Both the control group and experimental group took the 2011 National Spanish Exam as the comparison instrument.  The National Spanish Exam consists of two parts, Achievement and Proficiency.  Each part is worth 200 points, making the entire exam 400 points.  Teachers are able to compare their students’ scores with the national average and determine in which percentile their students fall in when compared nationwide.

The control group, the traditional, output-based group, took the Achievement portion of the National Spanish Exam on May 7th, 2012; they took the Proficiency portion on May 8th, 2012.  Due to time constraints on the study, the experimental group took the Achievement portion of the exam on March 13th, 2013 and they took the Proficiency portion on May 14th, 2013.

The test scores of each group are found in Table 2.  Students have been given pseudonyms to protect their identity.

Table 2, Test results Control Group Achievement Score (/200) Proficiency (/200) Total Score (/400) Experimental Group Achievement Score (/200) Proficiency (/200) Total Score (/400) Alyssa 88 78 166 Aaron 68 100 168

Kennedy 56 82 138 Bryan 80 112 192

Sophia* 180 172 352 Kalynn 156 176 332

Abigail 56 90 146 Russell 124 132 256

Megan 100 104 204 Michael 108 118 226

Sandra 80 90 170 Kenny 130 138 268

Michael 68 124 192 Madison 76 112 188

Alexander 88 80 168 Abby 76 96 172

Trevor 100 141 241 Justin 132 152 284

Marie 76 98 174 Joseph 128 134 262

Joseph 80 108 188 Jessica 156 150 306

Theresa 68 98 166 Darnell 88 100 188

Wayne 96 142 238 Lauren 160 172 332

James 108 150 258 Noel 76 108 184

Kyle 100 108 208 Tomas 92 98 190

Makayla 108 122 230 Anne 76 104 180

Brooke 64 60 124 Terrence** 76 84 160

Madison 56 72 128 Marie 88 106 194

Allie 104 146 250 Anthony 100 136 236

Nikki 112 106 218 Allan 176 180 356

Olivia 76 102 178 Nicole 104 140 244

Scott 44 96 140 Tiffany 92 110 202

Austin 56 90 146 Dirk 108 88 196

Alexander 76 96 172 Susan 60 78 138

Gabby 88 122 210 Javor 120 158 278

Chase 68 122 190 Michaela 124 156 280

Ricky 76 88 164 Katherine 80 144 224

Owen 96 96 192 Brittany 88 138 226

Jared 116 144 260 Madeline 72 158 230

Chris 80 48 128     30 students    29 students    Group Average 85.4 105.8 191.3 Group Average 103.9 126.8 230.7

* indicates Heritage speaker **indicates student with IEP  Table 2 shows the test scores for each group.  Table 3 shows statistical comparisons of performance scores.  As shown in table 2, the mean achievement score for the control group was 85.4 out of 200 while the mean achievement score for the experimental group was 103.9 out of 200.  We are able to see that the experimental group outperformed the control group on the achievement test by 18.5 points.  This is very interesting considering the control group had received more explicit grammar and vocabulary instruction than the experimental group had, yet the experimental group performed significantly better on the vocabulary and grammar portion of the exam.  On the proficiency test, which tests listening and reading comprehension, the control group’s mean score was 105.8 and the experimental group’s mean score is 126.8, showing that the experimental group outperformed the control group by 21 points.  Table 2 also shows the total scores on the National Spanish Exam for each group.  The control group’s mean total score on the exam is 191.3 and the control group’s mean total score is 230.7.  The experimental group, which focused on comprehensible input through storytelling and reading, outperformed the control group, which focused on exercised encouraging output, and communicative and grammatical competence, by 39.4 points on the National Spanish Exam. Table 3 presents statistical comparisons of the overall mean score for each group.

Table 3 Statistical Comparison of Control and Experimental Groups on National Spanish Exam National Spanish Exam Control group Experimental group t score Effect-size r Cohen’s d  Mean (Standard Deviation) Mean (Standard Deviation)

Overall Test Score  191.3 (49.02)  230.75 (55.27) t = 2.89 0.49 1.13

The mean score for the control group was 191.3 while the experimental group was 230.75.  There was a 39.4 point difference between the two groups.  This different may or may not have been a statistically significant difference.  In assessing the effect sizes of the scores, it was found that the effect-size r is 0.49.  According to Gall, Gall and Borg (2010), an effect size of 0.33 or larger has practical significance.  Therefore, with an effect size of 0.49, we can say that the difference between the two groups is significant.  According to Gravetter & Wallnau (2011, p. 233), a d of 0.8 or larger is a large effect.  When looking at the Cohen’s d of 1.13, we also see that the difference between the two groups is statistically significant.

The national average score on the 2011 National Spanish Exam was 239.2.  In the control group, 5 students scored above the national average.  In the experimental group, 11 students scored above the national average.   The National Spanish Exam takes student scores across the nation and puts them into a percentile, with certain categories being reserved for specific percentiles: • Gold – above the 95th percentile • Silver – from the 85th percentile through the 94th percentile • Bronze – from the 75th percentile through the 84th percentile • Honorable Mention – from the 50th percentile through the 74th percentile. In these two groups, the control group had 3 students in the Honorable Mention category, 0 in the Bronze and Silver and 1 in the Gold.  The experimental group had 7 students in the Honorable Mention, 1 in the Bronze, 2 in the Silver, and 1 in the Gold.  It is important to note that the Gold student in the control group was the heritage speaker, the Gold student in the experimental group was not.  The heritage speaker in the control group received an overall score of 352, while the non-heritage speaker in the experimental group, with no outside experience with Spanish, received a score of 356.

When we look at the lowest scores in each group, we find that the lowest score in the experimental group was 138.  In the control group, there are three students with lower scores than 138: 128, 124 and 128.  Not only was the overall mean score of the experimental group significantly higher than that of the control group, but even the lowest performing students in the experimental group outperformed the lowest performing students in the control group.

When comparing the test scores between the two groups, the data supports the hypothesis of this study that the students in the comprehensible input-based, TPRS classroom would outperform the students in the output and grammar-based traditional classroom.  The data and effect sizes show that the TPRS classroom significantly outperformed the traditional classroom and supports the hypothesis that comprehensible input-based methods are more effective ways to teach world languages than more output-based, traditional methods.

When analyzing the data between different groups’ performance, it is always important to consider the extraneous variables in the study.  There were three extraneous variables in this study: 1. Eight gifted students in the control group, zero in the experimental group 2. Student teacher for part of the school year in the experimental group 3. Experimental had to take the exam two months before the control group, allowing the control group to receive more instruction before the implementation of the National Spanish Exam.

All three of these extraneous variables could have given the control group an advantage over the experimental group.  However, as observed in the data analysis, the experimental group outperformed the control group in a way that was statistically significant.  This raises the questions:  Had the extraneous variables not been present, would the experimental group have outperformed the control group in more significant ways, thus further supporting the hypothesis of the study?