Instruction hours and language improvement according to CAL

Here is the podcast

The US Center for Applied Linguistics (CAL)  announced its latest digest,“Effects of Instructional Hours and Intensity of Instruction on  NRS Level Gain in Listening and Speaking.” The digest reports on a descriptive study examining the relationship between student performance ..and class attendance. ..Results showed that  the greater the number of instructional hours, the higher the percentage of students who made an NRS level gain. There was also a general trend toward greater NRS level gain for students with high levels of instructional intensity than for those with low instructional intensity. ..

To the people at CAL this is a "seminal report" justifying all their hard work. But is it really?

More hours of classroom instruction, and more intense instruction, should lead to better results, otherwise why bother? The real question is how much improvement is required to justify the time and money expended by learners, teachers and tax-payers. Here is my analysis based on information contained in the report.

We are told in the survey that 1.2 million adults are enrolled in federally funded adult ESL classes in the United States and that 36% of these students attained a measurable  educational level gain after a course of instruction.The scale used is here. In a survey of 6,599 adults, 60% showed improvement. Obviously being in a survey has a big impact on improvement results! The fact is that there are probably close to 20 million adults in the US who need to improve their English literacy, just among the immigrant population.

Almost half (49%) of the ESL learners in the survey were at level 0 and 1 on the scale, i.e. "no ability whatsoever' or "functions minimally if at all in English." Almost 20% were Low and High Beginner level learners (2 and 3 on the scale). Level 3 is described as "understands simple learned phrases, spoken slowly with frequent repetitions". At the other end of the scale 7% of the adults surveyed were Advanced or level 6 on the  scale, described as " can satisfy most survival needs and limited social demands." So, even the advanced learners were still at a basic level.

The results appear in tables below. It appears that contrary to the claims of the Center for Applied Linguistics, the biggest factor affecting grade improvement was not hours of instruction but the level of the learner. Beginner learners (level 2 and 3 on the scale) improved the most and were the least affected by the amount of instruction. Of those Low and High Beginners who had the least amount of instruction ( between 2 and 60 hours) almost 75% still managed to improve, whereas this only went up to 84% for those who had between 140 and up to 512 hours of instruction, i.e. at least 3 times as many hours of classroom instruction . We are told in the report that 78%, or almost 4 out of 5 of these Low and High Beginner learners improved regardless of the number of hours of instruction. 

The largest group, those with essentially no English skills(49%), as well as the most advanced group (7%), showed the lowest level of improvement, but seemed to benefit the greatest from instruction. The report does not explain this nor the fact that the rate of improvement  sometimes declines with increased instruction.(see tables below)

Intensity of instruction does not have a great affect on results. The largest group ( 57%) studied an average of 4.5 hours per week and 61% of these learners showed measurable improvement on the scale. However 31% of the survey group had less than 2.8 hours per week of instruction and yet 56%  still managed to improve. The intense group, roughly 12% of the learners, studied more than 9.3 hours per week. Despite more than double the hours of instruction, compared to the middle group, the percentage of learners with measurable improvement only increased from 61% to 66%. Again it was the Low and High Beginners who improved the most, with the least impact from instructional intensity.

To me the conclusion is that class instruction obviously does help, but not as much as CAL and teachers like to believe. Instead, I suspect that what really matters is what the learner does outside the classroom. As the report says, an adult ESL learner has limited time to spend, "typically 4 and 8 hours per week". Surely to help these learners it is better to focus on finding ways to enable these learners to create more time for learning. In other words we should find ways to make it easier and more effective for them to learn outside the classroom, and to encourage them to do so, instead of trying to justify bringing them to class. Classroom time does not seem to have  a decisive impact on their improvement. To paraphrase Rubem Alves, we need to make them hungry rather than giving them cheese.

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

Table 6. NRS Level Gain Related to Instructional Hours by NRS ESL Educational Functioning Level
Instr.                 Hours	Beginning                   ESL                   Literacy                 n=1,720	Low                   Beg.                   ESL                 n=407	High                   Beg.                   ESL                 n=543	Low                   Inter.                   ESL                 n=614	High                   Inter.                   ESL                 n=408	Advanced                   ESL                 n=252
Below 60                 Hours	n=536                 46%	n=112                 75%	n=160                 72%	n=146                 60%	n=96                 54%	n=51                 50%
60 to 79 Hours	n=284                 52%	n=86                 79%	n=85                 69%	n=93                 51%	n=68                 49%	n=44                 48%
80 to 99 Hours	n=247                 56%	n=57                 72%	n=95                 79%	n=105                 60%	n=74                 58%	n=33                 45%
100 to 119                 Hours	n=159                 54%	n=34                 81%	n=52                 81%	n=75                 68%	n=42                 58%	n=29                 64%
120 to 139                 Hours	n=120                 62%	n=37                 86%	n=52                 87%	n=49                 66%	n=43                 64%	n=30                 67%
140 or More                 Hours	n=374                 67%	n=81                 84%	n=99                 83%	n=146                 71%	n=85                 62%	n=65                 66%

 

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

Table 8. NRS Level Gain by Intensity of Instruction
Intensity                     Level	Beginning                     ESL                     Literacy                     n=1,720	Low                     Beg.                     ESL                     n=407	High                     Beg.                     ESL                     n=543	Low                     Inter.                     ESL                     n=614	High                     Inter.                     ESL                     n=408	Advanced                     ESL                     n=252
Low Intensity                     (<0.50)	n=548                     50%	n=116                     77%	n=165                     75%	n=140                     55%	n=121                     57%	n=52                     49%
Mid Intensity                     (0.50-0.99)	n=935                     55%	n=247                     78%	n=321                     77%	n=386                     64%	n=236                     56%	n=140                     54%
High Intensity                     (>1.00)	n=237                     60%	n=44                     83%	n=57                     83%	n=88                     68%	n=51                     61%	n=60

       

Comments

[Dmitriy]

Hi Steve,

always when I read your posts about classic approach to language learning I am sorry that your approach is less popular. Russia has great demand for English teaching. So LingQ need promotion.

In Russia we have our own search engines. I have just registered www.thelinguist.com in www.yandex.ru, it is the most popular search engine in Russia (google is less popular). If any site is absent in yandex.ru then Russians hardly know about this site.:)

I know you have russian programmers which make LinqQ, may be they could help you with promotion.

[LFJ]

university professors are always coming up with some new research presentation- to keep their jobs!

but i think they are more worried about modern instructional tools and pedagogical theories, and putting out the next research result, than what is most important- the students learning!

but its a tough thing to swallow for teachers that the role of teachers is not to be in charge on all aspects. the role of teachers is simple, yet can be extremely powerful. its almost like a power struggle though. they want to be the ones to teach people language rather than let the students learn.

maybe an ego problem?

[Ty]

And no one ever learned a language using old fashioned method. Let's get real here. No matter what you use to learn a language, audiolingual or something like LingQ or the Lozanov method, you're only going to get better with real-world live practice. The teacher and classroom instruction provide the planks. It's up to the student to use those planks to build the floor.

[Daniel]

I'm sure a person with a bawdy mind could make a lot of comments about this "seminal report" and CAL's self gratification. This report seems to be the classic use of statistics: to prop up an argument in the way a drunk props himself up using a lamp post.

[Israel]

Hi Steve,

perhaps this reply could be out of place; anyway, I would like to ask you some questions about your method of learning any language. I think like you that words and phrases are the most important things of the language. The more words and phrases you know, the better for writing and speaking naturally as the natives do. From my point of view the learning process can be resumed to know how to use patterns composed by words. But the problem is how to know whether a phrase is a common pattern or not and in what contexts is used. For example, some publications use the technique of annotating the texts with explications about commons patters that appears in them so the learner can use them later. I think a good added value to your system could be to annotate some texts in a similar way so you can offer another service.
What is your opinion about this?

Best regards,
Israel.

[Steve Kaufmann]

Israel,

Rubem Alves, the Brazilian educator that I am listening to and reading right now as part of my Portuguese learning effort, says a number of things that I agree with.

1) The importance of an emotional involvement with the the subject matter.It is most important that you like what you are reading and listening to. This engages you emotionally.

2) The importance of context. What we study must be relevant to us, and hopefully interesting.

3) The learner needs first of all to learn to see. The learner is at the centre of the learning adventure, not a teacher or text book editor.

At LingQ we are hoping to collect an unlimited amount of language content, in order to offer unlimited choice of content for our learners to explore. We also encourage the learner to import content from sources of interest to him or her. Unless phrases are so common as to be listed in a dictionary, there is no way we can anticipate which phrases the learner will "see" and want to save.

In my experience, most notes provided by these annotators of language books are uninteresting and a distraction. I think it is more important to enjoy the content, connect with it emotionally and learn to see, observe, the language and choose the phrases you want.

With words we can easily indicate the frequency level, with phrases we cannot. We do not intend to go over all the content on our site to create notes that may or may not be of interest to some of our learners. On the other hand learners can ask on the Forum and save any replies they get there from learners or tutors.