In this post, I’m going to provide a quantitative analysis of what Pimsleur teaches and evaluate how efficient the Pimsleur method is, in terms of time spent and content learned. I’m then going to do something similar for Duolingo and share my thoughts on the pros and cons of each method.
Before I get into it, though, I want to get a few facts out in the open.
I have a lot of respect for Pimsleur, and I really like their products. My website sells the Assimil product line, but I’m a huge fan (and user) of Pimsleur (and Michel Thomas, for what that’s worth.)
For what it’s worth, I think that Duolingo is a great utility, and I’m a semi-regular user of Duolingo as well.
I should also point out that this post is not intended as a take-down of the Pimsleur or Duolingo methods; that’d be far too self-serving for a guy who sells a competitive product, so let me make a few prefaces before I go any further:
Although I’m going to get into some criticisms of Pimsleur and Duolingo, I should note that no system is perfect, and at some point, I’ll get around to doing a quantitative analysis of Assimil products as well.
Finally, my goal in this review is to be as fact-driven and objective as possible. If anyone sees any holes or errors in my methodology or my findings, drop me a line so that I can improve upon what I’ve tried to do here.
So with that all covered, let’s get into the analysis. I’ll begin with a quick overview of how the Pimsleur Language Programs came to be. (I’ll avoid boring you with Duolingo’s history, since it’s more well-documented.)
For anyone not familiar with Pimsleur Language Programs, they are the creation of Dr. Paul Pimsleur; he created his program in the late 1960s around the same time that he published an academic paper titled “A Memory Schedule”.
It was a pretty short paper, describing Dr. Pimsleur’s observations of how time affected a student’s ability to correctly remember newly acquired language knowledge.
Interestingly enough, Dr. Pimsleur never actually published the underlying data upon which his paper was written. That’s not a knock on his work or the efficacy of its outcome. I think that the Pimsleur method was novel and effective when it was first introduced; it’s just a statement of fact.
Sadly, Dr. Paul Pimsleur passed away in 1976 at the age of 48 from a heart attack, and his intellectual property was acquired by publishing powerhouse Simon & Schuster in 1995.
Dr. Paul Pimsleur pioneered the Pimsleur Method in the 1960s, long before computers were widely available, let alone affordable. The simplest way to describe his method is the combination of traditional “listen and repeat” learning methods combined with a simple form of spaced repetition. The Pimsleur Method also makes use of a technique called back-chaining. It’s not a technique that Pimsleur invented, but it’s one that he made great use of and helped to popularize.
I should also note that when I say that Pimsleur’s spaced repetition was “simple”, I am referring to the technology available when the Pimsleur Method was first introduced. In no way is the term “simple” a knock on the man or his work.
In summary form, this is how the Pimsleur Method works:
The description above is just a simple summary. My intention here is not to review the Pimsleur Method; there are no shortage of reviews or in-depth evaluations on the Internet. My intention here is to provide a quantitative analysis of the method and evaluate how efficient it is, in terms of time spent and content learned.
There’s no question that the Pimsleur Method works. It works very well. I have used Pimsleur Language Programs to learn Russian, and I’m dabbling with Turkish and Arabic. That said, as many reviewers have pointed out, the lessons can be a bit long in the tooth, and I don’t know how often the dialogues are updated.
In the modern age of apps, with computer-based spaced repetition systems (SRS) that react to the student’s triumphs and errors, I think it’s a fair question to ask how efficient the Pimsleur Method is when it comes to use of time.
I purchased the Pimsleur Russian series, levels I – III as MP3 files. Over the course of 3 – 4 months, I listened to each lesson just about every day. In total, there are 90 lessons, and each one is about 30 minutes in length.
If I listened to every lesson just once, I would have invested 45 hours into the complete program. In reality, I listened to about half the lessons more than once, so I’d say that my total amount of time invested is closer to 60 hours. That said, let’s give Pimsleur the maximum benefit and say that I wasn’t a great student, and that most people would only have to listen to each lesson once. (This is a generous assumption since Pimsleur’s narration advises to listen to each lesson as often as necessary to get a level of comfort with the content covered.)
I think that Pimsleur did a great job forcing me to speak the Russian language and practice my pronunciation. That said, after I completed Pimsleur, I did Michel Thomas’ Russian program, and I’m now about one-third of the way through with Assimil’s Russian program. With these additional resources, I came to understand how little grammar Pimsleur addresses, and I became really curious as to what, exactly, I picked up in those 45 – 60 hours.
So I went back and had every single Russian Pimsleur lesson transcribed. I know that sounds insane, but I hired an English-Russian transcriptionist to type out everything that was said on every lesson. Interestingly, I found that Pimsleur Russian covers 101 dialogues over the course of the 90 lessons.
In retrospect, I think that makes sense. There were several cases where a related dialogue was introduced at the end of one lesson, then reinforced at the beginning of the subsequent lesson.
I should also add that many reviewers note that the structure of lessons and dialogues across Pimsleur programs is pretty consistent. So while my review is limited to Pimsleur Russian, the general findings and trends are likely to be pretty consistent for most, if not all, Pimsleur Language Programs.
So over the course of 45 hours, I learned 101 dialogues. But how many actual unique pieces of information did I learn?
I took all of the dialogues and de-duped them by speaking line. (“De-duped” means removing duplicate entries.) If one speaking line in a dialogue was: “Yes, thank you,” and if that speaking line appeared more than once, it was de-duped. However, if one line read: “Yes, thank you,” and another line read: “Yes, thanks,” I would be left with two unique lines.
With this approach, I was left with 693 unique spoken lines. That means that I learned 15.4 lines per hour of time invested. That seems a bit inefficient, but a “spoken line” is not really a fair unit of measure. You’ll understand why when you see examples of unique spoken lines learned:
As you can see from the real examples above, there’s little consistency in the length or diversity of spoken lines. Some consist of simple phrases, others are comprised of sentences with more complex clauses, and three of the six examples above are comprised of two full sentences.
A better analysis would be to evaluate how many unique words the course covered. This isn’t the fairest way to evaluate the quality of a course, but it’s the most consistent way to evaluate the efficiency of knowledge learned for time invested.
Fortunately, when I had all of the transcriptions done, I also had all of the Russian phrases translated into English. This means that I can look at all the unique words through the lenses of both English and Russian.
Using English translations of the Russian content, there are a total of 4,881 words in those 693 spoken lines. That means that, on average, each line is about seven words long.
Using the actual Russian content, a total of 4,091 words were used in those 693 spoken lines, leading to an average of about six words per spoken line.
In case you’re wondering why there’s a discrepancy, you should understand that there are many differences between any two languages. There are many differences between English and Russian, but one of the most notable is that the verb “to be” (or “быть” in Russian) is not used in the present tense. This is just one example, but a notable one that illustrates why and how there could be discrepancies in word usages between the Russian content and its English translation.
In case you’re curious regarding which words were used the most often, here’s a word cloud of all the words that Pimsleur uses in its Russian course. Again, since Pimsleur Language Programs tend to follow very similar formats across languages, what you see below is probably indicative of most Pimsleur products. For those of you who don’t speak Russian, I’ve also provided an English word cloud of all the word translations.
For those of you who can’t read or write Russian, here’s the English translation of the Pimsleur vocabulary word cloud:
As you may notice, there are some differences between Russian and English, but by and large, the word clouds are fairly similar. The words “I” and “you” and “to” stand out in both translations, although the Russian word “no” is more prominent than in the English translation. In general, interesting, but if you’re looking for more insights, read on. (I get into why there are variances between the English and Russian translations in just a little bit below.)
That’s a pretty easy question to answer: around 500 – 600.
Ok, so what do I mean by that, and why is the upper end of my range 20% higher than the lower end of my range?
First I’ll explain using the English, and then I’ll explain using the Russian data set. When looking at translations of the Russian content that Pimsleur teaches, my analysis shows that there are 490 unique words that are taught in those 45 hours of instruction. That said, this figure is somewhat misleadingly high.
Why is that? There are two reasons: words and lemmas. For example, one of the unique English “words” taught is “Aleksandrovich”, which is actually a Russian surname. As for lemmas, here are three examples of words that are actually all derivatives of the same lemma, arrive: arrived, arrives, arriving. Pimsleur doesn’t teach much formal grammar, but one could argue that there’s really only one word there, and that it’s simply used differently based on grammar rules- in this case, verb conjugation.
In the actual Russian course content, there are a total of 628 unique words that are taught. The issue of names is the same in Russian as in English, but if anything, the issue of lemmas is more acute. This owes to the fact that Russian grammar is more complex than English. (That is to say that Russian has more formal grammar rules than does English; I wouldn’t want to imply that one language is “more anything” than another.)
For example, Russian has a formal and informal pronoun for “you” (“вы” and “ты”, respectively). Additionally, like many European languages, Russian nouns have gender – masculine, feminine or neuter. In some verb tenses, verbs are conjugated differently based on the gender of their direct objects. For example, the Russian verb for “was” is “был”, but can appear as “была” (for feminine nouns), “были” (for plural nouns) and “было” (for neuter nouns). In case you’re curious, “был” is used for masculine nouns.
Additionally, in all tenses, Russian adjectives take suffixes that agree in number and gender with the nouns they modify. For example, the Russian adjective for “good” can appear as “хорошо” for masculine or neuter nouns and “хорошая” for feminine nouns. These complexities lead to a higher word count for the Russian content, when compared to its English translations.
So where does that leave us? I’ll be as generous as possible and say that all 628 “words” that are taught in the Russian Pimsleur Language Program constitute “useful things”. So that means that Pimsleur taught me 628 “useful things” in the Russian language over the course of 45 hours.
That means that I learned about 14 “useful things” (words) per hour for each of the 45 hours that I spent on the course.
(Actually, since I listened to many of the lessons more than once, I’d say I probably invested more like 60 hours of time, which means that I learned a little less than 10.5 “useful things” per hour of time invested. But like I said, I wanted to be as generous to Pimsleur as possible in this analysis.)
Now before I go any further, I should be the first to point out that while this sounds disturbingly inefficient, it’s not a fair statement. There’s a huge differences between learning words and learning actual phrases.
Also, since I modified my question to evaluate how many “useful things” Pimsleur taught me, I should factor in those spoken lines. So let’s do that now.
As I said before, the Pimsleur Russian Language Program is comprised of 693 unique spoken lines, but many lines are more than one sentence long. Breaking them out into individual sentences and then de-duping any repeated sentences, we’re left with 823 unique sentences. Now, I should point out that some of these “sentences” exist of things like, “Yes”, and “Oh, yes!” and the like, but that’s ok. These are fully grammatically correct statements, and again, I want to be as generous as possible.
In the spirit of generosity, I’ll even add the unique sentences to all the unique words and use only the Russian course content, since the word count is higher. So we have:
628 “Useful things” (words)
823 “Sentences” (grammatically correct statements)
1,451 Total things that Pimsleur taught me
So over the course of 45 hours, that works out to 32.2 things learned per hour. (That’d be 24.18 things learned per hour using the more realistic 60-hour estimate I referenced earlier.)
Quite honestly, I’m not sure.
I thought it’d be interesting to look at the words that Pimsleur teaches and compare them to a frequency data set. Of course, there are many word frequency data sets floating around out there, and all of them come with various trade-offs and caveats. In order to keep things as apples-to-apples as possible, I decided to compare the Pimsleur data sets with an index of word frequencies for both Russian and English lemmas as published by Rutledge.
I chose Rutledge for two simple reasons:
1. Rutledge is a well-respected academic publisher that puts a lot of time and care into developing its word frequency lists.
2. Rutledge had word frequency lists available for both English and Russian.
In case you’re curious, here are the Amazon links to the actual published sources of the data from Rutledge for English and the Russian. (These are straight links, no affiliate stuff. I just want folks to know where my data was coming from.)
It took a little bit of data massaging to get the data sets as comparable as possible. I had to manually go back into the English and Russian data sets and assign a lemma to each word from the Pimsleur data set so that I could do an apples-to-apples comparison across Pimsleur and Rutledge.
Here’s what I came up with:
All the words in the data set are Russian lemmas from Pimsleur’s Russian course, levels I – III. Words marked in blue are those words from the Pimsleur course that also appear in the top 500 words from the Rutledge data set.
I find it very interesting that among Rutledge’s top 500 words, there are relatively few of Pimsleur’s words. Now, in some ways, I think this can make sense. For example, the word “help” is one of Pimsleur’s top words, but the word “help” isn’t even in the top 200 most frequent words from the Rutledge data set. This makes a lot of sense, of course, since word frequency lists are measured across native speakers, while helpful words for a new learner of a language might vary.
But getting back to my original point, how efficient is the Pimsleur method? Efficiency and effectiveness are two different things. Sadly, there’s been very little research done on effectiveness, and almost no work done on efficiency. That said, one of the few studies that looked at efficiency was done by Duolingo.
According to this post:
On average, it takes 34 hours of Duolingo to learn the equivalent of one semester of college. Since a semester of college generally takes a lot more than 34 hours of work, this suggests that Duolingo is more effective than a college course.
The study was done by an external research team that previously evaluated the effectiveness of other methods such as Rosetta Stone. It is of note that it took 55 hours of study with Rosetta Stone to reach the equivalent of one semester of college. So not only is Duolingo free-er than Rosetta Stone, the study suggests it’s also better.
That sounds pretty impressive, but having read the entire 25-page paper published by Duolingo, here are a few things to bear in mind:
The study provided native-English speakers of non-Hispanic origin a college-level test on Spanish before and after completing the Duolingo Spanish course.
Overall improvement was 91.4 “points”, and this was statistically significant. It’s not clear what “points” are, or what the total possible points could have been. I have trouble believing that the scale consisted of 100 points, because that would imply that on average, everyone in the study would have scored an A- on the Spanish test post-study.
Additionally, out of an initial population of 556 people, 170 were excluded for a variety of statistical reasons. Of the remaining 386 participants in the study:
The average age of eligible participants was 32.0 years with 48.4% females, and 99.0% of them were novice to intermediate (self-report) users of Spanish.
That’s interesting. The paper doesn’t get into much detail on defining “novice to intermediate”, but I suspect the definition is not “speaks absolutely zero Spanish”. That would imply that the study participants had a bit of a head start, which means that comparing them to college students starting from zero may not be a fair comparison. (Of course, that assumes that all college students who study Spanish start from zero, which is also probably a flawed assumption.)
The study goes on to state that 11.0% of study participants’ spouses, partners or close friends spoke Spanish, and that 4.5% of participants’ parents, grandparents or great grandparents spoke Spanish.
More than a quarter (28.5%) of participants knew at least one other foreign language.
Finally, the paper concludes that the average Duolingo user could learn the equivalent of one college semester of Spanish with 26 – 49 hours of time spent on Duolingo.
Now don’t get me wrong. I think Duolingo is a pretty interesting app, and I applaud them for using an outside research firm to evaluate the efficacy of their approach to language learning, but:
Now that said, since I’m a little bit crazy, and since I’m using Duolingo on a semi-regular basis to practice my Russian, I decided to grab a list of the Duolingo vocabulary set for Russian. (Duolingo makes all vocabulary courses for all its languages available through its Tiny Cards app, and that’s where I got the data set.)
With the Duolingo data, I wasn’t able to form a word cloud, since I don’t have all the unique sentences that are used for all the Russian lessons. All I have is a list of words. As was the case with the Pimsleur data set, I did my best to normalize the data and account for lemmas.
This time, I thought I’d mix things up a bit and put together some pie charts. So here’s what happens when we compare all the words in the Duolingo Russian course with the top 500 words from the Rutledge data set:
What does this mean? It means that 67% of the top 500 words in the Rutledge data set are not found in the Duolingo Russian course. In case you’re curious, the green, yellow and orange data shows the percentage covered in the top 10, 25, and 100 words.
Now this does NOT mean that there are tons of words that Duolingo doesn’t teach. It just means that the emphasis on those words, as defined by their presence in Rutledge’s top 500 most frequent words, is prioritized differently.
By the way, after playing around with this for a while, I did a Pimsleur infographic to compare, and this is what I came up with:
As you can see, Pimsleur’s coverage is actually a lot better than Duolingo’s. This makes sense, too, since Pimsleur’s been around for so long, and it was designed by language teachers (as opposed to app developers.)
Great question. The short answer: we collectively need more research done to evaluate the efficiency and effectiveness of language learning programs, and those studies need to look at verb, written and reading comprehension of a language – not just use a standard, written college test.
For my own part, whether we go with the 45-hour estimate of the 60-hour estimate, I’m pretty happy with the fact that Pimsleur taught me around 30 “useful things” per hour. That’s a new nugget of knowledge every 2 minutes, on average, of time spent invested with the program.
That feels like a pretty good use of my time. Moreover, while I’ve dabbled with Duolingo’s Russian course, Pimsleur’s course actually taught me how to speak. (It didn’t teach me how to read, which Duolingo does fairly well, but it got my jaw yapping, which is important.)
Overall, I’d say that Pimsleur is a very solid language program. I have my personal views on the underlying content it teaches, but I’ll keep them to myself.
In case you’re curious on my views about Duolingo, rather than repeat myself, you might want to check out this related post on Why Apps Suck. (You might think I’d be down on Duolingo, but you might be surprised to see that I’m actually not….)