Combating Knowledge Interference (Flashcard Refactoring)

I came across this computer networking Anki flashcard I’ve forgotten over 7 times:

(NW for Sysadmins: Ethernet) Address Resolution Protocol (ARP) – Maps Ethernet addresses to IPv4 addresses and back.

The card uses cloze deletions [] like this:

  • (NW for Sysadmins: Ethernet) [Address Resolution Protocol (ARP)] – [Maps Ethernet addresses to IPv4 addresses and back].
  • (NW for Sysadmins: Ethernet) Address Resolution Protocol (ARP) – Maps [Ethernet addresses] to [IPv4 addresses] and back.

Interestingly, I haven’t forgotten a single review for the right-hand side clozes. Turns out this was the one causing me trouble:

(NW for Sysadmins: Ethernet) […] – Maps Ethernet addresses to IPv4 addresses and back.

Why? The issue seems to be another card in my deck that is very similar, and I’m confusing the two. The other card quizzes a networking concept called “Neighbour Discovery (ND)”, which performs a similar function to ARP except it maps IPv6 addresses to Ethernet and back rather than IPv4 addresses. This is a good example of interference, which refers to the fact that learning similar things can make you confuse them (see Rule 11 of Poitr Wozniak’s classic article on the 20 rules of formulating knowledge).

So the solution I’m opting for is pretty simple, just add a hint:

  • (NW for Sysadmins: Ethernet) ARP (hint: not ND) – Maps Ethernet addresses to IPv4 addresses and back.

One other small improvement is adding another card for the acronym alone:

  • Front: (NW for Sysadmins: Ethernet) ARP (Unpack Acronym)
  • Back: Address Resolution Protocol

These interference issues are tricky because you can’t really anticipate them in advance. You have to discover them as you review your cards.

Another annoyance is I’m not 100% sure that interference was actually the problem. Ideally, I would have discovered this troublesome card during review, so I could know for sure why I’m failing.

So here are some lessons learned from this little exercise:

  • Use hints as an effective tool for reducing interference.
  • Keep an eye out for interference during review of your knowledge. As soon as you encounter it, note it. In the case of Anki, there is a “mark card” feature. I also recommend actually writing text within the card to remind yourself exactly how you failed the card when you fix it later. It would be nice to be able to see basic card statistics, like number of lapses, during review without having to go into card statistics. I inquired on reddit whether there was an addon for this and while there are some good options for desktop, it doesn’t seem like there’s anything that quite meets this need for mobile (where I do all of my reviews).
  • As you get better at knowledge building, interference will become your most common problem. As Poitr Wozniak says, interference is “probably the single greatest cause of forgetting in collections of an experienced user” of spaced repetition systems since it is hard (impossible?) to avoid even if you are really good. You typically discover it during knowledge review time, not knowledge construction time.

Flashcard Refactoring

I’ve started a weekly habit of flashcard review. I want to share with you my thought process for modifying my cards, because I think this will be valuable to help you improve your own knowledge construction skills.

I also want my flashcard development out in the open so you can call me out when I make mistakes and provide suggestions for further improvements. Please do reach out! I am by no means the ultimate expert in knowledge construction.

So, I will be doing a regular series I call “Flashcard Refactoring” (Refactoring comes from the programming term which basically means revising and improving your code).

To sniff out poor flashcards, I ran prop:lapses>7 in the Anki browser to get all the cards I’ve forgotten over 7 times. Here’s one I came across about a command in the Linux command line to a suspend a job:

  • Side 1: ^Z (Linux Command Line)
  • Side 2: Suspend a job running in the foreground (Linux Command Line)

The card is reversible, so are two cards in total: one with Side 1 as the question, and another with Side 2 as the question.

At first glance, it doesn’t look too bad. It’s fairly concise. But one quick and easy change is reduce words in Side 2, in accordance with the 12th rule of Formulating Knowledge (“Optimize Wording”):

  • Side 2: Suspend foreground job (Linux Command Line)

This is a nice little improvement, but why am I really forgetting this card? I think it’s because ^Z doesn’t really have any meaning – it seems arbitrary and it has no clear connection to suspending foreground jobs.

So, I’ll create a fake connection, i.e. a mnemonic.

The mnemonic that immediately came to mind was the fact that the beginning of “Suspend” kind of sounds like a “Z”, e.g. “Zuspend”. I think this is all that’s required for this to stick in my memory (but only time will tell).

When you come up with a mnemonic, it’s a good idea to create a separate card for it, so I added the following to my deck: 

  • Q: Mnemonic for remembering ^Z suspends foreground job in Linux Command Line.
  • A: Zuspend

To receive content like this weekly in your inbox, subscribe to my Spaced Repetition Newsletter, which provides latest news, tips, and ideas about spaced repetition, using learning tools like Anki, and improving your learning productivity.

Anki / Spaced Repetition Tip: Review your Weak Flashcards

To receive content like this weekly in your inbox, subscribe to my Spaced Repetition Newsletter, which provides latest news, tips, and ideas about spaced repetition, using learning tools like Anki, and improving your learning productivity.

I’ve been a long-time user of spaced repetition tools. I’ll never forget first hearing about SuperMemo from a close friend as I started my undergraduate degree in 2005. I was immediately sold on the value of spaced repetition, and I particularly liked the idea of computers automatically taking care of review scheduling for you. I started using SuperMemo as a central tool for studying, and saw my academic performance skyrocket.

Over the years, I’ve slowly improved my skill in designing flashcards. It is by no means a trivial skill: it took me years to get pretty good at it, and to this day I still often make flashcards that are complete failures.

I believe there will eventually be an open collaborative platform for flashcard development and sharing, where experts can contribute and refine perfectly crafted cards. Users contribute their deck statistics, revealing poorly formed cards and contributing to our understanding of optimal flashcards.

But until that day, it pays to develop your flashcard creation skills.

Flashcard quality is top of mind for me since I’ve revisited the classic article by Peter Wozniak (of SuperMemo fame), “Effective Learning: Twenty Rules of Formulating Knowledge)”. It is a must-read for anyone that creates flashcards for learning (i.e. almost everyone at some point in their life). I’ve published my summary notes on this article (aside: my notetaking tool of choice is Roam my notes are easy to copy-paste into your own Roam database if you happen to use it as well).

One great way to improve your flashcard development skills, while simultaneously improving the quality of your deck, is to review your old cards regularly. Review your top 10-20 most problematic cards weekly, and for each one you encounter, do one of the following things:

  • Revise: With the Twenty Rules of Formulating Knowledge by your side, refine your card or break it down into a larger number of small, easy to digest cards.
  • Suspend: If you don’t think you need to have a card in spaced repetition anymore, but don’t want to delete it entirely, suspending is a good option.
  • Delete: If you know the knowledge is completely useless to you, trash the card entirely.

But what cards should you review? If you’re like me, you have a pretty big collection, and it’s just not feasible to review all your cards every week to find the weak ones.

Anki makes it quite easy to find these problematic cards. Two main search commands in the Anki Browser are useful here:

  • tag:leech – this finds all of the “leeches” in your Anki deck, which are cards that you keep forgetting. By default, Anki tags your card as “leech” when you fail a card 8 times.
  • prop:lapses>n – this reveals all of the cards you have failed (“lapsed“) over n times. You can set n to whatever number you like. Start with high-n cards and work your way down.

In addition to using these search techniques, I try to make a habit of “marking” cards that are problematic or poorly formed in some way, during review. If it’s an easy correction (e.g. obvious suspension, or small text changes), I’ll make the change right away in the mobile app. Otherwise, I will simply mark the card and filter it out during weekly review to make improvements.

When you do revise your cards, I recommend “resetting” the card so it’s like a “do-over” – the card should be reviewed again as if you just created it. This serves two purposes: it ensures that the card will no longer show up in your “problem cards” lists when you do the above queries. It also provides you with more opportunities to review your new formulation of the knowledge.

Unfortunately, it seems the only way to do this in Anki is do create new card(s) with the information you want and delete the old one. There is an option for “rescheduling” the card, but this only restarts the review process and doesn’t delete your review history. As a result, the card will still appear as one of your problem cards if you do a query like prop:lapses>n. Luckily, it’s not much extra effort to do this.

I have to admit that I do not entirely practice what I preach here. Weekly review of my cards is something I haven’t fully incorporated yet, but I’m resolving to start doing it today. In the next weeks, I’m going to experiment with a Flashcard Refactoring series to illustrate the card refinement process. Stay tuned!

To receive content like this weekly in your inbox, subscribe to my Spaced Repetition Newsletter, which provides latest news, tips, and ideas about spaced repetition, using learning tools like Anki, and improving your learning productivity.

Roam Notes on Poitr Wozniak (Supermemo) Twenty Rules of Formulating Knowledge

  • "Author::" [[Poitr Wozniak]]
  • "Source::" https://www.supermemo.com/en/archives1990-2015/articles/20rules
  • "Recommended By::"
  • "Tags:: " #Flashcards #[[Spaced Repetition]] #[[flashcard design]] #Learning
  • Summary

  • The rules are listed in order from most important / common to least.
  • Rule 1: Do not learn if you do not understand. Trying to memorize things you don’t understand increase the time to learn and more importantly, reduces the value of the knowledge to nothing (e.g. memorizing a German history book when you don’t know German – you won’t know any of its history). #[[Flashcard Tip: Don’t add Things you Don’t Understand]]
  • Rule 2: Learn before you memorize. He recommends building an overall picture of the learned knowledge before memorizing. You’ll reduce learning time when the individual pieces fit a single coherent structure. So, read the chapter first, then add the cards. #[[Flashcard Tip: Learn Before you Memorize]]
    • Notes: Why can’t you learn with [[Flashcards]] alone? Perhaps this is efficient if presented in the proper order. Also, perhaps the cards need to change when first learning when compared to committing to long-term memory. If so, how do they change? In other words, how are "questions for learning" different than "questions for retention"? #[[Personal Ideas]]
  • Rule 3: Build upon the basics. Start simple, and build from there. Don’t hesitate to memorize basic, obvious things. The cost of memorizing them is small, because they’re easy to answer. "usually you spend 50% of your time repeating just 3-5% of the learned material" source #[[Flashcard Tip: Build Upon the Basics]]
    • Notes: The basics provide [[scaffolding]] that you can build upon. This reminds me of the [[80-20 rule]], where a big chunk of your time is spent on a small number of [[flashcards]]. #[[Flashcard Tip: Track Down and Eliminate Your Problem Cards]].
  • Rule 4: Stick to the minimum information principle. Formulate knowledge as simply as possible. Simple is easy to remember, and having a complex answer means there is more to remember – a larger number of simpler cards covering the same knowledge lets you review each sub-component at its own appropriate pace. #[[Minimum Information Principle]] #[[Flashcard Tip: Follow the Minimum Information Principle]]
  • Rule 5: Cloze deletion is easy and effective. #[[Flashcard Tip: Use Cloze Deletion]]
  • Rule 6: Use imagery. Our brains are wired for them. They usually take more time to create though compared to a basic verbal card, so weigh the benefits. #[[Flashcard Tip: Use Images]]
  • Rule 7: Use mnemonic techniques. He makes an interesting point that these do not solve the problem of forgetting, since the bottleneck is long-lasting and useful memory, not quickly memorizing knowledge. For that, you need #[[Spaced Repetition]]. "Experience shows that with a dose of training you will need to consciously apply mnemonic techniques in only 1-5% of your items". #[[Flashcard Tip: Save Mnemonics for Difficult Cards]] #mnemonics
  • Rule 8: Graphic deletion is as good as cloze deletion. #[[Flashcard Tip: Use Image Occlusion]]
  • Rule 9: Avoid sets. Sets are unordered collections of objects. Very difficult to memorize. If you must, use [[enumerations]] instead, which are ordered in some way. #sets #[[Flashcard Tip: Avoid sets]]
  • Rule 10: Avoid enumerations #enumerations #[[Flashcard Tip: Avoid Enumerations]]
    • He includes a nice method for [[memorizing text]] such as [[poems]] or [[prayers]], without using [[cloze deletion]]
  • Rule 11: Combat interference: #[[memory interference]] #[[Flashcard Tip: Combat Interference]]
    • Learning similar things tends to make you confuse them. [[memory interference]] – "knowledge of one item tends to make it harder to remember another item".
    • "**Interference is probably the single greatest cause of forgetting in collections of an experienced user of **[[SuperMemo]]."
    • The only strategy to work against this is detect and eliminate. It’s hard to know you’ll face interference at card creation time.
  • Rule 12: Optimize wording #[[Flashcard Tip: Optimize Wording]]
    • Shave down the number of words you use. Make your cards as clear and concise as possible. Focus on the piece of information that is important.
  • Rule 13: Refer to other memories #[[Flashcard Tip: Refer to Other Memories]]
    • When you add a new card, try incorporating things you’ve learned from other cards.
  • Rule 14: Personalize and provide examples: #[[Flashcard Tip: Personalize and Provide Examples]]
    • Link your cards to your personal life.
  • Rule 15: Rely on emotional states: #[[Flashcard Tip: Rely on Emotional States]]
    • We remember things better that are vivid or shocking.
  • Rule 16: Context cues simplify wording: #[[Flashcard Tip: Use Context Cues]]
    • They often reduce the number of words you need
  • Rule 17: Redundancy does not contradict minimum information principle #[[Flashcard Tip: Use Redundancy]]
    • Redundancy – more information than needed or duplicate information.
    • It can be good, and minimum information principle does not mean minimum number of characters in your deck.
  • Rule 18: Provide source: #[[Flashcard Tip: Provide Sources]]
  • Rule 19: Provide date stamping: #[[Flashcard Tip: Use Date Stamps]]
    • Particularly for knowledge that changes over time and can become obsolete.
  • Rule 20: Prioritize: #[[Flashcard Tip: Prioritize]]
    • There is way more knowledge in the world than you’ll be able to absorb and remember long-term.
    • Focus on adding knowledge that is most relevant and important to you.

How to Deal with Lists in Anki

Photo by Glenn Carstens-Peters on Unsplash

When adding cards to Anki (or any other spaced repetition system), you’ll find you often need to deal with lists. In the spaced repetition community, these are sometimes called “Sets” (see Anki Essentials, Chapter 7, “The 20 Rules”).

For example, here’s a section from a book I’m working through called The Art of Doing Science and Engineering by Richard Hamming, where he summarizes the main advantages of computers over humans:

“People are sure the machine can never compete, ignoring all the advantages the machines have…These are: economics, speed, accuracy, reliability, rapidity of control, freedom from boredom, bandwidth in and out, ease of retraining, hostile environments, and personnel problems.”

This is a frustrating passage, because it’s packed with valuable information but tough to turn it into flashcards.

One easy approach is to cloze delete each individual option, and then cloze delete the entire list to make sure you can recall the whole thing. While this makes the cards easy to create, you’ll run into these issues when reviewing them:

  • It’s a lot to read, and you have to read every option to know what the missing one is.
  • It’s hard to remember. Even if you do individual cloze deletions for each point, you will still be very likely to fail on the question where you are asked to recall the entire list. Also, many lists (including the example above), are open ended and not necessarily exhaustive. So, when it comes time to review, you’re not quite sure what’s on the list, and you may list things that are “correct” but just not included on that particular list.
  • It violates a fundamental rule of flashcard creation. The “Minimum Information Principle” states that your questions and answers should be as simple as possible. A large number of simple cards is far more efficient to remember than a small number of big, demanding, awkward cards.
  • It’s hard to rate yourself. How do you rate your performance when you get 9 out of the 10 answers in the list correct? Do you fail yourself if you only got one wrong?

So, what are our options for dealing with these annoying lists?

Break it Down

First, ask yourself: do I really need to memorize the list? Do you really need to be able to spout off the list of advantages of computers over humans, and not miss any of points? Probably not.

Then ask yourself, what do I really want to get out of this list? In my example, I want advantages of computers over humans to pop up in my mind when the time is right. For example, when I’m working on a process at work that is repetitive, rote, and requires accuracy, I want to recognize computer scripts as a good solution.

So, I would not Ankify this specific list at all. Instead, I would look in the source to Ankify material focused on each individual point. For example, I could add specific questions on the relative costs of computing compared to manual effort, or details about the speed of computing compared to hand calculations. Doing this will give a deeper understanding of the advantages of computers over humans. It will probably also increase the likelihood of being able to recall the entire list at once (even though it’s unlikely I’ll need to do that).

You can also lists into sub-categories and then add those smaller lists to spaced repetition. Aim for each sub-category to have 1-2 items.

Take Advantage of Special List Structure

Sometimes you can take advantage of the inherent structure of a list to break the flashcard down. For example, your list can take a special form of a “1-n relationship”, as discussed in this thread .

A 1-n relationship is best understood by example. Suppose you want to memorize the Provinces in Canada. This list has the special property where each province in the list is associated with one and only one country (i.e. 1 country – n provinces). With lists like this, you can add questions like “what country does the province of Manitoba belong to?”, “What country does the province of Alberta belong to?”, etc. Of course, this fails if there is a province in another country with the same name, (that would mean it’s not a 1-n relationship), but you get the point.

Turn the List into an Enumeration

One of the problems with the example list of computer advantages is the order of the items has no meaning. There is no reason why “personnel problems” should be last rather than first. This makes it far more difficult to memorize because it’s just a bunch of seemingly random points.

Enumerations are ordered lists. Although they can still be difficult to memorize, they at least have some structure that your mind can latch on to. You should always try to convert your list into an enumeration if it’s not one already.

Ideally, your ordering will contain useful knowledge itself. For example, you could list the advantages of computers over humans, from most important to least important according to the author.

There are many other possibilities for turning lists into enumerations, depending on what you are trying to memorize, such as chronological order, area of the body, size, popularity, population, etc. If you can’t find a meaningful ordering, you can always list items alphabetically.

Enumerations are a free lunch, because they make list easier to memorize and provide bonus knowledge about the material. For example, wouldn’t it be much better to know the order of the planets in the solar system from closest to the sun to furthest, rather than just knowing the individual planets?

Use Cloze Overlapper

After you have turned something into an enumeration. How should you design your repetitions?

One option is to do one cloze deletion per item in the list. For example, take the list (a, b, c, d, e, f). Your cloze deletions are then ([…], b, c, d, e, f), (a, […], c, d, e, f), (a, b, […], d, e, f), etc.

I have done this many times in the past, and while I can usually answer the questions quickly and accurately, I find it hard to retain the information. When asked to reproduce the whole list, I fail. This is because the questions are too easy – removing only one item provides way too much context.

Until recently, I thought this was the best option. Then I came across the “overlapping cloze”: close deletions on individual items in the list, but with only a small amount of context given. For example, something like this: ([…], …, …, …, …), (a, […], …, …, …), (…, b, […], …, …), etc.

It’s actually quite painful to create overlapping clozes in vanilla Anki. Luckily, there is an add-on for this: Cloze Overlapper. This video by Glutanimate, the author of the add-on, provides an overview of issues with lists in and illustrates how how to use the add-on (aside: the author is the developer of another great add-on, image occlusion enhanced). I haven’t tried out Cloze Overlapper yet, so I can’t fully vouch for the technique or the add-on, but it seems to be highly regarded in the spaced repetition community.

Use Mnemonic Techniques

With a bit of practice, mnemonic techniques can help you achieve incredible feats of memory. Many are easy to learn. Even doing a little bit of studying on these techniques can produce big payoffs for your memory.

The technique I probably use most is the link method. For example, one flashcard I have on computer networking asks me to recall the main types of datalink layer errors. I added it as an unordered list: frame errors, drops, collisions, and overruns.

It’s actually a pretty poor card, but I haven’t had much trouble memorizing it because of the mnemonic I used:

  • I think of Link (from Zelda) with a shirt that says ERROR on it (dataLINK layer ERRORS).
  • Then I imagine him putting on some glasses (frame errors)
  • Link throws the glasses down to the ground (drops)
  • A giant truck hits Link (collisions)
  • Finally, the truck runs over the glasses that are on the ground (overruns)

Sounds silly, but it works. Our minds are extremely good at retaining images.

I am by no means an expert in these techniques, but I’ve found that learning a little bit helps a lot. If you go deeper, I’m sure you’ll see even bigger benefits. Other useful memory systems include peg and Method of Loci.

If you use a mnemonic technique for a list in your spaced repetition system, be sure to add flashcards to remember mnemonic itself.

Conclusion

Avoid lists if you can, but if you must, try to break them down into sub-categories, make sure they’re ordered (preferably with some relevant meaning to the ordering), use cloze overlapping to implement the reviews and limit context, and use mnemonic techniques. Together, this will give you a solid arsenal for committing lists to memory without much pain.

Resources

Notes on Developing Transformative Tools for Thought

Occasionally, special tools come along that amplify our minds, enabling new kinds of thought not otherwise possible. Computers, writing, speaking, and the printing press are all examples of these “Tools for Thought” that surge human potential. 

This essay from Andy Matuschak and Michael Nielsen explores whether we can accelerate the development of these kind of tools. They also provide a taste of potential tools for thought with their prototype “mnemonic medium”, an interactive post on quantum mechanics called “Quantum Country” with embedded flashcards combined with a spaced repetition system delivered through email follow-ups. 

Their essay is a must-read for anyone interested in spaced-repetition or productive learning. There are several points I found thought-provoking. I believe each of these insights indicate a need for a new Tool for Thought for flashcard development and sharing.

Spaced repetition creates exponential returns to studying

Based on Quantum Country user data, Matuschak and Nielsen estimate that devoting only 50% more time to spaced repetition after reading the essay resulted in users recalling the key points for months or years. 

In other words, relatively small investments in spaced repetition after reading an article produces outsized results – more evidence to place on top of the mountain of research suggesting spaced repetition works. 

Good flashcard development is difficult

Matuschak and Nielsen note that it takes a surprising amount of skill and time to build quality flashcards, especially for abstract concepts. This is probably a big reason why most people fail to adopt spaced-repetition tools like Anki. Since flashcard development is a skill that you develop over much time and effort, new users tend to add cards in a way that inevitably leads to frustration and failure. 

This may partially explain the efficacy of Quantum Country: the authors are experts in both quantum mechanics and flashcard development – a rare but essential combination of skills for their essay to work. 

Flashcards written by others can be useable

Some people the spaced repetition community don’t believe in using flashcards created by others, and with good reason. They’re often poorly written. They’re idiosyncratic. They’re missing crucial contextual information that you lack as someone who hasn’t read the original source material. I used to be one of these non-believers.

But the effectiveness of the Quantum Country essay suggests that shared flashcards can work well. This has benefits of saving users of the burden of flash card creation, as well as preventing new user frustration from poor flashcard building skills and poor domain knowledge.

Matuschak and Nielsen hypothesize that the quality of their flashcards is what makes this work. I agree, but I have a few more hypotheses: 

  • Their flashcards are introduced in a logical progression as users read the essay. In contrast, shared decks in Anki shuffle cards randomly and are not encoded with dependency information.
  • Their flashcards are clearly connected to a source (i.e. the essay), providing important context for the user.
  • Users learn the material before they review flashcards. This is in line with the common wisdom that flashcards don’t work if you don’t already understand the material – they are a tool for retention, not learning. Aside: is this common wisdom true? I’m not so sure. Socrates taught using Q and A, so why can’t you teach a subject entirely with flashcards? If it is possible, what are the prerequisites to making it work? 

Elaborative encoding

Matuschak and Nielsen note elaborative encoding as another learning tool shown to be extremely powerful in promoting memory. Essentially, it means connecting new ideas you want to remember with old ideas you know well, providing a fast path in your brain to new information. 

Remember this concept while developing your flashcards. Whenever you add a new card, think about what you already know well and how you can connect this to the new knowledge.

A New Tool for Thought?

Matuschak and Nielsen’s article has renewed my interest in a tool for thought idea I’ve been pondering for quite a while: a platform for collaborative flashcards development and sharing. I believe such a tool, if properly developed, can address the issues that limit the use of spaced repetition:

  • Spaced repetition practitioners currently need to develop their own flashcards, which requires a significant amount of time, domain expertise, and flashcard-building skill. There needs to be a place where experts can create shared flashcards, and there should be a proper incentive structure encouraging creators to improve these flashcards over time.
  • Flashcards are not clearly connected to original sources. Spaced repetition practitioners should be able to pull up pre-built flashcards for a source document they are working through. 
  • Current tools do not provide information that link flashcards together (other than knowing two flashcards are part of the same deck, or have the same tag). At the very least, flashcards should have a notion of “depends on” or “prerequisite to”. This would make shared decks more useful by showing the intended progression of knowledge. It would also aid elaborative encoding (e.g. examining cards you’ve reviewed and linking them up to cards “nearby” in a knowledge graph)

I strongly believe a tool like this needs to exist, as you may have guessed if you noticed the Download my Brain feature I built for this site that provides a platform for sharing my personal Anki decks. I have started work on a more generic tool for collaborative flashcard construction and sharing and will keep you posted once I have something ready for production. 

Thanks to Andy Matuschak and Michael Nielsen for the inspiration to follow this path.

If you’re interested in space repetition and techniques for productive learning, subscribe to my Spaced Repetition Newsletter. Check out my shared data science Anki decks here: Download my Brain.

Spaced Repetition for Efficient Learning by Gwern Branwen

If you’re interested in spaced repetition and haven’t read it yet, Gwern Branwen’s essay is a fantastic review, including research into the benefits of testing, what spaces repetition is used for, software, and other observations. There are lots of resources here for people that want to know the research behind why spaced repetition works, including many studies on the effects of testing and the effects of spacing your learning. 

Here are a few points I found most interesting in Gwern’s essay:

People underestimate the benefits of spaced repetition: Gwern references fascinating research on how students and teachers grossly underestimate how much better spaced repetition is compared to cramming for learning. 

Spaced repetition as a tool for maintenance of knowledge, not gaining new knowledge: According to some of the research referenced in the essay, spaced repetition doesn’t teach you new skills or abilities. Rather, it just helps you maintain existing skills.

  • Skills like gymnastics and music performance raise an important point about the testing effect and spaced repetition: they are for the maintenance of memories or skills, they do not increase it beyond what was already learned. If one is a gifted amateur when one starts reviewing, one remains a gifted amateur.” 
  • I’m resistant to this idea that you can’t learn using flashcards. I agree if new flashcards are thrown at you randomly, then yes that is a very inefficient way to learn. But what if flashcards are presented to you in a specific order when you are learning them, and “advanced” cards are not shown to you until you’ve learned the prerequisite “beginner” cards? I don’t know the research on this, but this strikes me as potentially a better way to learn than reading, if the cards are formulated properly.  To my knowledge, none of the spaced repetition tools out there (e.g. Supermemo, Anki) allow for this kind of “card dependency” – if you know of a tool that does this, let me know.

Tradeoff between lookup time and mental space: A key question that you have to ask yourself when using spaced repetition is what information should I add to my spaced repetition system? Why add anything at all to it when you can just look it up on the internet?

  • The answer is that lookup costs can be large, especially for information that you use a lot. Often when you want to apply your knowledge, there isn’t enough time to look it up, or the time to look-up impedes your thinking. An extreme example of this would be trying to recall an important piece of knowledge in a job interview – good luck pulling out your phone in front of the interviewer to get the right answer.
  • To figure out what to add, you have to strike a balance with the tradeoff between lookup costs and the cost of adding the item to your spaced repetition system and reviewing it. Gwern provides a 5 minute rule as a criteria for deciding what to add: if you think you’ll spend over 5 minutes over your lifetime looking something up or not having the knowledge in your head will wind up somehow costing you more than 5 minutes, then it’s worth it to add it to spaced repetition.

Idea – dynamically generated cards: Gwern offers some interesting ideas about the possibility of dynamically generated cards. For example, having a card that teaches multiplication by randomly generating numbers to multiply. Similar ideas apply to chess, go, programming, grammar, and more.

If you’re interested in space repetition and techniques for productive learning, subscribe to my Spaced Repetition Newsletter. Check out my shared data science Anki decks here: Download my Brain.