The Triple-Pass Method to Remember What you Learn, Forever

You read books. You watch videos. You listen to podcasts. There is a firehose of high-quality information available at your fingertips.

But how much do you actually get out of your media consumption? How much do you remember? For a long time, my answer was “not a whole lot”. I was often shocked at how few key points I could recall from material I had read only days or weeks earlier.

To me, this is unacceptable. Why bother doing so much reading if I’m just going to forget it all? Sure, if reading for entertainment, then no big deal. But most of the material I spend my time on is highly relevant to my professional and personal life. I want this knowledge to compound.

I’ve spent years experimenting, testing, and tweaking systems to get more value out of the media I consume and remember key points forever. I’ve eventually landed on a solution that I want to share with you: the “triple-pass” system. Using this system, I’m confident I’m getting everything I can out of my reading, and not a moment is wasted.

High-level Overview of the Triple-Pass System

At the highest level, the triple-pass system consists of the following stages:

  • First Pass: Active Media Consumption. Take notes and highlight important points from the media you consume (e.g. books, articles, videos).
  • Second Pass: Consolidate and Summarize Notes. Export notes and highlights from the first pass to a central note-taking system, then review, refine, and consolidate. This pass produces what Tiago Forte calls your “second brain“: an external system storing your knowledge in a format that can be easily searched and retrieved for use.
  • Third Pass: Commit to Long-Term Memory using Spaced Repetition. Add the most important parts of your notes from the second pass to a spaced repetition system. This stores the information efficiently in your long-term memory so you can access it in-the-moment when needed.

I go into each of these three stages in more detail below, including the technologies I use at each stage.

First Pass: Active Media Consumption

At this stage, you consume the media that you want to absorb. The key feature about this stage is that it is active media consumption, i.e. highlighting and taking notes as you read.

Here are some tips for getting the most out of the first pass:

  • Err on the side of more highlighting rather than less. It’s good to have a lot of context in your highlights. You can always eliminate things that are redundant or unnecessary in the second pass.
  • Highlight chapter titles. Titles provide useful context for organizing your notes and understanding the broader picture for an excerpt.
  • Prioritize the new, useful, and the interesting. There is no need for anything useless and uninteresting to exist in your knowledge base. Also, avoid including things you already know well.
  • Look for scaffolding. Keep an eye out for material that provides a “platform” to tackle more advanced concepts. For example, I always highlight definitions of important terms I’m unfamiliar with. I also like information about people, places, or things, since I can use these as context to learn related topics faster.

Technology I use for the First Pass

The tools I use at this stage vary depending on the type of media I’m consuming. There are two main criteria: i) the tool must allow me to record highlights and notes digitally, and ii) the tool must allow me to export materials to my note taking system (Roam) with little effort.

  • Books: Usually I purchase books on Kindle, which has excellent highlighting. I then use Readwise to automatically sync highlights to Evernote, and can copy and paste from Evernote to Roam (apparently Readwise now has a direct export to Roam, so it looks like I can cut out the Evernote middleman). For physical books, I use the Readwise mobile application, which has an OCR feature that lets you take a picture of the page you are reading and highlight it.
  • Articles / Blogs: I try to read all articles and blog posts on Instapaper, which allows me to highlight and take notes on articles. Again, Readwise allows me to import these notes into Evernote, which I copy / paste into Roam.
  • PDFs: At the moment I don’t have a good solution for PDFs. Since I can’t highlight them easily, I’ll often just take notes directly to Roam as I read, with the PDF open in one window and Roam open in another.
  • Videos: Typically I’ll sit at my computer when watching videos. So, I keep Roam open while watching the video and take notes, with timestamps. Here’s an example of video notes I put together for a Jeff Bezos Lecture on innovation.
  • Podcasts: I do listen to some podcasts (especially Conversations with Tyler), although I haven’t found a great tool for taking podcast notes. I usually listen to podcasts when I’m on the go, so note taking in the browser is typically not possible. Usually, if there’s a podcast that I listen to that’s really good, I’ll just revisit it and take notes while I listen at my computer desk.

Second Pass: Consolidate and Summarize Notes

In this pass, edit the highlights and notes that you’ve exported to your note-taking system. Activities here can include:

  • Deleting irrelevant or redundant notes
  • Summarizing excerpts into key points, while keeping some direct quotes from the material that is notable or quote-worthy
  • Reformulating material into your own words
  • Bold-facing the most important points so your notes are “glanceable”
  • Creating commentary on the material, expanding on points that you liked, critiquing points you disagreed with, filling in missing arguments, creating connections to other material in your knowledge base, or creating a high-level “book-review” style summary
  • Marking particularly important material for long-term memory

A key advantage of the second pass is it produces a valuable digital asset that you can draw on the rest of your life: you now have a searchable, quick-to-read summary of what you have read. The knowledge is now part of your “second brain” for easy access and connection to your existing knowledge.

Yet another advantage of this second pass is that the act of editing helps you absorb the material. This is because it requires elaboration and recall, which are both well-known to foster learning and memory.

Technology I use for the Second Pass

My note taking tool of choice is Roam. It is a fantastic piece of software, although it’s difficult to explain its value in words (you really just have to try it). I recommend looking into it if you are not already heavily invested in an existing note taking app. I find it allows me to easily make connections between knowledge, and its incredible functionality has led me to ditch Evernote, Asana, and 90% of Google Drive.

Third Pass: Commit to Long-Term Memory Using Spaced Repetition

In this third and final pass, add material material flagged “long-term memory” in the second pass to a spaced repetition system.

Spaced repetition is reviewing material at increasing intervals of time, allowing you to remember material with minimal effort. I won’t go into more detail here, but I highly recommend this overview by Gwern Branwen. You can also subscribe to my Spaced Repetition Newsletter.

The advantage of this third pass is access to your knowledge in the moment, when it matters, without any external note taking tools. This is useful for the many situations where it’s not feasible to consult your notes. For example: job interviews, meetings, or creative work where speed of thought is important and you need lots of in-memory scaffolding to make progress.

One nice feature of this third pass is you’ve thoroughly vetted the material in the first two passes. This means you’ll be less likely to add things you don’t need to your spaced repetition system, and you’ll only add information you understand (see rule 1 of flashcard knowledge construction: do not learn what you do not understand).

Technology I use for the Third Pass

Personally, I use Anki, which is probably the most popular spaced repetition software tool today. Some examples of other options include Mnemosyne, SuperMemo or even paper flashcards if you’re a technophobe.

Examples of the Triple-Pass System in Action

To get a feel for what the end result of this system looks like, here are a couple of examples:

This Seems like a lot of Work…

It’s true that using this system will almost certainly mean you’ll take more time to consume media. Compared to just passively reading a book, the highlighting, summarizing, and spaced repetition all add time.

But before you dismiss it, ask yourself a couple questions.

First: are you a genius that effortlessly absorbs the materials you consume? I don’t mean this sarcastically. People like this exist, like Tyler Cowen. If yes, there’s no real benefit to you from this system. It would just slow you down.

Second: why are you consuming the media in the first place? Is it something you really want to remember? If the answer is yes, then I believe using a system like this is a no-brainer.

Yes, it takes some extra time. But if you are consuming high-quality material relevant to your life, the benefits of that extra 10-30% of effort is well worth it.

To receive content like this weekly in your inbox, subscribe to my Spaced Repetition Newsletter, which provides latest news, tips, and ideas about spaced repetition, using learning tools like Anki, and improving your learning productivity.

Tips From Anki Flashcard Refactoring: Add Enough Knowledge to your Deck and Review your Sources

My flashcard refactoring for today is a reminder of the classic knowledge construction advice: do not add what you do not understand. It is also a reminder of the importance of providing enough related cards in your deck for a piece of knowledge.

Here’s the card I came across that was giving me trouble, related to SQL programming (double-sided):

  • Side 1: Oracle SQL syntax for creating object table
  • Side 2: CREATE TABLE (table name) OF (object type)

When revisiting this card, I realized that I didn’t have a good concept of what “object tables” are, so this is definitely a case of not understanding the material before committing it to spaced repetition.

But the thing is, I wouldn’t have added it if I didn’t have a good understanding of object tables, at the time of adding knowledge to my spaced repetition system. The problem is I forgot the concept of “object tables”, and seeing the answer to this card was not enough to bring it back. I didn’t have any other cards in my deck about “object tables” and how they differ from other related concepts in Oracle SQL such as nested tables.

In a situation like this, it helps to go back to the source, clarify any misunderstanding, and add new cards that solidify your knowledge.

So, in this case, I looked up Oracle documentation and found a great article almost immediately that clarified the meaning. It also provided a bunch of useful nomenclature for closely related concepts, providing further scaffolding for the knowledge. This lead me to add a bunch of cards:

  • Card 1 (Cloze): Objects can be stored in two types of tables: [object tables] and [relational tables].
  • Card 2 (Basic 1-sided Q&A):
    • Q: What’s the difference between object tables and relational tables? (Oracle SQL)
    • A: Object tables store only objects Relational tables store objects with other table data
  • Card 3 (Basic 1-sided Q&A):
    • Q: What does each row represent in an object table? (Oracle SQL)
    • A: An Object

So to recap, here the main lessons from this refactoring:

  1. Don’t add stuff to spaced repetition that you don’t understand
  2. Make sure you add enough knowledge about the concept in your deck, so there is sufficient context for you to understand again when you forget
  3. When dealing with 1 or 2, the solution is to go back to the original source to understand the knowledge and add more relevant material.

To receive content like this weekly in your inbox, subscribe to my Spaced Repetition Newsletter, which provides latest news, tips, and ideas about spaced repetition, using learning tools like Anki, and improving your learning productivity.

Tips from Flashcard Refactoring

Include your Sources, Have a Single Answer, and Break-Down Your Cards

Here’s a flashcard related to Oracle SQL that was giving me trouble (lapsed 8 times and was automatically marked as a leech):

  • Side 1: Collection (Oracle SQL)
  • Side 2: Data types in Oracle SQL that lets you internalize parent-child relationships between tables in the parent table.

This was a double-sided card, so both Side 1 and 2 serve as the question. Let’s see if we can improve this one.

First things first: do I need this card at all? Yes: SQL is highly relevant to my career in Data Science, and the organization I work for relies heavily on Oracle database. It’s important knowledge for me that I didn’t want to remove.

Next, figure out the issue with the card. Looking at the card statistics, it turns out I was always getting Side 2 wrong. After some consideration, I realized that this is actually a poor definition of a “Collection”. In fact, it’s not really the “definition” of a Collection, but a characteristic of a Collection. In other words, the flashcard doesn’t have a unique answer: it’s true that a Collection internalizes parent-child relationships, but it does a lot of other things too.

I consulted the original source of the material and there isn’t a clear definition of a Collection there. I did some Googling for other sources and apparently there isn’t really a great definition of an Oracle Collection. It turns out that Collection refers to a generic programming idea not specific to Oracle.

So, rather than trying to define Collection, I’ve opted to break the existing card down into multiple cards, following Rule Number 4 of Knowledge Construction: stick to the minimum information principle, which means if you can break a card into multiple simpler, easier-to-answer cards, do it.

Card 1 (one-sided):

  • Side 1: What Oracle SQL data type lets you internalize parent-child relationships in the parent table?
  • Side 2: Collection

Card 2 (one-sided):

  • Side 1: What kind of relationship does an Oracle SQL Collection help you represent?
  • Side 2: Parent-child (aka “one to many”)

Card 3 (one-sided):

  • Side 1: Does the Oracle SQL Collection data type internalize parent-child relationships in the parent table or child table?
  • Side 2: Parent table

I also tracked down a good definition of the generic “Collection” concept in Computer Science, and added it:

Card 4-5 (double-sided):

  • Side 1: Collection (Computer Science)
  • Side 2: Object that groups multiple items together as a single unit (Computer Science)

I feel confident these cards will be easier to remember, cost less time and frustration, and help me remember the concept much better.

Lessons learned:

  • Flashcards should have a single answer. Multiple correct answers for a card is a recipe for confusion and frustration. Interestingly, this isn’t included in Poitr Wozniak’s Twenty Rules for Formulating Knowledge, although you could interpret this as a form of interference (Rule #11)
  • Keep track of your source material when making cards. It makes it easy to look up more details when needed. 
  • Browse related sources through Google search if you’re unsure about what to do to an item. This will give you more context around the card to see whether the knowledge is even required at all. You may also come across a clarification or better formulation. In the example above, I discovered the generic concept of “Collection” in programming and realized that it was futile to try to include a definition specific to Oracle SQL.
  • Break cards down into a larger number of simpler cards. This is classic knowledge construction advice that is often not heeded, because it feels like more cards means more work. Counterintuitively, it is really a free lunch: you remember the concept better, you spend less time reviewing than you would have with the single complicated card, and reviews become much more enjoyable. 

To receive content like this weekly in your inbox, subscribe to my Spaced Repetition Newsletter, which provides latest news, tips, and ideas about spaced repetition, using learning tools like Anki, and improving your learning productivity.

Combating Knowledge Interference (Flashcard Refactoring)

I came across this computer networking Anki flashcard I’ve forgotten over 7 times:

(NW for Sysadmins: Ethernet) Address Resolution Protocol (ARP) – Maps Ethernet addresses to IPv4 addresses and back.

The card uses cloze deletions [] like this:

  • (NW for Sysadmins: Ethernet) [Address Resolution Protocol (ARP)] – [Maps Ethernet addresses to IPv4 addresses and back].
  • (NW for Sysadmins: Ethernet) Address Resolution Protocol (ARP) – Maps [Ethernet addresses] to [IPv4 addresses] and back.

Interestingly, I haven’t forgotten a single review for the right-hand side clozes. Turns out this was the one causing me trouble:

(NW for Sysadmins: Ethernet) […] – Maps Ethernet addresses to IPv4 addresses and back.

Why? The issue seems to be another card in my deck that is very similar, and I’m confusing the two. The other card quizzes a networking concept called “Neighbour Discovery (ND)”, which performs a similar function to ARP except it maps IPv6 addresses to Ethernet and back rather than IPv4 addresses. This is a good example of interference, which refers to the fact that learning similar things can make you confuse them (see Rule 11 of Poitr Wozniak’s classic article on the 20 rules of formulating knowledge).

So the solution I’m opting for is pretty simple, just add a hint:

  • (NW for Sysadmins: Ethernet) ARP (hint: not ND) – Maps Ethernet addresses to IPv4 addresses and back.

One other small improvement is adding another card for the acronym alone:

  • Front: (NW for Sysadmins: Ethernet) ARP (Unpack Acronym)
  • Back: Address Resolution Protocol

These interference issues are tricky because you can’t really anticipate them in advance. You have to discover them as you review your cards.

Another annoyance is I’m not 100% sure that interference was actually the problem. Ideally, I would have discovered this troublesome card during review, so I could know for sure why I’m failing.

So here are some lessons learned from this little exercise:

  • Use hints as an effective tool for reducing interference.
  • Keep an eye out for interference during review of your knowledge. As soon as you encounter it, note it. In the case of Anki, there is a “mark card” feature. I also recommend actually writing text within the card to remind yourself exactly how you failed the card when you fix it later. It would be nice to be able to see basic card statistics, like number of lapses, during review without having to go into card statistics. I inquired on reddit whether there was an addon for this and while there are some good options for desktop, it doesn’t seem like there’s anything that quite meets this need for mobile (where I do all of my reviews).
  • As you get better at knowledge building, interference will become your most common problem. As Poitr Wozniak says, interference is “probably the single greatest cause of forgetting in collections of an experienced user” of spaced repetition systems since it is hard (impossible?) to avoid even if you are really good. You typically discover it during knowledge review time, not knowledge construction time.

To receive content like this weekly in your inbox, subscribe to my Spaced Repetition Newsletter, which provides latest news, tips, and ideas about spaced repetition, using learning tools like Anki, and improving your learning productivity.

Flashcard Refactoring

I’ve started a weekly habit of flashcard review. I want to share with you my thought process for modifying my cards, because I think this will be valuable to help you improve your own knowledge construction skills.

I also want my flashcard development out in the open so you can call me out when I make mistakes and provide suggestions for further improvements. Please do reach out! I am by no means the ultimate expert in knowledge construction.

So, I will be doing a regular series I call “Flashcard Refactoring” (Refactoring comes from the programming term which basically means revising and improving your code).

To sniff out poor flashcards, I ran prop:lapses>7 in the Anki browser to get all the cards I’ve forgotten over 7 times. Here’s one I came across about a command in the Linux command line to a suspend a job:

  • Side 1: ^Z (Linux Command Line)
  • Side 2: Suspend a job running in the foreground (Linux Command Line)

The card is reversible, so are two cards in total: one with Side 1 as the question, and another with Side 2 as the question.

At first glance, it doesn’t look too bad. It’s fairly concise. But one quick and easy change is reduce words in Side 2, in accordance with the 12th rule of Formulating Knowledge (“Optimize Wording”):

  • Side 2: Suspend foreground job (Linux Command Line)

This is a nice little improvement, but why am I really forgetting this card? I think it’s because ^Z doesn’t really have any meaning – it seems arbitrary and it has no clear connection to suspending foreground jobs.

So, I’ll create a fake connection, i.e. a mnemonic.

The mnemonic that immediately came to mind was the fact that the beginning of “Suspend” kind of sounds like a “Z”, e.g. “Zuspend”. I think this is all that’s required for this to stick in my memory (but only time will tell).

When you come up with a mnemonic, it’s a good idea to create a separate card for it, so I added the following to my deck: 

  • Q: Mnemonic for remembering ^Z suspends foreground job in Linux Command Line.
  • A: Zuspend

To receive content like this weekly in your inbox, subscribe to my Spaced Repetition Newsletter, which provides latest news, tips, and ideas about spaced repetition, using learning tools like Anki, and improving your learning productivity.

Anki / Spaced Repetition Tip: Review your Weak Flashcards

To receive content like this weekly in your inbox, subscribe to my Spaced Repetition Newsletter, which provides latest news, tips, and ideas about spaced repetition, using learning tools like Anki, and improving your learning productivity.

I’ve been a long-time user of spaced repetition tools. I’ll never forget first hearing about SuperMemo from a close friend as I started my undergraduate degree in 2005. I was immediately sold on the value of spaced repetition, and I particularly liked the idea of computers automatically taking care of review scheduling for you. I started using SuperMemo as a central tool for studying, and saw my academic performance skyrocket.

Over the years, I’ve slowly improved my skill in designing flashcards. It is by no means a trivial skill: it took me years to get pretty good at it, and to this day I still often make flashcards that are complete failures.

I believe there will eventually be an open collaborative platform for flashcard development and sharing, where experts can contribute and refine perfectly crafted cards. Users contribute their deck statistics, revealing poorly formed cards and contributing to our understanding of optimal flashcards.

But until that day, it pays to develop your flashcard creation skills.

Flashcard quality is top of mind for me since I’ve revisited the classic article by Peter Wozniak (of SuperMemo fame), “Effective Learning: Twenty Rules of Formulating Knowledge)”. It is a must-read for anyone that creates flashcards for learning (i.e. almost everyone at some point in their life). I’ve published my summary notes on this article (aside: my notetaking tool of choice is Roam my notes are easy to copy-paste into your own Roam database if you happen to use it as well).

One great way to improve your flashcard development skills, while simultaneously improving the quality of your deck, is to review your old cards regularly. Review your top 10-20 most problematic cards weekly, and for each one you encounter, do one of the following things:

  • Revise: With the Twenty Rules of Formulating Knowledge by your side, refine your card or break it down into a larger number of small, easy to digest cards.
  • Suspend: If you don’t think you need to have a card in spaced repetition anymore, but don’t want to delete it entirely, suspending is a good option.
  • Delete: If you know the knowledge is completely useless to you, trash the card entirely.

But what cards should you review? If you’re like me, you have a pretty big collection, and it’s just not feasible to review all your cards every week to find the weak ones.

Anki makes it quite easy to find these problematic cards. Two main search commands in the Anki Browser are useful here:

  • tag:leech – this finds all of the “leeches” in your Anki deck, which are cards that you keep forgetting. By default, Anki tags your card as “leech” when you fail a card 8 times.
  • prop:lapses>n – this reveals all of the cards you have failed (“lapsed“) over n times. You can set n to whatever number you like. Start with high-n cards and work your way down.

In addition to using these search techniques, I try to make a habit of “marking” cards that are problematic or poorly formed in some way, during review. If it’s an easy correction (e.g. obvious suspension, or small text changes), I’ll make the change right away in the mobile app. Otherwise, I will simply mark the card and filter it out during weekly review to make improvements.

When you do revise your cards, I recommend “resetting” the card so it’s like a “do-over” – the card should be reviewed again as if you just created it. This serves two purposes: it ensures that the card will no longer show up in your “problem cards” lists when you do the above queries. It also provides you with more opportunities to review your new formulation of the knowledge.

Unfortunately, it seems the only way to do this in Anki is do create new card(s) with the information you want and delete the old one. There is an option for “rescheduling” the card, but this only restarts the review process and doesn’t delete your review history. As a result, the card will still appear as one of your problem cards if you do a query like prop:lapses>n. Luckily, it’s not much extra effort to do this.

I have to admit that I do not entirely practice what I preach here. Weekly review of my cards is something I haven’t fully incorporated yet, but I’m resolving to start doing it today. In the next weeks, I’m going to experiment with a Flashcard Refactoring series to illustrate the card refinement process. Stay tuned!

To receive content like this weekly in your inbox, subscribe to my Spaced Repetition Newsletter, which provides latest news, tips, and ideas about spaced repetition, using learning tools like Anki, and improving your learning productivity.

How to Deal with Lists in Anki

Photo by Glenn Carstens-Peters on Unsplash

When adding cards to Anki (or any other spaced repetition system), you’ll find you often need to deal with lists. In the spaced repetition community, these are sometimes called “Sets” (see Anki Essentials, Chapter 7, “The 20 Rules”).

For example, here’s a section from a book I’m working through called The Art of Doing Science and Engineering by Richard Hamming, where he summarizes the main advantages of computers over humans:

“People are sure the machine can never compete, ignoring all the advantages the machines have…These are: economics, speed, accuracy, reliability, rapidity of control, freedom from boredom, bandwidth in and out, ease of retraining, hostile environments, and personnel problems.”

This is a frustrating passage, because it’s packed with valuable information but tough to turn it into flashcards.

One easy approach is to cloze delete each individual option, and then cloze delete the entire list to make sure you can recall the whole thing. While this makes the cards easy to create, you’ll run into these issues when reviewing them:

  • It’s a lot to read, and you have to read every option to know what the missing one is.
  • It’s hard to remember. Even if you do individual cloze deletions for each point, you will still be very likely to fail on the question where you are asked to recall the entire list. Also, many lists (including the example above), are open ended and not necessarily exhaustive. So, when it comes time to review, you’re not quite sure what’s on the list, and you may list things that are “correct” but just not included on that particular list.
  • It violates a fundamental rule of flashcard creation. The “Minimum Information Principle” states that your questions and answers should be as simple as possible. A large number of simple cards is far more efficient to remember than a small number of big, demanding, awkward cards.
  • It’s hard to rate yourself. How do you rate your performance when you get 9 out of the 10 answers in the list correct? Do you fail yourself if you only got one wrong?

So, what are our options for dealing with these annoying lists?

Break it Down

First, ask yourself: do I really need to memorize the list? Do you really need to be able to spout off the list of advantages of computers over humans, and not miss any of points? Probably not.

Then ask yourself, what do I really want to get out of this list? In my example, I want advantages of computers over humans to pop up in my mind when the time is right. For example, when I’m working on a process at work that is repetitive, rote, and requires accuracy, I want to recognize computer scripts as a good solution.

So, I would not Ankify this specific list at all. Instead, I would look in the source to Ankify material focused on each individual point. For example, I could add specific questions on the relative costs of computing compared to manual effort, or details about the speed of computing compared to hand calculations. Doing this will give a deeper understanding of the advantages of computers over humans. It will probably also increase the likelihood of being able to recall the entire list at once (even though it’s unlikely I’ll need to do that).

You can also lists into sub-categories and then add those smaller lists to spaced repetition. Aim for each sub-category to have 1-2 items.

Take Advantage of Special List Structure

Sometimes you can take advantage of the inherent structure of a list to break the flashcard down. For example, your list can take a special form of a “1-n relationship”, as discussed in this thread .

A 1-n relationship is best understood by example. Suppose you want to memorize the Provinces in Canada. This list has the special property where each province in the list is associated with one and only one country (i.e. 1 country – n provinces). With lists like this, you can add questions like “what country does the province of Manitoba belong to?”, “What country does the province of Alberta belong to?”, etc. Of course, this fails if there is a province in another country with the same name, (that would mean it’s not a 1-n relationship), but you get the point.

Turn the List into an Enumeration

One of the problems with the example list of computer advantages is the order of the items has no meaning. There is no reason why “personnel problems” should be last rather than first. This makes it far more difficult to memorize because it’s just a bunch of seemingly random points.

Enumerations are ordered lists. Although they can still be difficult to memorize, they at least have some structure that your mind can latch on to. You should always try to convert your list into an enumeration if it’s not one already.

Ideally, your ordering will contain useful knowledge itself. For example, you could list the advantages of computers over humans, from most important to least important according to the author.

There are many other possibilities for turning lists into enumerations, depending on what you are trying to memorize, such as chronological order, area of the body, size, popularity, population, etc. If you can’t find a meaningful ordering, you can always list items alphabetically.

Enumerations are a free lunch, because they make list easier to memorize and provide bonus knowledge about the material. For example, wouldn’t it be much better to know the order of the planets in the solar system from closest to the sun to furthest, rather than just knowing the individual planets?

Use Cloze Overlapper

After you have turned something into an enumeration. How should you design your repetitions?

One option is to do one cloze deletion per item in the list. For example, take the list (a, b, c, d, e, f). Your cloze deletions are then ([…], b, c, d, e, f), (a, […], c, d, e, f), (a, b, […], d, e, f), etc.

I have done this many times in the past, and while I can usually answer the questions quickly and accurately, I find it hard to retain the information. When asked to reproduce the whole list, I fail. This is because the questions are too easy – removing only one item provides way too much context.

Until recently, I thought this was the best option. Then I came across the “overlapping cloze”: close deletions on individual items in the list, but with only a small amount of context given. For example, something like this: ([…], …, …, …, …), (a, […], …, …, …), (…, b, […], …, …), etc.

It’s actually quite painful to create overlapping clozes in vanilla Anki. Luckily, there is an add-on for this: Cloze Overlapper. This video by Glutanimate, the author of the add-on, provides an overview of issues with lists in and illustrates how how to use the add-on (aside: the author is the developer of another great add-on, image occlusion enhanced). I haven’t tried out Cloze Overlapper yet, so I can’t fully vouch for the technique or the add-on, but it seems to be highly regarded in the spaced repetition community.

Use Mnemonic Techniques

With a bit of practice, mnemonic techniques can help you achieve incredible feats of memory. Many are easy to learn. Even doing a little bit of studying on these techniques can produce big payoffs for your memory.

The technique I probably use most is the link method. For example, one flashcard I have on computer networking asks me to recall the main types of datalink layer errors. I added it as an unordered list: frame errors, drops, collisions, and overruns.

It’s actually a pretty poor card, but I haven’t had much trouble memorizing it because of the mnemonic I used:

  • I think of Link (from Zelda) with a shirt that says ERROR on it (dataLINK layer ERRORS).
  • Then I imagine him putting on some glasses (frame errors)
  • Link throws the glasses down to the ground (drops)
  • A giant truck hits Link (collisions)
  • Finally, the truck runs over the glasses that are on the ground (overruns)

Sounds silly, but it works. Our minds are extremely good at retaining images.

I am by no means an expert in these techniques, but I’ve found that learning a little bit helps a lot. If you go deeper, I’m sure you’ll see even bigger benefits. Other useful memory systems include peg and Method of Loci.

If you use a mnemonic technique for a list in your spaced repetition system, be sure to add flashcards to remember mnemonic itself.

Conclusion

Avoid lists if you can, but if you must, try to break them down into sub-categories, make sure they’re ordered (preferably with some relevant meaning to the ordering), use cloze overlapping to implement the reviews and limit context, and use mnemonic techniques. Together, this will give you a solid arsenal for committing lists to memory without much pain.

Resources

To receive content like this weekly in your inbox, subscribe to my Spaced Repetition Newsletter, which provides latest news, tips, and ideas about spaced repetition, using learning tools like Anki, and improving your learning productivity.

Why Learning By Teaching Works and How to Use It

Insights from Learning by Teaching Human Pupils and Teachable Agents: The Importance of Recursive Feedback by Sandra Okita and Daniel Schwartz.

Want to commit key points from this blog post to long-term memory? You can access an Anki Deck here.

Special thank-you to Andy Matuschak for pointing me to this paper.

Brief Overview of the Okita and Schwartz Paper

You have probably heard the advice that you should teach something if you really want to understand it. But does it really work, and if so, how?

Okita and Schwartz provide an extremely readable primer on the research behind Learning by Teaching and why it is so effective as a learning strategy.

They also explore a little-known benefit of learning by teaching called “recursive feedback” – feedback from watching your pupils use what you taught them. Using two separate experiments, Okita and Schwartz find that this special kind of feedback significantly improves teacher learning.

In this post, I summarize key insights from the paper on why learning by teaching works, and conclude with some thoughts on specific strategies for learning by teaching to improve your learning.

Why Learning by Teaching Works

Okita and Schwartz divide the learning by teaching process into three distinct phases: preparation (the teacher’s preparations to teach), teaching (the actual act of teaching), and observation (when the teacher receives recursive feedback by seeing the student apply what they learned from the teacher). It turns out that each of these phases contribute to the teacher’s learning.

Preparing

Preparing to teach students helps teachers learn due to three main forces:

  • Motivation: Preparation introduces strong emotions and motivation to truly understand the material and do it justice. This includes a sense of responsibility, the anticipation of a public performance, and a desire to avoid embarrassment or “looking stupid”.
  • Generation: Preparing to teach involves a “generation effect“. The generation effect means that you remember information better when you generate it from your mind rather than passively reading it. Preparing to teach is generative because you have to frequently retrieve and elaborate on information from your memory as you think about how you are going to teach your students.
  • Meta-cognitive vigilance: Meta-cognition refers to an awareness of your own thoughts. It turns out that preparing to teach generates this kind of awareness through the teacher’s self-explaining and self-generated questions as they anticipate the needs and questions of their pupils. This, in turn, helps teachers identify areas where they have conflicting knowledge or incomplete understanding.

Teaching

Three main aspects of this phase helps teachers learn:

  • Explanation. The act of teaching involves explaining ideas to others. This leads to “explanation effects” that occur from explaining ideas rather than just receiving them.
  • Question answering. Students often have questions of their teachers, which can reveal gaps in the teachers understanding or encourage the teacher to extend their understanding. In fact, Okita and Schwartz cite a study by Roscoe and Chi (2007) that found “tutees’ questions were responsible for about two-thirds of tutors’ reflective knowledge-building activity”.
  • Physiological arousal. Anyone that has experienced public speaking knows that it is a very stimulating activity that focuses your mind. As you might expect, this kind of arousal and attention improves learning.

Observing

In this phase, the teacher receives recursive feedback by watching the student apply what they learned. You might think it’s obvious that feedback improves learning, but it turns out that’s not always the case.

For example, direct feedback, where a student tells a teacher directly how well they did, can be counterproductive because people often take this kind of feedback personally, leading to “self-handicapping, learned hopelessness, or social comparison”. When this happens, the recipient of the feedback tends to adopt attitudes like, “well, I guess I’m not the kind of person who’s good at math“. As a result, they give up or opt for an easier subject. As much as we all like to believe we would receive feedback objectively and use it to our advantage, our egos often get involved, whether we like it or not.

Thankfully, recursive feedback side-steps this issue. Since the teacher is not directly receiving feedback from the student, their egos are uninvolved. Instead, they see for themselves what worked and what didn’t. No one explicitly tells them, “this is where you need to improve, this is what you did wrong”. Okita and Schwartz note that this “creates an ego-protective buffer, because the pupil rather than the tutor takes the immediate responsibility for being wrong”.

I also believe recursive feedback helps build empathy skills. A lack of empathy, in my opinion, is often a major barrier to learning and self-improvement. Think of how many people you have encountered in schools or workplaces that have tried to explain something to you, but assume an unreasonable amount of knowledge, leave out critical bits of information, or explain things in ways that are otherwise unhelpful (e.g. using tons of jargon you’re unfamiliar with)? Often these people are very intelligent, and would be far more effective if they gained some empathy.

How can we use these lessons about Learning by Teaching to improve our own learning?

The main lesson I took away from the Okita and Schwartz paper is that Learning by Teaching is not just folk wisdom. It works. So, as learners, we should try to use it wherever we can. Here are some ideas:

  • Develop a course or become a tutor. If there is a topic you really want to master, and you have enough time and resources, you can run an online or in-person course. In addition to the learning benefits, you can make some side income.
  • Observe your students. To take advantage of recursive feedback, you need to be able to see your students apply their knowledge. Use tests, presentations, or projects to see what your students learned.
  • Keep it interactive. Interaction with your students means more questions from them and, as a result, more learning for you. Okita and Schwartz note “Tutoring sessions should be arranged to limit the natural tendency of tutors to slip into didactics, which can eliminate opportunities for pupils to ask questions“. Students will often ask useful questions that help you better understand the material.
  • Write online. Writing online is a form of teaching, whether it’s blog posts, twitter, email newsletters, or something else. It also provides ample opportunity to receive valuable feedback about your work.
  • Develop a smart and interesting following. The quality of pupil questions impacts teacher learning. This means that, if you have an email list or a blog, or some other kind of online following, you learning would benefit from attracting high-quality people. Check out David Perell’s work on writing online, where he emphasizes the importance of writing for intelligent readers and leveraging their feedback to improve yourself.
  • Encourage your students to keep in touch about the projects they work on and how they applied the knowledge you taught them. Your students will appreciate this, and you’ll also get valuable recursive feedback.
  • Build things and watch people use what you’ve built. Okita and Schwartz point out that recursive feedback applies to areas other than Learning by Teaching. For example, producing products, tools, crafts, art, and then getting recursive feedback by observing how your “customer” uses it. One application of this idea that comes to mind is Agile software methodology, which relies on building software quickly, getting feedback on it as soon as possible, and constantly iterating and improving based on that feedback. If you produce something that others consume, you’ll gain by watching them use it.

To receive content like this weekly in your inbox, subscribe to my Spaced Repetition Newsletter, which provides latest news, tips, and ideas about spaced repetition, using learning tools like Anki, and improving your learning productivity.

“The Artists of Data Science” Podcast Interview

My friend Harpreet Sahota just launched a new podcast called “The Artists of Data Science”. You can hear my interview here. We cover a variety of topics, including my path to becoming a data science, spaced repetition, compounding, finding good data science projects, the strangest question I’ve been asked in a job interview, and more!

Check out his other interviews there as well, including Vin Vashishta, Andreas Kretz, Angela Baltes, Daniel Bourke, David Tello, Pooja Sund, Alexander Lim, Lediona Nishani, Mikiko Bazeley, and Amit Jain.

Notes on Developing Transformative Tools for Thought

Occasionally, special tools come along that amplify our minds, enabling new kinds of thought not otherwise possible. Computers, writing, speaking, and the printing press are all examples of these “Tools for Thought” that surge human potential. 

This essay from Andy Matuschak and Michael Nielsen explores whether we can accelerate the development of these kind of tools. They also provide a taste of potential tools for thought with their prototype “mnemonic medium”, an interactive post on quantum mechanics called “Quantum Country” with embedded flashcards combined with a spaced repetition system delivered through email follow-ups. 

Their essay is a must-read for anyone interested in spaced-repetition or productive learning. There are several points I found thought-provoking. I believe each of these insights indicate a need for a new Tool for Thought for flashcard development and sharing.

Spaced repetition creates exponential returns to studying

Based on Quantum Country user data, Matuschak and Nielsen estimate that devoting only 50% more time to spaced repetition after reading the essay resulted in users recalling the key points for months or years. 

In other words, relatively small investments in spaced repetition after reading an article produces outsized results – more evidence to place on top of the mountain of research suggesting spaced repetition works. 

Good flashcard development is difficult

Matuschak and Nielsen note that it takes a surprising amount of skill and time to build quality flashcards, especially for abstract concepts. This is probably a big reason why most people fail to adopt spaced-repetition tools like Anki. Since flashcard development is a skill that you develop over much time and effort, new users tend to add cards in a way that inevitably leads to frustration and failure. 

This may partially explain the efficacy of Quantum Country: the authors are experts in both quantum mechanics and flashcard development – a rare but essential combination of skills for their essay to work. 

Flashcards written by others can be useable

Some people the spaced repetition community don’t believe in using flashcards created by others, and with good reason. They’re often poorly written. They’re idiosyncratic. They’re missing crucial contextual information that you lack as someone who hasn’t read the original source material. I used to be one of these non-believers.

But the effectiveness of the Quantum Country essay suggests that shared flashcards can work well. This has benefits of saving users of the burden of flash card creation, as well as preventing new user frustration from poor flashcard building skills and poor domain knowledge.

Matuschak and Nielsen hypothesize that the quality of their flashcards is what makes this work. I agree, but I have a few more hypotheses: 

  • Their flashcards are introduced in a logical progression as users read the essay. In contrast, shared decks in Anki shuffle cards randomly and are not encoded with dependency information.
  • Their flashcards are clearly connected to a source (i.e. the essay), providing important context for the user.
  • Users learn the material before they review flashcards. This is in line with the common wisdom that flashcards don’t work if you don’t already understand the material – they are a tool for retention, not learning. Aside: is this common wisdom true? I’m not so sure. Socrates taught using Q and A, so why can’t you teach a subject entirely with flashcards? If it is possible, what are the prerequisites to making it work? 

Elaborative encoding

Matuschak and Nielsen note elaborative encoding as another learning tool shown to be extremely powerful in promoting memory. Essentially, it means connecting new ideas you want to remember with old ideas you know well, providing a fast path in your brain to new information. 

Remember this concept while developing your flashcards. Whenever you add a new card, think about what you already know well and how you can connect this to the new knowledge.

A New Tool for Thought?

Matuschak and Nielsen’s article has renewed my interest in a tool for thought idea I’ve been pondering for quite a while: a platform for collaborative flashcards development and sharing. I believe such a tool, if properly developed, can address the issues that limit the use of spaced repetition:

  • Spaced repetition practitioners currently need to develop their own flashcards, which requires a significant amount of time, domain expertise, and flashcard-building skill. There needs to be a place where experts can create shared flashcards, and there should be a proper incentive structure encouraging creators to improve these flashcards over time.
  • Flashcards are not clearly connected to original sources. Spaced repetition practitioners should be able to pull up pre-built flashcards for a source document they are working through. 
  • Current tools do not provide information that link flashcards together (other than knowing two flashcards are part of the same deck, or have the same tag). At the very least, flashcards should have a notion of “depends on” or “prerequisite to”. This would make shared decks more useful by showing the intended progression of knowledge. It would also aid elaborative encoding (e.g. examining cards you’ve reviewed and linking them up to cards “nearby” in a knowledge graph)

I strongly believe a tool like this needs to exist, as you may have guessed if you noticed the Download my Brain feature I built for this site that provides a platform for sharing my personal Anki decks. I have started work on a more generic tool for collaborative flashcard construction and sharing and will keep you posted once I have something ready for production. 

Thanks to Andy Matuschak and Michael Nielsen for the inspiration to follow this path.

If you’re interested in space repetition and techniques for productive learning, subscribe to my Spaced Repetition Newsletter. Check out my shared data science Anki decks here: Download my Brain.