How to Deal with Lists in Anki

Photo by Glenn Carstens-Peters on Unsplash

When adding cards to Anki (or any other spaced repetition system), you’ll find you often need to deal with lists. In the spaced repetition community, these are sometimes called “Sets” (see Anki Essentials, Chapter 7, “The 20 Rules”).

For example, here’s a section from a book I’m working through called The Art of Doing Science and Engineering by Richard Hamming, where he summarizes the main advantages of computers over humans:

“People are sure the machine can never compete, ignoring all the advantages the machines have…These are: economics, speed, accuracy, reliability, rapidity of control, freedom from boredom, bandwidth in and out, ease of retraining, hostile environments, and personnel problems.”

This is a frustrating passage, because it’s packed with valuable information but tough to turn it into flashcards.

One easy approach is to cloze delete each individual option, and then cloze delete the entire list to make sure you can recall the whole thing. While this makes the cards easy to create, you’ll run into these issues when reviewing them:

  • It’s a lot to read, and you have to read every option to know what the missing one is.
  • It’s hard to remember. Even if you do individual cloze deletions for each point, you will still be very likely to fail on the question where you are asked to recall the entire list. Also, many lists (including the example above), are open ended and not necessarily exhaustive. So, when it comes time to review, you’re not quite sure what’s on the list, and you may list things that are “correct” but just not included on that particular list.
  • It violates a fundamental rule of flashcard creation. The “Minimum Information Principle” states that your questions and answers should be as simple as possible. A large number of simple cards is far more efficient to remember than a small number of big, demanding, awkward cards.
  • It’s hard to rate yourself. How do you rate your performance when you get 9 out of the 10 answers in the list correct? Do you fail yourself if you only got one wrong?

So, what are our options for dealing with these annoying lists?

Break it Down

First, ask yourself: do I really need to memorize the list? Do you really need to be able to spout off the list of advantages of computers over humans, and not miss any of points? Probably not.

Then ask yourself, what do I really want to get out of this list? In my example, I want advantages of computers over humans to pop up in my mind when the time is right. For example, when I’m working on a process at work that is repetitive, rote, and requires accuracy, I want to recognize computer scripts as a good solution.

So, I would not Ankify this specific list at all. Instead, I would look in the source to Ankify material focused on each individual point. For example, I could add specific questions on the relative costs of computing compared to manual effort, or details about the speed of computing compared to hand calculations. Doing this will give a deeper understanding of the advantages of computers over humans. It will probably also increase the likelihood of being able to recall the entire list at once (even though it’s unlikely I’ll need to do that).

You can also lists into sub-categories and then add those smaller lists to spaced repetition. Aim for each sub-category to have 1-2 items.

Take Advantage of Special List Structure

Sometimes you can take advantage of the inherent structure of a list to break the flashcard down. For example, your list can take a special form of a “1-n relationship”, as discussed in this thread .

A 1-n relationship is best understood by example. Suppose you want to memorize the Provinces in Canada. This list has the special property where each province in the list is associated with one and only one country (i.e. 1 country – n provinces). With lists like this, you can add questions like “what country does the province of Manitoba belong to?”, “What country does the province of Alberta belong to?”, etc. Of course, this fails if there is a province in another country with the same name, (that would mean it’s not a 1-n relationship), but you get the point.

Turn the List into an Enumeration

One of the problems with the example list of computer advantages is the order of the items has no meaning. There is no reason why “personnel problems” should be last rather than first. This makes it far more difficult to memorize because it’s just a bunch of seemingly random points.

Enumerations are ordered lists. Although they can still be difficult to memorize, they at least have some structure that your mind can latch on to. You should always try to convert your list into an enumeration if it’s not one already.

Ideally, your ordering will contain useful knowledge itself. For example, you could list the advantages of computers over humans, from most important to least important according to the author.

There are many other possibilities for turning lists into enumerations, depending on what you are trying to memorize, such as chronological order, area of the body, size, popularity, population, etc. If you can’t find a meaningful ordering, you can always list items alphabetically.

Enumerations are a free lunch, because they make list easier to memorize and provide bonus knowledge about the material. For example, wouldn’t it be much better to know the order of the planets in the solar system from closest to the sun to furthest, rather than just knowing the individual planets?

Use Cloze Overlapper

After you have turned something into an enumeration. How should you design your repetitions?

One option is to do one cloze deletion per item in the list. For example, take the list (a, b, c, d, e, f). Your cloze deletions are then ([…], b, c, d, e, f), (a, […], c, d, e, f), (a, b, […], d, e, f), etc.

I have done this many times in the past, and while I can usually answer the questions quickly and accurately, I find it hard to retain the information. When asked to reproduce the whole list, I fail. This is because the questions are too easy – removing only one item provides way too much context.

Until recently, I thought this was the best option. Then I came across the “overlapping cloze”: close deletions on individual items in the list, but with only a small amount of context given. For example, something like this: ([…], …, …, …, …), (a, […], …, …, …), (…, b, […], …, …), etc.

It’s actually quite painful to create overlapping clozes in vanilla Anki. Luckily, there is an add-on for this: Cloze Overlapper. This video by Glutanimate, the author of the add-on, provides an overview of issues with lists in and illustrates how how to use the add-on (aside: the author is the developer of another great add-on, image occlusion enhanced). I haven’t tried out Cloze Overlapper yet, so I can’t fully vouch for the technique or the add-on, but it seems to be highly regarded in the spaced repetition community.

Use Mnemonic Techniques

With a bit of practice, mnemonic techniques can help you achieve incredible feats of memory. Many are easy to learn. Even doing a little bit of studying on these techniques can produce big payoffs for your memory.

The technique I probably use most is the link method. For example, one flashcard I have on computer networking asks me to recall the main types of datalink layer errors. I added it as an unordered list: frame errors, drops, collisions, and overruns.

It’s actually a pretty poor card, but I haven’t had much trouble memorizing it because of the mnemonic I used:

  • I think of Link (from Zelda) with a shirt that says ERROR on it (dataLINK layer ERRORS).
  • Then I imagine him putting on some glasses (frame errors)
  • Link throws the glasses down to the ground (drops)
  • A giant truck hits Link (collisions)
  • Finally, the truck runs over the glasses that are on the ground (overruns)

Sounds silly, but it works. Our minds are extremely good at retaining images.

I am by no means an expert in these techniques, but I’ve found that learning a little bit helps a lot. If you go deeper, I’m sure you’ll see even bigger benefits. Other useful memory systems include peg and Method of Loci.

If you use a mnemonic technique for a list in your spaced repetition system, be sure to add flashcards to remember mnemonic itself.

Conclusion

Avoid lists if you can, but if you must, try to break them down into sub-categories, make sure they’re ordered (preferably with some relevant meaning to the ordering), use cloze overlapping to implement the reviews and limit context, and use mnemonic techniques. Together, this will give you a solid arsenal for committing lists to memory without much pain.

Resources

Notes on The Pseudonymous Economy – Balaji Srinivasan

  • Overview: [[Balaji Srinivasan]] looks at [[pseudonymity]], why have a pseudonymous economy, how might it work, and how could we build it.
  • [[pseudonymity]] is not [[anonymity]]. pseudonymity is not giving your real name, but you maintain it over time and develop a reputation on it. It provides [[accountability]], but it’s also a shield against character assassination. #[[To Ankify]]
  • [[pseudonymity]] is a continuum, not binary. They are widely used today, including by people high up in government.
  • 4:00 The big next step is not just pseudononymous communication, but pseudonomyous transaction.
    • Drivers of increased use of pseudonyms: technological support, employment law (don’t want to be discriminated against), social media mobs.
  • 5:30 Pseudonymity provides not just freedom of speech, but freedom after speech. Opponents have to attack your idea, they can’t attack you.
    • Negative press is an attack on your social network.
    • Your bank account is your stored wealth, your real name is your stored reputation. Only you can debit your bank account, but anyone can debit your reputation.
  • 9:20 A possibility is separating your earning, speaking, and real names. Use your real name on official forums like social security numbers. When speaking use a pseudonym. To earn (this is the new part) – earn under a pseudonym, with a professional identity and you speak about things relevant to that identity.
    • You might have many different speaking names, many different earning names.
  • 10:20 **We can already move wealth to a pseudonym, can we move reputation too? **Right now, you can only have 100% of followers and write under your existing name, or 0% of followers starting completely fresh with a new pseudonym. #Ideas #[[startup ideas]]
    • What if we transfer attestations rather than everything (i.e. the entire list of followers)? For example, build a new pseudonym platform that confirms that you’re verified (blue check) but nothing else. You can have multiple attestations (e.g. followed by @jack).
    • 17:00 But how do you ensure people will follow? [[Balaji Srinivasan]] calls this “[[autofollow]]”. People would be on a network like this to follow pseudonymous accounts – they will autofollow anyone with over 100k followers, anyone followed by @jack, etc. You then automatically get followers.
    • An application like this would essentially allow you to move along a reputation-anonymity continuum, trading off a small amount of anonymity for higher reputation.

Notes on Why You Should Become Internet Famous, by Patrick McKenzie and David Perell

  • Overview: [[David Perrell]] and [[Patrick MacKenzie]] discuss various topics related to writing and promoting yourself online.
  • 4:45 You should think about how your writing is framed online if you are looking for it to be used to promote your commercial work. “Blogging” has a poor brand within many high status employers, whereas “memos” and “essays” do not. Try to categorize your writing as these things rather than a blog post. These small changes can make a big difference. Don’t call it a “blog”, don’t put a date on it if it’s not stuff that will decay in importance in 48 hours, choose to write about things that will stay relevant, etc. #blogging #promotion #Portfolio
  • 9:00 Try to create a tighter brand for your writing. Find a niche and try to focus on that, although experimenting isn’t that bad an idea, at least at first. Good to also think about your interests that happen to be booming in the economy (for example, [[Patrick MacKenzie]] focused on the intersection of [[marketing]] and [[engineering]]). #branding #niche
  • 13:00 Even if absolutely no-one reads your essay, it’s still worth writing it because it produces an asset you can use. For example, you could use it as a proof of work to a future employer and that would make it a tremendous ROI. Don’t feel like you have to have a big audience before [[writing]].
  • 15:00 find where people you are writing for hang out. Hang out there, and gradually inject your stuff in there. Gradually, and in an authentic way, introduce your thoughts into people’s with large following’s posts. Some people don’t get annoyed by this – some people with large followings actively signal boost.
  • 23:40 Benefits of long-form [[writing]]. A lot of the common advice about writing short-form on the internet comes from people with certain incentives for following that model (e.g. [[BuzzFeed]]). You really need to understand your personal [[goals]] and what the incentive structure are for achieving that goal.
  • 28:30 As soon as you put something on a dashboard, people start to change their behaviour based on what’s on that dashboard. So – be careful of what you put on [[dashboards]]. #KPIs #metrics
  • 30:00 Discusses the so-called “death of blogs”. [[Patrick MacKenzie]] points out that people that wrote blogs back in 2010 are still around, but now they have moved on to incredible titles, like CEO or Senior Staff Engineer at Google. As a result, they no longer blog or call themselves bloggers. #blogging
  • 37:20 [[Patrick MacKenzie]] is a fan of [[Ramit Sethi]], he is one of his favourite marketers and knows how to do the internet well.
  • 38:00 Discussion of recommendations for email lists. Email is something you can own – it’s difficult to take that away from you. There’s also something powerful that someone put up their hand and said “yes, I want to hear, in my inbox, what you are talking about”. It’s also less “risky”, since no-one forwards a newsletter for someone else to dunk on it (unlike what happens on Twitter). #[[Email Lists]] #Independence #control
  • 43:00 What makes an idea worth publishing?
  • 47:00 What do the best marketers miss?
  • 49:00 Culture of [[Stripe]] and how they maintain a high level of craft. Are [[craft]] and [[metrics]] somehow opposed or a different language? Incentives matter – if you incentivize shipping the best version of something, that’s what you get. If you incentivize always meeting deadlines, that’s what you get. #incentives
  • 54:35 What does making writing at [[Stripe]] do for the company? 99% of their word count is internal. Some companies in hyper [[Growth]] mode are doubling their number of employees each year. As long as that continues, over half of your staff have less than 1 year experience with the company. Big problem is spinning people up to speed and getting them into the culture – “[[democracy of the dead]]” – people who were there before can have tremendous impact by producing highly leveraged artifacts, like written documents. Writing helps transfer knowledge #[[Corporate Knowledge]] #[[Organizational Memory]] #leverage
  • 1:00:00 When should you go to market after building an audience? [[Writing]] and developing an audience first before developing software is valuable because you build a list of people ready to buy. You also learn more about how to build a great product by writing deeply about it. The MVP for sellable word products is much lower bar than [[MVP]] for software. #[[Audience Building]]
  • 1:07:30 What is a small software business that you admire? How do they use online writing and content to grow and validate the business? There are tens of thousands of profitable, successful software / SAS business that you’ve never heard of. It will be boring, very successful, and will be sold, and hardly anyone that isn’t a customer or an employee will be aware of it. E.g. Moraware which provides software for kitchen counter installation companies.
  • 1:14:00 What is the 1 thing you think the [[US]] should import from Japanese [[culture]]? [[Earnestness]] and [[optimism]] of ones [[work]]. A non-ironic embrace of loving what you do. There’s a lot of [[cynicism]] in the US. “Choose to do what you love” is bad advice. Better advice is “Learn to love what you do”. #Japan #happiness
  • 1:18:30 How did you go from being a kid with “I want to be a baseball player” type interests to who you are now – an expert in niche SAS? People underestimate the ability they have to change, particularly the way they think. [[Stripe]] hires for people that are ambitious and optimist. Hanging around people like that makes you more ambitious and optimistic! #Ambition #optimism
  • 1:23:00 The amount of luck you have in life is how much value you create times how many people you tell about it. Explain what you mean by that. It’s a mistake to think that if you just do great work, you’ll be recognized for that. It’s a key professional skill and you should probably get good at it. #luck #marketing

Notes on Anthony Pompliano & David Perell: Growing Your Audience

  • Overview: [[Anthony Pompliano]] and [[David Perrell]] discuss [[Anthony Pompliano]]’s strategy for how he built his following online.
  • 1:34 presentation starts, [[Anthony Pompliano]] introduces himself.
  • 2:18 shows his results from building his online presence – 3 people built it, 60 million impressions per month.
  • 3:00 Main components of his stack: [[The Pomp Podcast]], [[The Pomp Letter]] (letter every weekday), [[Lunch Money]] (YouTube daily show). All of this is driven by [[Twitter]]. “Everything starts and ends with Twitter” – it’s a traffic engine for every other product and platform.
  • 6:42 “Audience is the new currency”. “Those who build the audience have the power.”
  • 8:30 It’s hard to build an audience. You want to ask yourself what the value you want to create, what does the end result look like, and do you have the time / energy / dedication to attempt this? Comes down to man hours and time.
  • 6 Principles for growing your audience
    • Persistence is the most important thing (9:40). It takes time, and audience building starts slow. True for everyone, even those with huge followings. #persistence
    • Productize yourself (11:25). Every hour is something you can dedicate to a different product. Break up your day and optimize every hour – “how am I dedicating time to production of an asset that can be monetized later on?” What is your “[[Return on Attention]]”? #production #Productivity
    • Focus on one platform at a time (17:00). Benefits: 1) [[focus]] (how can I win on this platform and how can I be successful?) 2) if you grow the first platform it helps you build the second one, because you push people from your first platform to your new platforms. The big [[social media]] managers and influencers will often advise to do a whole bunch of platforms at once. When you don’t have the resources of [[Gary Vanyrchuk]] (had 15 people working for him on a holiday!) then it’s way better to focus on one platform and get good at that.
    • More content is better than less (22:25). [[Anthony Pompliano]] started out just Tweeting articles and pulling out the most relevant tidbit and just tweeting that, because he was unconfident and didn’t know what to Tweet about. People don’t listen to all your stuff. It’s like Netflix, give as much as possible and give the audience choice. More content means more people being interested in your content which means more followers. #[[idea generation]] #[[growth hacking]] #Twitter
    • Create once, publish 5 times (26:00). E.g. podcasts – publish as audio, publish as video, take cuts out of it and publish interviews, tweet out links to videos. That is a mechanism that should always exist try to do this as much as possible. Streamline thought creation process and expand how you can use that content to a bunch of platforms. It helps to have a set process and organization – “when we record a podcast, then we do x, y, z”. #marketing #leverage #Strategy
    • You owe the audience everything (29:15). [[Anthony Pompliano]] spends a ton of time going through comments and responding. Often fairly safe, canned responses and emojis. Occasionally, you give a detailed thoughtful response. This gets the audience to become emotionally involved, and feel like they are involved and important. Not scalable, so he’s now at the point where he can’t respond to everything. The audience is why you can monetize, it’s the reason why you’re here. #engagement
  • Actionable Ideas: [[Twitter]] (37:21) #Growth #[[Audience Building]]
    • Create 10+ tweets per day
    • Use spacing and punctuation – make it easier to read
    • Lists increase [[virality]] – super clear, digestible.
    • Links hurt [[virality]] – links are the exact opposite, you have to click the link to get the information. Also, Twitter doesn’t want users to leave their platform. Put the content first and then the link later. Rather than sharing, actually talk about the top things you took away from the article and summarize things.
    • Hijack viral tweets / large accounts. [[Anthony Pompliano]] would put up alerts for big accounts like [[Donald Trump]] and he would race to respond as fast as possible. He would do something to get a lot of engagement, audience looks and checks out his profile. The big accounts are doing your marketing for you.
    • Reply to everyone
    • DMs are the real [[Linkedin]]. People are responsive, and there’s very few people on Twitter that have someone else running their account for them.
    • Set up your profile up for success. Where do you want people to go? Put the link in your profile. Write it out so people know the kind of content they’ll get. You can get tons of traffic on your profile if your tweeting gets lots of impressions.
    • A good gif is worth 1,000 favourites.
  • Actionable Ideas: [[Email Lists]] (43:25)
    • Set a consistent schedule & don’t miss. People subscribe expecting something, so don’t miss. If you do a paid list like [[Substack]], focus on getting free people then converting them to paid, don’t focus on getting people immediately to paying. Most people want to window shop. You have to ease them in.
    • Build your free list – give more than you extract. You are in their inbox, so you better be sure you’re providing value.
    • Don’t be afraid to ask people to subscribe, whether it’s other social platforms or to your paid list. Know what you’re worth.
    • Find like-minded audiences & do link swaps. Figure out how to work with the people in those audiences. “If you like X, you’ll definitely like this thing Y I’m doing”
    • Identify “inflection points” and promote them in advance. Sometimes you know you’re going to be putting out something good. Rather than waiting until you publish, promote in advance: “on Monday, I’m writing something about X, if you want to read it, subscribe”. After you publish it, say “I wrote something about X, if you want to read it, subscribe so you get more like this”.
    • Use email as a distribution point for all content. In the bottom of the email, put links to all your other things – podcast episode of the day, sponsors. Just use a template and fill it in. Passive links pay off
    • Upsell, Upsell, Upsell.
  • Actionable Ideas: [[YouTube]] (48:13)
    • Use the banner to explain what you cover and how often.
    • Don’t be afraid to ask people to subscribe.
    • Create more videos.
    • Optimize the video title, description, and thumbnail for SEO. [[YouTube]] is the second biggest search engine in the world. Look in [[Google Trends]] to see what people are searching for.
    • Use the description for passive links (link to all your other social platforms). Use that distribution you already have.
    • Pin the comment to top of comments with top link. Single best thing to do on YouTube if you want to promote other things. E.g. “pomp writes a daily letter to these kinds of people, here’s the link”. Whatever you’re optimizing for, pin it at the top of the comment.
    • Create an outro with each video. That 15-20 second outtro can just be a pre-recorded clip and you just drop it in.
    • Use search trends to your advantage
    • Leverage guests with large audiences
  • Actionable Ideas: [[Podcasts]] (55:05)
    • Set a frequency and don’t miss an episode
    • Target guests with large audiences
    • Authors with new books are always looking to promote their work
    • Record via Zoom to get audio and video
    • Good microphone and good lighting is important
    • Use micro content to promote individual episodes
    • Transcripts and show notes for SEO
  • “If you’re going to try this, make sure you are ready to dedicate hours a day for a decade”. Just trying it sometimes or just doing one day a week – you will get frustrated, won’t build an audience. #persistence
  • Businesses of the future will build the customer base first and then build the product. When you write and build an audience you get feedback about what they want. As a result when you launch a product, you don’t have to pay for any advertising. #Strategy

Why Learning By Teaching Works and How to Use It

Insights from Learning by Teaching Human Pupils and Teachable Agents: The Importance of Recursive Feedback by Sandra Okita and Daniel Schwartz.

Want to commit key points from this blog post to long-term memory? You can access an Anki Deck here.

Special thank-you to Andy Matuschak for pointing me to this paper.

Brief Overview of the Okita and Schwartz Paper

You have probably heard the advice that you should teach something if you really want to understand it. But does it really work, and if so, how?

Okita and Schwartz provide an extremely readable primer on the research behind Learning by Teaching and why it is so effective as a learning strategy.

They also explore a little-known benefit of learning by teaching called “recursive feedback” – feedback from watching your pupils use what you taught them. Using two separate experiments, Okita and Schwartz find that this special kind of feedback significantly improves teacher learning.

In this post, I summarize key insights from the paper on why learning by teaching works, and conclude with some thoughts on specific strategies for learning by teaching to improve your learning.

Why Learning by Teaching Works

Okita and Schwartz divide the learning by teaching process into three distinct phases: preparation (the teacher’s preparations to teach), teaching (the actual act of teaching), and observation (when the teacher receives recursive feedback by seeing the student apply what they learned from the teacher). It turns out that each of these phases contribute to the teacher’s learning.

Preparing

Preparing to teach students helps teachers learn due to three main forces:

  • Motivation: Preparation introduces strong emotions and motivation to truly understand the material and do it justice. This includes a sense of responsibility, the anticipation of a public performance, and a desire to avoid embarrassment or “looking stupid”.
  • Generation: Preparing to teach involves a “generation effect“. The generation effect means that you remember information better when you generate it from your mind rather than passively reading it. Preparing to teach is generative because you have to frequently retrieve and elaborate on information from your memory as you think about how you are going to teach your students.
  • Meta-cognitive vigilance: Meta-cognition refers to an awareness of your own thoughts. It turns out that preparing to teach generates this kind of awareness through the teacher’s self-explaining and self-generated questions as they anticipate the needs and questions of their pupils. This, in turn, helps teachers identify areas where they have conflicting knowledge or incomplete understanding.

Teaching

Three main aspects of this phase helps teachers learn:

  • Explanation. The act of teaching involves explaining ideas to others. This leads to “explanation effects” that occur from explaining ideas rather than just receiving them.
  • Question answering. Students often have questions of their teachers, which can reveal gaps in the teachers understanding or encourage the teacher to extend their understanding. In fact, Okita and Schwartz cite a study by Roscoe and Chi (2007) that found “tutees’ questions were responsible for about two-thirds of tutors’ reflective knowledge-building activity”.
  • Physiological arousal. Anyone that has experienced public speaking knows that it is a very stimulating activity that focuses your mind. As you might expect, this kind of arousal and attention improves learning.

Observing

In this phase, the teacher receives recursive feedback by watching the student apply what they learned. You might think it’s obvious that feedback improves learning, but it turns out that’s not always the case.

For example, direct feedback, where a student tells a teacher directly how well they did, can be counterproductive because people often take this kind of feedback personally, leading to “self-handicapping, learned hopelessness, or social comparison”. When this happens, the recipient of the feedback tends to adopt attitudes like, “well, I guess I’m not the kind of person who’s good at math“. As a result, they give up or opt for an easier subject. As much as we all like to believe we would receive feedback objectively and use it to our advantage, our egos often get involved, whether we like it or not.

Thankfully, recursive feedback side-steps this issue. Since the teacher is not directly receiving feedback from the student, their egos are uninvolved. Instead, they see for themselves what worked and what didn’t. No one explicitly tells them, “this is where you need to improve, this is what you did wrong”. Okita and Schwartz note that this “creates an ego-protective buffer, because the pupil rather than the tutor takes the immediate responsibility for being wrong”.

I also believe recursive feedback helps build empathy skills. A lack of empathy, in my opinion, is often a major barrier to learning and self-improvement. Think of how many people you have encountered in schools or workplaces that have tried to explain something to you, but assume an unreasonable amount of knowledge, leave out critical bits of information, or explain things in ways that are otherwise unhelpful (e.g. using tons of jargon you’re unfamiliar with)? Often these people are very intelligent, and would be far more effective if they gained some empathy.

How can we use these lessons about Learning by Teaching to improve our own learning?

The main lesson I took away from the Okita and Schwartz paper is that Learning by Teaching is not just folk wisdom. It works. So, as learners, we should try to use it wherever we can. Here are some ideas:

  • Develop a course or become a tutor. If there is a topic you really want to master, and you have enough time and resources, you can run an online or in-person course. In addition to the learning benefits, you can make some side income.
  • Observe your students. To take advantage of recursive feedback, you need to be able to see your students apply their knowledge. Use tests, presentations, or projects to see what your students learned.
  • Keep it interactive. Interaction with your students means more questions from them and, as a result, more learning for you. Okita and Schwartz note “Tutoring sessions should be arranged to limit the natural tendency of tutors to slip into didactics, which can eliminate opportunities for pupils to ask questions“. Students will often ask useful questions that help you better understand the material.
  • Write online. Writing online is a form of teaching, whether it’s blog posts, twitter, email newsletters, or something else. It also provides ample opportunity to receive valuable feedback about your work.
  • Develop a smart and interesting following. The quality of pupil questions impacts teacher learning. This means that, if you have an email list or a blog, or some other kind of online following, you learning would benefit from attracting high-quality people. Check out David Perell’s work on writing online, where he emphasizes the importance of writing for intelligent readers and leveraging their feedback to improve yourself.
  • Encourage your students to keep in touch about the projects they work on and how they applied the knowledge you taught them. Your students will appreciate this, and you’ll also get valuable recursive feedback.
  • Build things and watch people use what you’ve built. Okita and Schwartz point out that recursive feedback applies to areas other than Learning by Teaching. For example, producing products, tools, crafts, art, and then getting recursive feedback by observing how your “customer” uses it. One application of this idea that comes to mind is Agile software methodology, which relies on building software quickly, getting feedback on it as soon as possible, and constantly iterating and improving based on that feedback. If you produce something that others consume, you’ll gain by watching them use it.

Subscribe to my Spaced Repetition Newsletter to get regular tips and ideas about spaced repetition, using Anki, and improving your learning speed.

“The Artists of Data Science” Podcast Interview

My friend Harpreet Sahota just launched a new podcast called “The Artists of Data Science”. You can hear my interview here. We cover a variety of topics, including my path to becoming a data science, spaced repetition, compounding, finding good data science projects, the strangest question I’ve been asked in a job interview, and more!

Check out his other interviews there as well, including Vin Vashishta, Andreas Kretz, Angela Baltes, Daniel Bourke, David Tello, Pooja Sund, Alexander Lim, Lediona Nishani, Mikiko Bazeley, and Amit Jain.

Notes on Developing Transformative Tools for Thought

Occasionally, special tools come along that amplify our minds, enabling new kinds of thought not otherwise possible. Computers, writing, speaking, and the printing press are all examples of these “Tools for Thought” that surge human potential. 

This essay from Andy Matuschak and Michael Nielsen explores whether we can accelerate the development of these kind of tools. They also provide a taste of potential tools for thought with their prototype “mnemonic medium”, an interactive post on quantum mechanics called “Quantum Country” with embedded flashcards combined with a spaced repetition system delivered through email follow-ups. 

Their essay is a must-read for anyone interested in spaced-repetition or productive learning. There are several points I found thought-provoking. I believe each of these insights indicate a need for a new Tool for Thought for flashcard development and sharing.

Spaced repetition creates exponential returns to studying

Based on Quantum Country user data, Matuschak and Nielsen estimate that devoting only 50% more time to spaced repetition after reading the essay resulted in users recalling the key points for months or years. 

In other words, relatively small investments in spaced repetition after reading an article produces outsized results – more evidence to place on top of the mountain of research suggesting spaced repetition works. 

Good flashcard development is difficult

Matuschak and Nielsen note that it takes a surprising amount of skill and time to build quality flashcards, especially for abstract concepts. This is probably a big reason why most people fail to adopt spaced-repetition tools like Anki. Since flashcard development is a skill that you develop over much time and effort, new users tend to add cards in a way that inevitably leads to frustration and failure. 

This may partially explain the efficacy of Quantum Country: the authors are experts in both quantum mechanics and flashcard development – a rare but essential combination of skills for their essay to work. 

Flashcards written by others can be useable

Some people the spaced repetition community don’t believe in using flashcards created by others, and with good reason. They’re often poorly written. They’re idiosyncratic. They’re missing crucial contextual information that you lack as someone who hasn’t read the original source material. I used to be one of these non-believers.

But the effectiveness of the Quantum Country essay suggests that shared flashcards can work well. This has benefits of saving users of the burden of flash card creation, as well as preventing new user frustration from poor flashcard building skills and poor domain knowledge.

Matuschak and Nielsen hypothesize that the quality of their flashcards is what makes this work. I agree, but I have a few more hypotheses: 

  • Their flashcards are introduced in a logical progression as users read the essay. In contrast, shared decks in Anki shuffle cards randomly and are not encoded with dependency information.
  • Their flashcards are clearly connected to a source (i.e. the essay), providing important context for the user.
  • Users learn the material before they review flashcards. This is in line with the common wisdom that flashcards don’t work if you don’t already understand the material – they are a tool for retention, not learning. Aside: is this common wisdom true? I’m not so sure. Socrates taught using Q and A, so why can’t you teach a subject entirely with flashcards? If it is possible, what are the prerequisites to making it work? 

Elaborative encoding

Matuschak and Nielsen note elaborative encoding as another learning tool shown to be extremely powerful in promoting memory. Essentially, it means connecting new ideas you want to remember with old ideas you know well, providing a fast path in your brain to new information. 

Remember this concept while developing your flashcards. Whenever you add a new card, think about what you already know well and how you can connect this to the new knowledge.

A New Tool for Thought?

Matuschak and Nielsen’s article has renewed my interest in a tool for thought idea I’ve been pondering for quite a while: a platform for collaborative flashcards development and sharing. I believe such a tool, if properly developed, can address the issues that limit the use of spaced repetition:

  • Spaced repetition practitioners currently need to develop their own flashcards, which requires a significant amount of time, domain expertise, and flashcard-building skill. There needs to be a place where experts can create shared flashcards, and there should be a proper incentive structure encouraging creators to improve these flashcards over time.
  • Flashcards are not clearly connected to original sources. Spaced repetition practitioners should be able to pull up pre-built flashcards for a source document they are working through. 
  • Current tools do not provide information that link flashcards together (other than knowing two flashcards are part of the same deck, or have the same tag). At the very least, flashcards should have a notion of “depends on” or “prerequisite to”. This would make shared decks more useful by showing the intended progression of knowledge. It would also aid elaborative encoding (e.g. examining cards you’ve reviewed and linking them up to cards “nearby” in a knowledge graph)

I strongly believe a tool like this needs to exist, as you may have guessed if you noticed the Download my Brain feature I built for this site that provides a platform for sharing my personal Anki decks. I have started work on a more generic tool for collaborative flashcard construction and sharing and will keep you posted once I have something ready for production. 

Thanks to Andy Matuschak and Michael Nielsen for the inspiration to follow this path.

If you’re interested in space repetition and techniques for productive learning, subscribe to my Spaced Repetition Newsletter. Check out my shared data science Anki decks here: Download my Brain.

The Hidden Power of Compounding (and 4 Ideas for Harnessing it)

Ways to take advantage of this powerful and often-overlooked force for improvement 

Ever wonder how successful people reach such heights? Think of a wildly successful person you admire. How did they get there? 

The typical answers are hard work, innate gifts (personality, natural ability) and luck. These factors play a role, but the most important factor is left out: they leverage the power of compounding.

What is Compounding?

For compounding to occur, only two things are required:

  • Growth: Something must be growing by some percentage each year
  • Time: The growth process happens over multiple years

You’ve probably heard of a specific type of compounding: compound interest. In this context, compounding means that growing your money by some percentage every year eventually snowballs into huge results if you give it enough time. 

For example, say you are able to earn 7% per year on your money. That doesn’t sound like a lot: if you invest $1000, that’s $70 in a year. Seems pretty modest.

Yes, you’ll earn $70 in year one, but in year 2 you earn $70 (7% of $1000) plus 7% on the $70 of additional money in your account that grew from year 1. As time passes, that 7% per year pumps out bigger and bigger piles of cash:

  • In Year 10: You’ve doubled the size of your account ($1,967)
  • In Year 20: You’ve quadrupled the size of your account ($3,870)
  • In Year 30: Your account is now almost 8 times your original investment of $1,000 ($7,612)

One lesson from this exercise is to save and invest some of your money. Your future self will thank you. 

Another lesson is the earlier you start, the better. More years means more opportunity for exponential growth. 

But there’s an interesting corollary to this exercise that is less obvious, and much more exciting: compounding isn’t just something that happens to your bank account – it applies to many other areas of life. 

“We’re all sort of blundering fools, but if you just get some rate of improvement and let it keep compounding, you can do pretty well…You always want to be on some sort of curve where you’re compounding.”

– Tyler Cowen, on David Perell’s Podcast

All you need for compounding is growth and time – there is nothing about dollars or bank accounts mentioned in that definition. If you can get some percentage growth rate in some area of interest every year, you’ll eventually reach heights you never dreamed possible.

Think about some area where you want to excel. As an example, let’s say you’re in sales. If you improved at sales 10% every year, you would be twice as good in 7 years, four times as good in 15 years ten times as good in 25 years, and seventeen times as good in 30 years. In any given year, you’re not making tremendous improvements, but over time persistence leads to tremendous outcomes.

Also, keep in mind that although 10% growth is a great rate for a bank account, who’s to say this is a good rate of growth for your sales career? Maybe a reasonable growth rate is much higher. If that’s the case, you can expect much more dramatic results. 

I believe this is how extraordinary people like Elon Musk reach rarefied heights: achieving a high growth rate in an area (e.g. managing a private space company) through intense focus and then relentlessly persisting to maintain that high growth rate over many years.

4 Ideas for Better Compounding

There are lots of ways you can compound in an area you want to improve. Look for anything that 1) gives you growth by some percentage or 2) helps you maintain that growth over multiple years. Some of the obvious ideas here include reading, taking lessons, attending talks, working with a mentor, and just simply doing work in the area you want to compound.

That being said, I have a few other tips that can both increase your percentage growth rate (ideas #2 and #3) and ensure you stick to it over the years (ideas #1 and #4). 

Idea #1 – Have a plan

The key to compounding is consistent, focused effort over multiple years. It’s hard to do that without being clear about where you want to improve. If you don’t have clear goals, you’ll forget them or lose discipline, stifling your compounding efforts.

So, I recommend writing down the areas you would like to compound over time. Check back on this list regularly (I check weekly) and make sure that every year you’re making some effort to improve your abilities by some percentage.

For example, here’s a list of areas I’m personally focused on compounding over time:

  • Data science
    • Statistics / math
    • Programming
  • Communication
    • Writing
    • Speaking
    • Sales / persuasion
  • Managing teams / project management
  • Personal brand / developing followers on my blog and email list
  • Health
  • Relationships
  • Cooking
  • Drumming

I keep a Google document of this list and keep track of specific things that I am doing within each category to propel myself forward. 

Idea #2 – Use spaced repetition

You’ve probably had this experience: you read a book or take a course and you want to retain it and apply as much as you can. Inevitably, the precious knowledge slowly exits your mind and 6 months later it feels like you didn’t learn anything in the first place.

Spaced repetition is a solution to this problem.

Spaced repetition is quizzing yourself on knowledge in increasing intervals of time. It’s extremely effective and time efficient. I have flashcards in my spaced repetition system that I just answered that I will not be quizzed on again in 2 years. This spacing allows you to hold tens of thousands of flashcards in your mind while only reviewing tens of flashcards a day. 

For more information about the science behind spaced repetition, check out this great summary by Gwern Branwen.

I have been a long-time user of spaced-repetition tools for committing things to long-term memory. During my university years, I wrote cards in Supermemo for all of my courses, and it was the secret weapon to my performance. Today, I use Anki, and it continues to be a key tool for compounding my knowledge in data science and retaining the vast amount of information required for success in the field.

I’m so excited about spaced repetition, I even have a feature on my website that I built to share my data science Anki decks with the world, organized by source: Download my Brain! I’m also running a Spaced Repetition Newsletter that will provide the latest news and links related to Anki, tips on using Anki, and ideas related to spaced repetition and productive learning.

Idea #3 – Keep a commonplace book

A commonplace book is a place where you store wisdom or interesting things you’ve read or heard from others, or have thought about yourself.  I’ve written previously about the benefits of keeping a commonplace book and a summary of my personal commonplace book system that involves a spaced-repetition-esque component (I wrote code to email myself 5 randomly selected commonplace book notes from my collection). Lots of others have written about them as well (here is a good article from Ryan Holiday, which is where I first heard about this idea). 

A commonplace book is similar to spaced repetition in that it’s a tool to help you retain more of the important things that you learn. It also provides you with a body of material you can draw from, which is particularly valuable if you are a writer. 

I personally use my commonplace book system to hold long-form wisdom that doesn’t lend itself well to flash card quizzing.

Idea #4 – Use commitment contracts

Once you have a plan for compounding in your improvement areas, you need to follow-through with that plan. Over the years I’ve tried many strategies to help deal with this issue of staying motivated and sticking to a plan, and the most effective tool I’ve found are commitment contracts. I’ve used commitment contracts to read and write more than I ever have in the past, despite being busier than ever with a full time job and a 3-year-old at home.

Commitment contracts are agreements where you commit to doing something, and failing leads to actual consequences. For example, I have a commitment contract to spend a minimum number of hours each week on various areas of interest. I track my time during the week, report back each week, and if I’m below my target I’m penalized $5. 

I have found this surprisingly effective, especially considering the low stakes. It doesn’t take much of a penalty to provide enough motivation to follow through with your targets. I think this may be tapping into our inherent loss-aversion: we are irrationally repulsed by a loss, much more than the size of the loss would indicate. Of course, you can make your failure penalties higher if you want, but I personally don’t find it necessary. 

The specific tool that I use for commitment contracts is StickK. I highly recommend it. 

Conclusion

I think these tips will help you out, but the most important point to remember is compounding only requires two things: growth and time. Growth ensures you make progress, and time is what allows you to reach heights you never thought possible. In any given year, you may not be making huge improvements, but over time your persistence leads to huge outcomes.

If you want to commit this article to long-term memory, download the Anki deck I put together for it here. If you are a user of Anki or other spaced repetition tools, you can subscribe to my Spaced Repetition Newsletter for the latest news, tips, and ideas about spaced repetition, using Anki, and improving your learning speed.

I made a speech based on this blog post that you can find here:

This is a speech version of this post from my local Toastmasters group

Spaced Repetition for Efficient Learning by Gwern Branwen

If you’re interested in spaced repetition and haven’t read it yet, Gwern Branwen’s essay is a fantastic review, including research into the benefits of testing, what spaces repetition is used for, software, and other observations. There are lots of resources here for people that want to know the research behind why spaced repetition works, including many studies on the effects of testing and the effects of spacing your learning. 

Here are a few points I found most interesting in Gwern’s essay:

People underestimate the benefits of spaced repetition: Gwern references fascinating research on how students and teachers grossly underestimate how much better spaced repetition is compared to cramming for learning. 

Spaced repetition as a tool for maintenance of knowledge, not gaining new knowledge: According to some of the research referenced in the essay, spaced repetition doesn’t teach you new skills or abilities. Rather, it just helps you maintain existing skills.

  • Skills like gymnastics and music performance raise an important point about the testing effect and spaced repetition: they are for the maintenance of memories or skills, they do not increase it beyond what was already learned. If one is a gifted amateur when one starts reviewing, one remains a gifted amateur.” 
  • I’m resistant to this idea that you can’t learn using flashcards. I agree if new flashcards are thrown at you randomly, then yes that is a very inefficient way to learn. But what if flashcards are presented to you in a specific order when you are learning them, and “advanced” cards are not shown to you until you’ve learned the prerequisite “beginner” cards? I don’t know the research on this, but this strikes me as potentially a better way to learn than reading, if the cards are formulated properly.  To my knowledge, none of the spaced repetition tools out there (e.g. Supermemo, Anki) allow for this kind of “card dependency” – if you know of a tool that does this, let me know.

Tradeoff between lookup time and mental space: A key question that you have to ask yourself when using spaced repetition is what information should I add to my spaced repetition system? Why add anything at all to it when you can just look it up on the internet?

  • The answer is that lookup costs can be large, especially for information that you use a lot. Often when you want to apply your knowledge, there isn’t enough time to look it up, or the time to look-up impedes your thinking. An extreme example of this would be trying to recall an important piece of knowledge in a job interview – good luck pulling out your phone in front of the interviewer to get the right answer.
  • To figure out what to add, you have to strike a balance with the tradeoff between lookup costs and the cost of adding the item to your spaced repetition system and reviewing it. Gwern provides a 5 minute rule as a criteria for deciding what to add: if you think you’ll spend over 5 minutes over your lifetime looking something up or not having the knowledge in your head will wind up somehow costing you more than 5 minutes, then it’s worth it to add it to spaced repetition.

Idea – dynamically generated cards: Gwern offers some interesting ideas about the possibility of dynamically generated cards. For example, having a card that teaches multiplication by randomly generating numbers to multiply. Similar ideas apply to chess, go, programming, grammar, and more.

If you’re interested in space repetition and techniques for productive learning, subscribe to my Spaced Repetition Newsletter. Check out my shared data science Anki decks here: Download my Brain.

How to Make Decisions: A Mental Framework and 5 Common Failures

Whether you like it or not, you are faced with a barrage of decisions every day. Human beings are essentially decision machines, and the quality of your life hinges on these choices you make. The better you are at it, the better your life will be.

Some decisions are low stakes. Sometimes, the stakes are tremendous. What should I let my kids do? Where should I invest? Should I switch jobs? Should I ride my bike or take a car? Should I move to that neighbourhood? Should I reach out to the family member I haven’t talked to in ages? Should I cross the street? 

It’s valuable to have a mental decision framework that guides you to increase your decision quality and decrease the stress of the process. It’s also valuable to be aware of some common decision-making traps to avoid. 

The Framework

The mental framework I use for making decisions looks like this:

As you can probably tell, this mental framework is heavily influenced by economics (specifically, cost-benefit analysis which is a decision framework used by economists). 

At the highest level, making a decision in this framework involves 3 steps:

  1. Estimate the benefits
  2. Estimate the costs
  3. If the benefits outweigh the costs, do it. If not, don’t. When you’re choosing between multiple courses of action, pick the one with the highest benefit-to-cost ratio 

Steps #1 and #2 each break down into understanding two sub-components: probability and impact.

  • Probability is the chance of the chance the benefit or cost happens
  • Impact is how good the benefit is or how bad the cost is if it happens

Higher probability or higher impact means you should place more weight on that benefit or cost. 

Most of us probably use a similar framework in our heads without even knowing it, but spelling it out like this helps clarify tough decisions and exposes hidden flaws in our decision-making.

5 Common Decision-Making Failures 

This model may seem simple at first glance, but it’s deceptively tough to apply. This is because we’re human beings, not computers or all-knowing gods. We will never come close to applying this decision-making model perfectly. But even though we’ll never reach perfection, there are many ways to improve. 

Here are five common failures you can keep an eye out for.

Common Failure #1: Loss aversion

Loss aversion means that we tend to hate losses much more than we love gains. As a result of this tendency, you irrationally underestimate benefits and overestimate costs in your decision-making. 

This leads to particularly skewed decisions when considering benefits that are huge, but low probability (in other words, high payoff but a chance of a “loss”). 

For example, say you’re applying for a big grant and you estimate your probability of getting it is less than 5%. Many would instantly choose to not write the grant without giving a second thought because the chances are slim. It’s true that winning the grant is unlikely, the payoff is huge. Applying might still be worthwhile and shouldn’t be dismissed without consideration. 

Or consider Elon Musk. When he started SpaceX, he didn’t say “well, the chance of building a profitable, successful space company is low, so let’s just forget it”. Instead, he thought “the probability of success with this company is only about 10 percent, but the upside if I’m successful is tremendous so I’m going to try”. 

Always remember that the emotional toll of losing is often irrationally high and doesn’t represent the actual cost of losing.

Common Failure #2: Not valuing your time

I remember in university, there was occasionally a “free pancake day”. Students would line up out the door for them, waiting about an hour in line to get these “free” pancakes. 

The pancakes are only “free” if you don’t value your time, which is a huge mistake. 

I personally believe you should value your time at least as high as your equivalent hourly salary at work. You may disagree with that number, but the point is that it’s absolutely worth something. This was one of the key takeaways from my economics background and probably the best piece of advice I can give.

Common Failure #3: Focusing only on money and measurable things

We often ignore benefits or costs that are not monetary, and in general we have a difficult time considering things that are not easy to measure. Here are some difficult-to-measure areas where there are often huge costs or benefits:

  • Health and wellness (e.g “I could never ride a bicycle because it’s too dangerous”)
  • Relationships with loved ones (e.g. cutting ties with a family member over a $100 dispute)
  • Education, skills, and learning (e.g. “our company doesn’t have the skills to perform this core business task so let’s just contract it out”)
  • Culture / morale (e.g. “We can’t fire Sarah. Sure, she’s a toxic, unethical person and makes everyone in the office miserable, but damn is she ever good at making widgets!”)

Common Failure #4: Not considering the costs of inaction

No matter what you do, you are making a decision. Choosing to not do anything or to “not make a decision” is itself a decision. As a result, the benefits and costs of inaction must be considered alongside the benefits and costs of all other courses of action. 

Common Failure #5: Ignoring hidden benefits and costs

Many benefits and costs are commonly ignored because they are less obvious. For example:

  • Transporting yourself by bicycle leads to improvements to your physical appearance, mood, strength, focus, and energy. 
  • Many people think the cost of owning a home is just a monthly mortgage payment. Hidden costs include property taxes, maintenance, and insurance, among many other expenses.
  • Many people think the cost of driving a car is just the cost of gas. Hidden costs here include insurance, parking, and maintenance.

Keep an eye out for costs and benefits that may be lurking slightly under the surface.

Conclusion

Decisions are hard. You’ll never make perfect decisions, but if you have a solid mental framework at your side and have some awareness of common mistakes, you can get better. More importantly, you can relieve some stress and feel more confident that you’re doing the best you can with what you have.

If you want to commit this article to long-term memory, download the Anki deck I put together for it here. If you are a user of Anki or other spaced repetition tools, you can subscribe to my Spaced Repetition Newsletter for the latest news, tips, and ideas about spaced repetition, using Anki, and improving your learning speed.