Spaced Repetition (2019) | Hacker News Reader

Spaced Repetition (2019)

gwern.net

83 pointsjacquesm66 comments

> Son & Simon 2012: Furthermore, even after acknowledging the benefits of spacing, changing teaching practices proved to be enormously difficult. Delaney et al (2010) wrote: “Anecdotally, high school teachers and college professors seem to teach in a linear fashion without repetition and give three or four noncumulative exams.” (p. 130). Focusing on the math domain, where one might expect a very easy-to-review-and-to-space strategy, Rohrer (2009) points out that mathematics textbooks usually present topics in a non-spaced, non-mixed fashion. Even much earlier, Vash (1989) had written: “Education policy setters know perfectly well that [spaced practice] works better [than massed practice]. They don’t care. It isn’t tidy. It doesn’t let teachers teach a unit and dust off their hands quickly with a nice sense of ‘Well, that’s done.’” (p. 1547).

It's strange they would use math as an example or where spaced reptition would work.

It's specifically one of the least applicable domains. Things that are mainly conceptual, that build on previous concepts, don't benefit from spaced repetition at all.

With math, the hard part is the conceptual understanding. But once you understand a concept, you understand it. You don't really need to review it again.

This is why the part of the article that lists what spaced repetition works well on is all factual/associative -- vocabulary, trivia, medical terminology, etc.

Spaced repetition is for simple associative facts -- "x is y" -- not conceptual things.

For whatever reason, once we've understood a concept, it seems to be part of our thinking forever. But of course the hard part is understanding it. Some people struggle for days, weeks, or even years in school to truly understand things like derivatives, or pointers, or trigonometry. Spaced repetition does nothing for that.

I'd like to respectfully disagree.

A big part of math is building up your toolkit: when I see x, I should do y. Feynman famously made this a big part of how he approached quantum mechanics, saying that it was his toolkit that allowed him to solve problems others saw as impossible.

While the initial understanding can't be done with spaced repetition, remembering what's in your math toolkit and how to use it can very much be aided by spaced repetition. I've personally used this to great effect teaching people math through my spaced repetition app.

Can you give an example?

I'm not saying math practice isn't useful -- after all, that's what homework problems are for, to figure out which concepts apply and use them. Repetition itself is useful.

But in my experience, this doesn't benefit from being spaced over time, and it's more about achieving full understanding of the concept.

You seem to be describing pattern recognition, but pattern recognition is conceptual -- you need time/experience to build that up, but it can be done in a day and doesn't need to be revisited days/weeks later in order to not be forgotten.

What kind of math content do you think benefits from repetition that is necessarily spaced over time?

In my experience, which is graduate level mathematics, spaced repetition is essential. Once you get beyond basic mathematics and algebra, recalling and understanding concepts and definitions is essential. This is true at the calculus level where you need to remember things like the chain rule, integration by parts, various theorems, etc. But it is even more true at the higher levels of mathematics where there are many more theorems, lemmas, and definitions to remember. I find that at that level, just having a huge depth of recall for definitions alone is incredibly useful. If you remember enough definitions you can start to piece things together. If you can recall theorems, lemmas, and some key proofs you can achieve quite a bit.

Personally, I only understood logarithms after memorizing the log laws using spaced repetition. I had been taught logarithms before descriptively, but only once the equations were in my head could my brain make sense of how they work.

Good example. I know people frown on memorization, but personally I would say the following are useful to know deeply:

- Log laws

- Exponent laws

- Derivative rules

- Probability laws

- Combinations/Permutations

- Matrix algebra

- more?

When you're reading through math derivations, or trying to work one out yourself, you absolutely need to have some basic literacy in these rules.

> But once you understand a concept, you understand it. You don't really need to review it again.

I disagree - if this were true I would still understand all of calculus and linear algebra, which I took years ago.

I think part of what you say is true - some concepts that are understood stick with you more than the sort of fact memorization you might find in a history class. But there are still many math skills that would benefit from spaced repetition. What are line integrals used for and how do I compute one? What does the determinant of a matrix represent and how can I find it? What is Stokes' Theorem?

It's one thing to understand a high level concept (e.g. an integral can be used to model continuous accrual of some quantity) versus details (e.g. how do I integrate common trig functions?).

I frequently used spaced repition in my maths/physics courses to great success. The types of things I would memorize:

- Key theorems

- Proofs

- Any kind of visualizations (e.g. statistical distributions)

- Properties

- Trig identities

A lot of time in math or math heavy fields, there are a lot of tiny details and nuances. Using an SRS properly really solidifies those concepts in your head (See my comment below on manufacturing 'aha' moments). It also helped because a lot of those key concepts were immediately available in my mind and I was 100% I had them correct. Memorization reduces a lot of mental friciton.

In math, there can be a lot to memorize! There is a lot of notation, and associated definitions and facts. Also formulas, and not to mention the values certain important constants.

Mathmatics textbooks work well as a sort of reference volume. If you want to repeat earlier sections, you should, and you can, but it doesn't mean the text book needs to repeat the content. The lesson plans can use the book non-linearly.

Depending on the specific math course, it can be the case that later topics contain earlier topics anyway, so a final with only problems from the end will also cover techniques from the beginning and middle. Certainly, that's not always the case.

Yup. It's also really difficult for students. I use it in my tutoring practice, and I always have to convince my students that it doesn't mean they're backsliding.

What helped me the most with sticking to SR and Anki over 6 months now was the tip by Michael Nielsen [0] to not use any plugins and to not personalize anything:

> I know many people who try Anki out, and then go down a rabbit hole learning as many features as possible so they can use it “efficiently”. Usually, they're chasing 1% improvements. Often, those people ultimately give up Anki as “too difficult”, which is often a synonym for “I got nervous I wasn't using it perfectly”. This is a pity. As discussed earlier, Anki offers something like a 20-fold improvement over (say) ordinary flashcards. And so they're giving up a 2,000% improvement because they were worried they were missing a few final 5%, 1% and (in many cases) 0.1% improvements. This kind of rabbit hole seems to be especially attractive to programmers.

[0] http://augmentingcognition.com/ltm.html

I've seen the same reaction to learning VI-like editors. I've heard a few times variations on "I don't want to spend my time worried I could have done an editor operation in 2 fewer keystrokes" I sure as heck don't want to do that either and I've been using vi-like editors for almost 30 years now.

Anki has some poor defaults for some parameters, which will cause you to be needlessly inundated with reviews.

Tweaking the parameters may just be a 1% improvement in terms of learning and retention. Perhaps even no improvement at all.

Where the improvement is seen is in the use of your time.

> Anki has some poor defaults for some parameters, which will cause you to be needlessly inundated with reviews.

The problem is that people incorrectly assume that they must "do something" about missed reviews. That's just not how it works; in fact, Anki will give you extra credit for recalling a card beyond the expected forgetting time, so working with a backlog of cards to review is a lot easier than it sounds.

The whole idea is that once the expected forgetting time for those "review" cards has been pushed far in the future, that's the best time to learn new, more challenging content. It works very well in my experience.

Where reducing the amount of reviews may well be justified is for cards that are "nice to have", but that you don't actually care about remembering. That's also the only sorts of cards for which "leech" protection makes sense, and that's a lot closer to a poor default in my view.

There are multiple problems. One is that if you have a card with a nice long interval, and you happen to lapse on it, the interval is trimmed to a very short value, by default.

That article should be required reading before one delves into Anki, it's like a 'best practice' guide for a fairly open-ended tool. Great stuff

I was pretty frustrated with existing spaced repetition software, so I went ahead and made my own[1].

It's markdown enabled (including latex), with a cross platform offline-first desktop app and mobile apps. The pro subscription offers syncing across devices.

It's sort of a mix between a zettelkasten and spaced repetition system.

[1] https://mochi.cards/

What did you dislike about Anki and how does this work better?

Can you import and export decks between them?

Mostly found the interface really clunky and confusing / difficult to use, which made me dread using the app regularly. Some things were also surprisingly difficult to accomplish / impossible to do. I also wanted to be able to write markdown for my cards. Aside from that I liked the concept of Anki (desktop app, offline first, decks sharable via an export file). Mochi supports Anki imports, but I don't believe Anki supports importing any other format other than .apkg and .csv.

I'm trying it out, and it does look nicer.

A few unasked for comments/questions:

1. I'm worried about committing to a closed system. I see you have an export, but at first glance it seems to make an unreadable .mochi file. A csv would be nice. Could I ever export back to Anki or something if I stopped using this?

2. Will you support other card types besides front/back?

3. I miss Anki's database view. Your deck view supports a "list" as well as "grid," but the list items are very tall, and I can't see any other columns.

4. Your "forgotten" multiplier works very differently from Anki's, I believe -- I think they just reset you to zero, right? This seems potentially very cool and potentially a big difference between the two apps. (In an attempt to split the difference between the two systems, I'm trying out 0.3 instead of 0.5.) Do you have any research showing this works?

1. The .mochi file is just a zip archive with a plain text description of the deck/cards (in extensible data notation[0]) and any media files used in the deck(s) exported. You can find documentation on the format of edn file on the FAQ[1].

2. At the moment you can also use cloze deletions. I do have plans for things like typing in an answer, or drawing with a touch screen, but I have no plans for something like multiple choice, but I could be convinced otherwise. I'm also open to suggestions.

3. Yeah, this is probably the best part of Anki IMO. I decided not to include it in the initial version of Mochi because I thought plain markdown documents would be easier for new users to "grok". I still plan to add this kind of templating thing in the future, but I still need some time to let the idea "bake".

4. You can actually get this behavior in Anki, but it's not the default. The initial inspiration for this change came from this blog post[2], but it is roughly equivalent to the Leitner System[3]. This other blog [4] also provided a lot of influence in some of the design of the SRS system.

[0] https://github.com/edn-format/edn

[1] https://mochi.cards/faq.html#how-can-i-create-my-own-mochi-f...

[2] https://eshapard.github.io/anki/anki-new-interval-after-a-la...

[3] https://en.wikipedia.org/wiki/Leitner_system

[4] https://massimmersionapproach.com/table-of-contents/anki/

Very nice.

Are there pre-made sets for various topics?

Can people share their card sets with other users?

While I can't speak for Mochi Cards, if you're looking for a spaced repetition app that allows sharing, I'd recommend my own [1]. Sharing with friends and teachers is built into the app.

If you're particularly interested in pre-made sets, both my app and Mochi Cards rely on importing from Anki and Quizlet, which both have a large amount of pre-made sets.

[1] https://get21stnight.com/

Sharing isn't built into the app yet, but you can export decks and share them.

Spaced Repetition is a super power! I think learning about learning is one of the best things you can do for your personal and professional growth. I've noticed a huge growth in my own learning journey since diving deeper into how spaced repetition and other techniques can help you.

I've also been more intentional about my time programming at work. Focusing on certain things to improve on a daily/weekly basis. Most of our day to day work is more akin to a performance than deliberate practice[1].

I also noticed that I didn't have a great way to use spaced repetition with programming skills, so I've been working on a platform that lets you practice python fundamentals with spaced repetition. It's pre-launch but I'm documenting progress on my newsletter[2].

1. https://deliberatepython.substack.com/p/is-programming-at-wo...

2. https://www.sendfox.com/deliberatepython

One of the best uses for Spaced Repitition I've found is to manufacture those "Aha!" lightbulb moments. The process works like this:

1. I create diagrams/other visuals from what seem to be important concepts from the book before reading it and without knowing what they are about.

2. I will use Anki to try and draw those visuals from memory. At this point, I learn the intimate details of the visual but have the constant question in my mind,"what is this thing?". I try to time starting this process 3-5 days before reading the book.

3. I read the book as I normally would, and when I hit a discussion about whatever visual I memorized my brain immediately goes "Oh! That's what that thing was!". Those euroka moments are actually great tools for remembering concepts and with the SR system I also have a visual to tie the concept too.

Other things I've used SRS for:

- Internalizing life principles. Anytime I feel "Oh thats important principle, I don't want to forget it". Sometimes its been from a book, othertimes its useful feedback from coworkers, or just anything thats a principle I want to hvae top of mind.

- Associating faces with names. When joining a new company and meeting dozens of new people, using Anki to remember peoples names works well.

This is interesting. Don't you risk the 'Eureka moment' being something that's not important? I wonder if one could, as a teacher, manufacture this with seemingly nonsensical one-liners that eventually become clear.

I sort of do this with my kids, dropping titbits to try and entice interest: "of course we are time travellers, moving at a second-per-second most of the time" but that could be more explicitly tailored to "... except under time-dilation". Perhaps the latter gives a hook "time dilation" to hang the concept on later when it's learned -- maybe like reserving disk space when downloading a large file. Or perhaps it's the 'transfer' of the concept to the hook, at some point, that makes for a 'repitition'.

If you're interesting in improving your ability to learn, I recommend Learning How To Learn on coursera: https://www.coursera.org/learn/learning-how-to-learn#syllabu...

It discusses the actual science of learning (it does not feature spaced repetition).

Isn't spaced repetition, like, the principle way of learning ... how can a course on learning not include it?

I didn't think we knew how information is laid down in/by neurons: could you precis this "actual science" that it teaches?

What's weird about the above comment is that this course specifically talks about spaced repetition and active recall as the only ways to learn effectively. I took this course and it revolutionised my life - but mostly because I started using Anki afterwards.

I just checked the course. Spaced Repetition is listed in the glossary of terms, so you're right that it's in there somewhere. I don't remember it as being a primary topic, whereas active recall is primary.

This course also revolutionized my life, but I find spaced repetition to introduce too much pointless overhead. Instead I focus on aggressive active recall until I understand the content and then I move on.

This is exactly why you should take the course. To clear up the misconception that spaced repetition is the principle way of learning.

This is probably one of the best investments most people can make.

Think about it. If you become a more effective learner it can have a huge compounding effect.

This course is one of my favorites. Thanks for sharing it!

I don’t remember if the course says spaced repetition exactly, but active recall testing is featured a lot, and active recall testing is very much an ajacent concept to that.

I think they're orthogonal. You could have spaced repetition without active recall and you could have active recall without ongoing spaced repetition.

I think active recall is the more important fundamental mechanism for learning and the focus on spaced repetition tools detracts from that.

One thing that bothers me is this section: Most existing software follows SuperMemo in using an expanding spacing algorithm, so it’s not worth worrying about; as Mnemosyne developer Peter Bienstman says, it’s not clear the more complex algorithms really help32, and the Anki developers were concerned about the larger errors SM3+ risks attempting to be more optimal. So too here.

None of these people criticizing SuperMemo have actually used it (as far as I know, or if they did, probably a decade or two ago). I know lots of people that have switched from Anki to SuperMemo with way better rep loads. If you look at supermemo.guru and any of the writings of the creator, it's quite clear he knows what he's talking about. It would be pretty insane that over 30 years of research, he couldn't come up with a significantly better algorithm (currently sm18).

I'm a huge fan of spaced repetition.

I base my tutoring practice around it, and I've developed an app around it [1]. I actually encourage my students (and my app users) to use spaced repetition not just for facts, but for processes as well, like coding or mathematics. Going back to old, difficult processes and repeating them has much the same beneficial effects as doing so for facts.

One thing that Gwern doesn't mention, but that I've been very cognizant of, is how hard it is to keep up spaced repetition over a long period of time. That's a big part of why I've implemented streaks and rewards in my app, and also allowed my app users to review things on their own schedule or follow the algorithm. Just forcing users to follow the algorithm continuously tends to result in very high dropout rates.

[1] https://get21stnight.com/

Another issue with spaced repetition is that it's not a teaching method - you just can't throw a pile of random quiz cards on people and tell them to keep flipping them until they get the answer right. That's the principle that too many language learning websites out there try to utilize.

If you haven't seen any of his posts before, gwern is a treasure. He delves so, so deeply into topics that he finds interesting. A modern Library of Alexandria.

I posted this because I once again became interested in learning more about how we learn and when Gwern has covered a subject it is as good or better than having a wikipedia entry on the subject. Some of the stuff described here will find its way into the music tutoring software, which is progressing nicely.

> Gwern has covered a subject it is as good or better than having a wikipedia entry on the subject.

I wish that Gwern were superhuman and could write his own entire Wikipedia. Imagine what an incredible reference that would be!

I'd pay for that. His treatment of subjects is so exhaustive and well researched that it puts any paid encyclopedia to shame.

FWIW, he has a Patreon: https://www.patreon.com/gwern

Good one! Thank you for pointing that out. Hm, ok I looked at that page, it requires me to sign up for a membership. I will contact him out-of-band and ask for his IBAN so I can wire him money directly.

Spaced repetition is working. I've been a fan since I started using Anki to learn my Czech language. I also made a similar system a part of my knowledge management system (https://rekowl.com/) to keep thing fresh in my head

I’ve got several spaced repetition links here:

https://github.com/melling/LanguageLearning/blob/master/READ...

I’ve been developing iOS language apps for many years. A recent one attempts to simply teach 100 basic words.

https://apps.apple.com/us/app/hundred-words/id1469449237

I don’t use spaced repetition but have considered it. I’m not sure if that’s more worthwhile than gamifying with a better reward system.

Anyway, I’m looking for ideas on how to make better educational apps. Thanks.

I'm developing an app around spaced repetition as well, for couples to learn each other's languages. I made the spaced repetition "multiplayer", so the native speaking partner can create appropriate flashcards or check in on the learner, or the learner can ask help from the partner.

As some have mentioned, and what I went through, is spaced repetition is as good as the content, and often I made cards for learning Chinese that were like...ancient-style speaking Chinese. And then for things like languages, spaced repetition can only be a foundation, but needs to be augmented by real world practice.

https://learncoupling.com

I find Readwise does a good job of using spaced repetition, when it comes to my reading across various platforms. Additionally, it integrates with all the popular note taking apps so that you can create a knowledge repository of the most valuable insights.

Maybe it will be interesting to someone. We are developing the Studylib New Tab extension, that helps to learn flashcards without long and tedious learning sessions. It shows flashcards on the Google Chrome New Tab screen and you can adjust the delay between sessions. Another advantage is that you can't forget to review your current study task. Here is the link: https://chrome.google.com/webstore/detail/studylib-new-tab-s...

I've been using spaced repetition via anki for about a year now. Two effects that I've run into:

1.) When there's a deadline looming i tend to drop doing it, and then after the deadline i have a bunch to catch up on.

2.) I find somewhat frequently that some of the things i learned that I thought would be good to know in the future, i no longer have a use for. It's easy enough to stop reviewing those bits, but I've wondered what kind of impact that has.

While I can't address the second point, the first point is something that I've run into a lot.

In my own spaced repetition app [1], I simply space out the amount that people have to catch up on if they develop a huge backlog. For anki, my best advice would be to take this opportunity to delete some of the cards that you really don't need, especially if you're just doing this for fun. Anki tends to be very conservative with how long it makes you review cards, anyways.

[1] https://get21stnight.com/

One feature I wish SR software had was a tool to let you know if you were going to make a "bad card". There's a definite skill curve in making material at first. You can't tell until you've repeated a card a few times though that it was a bad card. By that time it's easily been a month.

Are there any spaced repetition apps that are:

1. Not Anki (it would be quite boring of you to ask...)

2. Free (Free as in no optional features that actually cost money)

3. Easy to use

I use Anki, but here are some alternatives.

Mnemosyne: https://mnemosyne-proj.org/

Drill mode: Not great for 3 unless you are already an orgmode user, but Drill-mode https://orgmode.org/worg/org-contrib/org-drill.html is an emacs plug for spaced repetition.

I few years back I co-founded Memorang[0] to do exactly this. While I'm no longer affiliated with the company, it technically is a free solution, albeit geared mainly towards medical students.

[0] https://www.memorangapp.com/flashcards/108640/Star+Wars+Spac...

We built https://traverse.link/ for this. Markdown based, take notes as you go, turn the ones you want to remember into flashcards with the click of a button

What other SRS apps are people using here? I've used Anki, but I dislike the GUI interface (especially the way it weirdly formats copied text) and would prefer a Linux friendly, more text-based system. Any suggestions?

I use Anki, but here are some alternatives.

Mnemosyne: https://mnemosyne-proj.org/

Drill mode: https://orgmode.org/worg/org-contrib/org-drill.html is an emacs plug for spaced repetition.

This is a little R package I wrote for doing spaced repetition in R: https://www.dontyouremember.com/

Call me old fashioned, but I still just use note cards. Obviously portability is an issue, but I don't travel much (1 long trip a year pre-covid), so never ends up being a real problem.

Note cards are great, but I think a really big factor with a good spaced repetition system is to be able to essentially pick the best card to show you at any time. Based upon your history, which card you are most likely to have forgotten and need a refresher on, and would be most beneficial to show you next.

This would be practically impossible with a physical system. In addition to the fact that your memory of a new cards fades faster than cards you saw a long time ago; taking the time to approximate such a system would be a large burden that software can just handle for you.

That's what I used when I was learning Mandarin Chinese in college, pre-smartphones.

But I did my own version of spaced repetition -- when a card was really hard, I'd put it back in the deck just 5 cards back. If it was easy, I'd put it all the way in the back of my ~200 cards, or eventually remove it entirely. Or wherever it was in between, I'd slot it at an appropriate distance. It actually worked really well.

I did something similar for both Spanish and Latin. I'd grab 20-50 cards to study from the top of the stack, work through them and repeat if needed. Then shift them to the back. The very easy ones ("ser" or "ir", for example because they're seen so much) would get moved out of the deck entirely, and only ever seen again for a full-deck review. New terms or concepts (conjugating for different tenses, for example) would go on top. I ended up with 1k cards in Latin over two semesters. Not sure how many for Spanish.

Reply on news.ycombinator.com