Sample efficiency
No human learns by reading all the text available on the internet. However, we can learn new knowledge and skills efficiently and be adaptable. Modern LLMs, on the other hand, need massive amounts of training data to match human performance on many tasks. How is it that humans appear to be so much more sample efficient? I believe this is a nuanced question. Naively, we might say this implies that humans must have a much better learning algorithm than what we use to train LLMs (backpropagation, optimisers, architecture, and so on). However, there are many counters to this point.
Firstly, the human brain has evolved over millions of years. How can we account for the amount of “data” that was involved in the evolutionary process? This might be a valid point for the types of tasks animals and humans have been doing for a long time – for vision or locomotion, say, we might just have inbuilt circuits encoded in our genome that are much better suited to these tasks. However, I don't think the argument follows for tasks humans have only more recently been doing, such as mathematics or coding. Evolution is not that fast! Of course, it might have indirectly learnt better algorithms which lead to sample efficiency and better generalisation. There is also another interesting point with regard to the genome: a full human genome can contain up to around 200 GB of information. The human brain, however, contains 86 billion neurons with 100 trillion connections – around two orders of magnitude larger than the number of bits stored in the genome. So, many neural circuits must be either very compressible or learnt after birth.
Furthermore, perhaps comparing the amount of text data a human comes across with how much an LLM ingests underplays the sample efficiency and generalisation capabilities of the models. For example, LLMs exhibit the emergent phenomenon of in-context learning. Here, with very few examples, a model can pick up a new skill or task – something that would be unimaginable for learning via gradient descent. Nonetheless – even though evolution might have played a role in creating circuits for primal skills, and even though LLMs might exhibit generalisation capabilities beyond what was imaginable a decade or two ago – I believe there is still a big gap between the learning and generalisation capabilities displayed by humans and by LLMs. It is hard to argue against the fact that even for newer tasks, for which evolution could not have adapted so fast, humans pick things up quicker. And the size comparison between the human brain and the genome, alongside mountains of neuroscience and psychology literature, points to some learning at scale really happening after birth.
So, back to the question – what is the difference? Well, I think we still have no idea. But if I were to guess, I don't think it is solely to do with a different algorithm at the neuron scale (as opposed to backpropagation). One direction research in increasing sample efficiency for LLMs has taken is to make the most of the learning signal available. Consider reinforcement learning: currently, many setups solely make use of a scalar reward value at the end of a rollout as the learning signal. But maybe one reason humans can learn so efficiently is that they already have common sense and a good understanding of the world. Can we inject the understanding LLMs have about the world to make them learn more efficiently? This is the key idea behind on-policy self-distillation.
The method goes as follows: we give the model a question and let it attempt a solution. The model then receives some feedback from the environment – for coding, this might be a compilation error or a test case that didn't pass. The model is then prompted again with its previous attempt, as well as the feedback received, so it can try to fix its mistake. The training objective is to match the log probabilities of the output the model produced with the feedback, with only the initial question. This way, the model's existing understanding of the world can be used to get a richer learning signal: by incorporating the feedback, the model can learn from the signal of where its attempt went wrong, not solely that it was. There are benefits to this method being on-policy beyond the scalability of not needing a strong teacher. Recent work has demonstrated that self-distillation can be beneficial for continual learning: by letting the model generate its own outputs on-policy, the update ends up being “closer” to the base model, retaining more of its existing capabilities.
Now, all of this sounds very exciting – but what does it work well for? This area is still very new and much is still being figured out, but one place it seems to shine is coding, with environment feedback from the compiler, interpreter or test cases. However, one field where there is perhaps an even bigger gap between human and model learning than in LLMs is robotics. Could self-distillation techniques be useful here? There is already work to suggest that when working with VLAs, providing feedback via text for on-policy self-distillation does not create a meaningful difference. But why? Firstly, I think text is not precise enough for robotics – and the way VLAs are trained perhaps doesn't include much very specific textual feedback, but rather general instructions. I can describe to you in text all about how to shoot a basketball like Michael Jordan, but none of my words can compare to actually watching it live. And of course, learning to shoot by watching someone cannot substitute for actually trying it yourself! But perhaps that is a different story.
Nonetheless, I think that in order to incorporate the ideas of on-policy self-distillation into VLAs, many changes would have to be made. Firstly, the context and memory of the model. For LLMs, the model can keep its previous attempt, the feedback and its current attempt in context all at once. I believe this sort of memory is required to achieve the rich-signal learning that self-distillation offers – maybe the previous attempt and feedback can somehow be compressed, but it is not obvious how. Perhaps even more crucially, for this to work we need true in-context learning from robotics foundation models. Although recent models such as π0.7 have increasing amounts of memory and emergent generalisation and learning capabilities, older open-source models don't seem suitable at all. Secondly, I believe both the memory of the observation and the correction have to come from the visual domain. In short, current publicly released models don't seem ready for the on-policy self-distillation techniques used for LLMs – but as models develop emergent in-context learning capabilities and longer contexts for memory, this could become interesting for robots as well.
Although much of this post has been about on-policy self-distillation, I still don't think we have found what is at the heart of the sample-efficiency difference between humans and LLMs. But perhaps we don't need to – after all, the AI models of today are nothing like humans, and the capabilities they gain might be both completely different and obtained via unrelated mechanisms.