We assume we predict forward in time
To decide how to act requires predicting how our actions lead to desired events. These predictions comprise a so-called model of the world, which we use to simulate how we act in the world, and what would occur as a result of our actions. Our use of world models to plan some argue distinguishes human from current machine intelligences.
A dominant assumption in work on world models is that we predict events forward in time – from current actions to future goals. This assumption, however, seems to be defied by everyday experience. It appears obvious, and has been observed in human self-report, that we often plan by beginning at a future goal, and work our way back toward current options – that is, we predict backward. Despite such evidence, the science of prediction and decision-making lacks an informed principle which describes why it would benefit us to predict events in either a forward or backward direction. In our article published this week in Nature Human Behaviour, we offer a new computational model that demonstrates how particular environments make forward or backward prediction more efficient to utilize.
We assume we predict forward even though everyday experience tells us we can predict from backward.
A motivating thought experiment
A simple thought experiment motivated our work on backward prediction. Imagine solving a 2D maze written on a sheet of paper. If the maze has many dead-ends and only one route leading to the exit, tracing forward from the entrance towards the target location will terminate in many dead-ends. However, starting at the target location and tracing back toward the entrance will avoid these failures. You can think of each attempt at solving the maze like a prediction we would call to mind during planning. We thus set out to create a task like the maze example to show how predicting backward can make the planning process more efficient.
A new model of when to plan forward or backward
We first had to translate the aforementioned intuitive example into a testable hypothesis. To do so, we formulated a computational model of backward prediction, called a predecessor representation, to complement the existing model for forward prediction, called a successor representation. Using these models, we showed that fewer backward than forward predictions are required to solve tasks where future events outnumber past events (which we call “divergent”). By contrast, we also showed that fewer forward than backward predictions are required for the converse “convergent” task, where past events outnumber future events. It is thus more efficient and less computationally costly to use the strategy requiring less predictions.
The direction we predict events adaptively shifts
To test our hypothesis in human participants, we created a series of tasks that manipulated the ratio of past to future states. A “state” here was signified by a unique emoji (like a trident in the figure below) presented on a computer screen. The task involved subjects learning how emojis probabilistically led to new emojis over many trials. Because only backward but not forward prediction is sensitive to how likely a participant begins in a starting state emoji, we manipulated how likely participants started in certain emojis (“base-rates”) during learning.
Example of a divergent prediction task, where future states at the bottom outnumber past states at the top, and where backward prediction is more efficient than forward prediction.
After many learning trials, we asked participants to use their knowledge to select actions that were most likely to lead to emojis that were instructed to give the participant large rewards. These questions were carefully designed such that patterns of choices differed between participants that used forward or backward predictions to inform their choice. Participants got no feedback in these test trials, and had to plan based on the predictions they previously learned.
We showed robust evidence for our model across six studies. First, we found evidence that humans used backward prediction in a diverging environment in a simple prediction task, where an initial action at a starting emoji led to two succeeding emojis, one after the other. Next, we flipped the task around to make a converging design, and showed that participants switched to engaging forward prediction, just as our model predicted.
Example of a convergent prediction task, where past states at the top outnumber future states at the bottom, and where forward prediction is more efficient than backward prediction.
Finally, we ran modified versions of these two tasks, except subjects needed to engage two actions to reach a final goal. Our hypotheses were supported, reflecting that humans adaptively switch between forward and backward prediction in complex multi-action planning tasks.
The road ahead
Although our work establishes that backward prediction is used adaptively in human decision-making, there’s plenty of work ahead to understand its algorithmic implementation. Reassuringly, recent rodent neuroscience work has found that dopamine activity may implement backward prediction for causal learning (see here).
Future work should investigate whether we predict in both forward and backward directions simultaneously, which compete to influence decision-making. This might be useful when there’s uncertainty over how the environment is structured, which bears on which direction is most efficient for prediction. Relatedly, our particular implementation of backward prediction was actually not strictly accurate (although more efficient) because of a base-rate bias inherent to its computation. Future models of backward prediction, akin to those used in the aforementioned neuroscience study, should inform task development where backward prediction adjusts for base-rates. The challenge will be to create tasks that can distinguish base-rate-adjusted backward prediction from forward prediction, which might require models of more than just choice behavior (e.g., of neural activity).