1. Answer to post 1

In the previous post, I reviewed the Rescorla-Wagner updating (Delta) rule and its contemporary instantiation. At the end, I asked the following question:

• How should you change the learning rate so that the expected win rate is always the average of all past outcomes?

We will go over the answer to this question before progressing to the use of the Delta rule in modeling human choice. To begin, refer back to the Delta rule written in the following form:

\[Pr(win)_{t+1} = (1 - \beta) \cdot Pr(win)_{t} + \beta \cdot \lambda_{t}\]

Here, we see that in the Delta rule the expected win probability for the next trial is equal to the exponentially weighted moving average of the past expectation and the current outcome. It is easy to show this through a visualization of the expectation over time. For example, imagine that we have a vector of outcomes \(\lambda = [1,0,0,1,1,1,0,1,1,1]\), where 0 and 1 represent losing and winning slot machine rolls, respectively. Note that in this example, the placement of these outcomes within the vector \(\lambda\) indicates their temporal order (i.e. \(\lambda_{1}=1\), \(\lambda_{2}=0\), etc.). Now, if we set an arbitrary learning rate such as \(\beta = 0.05\), what is the expected win rate after iterating through outcomes \(\lambda\)? The R code below demonstrates the use of the Delta rule and an alternative exponential weighting scheme–which takes the form of a power series–to determine the expectation on each trial:

# For pretty plots
library(ggplot2)

# Assign lambda (lambda[1] == first trial)
lambda <- c(1,0,0,1,1,1,0,1,1,1)

# Set learning rate
beta <- 0.05

### Iterative prediction error (Delta rule) approach ###

# Function that iterates the Rescorla-Wagner rule 
  # This function is slightly modified from the last post
  # to ensure that that final expectation is stored
rw_update <- function(lambda, beta, init) {
  # To store expected value for each trial
  Pr_win <- vector(length=length(lambda)+1)
  # Set initial value
  Pr_win[1] <- init
  for (t in 1:(length(lambda))) {
    Pr_win[t+1] <- Pr_win[t] + beta * (lambda[t] - Pr_win[t])
  }
  return(Pr_win)
}

# With initial expectation = 0, iterate Delta rule
delta_results <- rw_update(lambda = lambda, 
                           beta   = beta, 
                           init   = 0)[-1]
                          #             ^
                          # Remove initial value (0)

### Power series approach ###

# Direct exponential weighting (saving all expectations)
power_ser <- NULL
for (i in 1:10) {
  power_ser[i] <- beta * sum((1-beta)^(0:(i-1))*lambda[i:1])
}

### Comparison of both approaches ###
all(round(delta_results, 8) == round(power_ser, 8))

## [1] TRUE

# data.frame for ggplot
all_data <- stack(data.frame(delta = delta_results,
                             pow_s = power_ser))
# Add trial 
all_data[["trial"]] <- rep(1:10, 2)
names(all_data)[2] <- "Approach"

# Visually
p <- ggplot(all_data, aes(x = trial, y = values, color = Approach)) + 
  geom_line() +
  facet_grid(facets = "Approach ~ .") + 
  ggtitle("Comparison of approaches") +
  xlab("Trial Number") +
  ylab("Expected Win Probability")
p

As you can see in the plots, both the Delta rule and the power series approach yield the same exact expectations when iterated for each trial. However, the power series form requires each past observation while the Delta rule only requires the last expectation–this feature makes the Delta rule form of the equation much more plausible as a processes that people may use to estimate the value of a choice. This is because the computational cost does not increase with the number of past observations.

Through this example, those familiar with economics or signal processing may find the Delta rule familiar. Essentially, we can think of the Delta rule as a smoothing function or a high- or low-pass filter–albeit in the time as opposed to frequency domain–which effectively attenuates the effect of past or current outcomes, respectively. What makes this specific form interesting is again the fact that it can be iterated (i.e. it is recursive), making it a realistic approximation to the computations performed by the brain when estimating some value.

With the above intuitions in mind, we will now get back to the question. How do we change the learning rate to ensure that the current expectation is always the simple average of all past outcomes? Since the above example showed that the Delta rule is really just a moving average where past outcomes are given exponentially decreasing weights, our goal is to make all outcomes equally represented. In other words, we want to weight past and current outcomes equally–this is a cumulative moving average. Using our slot machine example, the cumulative moving average formula (in its most common form) is written as follows:

\[Pr(win)_{t+1} = \frac{\lambda_{t} + (t-1) \cdot Pr(win)_{t}}{t}\]

We can re-write the above equation into one that is more familiar to us:

\[Pr(win)_{t+1} = (1-\frac{1}{t}) \cdot Pr(win)_{t} + \frac{1}{t} \cdot \lambda_{t} \]

In the above form, you can see that the cumulative moving average can be computed using the Delta rule by setting the learning rate to \(\frac{1}{t}\), where \(t\) is the trial number. Looking at the equation, you will notice that as \(t\) increases, the weight \((1-\frac{1}{t})\) placed on the past probability estimate \(Pr(win)_{t}\) becomes larger while the weight \(\frac{1}{t}\) on the current outcome \(\lambda_{t}\) shrinks. This behavior ensures that past outcomes are not discounted at a higher rate than current ones.

2. Choice mechanisms

Figure 1

With the Delta rule, we can approximate how people with a certain learning rate may update their expectations about an outcome on a trial-by-trial basis, but how does this translate to choice? In the slot machine example, we only had a single choice (i.e. pull the lever and see if you win), so this question never applied to us. But what if we have 2 slot machines and we want to select the one that will win most frequently? In this case, we can use the Delta rule to update win probability expectations for each slot machine separately, but what do we do with these values after they are computed? Below, I will describe three different methods that can be used to translate expected values to choice.

3. Implementation

We now have a full model describing each of the following steps:

• Evaluating an outcome,
• Updating previous representations of choice options, and
• Generating a probability of selecting each choice on the next trial.

This model is simple, but it provides the basic building blocks for more complex models that are found in neuroscience, cognitive science, and decision making literature today. In the next post, we will explore various methods which can be used to estimate the free parameters in the model (e.g. the learning rate) when all we have are the person’s choices.