But, a tendency to more often select a particular response would then lead to negative ε, even if subjects might, in the smaller proportion of exploratory trials, be more likely to explore uncertain actions. In contrast, the RT swing analysis permits examining the degree to which trial-to-trial variations are accounted for by the exploration term in the model as a function of relative uncertainty

and fitted ε. The use of a continuous RT allows us to detect not only when RTs change toward the direction of greater uncertainty, but the degree of that change and its correlation with the degree of relative uncertainty. This analysis is consistent with our observation that explorers continued to be fit by positive ε even in the simulations based on categorical responses—meaning high throughput screening that when sufficiently uncertain they were more likely to shift qualitatively from a slow to a fast response or vice-versa, rather than only make small RT adjustments within a response class. Second, as noted above, we used a task with static reward contingencies check details within a block, but changing contingencies between blocks, to estimate the effect of uncertainty given the history

of action-outcome samples without the additional complication of participants’ perceptions and beliefs about how rapidly contingencies are changing within blocks. Third, because it is difficult to integrate both frequency and magnitude for different RTs to compute expected value within a block, subjects cannot explicitly discover the programmed expected value functions (and hence behavior is suboptimal). Combining variation in both frequency and magnitude encourages subjects to sample the space of RTs to determine whether they might do better. Fifteen (eight female) right-handed adults (age 18–27, mean 20) with normal or corrected-to-normal vision and free of psychiatric and neurological conditions, contraindications for MRI, and medication

affecting the central nervous system were recruited. Participants gave written informed consent and were compensated for participation according to guidelines established and approved by the Research Thiamine-diphosphate kinase Protections Office of Brown University. Participants were paid $15/hr for their time. In order to investigate explore/exploit decisions, we employed a task used previously (Frank et al., 2009 and Moustafa et al., 2008) to study the influence of relative uncertainty on exploratory judgments. The task is a variant of the basic paradigm used to study exploration, in that multiple response options are available with different expected values that are known with different degrees of certainty based on previous sampling. The participants attempt to select responses that maximize their reward. Importantly, however, the present task separates learning into individual blocks within which the expected values of the different response options remain constant.