[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

A Neural Mechanism for Reward Discounting: Insights from Modeling Hippocampal–Striatal Interactions

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Decision-making often requires taking into consideration immediate gains as well as delayed rewards. Studies of behavior have established that anticipated rewards are discounted according to a decreasing hyperbolic function. Although mathematical explanations for reward delay discounting have been offered, little has been proposed in terms of neural network mechanisms underlying discounting. There has been much recent interest in the potential role of the hippocampus. Here, we demonstrate that a previously established neural network model of hippocampal region CA3 contains a mechanism that could explain discounting in downstream reward-prediction systems (e.g., basal ganglia). As part of its normal function, the model forms codes for stimuli that are similar to future predicted stimuli. This similarity provides a means for reward predictions associated with future stimuli to influence current decision-making. Simulations show that this “predictive similarity” decreases as the stimuli are separated in time, at a rate that is consistent with hyperbolic discounting.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Abbott LF, Blum KI. Functional significance of long-term potentiation for sequence learning and prediction. Cereb Cortex. 1996;6:406–16.

    Article  PubMed  CAS  Google Scholar 

  2. Addis DR, Cheng T, Roberts RP, Schacter DL. Hippocampal contributions to the episodic simulation of specific and general future events. Hippocampus .2011;21;1045–52.

    Article  PubMed  Google Scholar 

  3. August DA, Levy WB. Temporal sequence compression by an integrate-and-fire model of hippocampal area CA3. J Comput Neurosci. 1999;6:71–90.

    Article  PubMed  CAS  Google Scholar 

  4. Bamford NS, Zhang H, Schmitz Y, Wu N-P, Cepeda C, Levine MS, Schmauss C, Zakharenko SS, Zablow L, Sulzer D. Heterosynaptic dopamine neurotransmission selects sets of corticostriatal terminals. Neuron. 2004;42:653–63.

    Article  PubMed  CAS  Google Scholar 

  5. Bi GQ, Poo MM. Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J Neurosci. 1998;18:10464–72.

    PubMed  CAS  Google Scholar 

  6. Cheung THC, Cardinal RN. Hippocampal lesions facilitate instrumental learning with delayed reinforcement but induce impulsive choice in rats. BMC Neurosci. 2005;6:36.

    Article  PubMed  Google Scholar 

  7. Figner B, Knoch D, Johnson EJ, Krosch AR, Lisanby SH, Fehr E, Weber EU. Lateral prefrontal cortex and self-control in intertemporal choice. Nat Neurosci. 2010;13:538–9.

    Article  PubMed  CAS  Google Scholar 

  8. Gamboz N, Brandimonte MA, De Vito S. The role of past in the simulation of autobiographical future episodes. Exp Psychol. 2010;57:419–28.

    Article  PubMed  Google Scholar 

  9. Green L, Myerson J, Ostaszewski P. Discounting of delayed rewards across the life span: age differences in individual discounting functions. Behav Process. 1999;46:89–96.

    Article  Google Scholar 

  10. Gupta R, Duff M, Denburg N, Cohen N, Bechara A, Tranel D. Declarative memory is critical for sustained advantageous complex decision-making. Neuropsychologia. 2009;47:1686–93.

    Article  PubMed  Google Scholar 

  11. Hare TA, Camerer CF, Rangel A. Self-control in decision-making involves modulation of the vmpfc valuation system. Science. 2009;324:646–8.

    Article  PubMed  CAS  Google Scholar 

  12. Hassabis D, Kumaran D, Vann SD, Maguire EA. Patients with hippocampal amnesia cannot imagine new experiences. Proc Natl Acad Sci USA. 2007;104:1726–31.

    Article  PubMed  CAS  Google Scholar 

  13. Hollerman JR, Tremblay L, Schultz W. Influence of reward expectation on behavior-related neuronal activity in primate striatum. J Neurophysiol. 1998;80:947–63.

    PubMed  CAS  Google Scholar 

  14. Joel D, Niv Y, Ruppin E. Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw. 2002;15:535–47.

    Article  PubMed  Google Scholar 

  15. Johnson A, van der Meer MAA, Redish AD. Integrating hippocampus and striatum in decision-making. Curr Opin Neurobiol. 2007;17:692–7.

    Article  PubMed  CAS  Google Scholar 

  16. Johnson A, Redish AD. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J Neurosci. 2007;27:12176–89.

    Article  PubMed  CAS  Google Scholar 

  17. Kable JW, Glimcher PW. The neural correlates of subjective value during intertemporal choice. Nat Neurosci. 2007;10:1625–33.

    Article  PubMed  CAS  Google Scholar 

  18. Kable JW, Glimcher PW. An "as soon as possible" effect in human intertemporal decision making: behavioral evidence and neural mechanisms. J Neurophysiol. 2010;103:2513–31.

    Article  PubMed  Google Scholar 

  19. Kacelnik A. Normative and descriptive models of decision making: time discounting and risk sensitivity. Ciba Found Symp. 1997;208:51–67; discussion 67–70.

    PubMed  CAS  Google Scholar 

  20. Kelley A, Domesick V. The distribution of the projection from the hippocampal formation to the nucleus accumbens in the rat: an anterograde and retrograde-horseradish peroxidase study. Neuroscience. 1982;7:2321–35.

    Article  PubMed  CAS  Google Scholar 

  21. Lansink CS, Goltstein PM, Lankelma JV, McNaughton BL, Pennartz CMA. Hippocampus leads ventral striatum in replay of place-reward information. PLoS Biol. 2009; 7:e1000173.

    Article  PubMed  Google Scholar 

  22. Levy W. A computational approach to hippocampal function. Comput Model Learn Simple Neural Syst. 1989;23:243–305.

    Article  Google Scholar 

  23. Levy WB. A sequence predicting CA3 is a flexible associator that learns and uses context to solve hippocampal-like tasks. Hippocampus. 1996;6:579–90.

    Article  PubMed  CAS  Google Scholar 

  24. Levy WB, Sanyal A, Rodriguez P, Sullivan DW, Wu XB. The formation of neural codes in the hippocampus: trace conditioning as a prototypical paradigm for studying the random recoding hypothesis. Biol Cybern. 2005;92:409–26.

    Article  PubMed  CAS  Google Scholar 

  25. Levy WB, Steward O. Temporal contiguity requirements for long-term associative potentiation/depression in the hippocampus. Neuroscience. 1983;8:791–7.

    Article  PubMed  CAS  Google Scholar 

  26. Liljenström H, Wu XB. Noise-enhanced performance in a cortical associative memory model. Int J Neural Syst. 1995;6:19–29.

    Article  PubMed  Google Scholar 

  27. Mariano TY, Bannerman DM, McHugh SB, Preston TJ, Rudebeck PH, Rudebeck SR, Rawlins JNP, Walton ME, Rushworth MFS, Baxter MG, Campbell TG. Impulsive choice in hippocampal but not orbitofrontal cortex-lesioned rats on a nonspatial decision-making maze task. Eur J Neurosci. 2009;30:472–84.

    Article  PubMed  CAS  Google Scholar 

  28. Markram H, Lübke J, Frotscher M, Sakmann B. Regulation of synaptic efficacy by coincidence of postsynaptic aps and epsps. Science. 1997;275:213–5.

    Article  PubMed  CAS  Google Scholar 

  29. Mazur J. An adjusting procedure for studying delayed reinforcement. Quant Anal Behav Eff Delay Interv Events Reinf Value. 1987;5:55.

    Google Scholar 

  30. Mazur JE, Biondi DR. Delay-amount tradeoffs in choices by pigeons and rats: hyperbolic versus exponential discounting. J Exp Anal Behav. 2009;91:197–211.

    Article  PubMed  Google Scholar 

  31. McClure SM, Ericson KM, Laibson DI, Loewenstein G, Cohen JD. Time discounting for primary rewards. J Neurosci. 2007;27:5796–804.

    Article  PubMed  CAS  Google Scholar 

  32. McClure SM, Laibson DI, Loewenstein G, Cohen JD. Separate neural systems value immediate and delayed monetary rewards. Science. 2004;306:503–7.

    Article  PubMed  CAS  Google Scholar 

  33. Mehta M, Barnes C, McNaughton B. Experience-dependent, asymmetric expansion of hippocampal place fields. Proc Natl Acad Sci USA. 1997;94:8918.

    Article  PubMed  CAS  Google Scholar 

  34. Mehta M, Quirk M, Wilson M. Experience-dependent asymmetric shape of hippocampal receptive fields. Neuron. 2000;25:707–15.

    Article  PubMed  CAS  Google Scholar 

  35. Mitman K, Laurent P, Levy W. Defining time in a minimal hippocampal CA3 model by matching time-span of associative synaptic modification and input pattern duration. In: Neural networks, 2003. Proceedings of the international joint conference on 2003. p. 1631–1636 IEEE vol 3.

  36. Monaco J, Levy W. T-maze training of a recurrent CA3 model reveals the necessity of novelty-based modulation of LTP in hippocampal region CA3. In Neural networks, 2003. Proceedings of the international joint conference on 2003. p. 1655–1660 IEEE vol 3.

  37. Muzzio IA, Levita L, Kulkarni J, Monaco J, Kentros C, Stead M, Abbott LF, Kandel ER. Attention enhances the retrieval and stability of visuospatial and olfactory representations in the dorsal hippocampus. PLoS Biol. 2009;7:e1000140.

    Article  PubMed  Google Scholar 

  38. Pennartz CMA, Lee E, Verheul J, Lipa P, Barnes CA, McNaughton BL. The ventral striatum in off-line processing: ensemble reactivation during sleep and modulation by hippocampal ripples. J Neurosci. 2004;24:6446–56.

    Article  PubMed  CAS  Google Scholar 

  39. Peters J, Büchel C. Episodic future thinking reduces reward delay discounting through an enhancement of prefrontal-mediotemporal interactions. Neuron. 2010;66:138–48.

    Article  PubMed  CAS  Google Scholar 

  40. Polyn S, Wu X, Levy W. Entorhinal/dentate excitation of CA3: a critical variable in hippocampal models. Neurocomputing. 2000;32:493–499.

    Article  Google Scholar 

  41. Rawlins JN, Feldon J, Butt S. The effects of delaying reward on choice preference in rats with hippocampal or selective septal lesions. Behav Brain Res. 1985;15:191–203.

    Article  PubMed  CAS  Google Scholar 

  42. Reynolds JNJ, Wickens JR. Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 2002;15:507–21.

    Article  PubMed  Google Scholar 

  43. Schacter DL, Addis DR. On the nature of medial temporal lobe contributions to the constructive simulation of future events. Philos Trans R Soc Lond B Biol Sci. 2009;364:1245–53.

    Article  PubMed  Google Scholar 

  44. Schultz W, Dayan P, Montague PR. A neural substrate of prediction and reward. Science. 1997;275:1593–9.

    Article  PubMed  CAS  Google Scholar 

  45. Shadmehr R, Orban de Xivry JJ, Xu-Wilson M, Shih T-Y. Temporal discounting of reward and the cost of time in motor control. J Neurosci. 2010;30:10507–16.

    Article  PubMed  CAS  Google Scholar 

  46. Shon AP, Wu XB, Sullivan DW, Levy WB. Initial state randomness improves sequence learning in a model hippocampal network. Phys Rev E Stat Nonlin Soft Matter Phys. 2002;65:031914.

    Article  PubMed  CAS  Google Scholar 

  47. Sompolinsky H, Kanter I. Temporal association in asymmetric neural networks. Phys Rev Lett. 1986;57:2861–2864.

    Article  PubMed  Google Scholar 

  48. Sullivan DW, Levy WB. Quantal synaptic failures enhance performance in a minimal hippocampal model. Network. 2004;15:45–67.

    Article  PubMed  CAS  Google Scholar 

  49. Suri RE, Schultz W. Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Exp Brain Res. 1998;121:350–4.

    Article  PubMed  CAS  Google Scholar 

  50. Sutton RS, Barto AG. Reinforcement learning: an introduction. IEEE Trans Neural Netw. 1998;9:1054.

    Article  Google Scholar 

  51. Wallenstein GV, Hasselmo ME. Gabaergic modulation of hippocampal population activity: sequence learning, place field development, and the phase precession effect. J Neurophysiol. 1997;78:393–408.

    PubMed  CAS  Google Scholar 

  52. Wu X, Baxter RA, Levy WB. Context codes and the effect of noisy learning on a simplified hippocampal CA3 model. Biol Cybern. 1996;74:159–65.

    Article  PubMed  CAS  Google Scholar 

  53. Wu X, Liljenstrom H. Regulating the nonlinear dynamics of olfactory cortex. Netw Comput Neural Syst. 1994;5:47–60.

    Article  Google Scholar 

Download references

Acknowledgments

I thank Chip Levy, Joe Monaco, Sean Polyn, Steve Yantis, Kechen Zhang, and the anonymous reviewers for helpful discussions and comments on an earlier version of this manuscript. This work was supported by NIH grant R01-DA013165 to Steve Yantis.

Author information

Authors and Affiliations

  1. Department of Psychological and Brain Sciences, The Johns Hopkins University, 136 Ames Hall, 3400 N. Charles St., Baltimore, MD, 21218, USA

    Patryk A. Laurent

Authors
  1. Patryk A. Laurent
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Patryk A. Laurent.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Laurent, P.A. A Neural Mechanism for Reward Discounting: Insights from Modeling Hippocampal–Striatal Interactions. Cogn Comput 5, 152–160 (2013). https://doi.org/10.1007/s12559-012-9178-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-012-9178-8

Keywords

Search

Navigation