NTT Research, Inc., a division of NTT yesterday announced that a paper by Physics & Informatics (PHI) Lab Scientist Dr. Gautam Reddy, which makes a novel contribution to the understanding of learning, has been published in the Proceedings of the National Academy of Sciences (PNAS), one of the world’s most cited and comprehensive multidisciplinary scientific journals. The paper, titled “A reinforcement-based mechanism for discontinuous learning,” uses data on spatial navigation of labyrinths by laboratory mice to provide a simple explanation for the “a-ha” moment of sudden insight. Dr. Reddy shows that the outcome of reinforcement learning (RL) algorithms, which operate gradually, can appear discontinuous for certain kinds of tasks. Specifically, this is true when the artificial RL agent learns the direct path from point A to point B in certain structured environments.
Labyrinths or mazes have a similar complex structure to knowledge graphs, which represent how humans mentally explore and navigate a knowledge database. A recent paper by neuroscientists at Caltech describes how mice navigating through a maze to find water show “rapid learning, sudden insight and efficient exploration.” Dr. Reddy analyzes the data in that paper and produces a learning rule to explain the phenomenon of sudden insight. He quantifies the relationship between learning rule, derived from biologically based learning discontinuities; learning curve, which represents the speed of reinforcement waves; and exploration bias, or the probability that a mouse will take a certain action at a certain intersection. The paper’s novel contribution is that RL algorithms can reproduce “a-ha” moment discontinuities in simulations.
“Congratulations to Dr. Reddy on his publication in this prestigious journal and for his creative discovery,” PHI Lab Director Yoshihisa Yamamoto said. “His important findings will advance our understanding of how we reason, solve problems and gain insight.”
Long-standing RL algorithms incrementally reinforce rewarding actions through accumulated experience. Yet, past and recent experiments find sharp discontinuities in learning. Dr. Reddy sheds light on the underlying mechanism.
“We show that the nonlinear dynamics of RL-based learning, together with continuous exploration, lead to discontinuous learning curves in tree-like structured environments,” Dr. Reddy said. “We develop a quantitative theory which explains the origin and highlights the generality of the phenomenon. The theory explains existing data and provides specific testable predictions.”
Looking ahead, Dr. Reddy plans to extend the experimental framework beyond tree-like graphs to topologies with loops. He also hopes to examine how an animal learns the neural representation of its location in a maze, a prerequisite to its evaluating differing actions at a particular location.
Along with these research initiatives, Dr. Reddy also serves as the NTT Research PHI Lab lead on a five-year joint research project with scientists at Harvard University to study animal neuro-responses, in the hope of informing future artificial intelligence (AI) systems. The PHI Lab’s mission is to uncover fundamental principles and novel technologies that advance information processing beyond state of the art. Its members and partners explore the space between quantum information science, neuroscience and optical technologies. In October 2020, PHI Lab Director Yamamoto co-authored a paper in Applied Physics Letters (APL) titled, “Coherent Ising Machines: Quantum optics and neural network perspectives,” which underscored the interdisciplinary nature of the PHI Lab’s path-breaking research agenda.
In addition to Harvard, eight other universities have agreed to conduct joint research with the PHI Lab. These include the California Institute of Technology (Caltech), Cornell University, Massachusetts Institute of Technology (MIT), Notre Dame University, Stanford University, Swinburne University of Technology, the Tokyo Institute of Technology and the University of Michigan. The NASA Ames Research Centre in Silicon Valley has also entered into a joint research agreement with the PHI Lab.
For companies dealing with research and healthcare, please visit:
Cognizant Launches Advanced AI Lab to Accelerate AI Research and Innovation
Nordic Bioscience’s precision medicine biomarker PRO-C3 available in China for Research Use
