Decoding a new neural machine interface for control of artificial limbs.
P. Zhou and M. M. Lowery and K. B. Englehart and H. Huang and G. Li and L. Hargrove and J. P. A. Dewald and T. A. Kuiken
An analysis of the motor control information content made available with a neural-machine interface (NMI) in four subjects is presented in this study. We have developed a novel NMI-called targeted muscle reinnervation (TMR)-to improve the function of artificial arms for amputees. TMR involves transferring the residual amputated nerves to nonfunctional muscles in amputees. The reinnervated muscles act as biological amplifiers of motor commands in the amputated nerves and the surface electromyogram (EMG) can be used to enhance control of a robotic arm. Although initial clinical success with TMR has been promising, the number of degrees of freedom of the robotic arm that can be controlled has been limited by the number of reinnervated muscle sites. In this study we assess how much control information can be extracted from reinnervated muscles using high-density surface EMG electrode arrays to record surface EMG signals over the reinnervated muscles. We then applied pattern classification techniques to the surface EMG signals. High accuracy was achieved in the classification of 16 intended arm, hand, and finger/thumb movements. Preliminary analyses of the required number of EMG channels and computational demands demonstrate clinical feasibility of these methods. This study indicates that TMR combined with pattern-recognition techniques has the potential to further improve the function of prosthetic limbs. In addition, the results demonstrate that the central motor control system is capable of eliciting complex efferent commands for a missing limb, in the absence of peripheral feedback and without retraining of the pathways involved.
Cortical modulations increase in early sessions with brain-machine interface.
M. Zacksenhouse and M. A. Lebedev and J. M. Carmena and J. E. O'Doherty and C. Henriquez and M. A. L. Nicolelis
BACKGROUND: During planning and execution of reaching movements, the activity of cortical motor neurons is modulated by a diversity of motor, sensory, and cognitive signals. Brain-machine interfaces (BMIs) extract part of these modulations to directly control artificial actuators. However, cortical modulations that emerge in the novel context of operating the BMI are poorly understood. METHODOLOGY/PRINCIPAL FINDINGS: Here we analyzed the changes in neuronal modulations that occurred in different cortical motor areas as monkeys learned to use a BMI to control reaching movements. Using spike-train analysis methods we demonstrate that the modulations of the firing-rates of cortical neurons increased abruptly after the monkeys started operating the BMI. Regression analysis revealed that these enhanced modulations were not correlated with the kinematics of the movement. The initial enhancement in firing rate modulations declined gradually with subsequent training in parallel with the improvement in behavioral performance. CONCLUSIONS/SIGNIFICANCE: We conclude that the enhanced modulations are related to computational tasks that are significant especially in novel motor contexts. Although the function and neuronal mechanism of the enhanced cortical modulations are open for further inquiries, we discuss their potential role in processing execution errors and representing corrective or explorative activity. These representations are expected to contribute to the formation of internal models of the external actuator and their decoding may facilitate BMI improvement.
Mixture of trajectory models for neural decoding of goal-directed movements.
B. M. Yu and C. Kemere and G. Santhanam and A. Afshar and S. I. Ryu and T. H. Meng and M. Sahani and K. V. Shenoy
Probabilistic decoding techniques have been used successfully to infer time-evolving physical state, such as arm trajectory or the path of a foraging rat, from neural data. A vital element of such decoders is the trajectory model, expressing knowledge about the statistical regularities of the movements. Unfortunately, trajectory models that both 1) accurately describe the movement statistics and 2) admit decoders with relatively low computational demands can be hard to construct. Simple models are computationally inexpensive, but often inaccurate. More complex models may gain accuracy, but at the expense of higher computational cost, hindering their use for real-time decoding. Here, we present a new general approach to defining trajectory models that simultaneously meets both requirements. The core idea is to combine simple trajectory models, each accurate within a limited regime of movement, in a probabilistic mixture of trajectory models (MTM). We demonstrate the utility of the approach by using an MTM decoder to infer goal-directed reaching movements to multiple discrete goals from multi-electrode neural data recorded in monkey motor and premotor cortex. Compared with decoders using simpler trajectory models, the MTM decoder reduced the decoding error by 38 (48) percent in two monkeys using 98 (99) units, without a necessary increase in running time. When available, prior information about the identity of the upcoming reach goal can be incorporated in a principled way, further reducing the decoding error by 20 (11) percent. Taken together, these advances should allow prosthetic cursors or limbs to be moved more accurately toward intended reach goals.
Signal acquisition and analysis for cortical control of neuroprosthetics.
S. I. H. Tillery and D. M. Taylor
Curr Opin Neurobiol
Work in cortically controlled neuroprosthetic systems has concentrated on decoding natural behaviors from neural activity, with the idea that if the behavior could be fully decoded it could be duplicated using an artificial system. Initial estimates from this approach suggested that a high-fidelity signal comprised of many hundreds of neurons would be required to control a neuroprosthetic system successfully. However, recent studies are showing hints that these systems can be controlled effectively using only a few tens of neurons. Attempting to decode the pre-existing relationship between neural activity and natural behavior is not nearly as important as choosing a decoding scheme that can be more readily deployed and trained to generate the desired actions of the artificial system. These artificial systems need not resemble or behave similarly to any natural biological system. Effective matching of discrete and continuous neural command signals to appropriately configured device functions will enable effective control of both natural and abstract artificial systems using compatible thought processes.
Instant neural control of a movement signal.
M. D. Serruya and N. G. Hatsopoulos and L. Paninski and M. R. Fellows and J. P. Donoghue
(2002 Mar 14)
The activity of motor cortex (MI) neurons conveys movement intent sufficiently well to be used as a control signal to operate artificial devices, but until now this has called for extensive training or has been confined to a limited movement repertoire. Here we show how activity from a few (7-30) MI neurons can be decoded into a signal that a monkey is able to use immediately to move a computer cursor to any new position in its workspace (14 degrees x 14 degrees visual angle). Our results, which are based on recordings made by an electrode array that is suitable for human use, indicate that neurally based control of movement may eventually be feasible in paralysed humans.
A high-performance brain-computer interface.
G. Santhanam and S. I. Ryu and B. M. Yu and A. Afshar and K. V. Shenoy
(2006 Jul 13)
Recent studies have demonstrated that monkeys and humans can use signals from the brain to guide computer cursors. Brain-computer interfaces (BCIs) may one day assist patients suffering from neurological injury or disease, but relatively low system performance remains a major obstacle. In fact, the speed and accuracy with which keys can be selected using BCIs is still far lower than for systems relying on eye movements. This is true whether BCIs use recordings from populations of individual neurons using invasive electrode techniques or electroencephalogram recordings using less- or non-invasive techniques. Here we present the design and demonstration, using electrode arrays implanted in monkey dorsal premotor cortex, of a manyfold higher performance BCI than previously reported. These results indicate that a fast and accurate key selection system, capable of operating with a range of keyboard sizes, is possible (up to 6.5 bits per second, or approximately 15 words per minute, with 96 electrodes). The highest information throughput is achieved with unprecedentedly brief neural recordings, even as recording quality degrades over time. These performance results and their implications for system design should substantially increase the clinical viability of BCIs in humans.
Interpreting spatial and temporal neural activity through a recurrent neural network brain-machine interface.
J. C. Sanchez and D. Erdogmus and M. A. L. Nicolelis and J. Wessberg and J. C. Principe
IEEE Trans Neural Syst Rehabil Eng
We propose the use of optimized brain-machine interface (BMI) models for interpreting the spatial and temporal neural activity generated in motor tasks. In this study, a nonlinear dynamical neural network is trained to predict the hand position of primates from neural recordings in a reaching task paradigm. We first develop a method to reveal the role attributed by the model to the sampled motor, premotor, and parietal cortices in generating hand movements. Next, using the trained model weights, we derive a temporal sensitivity measure to asses how the model utilized the sampled cortices and neurons in real-time during BMI testing.
Contribution of transcranial magnetic stimulation to the understanding of cortical mechanisms involved in motor control.
J. Reis and O. B. Swayne and Y. Vandermeeren and M. Camus and M. A. Dimyan and M. Harris-Love and M. A. Perez and P. Ragert and J. C. Rothwell and L. G. Cohen
(2008 Jan 15)
Transcranial magnetic stimulation (TMS) was initially used to evaluate the integrity of the corticospinal tract in humans non-invasively. Since these early studies, the development of paired-pulse and repetitive TMS protocols allowed investigators to explore inhibitory and excitatory interactions of various motor and non-motor cortical regions within and across cerebral hemispheres. These applications have provided insight into the intracortical physiological processes underlying the functional role of different brain regions in various cognitive processes, motor control in health and disease and neuroplastic changes during recovery of function after brain lesions. Used in combination with neuroimaging tools, TMS provides valuable information on functional connectivity between different brain regions, and on the relationship between physiological processes and the anatomical configuration of specific brain areas and connected pathways. More recently, there has been increasing interest in the extent to which these physiological processes are modulated depending on the behavioural setting. The purpose of this paper is (a) to present an up-to-date review of the available electrophysiological data and the impact on our understanding of human motor behaviour and (b) to discuss some of the gaps in our present knowledge as well as future directions of research in a format accessible to new students and/or investigators. Finally, areas of uncertainty and limitations in the interpretation of TMS studies are discussed in some detail.
Superlinear population encoding of dynamic hand trajectory in primary motor cortex.
L. Paninski and S. Shoham and M. R. Fellows and N. G. Hatsopoulos and J. P. Donoghue
(2004 Sep 29)
Neural activity in primary motor cortex (MI) is known to correlate with hand position and velocity. Previous descriptions of this tuning have (1) been linear in position or velocity, (2) depended only instantaneously on these signals, and/or (3) not incorporated the effects of interneuronal dependencies on firing rate. We show here that many MI cells encode a superlinear function of the full time-varying hand trajectory. Approximately 20% of MI cells carry information in the hand trajectory beyond just the position, velocity, and acceleration at a single time lag. Moreover, approximately one-third of MI cells encode the trajectory in a significantly superlinear manner; as one consequence, even small position changes can dramatically modulate the gain of the velocity tuning of MI cells, in agreement with recent psychophysical evidence. We introduce a compact nonlinear "preferred trajectory" model that predicts the complex structure of the spatiotemporal tuning functions described in previous work. Finally, observing the activity of neighboring cells in the MI network significantly increases the predictability of the firing rate of a single MI cell; however, we find interneuronal dependencies in MI to be much more locked to external kinematic parameters than those described recently in the hippocampus. Nevertheless, this neighbor activity is approximately as informative as the hand velocity, supporting the view that neural encoding in MI is best understood at a population level.
Decoding trajectories from posterior parietal cortex ensembles.
G. H. Mulliken and S. Musallam and R. A. Andersen
(2008 Nov 26)
High-level cognitive signals in the posterior parietal cortex (PPC) have previously been used to decode the intended endpoint of a reach, providing the first evidence that PPC can be used for direct control of a neural prosthesis (Musallam et al., 2004). Here we expand on this work by showing that PPC neural activity can be harnessed to estimate not only the endpoint but also to continuously control the trajectory of an end effector. Specifically, we trained two monkeys to use a joystick to guide a cursor on a computer screen to peripheral target locations while maintaining central ocular fixation. We found that we could accurately reconstruct the trajectory of the cursor using a relatively small ensemble of simultaneously recorded PPC neurons. Using a goal-based Kalman filter that incorporates target information into the state-space, we showed that the decoded estimate of cursor position could be significantly improved. Finally, we tested whether we could decode trajectories during closed-loop brain control sessions, in which the real-time position of the cursor was determined solely by a monkey's neural activity in PPC. The monkey learned to perform brain control trajectories at 80% success rate (for 8 targets) after just 4-5 sessions. This improvement in behavioral performance was accompanied by a corresponding enhancement in neural tuning properties (i.e., increased tuning depth and coverage of encoding parameter space) as well as an increase in off-line decoding performance of the PPC ensemble.
Neuronal interactions improve cortical population coding of movement direction.
E. M. Maynard and N. G. Hatsopoulos and C. L. Ojakangas and B. D. Acuna and J. N. Sanes and R. A. Normann and J. P. Donoghue
(1999 Sep 15)
Interactions among groups of neurons in primary motor cortex (MI) may convey information about motor behavior. We investigated the information carried by interactions in MI of macaque monkeys using a novel multielectrode array to record simultaneously from 12-16 neurons during an arm-reaching task. Pairs of simultaneously recorded cells revealed significant correlations in their trial-to-trial firing rate variation when estimated over broad (600 msec) time intervals. This covariation was only weakly related to the preferred directions of the individual MI neurons estimated from the firing rate and did not vary significantly with interelectrode distance. Most significantly, in a portion of cell pairs, correlation strength varied with the direction of the arm movement. We evaluated to what extent correlated activity provided additional information about movement direction beyond that available in single neuron firing rate. A multivariate statistical model successfully classified direction from single trials of neural data. However, classification was consistently better when correlations were incorporated into the model as compared to one in which neurons were treated as independent encoders. Information-theoretic analysis demonstrated that interactions caused by correlated activity carry additional information about movement direction beyond that based on the firing rates of independently acting neurons. These results also show that cortical representations incorporating higher order features of population activity would be richer than codes based solely on firing rate, if such information can exploited by the nervous system.
Decoding of temporal intervals from cortical ensemble activity.
M. A. Lebedev and J. E. O'Doherty and M. A. L. Nicolelis
Neurophysiological, neuroimaging, and lesion studies point to a highly distributed processing of temporal information by cortico-basal ganglia-thalamic networks. However, there are virtually no experimental data on the encoding of behavioral time by simultaneously recorded cortical ensembles. We predicted temporal intervals from the activity of hundreds of neurons recorded in motor and premotor cortex as rhesus monkeys performed self-timed hand movements. During the delay periods, when animals had to estimate temporal intervals and prepare hand movements, neuronal ensemble activity encoded both the time that elapsed from the previous hand movement and the time until the onset of the next. The neurons that were most informative of these temporal intervals increased or decreased their rates throughout the delay until reaching a threshold value, at which point a movement was initiated. Variability in the self-timed delays was explainable by the variability of neuronal rates, but not of the threshold. In addition to predicting temporal intervals, the same neuronal ensemble activity was informative for generating predictions that dissociated the delay periods of the task from the movement periods. Left hemispheric areas were the best source of predictions in one bilaterally implanted monkey overtrained to perform the task with the right hand. However, after that monkey learned to perform the task with the left hand, its left hemisphere continued and the right hemisphere started contributing to the prediction. We suggest that decoding of temporal intervals from bilaterally recorded cortical ensembles could improve the performance of neural prostheses for restoration of motor function.
Brain-machine interfaces: past, present and future.
M. A. Lebedev and M. A. L. Nicolelis
Since the original demonstration that electrical activity generated by ensembles of cortical neurons can be employed directly to control a robotic manipulator, research on brain-machine interfaces (BMIs) has experienced an impressive growth. Today BMIs designed for both experimental and clinical studies can translate raw neuronal signals into motor commands that reproduce arm reaching and hand grasping movements in artificial actuators. Clearly, these developments hold promise for the restoration of limb mobility in paralyzed subjects. However, as we review here, before this goal can be reached several bottlenecks have to be passed. These include designing a fully implantable biocompatible recording device, further developing real-time computational algorithms, introducing a method for providing the brain with sensory feedback from the actuators, and designing and building artificial prostheses that can be controlled directly by brain-derived signals. By reaching these milestones, future BMIs will be able to drive and control revolutionary prostheses that feel and act like the human arm.
Cortical ensemble adaptation to represent velocity of an artificial actuator controlled by a brain-machine interface.
M. A. Lebedev and J. M. Carmena and J. E. O'Doherty and M. Zacksenhouse and C. S. Henriquez and J. C. Principe and M. A. L. Nicolelis
(2005 May 11)
Monkeys can learn to directly control the movements of an artificial actuator by using a brain-machine interface (BMI) driven by the activity of a sample of cortical neurons. Eventually, they can do so without moving their limbs. Neuronal adaptations underlying the transition from control of the limb to control of the actuator are poorly understood. Here, we show that rapid modifications in neuronal representation of velocity of the hand and actuator occur in multiple cortical areas during the operation of a BMI. Initially, monkeys controlled the actuator by moving a hand-held pole. During this period, the BMI was trained to predict the actuator velocity. As the monkeys started using their cortical activity to control the actuator, the activity of individual neurons and neuronal populations became less representative of the animal's hand movements while representing the movements of the actuator. As a result of this adaptation, the animals could eventually stop moving their hands yet continue to control the actuator. These results show that, during BMI control, cortical ensembles represent behaviorally significant motor parameters, even if these are not associated with movements of the animal's own limb.
Functional network reorganization during learning in a brain-computer interface paradigm.
B. Jarosiewicz and S. Chase and G. Fraser and M. Velliste and R. Kass and A. Schwartz
Proc Natl Acad Sci U S A
(2008 Dec 1)
Efforts to study the neural correlates of learning are hampered by the size of the network in which learning occurs. To understand the importance of learning-related changes in a network of neurons, it is necessary to understand how the network acts as a whole to generate behavior. Here we introduce a paradigm in which the output of a cortical network can be perturbed directly and the neural basis of the compensatory changes studied in detail. Using a brain-computer interface, dozens of simultaneously recorded neurons in the motor cortex of awake, behaving monkeys are used to control the movement of a cursor in a three-dimensional virtual-reality environment. This device creates a precise, well-defined mapping between the firing of the recorded neurons and an expressed behavior (cursor movement). In a series of experiments, we force the animal to relearn the association between neural firing and cursor movement in a subset of neurons and assess how the network changes to compensate. We find that changes in neural activity reflect not only an alteration of behavioral strategy but also the relative contributions of individual neurons to the population error signal.
Neuronal ensemble control of prosthetic devices by a human with tetraplegia.
L. R. Hochberg and M. D. Serruya and G. M. Friehs and J. A. Mukand and M. Saleh and A. H. Caplan and A. Branner and D. Chen and R. D. Penn and J. P. Donoghue
(2006 Jul 13)
Neuromotor prostheses (NMPs) aim to replace or restore lost motor functions in paralysed humans by routeing movement-related signals from the brain, around damaged parts of the nervous system, to external effectors. To translate preclinical results from intact animals to a clinically useful NMP, movement signals must persist in cortex after spinal cord injury and be engaged by movement intent when sensory inputs and limb movement are long absent. Furthermore, NMPs would require that intention-driven neuronal activity be converted into a control signal that enables useful tasks. Here we show initial results for a tetraplegic human (MN) using a pilot NMP. Neuronal ensemble activity recorded through a 96-microelectrode array implanted in primary motor cortex demonstrated that intended hand motion modulates cortical spiking patterns three years after spinal cord injury. Decoders were created, providing a 'neural cursor' with which MN opened simulated e-mail and operated devices such as a television, even while conversing. Furthermore, MN used neural control to open and close a prosthetic hand, and perform rudimentary actions with a multi-jointed robotic arm. These early results suggest that NMPs based upon intracortical neuronal ensemble spiking activity could provide a valuable new neurotechnology to restore independence for humans with paralysis.
Primate reaching cued by multichannel spatiotemporal cortical microstimulation.
N. A. Fitzsimmons and W. Drake and T. L. Hanson and M. A. Lebedev and M. A. L. Nicolelis
(2007 May 23)
Both humans and animals can discriminate signals delivered to sensory areas of their brains using electrical microstimulation. This opens the possibility of creating an artificial sensory channel that could be implemented in neuroprosthetic devices. Although microstimulation delivered through multiple implanted electrodes could be beneficial for this purpose, appropriate microstimulation protocols have not been developed. Here, we report a series of experiments in which owl monkeys performed reaching movements guided by spatiotemporal patterns of cortical microstimulation delivered to primary somatosensory cortex through chronically implanted multielectrode arrays. The monkeys learned to discriminate microstimulation patterns, and their ability to learn new patterns and new behavioral rules improved during several months of testing. Significantly, information was conveyed to the brain through the interplay of microstimulation patterns delivered to multiple electrodes and the temporal order in which these electrodes were stimulated. This suggests multichannel microstimulation as a viable means of sensorizing neural prostheses.
Bridging the brain to the world: a perspective on neural interface systems.
J. P. Donoghue
(2008 Nov 6)
Neural interface (NI) systems hold the potential to return lost functions to persons with paralysis. Impressive progress has been made, including evaluation of neural control signals, sensor testing in humans, signal decoding advances, and proof-of-concept validation. Most importantly, the field has demonstrated that persons with paralysis can use prototype systems for spelling, "point and click," and robot control. Human and animal NI research is advancing knowledge about neural information processing and plasticity in healthy, diseased, and injured nervous systems. This emerging field promises a range of neurotechnologies able to return communication, independence, and control to people with movement limitations.
Reduction of single-neuron firing uncertainty by cortical ensembles during motor skill learning.
D. Cohen and M. A. L. Nicolelis
(2004 Apr 7)
Motor skill learning is usually characterized by shortening of response time and performance of faster, more stereotypical movements. However, little is known about the changes in neural activity that underlie these behavioral changes. Here we used chronically implanted electrode arrays to record neuronal activity in the rat primary motor cortex (MI) as animals learned to execute movements in two directions. Strong modulation of MI single-neuron activity was observed while movement duration of the animal decreased. Despite many learning-induced changes, the precision with which single neurons fire did not improve with learning. Hence, prediction of movement direction from single neurons was bounded. In contrast, prediction of movement direction using neuronal ensembles improved significantly with learning, suggesting that, with practice, neuronal ensembles learn to overcome the uncertainty introduced by single-neuron stochastic activity.
Using multi-neuron population recordings for neural prosthetics.
J. K. Chapin
Classical single-neuron recording methods led to 'neuron-centric' concepts of neural coding, whereas more recent multi-neuron population recordings have inspired 'population-centric' concepts of distributed processing in neural systems. Because most neocortical neurons code information coarsely, sensory or motor processing tends to be widely distributed across neuronal populations. Dynamic fluctuations in neural population functions thus involve subtle changes in the overall pattern of neural activity. Mathematical analysis of neural population codes allows extraction of 'motor signals' from neuronal population recordings in the motor cortices, which can then be used in real-time to directly control movement of a robot arm. This technique holds promise for the development of neurally controlled prosthetic devices and provides insights into how information is distributed across several brain regions.
Learning to control a brain-machine interface for reaching and grasping by primates.
J. M. Carmena and M. A. Lebedev and R. E. Crist and J. E. O'Doherty and D. M. Santucci and D. F. Dimitrov and P. G. Patil and C. S. Henriquez and M. A. L. Nicolelis
Reaching and grasping in primates depend on the coordination of neural activity in large frontoparietal ensembles. Here we demonstrate that primates can learn to reach and grasp virtual objects by controlling a robot arm through a closed-loop brain-machine interface (BMIc) that uses multiple mathematical models to extract several motor parameters (i.e., hand position, velocity, gripping force, and the EMGs of multiple arm muscles) from the electrical activity of frontoparietal neuronal ensembles. As single neurons typically contribute to the encoding of several motor parameters, we observed that high BMIc accuracy required recording from large neuronal ensembles. Continuous BMIc operation by monkeys led to significant improvements in both model predictions and behavioral performance. Using visual feedback, monkeys succeeded in producing robot reach-and-grasp movements even when their arms did not move. Learning to operate the BMIc was paralleled by functional reorganization in multiple cortical areas, suggesting that the dynamic properties of the BMIc were incorporated into motor and sensory cortical representations.
Selecting the signals for a brain-machine interface.
R. A. Andersen and S. Musallam and B. Pesaran
Curr Opin Neurobiol
Brain-machine interfaces are being developed to assist paralyzed patients by enabling them to operate machines with recordings of their own neural activity. Recent studies show that motor parameters, such as hand trajectory, and cognitive parameters, such as the goal and predicted value of an action, can be decoded from the recorded activity to provide control signals. Neural prosthetics that use simultaneously a variety of cognitive and motor signals can maximize the ability of patients to communicate and interact with the outside world. Although most studies have recorded electroencephalograms or spike activity, recent research shows that local field potentials (LFPs) offer a promising additional signal. The decode performances of LFPs and spike signals are comparable and, because LFP recordings are more long lasting, they might help to increase the lifetime of the prosthetics.