Paul Nuyujukian

All Publications

• Cortical control of a tablet computer by people with paralysis. PloS one Nuyujukian, P., Albites Sanabria, J., Saab, J., Pandarinath, C., Jarosiewicz, B., Blabe, C. H., Franco, B., Mernoff, S. T., Eskandar, E. N., Simeral, J. D., Hochberg, L. R., Shenoy, K. V., Henderson, J. M. 2018; 13 (11): e0204566

  Abstract

  General-purpose computers have become ubiquitous and important for everyday life, but they are difficult for people with paralysis to use. Specialized software and personalized input devices can improve access, but often provide only limited functionality. In this study, three research participants with tetraplegia who had multielectrode arrays implanted in motor cortex as part of the BrainGate2 clinical trial used an intracortical brain-computer interface (iBCI) to control an unmodified commercial tablet computer. Neural activity was decoded in real time as a point-and-click wireless Bluetooth mouse, allowing participants to use common and recreational applications (web browsing, email, chatting, playing music on a piano application, sending text messages, etc.). Two of the participants also used the iBCI to "chat" with each other in real time. This study demonstrates, for the first time, high-performance iBCI control of an unmodified, commercially available, general-purpose mobile computing device by people with tetraplegia.

  View details for DOI 10.1371/journal.pone.0204566

  View details for PubMedID 30462658

• High performance communication by people with paralysis using an intracortical brain-computer interface. eLife Pandarinath, C., Nuyujukian, P., Blabe, C. H., Sorice, B. L., Saab, J., Willett, F. R., Hochberg, L. R., Shenoy, K. V., Henderson, J. M. 2017; 6

  Abstract

  Brain-computer interfaces (BCIs) have the potential to restore communication for people with tetraplegia and anarthria by translating neural activity into control signals for assistive communication devices. While previous pre-clinical and clinical studies have demonstrated promising proofs-of-concept (Serruya et al., 2002; Simeral et al., 2011; Bacher et al., 2015; Nuyujukian et al., 2015; Aflalo et al., 2015; Gilja et al., 2015; Jarosiewicz et al., 2015; Wolpaw et al., 1998; Hwang et al., 2012; Spüler et al., 2012; Leuthardt et al., 2004; Taylor et al., 2002; Schalk et al., 2008; Moran, 2010; Brunner et al., 2011; Wang et al., 2013; Townsend and Platsko, 2016; Vansteensel et al., 2016; Nuyujukian et al., 2016; Carmena et al., 2003; Musallam et al., 2004; Santhanam et al., 2006; Hochberg et al., 2006; Ganguly et al., 2011; O'Doherty et al., 2011; Gilja et al., 2012), the performance of human clinical BCI systems is not yet high enough to support widespread adoption by people with physical limitations of speech. Here we report a high-performance intracortical BCI (iBCI) for communication, which was tested by three clinical trial participants with paralysis. The system leveraged advances in decoder design developed in prior pre-clinical and clinical studies (Gilja et al., 2015; Kao et al., 2016; Gilja et al., 2012). For all three participants, performance exceeded previous iBCIs (Bacher et al., 2015; Jarosiewicz et al., 2015) as measured by typing rate (by a factor of 1.4-4.2) and information throughput (by a factor of 2.2-4.0). This high level of performance demonstrates the potential utility of iBCIs as powerful assistive communication devices for people with limited motor function.Clinical Trial No: NCT00912041.

  View details for DOI 10.7554/eLife.18554

  View details for PubMedID 28220753

  View details for PubMedCentralID PMC5319839

• A high-performance neural prosthesis enabled by control algorithm design NATURE NEUROSCIENCE Gilja, V., Nuyujukian, P., Chestek, C. A., Cunningham, J. P., Yu, B. M., Fan, J. M., Churchland, M. M., Kaufman, M. T., Kao, J. C., Ryu, S. I., Shenoy, K. V. 2012; 15 (12): 1752-1757

  Abstract

  Neural prostheses translate neural activity from the brain into control signals for guiding prosthetic devices, such as computer cursors and robotic limbs, and thus offer individuals with disabilities greater interaction with the world. However, relatively low performance remains a critical barrier to successful clinical translation; current neural prostheses are considerably slower, with less accurate control, than the native arm. Here we present a new control algorithm, the recalibrated feedback intention-trained Kalman filter (ReFIT-KF) that incorporates assumptions about the nature of closed-loop neural prosthetic control. When tested in rhesus monkeys implanted with motor cortical electrode arrays, the ReFIT-KF algorithm outperformed existing neural prosthetic algorithms in all measured domains and halved target acquisition time. This control algorithm permits sustained, uninterrupted use for hours and generalizes to more challenging tasks without retraining. Using this algorithm, we demonstrate repeatable high performance for years after implantation in two monkeys, thereby increasing the clinical viability of neural prostheses.

  View details for DOI 10.1038/nn.3265

  View details for Web of Science ID 000311706700023

  View details for PubMedID 23160043

  View details for PubMedCentralID PMC3638087

• Brain-machine interface cursor position only weakly affects monkey and human motor cortical activity in the absence of arm movements. Scientific reports Stavisky, S. D., Kao, J. C., Nuyujukian, P., Pandarinath, C., Blabe, C., Ryu, S. I., Hochberg, L. R., Henderson, J. M., Shenoy, K. V. 2018; 8 (1): 16357

  Abstract

  Brain-machine interfaces (BMIs) that decode movement intentions should ignore neural modulation sources distinct from the intended command. However, neurophysiology and control theory suggest that motor cortex reflects the motor effector's position, which could be a nuisance variable. We investigated motor cortical correlates of BMI cursor position with or without concurrent arm movement. We show in two monkeys that subtracting away estimated neural correlates of position improves online BMI performance only if the animals were allowed to move their arm. To understand why, we compared the neural variance attributable to cursor position when the same task was performed using arm reaching, versus arms-restrained BMI use. Firing rates correlated with both BMI cursor and hand positions, but hand positional effects were greater. To examine whether BMI position influences decoding in people with paralysis, we analyzed data from two intracortical BMI clinical trial participants and performed an online decoder comparison in one participant. We found only small motor cortical correlates, which did not affect performance. These results suggest that arm movement and proprioception are the major contributors to position-related motor cortical correlates. Cursor position visual feedback is therefore unlikely to affect the performance of BMI-driven prosthetic systems being developed for people with paralysis.

  View details for DOI 10.1038/s41598-018-34711-1

  View details for PubMedID 30397281

• An Open-Source Realtime Computational Platform (Short WIP Paper) Mehrotra, P., Dasgupta, S., Robertson, S., Nuyujukian, P. ASSOC COMPUTING MACHINERY. 2018: 109–12

  View details for DOI 10.1145/3211332.3211344

  View details for Web of Science ID 000452799600012

• Neural Population Dynamics Underlying Motor Learning Transfer NEURON Vyas, S., Even-Chen, N., Stavisky, S. D., Ryu, S. I., Nuyujukian, P., Shenoy, K. V. 2018; 97 (5): 1177-+

  Abstract

  Covert motor learning can sometimes transfer to overt behavior. We investigated the neural mechanism underlying transfer by constructing a two-context paradigm. Subjects performed cursor movements either overtly using arm movements, or covertly via a brain-machine interface that moves the cursor based on motor cortical activity (in lieu of arm movement). These tasks helped evaluate whether and how cortical changes resulting from "covert rehearsal" affect overt performance. We found that covert learning indeed transfers to overt performance and is accompanied by systematic population-level changes in motor preparatory activity. Current models of motor cortical function ascribe motor preparation to achieving initial conditions favorable for subsequent movement-period neural dynamics. We found that covert and overt contexts share these initial conditions, and covert rehearsal manipulates them in a manner that persists across context changes, thus facilitating overt motor learning. This transfer learning mechanism might provide new insights into other covert processes like mental rehearsal.

  View details for DOI 10.1016/j.neuron.2018.01.040

  View details for Web of Science ID 000426974100018

  View details for PubMedID 29456026

  View details for PubMedCentralID PMC5843544

• Feasibility of Automatic Error Detect-and-Undo System in Human Intracortical Brain-Computer Interfaces. IEEE transactions on bio-medical engineering Even-Chen, N., Stavisky, S. D., Pandarinath, C., Nuyujukian, P., Blabe, C. H., Hochberg, L. R., Henderson, J. M., Shenoy, K. V. 2018; 65 (8): 1771–84

  Abstract

  Brain-computer interfaces (BCIs) aim to help people with impaired movement ability by directly translating their movement intentions into command signals for assistive technologies. Despite large performance improvements over the last two decades, BCI systems still make errors that need to be corrected manually by the user. This decreases system performance and is also frustrating for the user. The deleterious effects of errors could be mitigated if the system automatically detected when the user perceives that an error was made and automatically intervened with a corrective action; thus, sparing users from having to make the correction themselves. Our previous preclinical work with monkeys demonstrated that task-outcome correlates exist in motor cortical spiking activity and can be utilized to improve BCI performance. Here, we asked if these signals also exist in the human hand area of motor cortex, and whether they can be decoded with high accuracy.We analyzed posthoc the intracortical neural activity of two BrainGate2 clinical trial participants who were neurally controlling a computer cursor to perform a grid target selection task and a keyboard-typing task.Our key findings are that: 1) there exists a putative outcome error signal reflected in both the action potentials and local field potentials of the human hand area of motor cortex, and 2) target selection outcomes can be classified with high accuracy (70-85%) of errors successfully detected with minimal (0-3%) misclassifications of success trials, based on neural activity alone.These offline results suggest that it will be possible to improve the performance of clinical intracortical BCIs by incorporating a real-time error detect-and-undo system alongside the decoding of movement intention.

  View details for DOI 10.1109/TBME.2017.2776204

  View details for PubMedID 29989931

• A High-Performance Neural Prosthesis Incorporating Discrete State Selection With Hidden Markov Models IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING Kao, J. C., Nuyujukian, P., Ryu, S. I., Shenoy, K. V. 2017; 64 (4): 935-945

  Abstract

  Communication neural prostheses aim to restore efficient communication to people with motor neurological injury or disease by decoding neural activity into control signals. These control signals are both analog (e.g., the velocity of a computer mouse) and discrete (e.g., clicking an icon with a computer mouse) in nature. Effective, high-performing, and intuitive-to-use communication prostheses should be capable of decoding both analog and discrete state variables seamlessly. However, to date, the highest-performing autonomous communication prostheses rely on precise analog decoding and typically do not incorporate high-performance discrete decoding. In this report, we incorporated a hidden Markov model (HMM) into an intracortical communication prosthesis to enable accurate and fast discrete state decoding in parallel with analog decoding. In closed-loop experiments with nonhuman primates implanted with multielectrode arrays, we demonstrate that incorporating an HMM into a neural prosthesis can increase state-of-the-art achieved bitrate by 13.9% and 4.2% in two monkeys ( ). We found that the transition model of the HMM is critical to achieving this performance increase. Further, we found that using an HMM resulted in the highest achieved peak performance we have ever observed for these monkeys, achieving peak bitrates of 6.5, 5.7, and 4.7 bps in Monkeys J, R, and L, respectively. Finally, we found that this neural prosthesis was robustly controllable for the duration of entire experimental sessions. These results demonstrate that high-performance discrete decoding can be beneficially combined with analog decoding to achieve new state-of-the-art levels of performance.

  View details for DOI 10.1109/TBME.2016.2582691

  View details for Web of Science ID 000398738300020

• A Nonhuman Primate Brain-Computer Typing Interface PROCEEDINGS OF THE IEEE Nuyujukian, P., Kao, J. C., Ryu, S. I., Shenoy, K. V. 2017; 105 (1): 66-72

  View details for DOI 10.1109/JPROC.2016.2586967

  View details for Web of Science ID 000391703100006

• Clinical translation of a high-performance neural prosthesis. Nature medicine Gilja, V., Pandarinath, C., Blabe, C. H., Nuyujukian, P., Simeral, J. D., Sarma, A. A., Sorice, B. L., Perge, J. A., Jarosiewicz, B., Hochberg, L. R., Shenoy, K. V., Henderson, J. M. 2015; 21 (10): 1142-1145

  Abstract

  Neural prostheses have the potential to improve the quality of life of individuals with paralysis by directly mapping neural activity to limb- and computer-control signals. We translated a neural prosthetic system previously developed in animal model studies for use by two individuals with amyotrophic lateral sclerosis who had intracortical microelectrode arrays placed in motor cortex. Measured more than 1 year after implant, the neural cursor-control system showed the highest published performance achieved by a person to date, more than double that of previous pilot clinical trial participants.

  View details for DOI 10.1038/nm.3953

  View details for PubMedID 26413781

• Single-trial dynamics of motor cortex and their applications to brain-machine interfaces NATURE COMMUNICATIONS Kao, J. C., Nuyujukian, P., Ryu, S. I., Churchland, M. M., Cunningham, J. P., Shenoy, K. V. 2015; 6

  Abstract

  Increasing evidence suggests that neural population responses have their own internal drive, or dynamics, that describe how the neural population evolves through time. An important prediction of neural dynamical models is that previously observed neural activity is informative of noisy yet-to-be-observed activity on single-trials, and may thus have a denoising effect. To investigate this prediction, we built and characterized dynamical models of single-trial motor cortical activity. We find these models capture salient dynamical features of the neural population and are informative of future neural activity on single trials. To assess how neural dynamics may beneficially denoise single-trial neural activity, we incorporate neural dynamics into a brain-machine interface (BMI). In online experiments, we find that a neural dynamical BMI achieves substantially higher performance than its non-dynamical counterpart. These results provide evidence that neural dynamics beneficially inform the temporal evolution of neural activity on single trials and may directly impact the performance of BMIs.

  View details for DOI 10.1038/ncomms8759

  View details for Web of Science ID 000358858800003

• Neural population dynamics in human motor cortex during movements in people with ALS ELIFE Pandarinath, C., Gilja, V., Blabe, C. H., Nuyujukian, P., Sarma, A. A., Sorice, B. L., Eskandar, E. N., Hochberg, L. R., Henderson, J. M., Shenoy, K. V. 2015; 4

  Abstract

  The prevailing view of motor cortex holds that motor cortical neural activity represents muscle or movement parameters. However, recent studies in non-human primates have shown that neural activity does not simply represent muscle or movement parameters; instead, its temporal structure is well-described by a dynamical system where activity during movement evolves lawfully from an initial pre-movement state. In this study, we analyze neuronal ensemble activity in motor cortex in two clinical trial participants diagnosed with Amyotrophic Lateral Sclerosis (ALS). We find that activity in human motor cortex has similar dynamical structure to that of non-human primates, indicating that human motor cortex contains a similar underlying dynamical system for movement generation.

  View details for DOI 10.7554/eLife.07436

  View details for Web of Science ID 000356720100001

  View details for PubMedID 26099302

  View details for PubMedCentralID PMC4475900

• A high performing brain-machine interface driven by low-frequency local field potentials alone and together with spikes. Journal of neural engineering Stavisky, S. D., Kao, J. C., Nuyujukian, P., Ryu, S. I., Shenoy, K. V. 2015; 12 (3): 036009-?

  Abstract

  Brain-machine interfaces (BMIs) seek to enable people with movement disabilities to directly control prosthetic systems with their neural activity. Current high performance BMIs are driven by action potentials (spikes), but access to this signal often diminishes as sensors degrade over time. Decoding local field potentials (LFPs) as an alternative or complementary BMI control signal may improve performance when there is a paucity of spike signals. To date only a small handful of LFP decoding methods have been tested online; there remains a need to test different LFP decoding approaches and improve LFP-driven performance. There has also not been a reported demonstration of a hybrid BMI that decodes kinematics from both LFP and spikes. Here we first evaluate a BMI driven by the local motor potential (LMP), a low-pass filtered time-domain LFP amplitude feature. We then combine decoding of both LMP and spikes to implement a hybrid BMI.Spikes and LFP were recorded from two macaques implanted with multielectrode arrays in primary and premotor cortex while they performed a reaching task. We then evaluated closed-loop BMI control using biomimetic decoders driven by LMP, spikes, or both signals together.LMP decoding enabled quick and accurate cursor control which surpassed previously reported LFP BMI performance. Hybrid decoding of both spikes and LMP improved performance when spikes signal quality was mediocre to poor.These findings show that LMP is an effective BMI control signal which requires minimal power to extract and can substitute for or augment impoverished spikes signals. Use of this signal may lengthen the useful lifespan of BMIs and is therefore an important step towards clinically viable BMIs.

  View details for DOI 10.1088/1741-2560/12/3/036009

  View details for PubMedID 25946198

• A high performing brain-machine interface driven by low-frequency local field potentials alone and together with spikes JOURNAL OF NEURAL ENGINEERING Stavisky, S. D., Kao, J. C., Nuyujukian, P., Ryu, S. I., Shenoy, K. V. 2015; 12 (3)

  Abstract

  Brain-machine interfaces (BMIs) seek to enable people with movement disabilities to directly control prosthetic systems with their neural activity. Current high performance BMIs are driven by action potentials (spikes), but access to this signal often diminishes as sensors degrade over time. Decoding local field potentials (LFPs) as an alternative or complementary BMI control signal may improve performance when there is a paucity of spike signals. To date only a small handful of LFP decoding methods have been tested online; there remains a need to test different LFP decoding approaches and improve LFP-driven performance. There has also not been a reported demonstration of a hybrid BMI that decodes kinematics from both LFP and spikes. Here we first evaluate a BMI driven by the local motor potential (LMP), a low-pass filtered time-domain LFP amplitude feature. We then combine decoding of both LMP and spikes to implement a hybrid BMI.Spikes and LFP were recorded from two macaques implanted with multielectrode arrays in primary and premotor cortex while they performed a reaching task. We then evaluated closed-loop BMI control using biomimetic decoders driven by LMP, spikes, or both signals together.LMP decoding enabled quick and accurate cursor control which surpassed previously reported LFP BMI performance. Hybrid decoding of both spikes and LMP improved performance when spikes signal quality was mediocre to poor.These findings show that LMP is an effective BMI control signal which requires minimal power to extract and can substitute for or augment impoverished spikes signals. Use of this signal may lengthen the useful lifespan of BMIs and is therefore an important step towards clinically viable BMIs.

  View details for DOI 10.1088/1741-2560/12/3/036009

  View details for Web of Science ID 000354998600010

  View details for PubMedID 25946198

  View details for PubMedCentralID PMC4457459

• Comparison of spike sorting and thresholding of voltage waveforms for intracortical brain-machine interface performance. Journal of neural engineering Christie, B. P., Tat, D. M., Irwin, Z. T., Gilja, V., Nuyujukian, P., Foster, J. D., Ryu, S. I., Shenoy, K. V., Thompson, D. E., Chestek, C. A. 2015; 12 (1): 016009-?

  Abstract

  Objective. For intracortical brain-machine interfaces (BMIs), action potential voltage waveforms are often sorted to separate out individual neurons. If these neurons contain independent tuning information, this process could increase BMI performance. However, the sorting of action potentials ('spikes') requires high sampling rates and is computationally expensive. To explicitly define the difference between spike sorting and alternative methods, we quantified BMI decoder performance when using threshold-crossing events versus sorted action potentials. Approach. We used data sets from 58 experimental sessions from two rhesus macaques implanted with Utah arrays. Data were recorded while the animals performed a center-out reaching task with seven different angles. For spike sorting, neural signals were sorted into individual units by using a mixture of Gaussians to cluster the first four principal components of the waveforms. For thresholding events, spikes that simply crossed a set threshold were retained. We decoded the data offline using both a Naïve Bayes classifier for reaching direction and a linear regression to evaluate hand position. Main results. We found the highest performance for thresholding when placing a threshold between -3 and -4.5 × Vrms. Spike sorted data outperformed thresholded data for one animal but not the other. The mean Naïve Bayes classification accuracy for sorted data was 88.5% and changed by 5% on average when data were thresholded. The mean correlation coefficient for sorted data was 0.92, and changed by 0.015 on average when thresholded. Significance. For prosthetics applications, these results imply that when thresholding is used instead of spike sorting, only a small amount of performance may be lost. The utilization of threshold-crossing events may significantly extend the lifetime of a device because these events are often still detectable once single neurons are no longer isolated.

  View details for DOI 10.1088/1741-2560/12/1/016009

  View details for PubMedID 25504690

• Comparison of spike sorting and thresholding of voltage waveforms for intracortical brain-machine interface performance. Journal of neural engineering Christie, B. P., Tat, D. M., Irwin, Z. T., Gilja, V., Nuyujukian, P., Foster, J. D., Ryu, S. I., Shenoy, K. V., Thompson, D. E., Chestek, C. A. 2015; 12 (1): 016009-?

  Abstract

  Objective. For intracortical brain-machine interfaces (BMIs), action potential voltage waveforms are often sorted to separate out individual neurons. If these neurons contain independent tuning information, this process could increase BMI performance. However, the sorting of action potentials ('spikes') requires high sampling rates and is computationally expensive. To explicitly define the difference between spike sorting and alternative methods, we quantified BMI decoder performance when using threshold-crossing events versus sorted action potentials. Approach. We used data sets from 58 experimental sessions from two rhesus macaques implanted with Utah arrays. Data were recorded while the animals performed a center-out reaching task with seven different angles. For spike sorting, neural signals were sorted into individual units by using a mixture of Gaussians to cluster the first four principal components of the waveforms. For thresholding events, spikes that simply crossed a set threshold were retained. We decoded the data offline using both a Naïve Bayes classifier for reaching direction and a linear regression to evaluate hand position. Main results. We found the highest performance for thresholding when placing a threshold between -3 and -4.5 × Vrms. Spike sorted data outperformed thresholded data for one animal but not the other. The mean Naïve Bayes classification accuracy for sorted data was 88.5% and changed by 5% on average when data were thresholded. The mean correlation coefficient for sorted data was 0.92, and changed by 0.015 on average when thresholded. Significance. For prosthetics applications, these results imply that when thresholding is used instead of spike sorting, only a small amount of performance may be lost. The utilization of threshold-crossing events may significantly extend the lifetime of a device because these events are often still detectable once single neurons are no longer isolated.

  View details for DOI 10.1088/1741-2560/12/1/016009

  View details for PubMedID 25504690

• A High-Performance Keyboard Neural Prosthesis Enabled by Task Optimization IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING Nuyujukian, P., Fan, J. M., Kao, J. C., Ryu, S. I., Shenoy, K. V. 2015; 62 (1): 21-29

  Abstract

  Communication neural prostheses are an emerging class of medical devices that aim to restore efficient communication to people suffering from paralysis. These systems rely on an interface with the user, either via the use of a continuouslymoving cursor (e.g., mouse) or the discrete selection of symbols (e.g., keyboard). In developing these interfaces, many design choices have a significant impact on the performance of the system. The objective of this study was to explore the design choices of a continuously-moving cursor neural prosthesis and optimize the interface to maximize information theoretic performance. We swept interface parameters of two keyboard-like tasks to find task and subject specific optimal parameters as measured by achieved bitrate using two rhesus macaques implanted with multielectrode arrays. In this report, we present the highest performing free-paced neural prosthesis under any recording modality with sustainable communication rates of up to 3.5 bits per second (bps). These findings demonstrate that meaningful high performance can be achieved using an intracortical neural prosthesis, and that, when optimized, these systems may be appropriate for use as communication devices for those with physical disabilities.

  View details for DOI 10.1109/TBME.2014.2354697

  View details for Web of Science ID 000346765500003

  View details for PubMedID 25203982

• Neural population dynamics in human motor cortex during movements in people with ALS. eLife Pandarinath, C., Gilja, V., Blabe, C. H., Nuyujukian, P., Sarma, A. A., Sorice, B. L., Eskandar, E. N., Hochberg, L. R., Henderson, J. M., Shenoy, K. V. 2015; 4

  Abstract

  The prevailing view of motor cortex holds that motor cortical neural activity represents muscle or movement parameters. However, recent studies in non-human primates have shown that neural activity does not simply represent muscle or movement parameters; instead, its temporal structure is well-described by a dynamical system where activity during movement evolves lawfully from an initial pre-movement state. In this study, we analyze neuronal ensemble activity in motor cortex in two clinical trial participants diagnosed with Amyotrophic Lateral Sclerosis (ALS). We find that activity in human motor cortex has similar dynamical structure to that of non-human primates, indicating that human motor cortex contains a similar underlying dynamical system for movement generation.

  View details for DOI 10.7554/eLife.07436

  View details for PubMedID 26099302

• Single-trial dynamics of motor cortex and their applications to brain-machine interfaces. Nature communications Kao, J. C., Nuyujukian, P., Ryu, S. I., Churchland, M. M., Cunningham, J. P., Shenoy, K. V. 2015; 6: 7759-?

  Abstract

  Increasing evidence suggests that neural population responses have their own internal drive, or dynamics, that describe how the neural population evolves through time. An important prediction of neural dynamical models is that previously observed neural activity is informative of noisy yet-to-be-observed activity on single-trials, and may thus have a denoising effect. To investigate this prediction, we built and characterized dynamical models of single-trial motor cortical activity. We find these models capture salient dynamical features of the neural population and are informative of future neural activity on single trials. To assess how neural dynamics may beneficially denoise single-trial neural activity, we incorporate neural dynamics into a brain-machine interface (BMI). In online experiments, we find that a neural dynamical BMI achieves substantially higher performance than its non-dynamical counterpart. These results provide evidence that neural dynamics beneficially inform the temporal evolution of neural activity on single trials and may directly impact the performance of BMIs.

  View details for DOI 10.1038/ncomms8759

  View details for PubMedID 26220660

• Performance sustaining intracortical neural prostheses. Journal of neural engineering Nuyujukian, P., Kao, J. C., Fan, J. M., Stavisky, S. D., Ryu, S. I., Shenoy, K. V. 2014; 11 (6): 066003-?

  Abstract

  Objective. Neural prostheses, or brain-machine interfaces, aim to restore efficient communication and movement ability to those suffering from paralysis. A major challenge these systems face is robust performance, particularly with aging signal sources. The aim in this study was to develop a neural prosthesis that could sustain high performance in spite of signal instability while still minimizing retraining time. Approach. We trained two rhesus macaques implanted with intracortical microelectrode arrays 1-4 years prior to this study to acquire targets with a neurally-controlled cursor. We measured their performance via achieved bitrate (bits per second, bps). This task was repeated over contiguous days to evaluate the sustained performance across time. Main results. We found that in the monkey with a younger (i.e., two year old) implant and better signal quality, a fixed decoder could sustain performance for a month at a rate of 4 bps, the highest achieved communication rate reported to date. This fixed decoder was evaluated across 22 months and experienced a performance decline at a rate of 0.24 bps yr(-1). In the monkey with the older (i.e., 3.5 year old) implant and poorer signal quality, a fixed decoder could not sustain performance for more than a few days. Nevertheless, performance in this monkey was maintained for two weeks without requiring additional online retraining time by utilizing prior days' experimental data. Upon analysis of the changes in channel tuning, we found that this stability appeared partially attributable to the cancelling-out of neural tuning fluctuations when projected to two-dimensional cursor movements. Significance. The findings in this study (1) document the highest-performing communication neural prosthesis in monkeys, (2) confirm and extend prior reports of the stability of fixed decoders, and (3) demonstrate a protocol for system stability under conditions where fixed decoders would otherwise fail. These improvements to decoder stability are important for minimizing training time and should make neural prostheses more practical to use.

  View details for DOI 10.1088/1741-2560/11/6/066003

  View details for PubMedID 25307561

• Performance sustaining intracortical neural prostheses JOURNAL OF NEURAL ENGINEERING Nuyujukian, P., Kao, J. C., Fan, J. M., Stavisky, S. D., Ryu, S. I., Shenoy, K. V. 2014; 11 (6)

  Abstract

  Objective. Neural prostheses, or brain-machine interfaces, aim to restore efficient communication and movement ability to those suffering from paralysis. A major challenge these systems face is robust performance, particularly with aging signal sources. The aim in this study was to develop a neural prosthesis that could sustain high performance in spite of signal instability while still minimizing retraining time. Approach. We trained two rhesus macaques implanted with intracortical microelectrode arrays 1-4 years prior to this study to acquire targets with a neurally-controlled cursor. We measured their performance via achieved bitrate (bits per second, bps). This task was repeated over contiguous days to evaluate the sustained performance across time. Main results. We found that in the monkey with a younger (i.e., two year old) implant and better signal quality, a fixed decoder could sustain performance for a month at a rate of 4 bps, the highest achieved communication rate reported to date. This fixed decoder was evaluated across 22 months and experienced a performance decline at a rate of 0.24 bps yr(-1). In the monkey with the older (i.e., 3.5 year old) implant and poorer signal quality, a fixed decoder could not sustain performance for more than a few days. Nevertheless, performance in this monkey was maintained for two weeks without requiring additional online retraining time by utilizing prior days' experimental data. Upon analysis of the changes in channel tuning, we found that this stability appeared partially attributable to the cancelling-out of neural tuning fluctuations when projected to two-dimensional cursor movements. Significance. The findings in this study (1) document the highest-performing communication neural prosthesis in monkeys, (2) confirm and extend prior reports of the stability of fixed decoders, and (3) demonstrate a protocol for system stability under conditions where fixed decoders would otherwise fail. These improvements to decoder stability are important for minimizing training time and should make neural prostheses more practical to use.

  View details for DOI 10.1088/1741-2560/11/6/066003

  View details for Web of Science ID 000345647700009

  View details for PubMedID 25307561

• A freely-moving monkey treadmill model. Journal of neural engineering D Foster, J., Nuyujukian, P., Freifeld, O., Gao, H., Walker, R., I Ryu, S., H Meng, T., Murmann, B., J Black, M., V Shenoy, K. 2014; 11 (4): 046020-?

  Abstract

  Objective. Motor neuroscience and brain-machine interface (BMI) design is based on examining how the brain controls voluntary movement, typically by recording neural activity and behavior from animal models. Recording technologies used with these animal models have traditionally limited the range of behaviors that can be studied, and thus the generality of science and engineering research. We aim to design a freely-moving animal model using neural and behavioral recording technologies that do not constrain movement. Approach. We have established a freely-moving rhesus monkey model employing technology that transmits neural activity from an intracortical array using a head-mounted device and records behavior through computer vision using markerless motion capture. We demonstrate the flexibility and utility of this new monkey model, including the first recordings from motor cortex while rhesus monkeys walk quadrupedally on a treadmill. Main results. Using this monkey model, we show that multi-unit threshold-crossing neural activity encodes the phase of walking and that the average firing rate of the threshold crossings covaries with the speed of individual steps. On a population level, we find that neural state-space trajectories of walking at different speeds have similar rotational dynamics in some dimensions that evolve at the step rate of walking, yet robustly separate by speed in other state-space dimensions. Significance. Freely-moving animal models may allow neuroscientists to examine a wider range of behaviors and can provide a flexible experimental paradigm for examining the neural mechanisms that underlie movement generation across behaviors and environments. For BMIs, freely-moving animal models have the potential to aid prosthetic design by examining how neural encoding changes with posture, environment and other real-world context changes. Understanding this new realm of behavior in more naturalistic settings is essential for overall progress of basic motor neuroscience and for the successful translation of BMIs to people with paralysis.

  View details for DOI 10.1088/1741-2560/11/4/046020

  View details for PubMedID 24995476

• A freely-moving monkey treadmill model JOURNAL OF NEURAL ENGINEERING Foster, J. D., Nuyujukian, P., Freifeld, O., Gao, H., Walker, R., Ryu, S. I., Meng, T. H., Murmann, B., Black, M. J., Shenoy, K. V. 2014; 11 (4)

  Abstract

  Objective. Motor neuroscience and brain-machine interface (BMI) design is based on examining how the brain controls voluntary movement, typically by recording neural activity and behavior from animal models. Recording technologies used with these animal models have traditionally limited the range of behaviors that can be studied, and thus the generality of science and engineering research. We aim to design a freely-moving animal model using neural and behavioral recording technologies that do not constrain movement. Approach. We have established a freely-moving rhesus monkey model employing technology that transmits neural activity from an intracortical array using a head-mounted device and records behavior through computer vision using markerless motion capture. We demonstrate the flexibility and utility of this new monkey model, including the first recordings from motor cortex while rhesus monkeys walk quadrupedally on a treadmill. Main results. Using this monkey model, we show that multi-unit threshold-crossing neural activity encodes the phase of walking and that the average firing rate of the threshold crossings covaries with the speed of individual steps. On a population level, we find that neural state-space trajectories of walking at different speeds have similar rotational dynamics in some dimensions that evolve at the step rate of walking, yet robustly separate by speed in other state-space dimensions. Significance. Freely-moving animal models may allow neuroscientists to examine a wider range of behaviors and can provide a flexible experimental paradigm for examining the neural mechanisms that underlie movement generation across behaviors and environments. For BMIs, freely-moving animal models have the potential to aid prosthetic design by examining how neural encoding changes with posture, environment and other real-world context changes. Understanding this new realm of behavior in more naturalistic settings is essential for overall progress of basic motor neuroscience and for the successful translation of BMIs to people with paralysis.

  View details for DOI 10.1088/1741-2560/11/4/046020

  View details for Web of Science ID 000340046500020

• Information Systems Opportunities in Brain-Machine Interface Decoders PROCEEDINGS OF THE IEEE Kao, J. C., Stavisky, S. D., Sussillo, D., Nuyujukian, P., Shenoy, K. V. 2014; 102 (5): 666-682

  View details for DOI 10.1109/JPROC.2014.2307357

  View details for Web of Science ID 000336915700004

• Self-recalibrating classifiers for intracortical brain-computer interfaces. Journal of neural engineering Bishop, W., Chestek, C. C., Gilja, V., Nuyujukian, P., Foster, J. D., Ryu, S. I., Shenoy, K. V., Yu, B. M. 2014; 11 (2): 026001-?

  Abstract

  Objective. Intracortical brain-computer interface (BCI) decoders are typically retrained daily to maintain stable performance. Self-recalibrating decoders aim to remove the burden this may present in the clinic by training themselves autonomously during normal use but have only been developed for continuous control. Here we address the problem for discrete decoding (classifiers). Approach. We recorded threshold crossings from 96-electrode arrays implanted in the motor cortex of two rhesus macaques performing center-out reaches in 7 directions over 41 and 36 separate days spanning 48 and 58 days in total for offline analysis. Main results. We show that for the purposes of developing a self-recalibrating classifier, tuning parameters can be considered as fixed within days and that parameters on the same electrode move up and down together between days. Further, drift is constrained across time, which is reflected in the performance of a standard classifier which does not progressively worsen if it is not retrained daily, though overall performance is reduced by more than 10% compared to a daily retrained classifier. Two novel self-recalibrating classifiers produce a [Formula: see text] increase in classification accuracy over that achieved by the non-retrained classifier to nearly recover the performance of the daily retrained classifier. Significance. We believe that the development of classifiers that require no daily retraining will accelerate the clinical translation of BCI systems. Future work should test these results in a closed-loop setting.

  View details for DOI 10.1088/1741-2560/11/2/026001

  View details for PubMedID 24503597

• Self-recalibrating classifiers for intracortical brain-computer interfaces. Journal of neural engineering Bishop, W., Chestek, C. C., Gilja, V., Nuyujukian, P., Foster, J. D., Ryu, S. I., Shenoy, K. V., Yu, B. M. 2014; 11 (2): 026001-?

  Abstract

  Objective. Intracortical brain-computer interface (BCI) decoders are typically retrained daily to maintain stable performance. Self-recalibrating decoders aim to remove the burden this may present in the clinic by training themselves autonomously during normal use but have only been developed for continuous control. Here we address the problem for discrete decoding (classifiers). Approach. We recorded threshold crossings from 96-electrode arrays implanted in the motor cortex of two rhesus macaques performing center-out reaches in 7 directions over 41 and 36 separate days spanning 48 and 58 days in total for offline analysis. Main results. We show that for the purposes of developing a self-recalibrating classifier, tuning parameters can be considered as fixed within days and that parameters on the same electrode move up and down together between days. Further, drift is constrained across time, which is reflected in the performance of a standard classifier which does not progressively worsen if it is not retrained daily, though overall performance is reduced by more than 10% compared to a daily retrained classifier. Two novel self-recalibrating classifiers produce a [Formula: see text] increase in classification accuracy over that achieved by the non-retrained classifier to nearly recover the performance of the daily retrained classifier. Significance. We believe that the development of classifiers that require no daily retraining will accelerate the clinical translation of BCI systems. Future work should test these results in a closed-loop setting.

  View details for DOI 10.1088/1741-2560/11/2/026001

  View details for PubMedID 24503597

• Intention estimation in brain-machine interfaces. Journal of neural engineering Fan, J. M., Nuyujukian, P., Kao, J. C., Chestek, C. A., Ryu, S. I., Shenoy, K. V. 2014; 11 (1): 016004-?

  Abstract

  The objective of this work was to quantitatively investigate the mechanisms underlying the performance gains of the recently reported 'recalibrated feedback intention-trained Kalman Filter' (ReFIT-KF).This was accomplished by designing variants of the ReFIT-KF algorithm and evaluating training and online data to understand the neural basis of this improvement. We focused on assessing the contribution of two training set innovations of the ReFIT-KF algorithm: intention estimation and the two-stage training paradigm.Within the two-stage training paradigm, we found that intention estimation independently increased target acquisition rates by 37% and 59%, respectively, across two monkeys implanted with multiunit intracortical arrays. Intention estimation improved performance by enhancing the tuning properties and the mutual information between the kinematic and neural training data. Furthermore, intention estimation led to fewer shifts in channel tuning between the training set and online control, suggesting that less adaptation was required during online control. Retraining the decoder with online BMI training data also reduced shifts in tuning, suggesting a benefit of training a decoder in the same behavioral context; however, retraining also led to slower online decode velocities. Finally, we demonstrated that one- and two-stage training paradigms performed comparably when intention estimation is applied.These findings highlight the utility of intention estimation in reducing the need for adaptive strategies and improving the online performance of BMIs, helping to guide future BMI design decisions.

  View details for PubMedID 24654266

• Intention estimation in brain-machine interfaces. Journal of neural engineering Fan, J. M., Nuyujukian, P., Kao, J. C., Chestek, C. A., Ryu, S. I., Shenoy, K. V. 2014; 11 (1): 016004-?

  Abstract

  The objective of this work was to quantitatively investigate the mechanisms underlying the performance gains of the recently reported 'recalibrated feedback intention-trained Kalman Filter' (ReFIT-KF).This was accomplished by designing variants of the ReFIT-KF algorithm and evaluating training and online data to understand the neural basis of this improvement. We focused on assessing the contribution of two training set innovations of the ReFIT-KF algorithm: intention estimation and the two-stage training paradigm.Within the two-stage training paradigm, we found that intention estimation independently increased target acquisition rates by 37% and 59%, respectively, across two monkeys implanted with multiunit intracortical arrays. Intention estimation improved performance by enhancing the tuning properties and the mutual information between the kinematic and neural training data. Furthermore, intention estimation led to fewer shifts in channel tuning between the training set and online control, suggesting that less adaptation was required during online control. Retraining the decoder with online BMI training data also reduced shifts in tuning, suggesting a benefit of training a decoder in the same behavioral context; however, retraining also led to slower online decode velocities. Finally, we demonstrated that one- and two-stage training paradigms performed comparably when intention estimation is applied.These findings highlight the utility of intention estimation in reducing the need for adaptive strategies and improving the online performance of BMIs, helping to guide future BMI design decisions.

  View details for PubMedID 24654266

• Hybrid decoding of both spikes and low-frequency local field potentials for brain-machine interfaces. Conference proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual Conference Stavisky, S. D., Kao, J. C., Nuyujukian, P., Ryu, S. I., Shenoy, K. V. 2014; 2014: 3041-3044

  Abstract

  The best-performing brain-machine interfaces (BMIs) to date decode movement intention from intracortically recorded spikes, but these signals may be lost over time. A way to increase the useful lifespan of BMIs is to make more comprehensive use of available neural signals. Recent studies have demonstrated that the local field potential (LFP), a potentially more robust signal, can also be used to control a BMI. However, LFP-driven performance has fallen short of the best spikes-driven performance. Here we report a biomimetic BMI driven by low-frequency LFP that enabled a rhesus monkey to acquire and hold randomly placed targets with 99% success rate. Although LFP-driven performance was still worse than when decoding spikes, to the best of our knowledge this represents the highest-performing LFP-based BMI. We also demonstrate a new hybrid BMI that decodes cursor velocity using both spikes and LFP. This hybrid decoder improved performance over spikes-only decoding. Our results suggest that LFP can complement spikes when spikes are available or provide an alternative control signal if spikes are absent.

  View details for DOI 10.1109/EMBC.2014.6944264

  View details for PubMedID 25570632

• Hybrid Decoding of Both Spikes and Low-Frequency Local Field Potentials for Brain-Machine Interfaces Stavisky, S. D., Kao, J. C., Nuyujukian, P., Ryu, S. I., Shenoy, K. V., IEEE IEEE. 2014: 3041–44

  View details for Web of Science ID 000350044703009

• Investigating the role of firing-rate normalization and dimensionality reduction in brain-machine interface robustness. Conference proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Conference Kao, J. C., Nuyujukian, P., Stavisky, S., Ryu, S. I., Ganguli, S., Shenoy, K. V. 2013; 2013: 293-298

  Abstract

  The intraday robustness of brain-machine interfaces (BMIs) is important to their clinical viability. In particular, BMIs must be robust to intraday perturbations in neuron firing rates, which may arise from several factors including recording loss and external noise. Using a state-of-the-art decode algorithm, the Recalibrated Feedback Intention Trained Kalman filter (ReFIT-KF) [1] we introduce two novel modifications: (1) a normalization of the firing rates, and (2) a reduction of the dimensionality of the data via principal component analysis (PCA). We demonstrate in online studies that a ReFIT-KF equipped with normalization and PCA (NPC-ReFIT-KF) (1) achieves comparable performance to a standard ReFIT-KF when at least 60% of the neural variance is captured, and (2) is more robust to the undetected loss of channels. We present intuition as to how both modifications may increase the robustness of BMIs, and investigate the contribution of each modification to robustness. These advances, which lead to a decoder achieving state-of-the-art performance with improved robustness, are important for the clinical viability of BMI systems.

  View details for DOI 10.1109/EMBC.2013.6609495

  View details for PubMedID 24109682

• Design and validation of a real-time spiking-neural-network decoder for brain-machine interfaces. Journal of neural engineering Dethier, J., Nuyujukian, P., Ryu, S. I., Shenoy, K. V., Boahen, K. 2013; 10 (3): 036008-?

  Abstract

  Objective. Cortically-controlled motor prostheses aim to restore functions lost to neurological disease and injury. Several proof of concept demonstrations have shown encouraging results, but barriers to clinical translation still remain. In particular, intracortical prostheses must satisfy stringent power dissipation constraints so as not to damage cortex. Approach. One possible solution is to use ultra-low power neuromorphic chips to decode neural signals for these intracortical implants. The first step is to explore in simulation the feasibility of translating decoding algorithms for brain-machine interface (BMI) applications into spiking neural networks (SNNs). Main results. Here we demonstrate the validity of the approach by implementing an existing Kalman-filter-based decoder in a simulated SNN using the Neural Engineering Framework (NEF), a general method for mapping control algorithms onto SNNs. To measure this system's robustness and generalization, we tested it online in closed-loop BMI experiments with two rhesus monkeys. Across both monkeys, a Kalman filter implemented using a 2000-neuron SNN has comparable performance to that of a Kalman filter implemented using standard floating point techniques. Significance. These results demonstrate the tractability of SNN implementations of statistical signal processing algorithms on different monkeys and for several tasks, suggesting that a SNN decoder, implemented on a neuromorphic chip, may be a feasible computational platform for low-power fully-implanted prostheses. The validation of this closed-loop decoder system and the demonstration of its robustness and generalization hold promise for SNN implementations on an ultra-low power neuromorphic chip using the NEF.

  View details for DOI 10.1088/1741-2560/10/3/036008

  View details for PubMedID 23574919

• Design and validation of a real-time spiking-neural-network decoder for brain-machine interfaces. Journal of neural engineering Dethier, J., Nuyujukian, P., Ryu, S. I., Shenoy, K. V., Boahen, K. 2013; 10 (3): 036008-?

  Abstract

  Objective. Cortically-controlled motor prostheses aim to restore functions lost to neurological disease and injury. Several proof of concept demonstrations have shown encouraging results, but barriers to clinical translation still remain. In particular, intracortical prostheses must satisfy stringent power dissipation constraints so as not to damage cortex. Approach. One possible solution is to use ultra-low power neuromorphic chips to decode neural signals for these intracortical implants. The first step is to explore in simulation the feasibility of translating decoding algorithms for brain-machine interface (BMI) applications into spiking neural networks (SNNs). Main results. Here we demonstrate the validity of the approach by implementing an existing Kalman-filter-based decoder in a simulated SNN using the Neural Engineering Framework (NEF), a general method for mapping control algorithms onto SNNs. To measure this system's robustness and generalization, we tested it online in closed-loop BMI experiments with two rhesus monkeys. Across both monkeys, a Kalman filter implemented using a 2000-neuron SNN has comparable performance to that of a Kalman filter implemented using standard floating point techniques. Significance. These results demonstrate the tractability of SNN implementations of statistical signal processing algorithms on different monkeys and for several tasks, suggesting that a SNN decoder, implemented on a neuromorphic chip, may be a feasible computational platform for low-power fully-implanted prostheses. The validation of this closed-loop decoder system and the demonstration of its robustness and generalization hold promise for SNN implementations on an ultra-low power neuromorphic chip using the NEF.

  View details for DOI 10.1088/1741-2560/10/3/036008

  View details for PubMedID 23574919

• Neural population dynamics during reaching NATURE Churchland, M. M., Cunningham, J. P., Kaufman, M. T., Foster, J. D., Nuyujukian, P., Ryu, S. I., Shenoy, K. V. 2012; 487 (7405): 51-?

  Abstract

  Most theories of motor cortex have assumed that neural activity represents movement parameters. This view derives from what is known about primary visual cortex, where neural activity represents patterns of light. Yet it is unclear how well the analogy between motor and visual cortex holds. Single-neuron responses in motor cortex are complex, and there is marked disagreement regarding which movement parameters are represented. A better analogy might be with other motor systems, where a common principle is rhythmic neural activity. Here we find that motor cortex responses during reaching contain a brief but strong oscillatory component, something quite unexpected for a non-periodic behaviour. Oscillation amplitude and phase followed naturally from the preparatory state, suggesting a mechanistic role for preparatory neural activity. These results demonstrate an unexpected yet surprisingly simple structure in the population response. This underlying structure explains many of the confusing features of individual neural responses.

  View details for DOI 10.1038/nature11129

  View details for Web of Science ID 000305982900048

  View details for PubMedID 22722855

  View details for PubMedCentralID PMC3393826

• A recurrent neural network for closed-loop intracortical brain-machine interface decoders JOURNAL OF NEURAL ENGINEERING Sussillo, D., Nuyujukian, P., Fan, J. M., Kao, J. C., Stavisky, S. D., Ryu, S., Shenoy, K. 2012; 9 (2)

  Abstract

  Recurrent neural networks (RNNs) are useful tools for learning nonlinear relationships in time series data with complex temporal dependences. In this paper, we explore the ability of a simplified type of RNN, one with limited modifications to the internal weights called an echostate network (ESN), to effectively and continuously decode monkey reaches during a standard center-out reach task using a cortical brain-machine interface (BMI) in a closed loop. We demonstrate that the RNN, an ESN implementation termed a FORCE decoder (from first order reduced and controlled error learning), learns the task quickly and significantly outperforms the current state-of-the-art method, the velocity Kalman filter (VKF), using the measure of target acquire time. We also demonstrate that the FORCE decoder generalizes to a more difficult task by successfully operating the BMI in a randomized point-to-point task. The FORCE decoder is also robust as measured by the success rate over extended sessions. Finally, we show that decoded cursor dynamics are more like naturalistic hand movements than those of the VKF. Taken together, these results suggest that RNNs in general, and the FORCE decoder in particular, are powerful tools for BMI decoder applications.

  View details for DOI 10.1088/1741-2560/9/2/026027

  View details for Web of Science ID 000302144100027

  View details for PubMedID 22427488

  View details for PubMedCentralID PMC3638090

• HermesE: A 96-Channel Full Data Rate Direct Neural Interface in 0.13 mu m CMOS IEEE JOURNAL OF SOLID-STATE CIRCUITS Gao, H., Walker, R. M., Nuyujukian, P., Makinwa, K. A., Shenoy, K. V., Murmann, B., Meng, T. H. 2012; 47 (4): 1043-1055

  View details for DOI 10.1109/JSSC.2012.2185338

  View details for Web of Science ID 000302494700023

• A framework for relating neural activity to freely moving behavior 34th Annual International Conference of the IEEE Engineering-in-Medicine-and-Biology-Society (EMBS) Foster, J. D., Nuyujukian, P., Freifeld, O., Ryu, S. I., Black, M. J., Shenoy, K. V. IEEE. 2012: 2736–2739

  Abstract

  Two research communities, motor systems neuroscience and motor prosthetics, examine the relationship between neural activity in the motor cortex and movement. The former community aims to understand how the brain controls and generates movement; the latter community focuses on how to decode neural activity as control signals for a prosthetic cursor or limb. Both have made progress toward understanding the relationship between neural activity in the motor cortex and behavior. However, these findings are tested using animal models in an environment that constrains behavior to simple, limited movements. These experiments show that, in constrained settings, simple reaching motions can be decoded from small populations of spiking neurons. It is unclear whether these findings hold for more complex, full-body behaviors in unconstrained settings. Here we present the results of freely-moving behavioral experiments from a monkey with simultaneous intracortical recording. We investigated neural firing rates while the monkey performed various tasks such as walking on a treadmill, reaching for food, and sitting idly. We show that even in such an unconstrained and varied context, neural firing rates are well tuned to behavior, supporting findings of basic neuroscience. Further, we demonstrate that the various behavioral tasks can be reliably classified with over 95% accuracy, illustrating the viability of decoding techniques despite significant variation and environmental distractions associated with unconstrained behavior. Such encouraging results hint at potential utility of the freely-moving experimental paradigm.

  View details for Web of Science ID 000313296502238

  View details for PubMedID 23366491

• A brain machine interface control algorithm designed from a feedback control perspective 34th Annual International Conference of the IEEE Engineering-in-Medicine-and-Biology-Society (EMBS) Gilja, V., Nuyujukian, P., Chestek, C. A., Cunningham, J. P., Yu, B. M., Fan, J. M., Ryu, S. I., Shenoy, K. V. IEEE. 2012: 1318–1322

  Abstract

  We present a novel brain machine interface (BMI) control algorithm, the recalibrated feedback intention-trained Kalman filter (ReFIT-KF). The design of ReFIT-KF is motivated from a feedback control perspective applied to existing BMI control algorithms. The result is two design innovations that alter the modeling assumptions made by these algorithms and the methods by which these algorithms are trained. In online neural control experiments recording from a 96-electrode array implanted in M1 of a macaque monkey, the ReFIT-KF control algorithm demonstrates large performance improvements over the current state of the art velocity Kalman filter, reducing target acquisition time by a factor of two, while maintaining a 500 ms hold period, thereby increasing the clinical viability of BMI systems.

  View details for Web of Science ID 000313296501144

  View details for PubMedID 23366141

• Long-term stability of neural prosthetic control signals from silicon cortical arrays in rhesus macaque motor cortex 39th Neural Interfaces Conference (NIC2010) Chestek, C. A., Gilja, V., Nuyujukian, P., Foster, J. D., Fan, J. M., Kaufman, M. T., Churchland, M. M., Rivera-Alvidrez, Z., Cunningham, J. P., Ryu, S. I., Shenoy, K. V. IOP PUBLISHING LTD. 2011

  Abstract

  Cortically-controlled prosthetic systems aim to help disabled patients by translating neural signals from the brain into control signals for guiding prosthetic devices. Recent reports have demonstrated reasonably high levels of performance and control of computer cursors and prosthetic limbs, but to achieve true clinical viability, the long-term operation of these systems must be better understood. In particular, the quality and stability of the electrically-recorded neural signals require further characterization. Here, we quantify action potential changes and offline neural decoder performance over 382 days of recording from four intracortical arrays in three animals. Action potential amplitude decreased by 2.4% per month on average over the course of 9.4, 10.4, and 31.7 months in three animals. During most time periods, decoder performance was not well correlated with action potential amplitude (p > 0.05 for three of four arrays). In two arrays from one animal, action potential amplitude declined by an average of 37% over the first 2 months after implant. However, when using simple threshold-crossing events rather than well-isolated action potentials, no corresponding performance loss was observed during this time using an offline decoder. One of these arrays was effectively used for online prosthetic experiments over the following year. Substantial short-term variations in waveforms were quantified using a wireless system for contiguous recording in one animal, and compared within and between days for all three animals. Overall, this study suggests that action potential amplitude declines more slowly than previously supposed, and performance can be maintained over the course of multiple years when decoding from threshold-crossing events rather than isolated action potentials. This suggests that neural prosthetic systems may provide high performance over multiple years in human clinical trials.

  View details for DOI 10.1088/1741-2560/8/4/045005

  View details for Web of Science ID 000292962800009

  View details for PubMedID 21775782

  View details for PubMedCentralID PMC3644617

• A closed-loop human simulator for investigating the role of feedback control in brain-machine interfaces JOURNAL OF NEUROPHYSIOLOGY Cunningham, J. P., Nuyujukian, P., Gilja, V., Chestek, C. A., Ryu, S. I., Shenoy, K. V. 2011; 105 (4): 1932-1949

  Abstract

  Neural prosthetic systems seek to improve the lives of severely disabled people by decoding neural activity into useful behavioral commands. These systems and their decoding algorithms are typically developed "offline," using neural activity previously gathered from a healthy animal, and the decoded movement is then compared with the true movement that accompanied the recorded neural activity. However, this offline design and testing may neglect important features of a real prosthesis, most notably the critical role of feedback control, which enables the user to adjust neural activity while using the prosthesis. We hypothesize that understanding and optimally designing high-performance decoders require an experimental platform where humans are in closed-loop with the various candidate decode systems and algorithms. It remains unexplored the extent to which the subject can, for a particular decode system, algorithm, or parameter, engage feedback and other strategies to improve decode performance. Closed-loop testing may suggest different choices than offline analyses. Here we ask if a healthy human subject, using a closed-loop neural prosthesis driven by synthetic neural activity, can inform system design. We use this online prosthesis simulator (OPS) to optimize "online" decode performance based on a key parameter of a current state-of-the-art decode algorithm, the bin width of a Kalman filter. First, we show that offline and online analyses indeed suggest different parameter choices. Previous literature and our offline analyses agree that neural activity should be analyzed in bins of 100- to 300-ms width. OPS analysis, which incorporates feedback control, suggests that much shorter bin widths (25-50 ms) yield higher decode performance. Second, we confirm this surprising finding using a closed-loop rhesus monkey prosthetic system. These findings illustrate the type of discovery made possible by the OPS, and so we hypothesize that this novel testing approach will help in the design of prosthetic systems that will translate well to human patients.

  View details for DOI 10.1152/jn.00503.2010

  View details for Web of Science ID 000289620500044

  View details for PubMedID 20943945

  View details for PubMedCentralID PMC3075301

• Combining Wireless Neural Recording and Video Capture for the Analysis of Natural Gait. International IEEE/EMBS Conference on Neural Engineering : [proceedings]. International IEEE EMBS Conference on Neural Engineering Foster, J. D., Freifeld, O., Nuyujukian, P., Ryu, S. I., Black, M. J., Shenoy, K. V. 2011; 2011: 613–16

  Abstract

  Neural control of movement is typically studied in constrained environments where there is a reduced set of possible behaviors. This constraint may unintentionally limit the applicability of findings to the generalized case of unconstrained behavior. We hypothesize that examining the unconstrained state across multiple behavioral contexts will lead to new insights into the neural control of movement and help advance the design of neural prosthetic decode algorithms. However, to pursue electrophysiological studies in such a manner requires a more flexible framework for experimentation. We propose that head-mounted neural recording systems with wireless data transmission, combined with markerless computer-vision based motion tracking, will enable new, less constrained experiments. As a proof-of-concept, we recorded and wirelessly transmitted broadband neural data from 32 electrodes in premotor cortex while acquiring single-camera video of a rhesus macaque walking on a treadmill. We demonstrate the ability to extract behavioral kinematics using an automated computer vision algorithm without use of markers and to predict kinematics from the neural data. Together these advances suggest that a new class of "freely moving monkey" experiments should be possible and should help broaden our understanding of the neural control of movement.

  View details for PubMedID 26019730

• Monkey Models for Brain-Machine Interfaces: The Need for Maintaining Diversity 33rd Annual International Conference of the IEEE Engineering-in-Medicine-and-Biology-Society (EMBS) Nuyujukian, P., Fan, J. M., Gilja, V., Kalanithi, P. S., Chestek, C. A., Shenoy, K. V. IEEE. 2011: 1301–1305

  Abstract

  Brain-machine interfaces (BMIs) aim to help disabled patients by translating neural signals from the brain into control signals for guiding prosthetic arms, computer cursors, and other assistive devices. Animal models are central to the development of these systems and have helped enable the successful translation of the first generation of BMIs. As we move toward next-generation systems, we face the question of which animal models will aid broader patient populations and achieve even higher performance, robustness, and functionality. We review here four general types of rhesus monkey models employed in BMI research, and describe two additional, complementary models. Given the physiological diversity of neurological injury and disease, we suggest a need to maintain the current diversity of animal models and to explore additional alternatives, as each mimic different aspects of injury or disease.

  View details for Web of Science ID 000298810001110

  View details for PubMedID 22254555

• Spiking Neural Network Decoder for Brain-Machine Interfaces. International IEEE/EMBS Conference on Neural Engineering : [proceedings]. International IEEE EMBS Conference on Neural Engineering Dethier, J., Gilja, V., Nuyujukian, P., Elassaad, S. A., Shenoy, K. V., Boahen, K. 2011

  Abstract

  We used a spiking neural network (SNN) to decode neural data recorded from a 96-electrode array in premotor/motor cortex while a rhesus monkey performed a point-to-point reaching arm movement task. We mapped a Kalman-filter neural prosthetic decode algorithm developed to predict the arm's velocity on to the SNN using the Neural Engineering Framework and simulated it using Nengo, a freely available software package. A 20,000-neuron network matched the standard decoder's prediction to within 0.03% (normalized by maximum arm velocity). A 1,600-neuron version of this network was within 0.27%, and run in real-time on a 3GHz PC. These results demonstrate that a SNN can implement a statistical signal processing algorithm widely used as the decoder in high-performance neural prostheses (Kalman filter), and achieve similar results with just a few thousand neurons. Hardware SNN implementations-neuromorphic chips-may offer power savings, essential for realizing fully-implantable cortically controlled prostheses.

  View details for DOI 10.1109/NER.2011.5910570

  View details for PubMedID 24352611

• A Brain-Machine Interface Operating with a Real-Time Spiking Neural Network Control Algorithm. Advances in neural information processing systems Dethier, J., Nuyujukian, P., Eliasmith, C., Stewart, T., Elassaad, S. A., Shenoy, K. V., Boahen, K. 2011; 2011: 2213-2221

  Abstract

  Motor prostheses aim to restore function to disabled patients. Despite compelling proof of concept systems, barriers to clinical translation remain. One challenge is to develop a low-power, fully-implantable system that dissipates only minimal power so as not to damage tissue. To this end, we implemented a Kalman-filter based decoder via a spiking neural network (SNN) and tested it in brain-machine interface (BMI) experiments with a rhesus monkey. The Kalman filter was trained to predict the arm's velocity and mapped on to the SNN using the Neural Engineering Framework (NEF). A 2,000-neuron embedded Matlab SNN implementation runs in real-time and its closed-loop performance is quite comparable to that of the standard Kalman filter. The success of this closed-loop decoder holds promise for hardware SNN implementations of statistical signal processing algorithms on neuromorphic chips, which may offer power savings necessary to overcome a major obstacle to the successful clinical translation of neural motor prostheses.

  View details for PubMedID 25309106

  View details for PubMedCentralID PMC4190036

• Spiking Neural Network Decoder for Brain-Machine Interfaces 5th International IEEE Engineering-in-Medicine-and-Biology-Society (EMBS) Conference on Neural Engineering (NER) Dethier, J., Gilja, V., Nuyujukian, P., Elassaad, S. A., Shenoy, K. V., Boahen, K. IEEE. 2011: 396–399

  View details for Web of Science ID 000298735800094

• Combining Wireless Neural Recording and Video Capture for the Analysis of Natural Gait 5th International IEEE Engineering-in-Medicine-and-Biology-Society (EMBS) Conference on Neural Engineering (NER) Foster, J. D., Freifeld, O., Nuyujukian, P., Ryu, S. I., Black, M. J., Shenoy, K. V. IEEE. 2011: 613–616

  View details for Web of Science ID 000298735800147

• Autonomous head-mounted electrophysiology systems for freely behaving primates CURRENT OPINION IN NEUROBIOLOGY Gilja, V., Chestek, C. A., Nuyujukian, P., Foster, J., Shenoy, K. V. 2010; 20 (5): 676-686

  Abstract

  Recent technological advances have led to new light-weight battery-operated systems for electrophysiology. Such systems are head mounted, run for days without experimenter intervention, and can record and stimulate from single or multiple electrodes implanted in a freely behaving primate. Here we discuss existing systems, studies that use them, and how they can augment traditional, physically restrained, 'in-rig' electrophysiology. With existing technical capabilities, these systems can acquire multiple signal classes, such as spikes, local field potential, and electromyography signals, and can stimulate based on real-time processing of recorded signals. Moving forward, this class of technologies, along with advances in neural signal processing and behavioral monitoring, have the potential to dramatically expand the scope and scale of electrophysiological studies.

  View details for DOI 10.1016/j.conb.2010.06.007

  View details for Web of Science ID 000283481100022

  View details for PubMedID 20655733

  View details for PubMedCentralID PMC3401169

• Embedded Neural Recording With TinyOS-Based Wireless-Enabled Processor Modules IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING Farshchi, S., Pesterev, A., Nuyujukian, P., Guenterberg, E., Mody, I., Judy, J. W. 2010; 18 (2): 134-141

  Abstract

  To create a wireless neural recording system that can benefit from the continuous advancements being made in embedded microcontroller and communications technologies, an embedded-system-based architecture for wireless neural recording has been designed, fabricated, and tested. The system consists of commercial-off-the-shelf wireless-enabled processor modules (motes) for communicating the neural signals, and a back-end database server and client application for archiving and browsing the neural signals. A neural-signal-acquisition application has been developed to enable the mote to either acquire neural signals at a rate of 4000 12-bit samples per second, or detect and transmit spike heights and widths sampled at a rate of 16670 12-bit samples per second on a single channel. The motes acquire neural signals via a custom low-noise neural-signal amplifier with adjustable gain and high-pass corner frequency that has been designed, and fabricated in a 1.5-microm CMOS process. In addition to browsing acquired neural data, the client application enables the user to remotely toggle modes of operation (real-time or spike-only), as well as amplifier gain and high-pass corner frequency.

  View details for DOI 10.1109/TNSRE.2009.2039606

  View details for Web of Science ID 000277184700005

  View details for PubMedID 20071270

• Wireless Neural Recording With Single Low-Power Integrated Circuit IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING Harrison, R. R., Kier, R. J., Chestek, C. A., Gilja, V., Nuyujukian, P., Ryu, S., Greger, B., Solzbacher, F., Shenoy, K. V. 2009; 17 (4): 322-329

  Abstract

  We present benchtop and in vivo experimental results from an integrated circuit designed for wireless implantable neural recording applications. The chip, which was fabricated in a commercially available 0.6- mum 2P3M BiCMOS process, contains 100 amplifiers, a 10-bit analog-to-digital converter (ADC), 100 threshold-based spike detectors, and a 902-928 MHz frequency-shift-keying (FSK) transmitter. Neural signals from a selected amplifier are sampled by the ADC at 15.7 kSps and telemetered over the FSK wireless data link. Power, clock, and command signals are sent to the chip wirelessly over a 2.765-MHz inductive (coil-to-coil) link. The chip is capable of operating with only two off-chip components: a power/command receiving coil and a 100-nF capacitor.

  View details for DOI 10.1109/TNSRE.2009.2023298

  View details for Web of Science ID 000268900300003

  View details for PubMedID 19497825

  View details for PubMedCentralID PMC2941647

• HermesC: Low-Power Wireless Neural Recording System for Freely Moving Primates IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING Chestek, C. A., Gilja, V., Nuyujukian, P., Kier, R. J., Solzbacher, F., Ryu, S. I., Harrison, R. R., Shenoy, K. V. 2009; 17 (4): 330-338

  Abstract

  Neural prosthetic systems have the potential to restore lost functionality to amputees or patients suffering from neurological injury or disease. Current systems have primarily been designed for immobile patients, such as tetraplegics functioning in a rather static, carefully tailored environment. However, an active patient such as amputee in a normal dynamic, everyday environment may be quite different in terms of the neural control of movement. In order to study motor control in a more unconstrained natural setting, we seek to develop an animal model of freely moving humans. Therefore, we have developed and tested HermesC-INI3, a system for recording and wirelessly transmitting neural data from electrode arrays implanted in rhesus macaques who are freely moving. This system is based on the integrated neural interface (INI3) microchip which amplifies, digitizes, and transmits neural data across a approximately 900 MHz wireless channel. The wireless transmission has a range of approximately 4 m in free space. All together this device consumes 15.8 mA and 63.2 mW. On a single 2 A-hr battery pack, this device runs contiguously for approximately six days. The smaller size and power consumption of the custom IC allows for a smaller package (51 x 38 x 38 mm (3)) than previous primate systems. The HermesC-INI3 system was used to record and telemeter one channel of broadband neural data at 15.7 kSps from a monkey performing routine daily activities in the home cage.

  View details for DOI 10.1109/TNSRE.2009.2023293

  View details for Web of Science ID 000268900300004

  View details for PubMedID 19497829

• Neural Prosthetic Systems: Current Problems and Future Directions Annual International Conference of the IEEE-Engineering-in-Medicine-and-Biology-Society Chestek, C. A., Cunningham, J. P., Gilja, V., Nuyujukian, P., Ryu, S. I., Shenoy, K. V. IEEE. 2009: 3369–3375

  Abstract

  By decoding neural activity into useful behavioral commands, neural prosthetic systems seek to improve the lives of severely disabled human patients. Motor decoding algorithms, which map neural spiking data to control parameters of a device such as a prosthetic arm, have received particular attention in the literature. Here, we highlight several outstanding problems that exist in most current approaches to decode algorithm design. These include two problems that we argue will unlikely result in further dramatic increases in performance, specifically spike sorting and spiking models. We also discuss three issues that have been less examined in the literature, and we argue that addressing these issues may result in dramatic future increases in performance. These include: non-stationarity of recorded waveforms, limitations of a linear mappings between neural activity and movement kinematics, and the low signal to noise ratio of the neural data. We demonstrate these problems with data from 39 experimental sessions with a non-human primate performing reaches and with recent literature. In all, this study suggests that research in cortically-controlled prosthetic systems may require reprioritization to achieve performance that is acceptable for a clinically viable human system.

  View details for Web of Science ID 000280543602209

  View details for PubMedID 19963796

• HermesC: RF wireless low-power neural recording system for freely behaving primates IEEE International Symposium on Circuits and Systems Chestek, C. A., Gija, V., Nuyujukian, P., Ryu, S. I., Shenoy, K. V., Kier, R. J., Solzbacher, F., Harrison, R. R. IEEE. 2008: 1752–1755

  View details for Web of Science ID 000258532101169

• A Wireless Neural Interface for Chronic Recording IEEE Biomedical Circuits and Systems Conference - Intelligent Biomedical Systems Harrison, R. R., Kier, R. J., Kim, S., Rieth, L., Warren, D. J., Ledbetter, N. M., Clark, G. A., Solzbacher, F., Chestek, C. A., Gilja, V., Nuyujukian, P., Ryu, S. I., Shenoy, K. V. IEEE. 2008: 125–128

  View details for Web of Science ID 000264876700033

• Wireless neural signal acquisition with single low-power integrated circuit IEEE International Symposium on Circuits and Systems Harrison, R. R., Kier, R. J., Greger, B., Solzbacher, F., Chestek, C. A., Gija, V., Nuyujukian, P., Ryu, S. I., Shenoy, K. V. IEEE. 2008: 1748–1751

  View details for Web of Science ID 000258532101168

• Bi-Fi: An embedded sensor/system architecture for remote biological monitoring IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE Farshchi, S., Pesterev, A., Nuyujukian, P. H., Mody, I., Judy, J. W. 2007; 11 (6): 611-618

  Abstract

  Wireless-enabled processor modules intended for communicating low-frequency phenomena (i.e., temperature, humidity, and ambient light) have been enabled to acquire and transmit multiple biological signals in real time, which has been achieved by using computationally efficient data acquisition, filtering, and compression algorithms, and interfacing the modules with biological interface hardware. The sensor modules can acquire and transmit raw biological signals at a rate of 32 kb/s, which is near the hardware limit of the modules. Furthermore, onboard signal processing enables one channel, sampled at a rate of 4000 samples/s at 12-bit resolution, to be compressed via adaptive differential-pulse-code modulation (ADPCM) and transmitted in real time. In addition, the sensors can be configured to filter and transmit individual time-referenced "spike" waveforms, or to transmit the spike height and width for alleviating network traffic and increasing battery life. The system is capable of acquiring eight channels of analog signals as well as data via an asynchronous serial connection. A back-end server archives the biological data received via networked gateway sensors, and hosts them to a client application that enables users to browse recorded data. The system also acquires, filters, and transmits oxygen saturation and pulse rate via a commercial-off-the-shelf interface board. The system architecture can be configured for performing real-time nonobtrusive biological monitoring of humans or rodents. This paper demonstrates that low-power, computational, and bandwidth-constrained wireless-enabled platforms can indeed be leveraged for wireless biosignal monitoring.

  View details for DOI 10.1109/TITB.2007.897600

  View details for Web of Science ID 000250929100002

  View details for PubMedID 18046936

• A TinyOS-enabled MICA2-based wireless neural interface IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING Farshchi, S., Nuyujukian, P. H., Pesterev, A., Mody, I., Judy, J. W. 2006; 53 (7): 1416-1424

  Abstract

  Existing approaches used to develop compact low-power multichannel wireless neural recording systems range from creating custom-integrated circuits to assembling commercial-off-the-shelf (COTS) PC-based components. Custom-integrated-circuit designs yield extremely compact and low-power devices at the expense of high development and upgrade costs and turn-around times, while assembling COTS-PC-technology yields high performance at the expense of large system size and increased power consumption. To achieve a balance between implementing an ultra-compact custom-fabricated neural transceiver and assembling COTS-PC-technology, an overlay of a neural interface upon the TinyOS-based MICA2 platform is described. The system amplifies, digitally encodes, and transmits neural signals real-time at a rate of 9.6 kbps, while consuming less than 66 mW of power. The neural signals are received and forwarded to a client PC over a serial connection. This data rate can be divided for recording on up to 6 channels, with a resolution of 8 bits/sample. This work demonstrates the strengths and limitations of the TinyOS-based sensor technology as a foundation for chronic remote biological monitoring applications and, thus, provides an opportunity to create a system that can leverage from the frequent networking and communications advancements being made by the global TinyOS-development community.

  View details for DOI 10.1109/TMBE.2006.873760

  View details for Web of Science ID 000238711600021

  View details for PubMedID 16830946

• A TinyOS-based wireless neural sensing, archiving, and hosting system 2nd International IEEE/EMBS Conference on Neural Engineering Farshchi, S., Nuyujukian, P. H., Pesterev, A., Mody, I., Judy, J. W. IEEE. 2005: 671–674

  View details for Web of Science ID 000229610400179