⚠️ This work is in early development and changing rapidly. It is not ready for scientific use.⚠️
Our ultimate goal is to facilitate the creation of fully open, reproducible, OS-independent, browser-based workflows for biomedical research primarily using sustainable, domain-independent visualization tools. In support of this goal, this repository is the development ground for optimization and demonstration of HoloViz and Bokeh tools within the realm of neuroscience.
Urgent objectives:
- Workflow Development: Host the development of workflows.
- Code Sharing: Promote consistency and facilitate sharing of code across different workflows.
- Collaboration: Foster collaborative efforts between the HoloViz+Bokeh development teams and scientific collaborators outside these groups. This cross-collaboration aims to effectively tailor the tools to the specific requirements of the neuroscience community.
- Issue Identification and Resolution: As part of ongoing development, identify and address any performance or user interface bottlenecks in the workflows to optimize their usage and effectiveness.
- Benchmarking and Testing Integration: Host benchmarking work that involves the use of real and simulated data to assess the performance and functionality of the tools under relevant conditions.
- Improvement and Refinement: Over time, enhance, improve, and refine the developed workflows based on user feedback and advancements in the field.
- Dissemination: Eventually, share workflows with the broader scientific community. It's unclear yet where these all will be showcased, but at least some will go to examples.holoviz.org.
- Education and Community Building: Undertake educational and community-building activities such as providing tutorials, workshops, other educational resources to help researchers effectively utilize the developed tools.
- Host Domain-Specific Package: It is possible that not all required code for workflows will be accepted or appropriate for integrations into domain-independent HoloViz/Bokeh packages. Therefore, this repo might end up hosting code to be packaged as a domain-specific extension. TBD!
Roadmap:
- High-level summary: Our current grant period is through 2024, but we want to have a first pass of prioritized improvements for generalized workflows to disseminate for feedback within Q4 2023. The remainder of Q4 2023 and all of 2024 will be for iterating on feedback, developing the specialized workflows, demonstrating biomedical use-cases, collaborating lab support, educational activities, and as time permits - wishlist features and new collaborations.
- A living task-goal roadmap is visible on this project board view - currently through Q3 and early Q4 2023.
This repository contains developmental versions of workflows, which can be categorized into two types: generalized and specialized. Generalized workflows aim to be broadly applicable and primarily utilize domain-independent tools such as Numpy, Pandas, Xarray, and others. These generalized workflows serve as the foundational building blocks for specialized workflows. On the other hand, specialized workflows are designed to cater to specific contexts and have no limitations on the use of domain-specific tools like MNE, Minian, and more.
Generalized Workflows:
| Title | Modality | Thumbnail | Info & Links | Description |
|---|---|---|---|---|
| Stacked Timeseries | eeg, ephys |  readme workflow |
Synchronized examination of stacked time-series with large data handling, scale bar, annotations, minimap, and signal grouping. | |
| eeg | readme workflow |
Synchronized examination of EEG with stacked time-series, large data handling, scale bar, annotations, minimap, and signal grouping. | ||
| Video Viewer | calcium imaging | readme workflow |
Efficient visualization of large Miniscope calcium imaging movies with, playback controls, 2D annotation, scale bar, time views, intensity histogram, and summary statistics. | |
| ephys | readme workflow |
Synchronized examination of multielectrode intracranial extracellular electrophysiology (ephys) with all the relevant goodies of the EEG viewer. | ||
| Waveform | ephys | readme workflow |
Oscilloscope-style display of action potential waveform snippets | |
| Spike Raster | ephys | readme workflow |
Efficient visualization of large-scale neuronal spike time data, with a simple API, aggregate views of spike counts, and spike-level metadata management |
- Multimodal - visualizing and aligning ca-imaging with simultaneously recorded (but
differently sampled) timeseries like EEG, EMG, and/or behavior. Alternatively,
visualizing behavioral video (eye tracking, maze running) with timeseries data.
- Linked eeg-sensor layout
- Linked ephys-sensor layout
Specialized Workflows:
- Spike Motif
- MNE Raw
- Minian CNMF
Incubation/Wishlist:
- General: Streaming data - extend the ephys, eeg, and/or video viewer workflows to
display live streaming data.
- General: Videos/Timeseries recorder
- Specialized: CNMF Temporal update parameter exploration app long timeseries
improvement
Before installing the workflow environments, make sure you have Miniconda installed. If not, you can download and install it from the official site.
-
Clone the Repository: Clone the
neurorepository to your local machine.git clone https://github.com/holoviz-topics/neuro.git
-
Navigate to Workflow: Change to the directory of the workflow you're interested in.
cd neuro/workflows/eeg_viewer -
Create Environment: Use
condato create a new environment from theenvironment.ymlfile.conda env create -f environment.yml
-
Activate Environment: After the environment is created, activate it.
conda activate eeg_viewer
Replace eeg_viewer with the appropriate workflow name for different workflows.
If you've already installed a workflow environment and the environment.yml file has been updated, follow these steps to update the environment:
-
Update Repository: Pull the latest changes from the repository.
git pull
-
Navigate to Workflow: Go to the directory of the workflow you're interested in.
cd neuro/workflows/eeg_viewer -
Update Environment: Update the existing Conda environment based on the latest
environment.ymlfile.conda env update -f environment.yml --prune
The --prune option will remove packages from the environment not present in the updated environment.yml file.
Replace eeg_viewer with the appropriate workflow name for different workflows.
- Task Management: As workflows are developed and honed, performance and UI bottlenecks will be identified and addressed. Some improvements for the workflows themselves will be within this repo, but many improvements will be in the appropriate underlying libraries within the HoloViz, Bokeh, or other GitHub Organizations. We will do our best to track the disparate tasks related to these efforts into this
project board.
- Abstracted project board tasks prefixed with 'GOAL:' are for roadmap generation and hours estimation.
- Communication:
- Meeting minutes: Logged in the Wiki > Meeting Notes whenever possible.
- HoloViz Discord #neuro channel for real-time chat
- holoviz-topics/neuro GitHub repo issue tracker
- Specifications: The Wiki has some data specifications and modality notes (in progress).
- Data Generation: To assist the development using real data, some workflows utilize simple data generators to help benchmark across data and parameter space. As the data generators/simulators can be useful to multiple workflows, they are kept as a separate and importable module (
/src/neurodatagen). - Visualization source code: If there is visualization code or utilities that we want to live separate from the individual workflows, we can store them in
/src/hvneurofor now. However, it's unclear whether this will be released as a new package, incorporated into existing libraries, or live in particular workflows. TBD - Repo Structure and dev patterns:
/workflows /example1 readme_example1.md workflow_example1.ipynb environment.yml /dev date_example-workflow_task.ipynb- Use
readme_<workflow>.mdfor any essential workflow-specific info or links. - Maintain
workflow_<workflow>.ipynbas the latest version of the workflow. - Each workflow should have an
environment.ymlwith which to create a conda env that will install theneurodatagenmodule in dev mode. - Use the
devdir in each workflow as shared scratch space within themainbranch. There is no expectation that anything here is maintained.
- Use
This work is a collaboration between developers and scientists, and some developer-scientists. While some contributions are visible through the GitHub repo, many other contributions are less visible yet equally important.
Funding:
- 2023 - 2024: Chan Zuckerberg Initiative. Learn more in the grant announcement.
- Project Lead: Demetris Roumis (@droumis on Discord)
Multiple (probably all) HoloViz+Bokeh developers believe that helping people through the furthering of clinically impactful science is a worthy pursuit and in need of a data visualization boost.
We hypothesize that the visualization within the process of working always benefits from having the option to suddenly become interactive and shareable - allowing for the poking or plucking, pushing or pulling, drilling in or out, grouping or separating, and sending or receiving of what would otherwise be a static snapshot of the data. The combined use of HoloViz and Bokeh tools provides the interactivity and shareability needed to support research as a collective action rather than a collection of solitary observations.