Obtaining Datasets with DANDI#
DANDI is an open source data archive for neuroscience datasets, called Dandisets. DANDI allows scientists to submit and download neural datasets to promote research collaboration and consistent and transparent data standards. DANDI also provides a solution to the difficulties that come from housing data in the many other general domains (i.e. Dropbox, Google Drive, etc.). Usefully for our purposes here, many of the datasets on DANDI are in NWB format. If you’d like to know more about DANDI, check out the DANDI handbook.
There are two primary ways to work with Dandisets:
You can download the datasets, either via the DANDI Web Application or using the DANDI Python client below. If you download via the website, you’ll need to create an account.
You can stream datasets directly from DANDI. We’ll show you how to do this online as well as on your local computer.
Below, we demonstrate how to do both of these. For additional information on either of these methods, please refer to the DANDI documentation.
Option 1: Downloading Dandisets using Python#
The cell below will download this dataset from DANDI. This dataset contains 32-channel extracellular recordings from mouse cortex. We’re using the download tool from dandi below.
Note: Downloading this dataset may take several minutes, depending on your internet connection.
Note #2: This step is only possible if you’re running this on the Dandihub or if you have completed the setup steps.
from dandi.download import download as dandi_download
import os
# Set the URL for the DANDI file
url = 'https://dandiarchive.org/dandiset/000006/draft'
# Download the file into the current working directory
# It will skip downloading any files you've already downloaded
dandi_download([url], output_dir = os.getcwd(), existing = "skip")
PATH SIZE DONE DONE% CHECKSUM STATUS MESSAGE
000006/dandiset.yaml skipped no change
000006/sub-anm369962/sub-anm369962_ses-20170309.nwb skipped already exists
000006/sub-anm369962/sub-anm369962_ses-20170316.nwb skipped already exists
000006/sub-anm369962/sub-anm369962_ses-20170310.nwb skipped already exists
000006/sub-anm369962/sub-anm369962_ses-20170314.nwb skipped already exists
000006/sub-anm369962/sub-anm369962_ses-20170313.nwb skipped already exists
000006/sub-anm369962/sub-anm369962_ses-20170317.nwb skipped already exists
000006/sub-anm369963/sub-anm369963_ses-20170227.nwb skipped already exists
000006/sub-anm369963/sub-anm369963_ses-20170226.nwb skipped already exists
000006/sub-anm369963/sub-anm369963_ses-20170228.nwb skipped already exists
000006/sub-anm369963/sub-anm369963_ses-20170301.nwb skipped already exists
000006/sub-anm369963/sub-anm369963_ses-20170302.nwb skipped already exists
000006/sub-anm369963/sub-anm369963_ses-20170306.nwb skipped already exists
000006/sub-anm369963/sub-anm369963_ses-20170309.nwb skipped already exists
000006/sub-anm369963/sub-anm369963_ses-20170310.nwb skipped already exists
000006/sub-anm369964/sub-anm369964_ses-20170321.nwb skipped already exists
000006/sub-anm369964/sub-anm369964_ses-20170320.nwb 7.7 MB 7.7 MB 100% ok done
000006/sub-anm369964/sub-anm369964_ses-20170322.nwb 5.0 MB 5.0 MB 100% ok done
000006/sub-anm369964/sub-anm369964_ses-20170323.nwb 426.4 kB 426.4 kB 100% ok done
000006/sub-anm372793/sub-anm372793_ses-20170504.nwb 357.8 kB 357.8 kB 100% ok done
000006/sub-anm372793/sub-anm372793_ses-20170508.nwb 357.4 kB 357.4 kB 100% ok done
000006/sub-anm372793/sub-anm372793_ses-20170512.nwb 470.8 kB 470.8 kB 100% ok done
000006/sub-anm372793/sub-anm372793_ses-20170513.nwb 5.5 MB 5.5 MB 100% ok done
000006/sub-anm372794/sub-anm372794_ses-20170621.nwb 654.2 kB 654.2 kB 100% ok done
000006/sub-anm372794/sub-anm372794_ses-20170622.nwb 602.8 kB 602.8 kB 100% ok done
000006/sub-anm372793/sub-anm372793_ses-20170514.nwb 5.5 MB 5.5 MB 100% ok done
000006/sub-anm372794/sub-anm372794_ses-20170624.nwb 4.8 MB 4.8 MB 100% ok done
000006/sub-anm372794/sub-anm372794_ses-20170625.nwb 5.7 MB 5.7 MB 100% ok done
000006/sub-anm372794/sub-anm372794_ses-20170626.nwb 4.9 MB 4.9 MB 100% ok done
000006/sub-anm372794/sub-anm372794_ses-20170627.nwb 553.4 kB 553.4 kB 100% ok done
000006/sub-anm372795/sub-anm372795_ses-20170715.nwb 323.2 kB 323.2 kB 100% ok done
000006/sub-anm372795/sub-anm372795_ses-20170714.nwb 465.5 kB 465.5 kB 100% ok done
000006/sub-anm372795/sub-anm372795_ses-20170716.nwb 813.2 kB 813.2 kB 100% ok done
000006/sub-anm372795/sub-anm372795_ses-20170718.nwb 452.3 kB 452.3 kB 100% ok done
000006/sub-anm372797/sub-anm372797_ses-20170617.nwb 338.9 kB 338.9 kB 100% ok done
000006/sub-anm372904/sub-anm372904_ses-20170615.nwb 388.2 kB 388.2 kB 100% ok done
000006/sub-anm372797/sub-anm372797_ses-20170615.nwb 6.0 MB 6.0 MB 100% ok done
000006/sub-anm372904/sub-anm372904_ses-20170617.nwb 384.9 kB 384.9 kB 100% ok done
000006/sub-anm372904/sub-anm372904_ses-20170616.nwb 362.6 kB 362.6 kB 100% ok done
000006/sub-anm372904/sub-anm372904_ses-20170618.nwb 9.4 MB 9.4 MB 100% ok done
000006/sub-anm372904/sub-anm372904_ses-20170619.nwb 4.1 MB 4.1 MB 100% ok done
000006/sub-anm372905/sub-anm372905_ses-20170715.nwb 344.0 kB 344.0 kB 100% ok done
000006/sub-anm372905/sub-anm372905_ses-20170716.nwb 554.0 kB 554.0 kB 100% ok done
000006/sub-anm372905/sub-anm372905_ses-20170717.nwb 448.4 kB 448.4 kB 100% ok done
000006/sub-anm372906/sub-anm372906_ses-20170608.nwb 6.9 MB 6.9 MB 100% ok done
000006/sub-anm372906/sub-anm372906_ses-20170610.nwb 374.8 kB 374.8 kB 100% ok done
000006/sub-anm372906/sub-anm372906_ses-20170611.nwb 338.7 kB 338.7 kB 100% ok done
000006/sub-anm372906/sub-anm372906_ses-20170612.nwb 324.3 kB 324.3 kB 100% ok done
000006/sub-anm372907/sub-anm372907_ses-20170608.nwb 438.9 kB 438.9 kB 100% ok done
000006/sub-anm372907/sub-anm372907_ses-20170610.nwb 349.7 kB 349.7 kB 100% ok done
000006/sub-anm372907/sub-anm372907_ses-20170613.nwb 259.0 kB 259.0 kB 100% ok done
000006/sub-anm372907/sub-anm372907_ses-20170612.nwb 444.5 kB 444.5 kB 100% ok done
000006/sub-anm372909/sub-anm372909_ses-20170520.nwb 352.9 kB 352.9 kB 100% ok done
000006/sub-anm372909/sub-anm372909_ses-20170522.nwb 5.7 MB 5.7 MB 100% ok done
Summary: 82.3 MB 82.3 MB 16 skipped 1 no change
+57.3 MB 58.95% 38 done 15 already exists
ETA: a minute
Once the cell above completes running, you will see a new folder 📁”00006” wherever you’re running this notebook. Usefully, the code above will also print a list of individual NWB files that have been downloaded in this folder.
Once the data is done downloading, you’re ready for the next step.
Option 2: Streaming the Dandiset#
The folks at NWB have also developed a clever way to stream Dandisets so that small bits of them can be viewed without downloading the entire dataset. This is particularly useful for very large datasets! This step is a great option for large data sets, or if you have limited hard drive space, and/or if you are having issues with Option 1 above.
Streaming via the Dandihub#
The easiest way to stream data is via the DANDI Jupyter Hub (https://hub.dandiarchive.org/). There are setup steps for this in the introduction. The code below should work without a hitch in the Dandihub.
Streaming locally, after configuring your environment#
With some configuration, you can stream data on your local computer. First, you need to set up your environment with the right version of a package called h5py. There are instructions here for how to do that. Once you’re done, you can restart the kernel for this notebook, and run the code below.
Code for Data Streaming#
First, we need to figure out the correct URL for the dataset on the Amazon S3 storage system. There is a tool to do so within the dandiapi, which we’ll use below to get the URL for one session from the data we downloaded above.
from dandi.dandiapi import DandiAPIClient
dandiset_id = '000006' # ephys dataset from the Svoboda Lab
filepath = 'sub-anm372795/sub-anm372795_ses-20170718.nwb' # 450 kB file
with DandiAPIClient() as client:
asset = client.get_dandiset(dandiset_id, 'draft').get_asset_by_path(filepath)
s3_path = asset.get_content_url(follow_redirects=1, strip_query=True)
print(s3_path)
https://dandiarchive.s3.amazonaws.com/blobs/43b/f3a/43bf3a81-4a0b-433f-b471-1f10303f9d35
Now, we can read this path, but we’ll stream it, rather than downloading it! The cell below will print some of the data about this experiment. It uses another package, PyNWB, which is specifically designed to work with NWB files in Python. As you might expect, this won’t be the last time we see this package. Below, we’ll use the NWBHDF5IO class from this package, which will allow us to read NWB files.
Note: The code below will not work unless you’re on the Dandihub or have properly configured your environment. If you’d like to run it in Dandihub, copy and paste the code into a new cell below.
from pynwb import NWBHDF5IO
with NWBHDF5IO(s3_path, mode='r', load_namespaces=True, driver='ros3') as io:
nwbfile = io.read()
print(nwbfile)
print(nwbfile.acquisition['lick_times'].time_series['lick_left_times'].data[:])
In addition, we can use a fancy widget to create an interactive display of this dataset while we are streaming it. More on this later!
from nwbwidgets import nwb2widget
nwb2widget(nwbfile)
The following section will go over the the structure of an NWBFile and how to access data from this new file type.
Additional resources#
There are many different ways to configure your Dandi file download. See this OpenScope page for more details.
Similarly, here are alternative ways to configure streaming.