Obtaining Datasets with DANDI#
DANDI is an open source data archive for neuroscience datasets, called Dandisets. DANDI allows scientists to submit and download neural datasets to promote research collaboration and consistent and transparent data standards. DANDI also provides a solution to the difficulties that come from housing data in the many other general domains (i.e. Dropbox, Google Drive, etc.). Usefully for our purposes here, many of the datasets on DANDI are in NWB format. If you’d like to know more about DANDI, check out the DANDI handbook.
There are two primary ways to work with Dandisets:
You can stream datasets directly from DANDI. We’ll show you how to do this online as well as on your local computer. This is the preferred method given that many Dandisets are quite large.
You can download the datasets, either via the DANDI Web Application or using the DANDI Python client below. If you download via the website, you’ll need to create an account.
Below, we demonstrate how to do both of these. For additional information on either of these methods, please refer to the DANDI documentation.
Option 1: Streaming the Dandiset#
The folks at NWB have also developed a clever way to stream Dandisets so that small bits of them can be viewed without downloading the entire dataset. This is particularly useful for very large datasets! This step is a great option for large data sets and/or if you have limited hard drive space.
Below, we’ll find the correct URL for the dataset on Amazon S3, then stream it directly.
from dandi.dandiapi import DandiAPIClient
dandiset_id = '000006' # ephys dataset from the Svoboda Lab
filepath = 'sub-anm372795/sub-anm372795_ses-20170718.nwb' # 450 kB file
with DandiAPIClient() as client:
asset = client.get_dandiset(dandiset_id, 'draft').get_asset_by_path(filepath)
s3_path = asset.get_content_url(follow_redirects=1, strip_query=True)
print(s3_path)
https://dandiarchive.s3.amazonaws.com/blobs/43b/f3a/43bf3a81-4a0b-433f-b471-1f10303f9d35
Now, we can read this path, but we’ll stream it, rather than downloading it! The cell below will print some of the data about this experiment. It uses another package, PyNWB, which is specifically designed to work with NWB files in Python. As you might expect, this won’t be the last time we see this package. Below, we’ll use the NWBHDF5IO class from this package, which will allow us to read NWB files.
We’ll stream the file using remfile, which lets us read parts of a file directly from remote storage without downloading the whole thing. This works in any Python environment with internet access – Colab, Binder, or your own computer – with no special setup required.
from pynwb import NWBHDF5IO
import h5py
import remfile
# Stream the file using remfile
rem_file = remfile.File(s3_path)
with h5py.File(rem_file, "r") as h5py_file:
with NWBHDF5IO(file=h5py_file, load_namespaces=True) as io:
nwbfile = io.read()
print(nwbfile)
print(nwbfile.acquisition['lick_times'].time_series['lick_left_times'].data[:])
root pynwb.file.NWBFile at 0x5135612432
Fields:
acquisition: {
lick_times <class 'pynwb.behavior.BehavioralEvents'>
}
devices: {
H-19 <class 'pynwb.device.Device'>
}
electrode_groups: {
H-19: 64 <class 'pynwb.ecephys.ElectrodeGroup'>
}
electrodes: electrodes <class 'pynwb.ecephys.ElectrodesTable'>
experiment_description: Extracellular electrophysiology recordings performed on mouse anterior lateral motor cortex (ALM) in delay response task. Neural activity from two neuron populations, pyramidal track upper and lower, were characterized, in relation to movement execution.
experimenter: ['Mike Economo']
file_create_date: [datetime.datetime(2019, 10, 7, 17, 45, 0, 289664, tzinfo=tzoffset(None, -18000))]
identifier: anm372795_2017-07-18_5
institution: Janelia Research Campus
intervals: {
trials <class 'pynwb.epoch.TimeIntervals'>
}
keywords: <StrDataset for HDF5 dataset "keywords": shape (4,), type "|O">
related_publications: ['doi:10.1038/s41586-018-0642-9']
session_description:
session_start_time: 2017-07-18 00:00:00-05:00
subject: subject pynwb.file.Subject at 0x5135607392
Fields:
genotype:
sex: U
species: Mus musculus
subject_id: anm372795
timestamps_reference_time: 2017-07-18 00:00:00-05:00
trials: trials <class 'pynwb.epoch.TimeIntervals'>
units: units <class 'pynwb.misc.Units'>
[1. 1. 1. ... 1. 1. 1.]
In addition to exploring this data with code, you can get an interactive, code-free view of any DANDI-hosted NWB file using Neurosift – just click “Open with Neurosift” next to the file on its DANDI Archive page. More on this later!
Option 2: Downloading Dandisets using Python#
The cell below will download this dataset from DANDI. This dataset contains 32-channel extracellular recordings from mouse cortex. We’re using the download tool from dandi below.
Note: Downloading this dataset may take several minutes, depending on your internet connection.
from dandi.download import download as dandi_download
import os
# Set the URL for the DANDI file
url = 'https://dandiarchive.org/dandiset/000006/draft'
# Download the file into the current working directory
# It will skip downloading any files you've already downloaded
dandi_download([url], output_dir = os.getcwd(), existing = "skip")
PATH SIZE DONE DONE% CHECKSUM STATUS MESSAGE
000006/dandiset.yaml done updated
000006/sub-anm369962/sub-anm369962_ses-20170309.nwb 796.9 kB 796.9 kB 100% ok done
000006/sub-anm369962/sub-anm369962_ses-20170316.nwb 609.6 kB 609.6 kB 100% ok done
000006/sub-anm369962/sub-anm369962_ses-20170310.nwb 6.6 MB 6.6 MB 100% ok done
000006/sub-anm369962/sub-anm369962_ses-20170314.nwb 7.5 MB 7.5 MB 100% ok done
000006/sub-anm369962/sub-anm369962_ses-20170313.nwb 11.5 MB 11.5 MB 100% ok done
000006/sub-anm369962/sub-anm369962_ses-20170317.nwb 8.2 MB 8.2 MB 100% ok done
000006/sub-anm369963/sub-anm369963_ses-20170227.nwb 6.5 MB 6.5 MB 100% ok done
000006/sub-anm369963/sub-anm369963_ses-20170226.nwb 6.4 MB 6.4 MB 100% ok done
000006/sub-anm369963/sub-anm369963_ses-20170228.nwb 276.6 kB 276.6 kB 100% ok done
000006/sub-anm369963/sub-anm369963_ses-20170301.nwb 493.5 kB 493.5 kB 100% ok done
000006/sub-anm369963/sub-anm369963_ses-20170302.nwb 281.0 kB 281.0 kB 100% ok done
000006/sub-anm369963/sub-anm369963_ses-20170306.nwb 6.7 MB 6.7 MB 100% ok done
000006/sub-anm369963/sub-anm369963_ses-20170309.nwb 450.9 kB 450.9 kB 100% ok done
000006/sub-anm369963/sub-anm369963_ses-20170310.nwb 604.8 kB 604.8 kB 100% ok done
000006/sub-anm369964/sub-anm369964_ses-20170321.nwb 499.9 kB 499.9 kB 100% ok done
000006/sub-anm369964/sub-anm369964_ses-20170320.nwb 7.7 MB 7.7 MB 100% ok done
000006/sub-anm369964/sub-anm369964_ses-20170322.nwb 5.0 MB 5.0 MB 100% ok done
000006/sub-anm369964/sub-anm369964_ses-20170323.nwb 426.4 kB 426.4 kB 100% ok done
000006/sub-anm372793/sub-anm372793_ses-20170504.nwb 357.8 kB 357.8 kB 100% ok done
000006/sub-anm372793/sub-anm372793_ses-20170508.nwb 357.4 kB 357.4 kB 100% ok done
000006/sub-anm372793/sub-anm372793_ses-20170512.nwb 470.8 kB 470.8 kB 100% ok done
000006/sub-anm372793/sub-anm372793_ses-20170513.nwb 5.5 MB 5.5 MB 100% ok done
000006/sub-anm372794/sub-anm372794_ses-20170621.nwb 654.2 kB 654.2 kB 100% ok done
000006/sub-anm372794/sub-anm372794_ses-20170622.nwb 602.8 kB 602.8 kB 100% ok done
000006/sub-anm372793/sub-anm372793_ses-20170514.nwb 5.5 MB 5.5 MB 100% ok done
000006/sub-anm372794/sub-anm372794_ses-20170624.nwb 4.8 MB 4.8 MB 100% ok done
000006/sub-anm372794/sub-anm372794_ses-20170625.nwb 5.7 MB 5.7 MB 100% ok done
000006/sub-anm372794/sub-anm372794_ses-20170626.nwb 4.9 MB 4.9 MB 100% ok done
000006/sub-anm372794/sub-anm372794_ses-20170627.nwb 553.4 kB 553.4 kB 100% ok done
000006/sub-anm372795/sub-anm372795_ses-20170715.nwb 323.2 kB 323.2 kB 100% ok done
000006/sub-anm372795/sub-anm372795_ses-20170714.nwb 465.5 kB 465.5 kB 100% ok done
000006/sub-anm372795/sub-anm372795_ses-20170716.nwb 813.2 kB 813.2 kB 100% ok done
000006/sub-anm372795/sub-anm372795_ses-20170718.nwb 452.3 kB 452.3 kB 100% ok done
000006/sub-anm372797/sub-anm372797_ses-20170617.nwb 338.9 kB 338.9 kB 100% ok done
000006/sub-anm372904/sub-anm372904_ses-20170615.nwb 388.2 kB 388.2 kB 100% ok done
000006/sub-anm372797/sub-anm372797_ses-20170615.nwb 6.0 MB 6.0 MB 100% ok done
000006/sub-anm372904/sub-anm372904_ses-20170617.nwb 384.9 kB 384.9 kB 100% ok done
000006/sub-anm372904/sub-anm372904_ses-20170616.nwb 362.6 kB 362.6 kB 100% ok done
000006/sub-anm372904/sub-anm372904_ses-20170618.nwb 9.4 MB 9.4 MB 100% ok done
000006/sub-anm372904/sub-anm372904_ses-20170619.nwb 4.1 MB 4.1 MB 100% ok done
000006/sub-anm372905/sub-anm372905_ses-20170715.nwb 344.0 kB 344.0 kB 100% ok done
000006/sub-anm372905/sub-anm372905_ses-20170716.nwb 554.0 kB 554.0 kB 100% ok done
000006/sub-anm372905/sub-anm372905_ses-20170717.nwb 448.4 kB 448.4 kB 100% ok done
000006/sub-anm372906/sub-anm372906_ses-20170608.nwb 6.9 MB 6.9 MB 100% ok done
000006/sub-anm372906/sub-anm372906_ses-20170610.nwb 374.8 kB 374.8 kB 100% ok done
000006/sub-anm372906/sub-anm372906_ses-20170611.nwb 338.7 kB 338.7 kB 100% ok done
000006/sub-anm372906/sub-anm372906_ses-20170612.nwb 324.3 kB 324.3 kB 100% ok done
000006/sub-anm372907/sub-anm372907_ses-20170608.nwb 438.9 kB 438.9 kB 100% ok done
000006/sub-anm372907/sub-anm372907_ses-20170610.nwb 349.7 kB 349.7 kB 100% ok done
000006/sub-anm372907/sub-anm372907_ses-20170613.nwb 259.0 kB 259.0 kB 100% ok done
000006/sub-anm372907/sub-anm372907_ses-20170612.nwb 444.5 kB 444.5 kB 100% ok done
000006/sub-anm372909/sub-anm372909_ses-20170520.nwb 352.9 kB 352.9 kB 100% ok done
000006/sub-anm372909/sub-anm372909_ses-20170522.nwb 5.7 MB 5.7 MB 100% ok done
Summary: 139.6 MB 139.6 MB 54 done 1 updated
100.00%
Once the cell above completes running, you will see a new folder 📁”00006” wherever you’re running this notebook. Usefully, the code above will also print a list of individual NWB files that have been downloaded in this folder.
The following section will go over the the structure of an NWBFile and how to access data from this new file type.
Additional resources#
There are many different ways to configure your Dandi file download. See this OpenScope page for more details.
Similarly, here are alternative ways to configure streaming.