diff --git a/mne-bids-0.15.tar.gz b/mne-bids-0.15.tar.gz new file mode 100644 index 0000000000000000000000000000000000000000..9d7d5084ce9797c9aa4c2e0f0e3f6e960eee954a Binary files /dev/null and b/mne-bids-0.15.tar.gz differ diff --git a/mne-bids-0.15/.gitignore b/mne-bids-0.15/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..7c67eba788887bc642784659f00293280f8bacc5 --- /dev/null +++ b/mne-bids-0.15/.gitignore @@ -0,0 +1,38 @@ +# Distribution / packaging +.Python +dist/ +*.egg* +build +coverage +.coverage* +*.xml +.venv +.ruff_cache +sg_execution_times.rst + +# Sphinx documentation +doc/_build/ +doc/generated/ +doc/auto_examples/ +doc/auto_tutorials/ +doc/modules/generated/ +doc/sphinxext/cachedir +pip-log.txt +.coverage +tags +doc/coverages +doc/samples +cover + +# Pycharm +.idea/ + +*.pyc + +.cache +.pytest_cache +.ipynb_checkpoints +.DS_Store +.vscode/ + +__pycache__ diff --git a/mne-bids-0.15/LICENSE b/mne-bids-0.15/LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..a3ee632513bd27ef2b1d32feceadc3deafa3252b --- /dev/null +++ b/mne-bids-0.15/LICENSE @@ -0,0 +1,29 @@ +BSD 3-Clause License + +Copyright (c) 2018, mne-bids developers +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/mne-bids-0.15/PKG-INFO b/mne-bids-0.15/PKG-INFO new file mode 100644 index 0000000000000000000000000000000000000000..f78d2bb6886c6c7afeae0552a4c345b3cd1c4ce5 --- /dev/null +++ b/mne-bids-0.15/PKG-INFO @@ -0,0 +1,156 @@ +Metadata-Version: 2.3 +Name: mne-bids +Version: 0.15.0 +Summary: MNE-BIDS: Organizing MEG, EEG, and iEEG data according to the BIDS specification and facilitating their analysis with MNE-Python +Project-URL: Homepage, https://mne.tools/mne-bids +Project-URL: Download, https://pypi.org/project/mne-bids/#files +Project-URL: Bug Tracker, https://github.com/mne-tools/mne-bids/issues/ +Project-URL: Documentation, https://mne.tools/mne-bids +Project-URL: Forum, https://mne.discourse.group/ +Project-URL: Source Code, https://github.com/mne-tools/mne-bids +Author: MNE-BIDS Developers +Maintainer-email: Stefan Appelhoff +License: BSD-3-Clause +License-File: LICENSE +Keywords: bids,brain imaging data structure,eeg,ieeg,meg,neuroimaging,neuroscience +Classifier: Intended Audience :: Developers +Classifier: Intended Audience :: Science/Research +Classifier: License :: OSI Approved +Classifier: Operating System :: MacOS +Classifier: Operating System :: Microsoft :: Windows +Classifier: Operating System :: POSIX :: Linux +Classifier: Programming Language :: Python +Classifier: Programming Language :: Python :: 3.9 +Classifier: Programming Language :: Python :: 3.10 +Classifier: Programming Language :: Python :: 3.11 +Classifier: Programming Language :: Python :: 3.12 +Classifier: Topic :: Scientific/Engineering +Classifier: Topic :: Software Development +Requires-Python: >=3.9 +Requires-Dist: mne>=1.5 +Requires-Dist: numpy>=1.21.2 +Requires-Dist: scipy>=1.7.1 +Provides-Extra: dev +Requires-Dist: defusedxml; extra == 'dev' +Requires-Dist: edfio>=0.2.1; extra == 'dev' +Requires-Dist: edflib-python>=1.0.6; extra == 'dev' +Requires-Dist: eeglabio>=0.0.2; extra == 'dev' +Requires-Dist: matplotlib; extra == 'dev' +Requires-Dist: matplotlib>=3.5.0; extra == 'dev' +Requires-Dist: mne-nirs; extra == 'dev' +Requires-Dist: nibabel>=3.2.1; extra == 'dev' +Requires-Dist: nilearn; extra == 'dev' +Requires-Dist: numpydoc; extra == 'dev' +Requires-Dist: openneuro-py; extra == 'dev' +Requires-Dist: pandas; extra == 'dev' +Requires-Dist: pandas>=1.3.2; extra == 'dev' +Requires-Dist: pillow; extra == 'dev' +Requires-Dist: pre-commit; extra == 'dev' +Requires-Dist: pybv>=0.7.5; extra == 'dev' +Requires-Dist: pydata-sphinx-theme; extra == 'dev' +Requires-Dist: pymatreader>=0.0.30; extra == 'dev' +Requires-Dist: pytest; extra == 'dev' +Requires-Dist: pytest-cov; extra == 'dev' +Requires-Dist: pytest-sugar; extra == 'dev' +Requires-Dist: ruff; extra == 'dev' +Requires-Dist: seaborn; extra == 'dev' +Requires-Dist: sphinx; extra == 'dev' +Requires-Dist: sphinx-copybutton; extra == 'dev' +Requires-Dist: sphinx-gallery; extra == 'dev' +Provides-Extra: doc +Requires-Dist: matplotlib; extra == 'doc' +Requires-Dist: mne-nirs; extra == 'doc' +Requires-Dist: nilearn; extra == 'doc' +Requires-Dist: numpydoc; extra == 'doc' +Requires-Dist: openneuro-py; extra == 'doc' +Requires-Dist: pandas; extra == 'doc' +Requires-Dist: pillow; extra == 'doc' +Requires-Dist: pydata-sphinx-theme; extra == 'doc' +Requires-Dist: seaborn; extra == 'doc' +Requires-Dist: sphinx; extra == 'doc' +Requires-Dist: sphinx-copybutton; extra == 'doc' +Requires-Dist: sphinx-gallery; extra == 'doc' +Provides-Extra: full +Requires-Dist: defusedxml; extra == 'full' +Requires-Dist: edfio>=0.2.1; extra == 'full' +Requires-Dist: edflib-python>=1.0.6; extra == 'full' +Requires-Dist: eeglabio>=0.0.2; extra == 'full' +Requires-Dist: matplotlib>=3.5.0; extra == 'full' +Requires-Dist: nibabel>=3.2.1; extra == 'full' +Requires-Dist: pandas>=1.3.2; extra == 'full' +Requires-Dist: pybv>=0.7.5; extra == 'full' +Requires-Dist: pymatreader>=0.0.30; extra == 'full' +Provides-Extra: test +Requires-Dist: defusedxml; extra == 'test' +Requires-Dist: edfio>=0.2.1; extra == 'test' +Requires-Dist: edflib-python>=1.0.6; extra == 'test' +Requires-Dist: eeglabio>=0.0.2; extra == 'test' +Requires-Dist: matplotlib>=3.5.0; extra == 'test' +Requires-Dist: nibabel>=3.2.1; extra == 'test' +Requires-Dist: pandas>=1.3.2; extra == 'test' +Requires-Dist: pybv>=0.7.5; extra == 'test' +Requires-Dist: pymatreader>=0.0.30; extra == 'test' +Requires-Dist: pytest; extra == 'test' +Requires-Dist: pytest-cov; extra == 'test' +Requires-Dist: pytest-sugar; extra == 'test' +Requires-Dist: ruff; extra == 'test' +Description-Content-Type: text/markdown + +[![Codecov](https://codecov.io/gh/mne-tools/mne-bids/branch/main/graph/badge.svg)](https://codecov.io/gh/mne-tools/mne-bids) +[![GitHub Actions](https://github.com/mne-tools/mne-bids/workflows/build/badge.svg)](https://github.com/mne-tools/mne-bids/actions) +[![CircleCI](https://circleci.com/gh/mne-tools/mne-bids.svg?style=shield)](https://circleci.com/gh/mne-tools/mne-bids) +[![PyPI Download count](https://pepy.tech/badge/mne-bids)](https://pepy.tech/project/mne-bids) +[![Latest PyPI release](https://img.shields.io/pypi/v/mne-bids.svg)](https://pypi.org/project/mne-bids/) +[![Latest conda-forge release](https://img.shields.io/conda/vn/conda-forge/mne-bids.svg)](https://anaconda.org/conda-forge/mne-bids/) +[![JOSS publication](https://joss.theoj.org/papers/5b9024503f7bea324d5e738a12b0a108/status.svg)](https://joss.theoj.org/papers/5b9024503f7bea324d5e738a12b0a108) + +MNE-BIDS +======== + +MNE-BIDS is a Python package that allows you to read and write +[BIDS](https://bids.neuroimaging.io/)-compatible datasets with the help of +[MNE-Python](https://mne.tools/stable/index.html). + +![Schematic: From raw data to BIDS using MNE-BIDS](https://mne.tools/mne-bids/assets/MNE-BIDS.png) + +Why? +---- + +MNE-BIDS links BIDS and MNE-Python with the goal to make your analyses faster to code, more robust, and facilitate data and code sharing with co-workers and collaborators. + +How? +---- + +The documentation can be found under the following links: + +- for the [stable release](https://mne.tools/mne-bids/) +- for the [latest (development) version](https://mne.tools/mne-bids/dev/index.html) + +Getting Help +------------ +[MNE Forum](https://mne.discourse.group) + +For any usage questions, please post to the +[MNE Forum](https://mne.discourse.group). Be sure to add the `mne-bids` tag to +your question. + +Citing +------ + +[![JOSS publication](https://joss.theoj.org/papers/5b9024503f7bea324d5e738a12b0a108/status.svg)](https://joss.theoj.org/papers/5b9024503f7bea324d5e738a12b0a108) + +If you use MNE-BIDS in your work, please cite our +[publication in JOSS](https://doi.org/10.21105/joss.01896): + +> Appelhoff, S., Sanderson, M., Brooks, T., Vliet, M., Quentin, R., Holdgraf, C., +Chaumon, M., Mikulan, E., Tavabi, K., Höchenberger, R., Welke, D., Brunner, C., +Rockhill, A., Larson, E., Gramfort, A., & Jas, M. (2019): **MNE-BIDS: Organizing +electrophysiological data into the BIDS format and facilitating their analysis.** +*Journal of Open Source Software,* 4:1896. DOI: [10.21105/joss.01896](https://doi.org/10.21105/joss.01896) + +Please also cite one of the following papers to credit BIDS, depending on which data type you used: + +- [MEG-BIDS](https://doi.org/10.1038/sdata.2018.110) +- [EEG-BIDS](https://doi.org/10.1038/s41597-019-0104-8) +- [iEEG-BIDS](https://doi.org/10.1038/s41597-019-0105-7) +- [NIRS-BIDS](https://doi.org/10.31219/osf.io/7nmcp) diff --git a/mne-bids-0.15/README.md b/mne-bids-0.15/README.md new file mode 100644 index 0000000000000000000000000000000000000000..331d741ff43b68aa68fe04706086465b465ec9ec --- /dev/null +++ b/mne-bids-0.15/README.md @@ -0,0 +1,58 @@ +[![Codecov](https://codecov.io/gh/mne-tools/mne-bids/branch/main/graph/badge.svg)](https://codecov.io/gh/mne-tools/mne-bids) +[![GitHub Actions](https://github.com/mne-tools/mne-bids/workflows/build/badge.svg)](https://github.com/mne-tools/mne-bids/actions) +[![CircleCI](https://circleci.com/gh/mne-tools/mne-bids.svg?style=shield)](https://circleci.com/gh/mne-tools/mne-bids) +[![PyPI Download count](https://pepy.tech/badge/mne-bids)](https://pepy.tech/project/mne-bids) +[![Latest PyPI release](https://img.shields.io/pypi/v/mne-bids.svg)](https://pypi.org/project/mne-bids/) +[![Latest conda-forge release](https://img.shields.io/conda/vn/conda-forge/mne-bids.svg)](https://anaconda.org/conda-forge/mne-bids/) +[![JOSS publication](https://joss.theoj.org/papers/5b9024503f7bea324d5e738a12b0a108/status.svg)](https://joss.theoj.org/papers/5b9024503f7bea324d5e738a12b0a108) + +MNE-BIDS +======== + +MNE-BIDS is a Python package that allows you to read and write +[BIDS](https://bids.neuroimaging.io/)-compatible datasets with the help of +[MNE-Python](https://mne.tools/stable/index.html). + +![Schematic: From raw data to BIDS using MNE-BIDS](https://mne.tools/mne-bids/assets/MNE-BIDS.png) + +Why? +---- + +MNE-BIDS links BIDS and MNE-Python with the goal to make your analyses faster to code, more robust, and facilitate data and code sharing with co-workers and collaborators. + +How? +---- + +The documentation can be found under the following links: + +- for the [stable release](https://mne.tools/mne-bids/) +- for the [latest (development) version](https://mne.tools/mne-bids/dev/index.html) + +Getting Help +------------ +[MNE Forum](https://mne.discourse.group) + +For any usage questions, please post to the +[MNE Forum](https://mne.discourse.group). Be sure to add the `mne-bids` tag to +your question. + +Citing +------ + +[![JOSS publication](https://joss.theoj.org/papers/5b9024503f7bea324d5e738a12b0a108/status.svg)](https://joss.theoj.org/papers/5b9024503f7bea324d5e738a12b0a108) + +If you use MNE-BIDS in your work, please cite our +[publication in JOSS](https://doi.org/10.21105/joss.01896): + +> Appelhoff, S., Sanderson, M., Brooks, T., Vliet, M., Quentin, R., Holdgraf, C., +Chaumon, M., Mikulan, E., Tavabi, K., Höchenberger, R., Welke, D., Brunner, C., +Rockhill, A., Larson, E., Gramfort, A., & Jas, M. (2019): **MNE-BIDS: Organizing +electrophysiological data into the BIDS format and facilitating their analysis.** +*Journal of Open Source Software,* 4:1896. DOI: [10.21105/joss.01896](https://doi.org/10.21105/joss.01896) + +Please also cite one of the following papers to credit BIDS, depending on which data type you used: + +- [MEG-BIDS](https://doi.org/10.1038/sdata.2018.110) +- [EEG-BIDS](https://doi.org/10.1038/s41597-019-0104-8) +- [iEEG-BIDS](https://doi.org/10.1038/s41597-019-0105-7) +- [NIRS-BIDS](https://doi.org/10.31219/osf.io/7nmcp) diff --git a/mne-bids-0.15/mne_bids/__init__.py b/mne-bids-0.15/mne_bids/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..45e95587b2c4c956731e13ede84488b020fe30d4 --- /dev/null +++ b/mne-bids-0.15/mne_bids/__init__.py @@ -0,0 +1,40 @@ +"""MNE software for easily interacting with BIDS compatible datasets.""" + +try: + from importlib.metadata import version + + __version__ = version("mne_bids") +except Exception: + __version__ = "0.0.0" + +from mne_bids import commands +from mne_bids.report import make_report +from mne_bids.path import ( + BIDSPath, + get_datatypes, + get_entity_vals, + print_dir_tree, + get_entities_from_fname, + search_folder_for_text, + get_bids_path_from_fname, + find_matching_paths, +) +from mne_bids.read import get_head_mri_trans, read_raw_bids +from mne_bids.utils import get_anonymization_daysback +from mne_bids.write import ( + make_dataset_description, + write_anat, + write_raw_bids, + mark_channels, + write_meg_calibration, + write_meg_crosstalk, + get_anat_landmarks, + anonymize_dataset, +) +from mne_bids.sidecar_updates import update_sidecar_json, update_anat_landmarks +from mne_bids.inspect import inspect_dataset +from mne_bids.dig import ( + template_to_head, + convert_montage_to_ras, + convert_montage_to_mri, +) diff --git a/mne-bids-0.15/mne_bids/assets/help-128px.png b/mne-bids-0.15/mne_bids/assets/help-128px.png new file mode 100644 index 0000000000000000000000000000000000000000..9bd4f7a80982e0d54370c3274293067661e093f7 Binary files /dev/null and b/mne-bids-0.15/mne_bids/assets/help-128px.png differ diff --git a/mne-bids-0.15/mne_bids/assets/help.svg b/mne-bids-0.15/mne_bids/assets/help.svg new file mode 100644 index 0000000000000000000000000000000000000000..7b3dbfc1be2aa6e419f14f0f7c131a0fe1b4ad0d --- /dev/null +++ b/mne-bids-0.15/mne_bids/assets/help.svg @@ -0,0 +1 @@ + diff --git a/mne-bids-0.15/mne_bids/commands/__init__.py b/mne-bids-0.15/mne_bids/commands/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..ffcea797c91d1dbc4c6f307c27d3d2d10736ceed --- /dev/null +++ b/mne-bids-0.15/mne_bids/commands/__init__.py @@ -0,0 +1 @@ +"""Initialize cli.""" diff --git a/mne-bids-0.15/mne_bids/commands/mne_bids_calibration_to_bids.py b/mne-bids-0.15/mne_bids/commands/mne_bids_calibration_to_bids.py new file mode 100644 index 0000000000000000000000000000000000000000..4cceccb1aa55cec0b93e1d8bda175b4405381242 --- /dev/null +++ b/mne-bids-0.15/mne_bids/commands/mne_bids_calibration_to_bids.py @@ -0,0 +1,67 @@ +r"""Write Elekta/Neuromag/MEGIN fine-calibration data to BIDS. + +example usage: +$ mne_bids calibration_to_bids --subject_id=01 --session=test +--bids_root=bids_root --file=sss_cal.dat + +""" +# Authors: Richard Höchenberger +# +# License: BSD-3-Clause + +from mne.utils import logger + +import mne_bids +from mne_bids import BIDSPath, write_meg_calibration + + +def run(): + """Run the calibration_to_bids command.""" + from mne.commands.utils import get_optparser + + parser = get_optparser( + __file__, + usage="usage: %prog options args", + prog_prefix="mne_bids", + version=mne_bids.__version__, + ) + + parser.add_option( + "--bids_root", + dest="bids_root", + help="The path of the folder containing the BIDS dataset", + ) + parser.add_option("--subject_id", dest="subject", help="Subject name") + parser.add_option("--session_id", dest="session", help="Session name") + parser.add_option("--file", dest="fname", help="The path of the crosstalk file") + parser.add_option( + "--verbose", + dest="verbose", + action="store_true", + help="Whether do generate additional diagnostic output", + ) + + opt, args = parser.parse_args() + if args: + parser.print_help() + parser.error( + f"Please do not specify arguments without flags. " f"Got: {args}.\n" + ) + + if opt.bids_root is None: + parser.print_help() + parser.error("You must specify bids_root") + if opt.subject is None: + parser.print_help() + parser.error("You must specify a subject") + + bids_path = BIDSPath(subject=opt.subject, session=opt.session, root=opt.bids_root) + + logger.info(f"Writing fine-calibration file {bids_path.basename} …") + write_meg_calibration( + calibration=opt.fname, bids_path=bids_path, verbose=opt.verbose + ) + + +if __name__ == "__main__": + run() diff --git a/mne-bids-0.15/mne_bids/commands/mne_bids_count_events.py b/mne-bids-0.15/mne_bids/commands/mne_bids_count_events.py new file mode 100644 index 0000000000000000000000000000000000000000..8c7efdc383f607c1a69f8ad43114d0c1b62967a9 --- /dev/null +++ b/mne-bids-0.15/mne_bids/commands/mne_bids_count_events.py @@ -0,0 +1,96 @@ +"""Count events in BIDS dataset. + +example usage: $ mne_bids count_events --bids_root bids_root_path + +""" + +# Authors: Alex Gramfort +# +# License: BSD-3-Clause +from pathlib import Path + +import mne_bids +from mne_bids.stats import count_events + + +def run(): + """Run the raw_to_bids command.""" + import pandas as pd + from mne.commands.utils import get_optparser + + parser = get_optparser( + __file__, + usage="usage: %prog options args", + prog_prefix="mne_bids", + version=mne_bids.__version__, + ) + + parser.add_option( + "--bids_root", dest="bids_root", help="The path of the BIDS compatible folder." + ) + + parser.add_option( + "--datatype", dest="datatype", default="auto", help="The datatype to consider." + ) + + parser.add_option( + "--describe", + dest="describe", + action="store_true", + help="If set print the descriptive statistics (min, max, etc.).", + ) + + parser.add_option( + "--output", + dest="output", + default=None, + help="Path to the CSV file where to store the results.", + ) + + parser.add_option( + "--overwrite", + dest="overwrite", + action="store_true", + help="If set, overwrite an existing output file.", + ) + + parser.add_option( + "--silent", + dest="silent", + action="store_true", + help="Whether to print the event counts on the screen.", + ) + + opt, args = parser.parse_args() + + if len(args) > 0: + parser.print_help() + parser.error(f'Do not specify arguments without flags. Found: "{args}".\n') + + if not all([opt.bids_root]): + parser.print_help() + parser.error( + "Arguments missing. You need to specify the --bids_root parameter." + ) + + if opt.output and Path(opt.output).exists() and not opt.overwrite: + parser.error("Output file exists. To overwrite, pass --overwrite") + + counts = count_events(opt.bids_root, datatype=opt.datatype) + + if opt.describe: + counts = counts.describe() + + if not opt.silent: + with pd.option_context( + "display.max_rows", 1000, "display.max_columns", 80, "display.width", 1000 + ): + print(counts) + + if opt.output: + counts.to_csv(opt.output) + print(f"\nOutput stored in {opt.output}") + + +if __name__ == "__main__": + run() diff --git a/mne-bids-0.15/mne_bids/commands/mne_bids_cp.py b/mne-bids-0.15/mne_bids/commands/mne_bids_cp.py new file mode 100644 index 0000000000000000000000000000000000000000..7d981f11bd5de9f4bee454163130cbe44b7302bb --- /dev/null +++ b/mne-bids-0.15/mne_bids/commands/mne_bids_cp.py @@ -0,0 +1,81 @@ +"""Rename files (making a copy) and update their internal pointers. + +example usage: $ mne_bids cp --input myfile.vhdr --output sub-01_task-test.vhdr +""" + +# Authors: Stefan Appelhoff +# +# License: BSD-3-Clause +import mne_bids +from mne_bids.copyfiles import copyfile_brainvision, copyfile_ctf, copyfile_eeglab + + +def run(): + """Run the cp command.""" + from mne.commands.utils import get_optparser + + accepted_formats_msg = "(accepted formats: BrainVision .vhdr, EEGLAB .set, CTF .ds)" + + parser = get_optparser( + __file__, + usage="usage: %prog -i INPUT -o OUTPUT", + prog_prefix="mne_bids", + version=mne_bids.__version__, + ) + + parser.add_option( + "-i", + "--input", + dest="input", + help=f"path to the input file. {accepted_formats_msg}", + metavar="INPUT", + ) + + parser.add_option( + "-o", + "--output", + dest="output", + help="path to the output file (MUST be same format as input file)", + metavar="OUTPUT", + ) + + parser.add_option( + "-v", + "--verbose", + dest="verbose", + help="set logging level to verbose", + action="store_true", + ) + + opt, args = parser.parse_args() + opt_dict = vars(opt) + + # Check the usage and raise error if invalid + if len(args) > 0: + parser.print_help() + parser.error( + f'Do not specify arguments without flags. Found: "{args}".\n' + "Did you forget to provide -i and -o?" + ) + + if not opt_dict.get("input") or not opt_dict.get("output"): + parser.print_help() + parser.error( + "Incorrect number of arguments. Supply one input and one " + f'output file. You supplied: "{opt}"' + ) + + # Attempt to do the copying. Errors will be raised by the copyfile + # functions if there are issues with the file formats + if opt.input.endswith(".vhdr"): + copyfile_brainvision(opt.input, opt.output, opt.verbose) + elif opt.input.endswith(".set"): + copyfile_eeglab(opt.input, opt.output) + elif opt.input.endswith(".ds"): + copyfile_ctf(opt.input, opt.output) + else: + parser.error(f'{accepted_formats_msg} You supplied: "{opt}"') + + +if __name__ == "__main__": + run() diff --git a/mne-bids-0.15/mne_bids/commands/mne_bids_crosstalk_to_bids.py b/mne-bids-0.15/mne_bids/commands/mne_bids_crosstalk_to_bids.py new file mode 100644 index 0000000000000000000000000000000000000000..391fbd361fbf0b008e6c746242d71b560ed3d883 --- /dev/null +++ b/mne-bids-0.15/mne_bids/commands/mne_bids_crosstalk_to_bids.py @@ -0,0 +1,63 @@ +"""Write Elekta/Neuromag/MEGIN crosstalk data to BIDS. + +example usage: +$ mne_bids crosstalk_to_bids --subject_id=01 --session=test +--bids_root=bids_root --file=ct_sparse.fif + +""" +# Authors: Richard Höchenberger +# +# License: BSD-3-Clause + +from mne.utils import logger + +import mne_bids +from mne_bids import BIDSPath, write_meg_crosstalk + + +def run(): + """Run the crosstalk_to_bids command.""" + from mne.commands.utils import get_optparser + + parser = get_optparser( + __file__, + usage="usage: %prog options args", + prog_prefix="mne_bids", + version=mne_bids.__version__, + ) + + parser.add_option( + "--bids_root", + dest="bids_root", + help="The path of the folder containing the BIDS dataset", + ) + parser.add_option("--subject_id", dest="subject", help="Subject name") + parser.add_option("--session_id", dest="session", help="Session name") + parser.add_option("--file", dest="fname", help="The path of the crosstalk file") + parser.add_option( + "--verbose", + dest="verbose", + action="store_true", + help="Whether do generate additional diagnostic output", + ) + + opt, args = parser.parse_args() + if args: + parser.print_help() + parser.error(f"Please do not specify arguments without flags. Got: {args}.\n") + + if opt.bids_root is None: + parser.print_help() + parser.error("You must specify bids_root") + if opt.subject is None: + parser.print_help() + parser.error("You must specify a subject") + + bids_path = BIDSPath(subject=opt.subject, session=opt.session, root=opt.bids_root) + + logger.info(f"Writing crosstalk file {bids_path.basename} …") + write_meg_crosstalk(fname=opt.fname, bids_path=bids_path, verbose=opt.verbose) + + +if __name__ == "__main__": + run() diff --git a/mne-bids-0.15/mne_bids/commands/mne_bids_inspect.py b/mne-bids-0.15/mne_bids/commands/mne_bids_inspect.py new file mode 100644 index 0000000000000000000000000000000000000000..d47339cad5c036bc500ab090c039a330c6933a7c --- /dev/null +++ b/mne-bids-0.15/mne_bids/commands/mne_bids_inspect.py @@ -0,0 +1,112 @@ +r"""Inspect MEG and EEG raw data, and interactively mark channels as bad. + +example usage: +$ mne_bids inspect --subject_id=01 --task=experiment --session=test \ +--datatype=meg --suffix=meg --bids_root=bids_root + +""" +# Authors: Richard Höchenberger +# +# License: BSD-3-Clause + +from mne.utils import logger + +import mne_bids +from mne_bids import BIDSPath, inspect_dataset + + +def run(): + """Run the mark_channels command.""" + from mne.commands.utils import get_optparser + + parser = get_optparser( + __file__, + usage="usage: %prog options args", + prog_prefix="mne_bids", + version=mne_bids.__version__, + ) + + parser.add_option( + "--bids_root", + dest="bids_root", + help="The path of the folder containing the BIDS dataset", + ) + parser.add_option("--subject_id", dest="subject", help="Subject name") + parser.add_option("--session_id", dest="session", help="Session name") + parser.add_option("--task", dest="task", help="Task name") + parser.add_option("--acq", dest="acquisition", help="Acquisition parameter") + parser.add_option("--run", dest="run", help="Run number") + parser.add_option("--proc", dest="processing", help="Processing label.") + parser.add_option("--rec", dest="recording", help="Recording name") + parser.add_option( + "--type", dest="datatype", help="Recording data type, e.g. meg, ieeg or eeg" + ) + parser.add_option( + "--suffix", + dest="suffix", + help="The filename suffix, i.e. the last part before the extension", + ) + parser.add_option( + "--ext", + dest="extension", + help="The filename extension, including the leading period, e.g. .fif", + ) + parser.add_option( + "--find_flat", + dest="find_flat", + help="Whether to auto-detect flat channels and time segments", + ) + parser.add_option( + "--l_freq", dest="l_freq", help="The high-pass filter cutoff frequency" + ) + parser.add_option( + "--h_freq", dest="h_freq", help="The low-pass filter cutoff frequency" + ) + parser.add_option( + "--verbose", + dest="verbose", + action="store_true", + help="Whether do generate additional diagnostic output", + ) + + opt, args = parser.parse_args() + if args: + parser.print_help() + parser.error( + f"Please do not specify arguments without flags. " f"Got: {args}.\n" + ) + + if opt.bids_root is None: + parser.print_help() + parser.error("You must specify bids_root") + + bids_path = BIDSPath( + subject=opt.subject, + session=opt.session, + task=opt.task, + acquisition=opt.acquisition, + run=opt.run, + processing=opt.processing, + recording=opt.recording, + datatype=opt.datatype, + suffix=opt.suffix, + extension=opt.extension, + root=opt.bids_root, + ) + + find_flat = True if opt.find_flat is None else bool(opt.find_flat) + l_freq = None if opt.l_freq is None else float(opt.l_freq) + h_freq = None if opt.h_freq is None else float(opt.h_freq) + + logger.info(f"Inspecting {bids_path.basename} …") + inspect_dataset( + bids_path=bids_path, + find_flat=find_flat, + l_freq=l_freq, + h_freq=h_freq, + verbose=opt.verbose, + ) + + +if __name__ == "__main__": + run() diff --git a/mne-bids-0.15/mne_bids/commands/mne_bids_mark_channels.py b/mne-bids-0.15/mne_bids/commands/mne_bids_mark_channels.py new file mode 100644 index 0000000000000000000000000000000000000000..045e1035ea34437f8e9c82ea8f17a118d3c6fe8f --- /dev/null +++ b/mne-bids-0.15/mne_bids/commands/mne_bids_mark_channels.py @@ -0,0 +1,144 @@ +"""Mark channels in an existing BIDS dataset as "bad". + +example usage: +$ mne_bids mark_channels --ch_name="MEG 0112" --description="noisy" \ + --ch_name="MEG 0131" --description="flat" \ + --subject_id=01 --task=experiment --session=test \ + --bids_root=bids_root + +""" +# Authors: Richard Höchenberger +# +# License: BSD-3-Clause + +from mne.utils import logger + +import mne_bids +from mne_bids import BIDSPath, mark_channels +from mne_bids.config import reader + + +def run(): + """Run the mark_channels command.""" + from mne.commands.utils import get_optparser + + parser = get_optparser( + __file__, + usage="usage: %prog options args", + prog_prefix="mne_bids", + version=mne_bids.__version__, + ) + + parser.add_option( + "--ch_name", + dest="ch_names", + action="append", + default=[], + help="The names of the bad channels. If multiple " + "channels are bad, pass the --ch_name parameter " + "multiple times.", + ) + parser.add_option( + "--status", + default="bad", + help='Status of the channels (Either "good", or "bad").', + ) + parser.add_option( + "--description", + dest="descriptions", + action="append", + default=[], + help="Descriptions as to why the channels are bad. " + "Must match the number of bad channels provided. " + "Pass multiple times to supply more than one " + "value in that case.", + ) + parser.add_option( + "--bids_root", + dest="bids_root", + help="The path of the folder containing the BIDS dataset", + ) + parser.add_option("--subject_id", dest="subject", help="Subject name") + parser.add_option("--session_id", dest="session", help="Session name") + parser.add_option("--task", dest="task", help="Task name") + parser.add_option("--acq", dest="acquisition", help="Acquisition parameter") + parser.add_option("--run", dest="run", help="Run number") + parser.add_option("--proc", dest="processing", help="Processing label.") + parser.add_option("--rec", dest="recording", help="Recording name") + parser.add_option( + "--type", dest="datatype", help="Recording data type, e.g. meg, ieeg or eeg" + ) + parser.add_option( + "--suffix", + dest="suffix", + help="The filename suffix, i.e. the last part before the extension", + ) + parser.add_option( + "--ext", + dest="extension", + help="The filename extension, including the leading period, e.g. .fif", + ) + parser.add_option( + "--verbose", + dest="verbose", + action="store_true", + help="Whether do generate additional diagnostic output", + ) + + opt, args = parser.parse_args() + if args: + parser.print_help() + parser.error(f"Please do not specify arguments without flags. Got: {args}.\n") + + if opt.bids_root is None: + parser.print_help() + parser.error("You must specify bids_root") + if opt.ch_names is None: + parser.print_help() + parser.error("You must specify some --ch_name parameters.") + + status = opt.status + ch_names = [] if opt.ch_names == [""] else opt.ch_names + bids_path = BIDSPath( + subject=opt.subject, + session=opt.session, + task=opt.task, + acquisition=opt.acquisition, + run=opt.run, + processing=opt.processing, + recording=opt.recording, + datatype=opt.datatype, + suffix=opt.suffix, + extension=opt.extension, + root=opt.bids_root, + ) + + bids_paths = bids_path.match() + # Only keep data we can actually read & write. + allowed_extensions = list(reader.keys()) + bids_paths = [p for p in bids_paths if p.extension in allowed_extensions] + + if not bids_paths: + logger.info( + "No matching files found. Please consider using a less " + "restrictive set of entities to broaden the search." + ) + return # XXX should be return with an error code? + + logger.info( + f'Marking channels {", ".join(ch_names)} as bad in ' + f"{len(bids_paths)} recording(s) …" + ) + for bids_path in bids_paths: + logger.info(f"Processing: {bids_path.basename}") + mark_channels( + bids_path=bids_path, + ch_names=ch_names, + status=status, + descriptions=opt.descriptions, + verbose=opt.verbose, + ) + + +if __name__ == "__main__": + run() diff --git a/mne-bids-0.15/mne_bids/commands/mne_bids_raw_to_bids.py b/mne-bids-0.15/mne_bids/commands/mne_bids_raw_to_bids.py new file mode 100644 index 0000000000000000000000000000000000000000..1050540202e6c7cb37adf0c466fe900cffcffcec --- /dev/null +++ b/mne-bids-0.15/mne_bids/commands/mne_bids_raw_to_bids.py @@ -0,0 +1,126 @@ +"""Write raw files to BIDS format. + +example usage: $ mne_bids raw_to_bids --subject_id sub01 --task rest +--raw data.edf --bids_root new_path + +""" + +# Authors: Teon Brooks +# Stefan Appelhoff +# +# License: BSD-3-Clause +from itertools import chain, repeat + +import mne_bids +from mne_bids import BIDSPath, write_raw_bids +from mne_bids.read import _read_raw + + +def run(): + """Run the raw_to_bids command.""" + from mne.commands.utils import get_optparser + + parser = get_optparser( + __file__, + usage="usage: %prog options args", + prog_prefix="mne_bids", + version=mne_bids.__version__, + ) + + parser.add_option( + "--subject_id", + dest="subject_id", + help="subject name in BIDS compatible format (01, 02, etc.)", + ) + parser.add_option( + "--task", dest="task", help="name of the task the data is based on" + ) + parser.add_option("--raw", dest="raw_fname", help="path to the raw MEG file") + parser.add_option( + "--bids_root", dest="bids_root", help="The path of the BIDS compatible folder." + ) + parser.add_option( + "--session_id", dest="session_id", help="session name in BIDS compatible format" + ) + parser.add_option("--run", dest="run", help="run number for this dataset") + parser.add_option( + "--acq", dest="acq", help="acquisition parameter for this dataset" + ) + parser.add_option("--events", dest="events", help="events file (events.tsv)") + parser.add_option( + "--event_id", dest="event_id", help="event id dict", metavar="eid" + ) + parser.add_option("--hpi", dest="hpi", help="path to the MEG marker points") + parser.add_option( + "--electrode", dest="electrode", help="path to head-native digitizer points" + ) + parser.add_option("--hsp", dest="hsp", help="path to headshape points") + parser.add_option("--config", dest="config", help="path to the configuration file") + parser.add_option( + "--overwrite", + dest="overwrite", + help="whether to overwrite existing data (BOOLEAN)", + ) + parser.add_option( + "--line_freq", + dest="line_freq", + help="The frequency of the line noise in Hz " + "(e.g. 50 or 60). If unknown, pass None", + ) + + opt, args = parser.parse_args() + + if len(args) > 0: + parser.print_help() + parser.error(f'Do not specify arguments without flags. Found: "{args}".\n') + + if not all([opt.subject_id, opt.task, opt.raw_fname, opt.bids_root]): + parser.print_help() + parser.error( + "Arguments missing. You need to specify at least the" + "following: --subject_id, --task, --raw, --bids_root." + ) + + bids_path = BIDSPath( + subject=opt.subject_id, + session=opt.session_id, + run=opt.run, + acquisition=opt.acq, + task=opt.task, + root=opt.bids_root, + ) + + allow_maxshield = False + if opt.raw_fname.endswith(".fif"): + allow_maxshield = "yes" + + raw = _read_raw( + opt.raw_fname, + hpi=opt.hpi, + electrode=opt.electrode, + hsp=opt.hsp, + config_path=opt.config, + allow_maxshield=allow_maxshield, + ) + if opt.line_freq is not None: + line_freq = None if opt.line_freq == "None" else float(opt.line_freq) + raw.info["line_freq"] = line_freq + + if opt.overwrite is not None: + truthy = [1, "True", "true"] + falsy = [0, "False", "false"] + bool_mapping = dict(chain(zip(truthy, repeat(True)), zip(falsy, repeat(False)))) + opt.overwrite = bool_mapping[opt.overwrite] + + write_raw_bids( + raw, + bids_path, + event_id=opt.event_id, + events=opt.events, + overwrite=opt.overwrite, + verbose=True, + ) + + +if __name__ == "__main__": + run() diff --git a/mne-bids-0.15/mne_bids/commands/mne_bids_report.py b/mne-bids-0.15/mne_bids/commands/mne_bids_report.py new file mode 100644 index 0000000000000000000000000000000000000000..2ca58bbcfaee94040be2d3e05f7a516616d6d19b --- /dev/null +++ b/mne-bids-0.15/mne_bids/commands/mne_bids_report.py @@ -0,0 +1,48 @@ +"""Write raw files to BIDS format. + +example usage: $ mne_bids report --bids_root bids_root_path + +""" + +# Authors: Alexandre Gramfort +# +# License: BSD-3-Clause +import mne_bids +from mne_bids import make_report + + +def run(): + """Run the raw_to_bids command.""" + from mne.commands.utils import get_optparser + + parser = get_optparser( + __file__, + usage="usage: %prog options args", + prog_prefix="mne_bids", + version=mne_bids.__version__, + ) + + parser.add_option( + "--bids_root", dest="bids_root", help="The path of the BIDS compatible folder." + ) + + opt, args = parser.parse_args() + + if len(args) > 0: + parser.print_help() + parser.error(f'Do not specify arguments without flags. Found: "{args}".\n') + + if not all([opt.bids_root]): + parser.print_help() + parser.error( + "Arguments missing. You need to specify the --bids_root parameter." + ) + + report = make_report(opt.bids_root) + print("-" * 36 + " REPORT " + "-" * 36) + print(report) + assert " " not in report + + +if __name__ == "__main__": + run() diff --git a/mne-bids-0.15/mne_bids/commands/run.py b/mne-bids-0.15/mne_bids/commands/run.py new file mode 100755 index 0000000000000000000000000000000000000000..d0996db58c4c1a2b7f8b1fc698c8c9ef43985cca --- /dev/null +++ b/mne-bids-0.15/mne_bids/commands/run.py @@ -0,0 +1,47 @@ +"""Command Line Interface for MNE-BIDS.""" + +# Authors: Teon Brooks +# Stefan Appelhoff +# +# License: BSD-3-Clause +import glob +import os.path as op +import subprocess +import sys + +import mne_bids + +mne_bin_dir = op.abspath(op.dirname(mne_bids.__file__)) +valid_commands = sorted(glob.glob(op.join(mne_bin_dir, "commands", "mne_bids_*.py"))) +valid_commands = [c.split(op.sep)[-1][9:-3] for c in valid_commands] + + +def print_help(): + """Print the help.""" + print("Usage : mne_bids command options\n") + print("Accepted commands :\n") + for c in valid_commands: + print("\t- %s" % c) + print( + "\nExample : mne_bids raw_to_bids --subject_id sub01 --task rest", + "--raw_file data.edf --bids_root new_path", + ) + sys.exit(0) + + +def main(): + """Run main command.""" + if len(sys.argv) == 1: + print_help() + elif "help" in sys.argv[1] or "-h" in sys.argv[1]: + print_help() + elif sys.argv[1] == "--version": + print("MNE-BIDS %s" % mne_bids.__version__) + elif sys.argv[1] not in valid_commands: + print('Invalid command: "%s"\n' % sys.argv[1]) + print_help() + sys.exit(0) + else: + cmd = sys.argv[1] + cmd_path = op.join(mne_bin_dir, "commands", "mne_bids_%s.py" % cmd) + sys.exit(subprocess.call([sys.executable, cmd_path] + sys.argv[2:])) diff --git a/mne-bids-0.15/mne_bids/config.py b/mne-bids-0.15/mne_bids/config.py new file mode 100644 index 0000000000000000000000000000000000000000..7941cbb2c83caf0cb95ce9bb61774f1f691d61a2 --- /dev/null +++ b/mne-bids-0.15/mne_bids/config.py @@ -0,0 +1,622 @@ +"""Configuration values for MNE-BIDS.""" + +from mne import io +from mne.io.constants import FIFF + +BIDS_VERSION = "1.7.0" + +PYBV_VERSION = "0.7.3" +EEGLABIO_VERSION = "0.0.2" + +DOI = """https://doi.org/10.21105/joss.01896""" + +EPHY_ALLOWED_DATATYPES = ["meg", "eeg", "ieeg", "nirs"] + +ALLOWED_DATATYPES = EPHY_ALLOWED_DATATYPES + ["anat", "beh"] + +MEG_CONVERT_FORMATS = ["FIF", "auto"] +EEG_CONVERT_FORMATS = ["BrainVision", "auto"] +IEEG_CONVERT_FORMATS = ["BrainVision", "auto"] +NIRS_CONVERT_FORMATS = ["auto"] +CONVERT_FORMATS = { + "meg": MEG_CONVERT_FORMATS, + "eeg": EEG_CONVERT_FORMATS, + "ieeg": IEEG_CONVERT_FORMATS, + "nirs": NIRS_CONVERT_FORMATS, +} + +# Orientation of the coordinate system dependent on manufacturer +ORIENTATION = { + ".con": "KitYokogawa", + ".ds": "CTF", + ".fif": "ElektaNeuromag", + ".pdf": "4DBti", + ".sqd": "KitYokogawa", +} + +EXT_TO_UNIT_MAP = {".con": "m", ".ds": "cm", ".fif": "m", ".pdf": "m", ".sqd": "m"} + +UNITS_MNE_TO_BIDS_MAP = { + "C": "oC", # temperature in deg. C +} + +meg_manufacturers = { + ".con": "KIT/Yokogawa", + ".ds": "CTF", + ".fif": "Elekta", + ".meg4": "CTF", + ".pdf": "4D Magnes", + ".sqd": "KIT/Yokogawa", +} + +eeg_manufacturers = { + ".vhdr": "Brain Products", + ".eeg": "Brain Products", + ".edf": "n/a", + ".EDF": "n/a", + ".bdf": "Biosemi", + ".BDF": "Biosemi", + ".set": "n/a", + ".fdt": "n/a", + ".lay": "Persyst", + ".dat": "Persyst", + ".EEG": "Nihon Kohden", + ".cnt": "Neuroscan", + ".CNT": "Neuroscan", + ".bin": "EGI", + ".cdt": "Curry", +} + +ieeg_manufacturers = { + ".vhdr": "Brain Products", + ".eeg": "Brain Products", + ".edf": "n/a", + ".EDF": "n/a", + ".set": "n/a", + ".fdt": "n/a", + ".mef": "n/a", + ".nwb": "n/a", + ".lay": "Persyst", + ".dat": "Persyst", + ".EEG": "Nihon Kohden", +} + +nirs_manufacturers = {".snirf": "SNIRF"} + +# file-extension map to mne-python readers +reader = { + ".con": io.read_raw_kit, + ".sqd": io.read_raw_kit, + ".fif": io.read_raw_fif, + ".pdf": io.read_raw_bti, + ".ds": io.read_raw_ctf, + ".vhdr": io.read_raw_brainvision, + ".edf": io.read_raw_edf, + ".EDF": io.read_raw_edf, + ".bdf": io.read_raw_bdf, + ".set": io.read_raw_eeglab, + ".lay": io.read_raw_persyst, + ".EEG": io.read_raw_nihon, + ".cnt": io.read_raw_cnt, + ".CNT": io.read_raw_cnt, + ".bin": io.read_raw_egi, + ".snirf": io.read_raw_snirf, + ".cdt": io.read_raw_curry, +} + + +# Merge the manufacturer dictionaries in a python2 / python3 compatible way +# MANUFACTURERS dictionary only includes the extension of the input filename +# that mne-python accepts (e.g. BrainVision has three files, but the reader +# takes the filename for `.vhdr`) +MANUFACTURERS = dict() +MANUFACTURERS.update(meg_manufacturers) +MANUFACTURERS.update(eeg_manufacturers) +MANUFACTURERS.update(ieeg_manufacturers) +MANUFACTURERS.update(nirs_manufacturers) + +# List of synthetic channels by manufacturer that are to be excluded from the +# channel list. Currently this is only for stimulus channels. +IGNORED_CHANNELS = { + "KIT/Yokogawa": ["STI 014"], + "BrainProducts": ["STI 014"], + "n/a": ["STI 014"], # for unknown manufacturers, ignore it + "Biosemi": ["STI 014"], +} + +allowed_extensions_meg = [".con", ".sqd", ".fif", ".pdf", ".ds"] +allowed_extensions_eeg = [ + ".vhdr", # BrainVision, accompanied by .vmrk, .eeg + ".edf", # European Data Format + ".bdf", # Biosemi + ".set", # EEGLAB, potentially accompanied by .fdt +] + +allowed_extensions_ieeg = [ + ".vhdr", # BrainVision, accompanied by .vmrk, .eeg + ".edf", # European Data Format + ".set", # EEGLAB, potentially accompanied by .fdt + ".mef", # MEF: Multiscale Electrophysiology File + ".nwb", # Neurodata without borders +] + +allowed_extensions_nirs = [ + ".snirf", # SNIRF +] + +# allowed extensions (data formats) in BIDS spec +ALLOWED_DATATYPE_EXTENSIONS = { + "meg": allowed_extensions_meg, + "eeg": allowed_extensions_eeg, + "ieeg": allowed_extensions_ieeg, + "nirs": allowed_extensions_nirs, +} + +# allow additional extensions that are not BIDS +# compliant, but we will convert to the +# recommended formats +ALLOWED_INPUT_EXTENSIONS = ( + allowed_extensions_meg + + allowed_extensions_eeg + + allowed_extensions_ieeg + + allowed_extensions_nirs + + [".lay", ".EEG", ".cnt", ".CNT", ".bin", ".cdt"] +) + +# allowed suffixes (i.e. last "_" delimiter in the BIDS filenames before +# the extension) +ALLOWED_FILENAME_SUFFIX = [ + "meg", + "markers", + "eeg", + "ieeg", + "T1w", + "FLASH", # datatype + "participants", + "scans", + "sessions", + "electrodes", + "optodes", + "channels", + "coordsystem", + "events", # sidecars + "headshape", + "digitizer", # meg-specific sidecars + "beh", + "physio", + "stim", # behavioral + "nirs", +] + +# converts suffix to known path modalities +SUFFIX_TO_DATATYPE = { + "meg": "meg", + "headshape": "meg", + "digitizer": "meg", + "markers": "meg", + "eeg": "eeg", + "ieeg": "ieeg", + "T1w": "anat", + "FLASH": "anat", +} + +# allowed BIDS extensions (extension in the BIDS filename) +ALLOWED_FILENAME_EXTENSIONS = ( + ALLOWED_INPUT_EXTENSIONS + + [".json", ".tsv", ".tsv.gz", ".nii", ".nii.gz"] + + [".pos", ".eeg", ".vmrk"] + + [".dat", ".EEG"] # extra datatype-specific metadata files. + + [".mrk"] # extra eeg extensions # KIT/Yokogawa/Ricoh marker coil +) + +# allowed BIDSPath entities +ALLOWED_PATH_ENTITIES = ( + "subject", + "session", + "task", + "run", + "processing", + "recording", + "space", + "acquisition", + "split", + "description", + "suffix", + "extension", +) +ALLOWED_PATH_ENTITIES_SHORT = { + "sub": "subject", + "ses": "session", + "task": "task", + "acq": "acquisition", + "run": "run", + "proc": "processing", + "space": "space", + "rec": "recording", + "split": "split", + "desc": "description", +} + +# Annotations to never remove during reading or writing +ANNOTATIONS_TO_KEEP = ("BAD_ACQ_SKIP",) + +BIDS_STANDARD_TEMPLATE_COORDINATE_SYSTEMS = [ + "ICBM452AirSpace", + "ICBM452Warp5Space", + "IXI549Space", + "fsaverage", + "fsaverageSym", + "fsLR", + "MNIColin27", + "MNI152Lin", + "MNI152NLin2009aSym", + "MNI152NLin2009bSym", + "MNI152NLin2009cSym", + "MNI152NLin2009aAsym", + "MNI152NLin2009bAsym", + "MNI152NLin2009cAsym", + "MNI152NLin6Sym", + "MNI152NLin6ASym", + "MNI305", + "NIHPD", + "OASIS30AntsOASISAnts", + "OASIS30Atropos", + "Talairach", + "UNCInfant", +] + +coordsys_standard_template_deprecated = [ + "fsaverage3", + "fsaverage4", + "fsaverage5", + "fsaverage6", + "fsaveragesym", + "UNCInfant0V21", + "UNCInfant1V21", + "UNCInfant2V21", + "UNCInfant0V22", + "UNCInfant1V22", + "UNCInfant2V22", + "UNCInfant0V23", + "UNCInfant1V23", + "UNCInfant2V23", +] + +# accepted BIDS formats, which may be subject to change +# depending on the specification +BIDS_IEEG_COORDINATE_FRAMES = ["ACPC", "Pixels"] +BIDS_MEG_COORDINATE_FRAMES = [ + "CTF", + "ElektaNeuromag", + "4DBti", + "KitYokogawa", + "ChietiItab", +] +BIDS_EEG_COORDINATE_FRAMES = ["CapTrak"] + +# accepted coordinate SI units +BIDS_COORDINATE_UNITS = ["m", "cm", "mm"] +coordsys_wildcard = ["Other"] +BIDS_SHARED_COORDINATE_FRAMES = ( + BIDS_STANDARD_TEMPLATE_COORDINATE_SYSTEMS + + coordsys_standard_template_deprecated + + coordsys_wildcard +) + +ALLOWED_SPACES = dict() +ALLOWED_SPACES["meg"] = ALLOWED_SPACES["eeg"] = ( + BIDS_SHARED_COORDINATE_FRAMES + + BIDS_MEG_COORDINATE_FRAMES + + BIDS_EEG_COORDINATE_FRAMES +) +ALLOWED_SPACES["ieeg"] = BIDS_SHARED_COORDINATE_FRAMES + BIDS_IEEG_COORDINATE_FRAMES +ALLOWED_SPACES["anat"] = None +ALLOWED_SPACES["beh"] = None + +# See: https://bids-specification.readthedocs.io/en/latest/appendices/entity-table.html#encephalography-eeg-ieeg-and-meg # noqa +ENTITY_VALUE_TYPE = { + "subject": "label", + "session": "label", + "task": "label", + "run": "index", + "processing": "label", + "recording": "label", + "space": "label", + "acquisition": "label", + "split": "index", + "description": "label", + "suffix": "label", + "extension": "label", +} + +# mapping from supported BIDs coordinate frames -> MNE +BIDS_TO_MNE_FRAMES = { + "CTF": "ctf_head", + "4DBti": "ctf_head", + "KitYokogawa": "ctf_head", + "ElektaNeuromag": "head", + "ChietiItab": "head", + "CapTrak": "head", + "ACPC": "ras", # assumes T1 is ACPC-aligned, if not the coordinates are lost # noqa + "fsaverage": "mni_tal", # XXX: note fsaverage and MNI305 are the same # noqa + "MNI305": "mni_tal", +} + +# mapping from supported MNE coordinate frames -> BIDS +# XXX: note that there are a lot fewer MNE available coordinate +# systems so the range of BIDS supported coordinate systems we +# can write is limited. +MNE_TO_BIDS_FRAMES = {"ctf_head": "CTF", "head": "CapTrak", "mni_tal": "fsaverage"} + +# these coordinate frames in mne-python are related to scalp/meg +# 'meg', 'ctf_head', 'ctf_meg', 'head', 'unknown' +# copied from "mne.transforms.MNE_STR_TO_FRAME" +MNE_STR_TO_FRAME = dict( + meg=FIFF.FIFFV_COORD_DEVICE, + mri=FIFF.FIFFV_COORD_MRI, + mri_voxel=FIFF.FIFFV_MNE_COORD_MRI_VOXEL, + head=FIFF.FIFFV_COORD_HEAD, + mni_tal=FIFF.FIFFV_MNE_COORD_MNI_TAL, + ras=FIFF.FIFFV_MNE_COORD_RAS, + fs_tal=FIFF.FIFFV_MNE_COORD_FS_TAL, + ctf_head=FIFF.FIFFV_MNE_COORD_CTF_HEAD, + ctf_meg=FIFF.FIFFV_MNE_COORD_CTF_DEVICE, + unknown=FIFF.FIFFV_COORD_UNKNOWN, +) +MNE_FRAME_TO_STR = {val: key for key, val in MNE_STR_TO_FRAME.items()} + +# see BIDS specification for description we copied over from each +BIDS_COORD_FRAME_DESCRIPTIONS = { + "acpc": "The origin of the coordinate system is at the Anterior " + "Commissure and the negative y-axis is passing through the " + "Posterior Commissure. The positive z-axis is passing through " + "a mid-hemispheric point in the superior direction.", + "pixels": "If electrodes are localized in 2D space (only x and y are " + "specified and z is n/a), then the positions in this file " + "must correspond to the locations expressed in pixels on " + "the photo/drawing/rendering of the electrodes on the brain. " + "In this case, coordinates must be (row,column) pairs, with " + "(0,0) corresponding to the upper left pixel and (N,0) " + "corresponding to the lower left pixel.", + "ctf": "ALS orientation and the origin between the ears", + "elektaneuromag": "RAS orientation and the origin between the ears", + "4dbti": "ALS orientation and the origin between the ears", + "kityokogawa": "ALS orientation and the origin between the ears", + "chietiitab": "RAS orientation and the origin between the ears", + "captrak": ( + "The X-axis goes from the left preauricular point (LPA) through " + "the right preauricular point (RPA). " + "The Y-axis goes orthogonally to the X-axis through the nasion (NAS). " + "The Z-axis goes orthogonally to the XY-plane through the vertex of " + "the head. " + 'This corresponds to a "RAS" orientation with the origin of the ' + "coordinate system approximately between the ears. " + "See Appendix VIII in the BIDS specification." + ), + "fsaverage": "Defined by FreeSurfer, the MRI (surface RAS) origin is " + "at the center of a 256×256×256 mm^3 anisotropic volume " + "(may not be in the center of the head).", + "icbm452airspace": 'Reference space defined by the "average of 452 ' + 'T1-weighted MRIs of normal young adult brains" ' + 'with "linear transforms of the subjects into the ' + "atlas space using a 12-parameter affine " + 'transformation"', + "icbm452warp5space": 'Reference space defined by the "average of 452 ' + 'T1-weighted MRIs of normal young adult brains" ' + '"based on a 5th order polynomial transformation ' + 'into the atlas space"', + "ixi549space": 'Reference space defined by the average of the "549 (...) ' + 'subjects from the IXI dataset" linearly transformed to ' + "ICBM MNI 452.", + "fsaveragesym": "The fsaverage is a dual template providing both " + "volumetric and surface coordinates references. The " + "volumetric template corresponds to a FreeSurfer variant " + "of MNI305 space. The fsaverageSym atlas also defines a " + "symmetric surface reference system (formerly described " + "as fsaveragesym).", + "fslr": "The fsLR is a dual template providing both volumetric and " + "surface coordinates references. The volumetric template " + "corresponds to MNI152NLin6Asym. Surface templates are given " + "at several sampling densities: 164k (used by HCP pipelines " + "for 3T and 7T anatomical analysis), 59k (used by HCP pipelines " + "for 7T MRI bold and DWI analysis), 32k (used by HCP pipelines " + "for 3T MRI bold and DWI analysis), or 4k (used by HCP " + "pipelines for MEG analysis) fsaverage_LR surface " + "reconstructed from the T1w image.", + "mnicolin27": "Average of 27 T1 scans of a single subject.", + "mni152lin": "Also known as ICBM (version with linear coregistration).", + "mni152nlin6sym": "Also known as symmetric ICBM 6th generation " + "(non-linear coregistration).", + "mni152nlin6asym": "A variation of MNI152NLin6Sym built by A. Janke that " + "is released as the MNI template of FSL. Volumetric " + "templates included with HCP-Pipelines correspond to " + "this template too.", + "mni305": "Also known as avg305.", + "nihpd": "Pediatric templates generated from the NIHPD sample. Available " + "for different age groups (4.5–18.5 y.o., 4.5–8.5 y.o., " + "7–11 y.o., 7.5–13.5 y.o., 10–14 y.o., 13–18.5 y.o. This " + "template also comes in either -symmetric or -asymmetric flavor.", + "oasis30antsoasisants": "See https://figshare.com/articles/ANTs_ANTsR_Brain_Templates/915436", # noqa: E501 + "oasis30atropos": "See https://mindboggle.info/data.html", + "talairach": "Piecewise linear scaling of the brain is implemented as " + "described in TT88.", + "uncinfant": "Infant Brain Atlases from Neonates to 1- and 2-year-olds.", +} + +for letter in ("a", "b", "c"): + for sym in ("Sym", "Asym"): + BIDS_COORD_FRAME_DESCRIPTIONS[f"mni152nlin2009{letter}{sym}"] = ( + "Also known as ICBM (non-linear coregistration with 40 iterations," + ) + " released in 2009). It comes in either three different flavours " + "each in symmetric or asymmetric version." + +REFERENCES = { + "mne-bids": "Appelhoff, S., Sanderson, M., Brooks, T., Vliet, M., " + "Quentin, R., Holdgraf, C., Chaumon, M., Mikulan, E., " + "Tavabi, K., Höchenberger, R., Welke, D., Brunner, C., " + "Rockhill, A., Larson, E., Gramfort, A. and Jas, M. (2019). " + "MNE-BIDS: Organizing electrophysiological data into the " + "BIDS format and facilitating their analysis. Journal of " + "Open Source Software 4: (1896)." + "https://doi.org/10.21105/joss.01896", + "meg": "Niso, G., Gorgolewski, K. J., Bock, E., Brooks, T. L., " + "Flandin, G., Gramfort, A., Henson, R. N., Jas, M., Litvak, " + "V., Moreau, J., Oostenveld, R., Schoffelen, J., Tadel, F., " + "Wexler, J., Baillet, S. (2018). MEG-BIDS, the brain " + "imaging data structure extended to magnetoencephalography. " + "Scientific Data, 5, 180110." + "https://doi.org/10.1038/sdata.2018.110", + "eeg": "Pernet, C. R., Appelhoff, S., Gorgolewski, K. J., " + "Flandin, G., Phillips, C., Delorme, A., Oostenveld, R. (2019). " + "EEG-BIDS, an extension to the brain imaging data structure " + "for electroencephalography. Scientific Data, 6, 103." + "https://doi.org/10.1038/s41597-019-0104-8", + "ieeg": "Holdgraf, C., Appelhoff, S., Bickel, S., Bouchard, K., " + "D'Ambrosio, S., David, O., … Hermes, D. (2019). iEEG-BIDS, " + "extending the Brain Imaging Data Structure specification " + "to human intracranial electrophysiology. Scientific Data, " + "6, 102. https://doi.org/10.1038/s41597-019-0105-7", + "nirs": "Luke, R., Oostenveld, R., Cockx, H., Niso, G., Shader, M., " + "Orihuela-Espina, F., Innes-Brown, H., Tucker, S., Boas, D., Gau, R., " + "Salo, T., Appelhoff, S., Markiewicz, C McAlpine, D., BIDS maintainers, " + "Pollonini, L. (2023). fNIRS-BIDS, the Brain Imaging Data Structure " + "Extended to Functional Near-Infrared Spectroscopy. PsyArXiv. " + "https://doi.org/10.31219/osf.io/7nmcp", +} + + +# Mapping subject information between MNE-BIDS and MNE-Python. +HAND_BIDS_TO_MNE = { + ("n/a",): 0, + ("right", "r", "R", "RIGHT", "Right"): 1, + ("left", "l", "L", "LEFT", "Left"): 2, + ("ambidextrous", "a", "A", "AMBIDEXTROUS", "Ambidextrous"): 3, +} + +HAND_MNE_TO_BIDS = {0: "n/a", 1: "R", 2: "L", 3: "A"} + +SEX_BIDS_TO_MNE = { + ("n/a", "other", "o", "O", "OTHER", "Other"): 0, + ("male", "m", "M", "MALE", "Male"): 1, + ("female", "f", "F", "FEMALE", "Female"): 2, +} + +SEX_MNE_TO_BIDS = {0: "n/a", 1: "M", 2: "F"} + + +def _map_options(what, key, fro, to): + if what == "sex": + mapping_bids_mne = SEX_BIDS_TO_MNE + mapping_mne_bids = SEX_MNE_TO_BIDS + elif what == "hand": + mapping_bids_mne = HAND_BIDS_TO_MNE + mapping_mne_bids = HAND_MNE_TO_BIDS + else: + raise ValueError("Can only map `sex` and `hand`.") + + if fro == "bids" and to == "mne": + # Many-to-one mapping + mapped_option = None + for bids_keys, mne_option in mapping_bids_mne.items(): + if key in bids_keys: + mapped_option = mne_option + break + elif fro == "mne" and to == "bids": + # One-to-one mapping + mapped_option = mapping_mne_bids.get(key, None) + else: + raise RuntimeError( + f"fro value {fro} and to value {to} are not " + "accepted. Use 'mne', or 'bids'." + ) + + return mapped_option + + +# Which JSON data can safely be transferred from a non-anonymized to an +# anonymized dataset without accidentally exposing personal identifiable +# information +ANONYMIZED_JSON_KEY_WHITELIST = [ + # Common + "Manufacturer", + "ManufacturersModelName", + "InstitutionName", + "InstitutionalDepartmentName", + "InstitutionAddress", + "DeviceSerialNumber", + # MRI + # Many of these are not standardized, but produced by dcm2niix. + "Modality", + "MagneticFieldStrength", + "ImagingFrequency", + "StationName", + "SeriesInstanceUID", + "StudyInstanceUID", + "StudyID", + "BodyPartExamined", + "PatientPosition", + "ProcedureStepDescription", + "SoftwareVersions", + "MRAcquisitionType", + "SeriesDescription", + "ProtocolName", + "ScanningSequence", + "SequenceVariant", + "ScanOptions", + "SequenceName", + "ImageType", + "SeriesNumber", + "AcquisitionNumber", + "SliceThickness", + "SAR", + "EchoTime", + "RepetitionTime", + "InversionTime", + "FlipAngle", + "PartialFourier", + "BaseResolution", + "ShimSetting", + "TxRefAmp", + "PhaseResolution", + "ReceiveCoilName", + "ReceiveCoilActiveElements", + "PulseSequenceDetails", + "ConsistencyInfo", + "PercentPhaseFOV", + "PercentSampling", + "PhaseEncodingSteps", + "AcquisitionMatrixPE", + "PixelBandwidth", + "DwellTime", + "ImageOrientationPatientDICOM", + "InPlanePhaseEncodingDirectionDICOM", + "ConversionSoftware", + "ConversionSoftwareVersion", + # Electrophys common + "TaskName", + "TaskDescription", + "Instructions", + "PowerLineFrequency", + "SamplingFrequency", + "SoftwareFilters", + "RecordingType", + "EEGChannelCount", + "EOGChannelCount", + "ECGChannelCount", + "EMGChannelCount", + "MiscChannelCount", + "TriggerChannelCount", + "RecordingDuration", + # EEG + "EEGReference", + "EEGPlacementScheme", + # MEG + "DewarPosition", + "DigitizedLandmarks", + "DigitizedHeadPoints", + "MEGChannelCount", + "MEGREFChannelCount", + "ContinuousHeadLocalization", + "HeadCoilFrequency", +] diff --git a/mne-bids-0.15/mne_bids/conftest.py b/mne-bids-0.15/mne_bids/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..46948d2df209f4313317ea0d0f82fc7ba3d9ff4e --- /dev/null +++ b/mne-bids-0.15/mne_bids/conftest.py @@ -0,0 +1,19 @@ +"""Configure all tests.""" + +import mne +import pytest + + +def pytest_configure(config): + """Configure pytest options.""" + # Fixtures + config.addinivalue_line("usefixtures", "monkeypatch_mne") + + +@pytest.fixture(scope="session") +def monkeypatch_mne(): + """Monkeypatch MNE to ensure we have download=False everywhere in tests.""" + mne.datasets.utils._MODULES_TO_ENSURE_DOWNLOAD_IS_FALSE_IN_TESTS = ( + "mne", + "mne_bids", + ) diff --git a/mne-bids-0.15/mne_bids/copyfiles.py b/mne-bids-0.15/mne_bids/copyfiles.py new file mode 100644 index 0000000000000000000000000000000000000000..dc82c1eaf02313c6fa354377a307550c35fc0755 --- /dev/null +++ b/mne-bids-0.15/mne_bids/copyfiles.py @@ -0,0 +1,642 @@ +"""Utility functions to copy raw data files. + +When writing BIDS datasets, we often move and/or rename raw data files. several +original data formats have properties that restrict such operations. That is, +moving/renaming raw data files naively might lead to broken files, for example +due to internal pointers that are not being updated. + +""" + +# Authors: Mainak Jas +# Alexandre Gramfort +# Teon Brooks +# Chris Holdgraf +# Stefan Appelhoff +# Matt Sanderson +# +# License: BSD-3-Clause +import os +import os.path as op +import re +import shutil as sh +from pathlib import Path + +import numpy as np +from mne.io import anonymize_info, read_raw_bdf, read_raw_brainvision, read_raw_edf +from mne.utils import logger, verbose +from scipy.io import loadmat, savemat + +from mne_bids.path import BIDSPath, _mkdir_p, _parse_ext +from mne_bids.utils import _check_anonymize, _get_mrk_meas_date, warn + + +def _copytree(src, dst, **kwargs): + """See: https://github.com/jupyterlab/jupyterlab/pull/5150.""" + try: + sh.copytree(src, dst, **kwargs) + except sh.Error as error: + # `copytree` throws an error if copying to + from NFS even though + # the copy is successful (see https://bugs.python.org/issue24564) + if "[Errno 22]" not in str(error) or not op.exists(dst): + raise + + +def _get_brainvision_encoding(vhdr_file): + """Get the encoding of .vhdr and .vmrk files. + + Parameters + ---------- + vhdr_file : str + Path to the header file. + + Returns + ------- + enc : str + Encoding of the .vhdr file to pass it on to open() function + either 'UTF-8' (default) or whatever encoding scheme is specified + in the header. + + """ + with open(vhdr_file, "rb") as ef: + enc = ef.read() + if enc.find(b"Codepage=") != -1: + enc = enc[enc.find(b"Codepage=") + 9 :] + enc = enc.split()[0] + enc = enc.decode() + src = "(read from header)" + else: + enc = "UTF-8" + src = "(default)" + logger.debug(f"Detected file encoding: {enc} {src}.") + return enc + + +def _get_brainvision_paths(vhdr_path): + """Get the .eeg/.dat and .vmrk file paths from a BrainVision header file. + + Parameters + ---------- + vhdr_path : str + Path to the header file. + + Returns + ------- + paths : tuple + Paths to the .eeg/.dat file at index 0 and the .vmrk file at index 1 of + the returned tuple. + + """ + fname, ext = _parse_ext(vhdr_path) + if ext != ".vhdr": + raise ValueError(f'Expecting file ending in ".vhdr",' f" but got {ext}") + + # Header file seems fine + # extract encoding from brainvision header file, or default to utf-8 + enc = _get_brainvision_encoding(vhdr_path) + + # ..and read it + with open(vhdr_path, encoding=enc) as f: + lines = f.readlines() + + # Try to find data file .eeg/.dat + eeg_file_match = re.search(r"DataFile=(.*\.(eeg|dat))", " ".join(lines)) + + if not eeg_file_match: + raise ValueError("Could not find a .eeg or .dat file link in" f" {vhdr_path}") + else: + eeg_file = eeg_file_match.groups()[0] + + # Try to find marker file .vmrk + vmrk_file_match = re.search(r"MarkerFile=(.*\.vmrk)", " ".join(lines)) + if not vmrk_file_match: + raise ValueError("Could not find a .vmrk file link in" f" {vhdr_path}") + else: + vmrk_file = vmrk_file_match.groups()[0] + + # Make sure we are dealing with file names as is customary, not paths + # Paths are problematic when copying the files to another system. Instead, + # always use the file name and keep the file triplet in the same directory + assert os.sep not in eeg_file + assert os.sep not in vmrk_file + + # Assert the paths exist + head, tail = op.split(vhdr_path) + eeg_file_path = op.join(head, eeg_file) + vmrk_file_path = op.join(head, vmrk_file) + assert op.exists(eeg_file_path) + assert op.exists(vmrk_file_path) + + # Return the paths + return (eeg_file_path, vmrk_file_path) + + +def copyfile_ctf(src, dest): + """Copy and rename CTF files to a new location. + + Parameters + ---------- + src : path-like + Path to the source raw .ds folder. + dest : path-like + Path to the destination of the new bids folder. + + See Also + -------- + copyfile_brainvision + copyfile_bti + copyfile_edf + copyfile_eeglab + copyfile_kit + + """ + _copytree(src, dest) + # list of file types to rename + file_types = ( + ".acq", + ".eeg", + ".dat", + ".hc", + ".hist", + ".infods", + ".bak", + ".meg4", + ".newds", + ".res4", + ) + + # Consider CTF files that are split having consecutively numbered extensions + extra_ctf_file_types = tuple( + f".{i}_meg4" for i in range(1, 21) + ) # cap at 20 is arbitrary + file_types += extra_ctf_file_types + + # Rename files in dest with the name of the dest directory + fnames = [f for f in os.listdir(dest) if f.endswith(file_types)] + bids_folder_name = op.splitext(op.split(dest)[-1])[0] + for fname in fnames: + ext = op.splitext(fname)[-1] + os.replace(op.join(dest, fname), op.join(dest, bids_folder_name + ext)) + + +def copyfile_kit(src, dest, subject_id, session_id, task, run, _init_kwargs): + """Copy and rename KIT files to a new location. + + Parameters + ---------- + src : path-like + Path to the source raw .con or .sqd folder. + dest : path-like + Path to the destination of the new bids folder. + subject_id : str | None + The subject ID. Corresponds to "sub". + session_id : str | None + The session identifier. Corresponds to "ses". + task : str | None + The task identifier. Corresponds to "task". + run : int | None + The run number. Corresponds to "run". + _init_kwargs : dict + Extract information of marker and headpoints + + See Also + -------- + copyfile_brainvision + copyfile_bti + copyfile_ctf + copyfile_edf + copyfile_eeglab + + """ + # create parent directories in case it does not exist yet + _mkdir_p(op.dirname(dest)) + + # KIT data requires the marker file to be copied over too + sh.copyfile(src, dest) + data_path = op.split(dest)[0] + datatype = "meg" + + if "mrk" in _init_kwargs and _init_kwargs["mrk"] is not None: + hpi = _init_kwargs["mrk"] + acq_map = dict() + if isinstance(hpi, list): + if _get_mrk_meas_date(hpi[0]) > _get_mrk_meas_date(hpi[1]): + raise ValueError("Markers provided in incorrect order.") + _, marker_ext = _parse_ext(hpi[0]) + acq_map = dict(zip(["pre", "post"], hpi)) + else: + _, marker_ext = _parse_ext(hpi) + acq_map[None] = hpi + for key, value in acq_map.items(): + marker_path = BIDSPath( + subject=subject_id, + session=session_id, + task=task, + run=run, + acquisition=key, + suffix="markers", + extension=marker_ext, + datatype=datatype, + ) + sh.copyfile(value, op.join(data_path, marker_path.basename)) + + for acq in ["elp", "hsp"]: + if acq in _init_kwargs and _init_kwargs[acq] is not None: + position_file = _init_kwargs[acq] + task, run, acq = None, None, acq.upper() + position_ext = ".pos" + position_path = BIDSPath( + subject=subject_id, + session=session_id, + task=task, + run=run, + acquisition=acq, + suffix="headshape", + extension=position_ext, + datatype=datatype, + ) + sh.copyfile(position_file, op.join(data_path, position_path.basename)) + + +def _replace_file(fname, pattern, replace): + """Overwrite file, replacing end of lines matching pattern with replace.""" + new_content = [] + for line in open(fname): + match = re.match(pattern, line) + if match: + line = match.group()[: -len(replace)] + replace + "\n" + new_content.append(line) + + with open(fname, "w", encoding="utf-8") as fout: + fout.writelines(new_content) + + +def _anonymize_brainvision(vhdr_file, date): + """Anonymize vmrk and vhdr files in place using `date` datetime object.""" + _, vmrk_file = _get_brainvision_paths(vhdr_file) + + # Go through VMRK + pattern = re.compile(r"^Mk\d+=New Segment,.*,\d+,\d+,\d+,\d{20}$") + replace = date.strftime("%Y%m%d%H%M%S%f") + _replace_file(vmrk_file, pattern, replace) + + # Go through VHDR + pattern = re.compile(r"^Impedance \[kOhm\] at \d\d:\d\d:\d\d :$") + replace = f'at {date.strftime("%H:%M:%S")} :' + _replace_file(vhdr_file, pattern, replace) + + +@verbose +def copyfile_brainvision(vhdr_src, vhdr_dest, anonymize=None, verbose=None): + """Copy a BrainVision file triplet to a new location and repair links. + + The BrainVision file format consists of three files: + .vhdr, .eeg/.dat, and .vmrk + The .eeg/.dat and .vmrk files associated with the .vhdr file will be + given names as in `vhdr_dest` with adjusted extensions. Internal file + pointers will be fixed. + + Parameters + ---------- + vhdr_src : path-like + The source path of the .vhdr file to be copied. + vhdr_dest : path-like + The destination path of the .vhdr file. + anonymize : dict | None + If None (default), no anonymization is performed. + If dict, data will be anonymized depending on the keys provided with + the dict: `daysback` is a required key, `keep_his` is an optional key. + + `daysback` : int + Number of days by which to move back the recording date in time. + In studies with multiple subjects the relative recording date + differences between subjects can be kept by using the same number + of `daysback` for all subject anonymizations. `daysback` should be + great enough to shift the date prior to 1925 to conform with BIDS + anonymization rules. + + `keep_his` : bool + By default (False), all subject information next to the recording + date will be overwritten as well. If True, keep subject information + apart from the recording date. + + %(verbose)s + + See Also + -------- + mne.io.anonymize_info + copyfile_bti + copyfile_ctf + copyfile_edf + copyfile_eeglab + copyfile_kit + + """ + # Get extension of the brainvision file + fname_src, ext_src = _parse_ext(vhdr_src) + fname_dest, ext_dest = _parse_ext(vhdr_dest) + if ext_src != ext_dest: + raise ValueError( + f"Need to move data with same extension, " + f' but got "{ext_src}" and "{ext_dest}"' + ) + + eeg_file_path, vmrk_file_path = _get_brainvision_paths(vhdr_src) + + # extract encoding from brainvision header file, or default to utf-8 + enc = _get_brainvision_encoding(vhdr_src) + + # raise warning if binary file has .dat extension + if ".dat" in eeg_file_path: + warn( + "The file extension of your binary EEG data file is .dat, while " + "the expected extension for raw data is .eeg. " + "This might imply it's preprocessed or processed data: " + "We copied the files and changed the extension to .eeg, " + "but please ensure that this is actually BIDS compatible data!" + ) + + # Copy data .eeg/.dat ... no links to repair + sh.copyfile(eeg_file_path, fname_dest + ".eeg") + + # Write new header and marker files, fixing the file pointer links + # For that, we need to replace an old "basename" with a new one + # assuming that all .eeg/.dat, .vhdr, .vmrk share one basename + __, basename_src = op.split(fname_src) + assert op.split(eeg_file_path)[-1] in [basename_src + ".eeg", basename_src + ".dat"] + assert basename_src + ".vmrk" == op.split(vmrk_file_path)[-1] + __, basename_dest = op.split(fname_dest) + search_lines = [ + "DataFile=" + basename_src + ".eeg", + "DataFile=" + basename_src + ".dat", + "MarkerFile=" + basename_src + ".vmrk", + ] + + with open(vhdr_src, encoding=enc) as fin: + with open(vhdr_dest, "w", encoding=enc) as fout: + for line in fin.readlines(): + if line.strip() in search_lines: + line = line.replace(basename_src, basename_dest) + fout.write(line) + + with open(vmrk_file_path, encoding=enc) as fin: + with open(fname_dest + ".vmrk", "w", encoding=enc) as fout: + for line in fin.readlines(): + if line.strip() in search_lines: + line = line.replace(basename_src, basename_dest) + fout.write(line) + + if anonymize is not None: + raw = read_raw_brainvision(vhdr_src, preload=False, verbose=0) + daysback, keep_his, _ = _check_anonymize(anonymize, raw, ".vhdr") + raw.info = anonymize_info(raw.info, daysback=daysback, keep_his=keep_his) + _anonymize_brainvision(fname_dest + ".vhdr", date=raw.info["meas_date"]) + + for ext in [".eeg", ".vhdr", ".vmrk"]: + _, fname = os.path.split(fname_dest + ext) + dirname = op.dirname(op.realpath(vhdr_dest)) + logger.info(f'Created "{fname}" in "{dirname}".') + if anonymize: + logger.info("Anonymized all dates in VHDR and VMRK.") + + +def copyfile_edf(src, dest, anonymize=None): + """Copy an EDF, EDF+, or BDF file to a new location, optionally anonymize. + + .. warning:: EDF/EDF+/BDF files contain two fields for recording dates: + A generic "startdate" field that supports only 2-digit years, + and a "Startdate" field as part of the "local recording + identification", which supports 4-digit years. + If you want to anonymize your file, MNE-BIDS will set the + "startdate" field to 85 (i.e., 1985), the earliest possible + date for that field. However, the "Startdate" field in the + file's "local recording identification" and the date in the + session's corresponding ``scans.tsv`` will be set correctly + according to the argument provided to the ``anonymize`` + parameter. Note that it is possible that not all EDF/EDF+/BDF + reading software parses the accurate recording date, and + that for some reading software, the wrong year (1985) may + be parsed. + + Parameters + ---------- + src : path-like + The source path of the .edf or .bdf file to be copied. + dest : path-like + The destination path of the .edf or .bdf file. + anonymize : dict | None + If None (default), no anonymization is performed. + If dict, data will be anonymized depending on the keys provided with + the dict: `daysback` is a required key, `keep_his` is an optional key. + + `daysback` : int + Number of days by which to move back the recording date in time. + In studies with multiple subjects the relative recording date + differences between subjects can be kept by using the same number + of `daysback` for all subject anonymizations. `daysback` should be + great enough to shift the date prior to 1925 to conform with BIDS + anonymization rules. Due to limitations of the EDF/BDF format, the + year of the anonymized date will always be set to 1985 in the + 'startdate' field of the file. The correctly-shifted year will be + written to the 'local recording identification' region of the + file header, which may not be parsed by all EDF/EDF+/BDF reader + software. + + `keep_his` : bool + By default (False), all subject information next to the recording + date will be overwritten as well. If True, keep subject information + apart from the recording date. Participant names and birthdates + will always be anonymized if present, regardless of this setting. + + See Also + -------- + mne.io.anonymize_info + copyfile_brainvision + copyfile_bti + copyfile_ctf + copyfile_eeglab + copyfile_kit + + """ + # Ensure source & destination extensions are the same + fname_src, ext_src = _parse_ext(src) + fname_dest, ext_dest = _parse_ext(dest) + + if ext_src.lower() != ext_dest.lower(): + raise ValueError( + f"Need to move data with same extension, " + f' but got "{ext_src}" and "{ext_dest}"' + ) + + if ext_dest in [".EDF", ".BDF"]: + warn( + "Upper-case extension for EDF/BDF files is not supported " + "in BIDS. Converting destination extension to lower-case." + ) + ext_dest = ext_dest.lower() + dest = Path(dest).with_suffix(ext_dest) + + # Copy data prior to any anonymization + sh.copyfile(src, dest) + + # Anonymize EDF/BDF data, if requested + if anonymize is not None: + if ext_src in [".bdf", ".BDF"]: + raw = read_raw_bdf(dest, preload=False, verbose=0) + elif ext_src in [".edf", ".EDF"]: + raw = read_raw_edf(dest, preload=False, verbose=0) + else: + raise ValueError(f"Unsupported file type ({ext_src})") + + # Get subject info, recording info, and recording date + with open(dest, "rb") as f: + f.seek(8) # id_info field starts 8 bytes in + id_info = f.read(80).decode("ascii").rstrip() + rec_info = f.read(80).decode("ascii").rstrip() + + # Parse metadata from file + if len(id_info) == 0 or len(id_info.split(" ")) != 4: + id_info = "X X X X" + if len(rec_info) == 0 or len(rec_info.split(" ")) != 5: + rec_info = "Startdate X X X X" + pid, sex, birthdate, name = id_info.split(" ") + start_date, admin_code, tech, equip = rec_info.split(" ")[1:5] + + # Try to anonymize the recording date + daysback, keep_his, _ = _check_anonymize(anonymize, raw, ".edf") + anonymize_info(raw.info, daysback=daysback, keep_his=keep_his) + start_date = "01-JAN-1985" + meas_date = "01.01.85" + + # Anonymize ID info and write to file + if keep_his: + # Always remove participant birthdate and name to be safe + id_info = [pid, sex, "X", "X"] + rec_info = ["Startdate", start_date, admin_code, tech, equip] + else: + id_info = ["0", "X", "X", "X"] + rec_info = ["Startdate", start_date, "X", "mne-bids_anonymize", "X"] + with open(dest, "r+b") as f: + f.seek(8) # id_info field starts 8 bytes in + f.write(bytes(" ".join(id_info).ljust(80), "ascii")) + f.write(bytes(" ".join(rec_info).ljust(80), "ascii")) + f.write(bytes(meas_date, "ascii")) + + +def copyfile_eeglab(src, dest): + """Copy an EEGLAB file to a new location. + + If the EEGLAB ``.set`` file comes with an accompanying ``.fdt`` binary file + that contains the actual data, this function will copy this file, too, and + update all internal pointers in the new ``.set`` file. + + Parameters + ---------- + src : path-like + Path to the source raw .set file. + dest : path-like + Path to the destination of the new .set file. + + See Also + -------- + copyfile_brainvision + copyfile_bti + copyfile_ctf + copyfile_edf + copyfile_kit + + """ + # Get extension of the EEGLAB file + _, ext_src = _parse_ext(src) + fname_dest, ext_dest = _parse_ext(dest) + if ext_src != ext_dest: + raise ValueError( + f"Need to move data with same extension" f" but got {ext_src}, {ext_dest}" + ) + + # Load the EEG struct + # NOTE: do *not* simplify cells, because this changes the underlying + # structure and potentially breaks re-reading of the file + uint16_codec = None + eeg = loadmat( + file_name=src, + simplify_cells=False, + appendmat=False, + uint16_codec=uint16_codec, + mat_dtype=True, + ) + oldstyle = False + if "EEG" in eeg: + eeg = eeg["EEG"] + oldstyle = True + + has_fdt_link = False + try: + # If the data field is a string, it points to a .fdt file in src dir + if isinstance(eeg["data"][0, 0][0], str): + has_fdt_link = True + except IndexError: + pass + + if has_fdt_link: + fdt_fname = eeg["data"][0, 0][0] + + assert fdt_fname.endswith(".fdt"), f"Unexpected fdt name: {fdt_fname}" + head, _ = op.split(src) + fdt_path = op.join(head, fdt_fname) + + # Copy the .fdt file and give it a new name + fdt_name_new = fname_dest + ".fdt" + sh.copyfile(fdt_path, fdt_name_new) + + # Now adjust the pointer in the .set file + # NOTE: Clunky numpy code is to match MATLAB structure for "savemat" + _, tail = op.split(fdt_name_new) + new_value = np.empty((1, 1), dtype=object) + new_value[0, 0] = np.atleast_1d(np.array(tail)) + eeg["data"] = new_value + + # Save the EEG dictionary as a Matlab struct again + mdict = dict(EEG=eeg) if oldstyle else eeg + savemat(file_name=dest, mdict=mdict, appendmat=False) + else: + # If no .fdt file, simply copy the .set file, no modifications + # necessary + sh.copyfile(src, dest) + + +def copyfile_bti(raw, dest): + """Copy BTi data. + + Parameters + ---------- + raw : mne.io.Raw + An MNE-Python raw object of BTi data. + dest : path-like + Destination to copy the BTi data to. + + See Also + -------- + copyfile_brainvision + copyfile_ctf + copyfile_edf + copyfile_eeglab + copyfile_kit + + """ + os.makedirs(dest, exist_ok=True) + for key, val in ( + ("pdf_fname", None), + ("config_fname", "config"), + ("head_shape_fname", "hs_file"), + ): + keyfile = raw._raw_extras[0].get(key) + + # Keep name of pdf file + if key == "pdf_fname": + val = op.basename(keyfile) + + # If no headshape file present, cannot copy it + if key == "head_shape_fname" and keyfile is None: + continue + + sh.copyfile(keyfile, op.join(dest, val)) diff --git a/mne-bids-0.15/mne_bids/data/space-ICBM452AirSpace_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-ICBM452AirSpace_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..6acfa03a12301663e3c272c530cee1508b89035a Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-ICBM452AirSpace_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-ICBM452AirSpace_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-ICBM452AirSpace_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..926d79c270e29dc47bacc5d1eadbfa5a98bccbb6 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-ICBM452AirSpace_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-ICBM452AirSpace_trans.fif b/mne-bids-0.15/mne_bids/data/space-ICBM452AirSpace_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..ceb4b1f667de9310c1b758e9350d82fbc92eda3e Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-ICBM452AirSpace_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-ICBM452AirSpace_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-ICBM452AirSpace_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..1cbaf18695f85ce46bded27c7392862ac2851b55 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-ICBM452AirSpace_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-ICBM452Warp5Space_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-ICBM452Warp5Space_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..ad2c1920f83321c6833a1e54a8629fbd71c8fdb6 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-ICBM452Warp5Space_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-ICBM452Warp5Space_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-ICBM452Warp5Space_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..926d79c270e29dc47bacc5d1eadbfa5a98bccbb6 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-ICBM452Warp5Space_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-ICBM452Warp5Space_trans.fif b/mne-bids-0.15/mne_bids/data/space-ICBM452Warp5Space_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..7405fbfade07947fb5eb6cf059e07051bd0afc53 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-ICBM452Warp5Space_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-ICBM452Warp5Space_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-ICBM452Warp5Space_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..1cbaf18695f85ce46bded27c7392862ac2851b55 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-ICBM452Warp5Space_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-IXI549Space_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-IXI549Space_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..ef056b5c0c5da5554cf629790897219bcf43833e Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-IXI549Space_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-IXI549Space_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-IXI549Space_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..703a181a4e1bdba4c65445bf53cc0209b0f660b2 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-IXI549Space_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-IXI549Space_trans.fif b/mne-bids-0.15/mne_bids/data/space-IXI549Space_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..167092587ae1ca6e1c708f833052a7a5c4070c45 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-IXI549Space_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-IXI549Space_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-IXI549Space_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..c91168d8e5a059ff2fbfffdcd6f44cbd458ab6f6 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-IXI549Space_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152Lin_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-MNI152Lin_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..0c3b9b110b3563d289c7d0a1816777774d9376c1 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152Lin_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152Lin_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152Lin_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..50dbe4684999eccafa37ce758e58290c8f26bae4 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152Lin_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152Lin_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152Lin_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..b25f8696a3c2618ac558e017c788b07b51c2df35 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152Lin_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152Lin_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152Lin_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..4fff658da866496c17270e1c9ef50a6013664958 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152Lin_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aAsym_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aAsym_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..510bf28cc133ad07e3628af6b1635b9c12aadba7 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aAsym_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aAsym_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aAsym_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..238fc12f7c033bb4e1f0b416b1cd0c31c3191708 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aAsym_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aAsym_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aAsym_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..fb5bdfe16a5b87b872f757b935818665c7ae809b Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aAsym_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aAsym_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aAsym_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..95e5edbaa232a48ea077c7018cc844606447540b Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aAsym_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aSym_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aSym_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..510bf28cc133ad07e3628af6b1635b9c12aadba7 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aSym_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aSym_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aSym_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..238fc12f7c033bb4e1f0b416b1cd0c31c3191708 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aSym_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aSym_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aSym_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..fb5bdfe16a5b87b872f757b935818665c7ae809b Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aSym_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aSym_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aSym_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..95e5edbaa232a48ea077c7018cc844606447540b Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009aSym_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bAsym_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bAsym_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..0e532a4c7f827a5f18870b986a60614e7717311b Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bAsym_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bAsym_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bAsym_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..3db48f74f33d0e137a93c5008508748f3b2c6fe7 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bAsym_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bAsym_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bAsym_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..f1ee8e21464733a0bb6333d53bac5341944a9985 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bAsym_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bAsym_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bAsym_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..fdf63c335d38651724e1effc523f2dc1a868b938 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bAsym_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bSym_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bSym_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..0e532a4c7f827a5f18870b986a60614e7717311b Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bSym_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bSym_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bSym_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..3db48f74f33d0e137a93c5008508748f3b2c6fe7 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bSym_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bSym_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bSym_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..f1ee8e21464733a0bb6333d53bac5341944a9985 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bSym_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bSym_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bSym_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..fdf63c335d38651724e1effc523f2dc1a868b938 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009bSym_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cAsym_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cAsym_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..cd4563ab8af315d903309b48ecceeaaafe5fbea7 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cAsym_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cAsym_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cAsym_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..2e6f140242a5b338b834f13ef45332401c373fab Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cAsym_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cAsym_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cAsym_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..51dd0afd1ad827bdcb81374bf363fea5863c9284 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cAsym_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cAsym_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cAsym_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..9f3b271be9c2d66a082d331ce200c15a717b35d9 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cAsym_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cSym_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cSym_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..cd4563ab8af315d903309b48ecceeaaafe5fbea7 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cSym_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cSym_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cSym_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..2e6f140242a5b338b834f13ef45332401c373fab Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cSym_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cSym_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cSym_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..51dd0afd1ad827bdcb81374bf363fea5863c9284 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cSym_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cSym_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cSym_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..9f3b271be9c2d66a082d331ce200c15a717b35d9 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin2009cSym_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin6ASym_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin6ASym_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..7b0787fefceb7ee99f10bfcb2871434dc979ae8b Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin6ASym_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin6ASym_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin6ASym_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..ed09e3804492468d9037618576db9c812e831080 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin6ASym_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin6ASym_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin6ASym_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..84a5d39422522cfbee91f28bff2fc71c43f11a00 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin6ASym_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin6ASym_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin6ASym_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..264ad65a0d8a6f8f9d640ae2a90d963afaf38e4a Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin6ASym_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin6Sym_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin6Sym_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..673671d648e5073a0af9d2ae14572f8ebcc4c439 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin6Sym_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin6Sym_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin6Sym_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..25ac50cd26d5ca98cbe31c469128844f4a16aea2 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin6Sym_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin6Sym_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin6Sym_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..e93e5fa7b8d754f9d4e1f7148f53385616114a80 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin6Sym_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI152NLin6Sym_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI152NLin6Sym_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..9f3b271be9c2d66a082d331ce200c15a717b35d9 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI152NLin6Sym_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI305_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-MNI305_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..0ba25e0a533f189e4ec965e0b458d42ed2f657e7 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI305_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI305_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI305_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..14b762d4664435153467346c493e5af58cb274dd Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI305_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI305_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI305_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..f369bbe06abec8be41e61d011e1c91dc22802a8e Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI305_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNI305_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNI305_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..ea19506eaaf7ec5a8f3e7e7f9f45a7ee957acb06 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNI305_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNIColin27_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-MNIColin27_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..b76e97fe77ab0cd8b049cf3cc1986dd87edb99e6 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNIColin27_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNIColin27_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNIColin27_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..18a9a0f74227504e7cd7f89ea283846db1766ed6 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNIColin27_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNIColin27_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNIColin27_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..a55bfbd0cbae9423f9420390b3bc721ab1e47b34 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNIColin27_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-MNIColin27_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-MNIColin27_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..de4b301c17eb56634c5320e2577c0a192a498f85 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-MNIColin27_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-NIHPD_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-NIHPD_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..44e25ac43bc989ac531e9bb10383de9c9ffeb407 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-NIHPD_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-NIHPD_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-NIHPD_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..238fc12f7c033bb4e1f0b416b1cd0c31c3191708 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-NIHPD_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-NIHPD_trans.fif b/mne-bids-0.15/mne_bids/data/space-NIHPD_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..c4049968bd01024d540742fab1b2959ba54a9b76 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-NIHPD_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-NIHPD_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-NIHPD_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..95e5edbaa232a48ea077c7018cc844606447540b Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-NIHPD_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-OASIS30AntsOASISAnts_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-OASIS30AntsOASISAnts_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..c3165847f44132ca8d2839ce8a98a7a4da7e3b1c Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-OASIS30AntsOASISAnts_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-OASIS30AntsOASISAnts_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-OASIS30AntsOASISAnts_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..fc263347cceb167498721993f6b4daf77a96acd2 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-OASIS30AntsOASISAnts_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-OASIS30AntsOASISAnts_trans.fif b/mne-bids-0.15/mne_bids/data/space-OASIS30AntsOASISAnts_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..082efa55e7ca3fcf3c7faaf8835e9c9ce5f39594 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-OASIS30AntsOASISAnts_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-OASIS30AntsOASISAnts_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-OASIS30AntsOASISAnts_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..5b69b00af07e3348eec1f0452bdab6926067da7f Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-OASIS30AntsOASISAnts_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-OASIS30Atropos_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-OASIS30Atropos_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..ab2b7490f45adda41e2496e576e3c4d366024f4c Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-OASIS30Atropos_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-OASIS30Atropos_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-OASIS30Atropos_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..fc263347cceb167498721993f6b4daf77a96acd2 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-OASIS30Atropos_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-OASIS30Atropos_trans.fif b/mne-bids-0.15/mne_bids/data/space-OASIS30Atropos_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..24eb8f93f50b38f2248eee17588d259a95c3a5ef Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-OASIS30Atropos_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-OASIS30Atropos_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-OASIS30Atropos_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..5b69b00af07e3348eec1f0452bdab6926067da7f Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-OASIS30Atropos_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-Talairach_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-Talairach_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..76b615bf24592cc5a8cef29dbdee92ef255a8ce3 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-Talairach_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-Talairach_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-Talairach_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..f3d4a84fc914a32426b2ea2fc034e829038ca6ab Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-Talairach_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-Talairach_trans.fif b/mne-bids-0.15/mne_bids/data/space-Talairach_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..17e565cbb88dfebfede96ff97bb1d529b058e0c6 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-Talairach_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-Talairach_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-Talairach_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..b2bd237328e81a6f0002938998a5fe4018121a5c Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-Talairach_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-UNCInfant_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-UNCInfant_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..18518a352e305a6fdb541f1351d3c47e1e0f306b Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-UNCInfant_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-UNCInfant_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-UNCInfant_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..170330b8f8dccc1a552d8398f8baf8ae1bd26419 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-UNCInfant_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-UNCInfant_trans.fif b/mne-bids-0.15/mne_bids/data/space-UNCInfant_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..a94c6fa3c93f45ad1d7d0f52c3b72fcd4b5c0449 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-UNCInfant_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-UNCInfant_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-UNCInfant_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..4fff658da866496c17270e1c9ef50a6013664958 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-UNCInfant_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-fsLR_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-fsLR_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..7b0787fefceb7ee99f10bfcb2871434dc979ae8b Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-fsLR_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-fsLR_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-fsLR_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..ed09e3804492468d9037618576db9c812e831080 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-fsLR_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-fsLR_trans.fif b/mne-bids-0.15/mne_bids/data/space-fsLR_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..84a5d39422522cfbee91f28bff2fc71c43f11a00 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-fsLR_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-fsLR_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-fsLR_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..264ad65a0d8a6f8f9d640ae2a90d963afaf38e4a Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-fsLR_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-fsaverageSym_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-fsaverageSym_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..4f0bd2e989fec02cfde5acbd1eec8fc8d39673c2 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-fsaverageSym_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-fsaverageSym_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-fsaverageSym_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..14b762d4664435153467346c493e5af58cb274dd Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-fsaverageSym_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-fsaverageSym_trans.fif b/mne-bids-0.15/mne_bids/data/space-fsaverageSym_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..055731ca3e23be7bae75a3d63e9436dd5838f507 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-fsaverageSym_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-fsaverageSym_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-fsaverageSym_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..ea19506eaaf7ec5a8f3e7e7f9f45a7ee957acb06 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-fsaverageSym_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-fsaverage_fiducials.fif b/mne-bids-0.15/mne_bids/data/space-fsaverage_fiducials.fif new file mode 100644 index 0000000000000000000000000000000000000000..0ba25e0a533f189e4ec965e0b458d42ed2f657e7 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-fsaverage_fiducials.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-fsaverage_ras-vox_trans.fif b/mne-bids-0.15/mne_bids/data/space-fsaverage_ras-vox_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..14b762d4664435153467346c493e5af58cb274dd Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-fsaverage_ras-vox_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-fsaverage_trans.fif b/mne-bids-0.15/mne_bids/data/space-fsaverage_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..f369bbe06abec8be41e61d011e1c91dc22802a8e Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-fsaverage_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/data/space-fsaverage_vox-mri_trans.fif b/mne-bids-0.15/mne_bids/data/space-fsaverage_vox-mri_trans.fif new file mode 100644 index 0000000000000000000000000000000000000000..ea19506eaaf7ec5a8f3e7e7f9f45a7ee957acb06 Binary files /dev/null and b/mne-bids-0.15/mne_bids/data/space-fsaverage_vox-mri_trans.fif differ diff --git a/mne-bids-0.15/mne_bids/dig.py b/mne-bids-0.15/mne_bids/dig.py new file mode 100644 index 0000000000000000000000000000000000000000..1b372fa00f51aa8048a3e463eb0b47356183f557 --- /dev/null +++ b/mne-bids-0.15/mne_bids/dig.py @@ -0,0 +1,793 @@ +"""Read/write BIDS compatible electrode/coords structures from MNE.""" + +# Authors: Adam Li +# Stefan Appelhoff +# Alex Rockhill +# +# License: BSD-3-Clause +import json +import os.path as op +import re +import warnings +from collections import OrderedDict +from pathlib import Path + +import mne +import numpy as np +from mne.io.constants import FIFF +from mne.io.pick import _picks_to_idx +from mne.utils import _check_option, _validate_type, get_subjects_dir, logger + +from mne_bids.config import ( + ALLOWED_SPACES, + BIDS_COORD_FRAME_DESCRIPTIONS, + BIDS_COORDINATE_UNITS, + BIDS_STANDARD_TEMPLATE_COORDINATE_SYSTEMS, + BIDS_TO_MNE_FRAMES, + MNE_FRAME_TO_STR, + MNE_STR_TO_FRAME, + MNE_TO_BIDS_FRAMES, +) +from mne_bids.path import BIDSPath +from mne_bids.tsv_handler import _from_tsv +from mne_bids.utils import ( + _import_nibabel, + _scale_coord_to_meters, + _write_json, + _write_tsv, + verbose, + warn, +) + +data_dir = Path(__file__).parent / "data" + + +def _handle_electrodes_reading(electrodes_fname, coord_frame, coord_unit): + """Read associated electrodes.tsv and populate raw. + + Handle xyz coordinates and coordinate frame of each channel. + """ + logger.info("Reading electrode " f"coords from {electrodes_fname}.") + electrodes_dict = _from_tsv(electrodes_fname) + ch_names_tsv = electrodes_dict["name"] + + def _float_or_nan(val): + if val == "n/a": + return np.nan + else: + return float(val) + + # convert coordinates to float and create list of tuples + electrodes_dict["x"] = [_float_or_nan(x) for x in electrodes_dict["x"]] + electrodes_dict["y"] = [_float_or_nan(x) for x in electrodes_dict["y"]] + electrodes_dict["z"] = [_float_or_nan(x) for x in electrodes_dict["z"]] + ch_names_raw = [ + x for i, x in enumerate(ch_names_tsv) if electrodes_dict["x"][i] != "n/a" + ] + ch_locs = np.c_[electrodes_dict["x"], electrodes_dict["y"], electrodes_dict["z"]] + + # convert coordinates to meters + ch_locs = _scale_coord_to_meters(ch_locs, coord_unit) + + # create mne.DigMontage + ch_pos = dict(zip(ch_names_raw, ch_locs)) + montage = mne.channels.make_dig_montage(ch_pos=ch_pos, coord_frame=coord_frame) + return montage + + +def _handle_coordsystem_reading(coordsystem_fpath, datatype): + """Read associated coordsystem.json. + + Handle reading the coordinate frame and coordinate unit + of each electrode. + """ + with open(coordsystem_fpath, encoding="utf-8-sig") as fin: + coordsystem_json = json.load(fin) + + if datatype == "meg": + coord_frame = coordsystem_json["MEGCoordinateSystem"] + coord_unit = coordsystem_json["MEGCoordinateUnits"] + coord_frame_desc = coordsystem_json.get("MEGCoordinateDescription", None) + elif datatype == "eeg": + coord_frame = coordsystem_json["EEGCoordinateSystem"] + coord_unit = coordsystem_json["EEGCoordinateUnits"] + coord_frame_desc = coordsystem_json.get("EEGCoordinateDescription", None) + elif datatype == "ieeg": + coord_frame = coordsystem_json["iEEGCoordinateSystem"] + coord_unit = coordsystem_json["iEEGCoordinateUnits"] + coord_frame_desc = coordsystem_json.get("iEEGCoordinateDescription", None) + + msg = f"Reading coordinate system frame {coord_frame}" + if coord_frame_desc: + msg += f": {coord_frame_desc}" + + return coord_frame, coord_unit + + +def _get_impedances(raw, names): + """Get the impedance values in kOhm from raw.impedances.""" + if not hasattr(raw, "impedances"): # pragma: no cover + return ["n/a"] * len(names) + no_info = {"imp": np.nan, "imp_unit": "kOhm"} + impedance_dicts = [raw.impedances.get(name, no_info) for name in names] + # If we encounter a unit not defined in `scalings`, return NaN + scalings = {"kOhm": 1, "Ohm": 0.001} + impedances = [ + imp_dict["imp"] * scalings.get(imp_dict["imp_unit"], np.nan) + for imp_dict in impedance_dicts + ] + # replace np.nan with BIDS 'n/a' representation + impedances = [i if not np.isnan(i) else "n/a" for i in impedances] + return impedances + + +def _write_electrodes_tsv(raw, fname, datatype, overwrite=False): + """Create an electrodes.tsv file and save it. + + Parameters + ---------- + raw : mne.io.Raw + The data as MNE-Python Raw object. + fname : str + Filename to save the electrodes.tsv to. + datatype : str + Type of the data recording. Can be ``meg``, ``eeg``, + or ``ieeg``. + overwrite : bool + Defaults to False. + Whether to overwrite the existing data in the file. + If there is already data for the given `fname` and overwrite is False, + an error will be raised. + + """ + # create list of channel coordinates and names + x, y, z, names = list(), list(), list(), list() + for ch in raw.info["chs"]: + if ch["kind"] == FIFF.FIFFV_STIM_CH: + logger.debug(f"Not writing stim chan {ch['ch_name']} " f"to electrodes.tsv") + continue + elif np.isnan(ch["loc"][:3]).any() or np.allclose(ch["loc"][:3], 0): + x.append("n/a") + y.append("n/a") + z.append("n/a") + else: + x.append(ch["loc"][0]) + y.append(ch["loc"][1]) + z.append(ch["loc"][2]) + names.append(ch["ch_name"]) + + # create OrderedDict to write to tsv file + if datatype == "ieeg": + # XXX: size should be included in the future + sizes = ["n/a"] * len(names) + data = OrderedDict( + [ + ("name", names), + ("x", x), + ("y", y), + ("z", z), + ("size", sizes), + ] + ) + elif datatype == "eeg": + data = OrderedDict( + [ + ("name", names), + ("x", x), + ("y", y), + ("z", z), + ] + ) + else: # pragma: no cover + raise RuntimeError(f"datatype {datatype} not supported.") + + # Add impedance values if available, currently only BrainVision: + # https://github.com/mne-tools/mne-python/pull/7974 + if hasattr(raw, "impedances"): + data["impedance"] = _get_impedances(raw, names) + + # note that any coordsystem.json file shared within sessions + # will be the same across all runs (currently). So + # overwrite is set to True always + # XXX: improve later when BIDS is updated + # check that there already exists a coordsystem.json + if Path(fname).exists() and not overwrite: + electrodes_tsv = _from_tsv(fname) + + # cast values to str to make equality check work + if any( + [ + list(map(str, vals1)) != list(vals2) + for vals1, vals2 in zip(data.values(), electrodes_tsv.values()) + ] + ): + raise RuntimeError( + f"Trying to write electrodes.tsv, but it already " + f"exists at {fname} and the contents do not match. " + f"You must differentiate this electrodes.tsv file " + f'from the existing one, or set "overwrite" to True.' + ) + _write_tsv(fname, data, overwrite=True) + + +def _write_optodes_tsv(raw, fname, overwrite=False, verbose=True): + """Create a optodes.tsv file and save it. + + Parameters + ---------- + raw : instance of Raw + The data as MNE-Python Raw object. + fname : str | BIDSPath + Filename to save the optodes.tsv to. + overwrite : bool + Whether to overwrite the existing file. + Defaults to False. + verbose : bool + Set verbose output to True or False. + """ + picks = _picks_to_idx(raw.info, "fnirs", exclude=[], allow_empty=True) + sources = np.zeros(picks.shape) + detectors = np.zeros(picks.shape) + for ii in picks: + # NIRS channel names take a specific form in MNE-Python. + # The channel names always reflect the source and detector + # pair, followed by the wavelength frequency. + # The following code extracts the source and detector + # numbers from the channel name. + ch1_name_info = re.match(r"S(\d+)_D(\d+) (\d+)", raw.info["chs"][ii]["ch_name"]) + sources[ii] = ch1_name_info.groups()[0] + detectors[ii] = ch1_name_info.groups()[1] + unique_sources = np.unique(sources) + n_sources = len(unique_sources) + unique_detectors = np.unique(detectors) + names = np.concatenate( + ( + ["S" + str(s) for s in unique_sources.astype(int)], + ["D" + str(d) for d in unique_detectors.astype(int)], + ) + ) + + xs = np.zeros(names.shape) + ys = np.zeros(names.shape) + zs = np.zeros(names.shape) + for i, source in enumerate(unique_sources): + s_idx = np.where(sources == source)[0][0] + xs[i] = raw.info["chs"][s_idx]["loc"][3] + ys[i] = raw.info["chs"][s_idx]["loc"][4] + zs[i] = raw.info["chs"][s_idx]["loc"][5] + for i, detector in enumerate(unique_detectors): + d_idx = np.where(detectors == detector)[0][0] + xs[i + n_sources] = raw.info["chs"][d_idx]["loc"][6] + ys[i + n_sources] = raw.info["chs"][d_idx]["loc"][7] + zs[i + n_sources] = raw.info["chs"][d_idx]["loc"][8] + + ch_data = { + "name": names, + "type": np.concatenate( + ( + np.full(len(unique_sources), "source"), + np.full(len(unique_detectors), "detector"), + ) + ), + "x": xs, + "y": ys, + "z": zs, + } + _write_tsv(fname, ch_data, overwrite, verbose) + + +def _write_coordsystem_json( + *, + raw, + unit, + hpi_coord_system, + sensor_coord_system, + fname, + datatype, + overwrite=False, +): + """Create a coordsystem.json file and save it. + + Parameters + ---------- + raw : mne.io.Raw + The data as MNE-Python Raw object. + unit : str + Units to be used in the coordsystem specification, + as in BIDS_COORDINATE_UNITS. + hpi_coord_system : str + Name of the coordinate system for the head coils. + sensor_coord_system : str | tuple of str + Name of the coordinate system for the sensor positions. + If a tuple of strings, should be in the form: + ``(BIDS coordinate frame, MNE coordinate frame)``. + fname : str + Filename to save the coordsystem.json to. + datatype : str + Type of the data recording. Can be ``meg``, ``eeg``, + or ``ieeg``. + overwrite : bool + Whether to overwrite the existing file. + Defaults to False. + + """ + if raw.get_montage() is None: + dig = list() + coords = dict() + else: + montage = raw.get_montage() + pos = montage.get_positions() + dig = list() if montage.dig is None else montage.dig + coords = dict( + NAS=list() if pos["nasion"] is None else pos["nasion"].tolist(), + LPA=list() if pos["lpa"] is None else pos["lpa"].tolist(), + RPA=list() if pos["rpa"] is None else pos["rpa"].tolist(), + ) + + # get the coordinate frame description + sensor_coord_system_descr = BIDS_COORD_FRAME_DESCRIPTIONS.get( + sensor_coord_system.lower(), "n/a" + ) + + # create the coordinate json data structure based on 'datatype' + if datatype == "meg": + hpi = {d["ident"]: d for d in dig if d["kind"] == FIFF.FIFFV_POINT_HPI} + if hpi: + for ident in hpi.keys(): + coords["coil%d" % ident] = hpi[ident]["r"].tolist() + + fid_json = { + "MEGCoordinateSystem": sensor_coord_system, + "MEGCoordinateUnits": unit, # XXX validate this + "MEGCoordinateSystemDescription": sensor_coord_system_descr, + "HeadCoilCoordinates": coords, + "HeadCoilCoordinateSystem": hpi_coord_system, + "HeadCoilCoordinateUnits": unit, # XXX validate this + "AnatomicalLandmarkCoordinates": coords, + "AnatomicalLandmarkCoordinateSystem": sensor_coord_system, + "AnatomicalLandmarkCoordinateUnits": unit, + } + elif datatype == "eeg": + fid_json = { + "EEGCoordinateSystem": sensor_coord_system, + "EEGCoordinateUnits": unit, + "EEGCoordinateSystemDescription": sensor_coord_system_descr, + "AnatomicalLandmarkCoordinates": coords, + "AnatomicalLandmarkCoordinateSystem": sensor_coord_system, + "AnatomicalLandmarkCoordinateUnits": unit, + } + elif datatype == "ieeg": + fid_json = { + # (Other, Pixels, ACPC) + "iEEGCoordinateSystem": sensor_coord_system, + "iEEGCoordinateSystemDescription": sensor_coord_system_descr, + "iEEGCoordinateUnits": unit, # m (MNE), mm, cm , or pixels + } + elif datatype == "nirs": + fid_json = { + "NIRSCoordinateSystem": sensor_coord_system, + "NIRSCoordinateSystemDescription": sensor_coord_system_descr, + "NIRSCoordinateUnits": unit, + } + + # note that any coordsystem.json file shared within sessions + # will be the same across all runs (currently). So + # overwrite is set to True always + # XXX: improve later when BIDS is updated + # check that there already exists a coordsystem.json + if Path(fname).exists() and not overwrite: + with open(fname, encoding="utf-8-sig") as fin: + coordsystem_dict = json.load(fin) + if fid_json != coordsystem_dict: + raise RuntimeError( + f"Trying to write coordsystem.json, but it already " + f"exists at {fname} and the contents do not match. " + f"You must differentiate this coordsystem.json file " + f'from the existing one, or set "overwrite" to True.' + ) + _write_json(fname, fid_json, overwrite=True) + + +def _write_dig_bids(bids_path, raw, montage=None, acpc_aligned=False, overwrite=False): + """Write BIDS formatted DigMontage from Raw instance. + + Handles coordsystem.json and electrodes.tsv writing + from DigMontage. + + Parameters + ---------- + bids_path : BIDSPath + Path in the BIDS dataset to save the ``electrodes.tsv`` + and ``coordsystem.json`` file for. ``datatype`` + attribute must be ``eeg``, or ``ieeg``. For ``meg`` + data, ``electrodes.tsv`` are not saved. + raw : mne.io.Raw + The data as MNE-Python Raw object. + montage : mne.channels.DigMontage | None + The montage to use rather than the one in ``raw`` if it + must be transformed from the "head" coordinate frame. + acpc_aligned : bool + Whether "mri" space is aligned to ACPC. + overwrite : bool + Whether to overwrite the existing file. + Defaults to False. + + """ + # write electrodes data for iEEG and EEG + unit = "m" # defaults to meters + + if montage is None: + montage = raw.get_montage() + else: # assign montage to raw but supress any coordinate transforms + montage = montage.copy() # don't modify original + montage_coord_frame = montage.get_positions()["coord_frame"] + fids = [ + d + for d in montage.dig # save to add back + if d["kind"] == FIFF.FIFFV_POINT_CARDINAL + ] + montage.remove_fiducials() # prevent coordinate transform + with warnings.catch_warnings(): + warnings.filterwarnings( + action="ignore", + category=RuntimeWarning, + message=".*nasion not found", + module="mne", + ) + raw.set_montage(montage) + for ch in raw.info["chs"]: + ch["coord_frame"] = MNE_STR_TO_FRAME[montage_coord_frame] + for d in raw.info["dig"]: + d["coord_frame"] = MNE_STR_TO_FRAME[montage_coord_frame] + with raw.info._unlock(): # add back fiducials + raw.info["dig"] = fids + raw.info["dig"] + + # get the accepted mne-python coordinate frames + coord_frame_int = int(montage.dig[0]["coord_frame"]) + mne_coord_frame = MNE_FRAME_TO_STR.get(coord_frame_int, None) + coord_frame = MNE_TO_BIDS_FRAMES.get(mne_coord_frame, None) + + if coord_frame == "CapTrak" and bids_path.datatype in ("eeg", "nirs"): + pos = raw.get_montage().get_positions() + if any([pos[fid_key] is None for fid_key in ("nasion", "lpa", "rpa")]): + raise RuntimeError( + "'head' coordinate frame must contain nasion " + "and left and right pre-auricular point " + "landmarks" + ) + + if ( + bids_path.datatype == "ieeg" + and bids_path.space in (None, "ACPC") + and mne_coord_frame == "ras" + ): + if not acpc_aligned: + raise RuntimeError( + "`acpc_aligned` is False, if your T1 is not aligned " + "to ACPC and the coordinates are in fact in ACPC " + "space there will be no way to relate the coordinates " + "to the T1. If the T1 is ACPC-aligned, use " + "`acpc_aligned=True`" + ) + coord_frame = "ACPC" + + if bids_path.space is None: # no space, use MNE coord frame + if coord_frame is None: # if no MNE coord frame, skip + warn( + "Coordinate frame could not be inferred from the raw object " + "and the BIDSPath.space was none, skipping the writing of " + "channel positions" + ) + return + else: # either a space and an MNE coord frame or just a space + if coord_frame is None: # just a space, use that + coord_frame = bids_path.space + else: # space and raw have coordinate frame, check match + if bids_path.space != coord_frame and not ( + coord_frame == "fsaverage" and bids_path.space == "MNI305" + ): # fsaverage == MNI305 + raise ValueError( + "Coordinates in the raw object or montage " + f"are in the {coord_frame} coordinate " + "frame but BIDSPath.space is " + f"{bids_path.space}" + ) + + # create electrodes/coordsystem files using a subset of entities + # that are specified for these files in the specification + coord_file_entities = { + "root": bids_path.root, + "datatype": bids_path.datatype, + "subject": bids_path.subject, + "session": bids_path.session, + "acquisition": bids_path.acquisition, + "space": None if bids_path.datatype == "nirs" else coord_frame, + } + channels_suffix = "optodes" if bids_path.datatype == "nirs" else "electrodes" + _channels_fun = ( + _write_optodes_tsv if bids_path.datatype == "nirs" else _write_electrodes_tsv + ) + channels_path = BIDSPath( + **coord_file_entities, suffix=channels_suffix, extension=".tsv" + ) + coordsystem_path = BIDSPath( + **coord_file_entities, suffix="coordsystem", extension=".json" + ) + + # Now write the data to the elec coords and the coordsystem + _channels_fun(raw, channels_path, bids_path.datatype, overwrite) + _write_coordsystem_json( + raw=raw, + unit=unit, + hpi_coord_system="n/a", + sensor_coord_system=coord_frame, + fname=coordsystem_path, + datatype=bids_path.datatype, + overwrite=overwrite, + ) + + +def _read_dig_bids(electrodes_fpath, coordsystem_fpath, datatype, raw): + """Read MNE-Python formatted DigMontage from BIDS files. + + Handles coordinatesystem.json and electrodes.tsv reading + to DigMontage. + + Parameters + ---------- + electrodes_fpath : str + Filepath of the electrodes.tsv to read. + coordsystem_fpath : str + Filepath of the coordsystem.json to read. + datatype : str + Type of the data recording. Can be ``meg``, ``eeg``, + or ``ieeg``. + raw : mne.io.Raw + The raw data as MNE-Python ``Raw`` object. The montage + will be set in place. + """ + bids_coord_frame, bids_coord_unit = _handle_coordsystem_reading( + coordsystem_fpath, datatype + ) + + if bids_coord_frame not in ALLOWED_SPACES[datatype]: + warn( + f'"{bids_coord_frame}" is not a BIDS-acceptable coordinate frame ' + f"for {datatype.upper()}. The supported coordinate frames are: " + "{}".format(ALLOWED_SPACES[datatype]) + ) + coord_frame = None + elif bids_coord_frame in BIDS_TO_MNE_FRAMES: + coord_frame = BIDS_TO_MNE_FRAMES.get(bids_coord_frame, None) + else: + warn( + f"{bids_coord_frame} is not an MNE-Python coordinate frame " + f"for {datatype.upper()} data and so will be set to 'unknown'" + ) + coord_frame = "unknown" + + # check coordinate units + if bids_coord_unit not in BIDS_COORDINATE_UNITS: + warn( + f"Coordinate unit is not an accepted BIDS unit for " + f"{electrodes_fpath}. Please specify to be one of " + f"{BIDS_COORDINATE_UNITS}. Skipping electrodes.tsv reading..." + ) + coord_frame = None + + # montage is interpretable only if coordinate frame was properly parsed + if coord_frame is not None: + # read in electrode coordinates as a DigMontage object + montage = _handle_electrodes_reading( + electrodes_fpath, coord_frame, bids_coord_unit + ) + else: + montage = None + + if montage is not None: + # determine if there are problematic channels + ch_pos = montage._get_ch_pos() + nan_chs = [] + for ch_name, ch_coord in ch_pos.items(): + if any(np.isnan(ch_coord)) and ch_name not in raw.info["bads"]: + nan_chs.append(ch_name) + if len(nan_chs) > 0: + warn( + f"There are channels without locations " + f"(n/a) that are not marked as bad: {nan_chs}" + ) + + # add montage to Raw object + # XXX: Starting with mne 0.24, this will raise a RuntimeWarning + # if channel types are included outside of + # (EEG/sEEG/ECoG/DBS/fNIRS). Probably needs a fix in the future. + with warnings.catch_warnings(): + warnings.filterwarnings( + action="ignore", + category=RuntimeWarning, + message=".*nasion not found", + module="mne", + ) + raw.set_montage(montage, on_missing="warn") + + # put back in unknown for unknown coordinate frame + if coord_frame == "unknown": + for ch in raw.info["chs"]: + ch["coord_frame"] = MNE_STR_TO_FRAME["unknown"] + for d in raw.info["dig"]: + d["coord_frame"] = MNE_STR_TO_FRAME["unknown"] + + +@verbose +def template_to_head(info, space, coord_frame="auto", unit="auto", verbose=None): + """Transform a BIDS standard template montage to the head coordinate frame. + + Parameters + ---------- + %(info_not_none)s The info is modified in place. + space : str + The name of the BIDS standard template. See + https://bids-specification.readthedocs.io/en/latest/appendices/coordinate-systems.html#standard-template-identifiers + for a list of acceptable spaces. + coord_frame : 'mri' | 'mri_voxel' | 'ras' + BIDS template coordinate systems do not specify a coordinate frame, + so this must be determined by inspecting the documentation for the + dataset or the ``electrodes.tsv`` file. If ``'auto'``, the coordinate + frame is assumed to be ``'mri_voxel'`` if the coordinates are strictly + positive, and ``'ras'`` (``"scanner RAS"``) otherwise. + + .. warning:: + + ``scanner RAS`` and ``surface RAS`` coordinates frames are similar + so be very careful not to assume a BIDS dataset's coordinates are + in one when they are actually in the other. The only way to tell + for template coordinate systems, currently, is if it is specified + in the dataset documentation. + + unit : 'm' | 'mm' | 'auto' + The unit that was used in the coordinate system specification. + If ``'auto'``, ``'m'`` will be inferred if the montage + spans less than ``-1`` to ``1``, and ``'mm'`` otherwise. If the + ``coord_frame`` is ``'mri_voxel'``, ``unit`` will be ignored. + %(verbose)s + + Returns + ------- + %(info_not_none)s The modified ``Info`` object. + trans : mne.transforms.Transform + The data transformation matrix from ``'head'`` to ``'mri'`` + coordinates. + + """ + _validate_type(info, mne.Info) + _check_option("space", space, BIDS_STANDARD_TEMPLATE_COORDINATE_SYSTEMS) + _check_option("coord_frame", coord_frame, ("auto", "mri", "mri_voxel", "ras")) + _check_option("unit", unit, ("auto", "m", "mm")) + montage = info.get_montage() + if montage is None: + raise RuntimeError("No montage found in the `raw` object") + montage.remove_fiducials() # we will add fiducials so remove any + pos = montage.get_positions() + if pos["coord_frame"] not in ("mni_tal", "unknown"): + raise RuntimeError( + "Montage coordinate frame '{}' not expected for a template " + "montage, should be 'unknown' or 'mni_tal'".format(pos["coord_frame"]) + ) + locs = np.array(list(pos["ch_pos"].values())) + locs = locs[~np.any(np.isnan(locs), axis=1)] # only channels with loc + if locs.size == 0: + raise RuntimeError("No channel locations found in the montage") + if unit == "auto": + unit = "m" if abs(locs - locs.mean(axis=0)).max() < 1 else "mm" + if coord_frame == "auto": + coord_frame = "mri_voxel" if locs.min() >= 0 else "ras" + # transform montage to head + # set to the right coordinate frame as specified by the user + for d in montage.dig: # ensure same coordinate frame + d["coord_frame"] = MNE_STR_TO_FRAME[coord_frame] + # do the transforms, first ras -> vox if needed + if montage.get_positions()["coord_frame"] == "ras": + ras_vox_trans = mne.read_trans(data_dir / f"space-{space}_ras-vox_trans.fif") + if unit == "m": # must be in mm here + for d in montage.dig: + d["r"] *= 1000 + montage.apply_trans(ras_vox_trans) + if montage.get_positions()["coord_frame"] == "mri_voxel": + vox_mri_trans = mne.read_trans(data_dir / f"space-{space}_vox-mri_trans.fif") + montage.apply_trans(vox_mri_trans) + assert montage.get_positions()["coord_frame"] == "mri" + if not (unit == "m" and coord_frame == "mri"): # if so, already in m + for d in montage.dig: + d["r"] /= 1000 # mm -> m + # now add fiducials (in mri coordinates) + fids = mne.io.read_fiducials(data_dir / f"space-{space}_fiducials.fif")[0] + montage.dig = fids + montage.dig # add fiducials + for fid in fids: # ensure also in mri + fid["coord_frame"] = MNE_STR_TO_FRAME["mri"] + info.set_montage(montage) # transform to head + # finally return montage + return info, mne.read_trans(data_dir / f"space-{space}_trans.fif") + + +@verbose +def convert_montage_to_ras(montage, subject, subjects_dir=None, verbose=None): + """Convert a montage from surface RAS (m) to scanner RAS (m). + + Parameters + ---------- + montage : mne.channels.DigMontage + The montage in the "mri" coordinate frame. Note: modified in place. + %(subject)s + %(subjects_dir)s + %(verbose)s + """ + nib = _import_nibabel("converting a montage to RAS") + + subjects_dir = get_subjects_dir(subjects_dir, raise_error=True) + T1_fname = op.join(subjects_dir, subject, "mri", "T1.mgz") + if not op.isfile(T1_fname): + raise RuntimeError( + f"Freesurfer subject ({subject}) and/or " + f"subjects_dir ({subjects_dir}, incorrectly " + "formatted, T1.mgz not found" + ) + T1 = nib.load(T1_fname) + + # transform from "mri" (Freesurfer surface RAS) to "ras" (scanner RAS) + mri_vox_t = np.linalg.inv(T1.header.get_vox2ras_tkr()) + mri_vox_t[:3, :3] *= 1000 # scale from mm to m + mri_vox_trans = mne.transforms.Transform(fro="mri", to="mri_voxel", trans=mri_vox_t) + + vox_ras_t = T1.header.get_vox2ras() + vox_ras_t[:3] /= 1000 # scale from mm to m + vox_ras_trans = mne.transforms.Transform(fro="mri_voxel", to="ras", trans=vox_ras_t) + montage.apply_trans( # mri->vox + vox->ras = mri->ras + mne.transforms.combine_transforms( + mri_vox_trans, vox_ras_trans, fro="mri", to="ras" + ) + ) + + +@verbose +def convert_montage_to_mri(montage, subject, subjects_dir=None, verbose=None): + """Convert a montage from scanner RAS (m) to surface RAS (m). + + Parameters + ---------- + montage : mne.channels.DigMontage + The montage in the "ras" coordinate frame. Note: modified in place. + %(subject)s + %(subjects_dir)s + %(verbose)s + + Returns + ------- + ras_mri_t : mne.transforms.Transform + The transformation matrix from ``'ras'`` (``scanner RAS``) to + ``'mri'`` (``surface RAS``). + """ + nib = _import_nibabel("converting a montage to MRI") + + subjects_dir = get_subjects_dir(subjects_dir, raise_error=True) + T1_fname = op.join(subjects_dir, subject, "mri", "T1.mgz") + if not op.isfile(T1_fname): + raise RuntimeError( + f"Freesurfer subject ({subject}) and/or " + f"subjects_dir ({subjects_dir}, incorrectly " + "formatted, T1.mgz not found" + ) + T1 = nib.load(T1_fname) + + # transform from "ras" (scanner RAS) to "mri" (Freesurfer surface RAS) + ras_vox_t = T1.header.get_ras2vox() + ras_vox_t[:3, :3] *= 1000 # scale from mm to m + ras_vox_trans = mne.transforms.Transform(fro="ras", to="mri_voxel", trans=ras_vox_t) + + vox_mri_t = T1.header.get_vox2ras_tkr() + vox_mri_t[:3] /= 1000 # scale from mm to m + vox_mri_trans = mne.transforms.Transform(fro="mri_voxel", to="mri", trans=vox_mri_t) + montage.apply_trans( # ras->vox + vox->mri = ras->mri + mne.transforms.combine_transforms( + ras_vox_trans, vox_mri_trans, fro="ras", to="mri" + ) + ) diff --git a/mne-bids-0.15/mne_bids/inspect.py b/mne-bids-0.15/mne_bids/inspect.py new file mode 100644 index 0000000000000000000000000000000000000000..9106cfbe28c3691f637180627de6dba09fa8548b --- /dev/null +++ b/mne-bids-0.15/mne_bids/inspect.py @@ -0,0 +1,453 @@ +"""Inspect and annotate BIDS raw data.""" +# Authors: Richard Höchenberger +# Stefan Appelhoff +# +# License: BSD-3-Clause + +from pathlib import Path + +import mne +import numpy as np +from mne.preprocessing import annotate_amplitude +from mne.utils import logger, verbose +from mne.viz import use_browser_backend + +from mne_bids import mark_channels, read_raw_bids +from mne_bids.config import ALLOWED_DATATYPE_EXTENSIONS +from mne_bids.read import _from_tsv, _read_events +from mne_bids.write import _events_tsv + + +@verbose +def inspect_dataset( + bids_path, + find_flat=True, + l_freq=None, + h_freq=None, + show_annotations=True, + verbose=None, +): + """Inspect and annotate BIDS raw data. + + This function allows you to browse MEG, EEG, and iEEG raw data stored in a + BIDS dataset. You can toggle the status of a channel (bad or good) by + clicking on the traces, and when closing the browse window, you will be + asked whether you want to save the changes to the existing BIDS dataset or + discard them. + + .. warning:: This functionality is still experimental and will be extended + in the future. Its API will likely change. Planned features + include automated bad channel detection and visualization of + MRI images. + + .. note:: Currently, only MEG, EEG, and iEEG data can be inspected. + + To add or modify annotations, press ``A`` to toggle annotation mode. + + Parameters + ---------- + bids_path : BIDSPath + A :class:`mne_bids.BIDSPath` containing at least a ``root``. All + matching files will be inspected. To select only a subset of the data, + set more :class:`mne_bids.BIDSPath` attributes. If ``datatype`` is not + set and multiple datatypes are found, they will be inspected in the + following order: MEG, EEG, iEEG. + To read a specific file, set all the :class:`mne_bids.BIDSPath` + attributes required to uniquely identify the file: If this ``BIDSPath`` + is accepted by :func:`mne_bids.read_raw_bids`, it will work here. + find_flat : bool + Whether to auto-detect channels producing "flat" signals, i.e., with + unusually low variability. Flat **segments** will be added to + ``*_events.tsv``, while channels with more than 5 percent of flat data + will be marked as ``bad`` in ``*_channels.tsv``. + + .. note:: + This function calls ``mne.preprocessing.annotate_amplitude`` + (MNE-Python 1.0 or newer) or ``mne.preprocessing.annotate_flat`` + (older versions of MNE-Python) + and will only consider segments of at least **50 ms consecutive + flatness** as "flat" (deviating from MNE-Python's default of 5 ms). + If more than 5 percent of a channel's data has been marked as flat, + the entire channel will be added to the list of bad channels. Only + flat time segments applying to channels **not** marked as bad will + be added to ``*_events.tsv``. + + l_freq : float | None + The high-pass filter cutoff frequency to apply when displaying the + data. This can be useful when inspecting data with slow drifts. If + ``None``, no high-pass filter will be applied. + h_freq : float | None + The low-pass filter cutoff frequency to apply when displaying the + data. This can be useful when inspecting data with high-frequency + artifacts. If ``None``, no low-pass filter will be applied. + show_annotations : bool + Whether to show annotations (events, bad segments, …) or not. If + ``False``, toggling annotations mode by pressing ``A`` will be disabled + as well. + %(verbose)s + + Examples + -------- + Disable flat channel & segment detection, and apply a filter with a + passband of 1–30 Hz. + + >>> from mne_bids import BIDSPath + >>> root = Path('./mne_bids/tests/data/tiny_bids').absolute() + >>> bids_path = BIDSPath(subject='01', task='rest', session='eeg', + ... suffix='eeg', extension='.vhdr', root=root) + >>> inspect_dataset(bids_path=bids_path, find_flat=False, # doctest: +SKIP + ... l_freq=1, h_freq=30) + """ + allowed_extensions = set( + ALLOWED_DATATYPE_EXTENSIONS["meg"] + + ALLOWED_DATATYPE_EXTENSIONS["eeg"] + + ALLOWED_DATATYPE_EXTENSIONS["ieeg"] + ) + + bids_paths = [ + p + for p in bids_path.match(check=True) + if (p.extension is None or p.extension in allowed_extensions) + and p.acquisition != "crosstalk" + ] + + for bids_path_ in bids_paths: + _inspect_raw( + bids_path=bids_path_, + l_freq=l_freq, + h_freq=h_freq, + find_flat=find_flat, + show_annotations=show_annotations, + ) + + +# XXX This this should probably be refactored into a class attribute someday. +_global_vars = dict(raw_fig=None, dialog_fig=None, mne_close_key=None) + + +def _inspect_raw(*, bids_path, l_freq, h_freq, find_flat, show_annotations): + """Raw data inspection.""" + # Delay the import + import matplotlib + import matplotlib.pyplot as plt + + extra_params = dict() + if bids_path.extension == ".fif": + extra_params["allow_maxshield"] = "yes" + raw = read_raw_bids(bids_path, extra_params=extra_params, verbose="error") + old_bads = raw.info["bads"].copy() + old_annotations = raw.annotations.copy() + + if find_flat: + raw.load_data() # Speeds up processing dramatically + flat_annot, flat_chans = annotate_amplitude( + raw=raw, flat=0, min_duration=0.05, bad_percent=5 + ) + new_annot = raw.annotations + flat_annot + raw.set_annotations(new_annot) + raw.info["bads"] = list(set(raw.info["bads"] + flat_chans)) + del new_annot, flat_annot + else: + flat_chans = [] + + show_options = bids_path.datatype == "meg" + + with use_browser_backend("matplotlib"): + fig = raw.plot( + title=f"{bids_path.root.name}: {bids_path.basename}", + highpass=l_freq, + lowpass=h_freq, + show_options=show_options, + block=False, + show=False, + verbose="warning", + ) + + # Add our own event handlers so that when the MNE Raw Browser is being + # closed, our dialog box will pop up, asking whether to save changes. + def _handle_close(event): + mne_raw_fig = event.canvas.figure + # Bads alterations are only transferred to `inst` once the figure is + # closed; Annotation changes are immediately reflected in `inst` + new_bads = mne_raw_fig.mne.info["bads"].copy() + new_annotations = mne_raw_fig.mne.inst.annotations.copy() + + if not new_annotations: + # Ensure it's not an empty list, but an empty set of Annotations. + new_annotations = mne.Annotations( + onset=[], + duration=[], + description=[], + orig_time=mne_raw_fig.mne.info["meas_date"], + ) + _save_raw_if_changed( + old_bads=old_bads, + new_bads=new_bads, + flat_chans=flat_chans, + old_annotations=old_annotations, + new_annotations=new_annotations, + bids_path=bids_path, + ) + _global_vars["raw_fig"] = None + + def _keypress_callback(event): + if event.key == _global_vars["mne_close_key"]: + _handle_close(event) + + fig.canvas.mpl_connect("close_event", _handle_close) + fig.canvas.mpl_connect("key_press_event", _keypress_callback) + + if not show_annotations: + # Remove annotations and kill `_toggle_annotation_fig` method, since + # we cannot directly and easily remove the associated `a` keyboard + # event callback. + fig._clear_annotations() + fig._toggle_annotation_fig = lambda: None + # Ensure it's not an empty list, but an empty set of Annotations. + old_annotations = mne.Annotations( + onset=[], duration=[], description=[], orig_time=raw.info["meas_date"] + ) + + if matplotlib.get_backend().lower() != "agg": + plt.show(block=True) + + _global_vars["raw_fig"] = fig + _global_vars["mne_close_key"] = fig.mne.close_key + + +def _annotations_almost_equal(old_annotations, new_annotations): + """Allow for a tiny bit of floating point precision loss.""" + if ( + np.array_equal(old_annotations.description, new_annotations.description) + and np.array_equal(old_annotations.orig_time, new_annotations.orig_time) + and np.allclose(old_annotations.onset, new_annotations.onset) + and np.allclose(old_annotations.duration, new_annotations.duration) + ): + return True + else: + return False + + +def _save_annotations(*, annotations, bids_path): + # Attach the new Annotations to our raw data so we can easily convert them + # to events, which will be stored in the *_events.tsv sidecar. + extra_params = dict() + if bids_path.extension == ".fif": + extra_params["allow_maxshield"] = "yes" + + raw = read_raw_bids( + bids_path=bids_path, extra_params=extra_params, verbose="warning" + ) + raw.set_annotations(annotations) + events, durs, descrs = _read_events( + events=None, event_id=None, bids_path=bids_path, raw=raw + ) + + # Write sidecar – or remove it if no events are left. + events_tsv_fname = bids_path.copy().update(suffix="events", extension=".tsv").fpath + + if len(events) > 0: + _events_tsv( + events=events, + durations=durs, + raw=raw, + fname=events_tsv_fname, + trial_type=descrs, + overwrite=True, + ) + elif events_tsv_fname.exists(): + logger.info( + f"No events remaining after interactive inspection, " + f"removing {events_tsv_fname.name}" + ) + events_tsv_fname.unlink() + + +def _save_raw_if_changed( + *, old_bads, new_bads, flat_chans, old_annotations, new_annotations, bids_path +): + """Save bad channel selection if it has been changed. + + Parameters + ---------- + old_bads : list + The original bad channels. + new_bads : list + The updated set of bad channels (i.e. **all** of them, not only the + changed ones). + flat_chans : list + The auto-detected flat channels. This is either an empty list or a + subset of ``new_bads``. + old_annotations : mne.Annotations + The original Annotations. + new_annotations : mne.Annotations + The new Annotations. + """ + assert set(flat_chans).issubset(set(new_bads)) + + if set(old_bads) == set(new_bads): + bads = None + bad_descriptions = [] + else: + bads = new_bads + bad_descriptions = [] + + # Generate entries for the `status_description` column. + channels_tsv_fname = ( + bids_path.copy().update(suffix="channels", extension=".tsv").fpath + ) + channels_tsv_data = _from_tsv(channels_tsv_fname) + + for ch_name in bads: + idx = channels_tsv_data["name"].index(ch_name) + if channels_tsv_data["status"][idx] == "bad": + # Channel was already marked as bad in the data, so retain + # existing description. + description = channels_tsv_data["status_description"][idx] + elif ch_name in flat_chans: + description = "Flat channel, auto-detected via MNE-BIDS" + else: + # Channel has been manually marked as bad during inspection + description = "Interactive inspection via MNE-BIDS" + + bad_descriptions.append(description) + del ch_name, description + + del ( + channels_tsv_data, + channels_tsv_fname, + ) + + if _annotations_almost_equal(old_annotations, new_annotations): + annotations = None + else: + annotations = new_annotations + + if bads is None and annotations is None: + # Nothing has changed, so we can just exit. + return None + + return _save_raw_dialog_box( + bads=bads, + bad_descriptions=bad_descriptions, + annotations=annotations, + bids_path=bids_path, + ) + + +def _save_raw_dialog_box(*, bads, bad_descriptions, annotations, bids_path): + """Display a dialog box asking whether to save the changes.""" + # Delay the imports + import matplotlib + import matplotlib.pyplot as plt + from matplotlib.widgets import Button + from mne.viz.utils import figure_nobar + + title = "Save changes?" + message = "You have modified " + if bads is not None and annotations is None: + message += "the bad channel selection " + figsize = (7.5, 2.5) + elif bads is None and annotations is not None: + message += "the bad segments selection " + figsize = (7.5, 2.5) + else: + message += "the bad channel and\nannotations selection " + figsize = (8.5, 3) + + message += ( + f"of\n" + f"{bids_path.basename}.\n\n" + f"Would you like to save these changes to the\n" + f"BIDS dataset?" + ) + icon_fname = str(Path(__file__).parent / "assets" / "help-128px.png") + icon = plt.imread(icon_fname) + + fig = figure_nobar(figsize=figsize) + fig.canvas.manager.set_window_title("MNE-BIDS Inspector") + fig.suptitle(title, y=0.95, fontsize="xx-large", fontweight="bold") + + gs = fig.add_gridspec(1, 2, width_ratios=(1.5, 5)) + + # The dialog box tet. + ax_msg = fig.add_subplot(gs[0, 1]) + ax_msg.text( + x=0, + y=0.8, + s=message, + fontsize="large", + verticalalignment="top", + horizontalalignment="left", + multialignment="left", + ) + ax_msg.axis("off") + + # The help icon. + ax_icon = fig.add_subplot(gs[0, 0]) + ax_icon.imshow(icon) + ax_icon.axis("off") + + # Buttons. + ax_save = fig.add_axes([0.6, 0.05, 0.3, 0.1]) + ax_dont_save = fig.add_axes([0.1, 0.05, 0.3, 0.1]) + + save_button = Button(ax=ax_save, label="Save") + save_button.label.set_fontsize("medium") + save_button.label.set_fontweight("bold") + + dont_save_button = Button(ax=ax_dont_save, label="Don't save") + dont_save_button.label.set_fontsize("medium") + dont_save_button.label.set_fontweight("bold") + + # Store references to keep buttons alive. + fig.save_button = save_button + fig.dont_save_button = dont_save_button + + # Define callback functions. + def _save_callback(event): + plt.close(event.canvas.figure) # Close dialog + _global_vars["dialog_fig"] = None + + if bads is not None: + _save_bads(bads=bads, descriptions=bad_descriptions, bids_path=bids_path) + if annotations is not None: + _save_annotations(annotations=annotations, bids_path=bids_path) + + def _dont_save_callback(event): + plt.close(event.canvas.figure) # Close dialog + _global_vars["dialog_fig"] = None + + def _keypress_callback(event): + if event.key in ["enter", "return"]: + _save_callback(event) + elif event.key == _global_vars["mne_close_key"]: + _dont_save_callback(event) + + # Connect events to callback functions. + save_button.on_clicked(_save_callback) + dont_save_button.on_clicked(_dont_save_callback) + fig.canvas.mpl_connect("close_event", _dont_save_callback) + fig.canvas.mpl_connect("key_press_event", _keypress_callback) + + if matplotlib.get_backend().lower() != "agg": + fig.show() + + _global_vars["dialog_fig"] = fig + + +def _save_bads(*, bads, descriptions, bids_path): + """Update the set of channels marked as bad. + + Parameters + ---------- + bads : list + The complete list of bad channels. + descriptions : list + The values to be written to the `status_description` column. + """ + # We first make all channels not passed as bad here to be marked as good. + mark_channels(bids_path=bids_path, ch_names=[], status="good") + mark_channels( + bids_path=bids_path, ch_names=bads, status="bad", descriptions=descriptions + ) diff --git a/mne-bids-0.15/mne_bids/path.py b/mne-bids-0.15/mne_bids/path.py new file mode 100644 index 0000000000000000000000000000000000000000..101d2715cef65fe01b8714e39d8f9ad30caf517c --- /dev/null +++ b/mne-bids-0.15/mne_bids/path.py @@ -0,0 +1,2490 @@ +"""BIDS compatible path functionality.""" + +# Authors: Adam Li +# Stefan Appelhoff +# +# License: BSD-3-Clause +import glob +import json +import os +import re +import shutil as sh +from copy import deepcopy +from datetime import datetime +from io import StringIO +from os import path as op +from pathlib import Path +from textwrap import indent +from typing import Optional + +import numpy as np +from mne.utils import _check_fname, _validate_type, logger, verbose + +from mne_bids.config import ( + ALLOWED_DATATYPE_EXTENSIONS, + ALLOWED_DATATYPES, + ALLOWED_FILENAME_EXTENSIONS, + ALLOWED_FILENAME_SUFFIX, + ALLOWED_PATH_ENTITIES, + ALLOWED_PATH_ENTITIES_SHORT, + ALLOWED_SPACES, + ENTITY_VALUE_TYPE, + reader, +) +from mne_bids.tsv_handler import _drop, _from_tsv, _to_tsv +from mne_bids.utils import ( + _check_empty_room_basename, + _check_key_val, + _ensure_tuple, + param_regex, + warn, +) + + +def _find_empty_room_candidates(bids_path): + """Get matching empty-room file for an MEG recording.""" + # Check whether we have a BIDS root. + bids_root = bids_path.root + if bids_root is None: + raise ValueError( + 'The root of the "bids_path" must be set. ' + 'Please use `bids_path.update(root="")` ' + "to set the root of the BIDS folder to read." + ) + + bids_path = bids_path.copy() + + datatype = "meg" # We're only concerned about MEG data here + bids_fname = bids_path.update(suffix=datatype).fpath + _, ext = _parse_ext(bids_fname) + emptyroom_dir = BIDSPath(root=bids_root, subject="emptyroom").directory + + if not emptyroom_dir.exists(): + return list() + + # Find the empty-room recording sessions. + emptyroom_session_dirs = [ + x + for x in emptyroom_dir.iterdir() + if x.is_dir() and str(x.name).startswith("ses-") + ] + if not emptyroom_session_dirs: # No session sub-directories found + emptyroom_session_dirs = [emptyroom_dir] + + # Now try to discover all recordings inside the session directories. + + allowed_extensions = list(reader.keys()) + # `.pdf` is just a "virtual" extension for BTi data (which is stored inside + # a dedicated directory that doesn't have an extension) + del allowed_extensions[allowed_extensions.index(".pdf")] + + candidate_er_fnames = [] + for session_dir in emptyroom_session_dirs: + dir_contents = glob.glob( + op.join(session_dir, datatype, f"sub-emptyroom_*_{datatype}*") + ) + for item in dir_contents: + item = Path(item) + if (item.suffix in allowed_extensions) or ( + not item.suffix and item.is_dir() + ): # Hopefully BTi? + candidate_er_fnames.append(item.name) + + candidates = list() + for er_fname in candidate_er_fnames: + # get entities from filenamme + er_bids_path = get_bids_path_from_fname(er_fname, check=False) + er_bids_path.subject = "emptyroom" # er subject entity is different + er_bids_path.root = bids_root + er_bids_path.datatype = "meg" + candidates.append(er_bids_path) + + return candidates + + +def _find_matched_empty_room(bids_path): + from mne_bids import read_raw_bids # avoid circular import. + + candidates = _find_empty_room_candidates(bids_path) + + # Walk through recordings, trying to extract the recording date: + # First, from the filename; and if that fails, from `info['meas_date']`. + best_er_bids_path = None + min_delta_t = np.inf + date_tie = False + failed_to_get_er_date_count = 0 + bids_path = bids_path.copy().update(datatype="meg") + raw = read_raw_bids(bids_path=bids_path) + if raw.info["meas_date"] is None: + raise ValueError( + "The provided recording does not have a measurement " + "date set. Cannot get matching empty-room file." + ) + ref_date = raw.info["meas_date"] + del bids_path, raw + for er_bids_path in candidates: + # get entities from filenamme + er_meas_date = None + + # Try to extract date from filename. + if er_bids_path.session is not None: + try: + er_meas_date = datetime.strptime(er_bids_path.session, "%Y%m%d") + except (ValueError, TypeError): + # There is a session in the filename, but it doesn't encode a + # valid date. + pass + + if er_meas_date is None: # No luck so far! Check info['meas_date'] + _, ext = _parse_ext(er_bids_path.fpath) + extra_params = None + if ext == ".fif": + extra_params = dict(allow_maxshield="yes") + + er_raw = read_raw_bids(bids_path=er_bids_path, extra_params=extra_params) + + er_meas_date = er_raw.info["meas_date"] + if er_meas_date is None: # There's nothing we can do. + failed_to_get_er_date_count += 1 + continue + + er_meas_date = er_meas_date.replace(tzinfo=ref_date.tzinfo) + delta_t = er_meas_date - ref_date + + if abs(delta_t.total_seconds()) == min_delta_t: + date_tie = True + elif abs(delta_t.total_seconds()) < min_delta_t: + min_delta_t = abs(delta_t.total_seconds()) + best_er_bids_path = er_bids_path + date_tie = False + + if failed_to_get_er_date_count > 0: + msg = ( + f"Could not retrieve the empty-room measurement date from " + f"a total of {failed_to_get_er_date_count} recording(s)." + ) + warn(msg) + + if date_tie: + msg = ( + "Found more than one matching empty-room measurement with the " + "same recording date. Selecting the first match." + ) + warn(msg) + + return best_er_bids_path + + +class BIDSPath: + """A BIDS path object. + + BIDS filename prefixes have one or more pieces of metadata in them. They + must follow a particular order, which is followed by this function. This + will generate the *prefix* for a BIDS filename that can be used with many + subsequent files, or you may also give a suffix that will then complete + the file name. + + BIDSPath allows dynamic updating of its entities in place, and operates + similar to `pathlib.Path`. In addition, it can query multiple paths + with matching BIDS entities via the ``match`` method. + + Note that not all parameters are applicable to each suffix of data. For + example, electrode location TSV files do not need a "task" field. + + Parameters + ---------- + subject : str | None + The subject ID. Corresponds to "sub". + session : str | None + The acquisition session. Corresponds to "ses". + task : str | None + The experimental task. Corresponds to "task". + acquisition: str | None + The acquisition parameters. Corresponds to "acq". + run : int | None + The run number. Corresponds to "run". + processing : str | None + The processing label. Corresponds to "proc". + recording : str | None + The recording name. Corresponds to "rec". + space : str | None + The coordinate space for anatomical and sensor location + files (e.g., ``*_electrodes.tsv``, ``*_markers.mrk``). + Corresponds to "space". + Note that valid values for ``space`` must come from a list + of BIDS keywords as described in the BIDS specification. + split : int | None + The split of the continuous recording file for ``.fif`` data. + Corresponds to "split". + description : str | None + This corresponds to the BIDS entity ``desc``. It is used to provide + additional information for derivative data, e.g., preprocessed data + may be assigned ``description='cleaned'``. + + .. versionadded:: 0.11 + suffix : str | None + The filename suffix. This is the entity after the + last ``_`` before the extension. E.g., ``'channels'``. + The following filename suffix's are accepted: + 'meg', 'markers', 'eeg', 'ieeg', 'T1w', + 'participants', 'scans', 'electrodes', 'coordsystem', + 'channels', 'events', 'headshape', 'digitizer', + 'beh', 'physio', 'stim' + extension : str | None + The extension of the filename. E.g., ``'.json'``. + datatype : str + The BIDS data type, e.g., ``'anat'``, ``'func'``, ``'eeg'``, ``'meg'``, + ``'ieeg'``. + root : path-like | None + The root directory of the BIDS dataset. + check : bool + If ``True``, enforces BIDS conformity. Defaults to ``True``. + + Attributes + ---------- + entities : dict + A dictionary of the BIDS entities and their values: + ``subject``, ``session``, ``task``, ``acquisition``, + ``run``, ``processing``, ``space``, ``recording``, + ``split``, ``description``, ``suffix``, and ``extension``. + datatype : str | None + The data type, i.e., one of ``'meg'``, ``'eeg'``, ``'ieeg'``, + ``'anat'``. + basename : str + The basename of the file path. Similar to `os.path.basename(fpath)`. + root : pathlib.Path + The root of the BIDS path. + directory : pathlib.Path + The directory path. + fpath : pathlib.Path + The full file path. + check : bool + Whether to enforce BIDS conformity. + + Examples + -------- + Generate a BIDSPath object and inspect it + + >>> bids_path = BIDSPath(subject='test', session='two', task='mytask', + ... suffix='ieeg', extension='.edf', datatype='ieeg') + >>> print(bids_path.basename) + sub-test_ses-two_task-mytask_ieeg.edf + >>> bids_path + BIDSPath( + root: None + datatype: ieeg + basename: sub-test_ses-two_task-mytask_ieeg.edf) + + Copy and update multiple entities at once + + >>> new_bids_path = bids_path.copy().update(subject='test2', + ... session='one') + >>> print(new_bids_path.basename) + sub-test2_ses-one_task-mytask_ieeg.edf + + Printing a BIDSPath will show a relative path when `root` is not set + + >>> print(new_bids_path) + sub-test2/ses-one/ieeg/sub-test2_ses-one_task-mytask_ieeg.edf + + Setting `suffix` without an identifiable datatype will make + BIDSPath try to guess the datatype + + >>> new_bids_path = new_bids_path.update(suffix='channels', + ... extension='.tsv') + >>> print(new_bids_path) + sub-test2/ses-one/ieeg/sub-test2_ses-one_task-mytask_channels.tsv + + You can set a new root for the BIDS dataset. Let's see what the + different properties look like for our object: + + >>> new_bids_path = new_bids_path.update(root='/bids_dataset') + >>> print(new_bids_path.root.as_posix()) + /bids_dataset + >>> print(new_bids_path.basename) + sub-test2_ses-one_task-mytask_channels.tsv + >>> print(new_bids_path) + /bids_dataset/sub-test2/ses-one/ieeg/sub-test2_ses-one_task-mytask_channels.tsv + >>> print(new_bids_path.directory.as_posix()) + /bids_dataset/sub-test2/ses-one/ieeg + + Notes + ----- + BIDS entities are generally separated with a ``"_"`` character, while + entity key/value pairs are separated with a ``"-"`` character. + There are checks performed to make sure that there are no ``'-'``, ``'_'``, + or ``'/'`` characters contained in any entity keys or values. + + To represent a filename such as ``dataset_description.json``, + one can set ``check=False``, and pass ``suffix='dataset_description'`` + and ``extension='.json'``. + + ``BIDSPath`` can also be used to represent file and folder names of data + types that are not yet supported through MNE-BIDS, but are recognized by + BIDS. For example, one can set ``datatype`` to ``dwi`` or ``func`` and + pass ``check=False`` to represent diffusion-weighted imaging and + functional MRI paths. + """ + + def __init__( + self, + subject=None, + session=None, + task=None, + acquisition=None, + run=None, + processing=None, + recording=None, + space=None, + split=None, + description=None, + root=None, + suffix=None, + extension=None, + datatype=None, + check=True, + ): + if all( + ii is None + for ii in [ + subject, + session, + task, + acquisition, + run, + processing, + recording, + space, + description, + root, + suffix, + extension, + ] + ): + raise ValueError("At least one parameter must be given.") + + self.check = check + + self.update( + subject=subject, + session=session, + task=task, + acquisition=acquisition, + run=run, + processing=processing, + recording=recording, + space=space, + split=split, + description=description, + root=root, + datatype=datatype, + suffix=suffix, + extension=extension, + ) + + @property + def entities(self): + """Return dictionary of the BIDS entities.""" + return { + "subject": self.subject, + "session": self.session, + "task": self.task, + "acquisition": self.acquisition, + "run": self.run, + "processing": self.processing, + "space": self.space, + "recording": self.recording, + "split": self.split, + "description": self.description, + } + + @property + def basename(self): + """Path basename.""" + basename = [] + for key, val in self.entities.items(): + if val is not None and key != "datatype": + # convert certain keys to shorthand + long_to_short_entity = { + val: key for key, val in ALLOWED_PATH_ENTITIES_SHORT.items() + } + key = long_to_short_entity[key] + basename.append(f"{key}-{val}") + + if self.suffix is not None: + if self.extension is not None: + basename.append(f"{self.suffix}{self.extension}") + else: + basename.append(self.suffix) + + basename = "_".join(basename) + return basename + + @property + def directory(self): + """Get the BIDS parent directory. + + If ``subject``, ``session`` and ``datatype`` are set, then they will be + used to construct the directory location. For example, if + ``subject='01'``, ``session='02'`` and ``datatype='ieeg'``, then the + directory would be:: + + /sub-01/ses-02/ieeg + + Returns + ------- + data_path : pathlib.Path + The path of the BIDS directory. + """ + # Create the data path based on the available entities: + # root, subject, session, and datatype + data_path = "" if self.root is None else self.root + if self.subject is not None: + data_path = op.join(data_path, f"sub-{self.subject}") + if self.session is not None: + data_path = op.join(data_path, f"ses-{self.session}") + # datatype will allow 'meg', 'eeg', 'ieeg', 'anat' + if self.datatype is not None: + data_path = op.join(data_path, self.datatype) + return Path(data_path) + + @property + def subject(self) -> Optional[str]: + """The subject ID.""" + return self._subject + + @subject.setter + def subject(self, value): + self.update(subject=value) + + @property + def session(self) -> Optional[str]: + """The acquisition session.""" + return self._session + + @session.setter + def session(self, value): + self.update(session=value) + + @property + def task(self) -> Optional[str]: + """The experimental task.""" + return self._task + + @task.setter + def task(self, value): + self.update(task=value) + + @property + def run(self) -> Optional[str]: + """The run number.""" + return self._run + + @run.setter + def run(self, value): + self.update(run=value) + + @property + def acquisition(self) -> Optional[str]: + """The acquisition parameters.""" + return self._acquisition + + @acquisition.setter + def acquisition(self, value): + self.update(acquisition=value) + + @property + def processing(self) -> Optional[str]: + """The processing label.""" + return self._processing + + @processing.setter + def processing(self, value): + self.update(processing=value) + + @property + def recording(self) -> Optional[str]: + """The recording name.""" + return self._recording + + @recording.setter + def recording(self, value): + self.update(recording=value) + + @property + def space(self) -> Optional[str]: + """The coordinate space for an anatomical or sensor position file.""" + return self._space + + @space.setter + def space(self, value): + self.update(space=value) + + @property + def description(self) -> Optional[str]: + """The description entity.""" + return self._description + + @description.setter + def description(self, value): + self.update(description=value) + + @property + def suffix(self) -> Optional[str]: + """The filename suffix.""" + return self._suffix + + @suffix.setter + def suffix(self, value): + self.update(suffix=value) + + @property + def root(self) -> Optional[Path]: + """The root directory of the BIDS dataset.""" + return self._root + + @root.setter + def root(self, value): + self.update(root=value) + + @property + def datatype(self) -> Optional[str]: + """The BIDS data type, e.g. ``'anat'``, ``'meg'``, ``'eeg'``.""" + return self._datatype + + @datatype.setter + def datatype(self, value): + self.update(datatype=value) + + @property + def split(self) -> Optional[str]: + """The split of the continuous recording file for ``.fif`` data.""" + return self._split + + @split.setter + def split(self, value): + self.update(split=value) + + @property + def extension(self) -> Optional[str]: + """The extension of the filename, including a leading period.""" + return self._extension + + @extension.setter + def extension(self, value): + self.update(extension=value) + + def __str__(self): + """Return the string representation of the path.""" + return str(self.fpath.as_posix()) + + def __repr__(self): + """Representation in the style of `pathlib.Path`.""" + root = self.root.as_posix() if self.root is not None else None + + return ( + f"{self.__class__.__name__}(\n" + f"root: {root}\n" + f"datatype: {self.datatype}\n" + f"basename: {self.basename})" + ) + + def __fspath__(self): + """Return the string representation for any fs functions.""" + return str(self.fpath) + + def __eq__(self, other): + """Compare str representations.""" + return str(self) == str(other) + + def __ne__(self, other): + """Compare str representations.""" + return str(self) != str(other) + + def copy(self): + """Copy the instance. + + Returns + ------- + bidspath : BIDSPath + The copied bidspath. + """ + return deepcopy(self) + + def mkdir(self, exist_ok=True): + """Create the directory structure of the BIDS path. + + Parameters + ---------- + exist_ok : bool + If ``False``, raise an exception if the directory already exists. + Otherwise, do nothing (default). + + Returns + ------- + self : BIDSPath + The BIDSPath object. + """ + self.directory.mkdir(parents=True, exist_ok=exist_ok) + return self + + @verbose + def rm(self, *, safe_remove=True, verbose=None): + """Safely delete a set of files from a BIDS dataset. + + Deleting a scan that conforms to the bids-validator will + remove the respective row in ``*_scans.tsv``, the + corresponding sidecar files, and the data file itself. + + Deleting all files of a subject will update the + ``*_participants.tsv`` file. + + + Parameters + ---------- + safe_remove : bool + If ``False``, directly delete and update the files. + Otherwise, displays the list of operations planned + and asks for user confirmation before + executing them (default). + %(verbose)s + + Returns + ------- + self : BIDSPath + The BIDSPath object. + + Examples + -------- + Remove one specific run: + + >>> bids_path = BIDSPath(subject='01', session='01', run="01", # doctest: +SKIP + ... root='/bids_dataset').rm() # doctest: +SKIP + Please, confirm you want to execute the following operations: + Delete: + /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-01_channels.tsv + /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-01_events.json + /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-01_events.tsv + /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-01_meg.fif + /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-01_meg.json + Update: + /bids_dataset/sub-01/ses-01/sub-01_ses-01_scans.tsv + I confirm [y/N]>? y + + Remove all the files of a specific subject: + + >>> bids_path = BIDSPath(subject='01', root='/bids_dataset', # doctest: +SKIP + ... check=False).rm() # doctest: +SKIP + Please, confirm you want to execute the following operations: + Delete: + /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_acq-calibration_meg.dat + /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_acq-crosstalk_meg.fif + /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_coordsystem.json + /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-02_channels.tsv + /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-02_events.json + /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-02_events.tsv + /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-02_meg.fif + /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-02_meg.json + /bids_dataset/sub-01/ses-01/sub-01_ses-01_scans.tsv + /bids_dataset/sub-01 + Update: + /bids_dataset/participants.tsv + I confirm [y/N]>? y + """ + # only proceed if root is defined + if self.root is None: + raise RuntimeError("The root must not be None to remove files.") + + # Planning: + paths_matched = self.match(ignore_json=False, check=self.check) + subjects = set() + paths_to_delete = list() + paths_to_update = {} + subjects_paths_to_delete = [] + participants_tsv_fpath = None + for bids_path in paths_matched: + paths_to_delete.append(bids_path) + # if a datatype is present, then check + # if a scan is deleted or not + if bids_path.datatype is not None: + # read in the corresponding scans file + scans_fpath = ( + bids_path.copy() + .update(datatype=None) + .find_matching_sidecar( + suffix="scans", + extension=".tsv", + on_error="raise", + ) + ) + paths_to_update.setdefault(scans_fpath, []).append(bids_path) + subjects.add(bids_path.subject) + + files_to_delete = set(p.fpath for p in paths_to_delete) + for subject in subjects: + # check existence of files in the subject dir + subj_path = BIDSPath(root=self.root, subject=subject) + subj_files = [ + fpath for fpath in subj_path.directory.rglob("*") if fpath.is_file() + ] + if set(subj_files) <= files_to_delete: + subjects_paths_to_delete.append(subj_path) + participants_tsv_fpath = self.root / "participants.tsv" + + # Informing: + pretty_delete_paths = "\n".join( + [ + str(p) + for p in paths_to_delete + + [p.directory for p in subjects_paths_to_delete] + ] + ) + pretty_update_paths = "\n".join( + [ + str(p) + for p in list(paths_to_update.keys()) + + ( + [participants_tsv_fpath] + if participants_tsv_fpath is not None + else [] + ) + ] + ) + summary = "" + if pretty_delete_paths: + summary += f"Delete:\n{pretty_delete_paths}\n" + if pretty_update_paths: + summary += f"Update:\n{pretty_update_paths}\n" + + if safe_remove: + choice = input( + "Please, confirm you want to execute the following operations:\n" + f"{summary}\nI confirm [y/N]" + ) + if choice.lower() != "y": + return + else: + logger.info(f"Executing the following operations:\n{summary}") + + # Execution: + for bids_path in paths_to_delete: + bids_path.fpath.unlink() + + for scans_fpath, bids_paths in paths_to_update.items(): + if not scans_fpath.exists(): + continue + # get the relative datatype of these bids files + bids_fnames = [op.join(p.datatype, p.fpath.name) for p in bids_paths] + + scans_tsv = _from_tsv(scans_fpath) + scans_tsv = _drop(scans_tsv, bids_fnames, "filename") + _to_tsv(scans_tsv, scans_fpath) + + subjects_to_delete = [] + for subj_path in subjects_paths_to_delete: + if subj_path.directory.exists(): + sh.rmtree(subj_path.directory) + subjects_to_delete.append(subj_path.subject) + if subjects_to_delete and participants_tsv_fpath.exists(): + participants_tsv = _from_tsv(participants_tsv_fpath) + participants_tsv = _drop( + participants_tsv, subjects_to_delete, "participant_id" + ) + _to_tsv(participants_tsv, participants_tsv_fpath) + + return self + + @property + def fpath(self): + """Full filepath for this BIDS file. + + Getting the file path consists of the entities passed in + and will get the relative (or full if ``root`` is passed) + path. + + Returns + ------- + bids_fpath : pathlib.Path + Either the relative, or full path to the dataset. + """ + # get the inner-most BIDS directory for this file path + data_path = self.directory + + # account for MEG data that are directory-based + # else, all other file paths attempt to match + if self.suffix == "meg" and self.extension == ".ds": + bids_fpath = op.join(data_path, self.basename) + elif self.suffix == "meg" and self.extension == ".pdf": + bids_fpath = op.join(data_path, op.splitext(self.basename)[0]) + else: + # if suffix and/or extension is missing, and root is + # not None, then BIDSPath will infer the dataset + # else, return the relative path with the basename + if ( + self.suffix is None or self.extension is None + ) and self.root is not None: + # get matching BIDSPaths inside the bids root + matching_paths = _get_matching_bidspaths_from_filesystem(self) + + # FIXME This will break + # FIXME e.g. with FIFF data split across multiple files. + # if extension is not specified and no unique file path + # return filepath of the actual dataset for MEG/EEG/iEEG data + if self.suffix is None or self.suffix in ALLOWED_DATATYPES: + # now only use valid datatype extension + if self.extension is None: + valid_exts = sum(ALLOWED_DATATYPE_EXTENSIONS.values(), []) + else: + valid_exts = [self.extension] + matching_paths = [ + p for p in matching_paths if _parse_ext(p)[1] in valid_exts + ] + + if self.split is None and ( + not matching_paths or "_split-" in matching_paths[0] + ): + # try finding FIF split files (only first one) + this_self = self.copy().update(split="01") + matching_paths = _get_matching_bidspaths_from_filesystem(this_self) + + # found no matching paths + if not matching_paths: + bids_fpath = op.join(data_path, self.basename) + # if paths still cannot be resolved, then there is an error + elif len(matching_paths) > 1: + matching_paths_str = "\n".join(sorted(matching_paths)) + msg = ( + "Found more than one matching data file for the " + "requested recording. While searching:\n" + f'{indent(repr(self), " ")}\n' + f"Found {len(matching_paths)} paths:\n" + f'{indent(matching_paths_str, " ")}\n' + "Cannot proceed due to the " + "ambiguity. This is likely a problem with your " + "BIDS dataset. Please run the BIDS validator on " + "your data." + ) + raise RuntimeError(msg) + else: + bids_fpath = matching_paths[0] + + else: + bids_fpath = op.join(data_path, self.basename) + + bids_fpath = Path(bids_fpath) + return bids_fpath + + def update(self, *, check=None, **kwargs): + """Update inplace BIDS entity key/value pairs in object. + + ``run`` and ``split`` are auto-converted to have two + digits. For example, if ``run=1``, then it will nbecome ``run='01'``. + + Also performs error checks on various entities to + adhere to the BIDS specification. Specifically: + - ``datatype`` should be one of: ``anat``, ``eeg``, ``ieeg``, ``meg`` + - ``extension`` should be one of the accepted file + extensions in the file path: ``.con``, ``.sqd``, ``.fif``, + ``.pdf``, ``.ds``, ``.vhdr``, ``.edf``, ``.bdf``, ``.set``, + ``.edf``, ``.set``, ``.mef``, ``.nwb`` + - ``suffix`` should be one of the acceptable file suffixes in: ``meg``, + ``markers``, ``eeg``, ``ieeg``, ``T1w``, + ``participants``, ``scans``, ``electrodes``, ``channels``, + ``coordsystem``, ``events``, ``headshape``, ``digitizer``, + ``beh``, ``physio``, ``stim`` + - Depending on the modality of the data (EEG, MEG, iEEG), + ``space`` should be a valid string according to Appendix VIII + in the BIDS specification. + + Parameters + ---------- + check : None | bool + If a boolean, controls whether to enforce BIDS conformity. This + will set the ``.check`` attribute accordingly. If ``None``, rely on + the existing ``.check`` attribute instead, which is set upon + `mne_bids.BIDSPath` instantiation. Defaults to ``None``. + **kwargs : dict + It can contain updates for valid BIDSPath entities: + 'subject', 'session', 'task', 'acquisition', 'processing', 'run', + 'recording', 'space', 'suffix', 'split', 'extension', + or updates for 'root' or 'datatype'. + + Returns + ------- + bidspath : BIDSPath + The updated instance of BIDSPath. + + Examples + -------- + If one creates a bids basename using + :func:`mne_bids.BIDSPath`: + + >>> bids_path = BIDSPath(subject='test', session='two', + ... task='mytask', suffix='channels', + ... extension='.tsv') + >>> print(bids_path.basename) + sub-test_ses-two_task-mytask_channels.tsv + >>> # Then, one can update this `BIDSPath` object in place + >>> bids_path.update(acquisition='test', suffix='ieeg', + ... datatype='ieeg', + ... extension='.vhdr', task=None) + BIDSPath( + root: None + datatype: ieeg + basename: sub-test_ses-two_acq-test_ieeg.vhdr) + >>> print(bids_path.basename) + sub-test_ses-two_acq-test_ieeg.vhdr + """ + # Update .check attribute + if check is not None: + self.check = check + + for key, val in kwargs.items(): + if key == "root": + _validate_type(val, types=("path-like", None), item_name=key) + continue + + if key == "datatype": + if val is not None and val not in ALLOWED_DATATYPES and self.check: + raise ValueError( + f"datatype ({val}) is not valid. " + f"Should be one of " + f"{ALLOWED_DATATYPES}" + ) + else: + continue + + if key not in ENTITY_VALUE_TYPE: + raise ValueError( + f"Key must be one of " f"{ALLOWED_PATH_ENTITIES}, got {key}" + ) + + if ENTITY_VALUE_TYPE[key] == "label": + _validate_type(val, types=(None, str), item_name=key) + else: + assert ENTITY_VALUE_TYPE[key] == "index" + _validate_type(val, types=(int, str, None), item_name=key) + if isinstance(val, str) and not val.isdigit(): + raise ValueError(f"{key} is not an index (Got {val})") + elif isinstance(val, int): + kwargs[key] = f"{val}" + + # ensure extension starts with a '.' + extension = kwargs.get("extension") + if extension is not None and not extension.startswith("."): + kwargs["extension"] = f".{extension}" + del extension + + # error check entities + old_kwargs = dict() + for key, val in kwargs.items(): + # check if there are any characters not allowed + if val is not None and key != "root": + if key == "suffix" and not self.check: + # suffix may skip a check if check=False to allow + # things like "dataset_description.json" + pass + else: + _check_key_val(key, val) + + # set entity value, ensuring `root` is a Path + if val is not None and key == "root": + val = Path(val).expanduser() + old_kwargs[key] = ( + getattr(self, f"{key}") if hasattr(self, f"_{key}") else None + ) + setattr(self, f"_{key}", val) + + # Perform a check of the entities and revert changes if check fails + try: + self._check() + except Exception as e: + old_check = self.check + self.check = False + self.update(**old_kwargs) + self.check = old_check + raise e + return self + + def match(self, ignore_json=True, check=False): + """Get a list of all matching paths in the root directory. + + Performs a recursive search, starting in ``.root`` (if set), based on + `BIDSPath.entities` object. Ignores ``.json`` files. + + Parameters + ---------- + ignore_json : bool + If ``True``, ignores json files. Defaults to ``True``. + check : bool + If ``True``, only returns paths that conform to BIDS. If ``False`` + (default), the ``.check`` attribute of the returned + `mne_bids.BIDSPath` object will be set to ``True`` for paths that + do conform to BIDS, and to ``False`` for those that don't. + + Returns + ------- + bids_paths : list of mne_bids.BIDSPath + The matching paths. + """ + if self.root is None: + raise RuntimeError( + "Cannot match basenames if `root` " + "attribute is not set. Please set the" + "BIDS root directory path to `root` via " + "BIDSPath.update()." + ) + + paths = _return_root_paths( + self.root, datatype=self.datatype, ignore_json=ignore_json + ) + + fnames = _filter_fnames( + paths, suffix=self.suffix, extension=self.extension, **self.entities + ) + + bids_paths = _fnames_to_bidspaths(fnames, self.root, check=check) + return bids_paths + + def _check(self): + """Deep check or not of the instance.""" + self.basename # run basename to check validity of arguments + + # perform error check on scans + if ( + self.suffix == "scans" and self.extension == ".tsv" + ) and _check_non_sub_ses_entity(self): + raise ValueError( + "scans.tsv file name can only contain " + "subject and session entities. BIDSPath " + f"currently contains {self.entities}." + ) + + # perform deeper check if user has it turned on + if self.check: + _check_empty_room_basename(self) + + if ( + self.acquisition in ("calibration", "crosstalk") + and self.task is not None + ): + raise ValueError( + f'task must be None if the acquisition is "calibration" or ' + f'"crosstalk", but received: {self.task}' + ) + + # ensure extension starts with a '.' + extension = self.extension + if extension is not None: + # check validity of the extension + if extension not in ALLOWED_FILENAME_EXTENSIONS: + raise ValueError( + f"Extension {extension} is not " + f"allowed. Use one of these extensions " + f"{ALLOWED_FILENAME_EXTENSIONS}." + ) + + # labels from space entity must come from list (appendix VIII) + space = self.space + if space is not None: + datatype = getattr(self, "datatype", None) + if datatype is None: + raise ValueError( + "You must define datatype if you want to " + "use space in your BIDSPath." + ) + + allowed_spaces_for_dtype = ALLOWED_SPACES.get(datatype, None) + if allowed_spaces_for_dtype is None: + raise ValueError( + f"space entity is not valid for datatype " f"{self.datatype}" + ) + elif space not in allowed_spaces_for_dtype: + raise ValueError( + f"space ({space}) is not valid for " + f"datatype ({self.datatype}).\n" + f"Should be one of " + f"{allowed_spaces_for_dtype}" + ) + else: + pass + + # error check suffix + suffix = self.suffix + if suffix is not None and suffix not in ALLOWED_FILENAME_SUFFIX: + raise ValueError( + f"Suffix {suffix} is not allowed. " + f"Use one of these suffixes " + f"{ALLOWED_FILENAME_SUFFIX}." + ) + + @verbose + def find_empty_room(self, use_sidecar_only=False, *, verbose=None): + """Find the corresponding empty-room file of an MEG recording. + + This will only work if the ``.root`` attribute of the + :class:`mne_bids.BIDSPath` instance has been set. + + Parameters + ---------- + use_sidecar_only : bool + Whether to only check the ``AssociatedEmptyRoom`` entry in the + sidecar JSON file or not. If ``False``, first look for the entry, + and if unsuccessful, try to find the best-matching empty-room + recording in the dataset based on the measurement date. + %(verbose)s + + Returns + ------- + BIDSPath | None + The path corresponding to the best-matching empty-room measurement. + Returns ``None`` if none was found. + """ + if self.datatype not in ("meg", None): + raise ValueError("Empty-room data is only supported for MEG datasets") + + if self.root is None: + raise ValueError( + 'The root of the "bids_path" must be set. ' + 'Please use `bids_path.update(root="")` ' + "to set the root of the BIDS folder to read." + ) + + # needed to deal with inheritance principle + sidecar_fname = ( + self.copy() + .update(datatype=None, suffix="meg") + .find_matching_sidecar(extension=".json") + ) + with open(sidecar_fname, encoding="utf-8") as f: + sidecar_json = json.load(f) + + if "AssociatedEmptyRoom" in sidecar_json: + logger.info( + 'Using "AssociatedEmptyRoom" entry from MEG sidecar ' + "file to retrieve empty-room path." + ) + emptytoom_path = sidecar_json["AssociatedEmptyRoom"] + er_bids_path = get_bids_path_from_fname(emptytoom_path) + er_bids_path.root = self.root + er_bids_path.datatype = "meg" + elif use_sidecar_only: + logger.info( + "The MEG sidecar file does not contain an " + '"AssociatedEmptyRoom" entry. Aborting search for an ' + "empty-room recording, as you passed use_sidecar_only=True" + ) + return None + else: + logger.info( + "The MEG sidecar file does not contain an " + '"AssociatedEmptyRoom" entry. Will try to find a matching ' + "empty-room recording based on the measurement date …" + ) + er_bids_path = _find_matched_empty_room(self) + + if er_bids_path is not None and not er_bids_path.fpath.exists(): + raise FileNotFoundError( + f"Empty-room BIDS path resolved but not found:\n" + f"{er_bids_path}\n" + "Check your BIDS dataset for completeness." + ) + + return er_bids_path + + def get_empty_room_candidates(self): + """Get the list of empty-room candidates for the given file. + + Returns + ------- + candidates : list of BIDSPath + The candidate files that will be checked if the sidecar does not + contain an "AssociatedEmptyRoom" entry. + + Notes + ----- + .. versionadded:: 0.12.0 + """ + return _find_empty_room_candidates(self) + + def find_matching_sidecar(self, suffix=None, extension=None, *, on_error="raise"): + """Get the matching sidecar JSON path. + + Parameters + ---------- + suffix : str | None + The filename suffix. This is the entity after the last ``_`` + before the extension. E.g., ``'ieeg'``. + extension : str | None + The extension of the filename. E.g., ``'.json'``. + on_error : 'raise' | 'warn' | 'ignore' + If no matching sidecar file was found and this is set to + ``'raise'``, raise a ``RuntimeError``. If ``'warn'``, emit a + warning, and if ``'ignore'``, neither raise an exception nor a + warning, and return ``None`` in both cases. + + Returns + ------- + sidecar_path : pathlib.Path | None + The path to the sidecar JSON file. + """ + return _find_matching_sidecar( + self, + suffix=suffix, + extension=extension, + on_error=on_error, + ) + + @property + def meg_calibration_fpath(self): + """Find the matching Elekta/Neuromag/MEGIN fine-calibration file. + + This requires that at least ``root`` and ``subject`` are set, and that + ``datatype`` is either ``'meg'`` or ``None``. + + Returns + ------- + path : pathlib.Path | None + The path of the fine-calibration file, or ``None`` if it couldn't + be found. + """ + if self.root is None or self.subject is None: + raise ValueError("root and subject must be set.") + if self.datatype not in (None, "meg"): + raise ValueError("Can only find fine-calibration file for MEG datasets.") + + path = BIDSPath( + subject=self.subject, + session=self.session, + acquisition="calibration", + suffix="meg", + extension=".dat", + datatype="meg", + root=self.root, + ).fpath + if not path.exists(): + path = None + + return path + + @property + def meg_crosstalk_fpath(self): + """Find the matching Elekta/Neuromag/MEGIN crosstalk file. + + This requires that at least ``root`` and ``subject`` are set, and that + ``datatype`` is either ``'meg'`` or ``None``. + + Returns + ------- + path : pathlib.Path | None + The path of the crosstalk file, or ``None`` if it couldn't be + found. + """ + if self.root is None or self.subject is None: + raise ValueError("root and subject must be set.") + if self.datatype not in (None, "meg"): + raise ValueError("Can only find crosstalk file for MEG datasets.") + + path = BIDSPath( + subject=self.subject, + session=self.session, + acquisition="crosstalk", + suffix="meg", + extension=".fif", + datatype="meg", + root=self.root, + ).fpath + if not path.exists(): + path = None + + return path + + +def _get_matching_bidspaths_from_filesystem(bids_path): + """Get matching file paths for a BIDSPath. + + Assumes suffix and/or extension is not provided. + """ + # extract relevant entities to find filepath + sub, ses = bids_path.subject, bids_path.session + datatype = bids_path.datatype + basename, bids_root = bids_path.basename, bids_path.root + + if datatype is None: + datatype = _infer_datatype(root=bids_root, sub=sub, ses=ses) + + data_dir = BIDSPath( + subject=sub, session=ses, datatype=datatype, root=bids_root + ).directory + + # For BTi data, just return the directory with a '.pdf' extension + # to facilitate reading in mne-bids + bti_dir = op.join(data_dir, f"{basename}") + if op.isdir(bti_dir): + logger.info(f"Assuming BTi data in {bti_dir}") + matching_paths = [f"{bti_dir}.pdf"] + # otherwise, search for valid file paths + else: + search_str = bids_root + # parse down the BIDS directory structure + if sub is not None: + search_str = op.join(search_str, f"sub-{sub}") + if ses is not None: + search_str = op.join(search_str, f"ses-{ses}") + if datatype is not None: + search_str = op.join(search_str, datatype) + else: + search_str = op.join(search_str, "**") + search_str = op.join(search_str, f"{basename}*") + + # Find all matching files in all supported formats. + valid_exts = ALLOWED_FILENAME_EXTENSIONS + matching_paths = glob.glob(search_str) + matching_paths = [p for p in matching_paths if _parse_ext(p)[1] in valid_exts] + return matching_paths + + +def _check_non_sub_ses_entity(bids_path): + """Check existence of non subject/session entities in BIDSPath.""" + if ( + bids_path.task + or bids_path.acquisition + or bids_path.run + or bids_path.space + or bids_path.recording + or bids_path.split + or bids_path.processing + ): + return True + return False + + +def _print_lines_with_entry(file, entry, folder, is_tsv, line_numbers, outfile): + """Print the lines that contain the entry. + + Parameters + ---------- + file : str + The text file to look though. + entry : str + The string to look in the text file for. + folder : str + The base folder for relative file path printing. + is_tsv : bool + If ``True``, things that format a tsv nice will be used. + line_numbers : bool + Whether to include line numbers in the printout. + outfile : io.StringIO | None + The argument to pass to `print` for `file`. If ``None``, + prints to the console, else a string is printed to. + """ + entry_lines = list() + with open(file, encoding="utf-8-sig") as fid: + if is_tsv: # format tsv files nicely + header = _truncate_tsv_line(fid.readline()) + if line_numbers: + header = f"1 {header}" + header = header.rstrip() + for i, line in enumerate(fid): + if entry in line: + if is_tsv: + line = _truncate_tsv_line(line) + if line_numbers: + line = str(i + 2) + (5 - len(str(i + 2))) * " " + line + entry_lines.append(line.rstrip()) + if entry_lines: + print(op.relpath(file, folder), file=outfile) + if is_tsv: + print(f" {header}", file=outfile) + if len(entry_lines) > 10: + entry_lines = entry_lines[:10] + entry_lines.append("...") + for line in entry_lines: + print(f" {line}", file=outfile) + + +def _truncate_tsv_line(line, lim=10): + """Truncate a line to the specified number of characters.""" + return "".join( + [ + str(val) + (lim - len(val)) * " " if len(val) < lim else f"{val[:lim - 1]} " + for val in line.split("\t") + ] + ) + + +def search_folder_for_text( + entry, folder, extensions=(".json", ".tsv"), line_numbers=True, return_str=False +): + """Find any particular string entry in the text files of a folder. + + .. note:: This is a search function like `grep + `_ + that is formatted nicely for BIDS datasets. + + Parameters + ---------- + entry : str + The string to search for + folder : path-like + The folder in which to search. + extensions : list | tuple | str + The extensions to search through. Default is ``json`` and + ``tsv`` which are the BIDS sidecar file types. + line_numbers : bool + Whether to include line numbers. + return_str : bool + If ``True``, return the fields with "n/a" as a str instead of + printing them. + + Returns + ------- + str | None + If `return_str` is ``True``, the fields are returned as a + string. Else, ``None`` is returned and the fields are printed. + """ + _validate_type(entry, str, "entry") + if not op.isdir(folder): + raise ValueError("{folder} is not a directory") + folder = Path(folder) # ensure pathlib.Path + + extensions = (extensions,) if isinstance(extensions, str) else extensions + _validate_type(extensions, (tuple, list)) + _validate_type(line_numbers, bool, "line_numbers") + _validate_type(return_str, bool, "return_str") + outfile = StringIO() if return_str else None + + for extension in extensions: + for file in folder.rglob("*" + extension): + _print_lines_with_entry( + file, entry, folder, extension == ".tsv", line_numbers, outfile + ) + + if outfile is not None: + return outfile.getvalue() + + +def _check_max_depth(max_depth): + """Check that max depth is a proper input.""" + msg = "`max_depth` must be a positive integer or None" + if not isinstance(max_depth, (int, type(None))): + raise ValueError(msg) + if max_depth is None: + max_depth = float("inf") + if max_depth < 0: + raise ValueError(msg) + # Use max_depth same as the -L param in the unix `tree` command + max_depth += 1 + return max_depth + + +def print_dir_tree(folder, max_depth=None, return_str=False): + """Recursively print a directory tree. + + Parameters + ---------- + folder : path-like + The folder for which to print the directory tree. + max_depth : int + The maximum depth into which to descend recursively for printing + the directory tree. + return_str : bool + If ``True``, return the directory tree as a str instead of + printing it. + + Returns + ------- + str | None + If `return_str` is ``True``, the directory tree is returned as a + string. Else, ``None`` is returned and the directory tree is printed. + """ + folder = _check_fname( + fname=folder, overwrite="read", must_exist=True, name="Folder", need_dir=True + ) + max_depth = _check_max_depth(max_depth) + + _validate_type(return_str, bool, "return_str") + outfile = StringIO() if return_str else None + + # Base length of a tree branch, to normalize each tree's start to 0 + baselen = len(str(folder).split(os.sep)) - 1 + + # Recursively walk through all directories + for root, dirs, files in os.walk(folder, topdown=True): + # Since we're using `topdown=True`, sorting `dirs` ensures that + # `os.walk` will continue walking through directories in alphabetical + # order. So although we're not actually using `dirs` anywhere below, + # sorting it here is imperative to ensure the correct (alphabetical) + # directory sort order in the output. + dirs.sort() + files.sort() + + # Check how far we have walked + branchlen = len(root.split(os.sep)) - baselen + + # Only print if this is up to the depth we asked + if branchlen <= max_depth: + if branchlen <= 1: + print(f"|{op.basename(root) + os.sep}", file=outfile) + else: + print( + "|{} {}".format( + (branchlen - 1) * "---", op.basename(root) + os.sep + ), + file=outfile, + ) + + # Only print files if we are NOT yet up to max_depth or beyond + if branchlen < max_depth: + for file in files: + print("|{} {}".format(branchlen * "---", file), file=outfile) + + if outfile is not None: + return outfile.getvalue() + + +def _parse_ext(raw_fname): + """Split a filename into its name and extension.""" + raw_fname = str(raw_fname) + fname, ext = os.path.splitext(raw_fname) + # BTi data is the only file format that does not have a file extension + if ext == "" or "c,rf" in fname: + logger.info( + 'Found no extension for raw file, assuming "BTi" format ' + "and appending extension .pdf" + ) + ext = ".pdf" + # If ending on .gz, check whether it is an .nii.gz file + elif ext == ".gz" and raw_fname.endswith(".nii.gz"): + ext = ".nii.gz" + fname = fname[:-4] # cut off the .nii + return fname, ext + + +def _infer_datatype_from_path(fname: Path): + # get the parent + if fname.exists(): + datatype = fname.parent.name + if any([datatype.startswith(entity) for entity in ["sub", "ses"]]): + datatype = None + elif fname.stem.split("_")[-1] in ("meg", "eeg", "ieeg"): + datatype = fname.stem.split("_")[-1] + else: + datatype = None + + return datatype + + +@verbose +def get_bids_path_from_fname(fname, check=True, verbose=None): + """Retrieve a BIDSPath object from a filename. + + Parameters + ---------- + fname : path-like + The path to parse a `BIDSPath` from. + check : bool + Whether to check if the generated `BIDSPath` complies with the BIDS + specification, i.e., whether all included entities and the suffix are + valid. + %(verbose)s + + Returns + ------- + bids_path : BIDSPath + The BIDSPath object. + """ + fpath = Path(fname) + fname = fpath.name + + entities = get_entities_from_fname(fname) + + # parse suffix and extension + last_entity = fname.split("-")[-1] + if "_" in last_entity: + suffix = last_entity.split("_")[-1] + suffix, extension = _get_bids_suffix_and_ext(suffix) + else: + suffix = None + extension = Path(fname).suffix # already starts with a period + if extension == "": + extension = None + + if extension is not None: + assert extension.startswith(".") # better safe than sorry + + datatype = _infer_datatype_from_path(fpath) + + # find root and datatype if it exists + if fpath.parent == "": + root = None + else: + root_level = 0 + # determine root if it's there + if entities["subject"] is not None: + root_level += 1 + if entities["session"] is not None: + root_level += 1 + if suffix != "scans": + root_level += 1 + + if root_level: + root = fpath.parent + for _ in range(root_level): + root = root.parent + + bids_path = BIDSPath( + root=root, + datatype=datatype, + suffix=suffix, + extension=extension, + **entities, + check=check, + ) + if verbose: + logger.info(f"From {fpath}, formed a BIDSPath: {bids_path}.") + return bids_path + + +@verbose +def get_entities_from_fname(fname, on_error="raise", verbose=None): + """Retrieve a dictionary of BIDS entities from a filename. + + Entities not present in ``fname`` will be assigned the value of ``None``. + + Parameters + ---------- + fname : BIDSPath | path-like + The path to parse. + on_error : 'raise' | 'warn' | 'ignore' + If any unsupported labels in the filename are found and this is set + to ``'raise'``, raise a ``RuntimeError``. If ``'warn'``, + emit a warning and continue, and if ``'ignore'``, + neither raise an exception nor a warning, and + return all entities found. For example, currently MNE-BIDS does not + support derivatives yet, but the ``desc`` entity label is used to + differentiate different derivatives and will work with this function + if ``on_error='ignore'``. + %(verbose)s + + Returns + ------- + params : dict + A dictionary with the keys corresponding to the BIDS entity names, and + the values to the entity values encoded in the filename. + + Examples + -------- + >>> fname = 'sub-01_ses-exp_run-02_meg.fif' + >>> get_entities_from_fname(fname) + {'subject': '01', \ +'session': 'exp', \ +'task': None, \ +'acquisition': None, \ +'run': '02', \ +'processing': None, \ +'space': None, \ +'recording': None, \ +'split': None, \ +'description': None} + """ + if on_error not in ("warn", "raise", "ignore"): + raise ValueError( + f"Acceptable values for on_error are: warn, raise, " + f"ignore, but got: {on_error}" + ) + + fname = str(fname) # to accept also BIDSPath or Path instances + + # filename keywords to the BIDS entity mapping + entity_vals = list(ALLOWED_PATH_ENTITIES_SHORT.values()) + fname_vals = list(ALLOWED_PATH_ENTITIES_SHORT.keys()) + + params = {key: None for key in entity_vals} + idx_key = 0 + for match in re.finditer(param_regex, op.basename(fname)): + key, value = match.groups() + + if on_error in ("raise", "warn"): + if key not in fname_vals: + msg = f'Unexpected entity "{key}" found in ' f'filename "{fname}"' + if on_error == "raise": + raise KeyError(msg) + elif on_error == "warn": + warn(msg) + continue + if fname_vals.index(key) < idx_key: + msg = ( + f"Entities in filename not ordered correctly." + f' "{key}" should have occurred earlier in the ' + f'filename "{fname}"' + ) + raise ValueError(msg) + idx_key = fname_vals.index(key) + + key_short_hand = ALLOWED_PATH_ENTITIES_SHORT.get(key, key) + params[key_short_hand] = value + return params + + +def _find_matching_sidecar(bids_path, suffix=None, extension=None, on_error="raise"): + """Try to find a sidecar file with a given suffix for a data file. + + Parameters + ---------- + bids_path : BIDSPath + Full name of the data file. + suffix : str | None + The filename suffix. This is the entity after the last ``_`` + before the extension. E.g., ``'ieeg'``. + extension : str | None + The extension of the filename. E.g., ``'.json'``. + on_error : 'raise' | 'warn' | 'ignore' + If no matching sidecar file was found and this is set to ``'raise'``, + raise a ``RuntimeError``. If ``'warn'``, emit a warning, and if + ``'ignore'``, neither raise an exception nor a warning, and return + ``None`` in both cases. + + Returns + ------- + sidecar_fname : str | None + Path to the identified sidecar file, or ``None`` if none could be found + and ``on_error`` was set to ``'warn'`` or ``'ignore'``. + + """ + if on_error not in ("warn", "raise", "ignore"): + raise ValueError( + f"Acceptable values for on_error are: warn, raise, " + f"ignore, but got: {on_error}" + ) + + bids_root = bids_path.root + + # search suffix is BIDS-suffix and extension + search_suffix = "" + if suffix is None and bids_path.suffix is not None: + search_suffix = bids_path.suffix + elif suffix is not None: + search_suffix = suffix + + # do not search for suffix if suffix is explicitly passed + bids_path = bids_path.copy() + bids_path.check = False + bids_path.update(suffix=None) + + if extension is None and bids_path.extension is not None: + search_suffix = search_suffix + bids_path.extension + elif extension is not None: + search_suffix = search_suffix + extension + + # do not search for extension if extension is explicitly passed + bids_path = bids_path.copy() + bids_path.check = False + bids_path = bids_path.update(extension=None) + + # We only use subject and session as identifier, because all other + # parameters are potentially not binding for metadata sidecar files + search_str_filename = f"sub-{bids_path.subject}" + if bids_path.session is not None: + search_str_filename += f"_ses-{bids_path.session}" + + # Find all potential sidecar files, doing a recursive glob + # from bids_root/sub-*, potentially taking into account the data type + search_dir = Path(bids_root) / f"sub-{bids_path.subject}" + # ** -> don't forget about potentially present session directories + if bids_path.datatype is None: + search_dir = search_dir / "**" + else: + search_dir = search_dir / "**" / bids_path.datatype + + search_str_complete = str(search_dir / f"{search_str_filename}*{search_suffix}") + + candidate_list = glob.glob(search_str_complete, recursive=True) + best_candidates = _find_best_candidates(bids_path.entities, candidate_list) + if len(best_candidates) == 1: + # Success + return Path(best_candidates[0]) + + # We failed. Construct a helpful error message. + # If this was expected, simply return None, otherwise, raise an exception. + msg = None + if len(best_candidates) == 0: + msg = ( + f"Did not find any {search_suffix} " + f"associated with {bids_path.basename}." + ) + elif len(best_candidates) > 1: + # More than one candidates were tied for best match + msg = ( + f"Expected to find a single {search_suffix} file " + f"associated with {bids_path.basename}, " + f"but found {len(candidate_list)}:\n\n" + "\n".join(candidate_list) + ) + msg += f'\n\nThe search_str was "{search_str_complete}"' + if on_error == "raise": + raise RuntimeError(msg) + elif on_error == "warn": + warn(msg) + + return None + + +def _get_bids_suffix_and_ext(str_suffix): + """Parse suffix for valid suffix and ext.""" + # no matter what the suffix is, suffix and extension are last + suffix = str_suffix + ext = None + if "." in str_suffix: + # handle case of multiple '.' in extension + split_str = str_suffix.split(".") + suffix = split_str[0] + ext = ".".join(split_str[1:]) + ext = f".{ext}" # prepend period + return suffix, ext + + +@verbose +def get_datatypes(root, verbose=None): + """Get list of data types ("modalities") present in a BIDS dataset. + + Parameters + ---------- + root : path-like + Path to the root of the BIDS directory. + %(verbose)s + + Returns + ------- + modalities : list of str + List of the data types present in the BIDS dataset pointed to by + `root`. + + """ + # Take all possible data types from "entity" table + # (Appendix in BIDS spec) + # https://bids-specification.readthedocs.io/en/latest/appendices/entity-table.html # noqa + datatype_list = ("anat", "func", "dwi", "fmap", "beh", "meg", "eeg", "ieeg", "nirs") + datatypes = list() + for root, dirs, files in os.walk(root): + for dir in dirs: + if dir in datatype_list and dir not in datatypes: + datatypes.append(dir) + + return datatypes + + +@verbose +def get_entity_vals( + root, + entity_key, + *, + ignore_subjects="emptyroom", + ignore_sessions=None, + ignore_tasks=None, + ignore_acquisitions=None, + ignore_runs=None, + ignore_processings=None, + ignore_spaces=None, + ignore_recordings=None, + ignore_splits=None, + ignore_descriptions=None, + ignore_modalities=None, + ignore_datatypes=None, + ignore_dirs=("derivatives", "sourcedata"), + with_key=False, + verbose=None, +): + """Get list of values associated with an `entity_key` in a BIDS dataset. + + BIDS file names are organized by key-value pairs called "entities" [1]_. + With this function, you can get all values for an entity indexed by its + key. + + Parameters + ---------- + root : path-like + Path to the "root" directory from which to start traversing to gather + BIDS entities from file- and folder names. This will commonly be the + BIDS root, but it may also be a subdirectory inside of a BIDS dataset, + e.g., the ``sub-X`` directory of a hypothetical subject ``X``. + + .. note:: This function searches the names of all files and directories + nested within ``root``. Depending on the size of your + dataset and storage system, searching the entire BIDS dataset + may take a **considerable** amount of time (seconds up to + several minutes). If you find yourself running into such + performance issues, consider limiting the search to only a + subdirectory in the dataset, e.g., to a single subject or + session only. + + entity_key : str + The name of the entity key to search for. + ignore_subjects : str | array-like of str | None + Subject(s) to ignore. By default, entities from the ``emptyroom`` + mock-subject are not returned. If ``None``, include all subjects. + ignore_sessions : str | array-like of str | None + Session(s) to ignore. If ``None``, include all sessions. + ignore_tasks : str | array-like of str | None + Task(s) to ignore. If ``None``, include all tasks. + ignore_acquisitions : str | array-like of str | None + Acquisition(s) to ignore. If ``None``, include all acquisitions. + ignore_runs : str | array-like of str | None + Run(s) to ignore. If ``None``, include all runs. + ignore_processings : str | array-like of str | None + Processing(s) to ignore. If ``None``, include all processings. + ignore_spaces : str | array-like of str | None + Space(s) to ignore. If ``None``, include all spaces. + ignore_recordings : str | array-like of str | None + Recording(s) to ignore. If ``None``, include all recordings. + ignore_splits : str | array-like of str | None + Split(s) to ignore. If ``None``, include all splits. + ignore_descriptions : str | array-like of str | None + Description(s) to ignore. If ``None``, include all descriptions. + + .. versionadded:: 0.11 + ignore_modalities : str | array-like of str | None + Modalities(s) to ignore. If ``None``, include all modalities. + ignore_datatypes : str | array-like of str | None + Datatype(s) to ignore. If ``None``, include all datatypes (i.e. + ``anat``, ``ieeg``, ``eeg``, ``meg``, ``func``, etc.) + ignore_dirs : str | array-like of str | None + Directories nested directly within ``root`` to ignore. If ``None``, + include all directories in the search. + + .. versionadded:: 0.9 + with_key : bool + If ``True``, returns the full entity with the key and the value. This + will for example look like ``['sub-001', 'sub-002']``. + If ``False`` (default), just returns the entity values. This + will for example look like ``['001', '002']``. + %(verbose)s + + Returns + ------- + entity_vals : list of str + List of the values associated with an `entity_key` in the BIDS dataset + pointed to by `root`. + + Examples + -------- + >>> root = Path('./mne_bids/tests/data/tiny_bids').absolute() + >>> entity_key = 'subject' + >>> get_entity_vals(root, entity_key) + ['01'] + >>> get_entity_vals(root, entity_key, with_key=True) + ['sub-01'] + + Notes + ----- + This function will scan the entire ``root``, except for a + ``derivatives`` subfolder placed directly under ``root``. + + References + ---------- + .. [1] https://bids-specification.rtfd.io/en/latest/common-principles.html#entities + + """ + root = _check_fname( + fname=root, + overwrite="read", + must_exist=True, + need_dir=True, + name="Root directory", + ) + root = Path(root).expanduser() + + entities = ( + "subject", + "task", + "session", + "acquisition", + "run", + "processing", + "space", + "recording", + "split", + "description", + "suffix", + ) + entities_abbr = ( + "sub", + "task", + "ses", + "acq", + "run", + "proc", + "space", + "rec", + "split", + "desc", + "suffix", + ) + entity_long_abbr_map = dict(zip(entities, entities_abbr)) + + if entity_key not in entities: + raise ValueError( + f'`key` must be one of: {", ".join(entities)}. ' f"Got: {entity_key}" + ) + + ignore_subjects = _ensure_tuple(ignore_subjects) + ignore_sessions = _ensure_tuple(ignore_sessions) + ignore_tasks = _ensure_tuple(ignore_tasks) + ignore_acquisitions = _ensure_tuple(ignore_acquisitions) + ignore_runs = _ensure_tuple(ignore_runs) + ignore_processings = _ensure_tuple(ignore_processings) + ignore_spaces = _ensure_tuple(ignore_spaces) + ignore_recordings = _ensure_tuple(ignore_recordings) + ignore_splits = _ensure_tuple(ignore_splits) + ignore_descriptions = _ensure_tuple(ignore_descriptions) + ignore_modalities = _ensure_tuple(ignore_modalities) + + ignore_dirs = _ensure_tuple(ignore_dirs) + existing_ignore_dirs = [ + root / d for d in ignore_dirs if (root / d).exists() and (root / d).is_dir() + ] + ignore_dirs = _ensure_tuple(existing_ignore_dirs) + + p = re.compile(rf"{entity_long_abbr_map[entity_key]}-(.*?)_") + values = list() + filenames = root.glob(f"**/*{entity_long_abbr_map[entity_key]}-*_*") + + for filename in filenames: + # Skip ignored directories + # XXX In Python 3.9, we can use Path.is_relative_to() here + if any( + [str(filename).startswith(str(ignore_dir)) for ignore_dir in ignore_dirs] + ): + continue + + if ignore_datatypes and filename.parent.name in ignore_datatypes: + continue + if ignore_subjects and any( + [filename.stem.startswith(f"sub-{s}_") for s in ignore_subjects] + ): + continue + if ignore_sessions and any( + [f"_ses-{s}_" in filename.stem for s in ignore_sessions] + ): + continue + if ignore_tasks and any([f"_task-{t}_" in filename.stem for t in ignore_tasks]): + continue + if ignore_acquisitions and any( + [f"_acq-{a}_" in filename.stem for a in ignore_acquisitions] + ): + continue + if ignore_runs and any([f"_run-{r}_" in filename.stem for r in ignore_runs]): + continue + if ignore_processings and any( + [f"_proc-{p}_" in filename.stem for p in ignore_processings] + ): + continue + if ignore_spaces and any( + [f"_space-{s}_" in filename.stem for s in ignore_spaces] + ): + continue + if ignore_recordings and any( + [f"_rec-{a}_" in filename.stem for a in ignore_recordings] + ): + continue + if ignore_splits and any( + [f"_split-{s}_" in filename.stem for s in ignore_splits] + ): + continue + if ignore_descriptions and any( + [f"_desc-{d}_" in filename.stem for d in ignore_descriptions] + ): + continue + if ignore_modalities and any( + [f"_{k}" in filename.stem for k in ignore_modalities] + ): + continue + + match = p.search(filename.stem) + value = match.group(1) + if with_key: + value = f"{entity_long_abbr_map[entity_key]}-{value}" + if value not in values: + values.append(value) + return sorted(values) + + +def _mkdir_p(path, overwrite=False): + """Create a directory, making parent directories as needed [1]. + + References + ---------- + .. [1] stackoverflow.com/questions/600268/mkdir-p-functionality-in-python + + """ + if overwrite and op.isdir(path): + sh.rmtree(path) + logger.info(f"Clearing path: {path}") + + os.makedirs(path, exist_ok=True) + if not op.isdir(path): + logger.info(f"Creating folder: {path}") + + +def _find_best_candidates(params, candidate_list): + """Return the best candidate, based on the number of param matches. + + Assign each candidate a score, based on how many entities are shared with + the ones supplied in the `params` parameter. The candidate with the highest + score wins. Candidates with entities that conflict with the supplied + `params` are disqualified. + + Parameters + ---------- + params : dict + The entities that the candidate should match. + candidate_list : list of str + A list of candidate filenames. + + Returns + ------- + best_candidates : list of str + A list of all the candidate filenames that are tied for first place. + Hopefully, the list will have a length of one. + """ + params = {key: value for key, value in params.items() if value is not None} + + best_candidates = [] + best_n_matches = 0 + for candidate in candidate_list: + n_matches = 0 + candidate_disqualified = False + candidate_params = get_entities_from_fname(candidate) + for entity, value in params.items(): + if entity in candidate_params: + if candidate_params[entity] is None: + continue + elif candidate_params[entity] == value: + n_matches += 1 + else: + # Incompatible entity found, candidate is disqualified + candidate_disqualified = True + break + if not candidate_disqualified: + if n_matches > best_n_matches: + best_n_matches = n_matches + best_candidates = [candidate] + elif n_matches == best_n_matches: + best_candidates.append(candidate) + return best_candidates + + +def _get_datatypes_for_sub(*, root, sub, ses=None): + """Retrieve data modalities for a specific subject and session.""" + subject_dir = op.join(root, f"sub-{sub}") + if ses is not None: + subject_dir = op.join(subject_dir, f"ses-{ses}") + + # TODO We do this to ensure we don't accidentally pick up any "spurious" + # TODO sub-directories. But is that really necessary with valid BIDS data? + modalities_in_dataset = get_datatypes(root=root) + subdirs = [f.name for f in os.scandir(subject_dir) if f.is_dir()] + available_modalities = [s for s in subdirs if s in modalities_in_dataset] + return available_modalities + + +def _infer_datatype(*, root, sub, ses): + # Check which suffix is available for this particular + # subject & session. If we get no or multiple hits, throw an error. + + modalities = _get_datatypes_for_sub(root=root, sub=sub, ses=ses) + + # We only want to handle electrophysiological data here. + allowed_recording_modalities = ["meg", "eeg", "ieeg"] + modalities = list(set(modalities) & set(allowed_recording_modalities)) + if not modalities: + raise ValueError("No electrophysiological data found.") + elif len(modalities) >= 2: + msg = ( + f"Found data of more than one recording datatype. Please " + f"pass the `suffix` parameter to specify which data to load. " + f"Found the following modalitiess: {modalities}" + ) + raise RuntimeError(msg) + + assert len(modalities) == 1 + return modalities[0] + + +def _path_to_str(var): + """Make sure var is a string or Path, return string representation.""" + if not isinstance(var, (Path, str)): + raise ValueError( + f"All path parameters must be either strings or " + f"pathlib.Path objects. Found type {type(var)}." + ) + else: + return str(var) + + +def _filter_fnames( + fnames, + *, + subject=None, + session=None, + task=None, + acquisition=None, + run=None, + processing=None, + recording=None, + space=None, + split=None, + description=None, + suffix=None, + extension=None, +): + """Filter a list of BIDS filenames / paths based on BIDS entity values. + + Input can be str or list of str. + + Parameters + ---------- + fnames : iterable of pathlib.Path | iterable of str + + Returns + ------- + list of pathlib.Path + + """ + subject = _ensure_tuple(subject) + session = _ensure_tuple(session) + task = _ensure_tuple(task) + acquisition = _ensure_tuple(acquisition) + run = _ensure_tuple(run) + processing = _ensure_tuple(processing) + space = _ensure_tuple(space) + recording = _ensure_tuple(recording) + split = _ensure_tuple(split) + description = _ensure_tuple(description) + suffix = _ensure_tuple(suffix) + extension = _ensure_tuple(extension) + + leading_path_str = r".*\/?" # nothing or something ending with a `/` + sub_str = r"sub-(" + "|".join(subject) + ")" if subject else r"sub-([^_]+)" + ses_str = r"_ses-(" + "|".join(session) + ")" if session else r"(|_ses-([^_]+))" + task_str = r"_task-(" + "|".join(task) + ")" if task else r"(|_task-([^_]+))" + acq_str = ( + r"_acq-(" + "|".join(acquisition) + ")" if acquisition else r"(|_acq-([^_]+))" + ) + run_str = r"_run-(" + "|".join(run) + ")" if run else r"(|_run-([^_]+))" + proc_str = ( + r"_proc-(" + "|".join(processing) + ")" if processing else r"(|_proc-([^_]+))" + ) + space_str = r"_space-(" + "|".join(space) + ")" if space else r"(|_space-([^_]+))" + rec_str = r"_rec-(" + "|".join(recording) + ")" if recording else r"(|_rec-([^_]+))" + split_str = r"_split-(" + "|".join(split) + ")" if split else r"(|_split-([^_]+))" + desc_str = ( + r"_desc-(" + "|".join(description) + ")" if description else r"(|_desc-([^_]+))" + ) + suffix_str = r"_(" + "|".join(suffix) + ")" if suffix else r"_([^_]+)" + ext_str = r"(" + "|".join(extension) + ")" if extension else r".([^_]+)" + + regexp = ( + leading_path_str + + sub_str + + ses_str + + task_str + + acq_str + + run_str + + proc_str + + space_str + + rec_str + + split_str + + desc_str + + suffix_str + + ext_str + ) + + # Convert to str so we can apply the regexp ... + fnames = [str(f) for f in fnames] + + # https://stackoverflow.com/a/51246151/1944216 + fnames_filtered = sorted(filter(re.compile(regexp).match, fnames)) + + # ... and return Paths. + fnames_filtered = [Path(f) for f in fnames_filtered] + return fnames_filtered + + +def find_matching_paths( + root, + subjects=None, + sessions=None, + tasks=None, + acquisitions=None, + runs=None, + processings=None, + recordings=None, + spaces=None, + splits=None, + descriptions=None, + suffixes=None, + extensions=None, + datatypes=None, + check=False, +): + """Get list of all matching paths for all matching entity values. + + Input can be str or list of str. None matches all found values. + + Performs a recursive search, starting in ``.root`` (if set), based on + `BIDSPath.entities` object. + + Parameters + ---------- + root : pathlib.Path | str + The root of the BIDS path. + subjects : str | array-like of str | None + The subject ID. Corresponds to "sub". + sessions : str | array-like of str | None + The acquisition session. Corresponds to "ses". + tasks : str | array-like of str | None + The experimental task. Corresponds to "task". + acquisitions: str | array-like of str | None + The acquisition parameters. Corresponds to "acq". + runs : str | array-like of str | None + The run number. Corresponds to "run". + processings : str | array-like of str | None + The processing label. Corresponds to "proc". + recordings : str | array-like of str | None + The recording name. Corresponds to "rec". + spaces : str | array-like of str | None + The coordinate space for anatomical and sensor location + files (e.g., ``*_electrodes.tsv``, ``*_markers.mrk``). + Corresponds to "space". + Note that valid values for ``space`` must come from a list + of BIDS keywords as described in the BIDS specification. + splits : str | array-like of str | None + The split of the continuous recording file for ``.fif`` data. + Corresponds to "split". + descriptions : str | array-like of str | None + This corresponds to the BIDS entity ``desc``. It is used to provide + additional information for derivative data, e.g., preprocessed data + may be assigned ``description='cleaned'``. + + .. versionadded:: 0.11 + suffixes : str | array-like of str | None + The filename suffix. This is the entity after the + last ``_`` before the extension. E.g., ``'channels'``. + The following filename suffix's are accepted: + 'meg', 'markers', 'eeg', 'ieeg', 'T1w', + 'participants', 'scans', 'electrodes', 'coordsystem', + 'channels', 'events', 'headshape', 'digitizer', + 'beh', 'physio', 'stim' + extensions : str | array-like of str | None + The extension of the filename. E.g., ``'.json'``. + datatypes : str | array-like of str | None + The BIDS data type, e.g., ``'anat'``, ``'func'``, ``'eeg'``, ``'meg'``, + ``'ieeg'``. + check : bool + If ``True``, only returns paths that conform to BIDS. If ``False`` + (default), the ``.check`` attribute of the returned + `mne_bids.BIDSPath` object will be set to ``True`` for paths that + do conform to BIDS, and to ``False`` for those that don't. + + Returns + ------- + bids_paths : list of mne_bids.BIDSPath + The matching paths. + + """ + fpaths = _return_root_paths(root, datatype=datatypes, ignore_json=False) + + fpaths_filtered = _filter_fnames( + fpaths, + subject=subjects, + session=sessions, + task=tasks, + acquisition=acquisitions, + run=runs, + processing=processings, + recording=recordings, + space=spaces, + split=splits, + description=descriptions, + suffix=suffixes, + extension=extensions, + ) + + bids_paths = _fnames_to_bidspaths(fpaths_filtered, root, check=check) + return bids_paths + + +def _return_root_paths(root, datatype=None, ignore_json=True): + """Return all paths in root. + + Can be filtered by datatype (which is present in the path but not in + the BIDSPath basename). Can also be list of datatypes. + root : pathlib.Path | str + The root of the BIDS path. + datatype : str | array-like of str | None + The BIDS data type, e.g., ``'anat'``, ``'func'``, ``'eeg'``, ``'meg'``, + ``'ieeg'``. + """ + root = Path(root) # if root is str + + if datatype is not None: + datatype = _ensure_tuple(datatype) + search_str = f'*/{"|".join(datatype)}/*' + else: + search_str = "*.*" + + paths = root.rglob(search_str) + # Only keep files (not directories), and omit the JSON sidecars + # if ignore_json is True. + if ignore_json: + paths = [p for p in paths if p.is_file() and p.suffix != ".json"] + else: + paths = [p for p in paths if p.is_file()] + + return paths + + +def _fnames_to_bidspaths(fnames, root, check=False): + """Make BIDSPaths from file names. + + To check whether the BIDSPath is conforming to BIDS if check=True, we + first instantiate without checking and then run the check manually, + allowing us to be more specific about the exception to catch. + + Parameters + ---------- + fnames : list of str + Filenames as list of strings. + root : path-like | None + The root directory of the BIDS dataset. + check : bool + If ``True``, only returns paths that conform to BIDS. If ``False`` + (default), the ``.check`` attribute of the returned + `mne_bids.BIDSPath` object will be set to ``True`` for paths that + do conform to BIDS, and to ``False`` for those that don't. + + Returns + ------- + bids_paths : list of mne_bids.BIDSPath + Bids paths. + + """ + bids_paths = [] + for fname in fnames: + datatype = _infer_datatype_from_path(fname) + bids_path = get_bids_path_from_fname(fname, check=False) + bids_path.root = root + bids_path.datatype = datatype + bids_path.check = True + + try: + bids_path._check() + except ValueError: + # path is not BIDS-compatible + if check: # skip! + continue + else: + bids_path.check = False + + bids_paths.append(bids_path) + return bids_paths diff --git a/mne-bids-0.15/mne_bids/pick.py b/mne-bids-0.15/mne_bids/pick.py new file mode 100644 index 0000000000000000000000000000000000000000..a351d131fc5a95b0dd4156eb4db78e4afae522ca --- /dev/null +++ b/mne-bids-0.15/mne_bids/pick.py @@ -0,0 +1,88 @@ +"""Define coil types for MEG.""" + +# Authors: Matt Sanderson +# +# License: BSD-3-Clause +from mne.io.constants import FIFF + + +def get_coil_types(): + """Return all known coil types. + + Returns + ------- + coil_types : dict + The keys contain the channel types, and the values contain the + corresponding values in the info['chs'][idx]['kind'] + + """ + return dict( + meggradaxial=( + FIFF.FIFFV_COIL_KIT_GRAD, + FIFF.FIFFV_COIL_CTF_GRAD, + # Support for gradient-compensated data: + int(FIFF.FIFFV_COIL_CTF_GRAD | (3 << 16)), + int(FIFF.FIFFV_COIL_CTF_GRAD | (2 << 16)), + FIFF.FIFFV_COIL_AXIAL_GRAD_5CM, + FIFF.FIFFV_COIL_BABY_GRAD, + ), + megrefgradaxial=( + FIFF.FIFFV_COIL_CTF_REF_GRAD, + FIFF.FIFFV_COIL_CTF_OFFDIAG_REF_GRAD, + FIFF.FIFFV_COIL_MAGNES_REF_GRAD, + FIFF.FIFFV_COIL_MAGNES_OFFDIAG_REF_GRAD, + ), + meggradplanar=( + FIFF.FIFFV_COIL_VV_PLANAR_T1, + FIFF.FIFFV_COIL_VV_PLANAR_T2, + FIFF.FIFFV_COIL_VV_PLANAR_T3, + ), + megmag=( + FIFF.FIFFV_COIL_POINT_MAGNETOMETER, + FIFF.FIFFV_COIL_VV_MAG_W, + FIFF.FIFFV_COIL_VV_MAG_T1, + FIFF.FIFFV_COIL_VV_MAG_T2, + FIFF.FIFFV_COIL_VV_MAG_T3, + FIFF.FIFFV_COIL_NM_122, + FIFF.FIFFV_COIL_MAGNES_MAG, + FIFF.FIFFV_COIL_BABY_MAG, + ), + megrefmag=( + FIFF.FIFFV_COIL_KIT_REF_MAG, + FIFF.FIFFV_COIL_CTF_REF_MAG, + FIFF.FIFFV_COIL_MAGNES_REF_MAG, + FIFF.FIFFV_COIL_BABY_REF_MAG, + FIFF.FIFFV_COIL_BABY_REF_MAG2, + FIFF.FIFFV_COIL_ARTEMIS123_REF_MAG, + FIFF.FIFFV_COIL_MAGNES_REF_MAG, + ), + eeg=(FIFF.FIFFV_COIL_EEG,), + misc=(FIFF.FIFFV_COIL_NONE,), + ) + + +def coil_type(info, idx, ch_type="n/a"): + """Get coil type. + + Parameters + ---------- + info : dict + Measurement info + idx : int + Index of channel + ch_type : str + Channel type to fall back upon if a more specific + type is not found + + Returns + ------- + type : 'meggradaxial' | 'megrefgradaxial' | 'meggradplanar' + 'megmag' | 'megrefmag' | 'eeg' | 'misc' + Type of coil + + """ + ch = info["chs"][idx] + for key, values in get_coil_types().items(): + if ch["coil_type"] in values: + return key + return ch_type diff --git a/mne-bids-0.15/mne_bids/read.py b/mne-bids-0.15/mne_bids/read.py new file mode 100644 index 0000000000000000000000000000000000000000..cf91f43d7b3b79f0dc07cfe646338c322e94b1d7 --- /dev/null +++ b/mne-bids-0.15/mne_bids/read.py @@ -0,0 +1,1183 @@ +"""Check whether a file format is supported by BIDS and then load it.""" + +# Authors: Mainak Jas +# Alexandre Gramfort +# Teon Brooks +# Chris Holdgraf +# Stefan Appelhoff +# +# License: BSD-3-Clause +import json +import os +import os.path as op +import re +from datetime import datetime, timezone +from difflib import get_close_matches +from pathlib import Path + +import mne +import numpy as np +from mne import events_from_annotations, io, pick_channels_regexp, read_events +from mne.coreg import fit_matched_points +from mne.transforms import apply_trans +from mne.utils import get_subjects_dir, logger + +from mne_bids.config import ( + ALLOWED_DATATYPE_EXTENSIONS, + ANNOTATIONS_TO_KEEP, + _map_options, + reader, +) +from mne_bids.dig import _read_dig_bids +from mne_bids.path import ( + BIDSPath, + _find_matching_sidecar, + _infer_datatype, + _parse_ext, + get_bids_path_from_fname, +) +from mne_bids.tsv_handler import _drop, _from_tsv +from mne_bids.utils import _get_ch_type_mapping, _import_nibabel, verbose, warn + + +def _read_raw( + raw_path, + electrode=None, + hsp=None, + hpi=None, + allow_maxshield=False, + config_path=None, + **kwargs, +): + """Read a raw file into MNE, making inferences based on extension.""" + _, ext = _parse_ext(raw_path) + + # KIT systems + if ext in [".con", ".sqd"]: + raw = io.read_raw_kit( + raw_path, elp=electrode, hsp=hsp, mrk=hpi, preload=False, **kwargs + ) + + # BTi systems + elif ext == ".pdf": + raw = io.read_raw_bti( + pdf_fname=raw_path, + config_fname=config_path, + head_shape_fname=hsp, + preload=False, + **kwargs, + ) + + elif ext == ".fif": + raw = reader[ext](raw_path, allow_maxshield, **kwargs) + + elif ext in [".ds", ".vhdr", ".set", ".edf", ".bdf", ".EDF", ".snirf", ".cdt"]: + raw_path = Path(raw_path) + raw = reader[ext](raw_path, **kwargs) + + # MEF and NWB are allowed, but not yet implemented + elif ext in [".mef", ".nwb"]: + raise ValueError( + f'Got "{ext}" as extension. This is an allowed ' + f"extension but there is no IO support for this " + f"file format yet." + ) + + # No supported data found ... + # --------------------------- + else: + raise ValueError( + f"Raw file name extension must be one " + f"of {ALLOWED_DATATYPE_EXTENSIONS}\n" + f"Got {ext}" + ) + return raw + + +def _read_events(events, event_id, raw, bids_path=None): + """Retrieve events (for use in *_events.tsv) from FIFF/array & Annotations. + + Parameters + ---------- + events : path-like | np.ndarray | None + If a string, a path to an events file. If an array, an MNE events array + (shape n_events, 3). If None, events will be generated from + ``raw.annotations``. + event_id : dict | None + The event id dict used to create a 'trial_type' column in events.tsv, + mapping a description key to an integer-valued event code. + raw : mne.io.Raw + The data as MNE-Python Raw object. + bids_path : BIDSPath | None + Can be used to determine if the data is a resting-state or empty-room + recording, and will suppress a warning about missing events in this + case. + + Returns + ------- + all_events : np.ndarray, shape = (n_events, 3) + The first column contains the event time in samples and the third + column contains the event id. The second column is ignored for now but + typically contains the value of the trigger channel either immediately + before the event or immediately after. + all_dur : np.ndarray, shape (n_events,) + The event durations in seconds. + all_desc : dict + A dictionary with the keys corresponding to the event descriptions and + the values to the event IDs. + + """ + # retrieve events + if isinstance(events, np.ndarray): + if events.ndim != 2: + raise ValueError("Events must have two dimensions, " f"found {events.ndim}") + if events.shape[1] != 3: + raise ValueError( + "Events must have second dimension of length 3, " + f"found {events.shape[1]}" + ) + events = events + elif events is None: + events = np.empty(shape=(0, 3), dtype=int) + else: + events = read_events(events).astype(int) + + if raw.annotations: + if event_id is None: + logger.info( + "The provided raw data contains annotations, but you did not " + 'pass an "event_id" mapping from annotation descriptions to ' + "event codes. We will generate arbitrary event codes. " + 'To specify custom event codes, please pass "event_id".' + ) + else: + special_annots = {"BAD_ACQ_SKIP"} + desc_without_id = sorted( + set(raw.annotations.description) - set(event_id.keys()) + ) + # auto-add entries to `event_id` for "special" annotation values + # (but only if they're needed) + if set(desc_without_id) & special_annots: + for annot in special_annots: + # use a value guaranteed to not be in use + event_id = {annot: max(event_id.values()) + 90000} | event_id + # remove the "special" annots from the list of problematic annots + desc_without_id = sorted(set(desc_without_id) - special_annots) + if desc_without_id: + raise ValueError( + f"The provided raw data contains annotations, but " + f'"event_id" does not contain entries for all annotation ' + f"descriptions. The following entries are missing: " + f'{", ".join(desc_without_id)}' + ) + + # If we have events, convert them to Annotations so they can be easily + # merged with existing Annotations. + if events.size > 0: + ids_without_desc = set(events[:, 2]) - set(event_id.values()) + if ids_without_desc: + raise ValueError( + f"No description was specified for the following event(s): " + f'{", ".join([str(x) for x in sorted(ids_without_desc)])}. ' + f"Please add them to the event_id dictionary, or drop them " + f"from the events array." + ) + + # Append events to raw.annotations. All event onsets are relative to + # measurement beginning. + id_to_desc_map = dict(zip(event_id.values(), event_id.keys())) + # We don't pass `first_samp`, as set_annotations() below will take + # care of this shift automatically. + new_annotations = mne.annotations_from_events( + events=events, + sfreq=raw.info["sfreq"], + event_desc=id_to_desc_map, + orig_time=raw.annotations.orig_time, + ) + + raw = raw.copy() # Don't alter the original. + annotations = raw.annotations.copy() + + # We use `+=` here because `Annotations.__iadd__()` does the right + # thing and also performs a sanity check on `Annotations.orig_time`. + annotations += new_annotations + raw.set_annotations(annotations) + del id_to_desc_map, annotations, new_annotations + + # Now convert the Annotations to events. + all_events, all_desc = events_from_annotations( + raw, + event_id=event_id, + regexp=None, # Include `BAD_` and `EDGE_` Annotations, too. + ) + all_dur = raw.annotations.duration + + # Warn about missing events if not rest or empty-room data + if (all_events.size == 0 and bids_path.task is not None) and ( + not bids_path.task.startswith("rest") + or not (bids_path.subject == "emptyroom" and bids_path.task == "noise") + ): + warn( + "No events found or provided. Please add annotations to the raw " + "data, or provide the events and event_id parameters. For " + "resting state data, BIDS recommends naming the task using " + 'labels beginning with "rest".' + ) + + return all_events, all_dur, all_desc + + +def _verbose_list_index(lst, val, *, allow_all=False): + # try to "return lst.index(val)" for list of str, but be more + # informative/verbose when it fails + try: + return lst.index(val) + except ValueError as exc: + # Use str cast here to deal with pathlib.Path instances + extra = get_close_matches(str(val), [str(ll) for ll in lst]) + if allow_all and not extra: + extra = lst + extra = f". Did you mean one of {extra}?" if extra else "" + raise ValueError(f"{exc}{extra}") from None + + +def _handle_participants_reading(participants_fname, raw, subject): + participants_tsv = _from_tsv(participants_fname) + subjects = participants_tsv["participant_id"] + row_ind = _verbose_list_index(subjects, subject, allow_all=True) + raw.info["subject_info"] = dict() # start from scratch + + # set data from participants tsv into subject_info + for col_name, value in participants_tsv.items(): + if col_name in ("sex", "hand"): + value = _map_options( + what=col_name, key=value[row_ind], fro="bids", to="mne" + ) + # We don't know how to translate to MNE, so skip. + if value is None: + if col_name == "sex": + info_str = "subject sex" + else: + info_str = "subject handedness" + warn( + f'Unable to map "{col_name}" value "{value}" to MNE. ' + f"Not setting {info_str}." + ) + elif col_name in ("height", "weight"): + try: + value = float(value[row_ind]) + except ValueError: + value = None + else: + if value[row_ind] == "n/a": + value = None + else: + value = value[row_ind] + + # add data into raw.Info + key = "his_id" if col_name == "participant_id" else col_name + if value is not None: + assert key not in raw.info["subject_info"] + raw.info["subject_info"][key] = value + + return raw + + +def _handle_scans_reading(scans_fname, raw, bids_path): + """Read associated scans.tsv and set meas_date.""" + scans_tsv = _from_tsv(scans_fname) + fname = bids_path.fpath.name + + if fname.endswith(".pdf"): + # for BTi files, the scan is an entire directory + fname = fname.split(".")[0] + + # get the row corresponding to the file + # use string concatenation instead of os.path + # to work nicely with windows + data_fname = Path(bids_path.datatype) / fname + fnames = scans_tsv["filename"] + fnames = [Path(fname) for fname in fnames] + if "acq_time" in scans_tsv: + acq_times = scans_tsv["acq_time"] + else: + acq_times = ["n/a"] * len(fnames) + + # There are three possible extensions for BrainVision + # First gather all the possible extensions + acq_suffixes = set(fname.suffix for fname in fnames) + # Add the filename extension for the bids folder + acq_suffixes.add(Path(data_fname).suffix) + + if all(suffix in (".vhdr", ".eeg", ".vmrk") for suffix in acq_suffixes): + ext = fnames[0].suffix + data_fname = Path(data_fname).with_suffix(ext) + row_ind = _verbose_list_index(fnames, data_fname) + + # check whether all split files have the same acq_time + # and throw an error if they don't + if "_split-" in fname: + split_idx = fname.find("split-") + pattern = re.compile( + bids_path.datatype + + "/" + + bids_path.basename[:split_idx] + + r"split-\d+_" + + bids_path.datatype + + bids_path.fpath.suffix + ) + split_fnames = list(filter(lambda x: pattern.match(x.as_posix()), fnames)) + split_acq_times = [] + for split_f in split_fnames: + split_acq_times.append(acq_times[_verbose_list_index(fnames, split_f)]) + if len(set(split_acq_times)) != 1: + raise ValueError("Split files must have the same acq_time.") + + # extract the acquisition time from scans file + acq_time = acq_times[row_ind] + if acq_time != "n/a": + # BIDS allows the time to be stored in UTC with a zero time-zone offset, as + # indicated by a trailing "Z" in the datetime string. If the "Z" is missing, the + # time is represented as "local" time. We have no way to know what the local + # time zone is at the *acquisition* site; so we simply assume the same time zone + # as the user's current system (this is what the spec demands anyway). + acq_time_is_utc = acq_time.endswith("Z") + + # microseconds part in the acquisition time is optional; add it if missing + if "." not in acq_time: + if acq_time_is_utc: + acq_time = acq_time.replace("Z", ".0Z") + else: + acq_time += ".0" + + date_format = "%Y-%m-%dT%H:%M:%S.%f" + if acq_time_is_utc: + date_format += "Z" + + acq_time = datetime.strptime(acq_time, date_format) + + if acq_time_is_utc: + # Enforce setting timezone to UTC without additonal conversion + acq_time = acq_time.replace(tzinfo=timezone.utc) + else: + # Convert time offset to UTC + acq_time = acq_time.astimezone(timezone.utc) + + logger.debug( + f"Loaded {scans_fname} scans file to set " f"acq_time as {acq_time}." + ) + # First set measurement date to None and then call call anonymize() to + # remove any traces of the measurement date we wish + # to replace – it might lurk out in more places than just + # raw.info['meas_date'], e.g. in info['meas_id]['secs'] and in + # info['file_id'], which are not affected by set_meas_date(). + # The combined use of set_meas_date(None) and anonymize() is suggested + # by the MNE documentation, and in fact we cannot load e.g. OpenNeuro + # ds003392 without this combination. + raw.set_meas_date(None) + raw.anonymize(daysback=None, keep_his=True) + raw.set_meas_date(acq_time) + + return raw + + +def _handle_info_reading(sidecar_fname, raw): + """Read associated sidecar JSON and populate raw. + + Handle PowerLineFrequency of recording. + """ + with open(sidecar_fname, encoding="utf-8-sig") as fin: + sidecar_json = json.load(fin) + + # read in the sidecar JSON's and raw object's line frequency + json_linefreq = sidecar_json.get("PowerLineFrequency") + raw_linefreq = raw.info["line_freq"] + + # If both are defined, warn if there is a conflict, else all is fine + if (json_linefreq is not None) and (raw_linefreq is not None): + if json_linefreq != raw_linefreq: + msg = ( + f"Line frequency in sidecar JSON does not match the info " + f"data structure of the mne.Raw object:\n" + f"Sidecar JSON is -> {json_linefreq}\n" + f"Raw is -> {raw_linefreq}\n\n" + ) + + if json_linefreq == "n/a": + msg += "Defaulting to the info from mne.Raw object." + raw.info["line_freq"] = raw_linefreq + else: + msg += "Defaulting to the info from sidecar JSON." + raw.info["line_freq"] = json_linefreq + + warn(msg) + + # Else, try to use JSON, fall back on mne.Raw + elif (json_linefreq is not None) and (json_linefreq != "n/a"): + raw.info["line_freq"] = json_linefreq + else: + pass # line freq is either defined or None in mne.Raw + + # get cHPI info + chpi = sidecar_json.get("ContinuousHeadLocalization") + if chpi is None: + # no cHPI info in the sidecar – leave raw.info unchanged + pass + elif chpi is True: + from mne.io.ctf import RawCTF + from mne.io.kit.kit import RawKIT + + msg = ( + "Cannot verify that the cHPI frequencies from " + "the MEG JSON sidecar file correspond to the raw data{}" + ) + + if isinstance(raw, RawCTF): + # Pick channels corresponding to the cHPI positions + hpi_picks = pick_channels_regexp(raw.info["ch_names"], "HLC00[123][123].*") + if len(hpi_picks) != 9: + raise ValueError( + f"Could not find all cHPI channels that we expected for " + f"CTF data. Expected: 9, found: {len(hpi_picks)}" + ) + logger.info(msg.format(" for CTF files.")) + + elif isinstance(raw, RawKIT): + logger.info(msg.format(" for KIT files.")) + + elif "HeadCoilFrequency" in sidecar_json: + hpi_freqs_json = sidecar_json["HeadCoilFrequency"] + try: + hpi_freqs_raw, _, _ = mne.chpi.get_chpi_info(raw.info) + except ValueError: + logger.info(msg.format(".")) + else: + # XXX: Set chpi info in mne.Raw to what is in the sidecar + if not np.allclose(hpi_freqs_json, hpi_freqs_raw): + warn( + f"The cHPI coil frequencies in the sidecar file " + f"{sidecar_fname}:\n {hpi_freqs_json}\n " + f"differ from what is stored in the raw data:\n" + f" {hpi_freqs_raw}.\n" + f"Defaulting to the info from mne.Raw object." + ) + else: + addmsg = ( + ".\n(Because no 'HeadCoilFrequency' data was found in the sidecar.)" + ) + logger.info(msg.format(addmsg)) + + else: + if raw.info["hpi_subsystem"]: + logger.info( + "Dropping cHPI information stored in raw data, " + "following specification in sidecar file" + ) + with raw.info._unlock(): + raw.info["hpi_subsystem"] = None + raw.info["hpi_meas"] = [] + + return raw + + +def _handle_events_reading(events_fname, raw): + """Read associated events.tsv and populate raw. + + Handle onset, duration, and description of each event. + """ + logger.info(f"Reading events from {events_fname}.") + events_dict = _from_tsv(events_fname) + + # Get the descriptions of the events + if "trial_type" in events_dict: + trial_type_col_name = "trial_type" + elif "stim_type" in events_dict: # Backward-compat with old datasets. + trial_type_col_name = "stim_type" + warn( + f'The events file, {events_fname}, contains a "stim_type" ' + f'column. This column should be renamed to "trial_type" for ' + f"BIDS compatibility." + ) + else: + trial_type_col_name = None + + if trial_type_col_name is not None: + # Drop events unrelated to a trial type + events_dict = _drop(events_dict, "n/a", trial_type_col_name) + + if "value" in events_dict: + # Check whether the `trial_type` <> `value` mapping is unique. + trial_types = events_dict[trial_type_col_name] + values = np.asarray(events_dict["value"], dtype=str) + for trial_type in np.unique(trial_types): + idx = np.where(trial_type == np.atleast_1d(trial_types))[0] + matching_values = values[idx] + + if len(np.unique(matching_values)) > 1: + # Event type descriptors are ambiguous; create hierarchical + # event descriptors. + logger.info( + f'The event "{trial_type}" refers to multiple event ' + f"values. Creating hierarchical event names." + ) + for ii in idx: + value = values[ii] + value = "na" if value == "n/a" else value + new_name = f"{trial_type}/{value}" + logger.info( + f" Renaming event: {trial_type} -> " f"{new_name}" + ) + trial_types[ii] = new_name + descriptions = np.asarray(trial_types, dtype=str) + else: + descriptions = np.asarray(events_dict[trial_type_col_name], dtype=str) + elif "value" in events_dict: + # If we don't have a proper description of the events, perhaps we have + # at least an event value? + # Drop events unrelated to value + events_dict = _drop(events_dict, "n/a", "value") + descriptions = np.asarray(events_dict["value"], dtype=str) + + # Worst case, we go with 'n/a' for all events + else: + descriptions = np.array(["n/a"] * len(events_dict["onset"]), dtype=str) + + # Deal with "n/a" strings before converting to float + onsets = np.array( + [np.nan if on == "n/a" else on for on in events_dict["onset"]], dtype=float + ) + durations = np.array( + [0 if du == "n/a" else du for du in events_dict["duration"]], dtype=float + ) + + # Keep only events where onset is known + good_events_idx = ~np.isnan(onsets) + onsets = onsets[good_events_idx] + durations = durations[good_events_idx] + descriptions = descriptions[good_events_idx] + del good_events_idx + + # Add events as Annotations, but keep essential Annotations present in + # raw file + annot_from_raw = raw.annotations.copy() + + annot_from_events = mne.Annotations( + onset=onsets, duration=durations, description=descriptions + ) + raw.set_annotations(annot_from_events) + + annot_idx_to_keep = [ + idx + for idx, descr in enumerate(annot_from_raw.description) + if descr in ANNOTATIONS_TO_KEEP + ] + annot_to_keep = annot_from_raw[annot_idx_to_keep] + + if len(annot_to_keep): + raw.set_annotations(raw.annotations + annot_to_keep) + + return raw + + +def _get_bads_from_tsv_data(tsv_data): + """Extract names of bads from data read from channels.tsv.""" + idx = [] + for ch_idx, status in enumerate(tsv_data["status"]): + if status.lower() == "bad": + idx.append(ch_idx) + + bads = [tsv_data["name"][i] for i in idx] + return bads + + +def _handle_channels_reading(channels_fname, raw): + """Read associated channels.tsv and populate raw. + + Updates status (bad) and types of channels. + """ + logger.info(f"Reading channel info from {channels_fname}.") + channels_dict = _from_tsv(channels_fname) + ch_names_tsv = channels_dict["name"] + + # Now we can do some work. + # The "type" column is mandatory in BIDS. We can use it to set channel + # types in the raw data using a mapping between channel types + channel_type_bids_mne_map = dict() + + # Get the best mapping we currently have from BIDS to MNE nomenclature + bids_to_mne_ch_types = _get_ch_type_mapping(fro="bids", to="mne") + ch_types_json = channels_dict["type"] + for ch_name, ch_type in zip(ch_names_tsv, ch_types_json): + # We don't map MEG channels for now, as there's no clear 1:1 mapping + # from BIDS to MNE coil types. + if ch_type.upper() in ( + "MEGGRADAXIAL", + "MEGMAG", + "MEGREFGRADAXIAL", + "MEGGRADPLANAR", + "MEGREFMAG", + "MEGOTHER", + ): + continue + + # Try to map from BIDS nomenclature to MNE, leave channel type + # untouched if we are uncertain + updated_ch_type = bids_to_mne_ch_types.get(ch_type, None) + + if updated_ch_type is None: + # XXX Try again with uppercase spelling – this should be removed + # XXX once https://github.com/bids-standard/bids-validator/issues/1018 # noqa:E501 + # XXX has been resolved. + # XXX x-ref https://github.com/mne-tools/mne-bids/issues/481 + updated_ch_type = bids_to_mne_ch_types.get(ch_type.upper(), None) + if updated_ch_type is not None: + msg = ( + "The BIDS dataset contains channel types in lowercase " + "spelling. This violates the BIDS specification and " + "will raise an error in the future." + ) + warn(msg) + + if updated_ch_type is None: + # We don't have an appropriate mapping, so make it a "misc" channel + channel_type_bids_mne_map[ch_name] = "misc" + warn( + f'No BIDS -> MNE mapping found for channel type "{ch_type}". ' + f'Type of channel "{ch_name}" will be set to "misc".' + ) + else: + # We found a mapping, so use it + channel_type_bids_mne_map[ch_name] = updated_ch_type + + # Special handling for (synthesized) stimulus channel + synthesized_stim_ch_name = "STI 014" + if ( + synthesized_stim_ch_name in raw.ch_names + and synthesized_stim_ch_name not in ch_names_tsv + ): + logger.info( + f'The stimulus channel "{synthesized_stim_ch_name}" is present in ' + f"the raw data, but not included in channels.tsv. Removing the " + f"channel." + ) + raw.drop_channels([synthesized_stim_ch_name]) + + # Rename channels in loaded Raw to match those read from the BIDS sidecar + if len(ch_names_tsv) != len(raw.ch_names): + warn( + f"The number of channels in the channels.tsv sidecar file " + f"({len(ch_names_tsv)}) does not match the number of channels " + f"in the raw data file ({len(raw.ch_names)}). Will not try to " + f"set channel names." + ) + else: + raw.rename_channels(dict(zip(raw.ch_names, ch_names_tsv))) + + # Set the channel types in the raw data according to channels.tsv + channel_type_bids_mne_map_available_channels = { + ch_name: ch_type + for ch_name, ch_type in channel_type_bids_mne_map.items() + if ch_name in raw.ch_names + } + ch_diff = set(channel_type_bids_mne_map.keys()) - set( + channel_type_bids_mne_map_available_channels.keys() + ) + if ch_diff: + warn( + f"Cannot set channel type for the following channels, as they " + f'are missing in the raw data: {", ".join(sorted(ch_diff))}' + ) + raw.set_channel_types(channel_type_bids_mne_map_available_channels) + + # Set bad channels based on _channels.tsv sidecar + if "status" in channels_dict: + bads_tsv = _get_bads_from_tsv_data(channels_dict) + bads_avail = [ch_name for ch_name in bads_tsv if ch_name in raw.ch_names] + + ch_diff = set(bads_tsv) - set(bads_avail) + if ch_diff: + warn( + f'Cannot set "bad" status for the following channels, as ' + f"they are missing in the raw data: " + f'{", ".join(sorted(ch_diff))}' + ) + + raw.info["bads"] = bads_avail + + return raw + + +@verbose +def read_raw_bids(bids_path, extra_params=None, verbose=None): + """Read BIDS compatible data. + + Will attempt to read associated events.tsv and channels.tsv files to + populate the returned raw object with raw.annotations and raw.info['bads']. + + Parameters + ---------- + bids_path : BIDSPath + The file to read. The :class:`mne_bids.BIDSPath` instance passed here + **must** have the ``.root`` attribute set. The ``.datatype`` attribute + **may** be set. If ``.datatype`` is not set and only one data type + (e.g., only EEG or MEG data) is present in the dataset, it will be + selected automatically. + + .. note:: + If ``bids_path`` points to a symbolic link of a ``.fif`` file + without a ``split`` entity, the link will be resolved before + reading. + + extra_params : None | dict + Extra parameters to be passed to MNE read_raw_* functions. + Note that the ``exclude`` parameter, which is supported by some + MNE-Python readers, is not supported; instead, you need to subset + your channels **after** reading. + %(verbose)s + + Returns + ------- + raw : mne.io.Raw + The data as MNE-Python Raw object. + + Raises + ------ + RuntimeError + If multiple recording data types are present in the dataset, but + ``datatype=None``. + + RuntimeError + If more than one data files exist for the specified recording. + + RuntimeError + If no data file in a supported format can be located. + + ValueError + If the specified ``datatype`` cannot be found in the dataset. + + """ + if not isinstance(bids_path, BIDSPath): + raise RuntimeError( + '"bids_path" must be a BIDSPath object. Please ' + "instantiate using mne_bids.BIDSPath()." + ) + + bids_path = bids_path.copy() + sub = bids_path.subject + ses = bids_path.session + bids_root = bids_path.root + datatype = bids_path.datatype + suffix = bids_path.suffix + + # check root available + if bids_root is None: + raise ValueError( + 'The root of the "bids_path" must be set. ' + 'Please use `bids_path.update(root="")` ' + "to set the root of the BIDS folder to read." + ) + + # infer the datatype and suffix if they are not present in the BIDSPath + if datatype is None: + datatype = _infer_datatype(root=bids_root, sub=sub, ses=ses) + bids_path.update(datatype=datatype) + if suffix is None: + bids_path.update(suffix=datatype) + + if bids_path.fpath.suffix == ".pdf": + bids_raw_folder = bids_path.directory / f"{bids_path.basename}" + + # try to find the processed data file ("pdf") + # see: https://www.fieldtriptoolbox.org/getting_started/bti/ + bti_pdf_patterns = ["0", "c,rf*", "hc,rf*", "e,rf*"] + pdf_list = [] + for pattern in bti_pdf_patterns: + pdf_list += sorted(bids_raw_folder.glob(pattern)) + + if len(pdf_list) == 0: + raise RuntimeError( + "Cannot find BTi 'processed data file' (pdf). Please open an issue on " + "the mne-bids repository to discuss with the developers:\n\n" + "https://github.com/mne-tools/mne-bids/issues/new/choose\n\n" + f"No matches for following patterns:\n\n{bti_pdf_patterns}\n\n" + f"In: {bids_raw_folder}" + ) + elif len(pdf_list) > 1: # pragma: no cover + logger.warn( + "Found more than one BTi 'processed data file' (pdf). " + f"Picking:\n\n {pdf_list[0]}\n\nout of the options:\n\n" + f"{pdf_list}\n\n" + ) + raw_path = pdf_list[0] + config_path = bids_raw_folder / "config" + else: + raw_path = bids_path.fpath + # Resolve for FIFF files + if ( + raw_path.suffix == ".fif" + and bids_path.split is None + and raw_path.is_symlink() + ): + target_path = raw_path.resolve() + logger.info(f"Resolving symbolic link: " f"{raw_path} -> {target_path}") + raw_path = target_path + config_path = None + + # Special-handle EDF filenames: we accept upper- and lower-case extensions + if raw_path.suffix.lower() == ".edf": + for extension in (".edf", ".EDF"): + candidate_path = raw_path.with_suffix(extension) + if candidate_path.exists(): + raw_path = candidate_path + break + + if not raw_path.exists(): + options = os.listdir(bids_path.directory) + matches = get_close_matches(bids_path.basename, options) + msg = f"File does not exist:\n{raw_path}" + if matches: + msg += ( + "\nDid you mean one of:\n" + + "\n".join(matches) + + "\ninstead of:\n" + + bids_path.basename + ) + raise FileNotFoundError(msg) + if config_path is not None and not config_path.exists(): + raise FileNotFoundError(f"config directory not found: {config_path}") + + if extra_params is None: + extra_params = dict() + elif "exclude" in extra_params: + del extra_params["exclude"] + logger.info('"exclude" parameter is not supported by read_raw_bids') + + if raw_path.suffix == ".fif" and "allow_maxshield" not in extra_params: + extra_params["allow_maxshield"] = True + raw = _read_raw( + raw_path, + electrode=None, + hsp=None, + hpi=None, + config_path=config_path, + **extra_params, + ) + + # Try to find an associated events.tsv to get information about the + # events in the recorded data + if ( + bids_path.subject == "emptyroom" and bids_path.task == "noise" + ) or bids_path.task.startswith("rest"): + on_error = "ignore" + else: + on_error = "warn" + + events_fname = _find_matching_sidecar( + bids_path, suffix="events", extension=".tsv", on_error=on_error + ) + + if events_fname is not None: + raw = _handle_events_reading(events_fname, raw) + + # Try to find an associated channels.tsv to get information about the + # status and type of present channels + channels_fname = _find_matching_sidecar( + bids_path, suffix="channels", extension=".tsv", on_error="warn" + ) + if channels_fname is not None: + raw = _handle_channels_reading(channels_fname, raw) + + # Try to find an associated electrodes.tsv and coordsystem.json + # to get information about the status and type of present channels + on_error = "warn" if suffix == "ieeg" else "ignore" + electrodes_fname = _find_matching_sidecar( + bids_path, suffix="electrodes", extension=".tsv", on_error=on_error + ) + coordsystem_fname = _find_matching_sidecar( + bids_path, suffix="coordsystem", extension=".json", on_error=on_error + ) + if electrodes_fname is not None: + if coordsystem_fname is None: + raise RuntimeError( + f"BIDS mandates that the coordsystem.json " + f"should exist if electrodes.tsv does. " + f"Please create coordsystem.json for" + f"{bids_path.basename}" + ) + if datatype in ["meg", "eeg", "ieeg"]: + _read_dig_bids( + electrodes_fname, coordsystem_fname, raw=raw, datatype=datatype + ) + + # Try to find an associated sidecar .json to get information about the + # recording snapshot + sidecar_fname = _find_matching_sidecar( + bids_path, suffix=datatype, extension=".json", on_error="warn" + ) + if sidecar_fname is not None: + raw = _handle_info_reading(sidecar_fname, raw) + + # read in associated scans filename + scans_fname = BIDSPath( + subject=bids_path.subject, + session=bids_path.session, + suffix="scans", + extension=".tsv", + root=bids_path.root, + ).fpath + + if scans_fname.exists(): + raw = _handle_scans_reading(scans_fname, raw, bids_path) + + # read in associated subject info from participants.tsv + participants_tsv_path = bids_root / "participants.tsv" + subject = f"sub-{bids_path.subject}" + if op.exists(participants_tsv_path): + raw = _handle_participants_reading( + participants_fname=participants_tsv_path, raw=raw, subject=subject + ) + else: + warn(f"participants.tsv file not found for {raw_path}") + raw.info["subject_info"] = dict() + + assert raw.annotations.orig_time == raw.info["meas_date"] + return raw + + +@verbose +def get_head_mri_trans( + bids_path, + extra_params=None, + t1_bids_path=None, + fs_subject=None, + fs_subjects_dir=None, + *, + kind=None, + verbose=None, +): + """Produce transformation matrix from MEG and MRI landmark points. + + Will attempt to read the landmarks of Nasion, LPA, and RPA from the sidecar + files of (i) the MEG and (ii) the T1-weighted MRI data. The two sets of + points will then be used to calculate a transformation matrix from head + coordinates to MRI coordinates. + + .. note:: The MEG and MRI data need **not** necessarily be stored in the + same session or even in the same BIDS dataset. See the + ``t1_bids_path`` parameter for details. + + Parameters + ---------- + bids_path : BIDSPath + The path of the electrophysiology recording. If ``datatype`` and + ``suffix`` are not present, they will be set to ``'meg'``, and a + warning will be raised. + + .. versionchanged:: 0.10 + A warning is raised it ``datatype`` or ``suffix`` are not set. + extra_params : None | dict + Extra parameters to be passed to :func:`mne.io.read_raw` when reading + the MEG file. + t1_bids_path : BIDSPath | None + If ``None`` (default), will try to discover the T1-weighted MRI file + based on the name and location of the MEG recording specified via the + ``bids_path`` parameter. Alternatively, you explicitly specify which + T1-weighted MRI scan to use for extraction of MRI landmarks. To do + that, pass a :class:`mne_bids.BIDSPath` pointing to the scan. + Use this parameter e.g. if the T1 scan was recorded during a different + session than the MEG. It is even possible to point to a T1 image stored + in an entirely different BIDS dataset than the MEG data. + fs_subject : str + The subject identifier used for FreeSurfer. + + .. versionchanged:: 0.10 + Does not default anymore to ``bids_path.subject`` if ``None``. + fs_subjects_dir : path-like | None + The FreeSurfer subjects directory. If ``None``, defaults to the + ``SUBJECTS_DIR`` environment variable. + + .. versionadded:: 0.8 + kind : str | None + The suffix of the anatomical landmark names in the JSON sidecar. + A suffix might be present e.g. to distinguish landmarks between + sessions. If provided, should not include a leading underscore ``_``. + For example, if the landmark names in the JSON sidecar file are + ``LPA_ses-1``, ``RPA_ses-1``, ``NAS_ses-1``, you should pass + ``'ses-1'`` here. + If ``None``, no suffix is appended, the landmarks named + ``Nasion`` (or ``NAS``), ``LPA``, and ``RPA`` will be used. + + .. versionadded:: 0.10 + %(verbose)s + + Returns + ------- + trans : mne.transforms.Transform + The data transformation matrix from head to MRI coordinates. + """ + nib = _import_nibabel("get a head to MRI transform") + + if not isinstance(bids_path, BIDSPath): + raise RuntimeError( + '"bids_path" must be a BIDSPath object. Please ' + "instantiate using mne_bids.BIDSPath()." + ) + + # check root available + meg_bids_path = bids_path.copy() + del bids_path + if meg_bids_path.root is None: + raise ValueError( + 'The root of the "bids_path" must be set. ' + 'Please use `bids_path.update(root="")` ' + "to set the root of the BIDS folder to read." + ) + + # if the bids_path is underspecified, only get info for MEG data + if meg_bids_path.datatype is None: + meg_bids_path.datatype = "meg" + warn( + 'bids_path did not have a datatype set. Assuming "meg". This ' + "will raise an exception in the future.", + module="mne_bids", + category=DeprecationWarning, + ) + if meg_bids_path.suffix is None: + meg_bids_path.suffix = "meg" + warn( + 'bids_path did not have a suffix set. Assuming "meg". This ' + "will raise an exception in the future.", + module="mne_bids", + category=DeprecationWarning, + ) + + # Get the sidecar file for MRI landmarks + t1w_bids_path = ( + (meg_bids_path if t1_bids_path is None else t1_bids_path) + .copy() + .update(datatype="anat", suffix="T1w", task=None) + ) + t1w_json_path = _find_matching_sidecar( + bids_path=t1w_bids_path, extension=".json", on_error="ignore" + ) + del t1_bids_path + + if t1w_json_path is not None: + t1w_json_path = Path(t1w_json_path) + + if t1w_json_path is None or not t1w_json_path.exists(): + raise FileNotFoundError( + f"Did not find T1w JSON sidecar file, tried location: " f"{t1w_json_path}" + ) + for extension in (".nii", ".nii.gz"): + t1w_path_candidate = t1w_json_path.with_suffix(extension) + if t1w_path_candidate.exists(): + t1w_bids_path = get_bids_path_from_fname(fname=t1w_path_candidate) + break + + if not t1w_bids_path.fpath.exists(): + raise FileNotFoundError( + f"Did not find T1w recording file, tried location: " + f'{t1w_path_candidate.name.replace(".nii.gz", "")}[.nii, .nii.gz]' + ) + + # Get MRI landmarks from the JSON sidecar + t1w_json = json.loads(t1w_json_path.read_text(encoding="utf-8")) + mri_coords_dict = t1w_json.get("AnatomicalLandmarkCoordinates", dict()) + + # landmarks array: rows: [LPA, NAS, RPA]; columns: [x, y, z] + suffix = f"_{kind}" if kind is not None else "" + mri_landmarks = np.full((3, 3), np.nan) + for landmark_name, coords in mri_coords_dict.items(): + if landmark_name.upper() == ("LPA" + suffix).upper(): + mri_landmarks[0, :] = coords + elif landmark_name.upper() == ("RPA" + suffix).upper(): + mri_landmarks[2, :] = coords + elif ( + landmark_name.upper() == ("NAS" + suffix).upper() + or landmark_name.lower() == ("nasion" + suffix).lower() + ): + mri_landmarks[1, :] = coords + else: + continue + + if np.isnan(mri_landmarks).any(): + raise RuntimeError( + f"Could not extract fiducial points from T1w sidecar file: " + f"{t1w_json_path}\n\n" + f"The sidecar file SHOULD contain a key " + f'"AnatomicalLandmarkCoordinates" pointing to an ' + f'object with the keys "LPA", "NAS", and "RPA". ' + f"Yet, the following structure was found:\n\n" + f"{mri_coords_dict}" + ) + + # The MRI landmarks are in "voxels". We need to convert them to the + # Neuromag RAS coordinate system in order to compare them with MEG + # landmarks. See also: `mne_bids.write.write_anat` + if fs_subject is None: + warn( + 'Passing "fs_subject=None" has been deprecated and will raise ' + "an error in future versions. Please explicitly specify the " + "FreeSurfer subject name.", + DeprecationWarning, + ) + fs_subject = f"sub-{meg_bids_path.subject}" + + fs_subjects_dir = get_subjects_dir(fs_subjects_dir, raise_error=False) + fs_t1_path = Path(fs_subjects_dir) / fs_subject / "mri" / "T1.mgz" + if not fs_t1_path.exists(): + raise ValueError( + f"Could not find {fs_t1_path}. Consider running FreeSurfer's " + f"'recon-all` for subject {fs_subject}." + ) + fs_t1_mgh = nib.load(str(fs_t1_path)) + t1_nifti = nib.load(str(t1w_bids_path.fpath)) + + # Convert to MGH format to access vox2ras method + t1_mgh = nib.MGHImage(t1_nifti.dataobj, t1_nifti.affine) + + # convert to scanner RAS + mri_landmarks = apply_trans(t1_mgh.header.get_vox2ras(), mri_landmarks) + + # convert to FreeSurfer T1 voxels (same scanner RAS as T1) + mri_landmarks = apply_trans(fs_t1_mgh.header.get_ras2vox(), mri_landmarks) + + # now extract transformation matrix and put back to RAS coordinates of MRI + vox2ras_tkr = fs_t1_mgh.header.get_vox2ras_tkr() + mri_landmarks = apply_trans(vox2ras_tkr, mri_landmarks) + mri_landmarks = mri_landmarks * 1e-3 + + # Get MEG landmarks from the raw file + _, ext = _parse_ext(meg_bids_path) + if extra_params is None: + extra_params = dict() + if ext == ".fif": + extra_params["allow_maxshield"] = "yes" + + raw = read_raw_bids(bids_path=meg_bids_path, extra_params=extra_params) + + if ( + raw.get_montage() is None + or raw.get_montage().get_positions() is None + or any( + [ + raw.get_montage().get_positions()[fid_key] is None + for fid_key in ("nasion", "lpa", "rpa") + ] + ) + ): + raise RuntimeError( + f"Could not extract fiducial points from ``raw`` file: " + f"{meg_bids_path}\n\n" + f"The ``raw`` file SHOULD contain digitization points " + "for the nasion and left and right pre-auricular points " + "but none were found" + ) + pos = raw.get_montage().get_positions() + meg_landmarks = np.asarray((pos["lpa"], pos["nasion"], pos["rpa"])) + + # Given the two sets of points, fit the transform + trans_fitted = fit_matched_points(src_pts=meg_landmarks, tgt_pts=mri_landmarks) + trans = mne.transforms.Transform(fro="head", to="mri", trans=trans_fitted) + return trans diff --git a/mne-bids-0.15/mne_bids/report/__init__.py b/mne-bids-0.15/mne_bids/report/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..47816cbbcf9b6c74f2358702c988d40cc9df0f3e --- /dev/null +++ b/mne-bids-0.15/mne_bids/report/__init__.py @@ -0,0 +1,3 @@ +"""Create a summary report of the BIDS dataset.""" + +from ._report import make_report diff --git a/mne-bids-0.15/mne_bids/report/_report.py b/mne-bids-0.15/mne_bids/report/_report.py new file mode 100644 index 0000000000000000000000000000000000000000..cb681c316c3cbf2e210beaf48368cd19e9c94853 --- /dev/null +++ b/mne-bids-0.15/mne_bids/report/_report.py @@ -0,0 +1,538 @@ +"""Make BIDS report from dataset and sidecar files.""" + +# Authors: Adam Li +# +# License: BSD-3-Clause +import json +import os.path as op +import textwrap +from pathlib import Path + +import jinja2 +import numpy as np +from mne.utils import logger, verbose + +from mne_bids.config import ALLOWED_DATATYPES, DOI +from mne_bids.path import ( + BIDSPath, + _find_matching_sidecar, + _parse_ext, + get_bids_path_from_fname, + get_datatypes, + get_entity_vals, +) +from mne_bids.tsv_handler import _from_tsv +from mne_bids.utils import warn + +jinja_env = jinja2.Environment( + loader=jinja2.PackageLoader( + package_name="mne_bids.report", package_path="templates" + ) +) + + +def _pretty_str(listed): + # make strings a sequence of ',' and 'and' + if not isinstance(listed, list): + listed = list(listed) + + if len(listed) <= 1: + return ",".join(listed) + return "{}, and {}".format(", ".join(listed[:-1]), listed[-1]) + + +def _range_str(minval, maxval, meanval, stdval, n_unknown, type): + if minval == "n/a": + return "ages all unknown" + + if n_unknown > 0: + unknown_str = f"; {n_unknown} with unknown {type}" + else: + unknown_str = "" + return ( + f"ages ranged from {round(minval, 2)} to {round(maxval, 2)} " + f"(mean = {round(meanval, 2)}, std = {round(stdval, 2)}{unknown_str})" + ) + + +def _summarize_participant_hand(hands): + n_unknown = len([hand for hand in hands if hand == "n/a"]) + + if n_unknown == len(hands): + return "handedness were all unknown" + + n_rhand = len([hand for hand in hands if hand.upper() == "R"]) + n_lhand = len([hand for hand in hands if hand.upper() == "L"]) + n_ambidex = len([hand for hand in hands if hand.upper() == "A"]) + + return ( + f"comprised of {n_rhand} right hand, {n_lhand} left hand " + f"and {n_ambidex} ambidextrous" + ) + + +def _summarize_participant_sex(sexs): + n_unknown = len([sex for sex in sexs if sex == "n/a"]) + + if n_unknown == len(sexs): + return "sex were all unknown" + + n_males = len([sex for sex in sexs if sex.upper() == "M"]) + n_females = len([sex for sex in sexs if sex.upper() == "F"]) + + return f"comprised of {n_males} male and {n_females} female participants" + + +def _length_recording_str(length_recordings): + import numpy as np + + if length_recordings is None: + return "" + + min_record_length = round(np.min(length_recordings), 2) + max_record_length = round(np.max(length_recordings), 2) + mean_record_length = round(np.mean(length_recordings), 2) + std_record_length = round(np.std(length_recordings), 2) + total_record_length = round(sum(length_recordings), 2) + + return ( + f"Recording durations ranged from {min_record_length} to " + f"{max_record_length} seconds " + f"(mean = {mean_record_length}, std = {std_record_length}), " + f"for a total of {total_record_length} seconds of data recorded " + f"over all scans." + ) + + +def _summarize_software_filters(software_filters): + if software_filters in [{}, "n/a"]: + return "" + + msg = "" + for key, value in software_filters.items(): + msg += f"{key}" + + if isinstance(value, dict) and value: + parameters = [] + for param_name, param_value in value.items(): + if param_name and param_value: + parameters.append(f"{param_value} {param_name}") + if parameters: + msg += " with parameters " + msg += ", ".join(parameters) + return msg + + +def _pretty_dict(template_dict): + """Remove problematic blank spaces.""" + for key, val in template_dict.items(): + if val == " ": + template_dict[key] = "n/a" + + +def _summarize_dataset(root): + """Summarize the dataset_desecription.json file. + + Required dataset descriptors include: + - Name + - BIDSVersion + + Added descriptors include: + - Authors + - DOI + + Parameters + ---------- + root : path-like + The path of the root of the BIDS compatible folder. + + Returns + ------- + template_dict : dict + A dictionary of values for various template strings. + """ + dataset_descrip_fpath = op.join(root, "dataset_description.json") + if not op.exists(dataset_descrip_fpath): + return dict() + + # read file and 'REQUIRED' components of it + with open(dataset_descrip_fpath, encoding="utf-8-sig") as fin: + dataset_description = json.load(fin) + + # create dictionary to pass into template string + name = dataset_description["Name"] + bids_version = dataset_description["BIDSVersion"] + authors = dataset_description["Authors"] + template_dict = { + "name": name, + "bids_version": bids_version, + "mne_bids_doi": DOI, + "authors": _pretty_str(authors), + } + _pretty_dict(template_dict) + return template_dict + + +def _summarize_participants_tsv(root): + """Summarize `participants.tsv` file in BIDS root directory. + + Parameters + ---------- + root : path-like + The path of the root of the BIDS compatible folder. + + Returns + ------- + template_dict : dict + A dictionary of values for various template strings. + """ + participants_tsv_fpath = op.join(root, "participants.tsv") + if not op.exists(participants_tsv_fpath): + return dict() + + participants_tsv = _from_tsv(str(participants_tsv_fpath)) + p_ids = participants_tsv["participant_id"] + logger.info(f"Summarizing participants.tsv {participants_tsv_fpath}...") + + # summarize sex count statistics + keys = ["M", "F", "n/a"] + p_sex = participants_tsv.get("sex") + # phrasing works for both sex and gender + p_gender = participants_tsv.get("gender") + sexs = ["n/a"] + if p_sex or p_gender: + # only summarize sex if it conforms to `keys` referenced above + p_sex = p_gender if p_sex is None else p_sex + if all([sex.upper() in keys for sex in p_sex if sex != "n/a"]): + sexs = p_sex + + # summarize hand count statistics + keys = ["R", "L", "A", "n/a"] + p_hands = participants_tsv.get("hand") + hands = ["n/a"] + if p_hands: + # only summarize handedness if it conforms to + # mne-bids handedness + if all([hand.upper() in keys for hand in p_hands if hand != "n/a"]): + hands = p_hands + + # summarize age statistics: mean, std, min, max + p_ages = participants_tsv.get("age") + min_age, max_age = "n/a", "n/a" + mean_age, std_age = "n/a", "n/a" + n_age_unknown = len(p_ages) if p_ages else len(p_ids) + if p_ages: + # only summarize age if they are numerics + if all([age.isnumeric() for age in p_ages if age != "n/a"]): + age_list = [float(age) for age in p_ages if age != "n/a"] + n_age_unknown = len(p_ids) - len(age_list) + if age_list: + min_age, max_age = np.min(age_list), np.max(age_list) + mean_age, std_age = np.mean(age_list), np.std(age_list) + + template_dict = { + "sexs": _summarize_participant_sex(sexs), + "hands": _summarize_participant_hand(hands), + "ages": _range_str(min_age, max_age, mean_age, std_age, n_age_unknown, "age"), + } + return template_dict + + +def _summarize_scans(root, session=None): + """Summarize scans in BIDS root directory. + + Summarizes scans only if there is a *_scans.tsv file. + + Parameters + ---------- + root : path-like + The path of the root of the BIDS compatible folder. + session : str, optional + The session for a item. Corresponds to "ses". + + Returns + ------- + template_dict : dict + A dictionary of values for various template strings. + + """ + root = Path(root) + if session is None: + search_str = "*_scans.tsv" + else: + search_str = f"*ses-{session}" f"*_scans.tsv" + scans_fpaths = list(root.rglob(search_str)) + if len(scans_fpaths) == 0: + warn( + "No *scans.tsv files found. Currently, " + "we do not generate a report without the scans.tsv files." + ) + return dict() + + logger.info(f"Summarizing scans.tsv files {scans_fpaths}...") + + # summarize sidecar.json, channels.tsv template + sidecar_dict = _summarize_sidecar_json(root, scans_fpaths) + channels_dict = _summarize_channels_tsv(root, scans_fpaths) + template_dict = dict() + template_dict.update(**sidecar_dict) + template_dict.update(**channels_dict) + + return template_dict + + +def _summarize_sidecar_json(root, scans_fpaths): + """Summarize scans in BIDS root directory. + + Parameters + ---------- + root : path-like + The path of the root of the BIDS compatible folder. + scans_fpaths : list + A list of all *_scans.tsv files in ``root``. The summary + will occur for all scans listed in the *_scans.tsv files. + + Returns + ------- + template_dict : dict + A dictionary of values for various template strings. + + """ + n_scans = 0 + powerlinefreqs, sfreqs = set(), set() + manufacturers = set() + length_recordings = [] + + # loop through each scan + for scan_fpath in scans_fpaths: + # load in the scans.tsv file + # and read metadata for each scan + scans_tsv = _from_tsv(scan_fpath) + scans = scans_tsv["filename"] + for scan in scans: + # summarize metadata of recordings + bids_path, ext = _parse_ext(scan) + datatype = op.dirname(scan) + if datatype not in ALLOWED_DATATYPES: + continue + + n_scans += 1 + + # convert to BIDSPath + if not isinstance(bids_path, BIDSPath): + bids_path = get_bids_path_from_fname(bids_path) + bids_path.root = root + + # XXX: improve to allow emptyroom + if bids_path.subject == "emptyroom": + continue + + sidecar_fname = _find_matching_sidecar( + bids_path=bids_path, suffix=datatype, extension=".json" + ) + with open(sidecar_fname, encoding="utf-8-sig") as fin: + sidecar_json = json.load(fin) + + # aggregate metadata from each scan + # REQUIRED kwargs + sfreq = sidecar_json["SamplingFrequency"] + powerlinefreq = str(sidecar_json["PowerLineFrequency"]) + software_filters = sidecar_json.get("SoftwareFilters") + if not software_filters: + software_filters = "n/a" + + # RECOMMENDED kwargs + manufacturer = sidecar_json.get("Manufacturer", "n/a") + record_duration = sidecar_json.get("RecordingDuration", "n/a") + + sfreqs.add(str(np.round(sfreq, 2))) + powerlinefreqs.add(str(powerlinefreq)) + if manufacturer != "n/a": + manufacturers.add(manufacturer) + length_recordings.append(record_duration) + + # XXX: length summary is only allowed, if no 'n/a' was found + if any([dur == "n/a" for dur in length_recordings]): + length_recordings = None + + template_dict = { + "n_scans": n_scans, + "manufacturer": _pretty_str(manufacturers), + "sfreq": _pretty_str(sfreqs), + "powerlinefreq": _pretty_str(powerlinefreqs), + "software_filters": _summarize_software_filters(software_filters), + "length_recordings": _length_recording_str(length_recordings), + } + return template_dict + + +def _summarize_channels_tsv(root, scans_fpaths): + """Summarize channels.tsv data in BIDS root directory. + + Currently, summarizes all REQUIRED components of channels + data, and some RECOMMENDED and OPTIONAL components. + + Parameters + ---------- + root : path-like + The path of the root of the BIDS compatible folder. + scans_fpaths : list + A list of all *_scans.tsv files in ``root``. The summary + will occur for all scans listed in the *_scans.tsv files. + + Returns + ------- + template_dict : dict + A dictionary of values for various template strings. + """ + root = Path(root) + + # keep track of channel type, status + ch_status_count = {"bad": [], "good": []} + ch_count = [] + + # loop through each scan + for scan_fpath in scans_fpaths: + # load in the scans.tsv file + # and read metadata for each scan + scans_tsv = _from_tsv(scan_fpath) + scans = scans_tsv["filename"] + for scan in scans: + # summarize metadata of recordings + bids_path, _ = _parse_ext(scan) + datatype = op.dirname(scan) + if datatype not in ["meg", "eeg", "ieeg"]: + continue + + # convert to BIDSPath + if not isinstance(bids_path, BIDSPath): + bids_path = get_bids_path_from_fname(bids_path) + bids_path.root = root + + # XXX: improve to allow emptyroom + if bids_path.subject == "emptyroom": + continue + + channels_fname = _find_matching_sidecar( + bids_path=bids_path, suffix="channels", extension=".tsv" + ) + + # summarize channels.tsv + channels_tsv = _from_tsv(channels_fname) + for status in ch_status_count.keys(): + ch_status = [ch for ch in channels_tsv["status"] if ch == status] + ch_status_count[status].append(len(ch_status)) + ch_count.append(len(channels_tsv["name"])) + + # create summary template strings for status + template_dict = { + "mean_chs": np.mean(ch_count), + "std_chs": np.std(ch_count), + "mean_good_chs": np.mean(ch_status_count["good"]), + "std_good_chs": np.std(ch_status_count["good"]), + "mean_bad_chs": np.mean(ch_status_count["bad"]), + "std_bad_chs": np.std(ch_status_count["bad"]), + } + for key, val in template_dict.items(): + template_dict[key] = round(val, 2) + return template_dict + + +@verbose +def make_report(root, session=None, verbose=None): + """Create a methods paragraph string from BIDS dataset. + + Summarizes the REQUIRED components in the BIDS specification + and also some RECOMMENDED components. Currently, the methods + paragraph summarize the: + + - dataset_description.json file + - (optional) participants.tsv file + - (optional) datatype-agnostic files for (M/I)EEG data, + which reads files from the ``*_scans.tsv`` file. + + Parameters + ---------- + root : path-like + The path of the root of the BIDS compatible folder. + session : str | None + The (optional) session for a item. Corresponds to "ses". + %(verbose)s + + Returns + ------- + paragraph : str + The paragraph wrapped with 80 characters per line + describing the summary of the subjects. + """ + # high level summary + subjects = get_entity_vals(root, entity_key="subject") + sessions = get_entity_vals(root, entity_key="session") + modalities = get_datatypes(root) + + # only summarize allowed modalities (MEG/EEG/iEEG) data + # map them to a pretty looking string + datatype_map = { + "meg": "MEG", + "eeg": "EEG", + "ieeg": "iEEG", + } + modalities = [ + datatype_map[datatype] + for datatype in modalities + if datatype in datatype_map.keys() + ] + + # REQUIRED: dataset_description.json summary + dataset_summary = _summarize_dataset(root) + + # RECOMMENDED: participants summary + participant_summary = _summarize_participants_tsv(root) + + # RECOMMENDED: scans summary + scans_summary = _summarize_scans(root, session=session) + + dataset_agnostic_summary = scans_summary.copy() + dataset_agnostic_summary["system"] = _pretty_str(modalities) + + # turn off 'recommended' report summary + # if files are not available to summarize + if not participant_summary: + participants_info = "" + else: + particpants_info_template = jinja_env.get_template("participants.jinja") + participants_info = particpants_info_template.render(**participant_summary) + logger.info(f"The participant template found: {participants_info}") + + if not scans_summary: + datatype_agnostic_info = "" + else: + datatype_agnostic_template = jinja_env.get_template("datatype_agnostic.jinja") + datatype_agnostic_info = datatype_agnostic_template.render( + **dataset_agnostic_summary + ) + + dataset_summary.update( + { + "n_subjects": len(subjects), + "participants_info": participants_info, + "n_sessions": len(sessions), + "sessions": _pretty_str(sessions), + } + ) + + # XXX: add channel summary for modalities (ieeg, meg, eeg) + # create the content and mne Template + # lower-case templates are "Recommended", + # while upper-case templates are "Required". + + dataset_summary_template = jinja_env.get_template("dataset_summary.jinja") + dataset_summary_info = dataset_summary_template.render(**dataset_summary) + + # Concatenate info and clean the paragraph + paragraph = f"{dataset_summary_info}\n{datatype_agnostic_info}" + paragraph = paragraph.replace("\n", " ") + while " " in paragraph: + paragraph = paragraph.replace(" ", " ") + + return "\n".join(textwrap.wrap(paragraph, width=80)) diff --git a/mne-bids-0.15/mne_bids/report/templates/dataset_summary.jinja b/mne-bids-0.15/mne_bids/report/templates/dataset_summary.jinja new file mode 100644 index 0000000000000000000000000000000000000000..aada6dfd987a687f7af6db3fadb01d6fd04d7499 --- /dev/null +++ b/mne-bids-0.15/mne_bids/report/templates/dataset_summary.jinja @@ -0,0 +1,13 @@ +{% if name == 'n/a' %}This +{% else %} +The {{ name }} +{% endif %} +dataset was created by {{ authors }} and conforms to BIDS version +{{ bids_version }}. This report was generated with +MNE-BIDS ({{ mne_bids_doi }}). The dataset consists of +{{ n_subjects }} participants ({{ participants_info }}) +{% if n_sessions %} +and {{ n_sessions }} recording sessions: +{{ sessions }}. +{% else %}. +{% endif %} diff --git a/mne-bids-0.15/mne_bids/report/templates/datatype_agnostic.jinja b/mne-bids-0.15/mne_bids/report/templates/datatype_agnostic.jinja new file mode 100644 index 0000000000000000000000000000000000000000..50cc2a5588a0e2e370b6a1fb9e142202683b1bbe --- /dev/null +++ b/mne-bids-0.15/mne_bids/report/templates/datatype_agnostic.jinja @@ -0,0 +1,20 @@ +Data was recorded using an {{ system }} system +{% if manufacturer %} +({{ manufacturer }}) +{% endif %} +sampled at {{ sfreq }} Hz +with line noise at {{ powerlinefreq }} Hz. +{% if software_filters %} +The following software filters were applied during recording: +{{ software_filters }}. +{% endif %} +{% if n_scans > 1 %} +There were {{n_scans}} scans in total. +{% else %} +There was {{n_scans}} scan in total. +{% endif %} +{{ length_recordings }} +For each dataset, there were on average {{mean_chs}} (std = {{std_chs}}) +recording channels per scan, out of which {{mean_good_chs}} +(std = {{std_good_chs}}) were used in analysis +({{mean_bad_chs}} +/- {{std_bad_chs}} were removed from analysis). diff --git a/mne-bids-0.15/mne_bids/report/templates/participants.jinja b/mne-bids-0.15/mne_bids/report/templates/participants.jinja new file mode 100644 index 0000000000000000000000000000000000000000..4fb1b9c67d838dd91142823f03251fdfce300edd --- /dev/null +++ b/mne-bids-0.15/mne_bids/report/templates/participants.jinja @@ -0,0 +1,3 @@ +{{ sexs }}; +{{hands}}; +{{ages}} diff --git a/mne-bids-0.15/mne_bids/sidecar_updates.py b/mne-bids-0.15/mne_bids/sidecar_updates.py new file mode 100644 index 0000000000000000000000000000000000000000..0942aa2a8888929de9edbd109a294c4dce0ed1fe --- /dev/null +++ b/mne-bids-0.15/mne_bids/sidecar_updates.py @@ -0,0 +1,389 @@ +"""Update BIDS directory structures and sidecar files meta data.""" +# Authors: Adam Li +# Austin Hurst +# Stefan Appelhoff +# mne-bids developers +# +# License: BSD-3-Clause + +import json +from collections import OrderedDict + +import numpy as np +from mne.channels import DigMontage, make_dig_montage +from mne.io import read_fiducials +from mne.io.constants import FIFF +from mne.utils import _check_on_missing, _on_missing, _validate_type, logger, verbose + +from mne_bids import BIDSPath +from mne_bids.utils import _write_json + + +# TODO: add support for tsv files +@verbose +def update_sidecar_json(bids_path, entries, verbose=None): + """Update sidecar files using a dictionary or JSON file. + + Will update metadata fields inside the path defined by + ``bids_path.fpath`` according to the ``entries``. If a + field does not exist in the corresponding sidecar file, + then that field will be created according to the ``entries``. + If a field does exist in the corresponding sidecar file, + then that field will be updated according to the ``entries``. + + For example, if ``InstitutionName`` is not defined in + the sidecar json file, then trying to update + ``InstitutionName`` to ``Martinos Center`` will update + the sidecar json file to have ``InstitutionName`` as + ``Martinos Center``. + + Parameters + ---------- + bids_path : BIDSPath + The set of paths to update. The :class:`mne_bids.BIDSPath` instance + passed here **must** have the ``.root`` attribute set. The + ``.datatype`` attribute **may** be set. If ``.datatype`` is + not set and only one data type (e.g., only EEG or MEG data) + is present in the dataset, it will be + selected automatically. This must uniquely identify + an existing file path, else an error will be raised. + entries : dict | str | pathlib.Path + A dictionary, or JSON file that defines the + sidecar fields and corresponding values to be updated to. + %(verbose)s + + Notes + ----- + This function can only update JSON files. + + Sidecar JSON files include files such as ``*_ieeg.json``, + ``*_coordsystem.json``, ``*_scans.json``, etc. + + You should double check that your update dictionary is correct + for the corresponding sidecar JSON file because it will perform + a dictionary update of the sidecar fields according to + the passed in dictionary overwriting any information that was + previously there. + + Raises + ------ + RuntimeError + If the specified ``bids_path.fpath`` cannot be found + in the dataset. + + RuntimeError + If the ``bids_path.fpath`` does not have ``.json`` + extension. + + Examples + -------- + Update a sidecar JSON file + + >>> from pathlib import Path + >>> root = Path('./mne_bids/tests/data/tiny_bids').absolute() + >>> bids_path = BIDSPath(subject='01', task='rest', session='eeg', + ... suffix='eeg', extension='.json', datatype='eeg', + ... root=root) + >>> entries = {'PowerLineFrequency': 60} + >>> update_sidecar_json(bids_path, entries, verbose=False) + + """ + # get all matching json files + bids_path = bids_path.copy() + if bids_path.extension != ".json": + raise RuntimeError( + 'Only works for ".json" files. The ' + "BIDSPath object passed in has " + f"{bids_path.extension} extension." + ) + + # get the file path + fpath = bids_path.fpath + if not fpath.exists(): + raise RuntimeError(f"Sidecar file does not " f"exist for {fpath}.") + + # sidecar update either from file, or as dictionary + if isinstance(entries, dict): + sidecar_tmp = entries + else: + with open(entries) as tmp_f: + sidecar_tmp = json.load(tmp_f, object_pairs_hook=OrderedDict) + + logger.debug(sidecar_tmp) + logger.debug(f"Updating {fpath}...") + + # load in sidecar filepath + with open(fpath) as tmp_f: + sidecar_json = json.load(tmp_f, object_pairs_hook=OrderedDict) + + # update sidecar JSON file with the fields passed in + sidecar_json.update(**sidecar_tmp) + + # write back the sidecar JSON + _write_json(fpath, sidecar_json, overwrite=True) + + +def _update_sidecar(sidecar_fname, key, val): + """Update a sidecar JSON file with a given key/value pair. + + Parameters + ---------- + sidecar_fname : str | os.PathLike + Full name of the data file + key : str + The key in the sidecar JSON file. E.g. "PowerLineFrequency" + val : str + The corresponding value to change to in the sidecar JSON file. + """ + with open(sidecar_fname, encoding="utf-8-sig") as fin: + sidecar_json = json.load(fin) + sidecar_json[key] = val + _write_json(sidecar_fname, sidecar_json, overwrite=True) + + +@verbose +def update_anat_landmarks( + bids_path, + landmarks, + *, + fs_subject=None, + fs_subjects_dir=None, + kind=None, + on_missing="raise", + verbose=None, +): + """Update the anatomical landmark coordinates of an MRI scan. + + This will change the ``AnatomicalLandmarkCoordinates`` entry in the + respective JSON sidecar file, or create it if it doesn't exist. + + Parameters + ---------- + bids_path : BIDSPath + Path of the MR image. + landmarks : mne.channels.DigMontage | path-like + An :class:`mne.channels.DigMontage` instance with coordinates for the + nasion and left and right pre-auricular points in MRI voxel + coordinates. Alternatively, the path to a ``*-fiducials.fif`` file as + produced by the MNE-Python coregistration GUI or via + :func:`mne.io.write_fiducials`. + + .. note:: :func:`mne_bids.get_anat_landmarks` provides a convenient and + reliable way to generate the landmark coordinates in the + required coordinate system. + + .. note:: If ``path-like``, ``fs_subject`` and ``fs_subjects_dir`` + must be provided as well. + + .. versionchanged:: 0.10 + Added support for ``path-like`` input. + fs_subject : str | None + The subject identifier used for FreeSurfer. Must be provided if + ``landmarks`` is ``path-like``; otherwise, it will be ignored. + fs_subjects_dir : path-like | None + The FreeSurfer subjects directory. If ``None``, defaults to the + ``SUBJECTS_DIR`` environment variable. Must be provided if + ``landmarks`` is ``path-like``; otherwise, it will be ignored. + kind : str | None + The suffix of the anatomical landmark names in the JSON sidecar. + A suffix might be present e.g. to distinguish landmarks between + sessions. If provided, should not include a leading underscore ``_``. + For example, if the landmark names in the JSON sidecar file are + ``LPA_ses-1``, ``RPA_ses-1``, ``NAS_ses-1``, you should pass + ``'ses-1'`` here. + If ``None``, no suffix is appended, the landmarks named + ``Nasion`` (or ``NAS``), ``LPA``, and ``RPA`` will be used. + + .. versionadded:: 0.10 + on_missing : 'ignore' | 'warn' | 'raise' + How to behave if the specified landmarks cannot be found in the MRI + JSON sidecar file. + + .. versionadded:: 0.10 + %(verbose)s + + Notes + ----- + .. versionadded:: 0.8 + """ + _validate_type(item=bids_path, types=BIDSPath, item_name="bids_path") + _validate_type( + item=landmarks, types=(DigMontage, "path-like"), item_name="landmarks" + ) + _check_on_missing(on_missing) + + # Do some path verifications and fill in some gaps the users might have + # left (datatype and extension) + # XXX We could be more stringent (and less user-friendly) and insist on a + # XXX full specification of all parts of the BIDSPath, thoughts? + bids_path_mri = bids_path.copy() + if bids_path_mri.datatype is None: + bids_path_mri.datatype = "anat" + + if bids_path_mri.datatype != "anat": + raise ValueError( + f'Can only operate on "anat" MRI data, but the provided bids_path ' + f"points to: {bids_path_mri.datatype}" + ) + + if bids_path_mri.suffix is None: + raise ValueError( + 'Please specify the "suffix" entity of the provided ' "bids_path." + ) + elif bids_path_mri.suffix not in ("T1w", "FLASH"): + raise ValueError( + f'Can only operate on "T1w" and "FLASH" images, but the bids_path ' + f"suffix indicates: {bids_path_mri.suffix}" + ) + + valid_extensions = (".nii", ".nii.gz") + tried_paths = [] + file_exists = False + if bids_path_mri.extension is None: + # No extension was provided, start searching … + for extension in valid_extensions: + bids_path_mri.extension = extension + tried_paths.append(bids_path_mri.fpath) + + if bids_path_mri.fpath.exists(): + file_exists = True + break + else: + # An extension was provided + tried_paths.append(bids_path_mri.fpath) + if bids_path_mri.fpath.exists(): + file_exists = True + + if not file_exists: + raise ValueError( + f"Could not find an MRI scan. Please check the provided " + f"bids_path. Tried the following filenames: " + f'{", ".join([p.name for p in tried_paths])}' + ) + + if not isinstance(landmarks, DigMontage): # it's pathlike + if fs_subject is None: + raise ValueError( + 'You must provide the "fs_subject" parameter when passing the ' + "path to fiducials" + ) + landmarks = _get_landmarks_from_fiducials_file( + bids_path=bids_path, + fname=landmarks, + fs_subject=fs_subject, + fs_subjects_dir=fs_subjects_dir, + ) + + positions = landmarks.get_positions() + coord_frame = positions["coord_frame"] + if coord_frame != "mri_voxel": + raise ValueError( + f"The landmarks must be specified in MRI voxel coordinates, but " + f'provided DigMontage is in "{coord_frame}"' + ) + + # Extract the cardinal points + name_to_coords_map = { + "LPA": positions["lpa"], + "NAS": positions["nasion"], + "RPA": positions["rpa"], + } + + # Check if coordinates for any cardinal point are missing, and convert to + # a list so we can easily store the data in JSON format + missing_points = [] + for name, coords in name_to_coords_map.items(): + if coords is None: + missing_points.append(name) + else: + # Funnily, np.float64 is JSON-serializabe, while np.float32 is not! + # Thus, cast to float64 to avoid issues (which e.g. may arise when + # fiducials were read from disk!) + name_to_coords_map[name] = list(coords.astype("float64")) + + if missing_points: + raise ValueError( + f"The provided DigMontage did not contain all required cardinal " + f"points (nasion and left and right pre-auricular points). The " + f"following points are missing: " + f'{", ".join(missing_points)}' + ) + + bids_path_json = bids_path.copy().update(extension=".json") + if not bids_path_json.fpath.exists(): # Must exist before we can update it + _write_json(bids_path_json.fpath, dict()) + + mri_json = json.loads(bids_path_json.fpath.read_text(encoding="utf-8")) + if "AnatomicalLandmarkCoordinates" not in mri_json: + _on_missing( + on_missing=on_missing, + msg=f"No AnatomicalLandmarkCoordinates section found in " + f"{bids_path_json.fpath.name}", + error_klass=KeyError, + ) + mri_json["AnatomicalLandmarkCoordinates"] = dict() + + for name, coords in name_to_coords_map.items(): + if kind is not None: + name = f"{name}_{kind}" + + if name not in mri_json["AnatomicalLandmarkCoordinates"]: + _on_missing( + on_missing=on_missing, + msg=f"Anatomical landmark not found in " + f"{bids_path_json.fpath.name}: {name}", + error_klass=KeyError, + ) + + mri_json["AnatomicalLandmarkCoordinates"][name] = coords + + update_sidecar_json(bids_path=bids_path_json, entries=mri_json) + + +def _get_landmarks_from_fiducials_file( + *, bids_path, fname, fs_subject, fs_subjects_dir +): + """Get anatomical landmarks from fiducials file, in MRI voxel space.""" + # avoid dicrular imports + from mne_bids.write import ( + _get_fid_coords, + _get_t1w_mgh, + _mri_landmarks_to_mri_voxels, + ) + + digpoints, coord_frame = read_fiducials(fname) + + # All of this should be guaranteed, but better be safe than sorry! + assert coord_frame == FIFF.FIFFV_COORD_MRI + assert digpoints[0]["ident"] == FIFF.FIFFV_POINT_LPA + assert digpoints[1]["ident"] == FIFF.FIFFV_POINT_NASION + assert digpoints[2]["ident"] == FIFF.FIFFV_POINT_RPA + + montage_loaded = make_dig_montage( + lpa=digpoints[0]["r"], + nasion=digpoints[1]["r"], + rpa=digpoints[2]["r"], + coord_frame="mri", + ) + landmark_coords_mri, _ = _get_fid_coords(dig_points=montage_loaded.dig) + landmark_coords_mri = np.asarray( + ( + landmark_coords_mri["lpa"], + landmark_coords_mri["nasion"], + landmark_coords_mri["rpa"], + ) + ) + + t1w_mgh = _get_t1w_mgh(fs_subject, fs_subjects_dir) + landmark_coords_voxels = _mri_landmarks_to_mri_voxels( + mri_landmarks=landmark_coords_mri * 1000, + t1_mgh=t1w_mgh, # in mm + ) + montage_voxels = make_dig_montage( + lpa=landmark_coords_voxels[0], + nasion=landmark_coords_voxels[1], + rpa=landmark_coords_voxels[2], + coord_frame="mri_voxel", + ) + + return montage_voxels diff --git a/mne-bids-0.15/mne_bids/stats.py b/mne-bids-0.15/mne_bids/stats.py new file mode 100644 index 0000000000000000000000000000000000000000..50931b005467581cd7002defc7e58e9d66ca3c57 --- /dev/null +++ b/mne-bids-0.15/mne_bids/stats.py @@ -0,0 +1,130 @@ +"""Some functions to extract stats from a BIDS dataset.""" + +# Authors: Alex Gramfort +# +# License: BSD-3-Clause + +from mne_bids import BIDSPath, get_datatypes +from mne_bids.config import EPHY_ALLOWED_DATATYPES + + +def count_events(root_or_path, datatype="auto"): + """Count events present in dataset. + + Parameters + ---------- + root_or_path : path-like | mne_bids.BIDSPath + If str or Path it is the root folder of the BIDS dataset. + If a BIDSPath is passed it allows to limit the count + to a subject, a session or a run by only considering + the event files that match this BIDSPath. + datatype : str + Type of the data recording. Can be ``meg``, ``eeg``, + ``ieeg`` or ``auto``. If ``auto`` and a :class:`mne_bids.BIDSPath` + isinstance is passed as ``root_or_path`` which has a ``datatype`` + attribute set, then this data type will be used. Otherwise, only + one data type should be present in the dataset to avoid any + ambiguity. + + Returns + ------- + counts : pandas.DataFrame + The pandas dataframe containing all the counts of trial_type + in all matching events.tsv files. + + Notes + ----- + .. versionchanged:: 0.15 + Table values were changed from floats (with NaN for missing values) + to Pandas nullable integer arrays. + """ + import pandas as pd + + if not isinstance(root_or_path, BIDSPath): + bids_path = BIDSPath(root=root_or_path) + else: + bids_path = root_or_path.copy() + + bids_path.update(suffix="events", extension=".tsv") + + datatypes = get_datatypes(bids_path.root) + this_datatypes = list(set(datatypes).intersection(EPHY_ALLOWED_DATATYPES)) + + if (datatype == "auto") and (bids_path.datatype is not None): + datatype = bids_path.datatype + + if datatype == "auto": + if len(this_datatypes) > 1: + raise ValueError( + f"Multiple datatypes present ({this_datatypes})." + f" You need to specity datatype got: {datatype})" + ) + elif len(this_datatypes) == 0: + raise ValueError("No valid datatype present.") + + datatype = this_datatypes[0] + + if datatype not in EPHY_ALLOWED_DATATYPES: + raise ValueError( + f"datatype ({datatype}) is not supported. " + f"It must be one of: {EPHY_ALLOWED_DATATYPES})" + ) + + bids_path.update(datatype=datatype) + + tasks = sorted(set([bp.task for bp in bids_path.match()])) + + all_counts = [] + + for task in tasks: + bids_path.update(task=task) + + all_df = [] + for bp in bids_path.match(): + df = pd.read_csv(str(bp), delimiter="\t") + df["subject"] = bp.subject + if bp.session is not None: + df["session"] = bp.session + if bp.run is not None: + df["run"] = bp.run + all_df.append(df) + + if not all_df: + continue + + df = pd.concat(all_df) + groups = ["subject"] + if bp.session is not None: + groups.append("session") + if bp.run is not None: + groups.append("run") + + if "stim_type" in df.columns: + # Deal with some old files that use stim_type rather than + # trial_type + df = df.rename(columns={"stim_type": "trial_type"}) + + # There are datasets out there without a `trial_type` or `stim_type` + # column. + if "trial_type" in df.columns: + groups.append("trial_type") + + counts = df.groupby(groups).size() + counts = counts.unstack(fill_value=-1) + counts.replace(-1, pd.NA, inplace=True) + + if "BAD_ACQ_SKIP" in counts.columns: + counts = counts.drop("BAD_ACQ_SKIP", axis=1) + + counts.columns = pd.MultiIndex.from_arrays( + [[task] * counts.shape[1], counts.columns] + ) + + all_counts.append(counts) + + if not all_counts: + raise ValueError("No events files found.") + + counts = pd.concat(all_counts, axis=1) + + return counts diff --git a/mne-bids-0.15/mne_bids/tsv_handler.py b/mne-bids-0.15/mne_bids/tsv_handler.py new file mode 100644 index 0000000000000000000000000000000000000000..3ce4b60d8c6bcb1d5394561bee671455b0c7fde2 --- /dev/null +++ b/mne-bids-0.15/mne_bids/tsv_handler.py @@ -0,0 +1,220 @@ +"""Private functions to handle tabular data.""" + +from collections import OrderedDict +from copy import deepcopy + +import numpy as np + + +def _combine_rows(data1, data2, drop_column=None): + """Add two OrderedDict's together and optionally drop repeated data. + + Parameters + ---------- + data1 : collections.OrderedDict + Original OrderedDict. + data2 : collections.OrderedDict + New OrderedDict to be added to the original. + drop_column : str, optional + Name of the column to check for duplicate values in. + Any duplicates found will be dropped from the original data array (ie. + most recent value are kept). + + Returns + ------- + data : collections.OrderedDict + The new combined data. + """ + data = deepcopy(data1) + # next extend the values in data1 with values in data2 + for key, value in data2.items(): + data[key].extend(value) + + # Make sure that if there are any columns in data1 that didn't get new + # data they are populated with "n/a"'s. + for key in set(data1.keys()) - set(data2.keys()): + data[key].extend(["n/a"] * len(next(iter(data2.values())))) + + if drop_column is None: + return data + + # Find any repeated values and remove all but the most recent value. + n_rows = len(data[drop_column]) + _, idxs = np.unique(data[drop_column][::-1], return_index=True) + for key in data: + data[key] = [data[key][n_rows - 1 - idx] for idx in idxs] + + return data + + +def _contains_row(data, row_data): + """Determine whether the specified row data exists in the OrderedDict. + + Parameters + ---------- + data : collections.OrderedDict + OrderedDict to check. + row_data : dict + Dictionary with column names as keys, and values being the column value + to match within a row. + + Returns + ------- + bool + True if `row_data` exists in `data`. + + Note + ---- + This function will return True if the supplied `row_data` contains less + columns than the number of columns in the existing data but there is still + a match for the partial row data. + + """ + mask = None + for key, row_value in row_data.items(): + # if any of the columns don't even exist in the keys + # this data_value will return False + data_value = np.array(data.get(key)) + + # Cast row_value to the same dtype as data_value to avoid a NumPy + # FutureWarning, see + # https://github.com/mne-tools/mne-bids/pull/372 + row_value = np.array(row_value, dtype=data_value.dtype) + + column_mask = np.isin(data_value, row_value) + mask = column_mask if mask is None else (mask & column_mask) + return np.any(mask) + + +def _drop(data, values, column): + """Remove rows from the OrderedDict. + + Parameters + ---------- + data : collections.OrderedDict + Data to drop values from. + values : list + List of values to drop. Any row containing this value in the specified + column will be dropped. + column : string + Name of the column to check for the existence of `value` in. + + Returns + ------- + new_data : collections.OrderedDict + Copy of the original data with 0 or more rows dropped. + + """ + new_data = deepcopy(data) + new_data_col = np.array(new_data[column]) + + # Cast `values` to the same dtype as `new_data_col` to avoid a NumPy + # FutureWarning, see + # https://github.com/mne-tools/mne-bids/pull/372 + dtype = new_data_col.dtype + if new_data_col.shape == (0,): + dtype = np.array(values).dtype + values = np.array(values, dtype=dtype) + + mask = np.isin(new_data_col, values, invert=True) + for key in new_data.keys(): + new_data[key] = np.array(new_data[key])[mask].tolist() + return new_data + + +def _from_tsv(fname, dtypes=None): + """Read a tsv file into an OrderedDict. + + Parameters + ---------- + fname : str + Path to the file being loaded. + dtypes : list, optional + List of types to cast the values loaded as. This is specified column by + column. + Defaults to None. In this case all the data is loaded as strings. + + Returns + ------- + data_dict : collections.OrderedDict + Keys are the column names, and values are the column data. + + """ + from .utils import warn # avoid circular import + + data = np.loadtxt( + fname, dtype=str, delimiter="\t", ndmin=2, comments=None, encoding="utf-8-sig" + ) + column_names = data[0, :] + info = data[1:, :] + data_dict = OrderedDict() + if dtypes is None: + dtypes = [str] * info.shape[1] + if not isinstance(dtypes, (list, tuple)): + dtypes = [dtypes] * info.shape[1] + if not len(dtypes) == info.shape[1]: + raise ValueError( + "dtypes length mismatch. " + f"Provided: {len(dtypes)}, Expected: {info.shape[1]}" + ) + empty_cols = 0 + for i, name in enumerate(column_names): + values = info[:, i].astype(dtypes[i]).tolist() + data_dict[name] = values + if len(values) == 0: + empty_cols += 1 + + if empty_cols == len(column_names): + warn(f"TSV file is empty: '{fname}'") + + return data_dict + + +def _to_tsv(data, fname): + """Write an OrderedDict into a tsv file. + + Parameters + ---------- + data : collections.OrderedDict + Ordered dictionary containing data to be written to a tsv file. + fname : str + Path to the file being written. + + """ + n_rows = len(data[list(data.keys())[0]]) + output = _tsv_to_str(data, n_rows) + + with open(fname, "w", encoding="utf-8-sig") as f: + f.write(output) + f.write("\n") + + +def _tsv_to_str(data, rows=5): + """Return a string representation of the OrderedDict. + + Parameters + ---------- + data : collections.OrderedDict + OrderedDict to return string representation of. + rows : int, optional + Maximum number of rows of data to output. + + Returns + ------- + str + String representation of the first `rows` lines of `data`. + + """ + col_names = list(data.keys()) + n_rows = len(data[col_names[0]]) + output = list() + # write headings. + output.append("\t".join(col_names)) + + # write column data. + max_rows = min(n_rows, rows) + for idx in range(max_rows): + row_data = list(str(data[key][idx]) for key in data) + output.append("\t".join(row_data)) + + return "\n".join(output) diff --git a/mne-bids-0.15/mne_bids/utils.py b/mne-bids-0.15/mne_bids/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..2c934b2b13eab8c42c600d36ee105e184c3f5d0e --- /dev/null +++ b/mne-bids-0.15/mne_bids/utils.py @@ -0,0 +1,542 @@ +"""Utility and helper functions for MNE-BIDS.""" + +# Authors: Mainak Jas +# Alexandre Gramfort +# Teon Brooks +# Chris Holdgraf +# Stefan Appelhoff +# Matt Sanderson +# +# License: BSD-3-Clause +import json +import os +import re +from datetime import date, datetime, timedelta, timezone +from os import path as op + +import numpy as np +from mne import pick_types +from mne.channels import make_standard_montage +from mne.io.kit.kit import get_kit_info +from mne.utils import logger, verbose +from mne.utils import warn as _warn + +from mne_bids.tsv_handler import _to_tsv + +# This regex matches key-val pairs. Any characters are allowed in the key and +# the value, except these special symbols: - _ . \ / +param_regex = re.compile(r"([^-_\.\\\/]+)-([^-_\.\\\/]+)") + + +def _ensure_tuple(x): + """Return a tuple.""" + if x is None: + return tuple() + elif isinstance(x, str): + return (x,) + else: + return tuple(x) + + +def _get_ch_type_mapping(fro="mne", to="bids"): + """Map between BIDS and MNE nomenclatures for channel types. + + Parameters + ---------- + fro : str + Mapping from nomenclature of `fro`. Can be 'mne', 'bids' + to : str + Mapping to nomenclature of `to`. Can be 'mne', 'bids' + + Returns + ------- + mapping : dict + Dictionary mapping from one nomenclature of channel types to another. + If a key is not present, a default value will be returned that depends + on the `fro` and `to` parameters. + + Notes + ----- + For the mapping from BIDS to MNE, MEG channel types are ignored for now. + Furthermore, this is not a one-to-one mapping: Incomplete and partially + one-to-many/many-to-one. + + Bio channels are supported in mne-python and are converted to MISC + because there is no "Bio" supported channel in BIDS. + """ + if fro == "mne" and to == "bids": + mapping = dict( + eeg="EEG", + misc="MISC", + stim="TRIG", + emg="EMG", + ecog="ECOG", + seeg="SEEG", + eog="EOG", + ecg="ECG", + resp="RESP", + bio="MISC", + dbs="DBS", + gsr="GSR", + temperature="TEMP", + # NIRS + fnirs_cw_amplitude="NIRSCWAMPLITUDE", + # MEG channels + meggradaxial="MEGGRADAXIAL", + megmag="MEGMAG", + megrefgradaxial="MEGREFGRADAXIAL", + meggradplanar="MEGGRADPLANAR", + megrefmag="MEGREFMAG", + ias="MEGOTHER", + syst="MEGOTHER", + exci="MEGOTHER", + ) + + elif fro == "bids" and to == "mne": + mapping = dict( + EEG="eeg", + MISC="misc", + TRIG="stim", + EMG="emg", + ECOG="ecog", + SEEG="seeg", + EOG="eog", + ECG="ecg", + RESP="resp", + GSR="gsr", + TEMP="temperature", + # NIRS + NIRSCWAMPLITUDE="fnirs_cw_amplitude", + NIRS="fnirs_cw_amplitude", + # No MEG channels for now (see Notes above) + # Many to one mapping + VEOG="eog", + HEOG="eog", + DBS="dbs", + ) + else: + raise ValueError( + "Only two types of mappings are currently supported: " + "from mne to bids, or from bids to mne. However, " + f'you specified from "{fro}" to "{to}"' + ) + + return mapping + + +def _handle_datatype(raw, datatype): + """Check if datatype exists in raw object or infer datatype if possible. + + Parameters + ---------- + raw : mne.io.Raw + Raw object. + datatype : str | None + Can be one of either ``'meg'``, ``'eeg'``, or ``'ieeg'``. If ``None``, + `mne.utils._handle_datatype()` will attempt to infer the datatype from + the ``raw`` object. In case of multiple data types in the ``raw`` + object, ``datatype`` must not be ``None``. + + Returns + ------- + datatype : str + One of either ``'meg'``, ``'eeg'``, or ``'ieeg'``. + """ + if datatype is not None: + _check_datatype(raw, datatype) + # MEG data is not supported by BrainVision or EDF files + if datatype in ["eeg", "ieeg"] and "meg" in raw: + logger.info( + f"{os.linesep}Both {datatype} and 'meg' data found. " + f"BrainVision and EDF do not support 'meg' data. " + f"The data will therefore be stored as 'meg' data. " + f"If you wish to store your {datatype} data in " + f"BrainVision or EDF, please remove the 'meg'" + f"channels from your recording.{os.linesep}" + ) + datatype = "meg" + else: + datatypes = list() + ieeg_types = ["seeg", "ecog", "dbs"] + if any(ieeg_type in raw for ieeg_type in ieeg_types): + datatypes.append("ieeg") + if "meg" in raw: + datatypes.append("meg") + if "eeg" in raw: + datatypes.append("eeg") + if "fnirs_cw_amplitude" in raw: + datatypes.append("nirs") + if len(datatypes) == 0: + raise ValueError( + "No MEG, EEG or iEEG channels found in data. " + "Please use raw.set_channel_types to set the " + "channel types in the data." + ) + elif len(datatypes) > 1: + if "meg" in datatypes and "ieeg" not in datatypes: + datatype = "meg" + elif "ieeg" in datatypes and "meg" not in datatypes: + datatype = "ieeg" + else: + raise ValueError( + f"Multiple data types (``{datatypes}``) were " + "found in the data. Please specify the " + "datatype using " + '`bids_path.update(datatype="")` ' + "or use raw.set_channel_types to set the " + "correct channel types in the raw object." + ) + else: + datatype = datatypes[0] + return datatype + + +def _age_on_date(bday, exp_date): + """Calculate age from birthday and experiment date. + + Parameters + ---------- + bday : datetime.datetime + The birthday of the participant. + exp_date : datetime.datetime + The date the experiment was performed on. + + """ + if exp_date < bday: + raise ValueError("The experimentation date must be after the birth date") + if exp_date.month > bday.month: + return exp_date.year - bday.year + elif exp_date.month == bday.month: + if exp_date.day >= bday.day: + return exp_date.year - bday.year + return exp_date.year - bday.year - 1 + + +def _check_types(variables): + """Make sure all vars are str or None.""" + for var in variables: + if not isinstance(var, (str, type(None))): + raise ValueError( + f"You supplied a value ({var}) of type " + f"{type(var)}, where a string or None was " + f"expected." + ) + + +def _write_json(fname, dictionary, overwrite=False): + """Write JSON to a file.""" + if op.exists(fname) and not overwrite: + raise FileExistsError( + f'"{fname}" already exists. ' "Please set overwrite to True." + ) + + json_output = json.dumps(dictionary, indent=4) + with open(fname, "w", encoding="utf-8") as fid: + fid.write(json_output) + fid.write("\n") + + logger.info(f"Writing '{fname}'...") + + +@verbose +def _write_tsv(fname, dictionary, overwrite=False, verbose=None): + """Write an ordered dictionary to a .tsv file.""" + if op.exists(fname) and not overwrite: + raise FileExistsError( + f'"{fname}" already exists. ' "Please set overwrite to True." + ) + _to_tsv(dictionary, fname) + + logger.info(f"Writing '{fname}'...") + + +def _write_text(fname, text, overwrite=False): + """Write text to a file.""" + if op.exists(fname) and not overwrite: + raise FileExistsError( + f'"{fname}" already exists. ' "Please set overwrite to True." + ) + with open(fname, "w", encoding="utf-8-sig") as fid: + fid.write(text) + fid.write("\n") + + logger.info(f"Writing '{fname}'...") + + +def _check_key_val(key, val): + """Perform checks on a value to make sure it adheres to the spec.""" + if any(ii in val for ii in ["-", "_", "/"]): + raise ValueError( + "Unallowed `-`, `_`, or `/` found in key/value pair" f" {key}: {val}" + ) + return key, val + + +def _get_mrk_meas_date(mrk): + """Find the measurement date from a KIT marker file.""" + info = get_kit_info(mrk, False)[0] + meas_date = info.get("meas_date", None) + if isinstance(meas_date, (tuple, list, np.ndarray)): + meas_date = meas_date[0] + if isinstance(meas_date, datetime): + meas_datetime = meas_date + elif meas_date is not None: + meas_datetime = datetime.fromtimestamp(meas_date) + else: + meas_datetime = datetime.min + return meas_datetime + + +def _infer_eeg_placement_scheme(raw): + """Based on the channel names, try to infer an EEG placement scheme. + + Parameters + ---------- + raw : mne.io.Raw + The data as MNE-Python Raw object. + + Returns + ------- + placement_scheme : str + Description of the EEG placement scheme. Will be "n/a" for unsuccessful + extraction. + + """ + placement_scheme = "n/a" + # Check if the raw data contains eeg data at all + if "eeg" not in raw: + return placement_scheme + + # How many of the channels in raw are based on the extended 10/20 system + sel = pick_types(raw.info, meg=False, eeg=True) + ch_names = [raw.ch_names[i] for i in sel] + channel_names = [ch.lower() for ch in ch_names] + montage1005 = make_standard_montage("standard_1005") + montage1005_names = [ch.lower() for ch in montage1005.ch_names] + + if set(channel_names).issubset(set(montage1005_names)): + placement_scheme = "based on the extended 10/20 system" + + return placement_scheme + + +def _scale_coord_to_meters(coord, unit): + """Scale units to meters (mne-python default).""" + if unit == "cm": + return np.divide(coord, 100.0) + elif unit == "mm": + return np.divide(coord, 1000.0) + else: + return coord + + +def _check_empty_room_basename(bids_path): + if bids_path.subject != "emptyroom": + return + # only check task entity for emptyroom when it is the sidecar/MEG file + if bids_path.suffix != "meg": + return + if bids_path.acquisition in ("calibration", "crosstalk"): + return + if bids_path.task != "noise": + raise ValueError( + f'task must be "noise" if subject is "emptyroom", but ' + f"received: {bids_path.task}" + ) + + +def _check_anonymize(anonymize, raw, ext): + """Check the `anonymize` dict.""" + # if info['meas_date'] None, then the dates are not stored + if raw.info["meas_date"] is None: + daysback = None + else: + if "daysback" not in anonymize or anonymize["daysback"] is None: + raise ValueError("`daysback` argument required to anonymize.") + daysback = anonymize["daysback"] + daysback_min, daysback_max = _get_anonymization_daysback(raw) + if daysback < daysback_min: + warn( + "`daysback` is too small; the measurement date " + "is after 1925, which is not recommended by BIDS." + "The minimum `daysback` value for changing the " + "measurement date of this data to before this date " + f"is {daysback_min}" + ) + if ext == ".fif" and daysback > daysback_max: + raise ValueError( + "`daysback` exceeds maximum value MNE " + "is able to store in FIF format, must " + f"be less than {daysback_max}" + ) + keep_his = anonymize["keep_his"] if "keep_his" in anonymize else False + keep_source = anonymize["keep_source"] if "keep_source" in anonymize else False + return daysback, keep_his, keep_source + + +def _get_anonymization_daysback(raw): + """Get the min and max number of daysback necessary to satisfy BIDS specs. + + Parameters + ---------- + raw : mne.io.Raw + Subject raw data. + + Returns + ------- + daysback_min : int + The minimum number of daysback necessary to be compatible with BIDS. + daysback_max : int + The maximum number of daysback that MNE can store. + """ + this_date = _stamp_to_dt(raw.info["meas_date"]).date() + daysback_min = (this_date - date(year=1924, month=12, day=31)).days + daysback_max = ( + this_date + - datetime.fromtimestamp(0).date() + + timedelta(seconds=np.iinfo(">i4").max) + ).days + return daysback_min, daysback_max + + +@verbose +def get_anonymization_daysback(raws, verbose=None): + """Get the group min and max number of daysback necessary for BIDS specs. + + .. warning:: It is important that you remember the anonymization + number if you would ever like to de-anonymize but + that it is not included in the code publication + as that would break the anonymization. + + BIDS requires that anonymized dates be before 1925. In order to + preserve the longitudinal structure and ensure anonymization, the + user is asked to provide the same `daysback` argument to each call + of `write_raw_bids`. To determine the minimum number of daysback + necessary, this function will calculate the minimum number based on + the most recent measurement date of raw objects. + + Parameters + ---------- + raw : mne.io.Raw | list of mne.io.Raw + Subject raw data or list of raw data from several subjects. + %(verbose)s + + Returns + ------- + daysback_min : int + The minimum number of daysback necessary to be compatible with BIDS. + daysback_max : int + The maximum number of daysback that MNE can store. + """ + if not isinstance(raws, list): + raws = list([raws]) + daysback_min_list = list() + daysback_max_list = list() + for raw in raws: + if raw.info["meas_date"] is not None: + daysback_min, daysback_max = _get_anonymization_daysback(raw) + daysback_min_list.append(daysback_min) + daysback_max_list.append(daysback_max) + if not daysback_min_list or not daysback_max_list: + raise ValueError( + "All measurement dates are None, pass any `daysback` value to anonymize." + ) + daysback_min = max(daysback_min_list) + daysback_max = min(daysback_max_list) + if daysback_min > daysback_max: + raise ValueError( + "The dataset spans more time than can be " + "accomodated by MNE, you may have to " + "not follow BIDS recommendations and use" + "anonymized dates after 1925" + ) + return daysback_min, daysback_max + + +def _stamp_to_dt(utc_stamp): + """Convert POSIX timestamp to datetime object in Windows-friendly way.""" + # This is a windows datetime bug for timestamp < 0. A negative value + # is needed for anonymization which requires the date to be moved back + # to before 1925. This then requires a negative value of daysback + # compared the 1970 reference date. + if isinstance(utc_stamp, datetime): + return utc_stamp + stamp = [int(s) for s in utc_stamp] + if len(stamp) == 1: # In case there is no microseconds information + stamp.append(0) + return datetime.fromtimestamp(0, tz=timezone.utc) + timedelta( + 0, stamp[0], stamp[1] + ) # day, sec, μs + + +def _check_datatype(raw, datatype): + """Check if datatype exists in given raw object. + + Parameters + ---------- + raw : mne.io.Raw + Raw object. + datatype : str + Can be one of either ``'meg'``, ``'eeg'``, or ``'ieeg'``. + + Returns + ------- + None + """ + supported_types = ("meg", "eeg", "ieeg", "nirs") + if datatype not in supported_types: + raise ValueError( + f"The specified datatype {datatype} is currently not supported. " + f"It should be one of either `meg`, `eeg` or `ieeg` (Got " + f"`{datatype}`. Please specify a valid datatype using " + f'`bids_path.update(datatype="")`.' + ) + datatype_matches = False + if datatype == "eeg" and datatype in raw: + datatype_matches = True + elif datatype == "meg" and datatype in raw: + datatype_matches = True + elif datatype == "nirs" and "fnirs_cw_amplitude" in raw: + datatype_matches = True + elif datatype == "ieeg": + ieeg_types = ("seeg", "ecog", "dbs") + if any(ieeg_type in raw for ieeg_type in ieeg_types): + datatype_matches = True + if not datatype_matches: + raise ValueError( + f"The specified datatype {datatype} was not found in the raw " + "object. Please specify the correct datatype using " + '`bids_path.update(datatype="")` or use ' + "raw.set_channel_types to set the correct channel types in " + "the raw object." + ) + + +def _import_nibabel(why="work with MRI data"): + try: + import nibabel # noqa + except ImportError as exc: + raise exc.__class__( + f"nibabel is required to {why} but could not be imported, " f"got: {exc}" + ) from None + else: + return nibabel + + +def warn( + message, + category=RuntimeWarning, + module="mne_bids", + ignore_namespaces=("mne", "mne_bids"), +): # noqa: D103 + """Emit a warning.""" + _warn( + message, + category=category, + module=module, + ignore_namespaces=ignore_namespaces, + ) + + +# Some of the defaults here will be wrong but it should be close enough +warn.__doc__ = getattr(_warn, "__doc__", None) diff --git a/mne-bids-0.15/mne_bids/write.py b/mne-bids-0.15/mne_bids/write.py new file mode 100644 index 0000000000000000000000000000000000000000..be0a15012c548ec55eb0aac2c5756a3855d7f1c1 --- /dev/null +++ b/mne-bids-0.15/mne_bids/write.py @@ -0,0 +1,3147 @@ +"""Make BIDS compatible directory structures and infer meta data from MNE.""" + +# Authors: Mainak Jas +# Alexandre Gramfort +# Teon Brooks +# Chris Holdgraf +# Stefan Appelhoff +# Matt Sanderson +# +# License: BSD-3-Clause +import json +import os +import os.path as op +import re +import shutil +import sys +import warnings +from collections import OrderedDict, defaultdict +from datetime import datetime, timedelta, timezone +from pathlib import Path + +import mne +import mne.preprocessing +import numpy as np +from mne import Epochs, channel_type +from mne.channels.channels import _get_meg_system, _unit2human +from mne.chpi import get_chpi_info +from mne.io import BaseRaw, read_fiducials +from mne.io.constants import FIFF +from mne.io.pick import _picks_to_idx +from mne.transforms import _get_trans, apply_trans, rotation, translation +from mne.utils import ( + Bunch, + ProgressBar, + _validate_type, + check_version, + get_subjects_dir, + logger, + verbose, +) +from scipy import linalg + +from mne_bids import ( + BIDSPath, + get_anonymization_daysback, + get_bids_path_from_fname, + read_raw_bids, +) +from mne_bids.config import ( + ALLOWED_DATATYPE_EXTENSIONS, + ALLOWED_INPUT_EXTENSIONS, + ANONYMIZED_JSON_KEY_WHITELIST, + BIDS_STANDARD_TEMPLATE_COORDINATE_SYSTEMS, + BIDS_VERSION, + CONVERT_FORMATS, + EXT_TO_UNIT_MAP, + IGNORED_CHANNELS, + MANUFACTURERS, + ORIENTATION, + PYBV_VERSION, + REFERENCES, + UNITS_MNE_TO_BIDS_MAP, + _map_options, + reader, +) +from mne_bids.copyfiles import ( + copyfile_brainvision, + copyfile_bti, + copyfile_ctf, + copyfile_edf, + copyfile_eeglab, + copyfile_kit, +) +from mne_bids.dig import _write_coordsystem_json, _write_dig_bids +from mne_bids.path import _mkdir_p, _parse_ext, _path_to_str +from mne_bids.pick import coil_type +from mne_bids.read import _find_matching_sidecar, _read_events +from mne_bids.sidecar_updates import update_sidecar_json +from mne_bids.tsv_handler import _combine_rows, _contains_row, _drop, _from_tsv +from mne_bids.utils import ( + _age_on_date, + _check_anonymize, + _get_ch_type_mapping, + _handle_datatype, + _import_nibabel, + _infer_eeg_placement_scheme, + _stamp_to_dt, + _write_json, + _write_text, + _write_tsv, + warn, +) + +_FIFF_SPLIT_SIZE = "2GB" # MNE-Python default; can be altered during debugging + + +def _is_numeric(n): + return isinstance(n, (np.integer, np.floating, int, float)) + + +def _channels_tsv(raw, fname, overwrite=False): + """Create a channels.tsv file and save it. + + Parameters + ---------- + raw : mne.io.Raw + The data as MNE-Python Raw object. + fname : str | mne_bids.BIDSPath + Filename to save the channels.tsv to. + overwrite : bool + Whether to overwrite the existing file. + Defaults to False. + + """ + # Get channel type mappings between BIDS and MNE nomenclatures + map_chs = _get_ch_type_mapping(fro="mne", to="bids") + + # Prepare the descriptions for each channel type + map_desc = defaultdict(lambda: "Other type of channel") + map_desc.update( + meggradaxial="Axial Gradiometer", + megrefgradaxial="Axial Gradiometer Reference", + meggradplanar="Planar Gradiometer", + megmag="Magnetometer", + megrefmag="Magnetometer Reference", + stim="Trigger", + eeg="ElectroEncephaloGram", + ecog="Electrocorticography", + seeg="StereoEEG", + ecg="ElectroCardioGram", + eog="ElectroOculoGram", + emg="ElectroMyoGram", + misc="Miscellaneous", + bio="Biological", + ias="Internal Active Shielding", + dbs="Deep Brain Stimulation", + fnirs_cw_amplitude="Near Infrared Spectroscopy (continuous wave)", + resp="Respiration", + gsr="Galvanic skin response (electrodermal activity, EDA)", + temperature="Temperature", + ) + get_specific = ("mag", "ref_meg", "grad") + + # get the manufacturer from the file in the Raw object + _, ext = _parse_ext(raw.filenames[0]) + manufacturer = MANUFACTURERS.get(ext, "") + ignored_channels = IGNORED_CHANNELS.get(manufacturer, list()) + + status, ch_type, description = list(), list(), list() + for idx, ch in enumerate(raw.info["ch_names"]): + status.append("bad" if ch in raw.info["bads"] else "good") + _channel_type = channel_type(raw.info, idx) + if _channel_type in get_specific: + _channel_type = coil_type(raw.info, idx, _channel_type) + ch_type.append(map_chs[_channel_type]) + description.append(map_desc[_channel_type]) + low_cutoff, high_cutoff = (raw.info["highpass"], raw.info["lowpass"]) + if raw._orig_units: + units = [raw._orig_units.get(ch, "n/a") for ch in raw.ch_names] + else: + units = [_unit2human.get(ch_i["unit"], "n/a") for ch_i in raw.info["chs"]] + units = [u if u not in ["NA"] else "n/a" for u in units] + + # Translate from MNE to BIDS unit naming + for idx, mne_unit in enumerate(units): + if mne_unit in UNITS_MNE_TO_BIDS_MAP: + bids_unit = UNITS_MNE_TO_BIDS_MAP[mne_unit] + units[idx] = bids_unit + + n_channels = raw.info["nchan"] + sfreq = raw.info["sfreq"] + + # default to 'n/a' for status description + # XXX: improve with API to modify the description + status_description = ["n/a"] * len(status) + + ch_data = OrderedDict( + [ + ("name", raw.info["ch_names"]), + ("type", ch_type), + ("units", units), + ("low_cutoff", np.full((n_channels), low_cutoff)), + ("high_cutoff", np.full((n_channels), high_cutoff)), + ("description", description), + ("sampling_frequency", np.full((n_channels), sfreq)), + ("status", status), + ("status_description", status_description), + ] + ) + ch_data = _drop(ch_data, ignored_channels, "name") + + if "fnirs_cw_amplitude" in raw: + ch_data["wavelength_nominal"] = [ + raw.info["chs"][i]["loc"][9] for i in range(len(raw.ch_names)) + ] + + picks = _picks_to_idx(raw.info, "fnirs", exclude=[], allow_empty=True) + + sources = np.empty(picks.shape, dtype=" 0 and raise_error: + if set(fid_coord_frames.keys()) != set(["nasion", "lpa", "rpa"]): + raise ValueError( + f"Some fiducial points are missing, got {fid_coords.keys()}" + ) + + if len(set(fid_coord_frames.values())) > 1: + raise ValueError( + "All fiducial points must be in the same coordinate system, " + f"got {len(fid_coord_frames)})" + ) + + coord_frame = fid_coord_frames.popitem()[1] if fid_coord_frames else None + + return fid_coords, coord_frame + + +def _events_tsv(events, durations, raw, fname, trial_type, overwrite=False): + """Create an events.tsv file and save it. + + This function will write the mandatory 'onset', and 'duration' columns as + well as the optional 'value' and 'sample'. The 'value' + corresponds to the marker value as found in the TRIG channel of the + recording. In addition, the 'trial_type' field can be written. + + Parameters + ---------- + events : np.ndarray, shape = (n_events, 3) + The first column contains the event time in samples and the third + column contains the event id. The second column is ignored for now but + typically contains the value of the trigger channel either immediately + before the event or immediately after. + durations : np.ndarray, shape (n_events,) + The event durations in seconds. + raw : mne.io.Raw + The data as MNE-Python Raw object. + fname : str | mne_bids.BIDSPath + Filename to save the events.tsv to. + trial_type : dict | None + Dictionary mapping a brief description key to an event id (value). For + example {'Go': 1, 'No Go': 2}. + overwrite : bool + Whether to overwrite the existing file. + Defaults to False. + + """ + # Start by filling all data that we know into an ordered dictionary + first_samp = raw.first_samp + sfreq = raw.info["sfreq"] + events = events.copy() + events[:, 0] -= first_samp + + # Onset column needs to be specified in seconds + data = OrderedDict( + [ + ("onset", events[:, 0] / sfreq), + ("duration", durations), + ("trial_type", None), + ("value", events[:, 2]), + ("sample", events[:, 0]), + ] + ) + + # Now check if trial_type is specified or should be removed + if trial_type: + trial_type_map = {v: k for k, v in trial_type.items()} + data["trial_type"] = [trial_type_map.get(i, "n/a") for i in events[:, 2]] + else: + del data["trial_type"] + + _write_tsv(fname, data, overwrite) + + +def _events_json(fname, overwrite=False): + """Create participants.json for non-default columns in accompanying TSV. + + Parameters + ---------- + fname : str | mne_bids.BIDSPath + Output filename. + overwrite : bool + Whether to overwrite the output file if it exists. + """ + new_data = { + "onset": { + "Description": ( + "Onset (in seconds) of the event from the beginning of the first data" + "point. Negative onsets account for events before the first stored " + "data point." + ), + "Units": "s", + }, + "duration": { + "Description": ( + "Duration of the event in seconds from onset. " + "Must be zero, positive, or 'n/a' if unavailable. " + "A zero value indicates an impulse event. " + ), + "Units": "s", + }, + "sample": { + "Description": ( + "The event onset time in number of sampling points." + "First sample is 0." + ), + }, + "value": { + "Description": ( + "The event code (also known as trigger code or event ID) " + "associated with the event." + ) + }, + "trial_type": {"Description": "The type, category, or name of the event."}, + } + + # make sure to append any JSON fields added by the user + fname = Path(fname) + if fname.exists(): + orig_data = json.loads( + fname.read_text(encoding="utf-8"), object_pairs_hook=OrderedDict + ) + new_data = {**orig_data, **new_data} + + _write_json(fname, new_data, overwrite) + + +def _readme(datatype, fname, overwrite=False): + """Create a README file and save it. + + This will write a README file containing an MNE-BIDS citation. + If a README already exists, the behavior depends on the + `overwrite` parameter, as described below. + + Parameters + ---------- + datatype : string + The type of data contained in the raw file ('meg', 'eeg', 'ieeg') + fname : str | mne_bids.BIDSPath + Filename to save the README to. + overwrite : bool + Whether to overwrite the existing file (defaults to False). + If overwrite is True, create a new README containing an + MNE-BIDS citation. If overwrite is False, append an + MNE-BIDS citation to the existing README, unless it + already contains that citation. + """ + if os.path.isfile(fname) and not overwrite: + with open(fname, encoding="utf-8-sig") as fid: + orig_data = fid.read() + mne_bids_ref = REFERENCES["mne-bids"] in orig_data + datatype_ref = REFERENCES[datatype] in orig_data + if mne_bids_ref and datatype_ref: + return + text = "{}References\n----------\n{}{}".format( + orig_data + "\n\n", + "" if mne_bids_ref else REFERENCES["mne-bids"] + "\n\n", + "" if datatype_ref else REFERENCES[datatype] + "\n", + ) + else: + text = "References\n----------\n{}{}".format( + REFERENCES["mne-bids"] + "\n\n", REFERENCES[datatype] + "\n" + ) + + _write_text(fname, text, overwrite=True) + + +def _participants_tsv(raw, subject_id, fname, overwrite=False): + """Create a participants.tsv file and save it. + + This will append any new participant data to the current list if it + exists. Otherwise a new file will be created with the provided information. + + Parameters + ---------- + raw : mne.io.Raw + The data as MNE-Python Raw object. + subject_id : str + The subject name in BIDS compatible format ('01', '02', etc.) + fname : str | mne_bids.BIDSPath + Filename to save the participants.tsv to. + overwrite : bool + Whether to overwrite the existing file. + Defaults to False. + If there is already data for the given `subject_id` and overwrite is + False, an error will be raised. + + """ + subject_age = "n/a" + sex = "n/a" + hand = "n/a" + weight = "n/a" + height = "n/a" + subject_info = raw.info.get("subject_info", None) + + if subject_id != "emptyroom" and subject_info is not None: + # add sex + sex = _map_options( + what="sex", key=subject_info.get("sex", 0), fro="mne", to="bids" + ) + + # add handedness + hand = _map_options( + what="hand", key=subject_info.get("hand", 0), fro="mne", to="bids" + ) + + # determine the age of the participant + age = subject_info.get("birthday", None) + meas_date = raw.info.get("meas_date", None) + if isinstance(meas_date, (tuple, list, np.ndarray)): + meas_date = meas_date[0] + + if meas_date is not None and age is not None: + bday = datetime(age[0], age[1], age[2], tzinfo=timezone.utc) + if isinstance(meas_date, datetime): + meas_datetime = meas_date + else: + meas_datetime = datetime.fromtimestamp(meas_date, tz=timezone.utc) + subject_age = _age_on_date(bday, meas_datetime) + else: + subject_age = "n/a" + + # add weight and height + weight = subject_info.get("weight", "n/a") + height = subject_info.get("height", "n/a") + + subject_id = "sub-" + subject_id + data = OrderedDict(participant_id=[subject_id]) + data.update( + { + "age": [subject_age], + "sex": [sex], + "hand": [hand], + "weight": [weight], + "height": [height], + } + ) + + if os.path.exists(fname): + orig_data = _from_tsv(fname) + # whether the new data exists identically in the previous data + exact_included = _contains_row( + data=orig_data, + row_data={ + "participant_id": subject_id, + "age": subject_age, + "sex": sex, + "hand": hand, + "weight": weight, + "height": height, + }, + ) + # whether the subject id is in the previous data + sid_included = subject_id in orig_data["participant_id"] + # if the subject data provided is different to the currently existing + # data and overwrite is not True raise an error + if (sid_included and not exact_included) and not overwrite: + raise FileExistsError( + f'"{subject_id}" already exists in ' + f"the participant list. Please set " + f"overwrite to True." + ) + + # Append any columns the original data did not have, and fill them with + # n/a's. + for key in data.keys(): + if key in orig_data: + continue + + orig_data[key] = ["n/a"] * len(orig_data["participant_id"]) + + # Append any additional columns that original data had. + # Keep the original order of the data by looping over + # the original OrderedDict keys + for key in orig_data.keys(): + if key in data: + continue + + # add original value for any user-appended columns + # that were not handled by mne-bids + p_id = data["participant_id"][0] + if p_id in orig_data["participant_id"]: + row_idx = orig_data["participant_id"].index(p_id) + data[key] = [orig_data[key][row_idx]] + + # otherwise add the new data as new row + data = _combine_rows(orig_data, data, "participant_id") + + # overwrite is forced to True as all issues with overwrite == False have + # been handled by this point + _write_tsv(fname, data, True) + + +def _participants_json(fname, overwrite=False): + """Create participants.json for non-default columns in accompanying TSV. + + Parameters + ---------- + fname : str | mne_bids.BIDSPath + Output filename. + overwrite : bool + Defaults to False. + Whether to overwrite the existing data in the file. + If there is already data for the given `fname` and overwrite is False, + an error will be raised. + + """ + new_data = { + "participant_id": {"Description": "Unique participant identifier"}, + "age": { + "Description": "Age of the participant at time of testing", + "Units": "years", + }, + "sex": { + "Description": "Biological sex of the participant", + "Levels": {"F": "female", "M": "male"}, + }, + "hand": { + "Description": "Handedness of the participant", + "Levels": {"R": "right", "L": "left", "A": "ambidextrous"}, + }, + "weight": {"Description": "Body weight of the participant", "Units": "kg"}, + "height": {"Description": "Body height of the participant", "Units": "m"}, + } + + # make sure to append any JSON fields added by the user + # Note: mne-bids will overwrite age, sex and hand fields + # if `overwrite` is True + fname = Path(fname) + if fname.exists(): + orig_data = json.loads( + fname.read_text(encoding="utf-8"), object_pairs_hook=OrderedDict + ) + new_data = {**orig_data, **new_data} + + _write_json(fname, new_data, overwrite) + + +def _scans_tsv(raw, raw_fname, fname, keep_source, overwrite=False): + """Create a scans.tsv file and save it. + + Parameters + ---------- + raw : mne.io.Raw + The data as MNE-Python Raw object. + raw_fname : str | mne_bids.BIDSPath + Relative path to the raw data file. + fname : str + Filename to save the scans.tsv to. + keep_source : bool + Wehter to store``raw.filenames`` in the ``source`` column. + overwrite : bool + Defaults to False. + Whether to overwrite the existing data in the file. + If there is already data for the given `fname` and overwrite is False, + an error will be raised. + + """ + # get measurement date in UTC from the data info + meas_date = raw.info["meas_date"] + if meas_date is None: + acq_time = "n/a" + elif isinstance(meas_date, datetime): + acq_time = meas_date.strftime("%Y-%m-%dT%H:%M:%S.%fZ") + + # for fif files check whether raw file is likely to be split + raw_fnames = [raw_fname] + if raw_fname.endswith(".fif"): + # check whether fif files were split when saved + # use the files in the target directory what should be written + # to scans.tsv + datatype, basename = raw_fname.split(os.sep) + raw_dir = op.join(op.dirname(fname), datatype) + raw_files = [f for f in os.listdir(raw_dir) if f.endswith(".fif")] + if basename not in raw_files: + raw_fnames = [] + split_base = basename.replace("_meg.fif", "_split-{}") + for raw_f in raw_files: + if len(raw_f.split("_split-")) == 2: + if split_base.format(raw_f.split("_split-")[1]) == raw_f: + raw_fnames.append(op.join(datatype, raw_f)) + raw_fnames.sort() + + data = OrderedDict( + [ + ( + "filename", + ["{:s}".format(raw_f.replace(os.sep, "/")) for raw_f in raw_fnames], + ), + ("acq_time", [acq_time] * len(raw_fnames)), + ] + ) + + # add source filename if desired + if keep_source: + data["source"] = [Path(src_fname).name for src_fname in raw.filenames] + + # write out a sidecar JSON if not exists + sidecar_json_path = Path(fname).with_suffix(".json") + sidecar_json_path = get_bids_path_from_fname(sidecar_json_path) + sidecar_json = {"source": {"Description": "Original source filename."}} + + if sidecar_json_path.fpath.exists(): + update_sidecar_json(sidecar_json_path, sidecar_json) + else: + _write_json(sidecar_json_path, sidecar_json) + + if os.path.exists(fname): + orig_data = _from_tsv(fname) + # if the file name is already in the file raise an error + if raw_fname in orig_data["filename"] and not overwrite: + raise FileExistsError( + f'"{raw_fname}" already exists in ' + f"the scans list. Please set " + f"overwrite to True." + ) + + for key in data.keys(): + if key in orig_data: + continue + + # add 'n/a' if any missing columns + orig_data[key] = ["n/a"] * len(next(iter(data.values()))) + + # otherwise add the new data + data = _combine_rows(orig_data, data, "filename") + + # overwrite is forced to True as all issues with overwrite == False have + # been handled by this point + _write_tsv(fname, data, True) + + +def _load_image(image, name="image"): + nib = _import_nibabel() + if type(image) not in nib.all_image_classes: + try: + image = _path_to_str(image) + except ValueError: + # image -> str conversion in the try block was successful, + # so load the file from the specified location. We do this + # here to keep the try block as short as possible. + raise ValueError( + f"`{name}` must be a path to an MRI data " + "file or a nibabel image object, but it " + f'is of type "{type(image)}"' + ) + else: + image = nib.load(image) + + image = nib.Nifti1Image(image.dataobj, image.affine) + # XYZT_UNITS = NIFT_UNITS_MM (10 in binary or 2 in decimal) + # seems to be the default for Nifti files + # https://nifti.nimh.nih.gov/nifti-1/documentation/nifti1fields/nifti1fields_pages/xyzt_units.html + if image.header["xyzt_units"] == 0: + image.header["xyzt_units"] = np.array(10, dtype="uint8") + return image + + +def _meg_landmarks_to_mri_landmarks(meg_landmarks, trans): + """Convert landmarks from head space to MRI space. + + Parameters + ---------- + meg_landmarks : np.ndarray, shape (3, 3) + The meg landmark data: rows LPA, NAS, RPA, columns x, y, z. + trans : mne.transforms.Transform + The transformation matrix from head coordinates to MRI coordinates. + + Returns + ------- + mri_landmarks : np.ndarray, shape (3, 3) + The mri RAS landmark data converted to from m to mm. + """ + # Transform MEG landmarks into MRI space, adjust units by * 1e3 + return apply_trans(trans, meg_landmarks, move=True) * 1e3 + + +def _mri_landmarks_to_mri_voxels(mri_landmarks, t1_mgh): + """Convert landmarks from MRI surface RAS space to MRI voxel space. + + Parameters + ---------- + mri_landmarks : np.ndarray, shape (3, 3) + The MRI RAS landmark data: rows LPA, NAS, RPA, columns x, y, z. + t1_mgh : nib.MGHImage + The image data in MGH format. + + Returns + ------- + vox_landmarks : np.ndarray, shape (3, 3) + The MRI voxel-space landmark data. + """ + # Get landmarks in voxel space, using the T1 data + vox2ras_tkr_t = t1_mgh.header.get_vox2ras_tkr() + ras_tkr2vox_t = linalg.inv(vox2ras_tkr_t) + vox_landmarks = apply_trans(ras_tkr2vox_t, mri_landmarks) + + return vox_landmarks + + +def _mri_voxels_to_mri_scanner_ras(mri_landmarks, img_mgh): + """Convert landmarks from MRI voxel space to MRI scanner RAS space. + + Parameters + ---------- + mri_landmarks : np.ndarray, shape (3, 3) + The MRI RAS landmark data: rows LPA, NAS, RPA, columns x, y, z. + img_mgh : nib.MGHImage + The image data in MGH format. + + Returns + ------- + ras_landmarks : np.ndarray, shape (3, 3) + The MRI scanner RAS landmark data. + """ + # Get landmarks in voxel space, using the T1 data + vox2ras = img_mgh.header.get_vox2ras() + ras_landmarks = apply_trans(vox2ras, mri_landmarks) # in scanner RAS + return ras_landmarks + + +def _mri_scanner_ras_to_mri_voxels(ras_landmarks, img_mgh): + """Convert landmarks from MRI scanner RAS space to MRI to MRI voxel space. + + Parameters + ---------- + ras_landmarks : np.ndarray, shape (3, 3) + The MRI RAS landmark data: rows LPA, NAS, RPA, columns x, y, z. + img_mgh : nib.MGHImage + The image data in MGH format. + + Returns + ------- + vox_landmarks : np.ndarray, shape (3, 3) + The MRI voxel-space landmark data. + """ + # Get landmarks in voxel space, using the T1 data + vox2ras = img_mgh.header.get_vox2ras() + ras2vox = linalg.inv(vox2ras) + vox_landmarks = apply_trans(ras2vox, ras_landmarks) # in vox + return vox_landmarks + + +def _sidecar_json( + raw, task, manufacturer, fname, datatype, emptyroom_fname=None, overwrite=False +): + """Create a sidecar json file depending on the suffix and save it. + + The sidecar json file provides meta data about the data + of a certain datatype. + + Parameters + ---------- + raw : mne.io.Raw + The data as MNE-Python Raw object. + task : str + Name of the task the data is based on. + manufacturer : str + Manufacturer of the acquisition system. For MEG also used to define the + coordinate system for the MEG sensors. + fname : str | mne_bids.BIDSPath + Filename to save the sidecar json to. + datatype : str + Type of the data as in ALLOWED_ELECTROPHYSIO_DATATYPE. + emptyroom_fname : str | mne_bids.BIDSPath + For MEG recordings, the path to an empty-room data file to be + associated with ``raw``. Only supported for MEG. + overwrite : bool + Whether to overwrite the existing file. + Defaults to False. + + """ + sfreq = raw.info["sfreq"] + try: + powerlinefrequency = raw.info["line_freq"] + powerlinefrequency = "n/a" if powerlinefrequency is None else powerlinefrequency + except KeyError: + raise ValueError( + "PowerLineFrequency parameter is required in the sidecar files. " + "Please specify it in info['line_freq'] before saving to BIDS, " + "e.g. by running: " + " raw.info['line_freq'] = 60" + "in your script, or by passing: " + " --line_freq 60 " + "in the command line for a 60 Hz line frequency. If the frequency " + "is unknown, set it to None" + ) + + if isinstance(raw, BaseRaw): + rec_type = "continuous" + elif isinstance(raw, Epochs): + rec_type = "epoched" + else: + rec_type = "n/a" + + # determine whether any channels have to be ignored: + n_ignored = len( + [ + ch_name + for ch_name in IGNORED_CHANNELS.get(manufacturer, list()) + if ch_name in raw.ch_names + ] + ) + # all ignored channels are trigger channels at the moment... + + n_megchan = len([ch for ch in raw.info["chs"] if ch["kind"] == FIFF.FIFFV_MEG_CH]) + n_megrefchan = len( + [ch for ch in raw.info["chs"] if ch["kind"] == FIFF.FIFFV_REF_MEG_CH] + ) + n_eegchan = len([ch for ch in raw.info["chs"] if ch["kind"] == FIFF.FIFFV_EEG_CH]) + n_ecogchan = len([ch for ch in raw.info["chs"] if ch["kind"] == FIFF.FIFFV_ECOG_CH]) + n_seegchan = len([ch for ch in raw.info["chs"] if ch["kind"] == FIFF.FIFFV_SEEG_CH]) + n_eogchan = len([ch for ch in raw.info["chs"] if ch["kind"] == FIFF.FIFFV_EOG_CH]) + n_ecgchan = len([ch for ch in raw.info["chs"] if ch["kind"] == FIFF.FIFFV_ECG_CH]) + n_emgchan = len([ch for ch in raw.info["chs"] if ch["kind"] == FIFF.FIFFV_EMG_CH]) + n_miscchan = len([ch for ch in raw.info["chs"] if ch["kind"] == FIFF.FIFFV_MISC_CH]) + n_stimchan = ( + len([ch for ch in raw.info["chs"] if ch["kind"] == FIFF.FIFFV_STIM_CH]) + - n_ignored + ) + n_dbschan = len([ch for ch in raw.info["chs"] if ch["kind"] == FIFF.FIFFV_DBS_CH]) + nirs_channels = [ch for ch in raw.info["chs"] if ch["kind"] == FIFF.FIFFV_FNIRS_CH] + n_nirscwchan = len(nirs_channels) + n_nirscwsrc = len( + np.unique([ch["ch_name"].split(" ")[0].split("_")[0] for ch in nirs_channels]) + ) + n_nirscwdet = len( + np.unique([ch["ch_name"].split(" ")[0].split("_")[1] for ch in nirs_channels]) + ) + + # Set DigitizedLandmarks to True if any of LPA, RPA, NAS are found + # Set DigitizedHeadPoints to True if any "Extra" points are found + # (DigitizedHeadPoints done for Neuromag MEG files only) + digitized_head_points = False + digitized_landmark = False + if datatype == "meg" and raw.info["dig"] is not None: + for dig_point in raw.info["dig"]: + if dig_point["kind"] in [ + FIFF.FIFFV_POINT_NASION, + FIFF.FIFFV_POINT_RPA, + FIFF.FIFFV_POINT_LPA, + ]: + digitized_landmark = True + elif dig_point["kind"] == FIFF.FIFFV_POINT_EXTRA and str( + raw.filenames[0] + ).endswith(".fif"): + digitized_head_points = True + software_filters = { + "SpatialCompensation": {"GradientOrder": raw.compensation_grade} + } + + # Compile cHPI information, if any. + system, _ = _get_meg_system(raw.info) + chpi = None + hpi_freqs = [] + if datatype == "meg": + # We need to handle different data formats differently + if system == "CTF_275": + try: + mne.chpi.extract_chpi_locs_ctf(raw) + chpi = True + except RuntimeError: + chpi = False + logger.info("Could not find cHPI information in raw data.") + elif system == "KIT": + try: + mne.chpi.extract_chpi_locs_kit(raw) + chpi = True + except (RuntimeError, ValueError): + chpi = False + logger.info("Could not find cHPI information in raw data.") + elif system in ["122m", "306m"]: + n_active_hpi = mne.chpi.get_active_chpi(raw, on_missing="ignore") + chpi = bool(n_active_hpi.sum() > 0) + if chpi: + hpi_freqs, _, _ = get_chpi_info(info=raw.info, on_missing="ignore") + hpi_freqs = list(hpi_freqs) + + # Define datatype-specific JSON dictionaries + ch_info_json_common = [ + ("TaskName", task), + ("Manufacturer", manufacturer), + ("PowerLineFrequency", powerlinefrequency), + ("SamplingFrequency", sfreq), + ("SoftwareFilters", "n/a"), + ("RecordingDuration", raw.times[-1]), + ("RecordingType", rec_type), + ] + + ch_info_json_meg = [ + ("DewarPosition", "n/a"), + ("DigitizedLandmarks", digitized_landmark), + ("DigitizedHeadPoints", digitized_head_points), + ("MEGChannelCount", n_megchan), + ("MEGREFChannelCount", n_megrefchan), + ("SoftwareFilters", software_filters), + ] + + if chpi is not None: + ch_info_json_meg.append(("ContinuousHeadLocalization", chpi)) + ch_info_json_meg.append(("HeadCoilFrequency", hpi_freqs)) + + if emptyroom_fname is not None: + ch_info_json_meg.append(("AssociatedEmptyRoom", str(emptyroom_fname))) + + ch_info_json_eeg = [ + ("EEGReference", "n/a"), + ("EEGGround", "n/a"), + ("EEGPlacementScheme", _infer_eeg_placement_scheme(raw)), + ("Manufacturer", manufacturer), + ] + + ch_info_json_ieeg = [ + ("iEEGReference", "n/a"), + ("ECOGChannelCount", n_ecogchan), + ("SEEGChannelCount", n_seegchan + n_dbschan), + ] + + ch_info_json_nirs = [("Manufacturer", manufacturer)] + + ch_info_ch_counts = [ + ("EEGChannelCount", n_eegchan), + ("EOGChannelCount", n_eogchan), + ("ECGChannelCount", n_ecgchan), + ("EMGChannelCount", n_emgchan), + ("MiscChannelCount", n_miscchan), + ("TriggerChannelCount", n_stimchan), + ] + + ch_info_ch_counts_nirs = [ + ("NIRSChannelCount", n_nirscwchan), + ("NIRSSourceOptodeCount", n_nirscwsrc), + ("NIRSDetectorOptodeCount", n_nirscwdet), + ] + + # Stitch together the complete JSON dictionary + ch_info_json = ch_info_json_common + if datatype == "meg": + append_datatype_json = ch_info_json_meg + elif datatype == "eeg": + append_datatype_json = ch_info_json_eeg + elif datatype == "ieeg": + append_datatype_json = ch_info_json_ieeg + elif datatype == "nirs": + append_datatype_json = ch_info_json_nirs + ch_info_ch_counts.extend(ch_info_ch_counts_nirs) + + ch_info_json += append_datatype_json + ch_info_json += ch_info_ch_counts + ch_info_json = OrderedDict(ch_info_json) + + _write_json(fname, ch_info_json, overwrite) + + return fname + + +def _deface(image, landmarks, deface): + nib = _import_nibabel("deface MRIs") + + inset, theta = (5, 15.0) + if isinstance(deface, dict): + if "inset" in deface: + inset = deface["inset"] + if "theta" in deface: + theta = deface["theta"] + + if not _is_numeric(inset): + raise ValueError(f"inset must be numeric (float, int). Got {type(inset)}") + + if not _is_numeric(theta): + raise ValueError(f"theta must be numeric (float, int). Got {type(theta)}") + + if inset < 0: + raise ValueError("inset should be positive, Got {inset}") + + if not 0 <= theta < 90: + raise ValueError("theta should be between 0 and 90 degrees. Got {theta}") + + # get image data, make a copy + image_data = image.get_fdata().copy() + + # make indices to move around so that the image doesn't have to + idxs = np.meshgrid( + np.arange(image_data.shape[0]), + np.arange(image_data.shape[1]), + np.arange(image_data.shape[2]), + indexing="ij", + ) + idxs = np.array(idxs) # (3, *image_data.shape) + idxs = np.transpose(idxs, [1, 2, 3, 0]) # (*image_data.shape, 3) + idxs = idxs.reshape(-1, 3) # (n_voxels, 3) + + # convert to RAS by applying affine + idxs = nib.affines.apply_affine(image.affine, idxs) + + # now comes the actual defacing + # 1. move center of voxels to (nasion - inset) + # 2. rotate the head by theta from vertical + x, y, z = nib.affines.apply_affine(image.affine, landmarks)[1] + idxs = apply_trans(translation(x=-x, y=-y + inset, z=-z), idxs) + idxs = apply_trans(rotation(x=-np.pi / 2 + np.deg2rad(theta)), idxs) + idxs = idxs.reshape(image_data.shape + (3,)) + mask = idxs[..., 2] < 0 # z < middle + image_data[mask] = 0.0 + + # smooth decided against for potential lack of anonymizaton + # https://gist.github.com/alexrockhill/15043928b716a432db3a84a050b241ae + + image = nib.Nifti1Image(image_data, image.affine, image.header) + return image + + +def _write_raw_fif(raw, bids_fname): + """Save out the raw file in FIF. + + Parameters + ---------- + raw : mne.io.Raw + Raw file to save out. + bids_fname : str | mne_bids.BIDSPath + The name of the BIDS-specified file where the raw object + should be saved. + + """ + raw.save( + bids_fname, + fmt=raw.orig_format, + split_size=_FIFF_SPLIT_SIZE, + split_naming="bids", + overwrite=True, + ) + + +def _write_raw_brainvision(raw, bids_fname, events, overwrite): + """Save out the raw file in BrainVision format. + + Parameters + ---------- + raw : mne.io.Raw + Raw file to save out. + bids_fname : str + The name of the BIDS-specified file where the raw object + should be saved. + events : ndarray + The events as MNE-Python format ndaray. + overwrite : bool + Whether or not to overwrite existing files. + """ + if not check_version("pybv", PYBV_VERSION): # pragma: no cover + raise ImportError( + f"pybv >= {PYBV_VERSION} is required for converting" + " file to BrainVision format" + ) + from pybv import write_brainvision + + # Subtract raw.first_samp because brainvision marks events starting from + # the first available data point and ignores the raw.first_samp + if events is not None: + events[:, 0] -= raw.first_samp + events = events[:, [0, 2]] # reorder for pybv required order + meas_date = raw.info["meas_date"] + if meas_date is not None: + meas_date = _stamp_to_dt(meas_date) + + # pybv needs to know the units of the data for appropriate scaling + # get voltage units as micro-volts and all other units "as is" + unit = [] + for chs in raw.info["chs"]: + if chs["unit"] == FIFF.FIFF_UNIT_V: + unit.append("µV") + else: + unit.append(_unit2human.get(chs["unit"], "n/a")) + unit = [u if u not in ["NA"] else "n/a" for u in unit] + + # We enforce conversion to float32 format + # XXX: pybv can also write to int16, to do that, we need to get + # original units of data prior to conversion, and add an optimization + # function to pybv that maximizes the resolution parameter while + # ensuring that int16 can represent the data in original units. + if raw.orig_format != "single": + warn( + f'Encountered data in "{raw.orig_format}" format. ' + "Converting to float32.", + RuntimeWarning, + ) + + # Writing to float32 µV with 0.1 resolution are the pybv defaults, + # which guarantees accurate roundtrip for values >= 1e-7 µV + fmt = "binary_float32" + resolution = 1e-1 + write_brainvision( + data=raw.get_data(), + sfreq=raw.info["sfreq"], + ch_names=raw.ch_names, + ref_ch_names=None, + fname_base=op.splitext(op.basename(bids_fname))[0], + folder_out=op.dirname(bids_fname), + overwrite=overwrite, + events=events, + resolution=resolution, + unit=unit, + fmt=fmt, + meas_date=None, + ) + + +def _write_raw_edf(raw, bids_fname, overwrite): + """Store data as EDF. + + Parameters + ---------- + raw : mne.io.Raw + Raw data to save. + bids_fname : str + The output filename. + overwrite : bool + Whether to overwrite an existing file or not. + """ + assert str(bids_fname).endswith(".edf") + raw.export(bids_fname, overwrite=overwrite) + + +def _write_raw_eeglab(raw, bids_fname, overwrite): + """Store data as EEGLAB. + + Parameters + ---------- + raw : mne.io.Raw + Raw data to save. + bids_fname : str + The output filename. + overwrite : bool + Whether to overwrite an existing file or not. + """ + assert str(bids_fname).endswith(".set") + raw.export(bids_fname, overwrite=overwrite) + + +@verbose +def make_dataset_description( + *, + path, + name, + hed_version=None, + dataset_type="raw", + data_license=None, + authors=None, + acknowledgements=None, + how_to_acknowledge=None, + funding=None, + ethics_approvals=None, + references_and_links=None, + doi=None, + generated_by=None, + source_datasets=None, + overwrite=False, + verbose=None, +): + """Create a dataset_description.json file for a BIDS dataset. + + The dataset_description.json file is required in BIDS and describes + several general aspects of the dataset. You can use this function + to freely add metadata fields to this file. See the BIDS specification + for information about what each metadata field means. + + Parameters + ---------- + path : str + A path to a folder where the description will be created. + name : str + The name of this BIDS dataset. + hed_version : str + If HED tags are used: The version of the HED schema used to validate + HED tags for study. + dataset_type : str + Must be either "raw" or "derivative". Defaults to "raw". + data_license : str | None + The license under which this dataset is published. + authors : list | str | None + List of individuals who contributed to the creation/curation of the + dataset. Must be a list of str (e.g., ['a', 'b', 'c']) or a single + comma-separated str (e.g., 'a, b, c'). + acknowledgements : str | None + A str acknowledging individuals who contributed to the + creation/curation of this dataset. + how_to_acknowledge : str | None + A str describing how to acknowledge this dataset. + funding : list | str | None + List of sources of funding (e.g., grant numbers). Must be a list of + str (e.g., ['a', 'b', 'c']) or a single comma-separated str + (e.g., 'a, b, c'). + ethics_approvals : list | str | None + List of ethics committee approvals of the research protocols + and/or protocol identifiers. Must be a list of str (e.g., + ['a', 'b', 'c']) or a single comma-separated str (e.g., 'a, b, c'). + references_and_links : list | str | None + List of references to publication that contain information on the + dataset, or links. Must be a list of str (e.g., ['a', 'b', 'c']) + or a single comma-separated str (e.g., 'a, b, c'). + doi : str | None + The Digital Object Identifier of the dataset (not the corresponding + paper). Must be of the form ``doi:`` (e.g., + doi:10.5281/zenodo.3686061). + generated_by : list of dict | None + Used to specify provenance of the dataset. See BIDS specification + for details. + source_datasets : list of dict | None + Used to specify the locations and relevant attributes of all source + datasets. Each dict in the list represents one source dataset and + may contain the following keys: ``URL``, ``DOI``, ``Version``. + overwrite : bool + Whether to overwrite existing files or data in files. + Defaults to False. + If overwrite is True, provided fields will overwrite previous data. + If overwrite is False, no existing data will be overwritten or + replaced. + %(verbose)s + + Notes + ----- + The required metadata field ``BIDSVersion`` will be automatically filled in + by mne_bids. + + """ + # Convert potential string input into list of strings + convert_vars = [authors, funding, references_and_links, ethics_approvals] + convert_vars = [ + [i.strip() for i in var.split(",")] if isinstance(var, str) else var + for var in convert_vars + ] + authors, funding, references_and_links, ethics_approvals = convert_vars + + # Perform input checks + if dataset_type not in ["raw", "derivative"]: + raise ValueError('`dataset_type` must be either "raw" or ' '"derivative."') + if isinstance(doi, str): + if not doi.startswith("doi:"): + warn( + "The `doi` field in dataset_description should be of the " + "form `doi:`" + ) + + # check generated_by and source_datasets + msg_type = "{} must be a list of dicts or None." + msg_key = "found unexpected key(s) in dict: {}" + + generated_by_keys = set(["Name", "Version", "Description", "CodeURL", "Container"]) + if isinstance(generated_by, list): + if not all([isinstance(i, dict) for i in generated_by]): + raise ValueError(msg_type.format("generated_by")) + for i in generated_by: + if "Name" not in i: + raise ValueError( + '"Name" is a required field for each dict in ' "generated_by" + ) + if not set(i.keys()).issubset(generated_by_keys): + raise ValueError(msg_key.format(i.keys() - generated_by_keys)) + else: + if generated_by is not None: + raise ValueError(msg_type.format("generated_by")) + + source_ds_keys = set(["URL", "DOI", "Version"]) + if isinstance(source_datasets, list): + if not all([isinstance(i, dict) for i in source_datasets]): + raise ValueError(msg_type.format("source_datasets")) + for i in source_datasets: + if not set(i.keys()).issubset(source_ds_keys): + raise ValueError(msg_key.format(i.keys() - source_ds_keys)) + else: + if source_datasets is not None: + raise ValueError(msg_type.format("source_datasets")) + + # Prepare dataset_description.json + fname = op.join(path, "dataset_description.json") + description = OrderedDict( + [ + ("Name", name), + ("BIDSVersion", BIDS_VERSION), + ("HEDVersion", hed_version), + ("DatasetType", dataset_type), + ("License", data_license), + ("Authors", authors), + ("Acknowledgements", acknowledgements), + ("HowToAcknowledge", how_to_acknowledge), + ("Funding", funding), + ("EthicsApprovals", ethics_approvals), + ("ReferencesAndLinks", references_and_links), + ("DatasetDOI", doi), + ("GeneratedBy", generated_by), + ("SourceDatasets", source_datasets), + ] + ) + + # Handle potentially existing file contents + if op.isfile(fname): + with open(fname, encoding="utf-8-sig") as fin: + orig_cols = json.load(fin) + if "BIDSVersion" in orig_cols and orig_cols["BIDSVersion"] != BIDS_VERSION: + warnings.warn( + "Conflicting BIDSVersion found in dataset_description.json! " + "Consider setting BIDS root to a new directory and redo " + "conversion after ensuring all software has been updated. " + "Original dataset description will not be overwritten." + ) + overwrite = False + for key in description: + if description[key] is None or not overwrite: + description[key] = orig_cols.get(key, None) + + # default author to make dataset description BIDS compliant + # if the user passed an author don't overwrite, + # if there was an author there, only overwrite if `overwrite=True` + if authors is None and (description["Authors"] is None or overwrite): + description["Authors"] = ["[Unspecified]"] + + # Only write data that is not None + pop_keys = [key for key, val in description.items() if val is None] + for key in pop_keys: + description.pop(key) + _write_json(fname, description, overwrite=True) + + +@verbose +def write_raw_bids( + raw, + bids_path, + events=None, + event_id=None, + *, + anonymize=None, + format="auto", + symlink=False, + empty_room=None, + allow_preload=False, + montage=None, + acpc_aligned=False, + overwrite=False, + verbose=None, +): + """Save raw data to a BIDS-compliant folder structure. + + .. warning:: * The original file is simply copied over if the original + file format is BIDS-supported for that datatype. Otherwise, + this function will convert to a BIDS-supported file format + while warning the user. For EEG and iEEG data, conversion + will be to BrainVision format; for MEG, conversion will be + to FIFF. + + * ``mne-bids`` will infer the manufacturer information + from the file extension. If your file format is non-standard + for the manufacturer, please update the manufacturer field + in the sidecars manually. + + Parameters + ---------- + raw : mne.io.Raw + The raw data. It must be an instance of `mne.io.Raw` that is not + already loaded from disk unless ``allow_preload`` is explicitly set + to ``True``. See warning for the ``allow_preload`` parameter. + bids_path : BIDSPath + The file to write. The `mne_bids.BIDSPath` instance passed here + **must** have the ``subject``, ``task``, and ``root`` attributes set. + If the ``datatype`` attribute is not set, it will be inferred from the + recording data type found in ``raw``. In case of multiple data types, + the ``.datatype`` attribute must be set. + Example:: + + bids_path = BIDSPath(subject='01', session='01', task='testing', + acquisition='01', run='01', datatype='meg', + root='/data/BIDS') + + This will write the following files in the correct subfolder ``root``:: + + sub-01_ses-01_task-testing_acq-01_run-01_meg.fif + sub-01_ses-01_task-testing_acq-01_run-01_meg.json + sub-01_ses-01_task-testing_acq-01_run-01_channels.tsv + sub-01_ses-01_acq-01_coordsystem.json + + and the following one if ``events`` is not ``None``:: + + sub-01_ses-01_task-testing_acq-01_run-01_events.tsv + + and add a line to the following files:: + + participants.tsv + scans.tsv + + Note that the extension is automatically inferred from the raw + object. + events : path-like | np.ndarray | None + Use this parameter to specify events to write to the ``*_events.tsv`` + sidecar file, additionally to the object's :class:`~mne.Annotations` + (which are always written). + If ``path-like``, specifies the location of an MNE events file. + If an array, the MNE events array (shape: ``(n_events, 3)``). + If a path or an array and ``raw.annotations`` exist, the union of + ``events`` and ``raw.annotations`` will be written. + Mappings from event names to event codes (listed in the third + column of the MNE events array) must be specified via the ``event_id`` + parameter; otherwise, an exception is raised. If + :class:`~mne.Annotations` are present, their descriptions must be + included in ``event_id`` as well. + If ``None``, events will only be inferred from the raw object's + :class:`~mne.Annotations`. + + .. note:: + If specified, writes the union of ``events`` and + ``raw.annotations``. If you wish to **only** write + ``raw.annotations``, pass ``events=None``. If you want to + **exclude** the events in ``raw.annotations`` from being written, + call ``raw.set_annotations(None)`` before invoking this function. + + .. note:: + Descriptions of all event codes must be specified via the + ``event_id`` parameter. + + event_id : dict | None + Descriptions or names describing the event codes, if you passed + ``events``. The descriptions will be written to the ``trial_type`` + column in ``*_events.tsv``. The dictionary keys correspond to the event + description,s and the values to the event codes. You must specify a + description for all event codes appearing in ``events``. If your data + contains :class:`~mne.Annotations`, you can use this parameter to + assign event codes to each unique annotation description (mapping from + description to event code). + anonymize : dict | None + If `None` (default), no anonymization is performed. + If a dictionary, data will be anonymized depending on the dictionary + keys: ``daysback`` is a required key, ``keep_his`` is optional. + + ``daysback`` : int + Number of days by which to move back the recording date in time. + In studies with multiple subjects the relative recording date + differences between subjects can be kept by using the same number + of ``daysback`` for all subject anonymizations. ``daysback`` should + be great enough to shift the date prior to 1925 to conform with + BIDS anonymization rules. + + ``keep_his`` : bool + If ``False`` (default), all subject information next to the + recording date will be overwritten as well. If ``True``, keep + subject information apart from the recording date. + + ``keep_source`` : bool + Whether to store the name of the ``raw`` input file in the + ``source`` column of ``scans.tsv``. By default, this information + is not stored. + + format : 'auto' | 'BrainVision' | 'EDF' | 'FIF' | 'EEGLAB' + Controls the file format of the data after BIDS conversion. If + ``'auto'``, MNE-BIDS will attempt to convert the input data to BIDS + without a change of the original file format. A conversion to a + different file format will then only take place if the original file + format lacks some necessary features. + Conversion may be forced to BrainVision, EDF, or EEGLAB for (i)EEG, + and to FIF for MEG data. + symlink : bool + Instead of copying the source files, only create symbolic links to + preserve storage space. This is only allowed when not anonymizing the + data (i.e., ``anonymize`` must be ``None``). + + .. note:: + Symlinks currently only work with FIFF files. In case of split + files, only a link to the first file will be created, and + :func:`mne_bids.read_raw_bids` will correctly handle reading the + data again. + + .. note:: + Symlinks are currently only supported on macOS and Linux. We will + add support for Windows 10 at a later time. + + empty_room : mne.io.Raw | BIDSPath | None + The empty-room recording to be associated with this file. This is + only supported for MEG data. + If :class:`~mne.io.Raw`, you may pass raw data that was not preloaded + (otherwise, pass ``allow_preload=True``); i.e., it behaves similar to + the ``raw`` parameter. The session name will be automatically generated + from the raw object's ``info['meas_date']``. + If a :class:`~mne_bids.BIDSPath`, the ``root`` attribute must be the + same as in ``bids_path``. Pass ``None`` (default) if you do not wish to + specify an associated empty-room recording. + + .. versionchanged:: 0.11 + Accepts :class:`~mne.io.Raw` data. + allow_preload : bool + If ``True``, allow writing of preloaded raw objects (i.e., + ``raw.preload`` is ``True``). Because the original file is ignored, you + must specify what ``format`` to write (not ``auto``). + + .. warning:: + BIDS was originally designed for unprocessed or minimally processed + data. For this reason, by default, we prevent writing of preloaded + data that may have been modified. Only use this option when + absolutely necessary: for example, manually converting from file + formats not supported by MNE or writing preprocessed derivatives. + Be aware that these use cases are not fully supported. + montage : mne.channels.DigMontage | None + The montage with channel positions if channel position data are + to be stored in a format other than "head" (the internal MNE + coordinate frame that the data in ``raw`` is stored in). + acpc_aligned : bool + It is difficult to check whether the T1 scan is ACPC aligned which + means that "mri" coordinate space is "ACPC" BIDS coordinate space. + So, this flag is required to be True when the digitization data + is in "mri" for intracranial data to confirm that the T1 is + ACPC-aligned. + overwrite : bool + Whether to overwrite existing files or data in files. + Defaults to ``False``. + + If ``True``, any existing files with the same BIDS parameters + will be overwritten with the exception of the ``*_participants.tsv`` + and ``*_scans.tsv`` files. For these files, parts of pre-existing data + that match the current data will be replaced. For + ``*_participants.tsv``, specifically, age, sex and hand fields will be + overwritten, while any manually added fields in ``participants.json`` + and ``participants.tsv`` by a user will be retained. + If ``False``, no existing data will be overwritten or + replaced. + + %(verbose)s + + Returns + ------- + bids_path : BIDSPath + The path of the created data file. + + .. note:: + If you passed empty-room raw data via ``empty_room``, the + :class:`~mne_bids.BIDSPath` of the empty-room recording can be + retrieved via ``bids_path.find_empty_room(use_sidecar_only=True)``. + + Notes + ----- + You should ensure that ``raw.info['subject_info']`` and + ``raw.info['meas_date']`` are set to proper (not-``None``) values to allow + for the correct computation of each participant's age when creating + ``*_participants.tsv``. + + This function will convert existing `mne.Annotations` from + ``raw.annotations`` to events. Additionally, any events supplied via + ``events`` will be written too. To avoid writing of annotations, + remove them from the raw file via ``raw.set_annotations(None)`` before + invoking ``write_raw_bids``. + + To write events encoded in a ``STIM`` channel, you first need to create the + events array manually and pass it to this function: + + .. + events = mne.find_events(raw, min_duration=0.002) + write_raw_bids(..., events=events) + + See the documentation of :func:`mne.find_events` for more information on + event extraction from ``STIM`` channels. + + When anonymizing ``.edf`` files, then the file format for EDF limits + how far back we can set the recording date. Therefore, all anonymized + EDF datasets will have an internal recording date of ``01-01-1985``, + and the actual recording date will be stored in the ``scans.tsv`` + file's ``acq_time`` column. + + ``write_raw_bids`` will generate a ``dataset_description.json`` file + if it does not already exist. Minimal metadata will be written there. + If one sets ``overwrite`` to ``True`` here, it will not overwrite an + existing ``dataset_description.json`` file. + If you need to add more data there, or overwrite it, then you should + call :func:`mne_bids.make_dataset_description` directly. + + When writing EDF or BDF files, all file extensions are forced to be + lower-case, in compliance with the BIDS specification. + + See Also + -------- + mne.io.Raw.anonymize + mne.find_events + mne.Annotations + mne.events_from_annotations + + """ + if not isinstance(raw, BaseRaw): + raise ValueError("raw_file must be an instance of BaseRaw, got %s" % type(raw)) + + if raw.preload is not False and not allow_preload: + raise ValueError( + "The data has already been loaded from disk. To write it to BIDS, " + 'pass "allow_preload=True" and the "format" parameter.' + ) + + if not isinstance(bids_path, BIDSPath): + raise RuntimeError( + '"bids_path" must be a BIDSPath object. Please ' + "instantiate using mne_bids.BIDSPath()." + ) + + _validate_type( + events, + types=("path-like", np.ndarray, None), + item_name="events", + type_name="path-like, NumPy array, or None", + ) + + if symlink and sys.platform in ("win32", "cygwin"): + raise NotImplementedError( + "Symbolic links are currently not supported " + "by MNE-BIDS on Windows operating systems." + ) + + if symlink and anonymize is not None: + raise ValueError("Cannot create symlinks when anonymizing data.") + + if bids_path.root is None: + raise ValueError( + 'The root of the "bids_path" must be set. Please call ' + '"bids_path.root = " to set the root of the BIDS dataset.' + ) + + if bids_path.subject is None: + raise ValueError( + 'The subject of the "bids_path" must be set. Please call ' + '"bids_path.subject = "' + ) + + if bids_path.task is None: + raise ValueError( + 'The task of the "bids_path" must be set. Please call ' + '"bids_path.task = "' + ) + + if events is not None and event_id is None: + raise ValueError("You passed events, but no event_id dictionary.") + + _validate_type( + item=empty_room, item_name="empty_room", types=(mne.io.BaseRaw, BIDSPath, None) + ) + _validate_type(montage, (mne.channels.DigMontage, None), "montage") + _validate_type(acpc_aligned, bool, "acpc_aligned") + + raw = raw.copy() + convert = False # flag if converting not copying + + # Load file, filename, extension + if not allow_preload: + raw_fname = raw.filenames[0] + if ".ds" in op.dirname(raw.filenames[0]): + raw_fname = op.dirname(raw.filenames[0]) + # point to file containing header info for multifile systems + raw_fname = str(raw_fname).replace(".eeg", ".vhdr") + raw_fname = str(raw_fname).replace(".fdt", ".set") + raw_fname = str(raw_fname).replace(".dat", ".lay") + _, ext = _parse_ext(raw_fname) + + # force all EDF/BDF files with upper-case extension to be written as + # lower case + if ext == ".EDF": + ext = ".edf" + elif ext == ".BDF": + ext = ".bdf" + + if ext not in ALLOWED_INPUT_EXTENSIONS: + raise ValueError( + f"The input data is in a file format not supported by " + f'BIDS: "{ext}". You can try to preload the data and call ' + f'write_raw_bids() with the "allow_preload=True" and the ' + f'"format" parameters.' + ) + + if symlink and ext != ".fif": + raise NotImplementedError( + "Symlinks are currently only supported for FIFF files." + ) + + raw_orig = reader[ext](**raw._init_kwargs) + else: + if format == "BrainVision": + ext = ".vhdr" + elif format == "EDF": + ext = ".edf" + elif format == "EEGLAB": + ext = ".set" + elif format == "FIF": + ext = ".fif" + else: + msg = ( + 'For preloaded data, you must set the "format" parameter ' + "to one of: BrainVision, EDF, EEGLAB, or FIF" + ) + if format != "auto": # the default was changed + msg += f', but got: "{format}"' + + raise ValueError(msg) + + raw_orig = raw + + # Check times + if not np.array_equal(raw.times, raw_orig.times): + if len(raw.times) == len(raw_orig.times): + msg = ( + "raw.times has changed since reading from disk, but " + "write_raw_bids() doesn't allow writing modified data." + ) + else: + msg = ( + "The raw data you want to write contains {comp} time " + "points than the raw data on disk. It is possible that you " + "{guess} your data." + ) + if len(raw.times) < len(raw_orig.times): + msg = msg.format(comp="fewer", guess="cropped") + elif len(raw.times) > len(raw_orig.times): + msg = msg.format(comp="more", guess="concatenated") + + msg += ( + " To write the data, please preload it and pass " + '"allow_preload=True" and the "format" parameter to ' + "write_raw_bids()." + ) + raise ValueError(msg) + + # Initialize BIDSPath + datatype = _handle_datatype(raw, bids_path.datatype) + bids_path = bids_path.copy().update( + datatype=datatype, suffix=datatype, extension=ext + ) + + # Check whether provided info and raw indicates valid MEG emptyroom data + data_is_emptyroom = False + if ( + bids_path.datatype == "meg" + and bids_path.subject == "emptyroom" + and bids_path.task == "noise" + ): + data_is_emptyroom = True + # check the session date provided is consistent with the value in raw + meas_date = raw.info.get("meas_date", None) + if meas_date is not None: + if not isinstance(meas_date, datetime): + meas_date = datetime.fromtimestamp(meas_date[0], tz=timezone.utc) + + if anonymize is not None and "daysback" in anonymize: + meas_date = meas_date - timedelta(anonymize["daysback"]) + er_date = meas_date.strftime("%Y%m%d") + bids_path = bids_path.copy().update(session=er_date) + else: + er_date = meas_date.strftime("%Y%m%d") + + if er_date != bids_path.session: + raise ValueError( + f"The date provided for the empty-room session " + f"({bids_path.session}) doesn't match the empty-room " + f"recording date found in the data's info structure " + f"({er_date})." + ) + + associated_er_path = None + + if isinstance(empty_room, mne.io.BaseRaw): + er_date = empty_room.info["meas_date"] + if not er_date: + raise ValueError( + "The empty-room raw data must have a valid measurement date " + 'set. Please update its info["meas_date"] field.' + ) + er_session = er_date.strftime("%Y%m%d") + er_bids_path = bids_path.copy().update( + subject="emptyroom", session=er_session, task="noise", run=None + ) + write_raw_bids( + raw=empty_room, + bids_path=er_bids_path, + events=None, + event_id=None, + anonymize=anonymize, + format=format, + symlink=symlink, + allow_preload=allow_preload, + montage=montage, + acpc_aligned=acpc_aligned, + overwrite=overwrite, + verbose=verbose, + ) + associated_er_path = er_bids_path.fpath + del er_bids_path, er_date, er_session + elif isinstance(empty_room, BIDSPath): + if bids_path.datatype != "meg": + raise ValueError('"empty_room" is only supported for ' "MEG data.") + if data_is_emptyroom: + raise ValueError( + "You cannot write empty-room data and pass " + '"empty_room" at the same time.' + ) + if bids_path.root != empty_room.root: + raise ValueError( + "The MEG data and its associated empty-room " + "recording must share the same BIDS root." + ) + associated_er_path = empty_room.fpath + + if associated_er_path is not None: + if not associated_er_path.exists(): + raise FileNotFoundError( + f"Empty-room data file not found: " f"{associated_er_path}" + ) + + # Turn it into a path relative to the BIDS root + associated_er_path = Path( + str(associated_er_path).replace(str(bids_path.root), "") + ) + # Ensure it works on Windows too + associated_er_path = associated_er_path.as_posix() + + # In case of an "emptyroom" subject, BIDSPath() will raise + # an exception if we don't provide a valid task ("noise"). Now, + # scans_fname, electrodes_fname, and coordsystem_fname must NOT include + # the task entity. Therefore, we cannot generate them with + # BIDSPath() directly. Instead, we use BIDSPath() directly + # as it does not make any advanced check. + + data_path = bids_path.mkdir().directory + + # create *_scans.tsv + session_path = BIDSPath( + subject=bids_path.subject, session=bids_path.session, root=bids_path.root + ) + scans_path = session_path.copy().update(suffix="scans", extension=".tsv") + + # create *_coordsystem.json + coordsystem_path = session_path.copy().update( + acquisition=bids_path.acquisition, + space=bids_path.space, + datatype=bids_path.datatype, + suffix="coordsystem", + extension=".json", + ) + + # For the remaining files, we can use BIDSPath to alter. + readme_fname = op.join(bids_path.root, "README") + participants_tsv_fname = op.join(bids_path.root, "participants.tsv") + participants_json_fname = participants_tsv_fname.replace(".tsv", ".json") + + sidecar_path = bids_path.copy().update(suffix=bids_path.datatype, extension=".json") + events_tsv_path = bids_path.copy().update(suffix="events", extension=".tsv") + events_json_path = events_tsv_path.copy().update(extension=".json") + channels_path = bids_path.copy().update(suffix="channels", extension=".tsv") + + # Anonymize + keep_source = False + if anonymize is not None: + daysback, keep_his, keep_source = _check_anonymize(anonymize, raw, ext) + raw.anonymize(daysback=daysback, keep_his=keep_his) + + if bids_path.datatype == "meg" and ext != ".fif": + warn("Converting to FIF for anonymization") + convert = True + bids_path.update(extension=".fif") + elif bids_path.datatype in ["eeg", "ieeg"]: + if ext not in [".vhdr", ".edf", ".bdf", ".EDF"]: + warn("Converting data files to BrainVision format for anonymization") + convert = True + bids_path.update(extension=".vhdr") + # Read in Raw object and extract metadata from Raw object if needed + orient = ORIENTATION.get(ext, "n/a") + unit = EXT_TO_UNIT_MAP.get(ext, "n/a") + manufacturer = MANUFACTURERS.get(ext, "n/a") + + # save readme file unless it already exists + # XXX: can include README overwrite in future if using a template API + # XXX: see https://github.com/mne-tools/mne-bids/issues/551 + _readme(bids_path.datatype, readme_fname, False) + + # save all participants meta data + _participants_tsv( + raw=raw, + subject_id=bids_path.subject, + fname=participants_tsv_fname, + overwrite=overwrite, + ) + _participants_json(participants_json_fname, True) + + # for MEG, we only write coordinate system + if bids_path.datatype == "meg" and not data_is_emptyroom: + if bids_path.space is None: + sensor_coord_system = orient + elif orient == "n/a": + sensor_coord_system = bids_path.space + elif bids_path.space in BIDS_STANDARD_TEMPLATE_COORDINATE_SYSTEMS: + sensor_coord_system = bids_path.space + elif orient != bids_path.space: + raise ValueError( + f"BIDSPath.space {bids_path.space} conflicts " + f"with filetype {ext} which has coordinate " + f"frame {orient}" + ) + _write_coordsystem_json( + raw=raw, + unit=unit, + hpi_coord_system=orient, + sensor_coord_system=sensor_coord_system, + fname=coordsystem_path.fpath, + datatype=bids_path.datatype, + overwrite=overwrite, + ) + _write_coordsystem_json( + raw=raw, + unit=unit, + hpi_coord_system=orient, + sensor_coord_system=sensor_coord_system, + fname=coordsystem_path.fpath, + datatype=bids_path.datatype, + overwrite=overwrite, + ) + elif bids_path.datatype in ["eeg", "ieeg", "nirs"]: + # We only write electrodes.tsv and accompanying coordsystem.json + # if we have an available DigMontage + if montage is not None or (raw.info["dig"] is not None and raw.info["dig"]): + _write_dig_bids(bids_path, raw, montage, acpc_aligned, overwrite) + else: + logger.info( + f"Writing of electrodes.tsv is not supported " + f'for data type "{bids_path.datatype}". Skipping ...' + ) + + # Write events. + if not data_is_emptyroom: + events_array, event_dur, event_desc_id_map = _read_events( + events, event_id, raw, bids_path=bids_path + ) + if events_array.size != 0: + _events_tsv( + events=events_array, + durations=event_dur, + raw=raw, + fname=events_tsv_path.fpath, + trial_type=event_desc_id_map, + overwrite=overwrite, + ) + _events_json(fname=events_json_path.fpath, overwrite=overwrite) + # Kepp events_array around for BrainVision writing below. + del event_desc_id_map, events, event_id, event_dur + + # make dataset description and add template data if it does not + # already exist. Always set overwrite to False here. If users + # want to edit their dataset_description, they can directly call + # this function. + make_dataset_description(path=bids_path.root, name=" ", overwrite=False) + + _sidecar_json( + raw, + task=bids_path.task, + manufacturer=manufacturer, + fname=sidecar_path.fpath, + datatype=bids_path.datatype, + emptyroom_fname=associated_er_path, + overwrite=overwrite, + ) + _channels_tsv(raw, channels_path.fpath, overwrite) + + # create parent directories if needed + _mkdir_p(os.path.dirname(data_path)) + + # If not already converting for anonymization, we may still need to do it + # if current format not BIDS compliant + if not convert: + convert = ext not in ALLOWED_DATATYPE_EXTENSIONS[bids_path.datatype] + + if convert and symlink: + raise RuntimeError( + "The input file format is not supported by the BIDS standard. " + "To store your data, MNE-BIDS would have to convert it. " + "However, this is not possible since you set symlink=True. " + "Deactivate symbolic links by passing symlink=False to allow " + "file format conversion." + ) + + # check if there is an BIDS-unsupported MEG format + if bids_path.datatype == "meg" and convert and not anonymize: + raise ValueError( + f"Got file extension {ext} for MEG data, " + f"expected one of " + f"{', '.join(sorted(ALLOWED_DATATYPE_EXTENSIONS['meg']))}" + ) + + if not convert: + logger.info(f"Copying data files to {bids_path.fpath.name}") + + # If users desire a certain format, will handle auto-conversion + if format != "auto": + if format == "BrainVision" and bids_path.datatype in ["ieeg", "eeg"]: + convert = True + bids_path.update(extension=".vhdr") + elif format == "EDF" and bids_path.datatype in ["ieeg", "eeg"]: + convert = True + bids_path.update(extension=".edf") + elif format == "EEGLAB" and bids_path.datatype in ["ieeg", "eeg"]: + convert = True + bids_path.update(extension=".set") + elif format == "FIF" and bids_path.datatype == "meg": + convert = True + bids_path.update(extension=".fif") + elif all(format not in values for values in CONVERT_FORMATS.values()): + raise ValueError( + f'The input "format" {format} is not an ' + f"accepted input format for `write_raw_bids`. " + f"Please use one of {CONVERT_FORMATS[datatype]} " + f"for {datatype} datatype." + ) + elif format not in CONVERT_FORMATS[datatype]: + raise ValueError( + f'The input "format" {format} is not an ' + f"accepted input format for {datatype} datatype. " + f"Please use one of {CONVERT_FORMATS[datatype]} " + f"for {datatype} datatype." + ) + + # raise error when trying to copy files (copyfile_*) into same location + # (src == dest, see https://github.com/mne-tools/mne-bids/issues/867) + if ( + bids_path.fpath.exists() + and not convert + and bids_path.fpath.as_posix() == Path(raw_fname).as_posix() + ): + raise FileExistsError( + f'Desired output BIDSPath ("{bids_path.fpath}") is the source' + " file. Please pass a different output BIDSPath, or set" + ' `format` to something other than "auto".' + ) + + # otherwise if the BIDSPath currently exists, check if we + # would like to overwrite the existing dataset + if bids_path.fpath.exists(): + if overwrite: + # Need to load data before removing its source + raw.load_data() + if bids_path.fpath.is_dir(): + shutil.rmtree(bids_path.fpath) + else: + bids_path.fpath.unlink() + else: + raise FileExistsError( + f'"{bids_path.fpath}" already exists. Please set overwrite to True.' + ) + + # File saving branching logic + if convert: + if bids_path.datatype == "meg": + _write_raw_fif( + raw, + ( + op.join(data_path, bids_path.basename) + if ext == ".pdf" + else bids_path.fpath + ), + ) + elif bids_path.datatype in ["eeg", "ieeg"] and format == "EDF": + warn("Converting data files to EDF format") + _write_raw_edf(raw, bids_path.fpath, overwrite=overwrite) + elif bids_path.datatype in ["eeg", "ieeg"] and format == "EEGLAB": + warn("Converting data files to EEGLAB format") + _write_raw_eeglab(raw, bids_path.fpath, overwrite=overwrite) + else: + warn("Converting data files to BrainVision format") + bids_path.update(suffix=bids_path.datatype, extension=".vhdr") + # XXX Should we write durations here too? + _write_raw_brainvision( + raw, bids_path.fpath, events=events_array, overwrite=overwrite + ) + elif ext == ".fif": + if symlink: + link_target = Path(raw.filenames[0]) + link_path = bids_path.fpath + link_path.symlink_to(link_target) + else: + _write_raw_fif(raw, bids_path) + # CTF data is saved and renamed in a directory + elif ext == ".ds": + copyfile_ctf(raw_fname, bids_path) + # BrainVision is multifile, copy over all of them and fix pointers + elif ext == ".vhdr": + copyfile_brainvision(raw_fname, bids_path, anonymize=anonymize) + elif ext in [".edf", ".EDF", ".bdf", ".BDF"]: + if anonymize is not None: + warn( + "EDF/EDF+/BDF files contain two fields for recording dates." + "Due to file format limitations, one of these fields only " + "supports 2-digit years. The date for that field will be " + "set to 85 (i.e., 1985), the earliest possible date. " + "The true anonymized date is stored in the scans.tsv file." + ) + copyfile_edf(raw_fname, bids_path, anonymize=anonymize) + # EEGLAB .set might be accompanied by a .fdt - find out and copy it too + elif ext == ".set": + copyfile_eeglab(raw_fname, bids_path) + elif ext == ".pdf": + raw_dir = op.join(data_path, op.splitext(bids_path.basename)[0]) + _mkdir_p(raw_dir) + copyfile_bti(raw_orig, raw_dir) + elif ext in [".con", ".sqd"]: + copyfile_kit( + raw_fname, + bids_path.fpath, + bids_path.subject, + bids_path.session, + bids_path.task, + bids_path.run, + raw._init_kwargs, + ) + else: + # ext may be .snirf + shutil.copyfile(raw_fname, bids_path) + + # write to the scans.tsv file the output file written + scan_relative_fpath = op.join(bids_path.datatype, bids_path.fpath.name) + _scans_tsv( + raw, + raw_fname=scan_relative_fpath, + fname=scans_path.fpath, + keep_source=keep_source, + overwrite=overwrite, + ) + logger.info(f"Wrote {scans_path.fpath} entry with " f"{scan_relative_fpath}.") + + return bids_path + + +def get_anat_landmarks(image, info, trans, fs_subject, fs_subjects_dir=None): + """Get anatomical landmarks in MRI voxel coordinates. + + This function transforms the fiducial points from "head" to MRI "voxel" + coordinate space. The landmarks obtained are defined w.r.t. the MRI passed + via the ``image`` parameter. + + Parameters + ---------- + image : path-like | mne_bids.BIDSPath | NibabelImageObject + Path to an MRI scan (e.g. T1w) of the subject. Can be in any format + readable by nibabel. Can also be a nibabel image object of an + MRI scan. Will be written as a .nii.gz file. + info : mne.Info + The measurement information from an electrophysiology recording of + the subject with the anatomical landmarks stored in its + :class:`mne.channels.DigMontage`. + trans : mne.transforms.Transform | path-like + The transformation matrix from head to MRI coordinates. Can + also be a string pointing to a ``.trans`` file containing the + transformation matrix. + fs_subject : str + The subject identifier used for FreeSurfer. Must be provided to write + the anatomical landmarks if they are not provided in MRI voxel space. + This is because the head coordinate of a + :class:`mne.channels.DigMontage` is aligned using FreeSurfer surfaces. + fs_subjects_dir : path-like | None + The FreeSurfer subjects directory. If ``None``, defaults to the + ``SUBJECTS_DIR`` environment variable. Must be provided to write + anatomical landmarks if they are not provided in MRI voxel space. + + Returns + ------- + landmarks : mne.channels.DigMontage + A montage with the landmarks in MRI voxel space. + """ + nib = _import_nibabel("get anatomical landmarks") + coords_dict, coord_frame = _get_fid_coords(info["dig"]) + if coord_frame != FIFF.FIFFV_COORD_HEAD: + raise ValueError( + "Fiducial coordinates in `info` must be in " + f"the head coordinate frame, got {coord_frame}" + ) + landmarks = np.asarray( + (coords_dict["lpa"], coords_dict["nasion"], coords_dict["rpa"]) + ) + + # get trans and ensure it is from head to MRI + trans, _ = _get_trans(trans, fro="head", to="mri") + landmarks = _meg_landmarks_to_mri_landmarks(landmarks, trans) + + # Get FS T1 image in MGH format + t1w_mgh = _get_t1w_mgh(fs_subject, fs_subjects_dir) + + # FS MGH image: go to T1 voxel space from surface RAS/TkReg RAS/freesurfer + landmarks = _mri_landmarks_to_mri_voxels(landmarks, t1w_mgh) + + # FS MGH image: go to T1 scanner space from T1 voxel space + landmarks = _mri_voxels_to_mri_scanner_ras(landmarks, t1w_mgh) + + # Input image: go to T1 voxel space from T1 scanner space + if isinstance(image, BIDSPath): + image = image.fpath + img_nii = _load_image(image, name="image") + img_mgh = nib.MGHImage(img_nii.dataobj, img_nii.affine) + landmarks = _mri_scanner_ras_to_mri_voxels(landmarks, img_mgh) + + landmarks = mne.channels.make_dig_montage( + lpa=landmarks[0], nasion=landmarks[1], rpa=landmarks[2], coord_frame="mri_voxel" + ) + + return landmarks + + +def _get_t1w_mgh(fs_subject, fs_subjects_dir): + """Return the T1w image in MGH format.""" + import nibabel as nib + + fs_subjects_dir = get_subjects_dir(fs_subjects_dir, raise_error=True) + t1_fname = Path(fs_subjects_dir) / fs_subject / "mri" / "T1.mgz" + if not t1_fname.exists(): + raise ValueError( + "Freesurfer recon-all subject folder " + "is incorrect or improperly formatted, " + f"got {Path(fs_subjects_dir) / fs_subject}" + ) + t1w_img = _load_image(str(t1_fname), name="T1.mgz") + t1w_mgh = nib.MGHImage(t1w_img.dataobj, t1w_img.affine) + return t1w_mgh + + +def _get_landmarks(landmarks, image_nii, kind=""): + import nibabel as nib + + if isinstance(landmarks, (str, Path)): + landmarks, coord_frame = read_fiducials(landmarks) + landmarks = np.array( + [landmark["r"] for landmark in landmarks], dtype=float + ) # unpack + else: + # Prepare to write the sidecar JSON, extract MEG landmarks + coords_dict, coord_frame = _get_fid_coords(landmarks.dig) + landmarks = np.asarray( + (coords_dict["lpa"], coords_dict["nasion"], coords_dict["rpa"]) + ) + + # check if coord frame is supported + if coord_frame not in (FIFF.FIFFV_MNE_COORD_MRI_VOXEL, FIFF.FIFFV_MNE_COORD_RAS): + raise ValueError(f"Coordinate frame not supported: {coord_frame}") + + # convert to voxels from scanner RAS to voxels + if coord_frame == FIFF.FIFFV_MNE_COORD_RAS: + # Make MGH image for header properties + img_mgh = nib.MGHImage(image_nii.dataobj, image_nii.affine) + landmarks = _mri_scanner_ras_to_mri_voxels(landmarks * 1e3, img_mgh) + + suffix = f"_{kind}" if kind else "" + + # Write sidecar.json + img_json = { + "LPA" + suffix: list(landmarks[0, :]), + "NAS" + suffix: list(landmarks[1, :]), + "RPA" + suffix: list(landmarks[2, :]), + } + return img_json, landmarks + + +@verbose +def write_anat( + image, bids_path, landmarks=None, deface=False, overwrite=False, verbose=None +): + """Put anatomical MRI data into a BIDS format. + + Given an MRI scan, format and store the MR data according to BIDS in the + correct location inside the specified :class:`mne_bids.BIDSPath`. If a + transformation matrix is supplied, this information will be stored in a + sidecar JSON file. + + .. note:: To generate the JSON sidecar with anatomical landmark + coordinates ("fiducials"), you need to pass the landmarks via + the ``landmarks`` parameter. :func:`mne_bids.get_anat_landmarks` + may be useful for getting the ``landmarks``. + + Parameters + ---------- + image : path-like | NibabelImageObject + Path to an MRI scan (e.g. T1w) of the subject. Can be in any format + readable by nibabel. Can also be a nibabel image object of an + MRI scan. Will be written as a .nii.gz file. + bids_path : BIDSPath + The file to write. The :class:`mne_bids.BIDSPath` instance passed here + **must** have the ``root`` and ``subject`` attributes set. + The suffix is assumed to be ``'T1w'`` if not present. It can + also be ``'FLASH'``, for example, to indicate FLASH MRI. + landmarks : mne.channels.DigMontage | path-like | dict | None + The montage or path to a montage with landmarks that can be + passed to provide information for defacing. Landmarks can be determined + from the head model using `mne coreg` GUI, or they can be determined + from the MRI using ``freeview``. If a dictionary is passed, then the + values must be instances of :class:`~mne.channels.DigMontage` or + path-like objects pointing to a :class:`~mne.channels.DigMontage` + stored on disk, and the keys of the must be strings + (e.g. ``'session-1'``) which will be used as naming suffix for the + landmarks in the sidecar JSON file. If ``None``, no sidecar JSON file + will be created. + deface : bool | dict + If False, no defacing is performed. + If ``True``, deface with default parameters using the provided + ``landmarks``. If multiple landmarks are provided, will + use the ones with the suffix ``'deface'``; if no landmarks with this + suffix exist, will use the first ones in the ``landmarks`` dictionary. + If dict, accepts the following keys: + + - `inset`: how far back in voxels to start defacing + relative to the nasion (default 5) + + - `theta`: is the angle of the defacing shear in degrees relative + to vertical (default 15). + + overwrite : bool + Whether to overwrite existing files or data in files. + Defaults to False. + If overwrite is True, any existing files with the same BIDS parameters + will be overwritten with the exception of the `participants.tsv` and + `scans.tsv` files. For these files, parts of pre-existing data that + match the current data will be replaced. + If overwrite is False, no existing data will be overwritten or + replaced. + %(verbose)s + + Returns + ------- + bids_path : BIDSPath + Path to the written MRI data. + """ + nib = _import_nibabel("write anatomical MRI data") + + write_sidecar = landmarks is not None + + if deface and landmarks is None: + raise ValueError("`landmarks` must be provided to deface the image") + + # Check if the root is available + if bids_path.root is None: + raise ValueError( + 'The root of the "bids_path" must be set. ' + 'Please use `bids_path.update(root="")` ' + "to set the root of the BIDS folder to read." + ) + # create a copy + bids_path = bids_path.copy() + + # BIDS demands anatomical scans have no task associated with them + bids_path.update(task=None) + + # XXX For now, only support writing a single run. + bids_path.update(run=None) + + # this file is anat + if bids_path.datatype is None: + bids_path.update(datatype="anat") + + # default to T1w + if not bids_path.suffix: + bids_path.update(suffix="T1w") + + # data is compressed Nifti + bids_path.update(extension=".nii.gz") + + # create the directory for the MRI data + bids_path.directory.mkdir(exist_ok=True, parents=True) + + # Try to read our MRI file and convert to MGH representation + image_nii = _load_image(image) + + # Check if we have necessary conditions for writing a sidecar JSON + if write_sidecar: + if not isinstance(landmarks, dict): + landmarks = {"": landmarks} + img_json = {} + for kind, this_landmarks in landmarks.items(): + img_json.update(_get_landmarks(this_landmarks, image_nii, kind=kind)[0]) + img_json = {"AnatomicalLandmarkCoordinates": img_json} + fname = bids_path.copy().update(extension=".json") + if op.isfile(fname) and not overwrite: + raise OSError( + "Wanted to write a file but it already exists and " + f'`overwrite` is set to False. File: "{fname}"' + ) + _write_json(fname, img_json, overwrite) + + if deface: + landmarks_deface = landmarks.get("deface") + if landmarks_deface is None: + # Take first one if none is specified for defacing. + landmarks_deface = next(iter(landmarks.items()))[1] + _, landmarks_deface = _get_landmarks(landmarks_deface, image_nii) + image_nii = _deface(image_nii, landmarks_deface, deface) + + # Save anatomical data + if op.exists(bids_path): + if overwrite: + os.remove(bids_path) + else: + raise OSError( + f"Wanted to write a file but it already exists and " + f'`overwrite` is set to False. File: "{bids_path}"' + ) + + nib.save(image_nii, bids_path.fpath) + + return bids_path + + +@verbose +def mark_channels(bids_path, *, ch_names, status, descriptions=None, verbose=None): + """Update status and description of channels in an existing BIDS dataset. + + Parameters + ---------- + bids_path : BIDSPath + The recording to update. The :class:`mne_bids.BIDSPath` instance passed + here **must** have the ``.root`` attribute set. The ``.datatype`` + attribute **may** be set. If ``.datatype`` is not set and only one data + type (e.g., only EEG or MEG data) is present in the dataset, it will be + selected automatically. + ch_names : str | list of str + The names of the channel(s) to mark with a ``status`` and possibly a + ``description``. Can be an empty list to indicate all channel names. + status : 'good' | 'bad' | list of str + The status of the channels ('good', or 'bad'). Default is 'bad'. If it + is a list, then must be a list of 'good', or 'bad' that has the same + length as ``ch_names``. + descriptions : None | str | list of str + Descriptions of the reasons that lead to the exclusion of the + channel(s). If a list, it must match the length of ``ch_names``. + If ``None``, no descriptions are added. + %(verbose)s + + Examples + -------- + Mark a single channel as bad. + + >>> root = Path('./mne_bids/tests/data/tiny_bids').absolute() + >>> bids_path = BIDSPath(subject='01', task='rest', session='eeg', + ... datatype='eeg', root=root) + >>> mark_channels(bids_path=bids_path, ch_names='C4', status='bad', + ... verbose=False) + + Mark multiple channels as bad, and add a description as to why. + + >>> bads = ['C3', 'PO10'] + >>> descriptions = ['very noisy', 'continuously flat'] + >>> mark_channels(bids_path, ch_names=bads, status='bad', + ... descriptions=descriptions, verbose=False) + + Mark all channels with a new description, while keeping them as a "good" + channel. + + >>> descriptions = ['resected', 'resected'] + >>> mark_channels(bids_path=bids_path, ch_names=['C3', 'C4'], + ... descriptions=descriptions, status='good', + ... verbose=False) + """ + if not isinstance(bids_path, BIDSPath): + raise RuntimeError( + '"bids_path" must be a BIDSPath object. Please ' + "instantiate using mne_bids.BIDSPath()." + ) + + if bids_path.root is None: + raise ValueError( + 'The root of the "bids_path" must be set. ' + 'Please use `bids_path.update(root="")` ' + "to set the root of the BIDS folder to read." + ) + + # Read sidecar file + channels_fname = _find_matching_sidecar( + bids_path, suffix="channels", extension=".tsv" + ) + tsv_data = _from_tsv(channels_fname) + + # if an empty list is passed in, then these are the entire list + # of channels + if ch_names == []: + ch_names = tsv_data["name"] + elif isinstance(ch_names, str): + ch_names = [ch_names] + + # set descriptions based on how it's passed in + if isinstance(descriptions, str): + descriptions = [descriptions] * len(ch_names) + elif not descriptions: + descriptions = [None] * len(ch_names) + + # make sure statuses is a list of strings + if isinstance(status, str): + status = [status] * len(ch_names) + + if len(ch_names) != len(descriptions): + raise ValueError("Number of channels and descriptions must match.") + + if len(status) != len(ch_names): + raise ValueError( + f"If status is a list of {len(status)} statuses, " + f"then it must have the same length as ch_names " + f"({len(ch_names)})." + ) + + if not all(status in ["good", "bad"] for status in status): + raise ValueError( + "Setting the status of a channel must only be " '"good", or "bad".' + ) + + # Read sidecar and create required columns if they do not exist. + if "status" not in tsv_data: + logger.info('No "status" column found in input file. Creating.') + tsv_data["status"] = ["good"] * len(tsv_data["name"]) + + if "status_description" not in tsv_data: + logger.info('No "status_description" column found in input file. ' "Creating.") + tsv_data["status_description"] = ["n/a"] * len(tsv_data["name"]) + + # Now actually mark the user-requested channels as bad. + for ch_name, status_, description in zip(ch_names, status, descriptions): + if ch_name not in tsv_data["name"]: + raise ValueError(f"Channel {ch_name} not found in dataset!") + + idx = tsv_data["name"].index(ch_name) + logger.info( + f"Processing channel {ch_name}:\n" + f" status: bad\n" + f" description: {description}" + ) + tsv_data["status"][idx] = status_ + + # only write if the description was passed in + if description is not None: + tsv_data["status_description"][idx] = description + + _write_tsv(channels_fname, tsv_data, overwrite=True) + + +@verbose +def write_meg_calibration(calibration, bids_path, *, verbose=None): + """Write the Elekta/Neuromag/MEGIN fine-calibration matrix to disk. + + Parameters + ---------- + calibration : path-like | dict + Either the path of the ``.dat`` file containing the file-calibration + matrix, or the dictionary returned by + :func:`mne.preprocessing.read_fine_calibration`. + bids_path : BIDSPath + A :class:`mne_bids.BIDSPath` instance with at least ``root`` and + ``subject`` set, and that ``datatype`` is either ``'meg'`` or + ``None``. + %(verbose)s + + Examples + -------- + >>> data_path = mne.datasets.testing.data_path(download=False) # doctest: +SKIP + >>> calibration_fname = op.join(data_path, 'SSS', 'sss_cal_3053.dat') # doctest: +SKIP + >>> bids_path = BIDSPath(subject='01', session='test', + ... root=op.join(data_path, 'mne_bids')) # doctest: +SKIP + >>> write_meg_calibration(calibration_fname, bids_path) # doctest: +SKIP + Writing fine-calibration file to ...sub-01_ses-test_acq-calibration_meg.dat... + """ # noqa: E501 + if bids_path.root is None or bids_path.subject is None: + raise ValueError("bids_path must have root and subject set.") + if bids_path.datatype not in (None, "meg"): + raise ValueError( + "Can only write fine-calibration information for MEG datasets." + ) + + _validate_type( + calibration, + types=("path-like", dict), + item_name="calibration", + type_name="path or dictionary", + ) + + if isinstance(calibration, dict) and ( + "ch_names" not in calibration + or "locs" not in calibration + or "imb_cals" not in calibration + ): + raise ValueError( + "The dictionary you passed does not appear to be a " + "proper fine-calibration dict. Please only pass the " + "output of " + "mne.preprocessing.read_fine_calibration(), or a " + "filename." + ) + + if not isinstance(calibration, dict): + calibration = mne.preprocessing.read_fine_calibration(calibration) + + out_path = BIDSPath( + subject=bids_path.subject, + session=bids_path.session, + acquisition="calibration", + suffix="meg", + extension=".dat", + datatype="meg", + root=bids_path.root, + ) + + logger.info(f"Writing fine-calibration file to {out_path}") + out_path.mkdir() + mne.preprocessing.write_fine_calibration( + fname=str(out_path), calibration=calibration + ) + + +@verbose +def write_meg_crosstalk(fname, bids_path, verbose=None): + """Write the Elekta/Neuromag/MEGIN crosstalk information to disk. + + Parameters + ---------- + fname : path-like + The path of the ``FIFF`` file containing the crosstalk information. + bids_path : BIDSPath + A :class:`mne_bids.BIDSPath` instance with at least ``root`` and + ``subject`` set, and that ``datatype`` is either ``'meg'`` or + ``None``. + %(verbose)s + + Examples + -------- + >>> data_path = mne.datasets.testing.data_path(download=False) # doctest: +SKIP + >>> crosstalk_fname = op.join(data_path, 'SSS', 'ct_sparse.fif') # doctest: +SKIP + >>> bids_path = BIDSPath(subject='01', session='test', + ... root=op.join(data_path, 'mne_bids')) # doctest: +SKIP + >>> write_meg_crosstalk(crosstalk_fname, bids_path) # doctest: +SKIP + Writing crosstalk file to ...sub-01_ses-test_acq-crosstalk_meg.fif + """ # noqa: E501 + if bids_path.root is None or bids_path.subject is None: + raise ValueError("bids_path must have root and subject set.") + if bids_path.datatype not in (None, "meg"): + raise ValueError( + "Can only write fine-calibration information for MEG datasets." + ) + + _validate_type(fname, types=("path-like",), item_name="fname") + + # MNE doesn't have public reader and writer functions for crosstalk data, + # so just copy the original file. Use shutil.copyfile() to only copy file + # contents, but not metadata & permissions. + out_path = BIDSPath( + subject=bids_path.subject, + session=bids_path.session, + acquisition="crosstalk", + suffix="meg", + extension=".fif", + datatype="meg", + root=bids_path.root, + ) + + logger.info(f"Writing crosstalk file to {out_path}") + out_path.mkdir() + shutil.copyfile(src=fname, dst=str(out_path)) + + +def _get_daysback( + *, bids_paths: list[BIDSPath], rng: np.random.Generator, show_progress_thresh: int +) -> int: + """Try to find a suitable "daysback" for anonymization. + + Parameters + ---------- + bids_paths + The BIDSPath instances to consider. Will be filtered down in this + function to reduce run time (only one file run per session). + rng + The RNG to use for selecting a `daysback` from the valid range. + show_progress_thresh + After narrowing down the files to query for their measurement date, + show a progress bar if >= this number of files remain. + """ + bids_paths_for_daysback = dict() + + # Only consider one run in each session to reduce the amount of files + # we need to access. + for bids_path in bids_paths: + subject = bids_path.subject + session = bids_path.session + datatype = bids_path.datatype + + if subject not in bids_paths_for_daysback: + bids_paths_for_daysback[subject] = [bids_path] + continue + elif session is None: + # Keep any one run for each data type + if datatype not in [p.datatype for p in bids_paths_for_daysback[subject]]: + bids_paths_for_daysback[subject].append(bids_path) + elif session is not None: + # Keep any one run for each data type and session + if all( + [ + session != p.session + for p in bids_paths_for_daysback[subject] + if datatype == p.datatype + ] + ): + bids_paths_for_daysback[subject].append(bids_path) + + bids_paths_to_consider = [] + for bids_path in bids_paths_for_daysback.values(): + bids_paths_to_consider.extend(bids_path) + + if len(bids_paths_to_consider) >= show_progress_thresh: + raws = [] + logger.info("\n") + for bids_path in ProgressBar( + iterable=bids_paths_to_consider, mesg="Determining daysback" + ): + raw = read_raw_bids(bids_path=bids_path, verbose="error") + raws.append(raw) + else: + raws = [ + read_raw_bids(bids_path=bp, verbose="error") + for bp in bids_paths_to_consider + ] + + daysback_min, daysback_max = get_anonymization_daysback(raws=raws, verbose=False) + + # Pick one randomly + daysback = rng.choice(np.arange(daysback_min, daysback_max + 1, dtype=int)) + daysback = int(daysback) + return daysback + + +def _check_crosstalk_path(bids_path: BIDSPath) -> bool: + is_crosstalk_path = ( + bids_path.datatype == "meg" + and bids_path.suffix == "meg" + and bids_path.acquisition == "crosstalk" + and bids_path.extension == ".fif" + ) + return is_crosstalk_path + + +def _check_finecal_path(bids_path: BIDSPath) -> bool: + is_finecal_path = ( + bids_path.datatype == "meg" + and bids_path.suffix == "meg" + and bids_path.acquisition == "calibration" + and bids_path.extension == ".dat" + ) + return is_finecal_path + + +@verbose +def anonymize_dataset( + bids_root_in, + bids_root_out, + daysback="auto", + subject_mapping="auto", + datatypes=None, + random_state=None, + verbose=None, +): + """Anonymize a BIDS dataset. + + This function creates a copy of a BIDS dataset, and tries to remove all + personally identifiable information from the copy. + + Parameters + ---------- + bids_root_in : path-like + The root directory of the input BIDS dataset. + bids_root_out : path-like + The directory to place the anonymized dataset into. + daysback : int | 'auto' + Number of days by which to move back the recording date in time. If + ``'auto'``, tries to randomly pick a suitable number. + subject_mapping : dict | callable | 'auto' | None + How to anonymize subject IDs. If a dictionary, maps the original IDs + (keys) to the anonymized IDs (values). If a function, must be one that + accepts the original IDs as a list of strings and returns a dictionary + with original IDs as keys and anonymized IDs as values. If ``'auto'``, + automatically produces a mapping (zero-padded numerical IDs) and prints + it on the screen. If ``None``, subject IDs are not changed. + datatypes : list of str | str | None + Which data type to anonymize. If can be ``meg``, ``eeg``, ``ieeg``, or + ``anat``. Multiple data types may be passed as a collection of strings. + If ``None``, try to anonymize the entire input dataset. + %(random_state)s + The RNG will be used to derive ``daysback`` and ``subject_mapping`` if + they are ``'auto'``. + %(verbose)s + """ + bids_root_in = Path(bids_root_in).expanduser() + bids_root_out = Path(bids_root_out).expanduser() + rng = np.random.default_rng(seed=random_state) + + if not bids_root_in.is_dir(): + raise FileNotFoundError( + f"The specified input directory does not exist: {bids_root_in}" + ) + + if bids_root_in == bids_root_out: + raise ValueError("Input and output directory must differ") + + if bids_root_out.exists(): + raise FileExistsError( + f"The specified output directory already exists. Please remove " + f"it to perform anonymization: {bids_root_out}" + ) + + if not isinstance(subject_mapping, dict): + participants_tsv = _from_tsv(bids_root_in / "participants.tsv") + participants_in = [ + participant.replace("sub-", "") + for participant in participants_tsv["participant_id"] + ] + + if subject_mapping == "auto": + # Don't change `emptyroom` subject ID + if "emptyroom" in participants_in: + n_participants = len(participants_in) - 1 + else: + n_participants = len(participants_in) + + participants_out = rng.permutation( + np.arange(start=1, stop=n_participants + 1, dtype=int) + ) + + # Zero-pad anonymized IDs + id_len = len(str(len(participants_out))) + + participants_out = [str(p).zfill(id_len) for p in participants_out] + + if "emptyroom" in participants_in: + # Append empty-room at the end + participants_in.remove("emptyroom") + participants_in.append("emptyroom") + participants_out.append("emptyroom") + + assert len(participants_in) == len(participants_out) + subject_mapping = dict(zip(participants_in, participants_out)) + elif callable(subject_mapping): + subject_mapping = subject_mapping(participants_in) + elif subject_mapping is None: + # identity mapping + subject_mapping = dict(zip(participants_in, participants_in)) + + if subject_mapping not in ("auto", None): + # Make sure we're mapping to strings + for k, v in subject_mapping.items(): + subject_mapping[k] = str(v) + + if "emptyroom" in subject_mapping and subject_mapping["emptyroom"] != "emptyroom": + warn( + f'You requested to change the "emptyroom" subject ID ' + f'(to {subject_mapping["emptyroom"]}). It is not ' + f"recommended to do this!" + ) + + allowed_datatypes = ("meg", "eeg", "ieeg", "anat") + allowed_suffixes = ("meg", "eeg", "ieeg", "T1w", "FLASH") + allowed_extensions = [] + for v in ALLOWED_DATATYPE_EXTENSIONS.values(): + allowed_extensions.extend(v) + allowed_extensions.extend([".nii", ".nii.gz"]) + + if isinstance(datatypes, str): + requested_datatypes = [datatypes] + elif datatypes is None: + requested_datatypes = allowed_datatypes + else: + requested_datatypes = datatypes + + for datatype in requested_datatypes: + if datatype not in allowed_datatypes: + raise ValueError(f"Unsupported data type: {datatype}") + del datatype, datatypes + + # Assemble list of candidate files for conversion + matches = bids_root_in.glob("sub-*/**/sub-*.*") + bids_paths_in = [] + for f in matches: + bids_path = get_bids_path_from_fname(f, verbose="error") + if bids_path.datatype in requested_datatypes and ( + ( + bids_path.suffix in allowed_suffixes + and bids_path.extension in allowed_extensions + ) + or (_check_finecal_path(bids_path) or _check_crosstalk_path(bids_path)) + ): + bids_paths_in.append(bids_path) + + # Ensure we convert empty-room recordings first, as we'll want to pass + # their anonymized path when writing the associated experimental recordings + if "meg" in requested_datatypes: + bids_paths_in_er_only = [ + bp + for bp in bids_paths_in + if bp.subject == "emptyroom" and bp.task == "noise" + ] + bids_paths_in_er_first = bids_paths_in_er_only.copy() + for bp in bids_paths_in: + if bp not in bids_paths_in_er_only: + bids_paths_in_er_first.append(bp) + + bids_paths_in = bids_paths_in_er_first + del bids_paths_in_er_first, bids_paths_in_er_only + + logger.info("\nAnonymizing BIDS dataset") + if daysback == "auto": + # Find recordings that can be read with MNE-Python to extract the + # recording dates + bids_paths = [ + bp + for bp in bids_paths_in + if ( + bp.datatype != "anat" + and not _check_crosstalk_path(bp) + and not _check_finecal_path(bp) + ) + ] + if bids_paths: + logger.info('Determining "daysback" for anonymization.') + + daysback = _get_daysback( + bids_paths=bids_paths, rng=rng, show_progress_thresh=20 + ) + else: + daysback = None + del bids_paths + + # Check subject_mapping + subjects_in_dataset = set([bp.subject for bp in bids_paths_in]) + subjects_missing_mapping_keys = [ + s for s in subjects_in_dataset if s not in subject_mapping + ] + if subjects_missing_mapping_keys: + raise IndexError( + f"The subject_mapping dictionary does not contain an entry for " + f'subject ID: {", ".join(subjects_missing_mapping_keys)}' + ) + + _, unique_vals_idx, counts = np.unique( + list(subject_mapping.values()), return_index=True, return_counts=True + ) + non_unique_vals_idx = unique_vals_idx[counts > 1] + if non_unique_vals_idx.size > 0: + keys = np.array(list(subject_mapping.values()))[non_unique_vals_idx] + raise ValueError( + f"The subject_mapping dictionary contains duplicated anonymized " + f'subjet IDs: {", ".join(keys)}' + ) + + # Produce some logging output + msg = f"\n" f" Input: {bids_root_in}\n" f" Output: {bids_root_out}\n" f"\n" + if daysback is None: + msg += "Not shifting recording dates (found anatomical scans only).\n" + else: + msg += ( + f"Shifting recording dates by {daysback} days " + f"({round(daysback / 365, 1)} years).\n" + ) + msg += "Using the following subject ID anonymization mapping:\n\n" + for orig_sub, anon_sub in subject_mapping.items(): + msg += f" sub-{orig_sub} → sub-{anon_sub}\n" + logger.info(msg) + del msg + + # Actual processing starts here + for bp_in in ProgressBar(iterable=bids_paths_in, mesg="Anonymizing"): + bp_out = bp_in.copy().update( + subject=subject_mapping[bp_in.subject], root=bids_root_out + ) + + bp_er_in = bp_er_out = None + + # Handle empty-room anonymization: we need to change the session to + # match the new date + if ( + bp_in.datatype == "meg" + and "emptyroom" in subject_mapping + and not (_check_finecal_path(bp_in) or _check_crosstalk_path(bp_in)) + ): + if bp_in.subject == "emptyroom": + er_session_in = bp_in.session + else: + # An experimental recording, so we need to find the associated + # empty-room + bp_er_in = bp_in.find_empty_room(use_sidecar_only=True, verbose="error") + if bp_er_in is None: + er_session_in = None + else: + er_session_in = bp_er_in.session + + # Update the session entity + if er_session_in is not None: + date_fmt = "%Y%m%d" + er_session_out = datetime.strptime(er_session_in, date_fmt) - timedelta( + days=daysback + ) + er_session_out = datetime.strftime(er_session_out, date_fmt) + + if bp_in.subject == "emptyroom": + bp_out.session = er_session_out + assert bp_er_out is None + else: + bp_er_out = bp_er_in.copy().update( + subject=subject_mapping["emptyroom"], + session=er_session_out, + root=bp_out.root, + ) + + if bp_in.datatype == "anat": + bp_anat_json = bp_in.copy().update(extension=".json") + anat_json = json.loads(bp_anat_json.fpath.read_text(encoding="utf-8")) + landmarks = anat_json["AnatomicalLandmarkCoordinates"] + landmarks_dig = mne.channels.make_dig_montage( + nasion=landmarks["NAS"], + lpa=landmarks["LPA"], + rpa=landmarks["RPA"], + coord_frame="mri_voxel", + ) + write_anat( + image=bp_in.fpath, + bids_path=bp_out, + landmarks=landmarks_dig, + deface=True, + verbose="error", + ) + elif _check_crosstalk_path(bp_in): + write_meg_crosstalk(fname=bp_in.fpath, bids_path=bp_out, verbose="error") + elif _check_finecal_path(bp_in): + write_meg_calibration( + calibration=bp_in.fpath, bids_path=bp_out, verbose="error" + ) + else: + raw = read_raw_bids(bids_path=bp_in, verbose="error") + write_raw_bids( + raw=raw, + bids_path=bp_out, + anonymize={ + "daysback": daysback, + "keep_his": False, + "keep_source": False, + }, + empty_room=bp_er_out, + verbose="error", + ) + + # Enrich sidecars + bp_in_json = bp_in.copy().update(extension=".json") + bp_out_json = bp_out.copy().update(extension=".json") + bp_in_events = bp_in.copy().update(suffix="events", extension=".tsv") + bp_out_events = bp_out.copy().update(suffix="events", extension=".tsv") + + # Enrich the JSON file + if bp_in_json.fpath.exists(): + json_in = json.loads(bp_in_json.fpath.read_text(encoding="utf-8")) + else: + json_in = dict() + + if bp_out_json.fpath.exists(): + json_out = json.loads(bp_out_json.fpath.read_text(encoding="utf-8")) + else: + json_out = dict() + + # Only transfer data that we believe doesn't contain any personally + # identifiable information + json_updates = dict() + for key, value in json_in.items(): + if key in ANONYMIZED_JSON_KEY_WHITELIST and key not in json_out: + json_updates[key] = value + del json_in, json_out + + if json_updates: + bp_out_json.fpath.touch(exist_ok=True) + update_sidecar_json( + bids_path=bp_out_json, entries=json_updates, verbose="error" + ) + + # Transfer trigger codes from original *_events.tsv file + if bp_in_events.fpath.exists(): + assert bp_out_events.fpath.exists() + events_tsv_in = _from_tsv(bp_in_events) + events_tsv_out = _from_tsv(bp_out_events) + + assert events_tsv_in["trial_type"] == events_tsv_out["trial_type"] + events_tsv_out["value"] = events_tsv_in["value"] + _write_tsv( + fname=bp_out_events.fpath, + dictionary=events_tsv_out, + overwrite=True, + verbose="error", + ) + + # Copy some additional files + additional_files = ( + "README", + "CHANGES", + "dataset_description.json", + "participants.json", + ) + for fname in additional_files: + in_path = bids_root_in / fname + if in_path.exists(): + shutil.copy(src=in_path, dst=bids_root_out) diff --git a/mne-bids-0.15/pyproject.toml b/mne-bids-0.15/pyproject.toml new file mode 100644 index 0000000000000000000000000000000000000000..1a87c0f1129934bdadb5dbc92e95530bbc73bd04 --- /dev/null +++ b/mne-bids-0.15/pyproject.toml @@ -0,0 +1,151 @@ +[build-system] +requires = ["hatchling", "hatch-vcs"] +build-backend = "hatchling.build" + +[project] +name = "mne-bids" +description = "MNE-BIDS: Organizing MEG, EEG, and iEEG data according to the BIDS specification and facilitating their analysis with MNE-Python" +dynamic = ["version"] +authors = [{ name = "MNE-BIDS Developers" }] +maintainers = [ + { name = "Stefan Appelhoff", email = "stefan.appelhoff@mailbox.org" }, +] +license = { text = "BSD-3-Clause" } +readme = { file = "README.md", content-type = "text/markdown" } +requires-python = ">=3.9" +keywords = [ + "meg", + "eeg", + "ieeg", + "bids", + "brain imaging data structure", + "neuroscience", + "neuroimaging", +] +classifiers = [ + "Topic :: Scientific/Engineering", + "Intended Audience :: Science/Research", + "Intended Audience :: Developers", + "License :: OSI Approved", + "Topic :: Software Development", + "Topic :: Scientific/Engineering", + "Operating System :: Microsoft :: Windows", + "Operating System :: POSIX :: Linux", + "Operating System :: MacOS", + "Programming Language :: Python", + "Programming Language :: Python :: 3.9", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", +] +scripts = { mne_bids = "mne_bids.commands.run:main" } +dependencies = ["mne>=1.5", "numpy>=1.21.2", "scipy>=1.7.1"] + +[project.optional-dependencies] +# Variants with dependencies that will get installed on top of those listed unter +# project.dependencies + +# Dependencies for using all mne_bids features +full = [ + "nibabel >= 3.2.1", + "pybv >= 0.7.5", + "eeglabio >= 0.0.2", + "pymatreader >= 0.0.30", + "matplotlib >= 3.5.0", + "pandas >= 1.3.2", + "EDFlib-Python >= 1.0.6", # drop once mne <1.7 is no longer supported + "edfio >= 0.2.1", + "defusedxml", # For reading EGI MFF data and BrainVision montages +] + +# Dependencies for running the test infrastructure +test = ["mne_bids[full]", "pytest", "pytest-cov", "pytest-sugar", "ruff"] + +# Dependencies for building the documentation +doc = [ + "nilearn", + "sphinx", + "sphinx_gallery", + "sphinx-copybutton", + "pydata-sphinx-theme", + "numpydoc", + "matplotlib", + "pillow", + "pandas", + "mne-nirs", + "seaborn", + "openneuro-py", +] + +# Dependencies for developer installations +dev = ["mne_bids[test,doc,full]", "pre-commit"] + +[project.urls] +"Homepage" = "https://mne.tools/mne-bids" +"Download" = "https://pypi.org/project/mne-bids/#files" +"Bug Tracker" = "https://github.com/mne-tools/mne-bids/issues/" +"Documentation" = "https://mne.tools/mne-bids" +"Forum" = "https://mne.discourse.group/" +"Source Code" = "https://github.com/mne-tools/mne-bids" + +[tool.hatch.metadata] +allow-direct-references = true # allow specifying URLs in our dependencies + +[tool.hatch.build] +exclude = [ + "/.*", + "**/tests", + "/paper", + "/examples", + "/doc", + "/Makefile", + "/CITATION.cff", + "/CONTRIBUTING.md", +] + +[tool.hatch.version] +source = "vcs" +raw-options = { version_scheme = "release-branch-semver" } + +[tool.ruff.lint] +select = ["E", "F", "W", "D", "I"] +exclude = ["__init__.py"] + +[tool.ruff.lint.pydocstyle] +convention = "numpy" + +[tool.coverage.run] +omit = ["*tests*"] + +[tool.coverage.report] +# Regexes for lines to exclude from consideration +exclude_lines = ["pragma: no cover", "if 0:", "if __name__ == .__main__.:"] + +[tool.pytest.ini_options] +addopts = """--durations=20 -ra --junit-xml=junit-results.xml --tb=short + --ignore=doc --ignore=examples --ignore=mne_bids/tests/data""" +filterwarnings = [ + "error", + "ignore:Estimation of line frequency only supports.*:RuntimeWarning", + "ignore:There are channels without locations (n/a)*:RuntimeWarning", + "ignore:Did not find any electrodes.tsv.*:RuntimeWarning", + "ignore:Did not find any coordsystem.json.*:RuntimeWarning", + "ignore:Did not find any events.tsv.*:RuntimeWarning", + "ignore:No events found or provided.*:RuntimeWarning", + "ignore:Participants file not found for.*:RuntimeWarning", + "ignore:Converting to FIF for anonymization:RuntimeWarning", + "ignore:Converting to BV for anonymization:RuntimeWarning", + "ignore:Converting data files to BrainVision format:RuntimeWarning", + "ignore:Writing of electrodes.tsv is not supported for datatype.*:RuntimeWarning", + "ignore:numpy.ufunc size changed.*:RuntimeWarning", + "ignore:tostring\\(\\) is deprecated.*:DeprecationWarning", + "ignore:MEG ref channel RMSP did not.*:RuntimeWarning", + "ignore:`product` is deprecated as of NumPy.*:DeprecationWarning", + # Python 3.10+ and NumPy 1.22 (and maybe also newer NumPy versions?) + "ignore:.*distutils\\.sysconfig module is deprecated.*:DeprecationWarning", + # numba with NumPy dev + "ignore:`np.MachAr` is deprecated.*:DeprecationWarning", + # old MNE _fake_click + "ignore:The .*_event function was deprecated in Matplotlib.*:", + "ignore:datetime\\.datetime\\.utcfromtimestamp.* is deprecated and scheduled for removal in a future version.*:DeprecationWarning", +] diff --git a/mne-bids-0.9.tar.gz b/mne-bids-0.9.tar.gz deleted file mode 100644 index e3023f8eeeba7c6c78c2aace29dafc782a113ad3..0000000000000000000000000000000000000000 Binary files a/mne-bids-0.9.tar.gz and /dev/null differ diff --git a/python-mne-bids.spec b/python-mne-bids.spec index 60267163aa03a122a79cf0b0eb93fa45d0355dd0..122efb09f102e57a81d2ef70ce786ab1f0758fd5 100644 --- a/python-mne-bids.spec +++ b/python-mne-bids.spec @@ -1,11 +1,11 @@ %global _empty_manifest_terminate_build 0 Name: python-mne-bids -Version: 0.9 +Version: 0.15 Release: 1 Summary: Organizing MEG, EEG, and iEEG data according to the BIDS specification and facilitating their analysis with MNE-Python License: BSD-3-Clause URL: https://github.com/mne-tools/mne-bids -Source0: https://files.pythonhosted.org/packages/a0/f2/1091a5e4e89746105577a8d00bf8ac1063f847f7363d8120a4d1d5d022e5/mne-bids-0.9.tar.gz +Source0: https://files.pythonhosted.org/packages/31/33/6c14b9aefaeeab0fe40925cedb37ceda62cb23fef514e6f498889dd7c44f/mne-bids-0.15.tar.gz BuildArch: noarch Requires: python3-mne @@ -27,6 +27,8 @@ Summary: Organizing MEG, EEG, and iEEG data according to the BIDS specification Provides: python-mne-bids BuildRequires: python3-devel BuildRequires: python3-setuptools +BuildRequires: python3-pip +BuildRequires: python3-wheel %description -n python3-mne-bids MNE-BIDS is a Python package that allows you to read and write BIDS-compatible datasets with the help of MNE-Python. @@ -37,13 +39,13 @@ Provides: python3-mne-bids-doc MNE-BIDS is a Python package that allows you to read and write BIDS-compatible datasets with the help of MNE-Python. %prep -%autosetup -n mne-bids-0.9 +%autosetup -n mne-bids-0.15 %build -%py3_build +%pyproject_build %install -%py3_install +%pyproject_install install -d -m755 %{buildroot}/%{_pkgdocdir} if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi @@ -77,6 +79,39 @@ mv %{buildroot}/doclist.lst . %{_docdir}/* %changelog +* Fri Nov 01 2024 zhangyulong - 0.15-1 +- Update package to version 0.15 + -MAINT: remove ref-names from git_archive.txt + -Fix InvalidDistribution: Metadata is required fields: Name, Version. + -read_raw_bids() tries to process task name without checking if a task exists bug + -Allow BIDSPath(check=False) to work for _check_key_val() enhancement + -Fix after switch to hatchling&hatch-vcs: heading in documentation is taking up too much space maint + -Adjust pip install + -MAINT: MNE-Python now uses edfio + -MAINT: update release protocol --> versioning scheme maint + -Add copyright holder to each file, and remove individual authors maint + -FutureWarning: with copy=False --> set default explicity + -Fix BUG: Error when writing calibration for empty-room file + -Fix read_raw_bids can't create events if taskname includes decimal dot + -maint: linters, CIs, packaging metadata code-health maint + -get_entity_vals() does support the 'recording' entity easy enhancement Nice First Contribution Opportunity + -Fix read_raw_bids() messes up the channel order and therefore loading raw fails bug + -read_raw_bids not accepting extra params (clean_names for CTF) API doc enhancement + -Fix doc build failure: pydata-sphinx-theme + -Fix problems with conversion of BTI datasets bug + -MNE-BIDS 0.13 release + -Export to EEGLAB format + -Fix EEGLab bug writing HDF5 persisting in MNE-BIDS bug + -Use Black for formatting maint + -write_raw_bids() cannot handle MaxFiltered data bug + -docs: dropdown menu on main no longer working + -Fix doc build fails: "tut-ieeg-localize" moved bug + -Problem loading mne-bids when MNE 1.0.0 and above is installed + -BIDS sample and value columns will be considered "arbitrary" columns soon + -restrict find_matching_paths to only look outside of derivatives/ Nice First Contribution Opportunity + -Fix command raw_to_bids takes overwrite and line_freq as text and fails for brainvision format bug + -Fix write_raw_bids does not produce a valid SET file bug + * Fri Dec 17 2021 Python_Bot - 0.9-1 - Package Init