Direction for generation compatible data for training/fine-tuning from edf file

[GoMapur]

Hi, I'm trying to pretrain the model with custom dataset. I saw there is some code under /data for pre-processing data. Is it possble for you to provide some direction on how to process raw EDF file and other necessary meta-data into the form compatible with this project?

Thank you!

[czlwang]

Hi,

I updated the README to specify how the paths to the pretraining data should be set.

Basically, you can do the preprocessing yourself, as described in appendix F, and set the paths to the data as specified. I think this will make the most sense.

But if it helps, I can also explain what preprocessing I did, and where the scripts are. I don't think the scripts will be runnable off the shelf, and many of them will reflect the idiosyncrasies of our data organization, but pieces of the pipeline could be helpful.

At a high level, our data arrives in the EDF format, we convert it to h5 files for storage and ease of access. We then filter out line noise, and segment the data into examples, each 5 seconds long, and write each example as an .npy file.

The code for these steps can be found mostly under data and datasets. Working backwards, check out data/write_data_to_disk.py, which writes a PretrainingWavsInMem dataset (datasets/pretraining_wavs_in_mem) to .npy examples. At the heart of this code is the ElectrodeSubjectData object (data/electrode_subject_data) which is meant to represent all the data for a single electrode in a single subject. This class uses the H5DataReader class (data/H5DataReader.py) to do all the preprocessing.

For getting from EDF to H5, check out data/edf2h5.py, which gives example usage of the relevant functions.

Please let me know if you have any follow up questions!

[GoMapur]

Thank you very much for your help!

[g12bftd]

Hey! I am looking to pre-train on the CHB-MIT dataset (https://physionet.org/content/chbmit/1.0.0/chb01/#files-panel). The dataset structure may be different to yours. In particular, we have 23 folders each representing 1 subject/patient. Each folder contains 9-42 continuous .edf files from the subject/patient in question. Each .edf file contains 1-2 hours of EEG signals (all 23 channels). Is this a dataset that can easily be fit into your training pipeline or would the pipeline require substantial changes? If the latter is the case, is there a publicly available dataset that would more closely match the format you used?