various fixes to make the tests run again

.github/workflows/ci.yml

@@ -37,7 +37,12 @@ jobs:
       - name: Run setup.sh
         run: ./setup.sh --dev --no-system
 
+      - name: Run tests
+        run: |
+          source venv/bin/activate
+          pytest -v -m "not skip_in_ci" tests/unit
+
       - name: Run MPI tests
         run: |
           source venv/bin/activate
-          mpirun -n 2 pytest -v tests/unit --with-mpi
+          mpirun -n 2 pytest -v tests/unit-mpi --with-mpi

.gitignore

@@ -31,6 +31,8 @@ bluerecording.egg-info
 *.nrrd
 neurodamus/
 positions/
+*.pkl
+venv/
 pkls/
 pkls_big/
 htmlcov/
@@ -39,3 +41,5 @@ htmlcov/
 **/LFP.h5
 *removeme*
 *DS_Store
+libsonatareport/
+neurodamus-models/

README.md

@@ -76,12 +76,24 @@ Neurodamus-models expects that you have modules available on your system for `py
 ---
 # Testing
 
-First, download `atlas.zip` from [Zenodo](https://doi.org/10.5281/zenodo.10927050) and unzip it into the folder `examples/data/atlas`. If you ran `source setup.sh --dev`, you can skip this step, as the dataset is already downloaded.  
+First, download `atlas.zip` from [Zenodo](https://doi.org/10.5281/zenodo.10927050) and extract it into the `examples/data/atlas` directory. If you ran `source setup.sh --dev`, you can skip this step because the dataset is already downloaded.
 
-To run the tests (assuming you installed the development version with `source setup.sh --dev`):
+To run the MPI tests (assuming you installed the development version with `source setup.sh --dev`):
 
 ```bash
-mpirun -n 2 pytest -v tests/unit --with-mpi
+mpirun -n 2 pytest -v tests/unit-mpi --with-mpi
+```
+
+To run the full test suite, you must first run the simulation located in `examples/compare-to-reference-solutions/data/simulation/`. After doing so, execute:
+
+```bash
+pytest tests/unit
+```
+
+If you have not run the simulation, you will need to skip a small number of tests:
+
+```bash
+pytest -v -m "not skip_in_ci" tests/unit
 ```
 
 ### Steps to produce electrode files

bluerecording/getPositions.py

@@ -78,7 +78,7 @@ def get_axon_points(m,center):
 
     targetLength = 1060
 
-    points = np.Inf
+    points = np.inf
 
     currentLen = 0
     runningLen = []
@@ -397,6 +397,11 @@ def getNewIndex(colIdx):
     Because we are saving the start and end points for each non-somatic segment, we need to add an additional entry to the dataframe column indices for section
     '''
 
+    colIdx = pd.MultiIndex.from_tuples(
+        [(nid - 1, sec) for nid, sec in colIdx],
+        names=colIdx.names
+    )
+
     newIdx = []
 
     for i, col in enumerate(colIdx): # Iterates through column indices of compartment report
@@ -457,7 +462,7 @@ def getPositions(path_to_simconfig, neurons_per_file, files_per_folder, path_to_
 
     for idx, i in enumerate(ids): # Iterates through node_ids and gets segment positions
 
-        m, center = getMorphology(population,i, path_to_simconfig)
+        m, center = getMorphology(population,i-1, path_to_simconfig)
 
         axonsFirst = checkAxonsFirst(m)
 

bluerecording/writeH5.py

@@ -439,8 +439,12 @@ def getNeuronSegmentMidpts(position):
 
 def getSegmentMidpts(positions,node_ids):
 
-    newPos = positions.groupby(level=0,axis=1,group_keys=False).apply(getNeuronSegmentMidpts)
-
+    newPos = (
+    positions.T
+        .groupby(level=0, group_keys=False)
+        .apply(lambda g: getNeuronSegmentMidpts(g.T).T)
+        .T
+    )
 
     return newPos
 

bluerecording/writeH5_prelim.py

@@ -7,7 +7,6 @@
 import bluepysnap as bp
 import json
 from .utils import *
-import pkg_resources
 import datetime
 
 class ElectrodeFileStructure(object):
@@ -244,7 +243,7 @@ def initializeH5File(path_to_simconfig,outputfile,electrode_csv):
     h5id.set_mdc_config(cc)
     #####
 
-    h5 = ElectrodeFileStructure(h5file, nodeIds, electrodes, population_name, circuit=circuitpath,version=pkg_resources.get_distribution("bluerecording").version,date=str(datetime.date.today())) # Initializes fields in h5 file
+    h5 = ElectrodeFileStructure(h5file, nodeIds, electrodes, population_name, circuit=circuitpath,version=__version__) # Initializes fields in h5 file
 
 
     write_all_neuron(sectionIdsFrame, population_name, h5, h5file, electrodes)  # For each node_id, initializes coefficient field in h5 file

examples/circuitTest/ReadMe.md

@@ -11,18 +11,26 @@ We assume that you are running these examples on a Linux system with slurm. If t
 ### Download data
 Download the file networks.zip from 10.5281/zenodo.10927050 and unzip it into the folder *data/simulation/configuration/networks*
 
-### Calculating Segment Positions
+### Running 1 time step simulation
 
-First, in the **data/simulation** subfolder, the compartment report is produced for the network by running the script **launch.sh**. Next, in the the **data/getPositions** subfolder, the segment positions for the cells are extracted by running the script **GetPositions.sh**.
+First, in the **data/simulation** subfolder, the compartment report is produced for the network by running the script **launch.sh**. This will create the `report.h5` needed in the following steps.
 
-### Extracellular signal calculation
+### Load spack and python envs
 
-##### Electrode File
+After, load your BlueRecording spack and python environments. The following scripts assume that these are loaded.
 
-In subfolder **test/electrodeFile**, the the electrode weights h5 file is created by running the scripts **WriteH5Prelim.sh** and **WriteH5.sh**. 
+### Get the positions
+
+Next, in the the **data/getPositions** subfolder, the segment positions for the cells are extracted by running the script **GetPositions.sh**.
+
+### Electrode File
+
+In subfolder **test/electrodeFile**, the the electrode weights h5 file is created by running the script **WriteH5.sh**. 
+
+### Online signal calculation
 
-##### Online signal calculation
 In subfolder **test/simulation**, the simulation is launched by running the script **launch.sh**. 
 
 ## Important note
+
 The user should note that in the circuit and simulation configuration files, the absolute paths listed should be modified to match the paths in the user's system. In the bash scripts, the slurm commands must be modified according to the user's system.

examples/circuitTest/data/getPositions/GetPositions.sh

@@ -1,33 +1,20 @@
-#!/bin/bash -l
-#SBATCH --job-name="EEG_1_CoordsV"
-#SBATCH --partition=prod
-#SBATCH --nodes=1
-#SBATCH -C clx
-#SBATCH --cpus-per-task=2
-#SBATCH --time=24:00:00
-##SBATCH --mail-type=ALL
-#SBATCH --account=proj85
-#SBATCH --no-requeue
-#SBATCH --output=EEG_1_CoordsV.out
-##SBATCH --error=EEG_1_CoordsV.err
-#SBATCH --exclusive
-#SBATCH --mem=0
-# SPDX-License-Identifier: GPL-3.0-or-later
+#!/usr/bin/env bash
 
-spack env activate bluerecording-dev
-source ~/bluerecording-dev/bin/activate
 
 NEURONS_PER_FILE=1000
-
 FILES_PER_FOLDER=50
 
-for i in {0..0}
-do
-
-    folder="positions/$(($i/$CHUNK_SIZE))"
-    mkdir -p $folder 2>/dev/null
+# Number of folders you want to pre-create
+NUM_FOLDERS=1
 
+for ((i=0; i<NUM_FOLDERS; i++)); do
+    folder="positions/$i"
+    mkdir -p "$folder"
 done
 
-srun -n 1 python ../../../scripts/run_get_positions.py "../simulation/simulation_config.json" "positions" $NEURONS_PER_FILE $FILES_PER_FOLDER
+python ../../../scripts/run_get_positions.py \
+    "../simulation/simulation_config.json" \
+    "positions" \
+    "$NEURONS_PER_FILE" \
+    "$FILES_PER_FOLDER"
 

examples/circuitTest/test/electrodeFile/WriteH5.sh

@@ -1,23 +1,17 @@
-#!/bin/bash -l
-#SBATCH --job-name="EEG_2_CoordsV"
-#SBATCH --partition=prod
-#SBATCH --nodes=1
-#SBATCH -C clx
-#SBATCH --cpus-per-task=2
-#SBATCH --time=24:00:00
-##SBATCH --mail-type=ALL
-#SBATCH --account=proj85
-#SBATCH --no-requeue
-#SBATCH --output=EEG_2_CoordsV.out
-#SBATCH --error=EEG_2_CoordsV.err
-#SBATCH --exclusive
-#SBATCH --mem=0
+#!/usr/bin/env bash
+
 # SPDX-License-Identifier: GPL-3.0-or-later
 
-spack env activate bluerecording-dev
-source ~/bluerecording-dev/bin/activate
+python ../../../scripts/run_initialize_h5.py 'electrodes.csv' '../../data/simulation/simulation_config.json' 'coeffs.h5' 
 
+# Parameters
 NEURONS_PER_FILE=1000
 FILES_PER_FOLDER=50
 
-srun -n 1 python ../../../scripts/run_write_weights.py '../../data/simulation/simulation_config.json' '../../data/getPositions/positions' 'coeffs.h5' $NEURONS_PER_FILE $FILES_PER_FOLDER
+# Run the Python script locally
+python ../../../scripts/run_write_weights.py \
+    "../../data/simulation/simulation_config.json" \
+    "../../data/getPositions/positions" \
+    "coeffs.h5" \
+    "$NEURONS_PER_FILE" \
+    "$FILES_PER_FOLDER"

pyproject.toml

@@ -5,7 +5,10 @@ description = "A tool for calculating extracellular recording lead fields"
 requires-python = ">=3.9"
 dependencies = [
     "h5py",
-    "mpi4py"
+    "mpi4py",
+    "pandas",
+    "morphio",
+    "bluepysnap","voxcell","scikit-learn"
 ]
 
 [project.optional-dependencies]

pytest.ini

@@ -0,0 +1,3 @@
+[pytest]
+markers =
+    skip_in_ci: tests that require data. We skip them in ci to keep it light

setup.sh

@@ -4,8 +4,6 @@
 INSTALL_MODE="normal"
 SKIP_SYSTEM=0
 
-set -e
-
 # -------------------------
 # Parse arguments (with --help)
 # -------------------------