From 918477c55826ab1806cda782f52c3be84ff95513 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Mon, 21 Jul 2025 13:30:58 +0100 Subject: [PATCH 01/56] Skip these tests on Apple M1 architecture, unresolved issue --- tests/testthat/test-smk-corTestDS.R | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/testthat/test-smk-corTestDS.R b/tests/testthat/test-smk-corTestDS.R index be006fe4..ba2560e8 100644 --- a/tests/testthat/test-smk-corTestDS.R +++ b/tests/testthat/test-smk-corTestDS.R @@ -12,6 +12,9 @@ # Set up # +# Skip these tests on Apple M1 mac +skip_on_os("mac", arch = "aarch64") + context("corTestDS::smk::setup") # From a1f1178e9da93340e1d0ffce26444f1e1a1a3633 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Mon, 21 Jul 2025 13:46:15 +0100 Subject: [PATCH 02/56] Add CRAN release badge, title, installation instructions and references --- README.md | 62 +++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 56 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index aac29667..f78a2c3e 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,35 @@ -dsBase -====== +## dsBase: 'DataSHIELD' Server Site Base Functions -DataSHIELD server side base R library. +[![License](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0.html) +[![](https://www.r-pkg.org/badges/version/dsBase?color=black)](https://cran.r-project.org/package=dsBase) +[![R build +status](https://github.com/datashield/dsBase/workflows/R-CMD-check/badge.svg)](https://github.com/datashield/dsBase/actions) -[![License](https://img.shields.io/badge/license-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0.html) +## Installation -About -===== +You can install the released version of dsBase from +[CRAN](https://cran.r-project.org/package=dsBase) with: + +``` r +install.packages("dsBase") +``` + +And the development version from +[GitHub](https://github.com/datashield/dsBase/) with: + + +``` r +install.packages("remotes") +remotes::install_github("datashield/dsBase", "") + +# Install v6.4.0 with the following +remotes::install_github("datashield/dsBase", "v6.4.0-dev") +``` + +For a full list of development branches, checkout https://github.com/datashield/dsBase/branches + + +## About DataSHIELD is a software package which allows you to do non-disclosive federated analysis on sensitive data. Our website (https://www.datashield.org) has in depth descriptions of what it is, how it works and how to install it. A key point to highlight is that DataSHIELD has a client-server infrastructure, so the dsBase package (https://github.com/datashield/dsBase) needs to be used in conjuction with the dsBaseClient package (https://github.com/datashield/dsBaseClient) - trying to use one without the other makes no sense. @@ -18,3 +41,30 @@ Detailed instructions on how to install DataSHIELD are at https://wiki.datashiel | obiba CRAN | Where you probably should install DataSHIELD from. | | releases | Stable releases. | | master branch | Mostly in sync with the latest release, changes rarely. | + +## References + +[1] Burton P, Wilson R, Butters O, Ryser-Welch P, Westerberg A, Abarrategui L, Villegas-Diaz R, + Avraam D, Marcon Y, Wheater S (2025). _dsBase: 'DataSHIELD' Server Site Base Functions_. R + package version 6.3.3. . + +[2] Gaye A, Marcon Y, Isaeva J, LaFlamme P, Turner A, Jones E, Minion J, Boyd A, Newby C, Nuotio + M, Wilson R, Butters O, Murtagh B, Demir I, Doiron D, Giepmans L, Wallace S, Budin-Ljøsne I, + Oliver Schmidt C, Boffetta P, Boniol M, Bota M, Carter K, deKlerk N, Dibben C, Francis R, + Hiekkalinna T, Hveem K, Kvaløy K, Millar S, Perry I, Peters A, Phillips C, Popham F, Raab G, + Reischl E, Sheehan N, Waldenberger M, Perola M, van den Heuvel E, Macleod J, Knoppers B, + Stolk R, Fortier I, Harris J, Woffenbuttel B, Murtagh M, Ferretti V, Burton P (2014). + “DataSHIELD: taking the analysis to the data, not the data to the analysis.” _International + Journal of Epidemiology_, *43*(6), 1929-1944. . + +[3] Wilson R, W. Butters O, Avraam D, Baker J, Tedds J, Turner A, Murtagh M, R. Burton P (2017). + “DataSHIELD – New Directions and Dimensions.” _Data Science Journal_, *16*(21), 1-21. + . + +[4] Avraam D, Wilson R, Aguirre Chan N, Banerjee S, Bishop T, Butters O, Cadman T, Cederkvist L, + Duijts L, Escribà Montagut X, Garner H, Gonçalves G, González J, Haakma S, Hartlev M, + Hasenauer J, Huth M, Hyde E, Jaddoe V, Marcon Y, Mayrhofer M, Molnar-Gabor F, Morgan A, + Murtagh M, Nestor M, Nybo Andersen A, Parker S, Pinot de Moira A, Schwarz F, + Strandberg-Larsen K, Morris AvSwertz, Welten M, Wheater S, Burton P (2024). “DataSHIELD: + mitigating disclosure risk in a multi-site federated analysis platform.” _Bioinformatics + Advances_, *5*(1), 1-21. . From 5582620c2861e62e94c166f7b4305f9f33499243 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Mon, 21 Jul 2025 13:56:04 +0100 Subject: [PATCH 03/56] Correct title and citation for main package --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index f78a2c3e..de46eade 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -## dsBase: 'DataSHIELD' Server Site Base Functions +## dsBase: 'DataSHIELD' Server Side Base Functions [![License](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0.html) [![](https://www.r-pkg.org/badges/version/dsBase?color=black)](https://cran.r-project.org/package=dsBase) @@ -45,7 +45,7 @@ Detailed instructions on how to install DataSHIELD are at https://wiki.datashiel ## References [1] Burton P, Wilson R, Butters O, Ryser-Welch P, Westerberg A, Abarrategui L, Villegas-Diaz R, - Avraam D, Marcon Y, Wheater S (2025). _dsBase: 'DataSHIELD' Server Site Base Functions_. R + Avraam D, Marcon Y, Bishop T, Gaye A, Escribà Montagut X, Wheater S (2025). _dsBase: 'DataSHIELD' Server Side Base Functions_. R package version 6.3.3. . [2] Gaye A, Marcon Y, Isaeva J, LaFlamme P, Turner A, Jones E, Minion J, Boyd A, Newby C, Nuotio From c5088ce5f11cb8f08115450a71ceb2e6a28189cd Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Mon, 21 Jul 2025 16:23:28 +0100 Subject: [PATCH 04/56] Add new workflow based on old azure-pipelines.yml file --- .github/workflows/R-CMD-check-source.yaml | 185 ++++++++++++++++++++++ 1 file changed, 185 insertions(+) create mode 100644 .github/workflows/R-CMD-check-source.yaml diff --git a/.github/workflows/R-CMD-check-source.yaml b/.github/workflows/R-CMD-check-source.yaml new file mode 100644 index 00000000..2f53f446 --- /dev/null +++ b/.github/workflows/R-CMD-check-source.yaml @@ -0,0 +1,185 @@ +################################################################################ +# DataSHIELD GHA test suite - dsBase +# Adapted from `azure-pipelines.yml` by Roberto Villegas-Diaz +# +# OVERVIEW +# +# Inside the root directory $(Pipeline.Workspace) will be a file tree like: +# /dsBase <- Checked out version of datashield/dsBase +# /testStatus <- Checked out version of datashield/testStatus +# /logs <- Where results of tests and logs are collated +# +# As of Jul 2025 this takes ~ 15 mins to run. +# +################################################################################ +name: DataSHIELD GHA test suite + +on: + push: + branches: + - master + - v6.3.3-dev + + schedule: + - cron: '0 0 * * 0' # Weekly on master + - cron: '0 1 * * *' # Nightly on v6.3.3-dev + +jobs: + run_devtools_check: + runs-on: ubuntu-latest + timeout-minutes: 120 + + # These should all be constant, except test_filter. This can be used to test + # subsets of test files in the testthat directory. Options are like: + # '*' <- Run all tests. + # 'asNumericDS*' <- Run all asNumericDS tests, i.e. all the arg, etc. tests. + # '*_smk_*' <- Run all the smoke tests for all functions. + env: + test_filter: '*' + _r_check_system_clock_: 0 + datetime: ${{ github.run_id }}-${{ github.run_attempt }} + projectName: dsBase + branchName: ${{ github.ref_name }} + + steps: + - name: Checkout dsBase + uses: actions/checkout@v4 + with: + path: dsBase + + - name: Checkout testStatus + uses: actions/checkout@v4 + with: + repository: datashield/testStatus + token: ${{ secrets.TESTSTATUS_TOKEN }} + ref: master + path: testStatus + + - name: Tweak local R env + run: | + echo "options(Ncpus=4)" >> ~/.Rprofile + + - name: Install R and dependencies + run: | + sudo apt-get update -qq + sudo apt-get install --no-install-recommends software-properties-common dirmngr -y + wget -qO- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc | sudo tee -a /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc + sudo add-apt-repository "deb https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/" + sudo apt-get update -qq + sudo apt-get upgrade -qq + sudo apt-get install -qq \ + pkg-config libxml2-dev libcurl4-openssl-dev libssl-dev libgit2-dev \ + libharfbuzz-dev libfribidi-dev libfontconfig1-dev libfreetype6-dev \ + libpng-dev libtiff5-dev libjpeg-dev r-base -y + sudo R -e "install.packages(c('devtools','RANN','stringr','lme4','dplyr','reshape2','polycor','splines','gamlss','gamlss.dist','mice','childsds'), dependencies=TRUE)" + + - name: Check man files up-to-date + run: | + orig_sum=$(find man -type f | sort -u | xargs cat | md5sum) + R -e "devtools::document()" + new_sum=$(find man -type f | sort -u | xargs cat | md5sum) + if [ "$orig_sum" != "$new_sum" ]; then + echo "Your committed manual files (man/*.Rd) are out of sync with the R files. Run devtools::document() locally then commit." + exit 1 + else + echo "Documentation up-to-date." + fi + working-directory: dsBase + continue-on-error: true + + - name: Run devtools::check + run: | + R -q -e "library('devtools'); devtools::check(args = c('--no-tests', '--no-examples'))" | tee check.Rout + grep -q "^0 errors" check.Rout && grep -q " 0 warnings" check.Rout && grep -q " 0 notes" check.Rout + working-directory: dsBase + continue-on-error: true + + - name: Run tests with coverage & JUnit report + run: | + mkdir -p logs + sudo R --verbose -e 'devtools::reload()' + sudo R -q -e ' + library(covr); + write.csv( + coverage_to_list( + covr::package_coverage( + type = c("none"), + code = c( + '"'"'library(testthat); + output_file <- file("test_console_output.txt"); + sink(output_file); + sink(output_file, type = "message"); + junit_rep <- JunitReporter$new(file = "test_results.xml"); + progress_rep <- ProgressReporter$new(max_failures = 999999); + multi_rep <- MultiReporter$new(reporters = list(progress_rep, junit_rep)); + testthat::test_package("${{ env.projectName }}", filter = "${{ env.test_filter }}", reporter = multi_rep, stop_on_failure = FALSE)'"'"' + ) + ) + ), + "coveragelist.csv" + )' + mv coveragelist.csv logs/ + mv test_results.xml logs/ + mv test_console_output.txt logs/ + grep -q " FAIL 0 " logs/test_console_output.txt + working-directory: dsBase + + - name: Check for JUnit errors + run: | + issue_count=$(sed 's/failures="0" errors="0"//' test_results.xml | grep -c errors= || true) + echo "Number of testsuites with issues: $issue_count" + sed 's/failures="0" errors="0"//' test_results.xml | grep errors= > issues.log || true + cat issues.log || true + exit $issue_count + working-directory: dsBase/logs + + - name: Write versions to file + run: | + echo "branch:${{ env.branchName }}" > ${{ env.datetime }}.txt + echo "os:$(lsb_release -ds)" >> ${{ env.datetime }}.txt + echo "R:$(R --version | head -n1)" >> ${{ env.datetime }}.txt + working-directory: dsBase/logs + + - name: Commit results to testStatus + if: github.repository == 'datashield/dsBase' && github.event_name != 'pull_request' + run: | + git config --local user.email "$GITHUB_ACTOR@users.noreply.github.com" + git config --global user.name "GitHub Actions" + cd testStatus + git checkout master + git pull + + mkdir -p logs/${{ env.projectName }}/${{ env.branchName }} + mkdir -p docs/${{ env.projectName }}/${{ env.branchName }}/latest + + cp ../dsBase/logs/coveragelist.csv logs/${{ env.projectName }}/${{ env.branchName }}/ + cp ../dsBase/logs/test_results.xml logs/${{ env.projectName }}/${{ env.branchName }}/ + cp ../dsBase/logs/${{ env.datetime }}.txt logs/${{ env.projectName }}/${{ env.branchName }}/ + + # Run status.py script here if applicable + # e.g. source/status.py logs/... logs/... status.html ... + + git add . + git commit -m "Auto test for ${{ env.projectName }}/${{ env.branchName }} @ ${{ env.datetime }}" + git push + + - name: Dump environment info + run: | + echo -e "\n#############################" + echo -e "ls /: ######################" + ls -al . + echo -e "\n#############################" + echo -e "lscpu: ######################" + lscpu + echo -e "\n#############################" + echo -e "memory: #####################" + free -m + echo -e "\n#############################" + echo -e "env: ########################" + env + echo -e "\n#############################" + echo -e "R sessionInfo(): ############" + R -e 'sessionInfo()' + sudo apt install tree -y + tree . + From 1a24b756e9d5a5316d84b7ffc9d459bfa0a5f90d Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Mon, 21 Jul 2025 16:26:37 +0100 Subject: [PATCH 05/56] Update token variable name --- .github/workflows/R-CMD-check-source.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/R-CMD-check-source.yaml b/.github/workflows/R-CMD-check-source.yaml index 2f53f446..ed453fc2 100644 --- a/.github/workflows/R-CMD-check-source.yaml +++ b/.github/workflows/R-CMD-check-source.yaml @@ -51,7 +51,7 @@ jobs: uses: actions/checkout@v4 with: repository: datashield/testStatus - token: ${{ secrets.TESTSTATUS_TOKEN }} + token: ${{ secrets.DATASHIELD_DEV }} ref: master path: testStatus From 74a120b7f125284bf56279f7e03a4d27f2b6d4b1 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Tue, 22 Jul 2025 09:32:16 +0100 Subject: [PATCH 06/56] Replace test_package call by test_local --- .github/workflows/R-CMD-check-source.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/R-CMD-check-source.yaml b/.github/workflows/R-CMD-check-source.yaml index ed453fc2..1d49b3d4 100644 --- a/.github/workflows/R-CMD-check-source.yaml +++ b/.github/workflows/R-CMD-check-source.yaml @@ -112,7 +112,7 @@ jobs: junit_rep <- JunitReporter$new(file = "test_results.xml"); progress_rep <- ProgressReporter$new(max_failures = 999999); multi_rep <- MultiReporter$new(reporters = list(progress_rep, junit_rep)); - testthat::test_package("${{ env.projectName }}", filter = "${{ env.test_filter }}", reporter = multi_rep, stop_on_failure = FALSE)'"'"' + testthat::test_local(filter = "${{ env.test_filter }}", reporter = multi_rep, stop_on_failure = FALSE)'"'"' ) ) ), From 408a736b1e928c9c9d296682bd630d7a3153a49e Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Tue, 22 Jul 2025 11:51:41 +0100 Subject: [PATCH 07/56] Use r-lib custom actions to setup deployment environment and customise the location of logs --- .github/workflows/R-CMD-check-source.yaml | 49 ++++++++++------------- 1 file changed, 22 insertions(+), 27 deletions(-) diff --git a/.github/workflows/R-CMD-check-source.yaml b/.github/workflows/R-CMD-check-source.yaml index 1d49b3d4..95e26eca 100644 --- a/.github/workflows/R-CMD-check-source.yaml +++ b/.github/workflows/R-CMD-check-source.yaml @@ -55,24 +55,19 @@ jobs: ref: master path: testStatus - - name: Tweak local R env - run: | - echo "options(Ncpus=4)" >> ~/.Rprofile + - uses: r-lib/actions/setup-pandoc@v2 - - name: Install R and dependencies - run: | - sudo apt-get update -qq - sudo apt-get install --no-install-recommends software-properties-common dirmngr -y - wget -qO- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc | sudo tee -a /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc - sudo add-apt-repository "deb https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/" - sudo apt-get update -qq - sudo apt-get upgrade -qq - sudo apt-get install -qq \ - pkg-config libxml2-dev libcurl4-openssl-dev libssl-dev libgit2-dev \ - libharfbuzz-dev libfribidi-dev libfontconfig1-dev libfreetype6-dev \ - libpng-dev libtiff5-dev libjpeg-dev r-base -y - sudo R -e "install.packages(c('devtools','RANN','stringr','lme4','dplyr','reshape2','polycor','splines','gamlss','gamlss.dist','mice','childsds'), dependencies=TRUE)" + - uses: r-lib/actions/setup-r@v2 + with: + r-version: release + http-user-agent: release + use-public-rspm: true + - uses: r-lib/actions/setup-r-dependencies@v2 + with: + extra-packages: any::rcmdcheck + needs: check + - name: Check man files up-to-date run: | orig_sum=$(find man -type f | sort -u | xargs cat | md5sum) @@ -89,14 +84,14 @@ jobs: - name: Run devtools::check run: | - R -q -e "library('devtools'); devtools::check(args = c('--no-tests', '--no-examples'))" | tee check.Rout - grep -q "^0 errors" check.Rout && grep -q " 0 warnings" check.Rout && grep -q " 0 notes" check.Rout + R -q -e "library('devtools'); devtools::check(args = c('--no-tests', '--no-examples'))" | tee ../check.Rout + grep -q "^0 errors" ../check.Rout && grep -q " 0 warnings" ../check.Rout && grep -q " 0 notes" ../check.Rout working-directory: dsBase continue-on-error: true - name: Run tests with coverage & JUnit report run: | - mkdir -p logs + mkdir -p ../logs sudo R --verbose -e 'devtools::reload()' sudo R -q -e ' library(covr); @@ -118,9 +113,9 @@ jobs: ), "coveragelist.csv" )' - mv coveragelist.csv logs/ - mv test_results.xml logs/ - mv test_console_output.txt logs/ + + mv coveragelist.csv ../logs/ + mv test_* ../logs/ grep -q " FAIL 0 " logs/test_console_output.txt working-directory: dsBase @@ -131,14 +126,14 @@ jobs: sed 's/failures="0" errors="0"//' test_results.xml | grep errors= > issues.log || true cat issues.log || true exit $issue_count - working-directory: dsBase/logs + working-directory: logs - name: Write versions to file run: | echo "branch:${{ env.branchName }}" > ${{ env.datetime }}.txt echo "os:$(lsb_release -ds)" >> ${{ env.datetime }}.txt echo "R:$(R --version | head -n1)" >> ${{ env.datetime }}.txt - working-directory: dsBase/logs + working-directory: logs - name: Commit results to testStatus if: github.repository == 'datashield/dsBase' && github.event_name != 'pull_request' @@ -152,9 +147,9 @@ jobs: mkdir -p logs/${{ env.projectName }}/${{ env.branchName }} mkdir -p docs/${{ env.projectName }}/${{ env.branchName }}/latest - cp ../dsBase/logs/coveragelist.csv logs/${{ env.projectName }}/${{ env.branchName }}/ - cp ../dsBase/logs/test_results.xml logs/${{ env.projectName }}/${{ env.branchName }}/ - cp ../dsBase/logs/${{ env.datetime }}.txt logs/${{ env.projectName }}/${{ env.branchName }}/ + cp logs/coveragelist.csv logs/${{ env.projectName }}/${{ env.branchName }}/ + cp logs/test_results.xml logs/${{ env.projectName }}/${{ env.branchName }}/ + cp logs/${{ env.datetime }}.txt logs/${{ env.projectName }}/${{ env.branchName }}/ # Run status.py script here if applicable # e.g. source/status.py logs/... logs/... status.html ... From 06daac3984e9313b452a7fdc1930b7e1439d3357 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Tue, 22 Jul 2025 12:00:21 +0100 Subject: [PATCH 08/56] Install missing package to run checks and tests --- .github/workflows/R-CMD-check-source.yaml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/workflows/R-CMD-check-source.yaml b/.github/workflows/R-CMD-check-source.yaml index 95e26eca..0135e808 100644 --- a/.github/workflows/R-CMD-check-source.yaml +++ b/.github/workflows/R-CMD-check-source.yaml @@ -65,7 +65,11 @@ jobs: - uses: r-lib/actions/setup-r-dependencies@v2 with: - extra-packages: any::rcmdcheck + extra-packages: | + any::rcmdcheck + cran::covr + cran::devtools + cran::testthat needs: check - name: Check man files up-to-date From 4ead1cfc1259cd7d4e70d818753863c7615cd416 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Tue, 22 Jul 2025 13:06:14 +0100 Subject: [PATCH 09/56] Add missing environment variable --- .github/workflows/R-CMD-check-source.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/R-CMD-check-source.yaml b/.github/workflows/R-CMD-check-source.yaml index 0135e808..7cd7d164 100644 --- a/.github/workflows/R-CMD-check-source.yaml +++ b/.github/workflows/R-CMD-check-source.yaml @@ -40,6 +40,8 @@ jobs: datetime: ${{ github.run_id }}-${{ github.run_attempt }} projectName: dsBase branchName: ${{ github.ref_name }} + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + R_KEEP_PKG_SOURCE: yes steps: - name: Checkout dsBase From a4f0077d45f6aaf44ad4ba511d69073b5c761275 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Tue, 22 Jul 2025 13:20:37 +0100 Subject: [PATCH 10/56] Add step with new approach to install devtools --- .github/workflows/R-CMD-check-source.yaml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.github/workflows/R-CMD-check-source.yaml b/.github/workflows/R-CMD-check-source.yaml index 7cd7d164..46fe2952 100644 --- a/.github/workflows/R-CMD-check-source.yaml +++ b/.github/workflows/R-CMD-check-source.yaml @@ -74,6 +74,13 @@ jobs: cran::testthat needs: check + - name: Install dependencies + run: | + install.packages('devtools') + install.packages('remotes') + remotes::install_deps(dependencies = TRUE) + shell: Rscript {0} + - name: Check man files up-to-date run: | orig_sum=$(find man -type f | sort -u | xargs cat | md5sum) From d66c37ac540d1631416ad45211b9cb6fab8e54cd Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Tue, 22 Jul 2025 13:37:39 +0100 Subject: [PATCH 11/56] Add missing workind directory --- .github/workflows/R-CMD-check-source.yaml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/.github/workflows/R-CMD-check-source.yaml b/.github/workflows/R-CMD-check-source.yaml index 46fe2952..bbe5c171 100644 --- a/.github/workflows/R-CMD-check-source.yaml +++ b/.github/workflows/R-CMD-check-source.yaml @@ -69,17 +69,16 @@ jobs: with: extra-packages: | any::rcmdcheck - cran::covr - cran::devtools - cran::testthat needs: check - name: Install dependencies run: | + list.files('.') install.packages('devtools') install.packages('remotes') remotes::install_deps(dependencies = TRUE) shell: Rscript {0} + working-directory: dsBase - name: Check man files up-to-date run: | From a8269e58951541d8d96843ecbc1f47b679be273f Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Tue, 22 Jul 2025 13:50:17 +0100 Subject: [PATCH 12/56] Simplify workflow removing unnecessary step --- .github/workflows/R-CMD-check-source.yaml | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/.github/workflows/R-CMD-check-source.yaml b/.github/workflows/R-CMD-check-source.yaml index bbe5c171..9baa59d6 100644 --- a/.github/workflows/R-CMD-check-source.yaml +++ b/.github/workflows/R-CMD-check-source.yaml @@ -69,17 +69,9 @@ jobs: with: extra-packages: | any::rcmdcheck + cran::devtools needs: check - - name: Install dependencies - run: | - list.files('.') - install.packages('devtools') - install.packages('remotes') - remotes::install_deps(dependencies = TRUE) - shell: Rscript {0} - working-directory: dsBase - - name: Check man files up-to-date run: | orig_sum=$(find man -type f | sort -u | xargs cat | md5sum) @@ -104,8 +96,8 @@ jobs: - name: Run tests with coverage & JUnit report run: | mkdir -p ../logs - sudo R --verbose -e 'devtools::reload()' - sudo R -q -e ' + R -q -e "library('devtools'); devtools::reload();"" + R -q -e ' library(covr); write.csv( coverage_to_list( From f4893025ea272d9d62e96dbf23c2d6465f6305ca Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Tue, 22 Jul 2025 14:01:45 +0100 Subject: [PATCH 13/56] Remove additional double quote mark --- .github/workflows/R-CMD-check-source.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/R-CMD-check-source.yaml b/.github/workflows/R-CMD-check-source.yaml index 9baa59d6..abdf22ff 100644 --- a/.github/workflows/R-CMD-check-source.yaml +++ b/.github/workflows/R-CMD-check-source.yaml @@ -96,7 +96,7 @@ jobs: - name: Run tests with coverage & JUnit report run: | mkdir -p ../logs - R -q -e "library('devtools'); devtools::reload();"" + R -q -e "library('devtools'); devtools::reload();" R -q -e ' library(covr); write.csv( From 7a81c04c98bcdc656590899406a3b54bd7a27ab5 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Tue, 22 Jul 2025 14:17:20 +0100 Subject: [PATCH 14/56] Ammend wrong file path and add commands to check current file tree --- .github/workflows/R-CMD-check-source.yaml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/workflows/R-CMD-check-source.yaml b/.github/workflows/R-CMD-check-source.yaml index abdf22ff..4f9945ad 100644 --- a/.github/workflows/R-CMD-check-source.yaml +++ b/.github/workflows/R-CMD-check-source.yaml @@ -117,10 +117,14 @@ jobs: ), "coveragelist.csv" )' - + + echo '========== . =========' + ls . + echo '========== .. =========' + ls .. mv coveragelist.csv ../logs/ mv test_* ../logs/ - grep -q " FAIL 0 " logs/test_console_output.txt + grep -q " FAIL 0 " ../logs/test_console_output.txt working-directory: dsBase - name: Check for JUnit errors From a21b0c4074df2570e0a81c6ac6971a1c25e00020 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Tue, 22 Jul 2025 14:36:11 +0100 Subject: [PATCH 15/56] Replace the location of testStatus for development purposes --- .github/workflows/R-CMD-check-source.yaml | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/.github/workflows/R-CMD-check-source.yaml b/.github/workflows/R-CMD-check-source.yaml index 4f9945ad..27e22931 100644 --- a/.github/workflows/R-CMD-check-source.yaml +++ b/.github/workflows/R-CMD-check-source.yaml @@ -52,7 +52,7 @@ jobs: - name: Checkout testStatus uses: actions/checkout@v4 with: - repository: datashield/testStatus + repository: villegar/testStatus token: ${{ secrets.DATASHIELD_DEV }} ref: master path: testStatus @@ -118,10 +118,6 @@ jobs: "coveragelist.csv" )' - echo '========== . =========' - ls . - echo '========== .. =========' - ls .. mv coveragelist.csv ../logs/ mv test_* ../logs/ grep -q " FAIL 0 " ../logs/test_console_output.txt @@ -144,7 +140,7 @@ jobs: working-directory: logs - name: Commit results to testStatus - if: github.repository == 'datashield/dsBase' && github.event_name != 'pull_request' + if: github.repository == 'villegar/dsBase' && github.event_name != 'pull_request' run: | git config --local user.email "$GITHUB_ACTOR@users.noreply.github.com" git config --global user.name "GitHub Actions" From f5ac4bcc47d2842e1fd0429ca2ae919681a94f8f Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Tue, 22 Jul 2025 14:37:39 +0100 Subject: [PATCH 16/56] Update path ../logs >>> logs --- .github/workflows/R-CMD-check-source.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/R-CMD-check-source.yaml b/.github/workflows/R-CMD-check-source.yaml index 27e22931..ac2d94e8 100644 --- a/.github/workflows/R-CMD-check-source.yaml +++ b/.github/workflows/R-CMD-check-source.yaml @@ -95,7 +95,7 @@ jobs: - name: Run tests with coverage & JUnit report run: | - mkdir -p ../logs + mkdir -p logs R -q -e "library('devtools'); devtools::reload();" R -q -e ' library(covr); @@ -118,9 +118,9 @@ jobs: "coveragelist.csv" )' - mv coveragelist.csv ../logs/ - mv test_* ../logs/ - grep -q " FAIL 0 " ../logs/test_console_output.txt + mv coveragelist.csv logs/ + mv test_* logs/ + grep -q " FAIL 0 " logs/test_console_output.txt working-directory: dsBase - name: Check for JUnit errors From 5d2c5cd816162322b952fd7912d4c196e1a665d9 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Tue, 22 Jul 2025 14:46:34 +0100 Subject: [PATCH 17/56] Update working directories --- .github/workflows/R-CMD-check-source.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/R-CMD-check-source.yaml b/.github/workflows/R-CMD-check-source.yaml index ac2d94e8..18957add 100644 --- a/.github/workflows/R-CMD-check-source.yaml +++ b/.github/workflows/R-CMD-check-source.yaml @@ -130,14 +130,14 @@ jobs: sed 's/failures="0" errors="0"//' test_results.xml | grep errors= > issues.log || true cat issues.log || true exit $issue_count - working-directory: logs + working-directory: dsBase/logs - name: Write versions to file run: | echo "branch:${{ env.branchName }}" > ${{ env.datetime }}.txt echo "os:$(lsb_release -ds)" >> ${{ env.datetime }}.txt echo "R:$(R --version | head -n1)" >> ${{ env.datetime }}.txt - working-directory: logs + working-directory: dsBase/logs - name: Commit results to testStatus if: github.repository == 'villegar/dsBase' && github.event_name != 'pull_request' From 4cfa058ca4ca290f89b5f2b4829dd12c63bab3c8 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Tue, 22 Jul 2025 15:04:24 +0100 Subject: [PATCH 18/56] Update git config command and paths to local files --- .github/workflows/R-CMD-check-source.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/R-CMD-check-source.yaml b/.github/workflows/R-CMD-check-source.yaml index 18957add..791b35e3 100644 --- a/.github/workflows/R-CMD-check-source.yaml +++ b/.github/workflows/R-CMD-check-source.yaml @@ -142,7 +142,7 @@ jobs: - name: Commit results to testStatus if: github.repository == 'villegar/dsBase' && github.event_name != 'pull_request' run: | - git config --local user.email "$GITHUB_ACTOR@users.noreply.github.com" + git config --global user.email "$GITHUB_ACTOR@users.noreply.github.com" git config --global user.name "GitHub Actions" cd testStatus git checkout master @@ -151,9 +151,9 @@ jobs: mkdir -p logs/${{ env.projectName }}/${{ env.branchName }} mkdir -p docs/${{ env.projectName }}/${{ env.branchName }}/latest - cp logs/coveragelist.csv logs/${{ env.projectName }}/${{ env.branchName }}/ - cp logs/test_results.xml logs/${{ env.projectName }}/${{ env.branchName }}/ - cp logs/${{ env.datetime }}.txt logs/${{ env.projectName }}/${{ env.branchName }}/ + cp ../dsBase/logs/*.csv logs/${{ env.projectName }}/${{ env.branchName }}/ + cp ../dsBase/logs/*.xml logs/${{ env.projectName }}/${{ env.branchName }}/ + cp ../dsBase/logs/${{ env.datetime }}.txt logs/${{ env.projectName }}/${{ env.branchName }}/ # Run status.py script here if applicable # e.g. source/status.py logs/... logs/... status.html ... From b87295090090ab74771393a30003b9af2c947730 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Tue, 22 Jul 2025 15:15:17 +0100 Subject: [PATCH 19/56] Rename workflow and update documentation --- ...R-CMD-check-source.yaml => dsBase_test_suite.yaml} | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) rename .github/workflows/{R-CMD-check-source.yaml => dsBase_test_suite.yaml} (97%) diff --git a/.github/workflows/R-CMD-check-source.yaml b/.github/workflows/dsBase_test_suite.yaml similarity index 97% rename from .github/workflows/R-CMD-check-source.yaml rename to .github/workflows/dsBase_test_suite.yaml index 791b35e3..d547ab57 100644 --- a/.github/workflows/R-CMD-check-source.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -1,18 +1,15 @@ ################################################################################ # DataSHIELD GHA test suite - dsBase # Adapted from `azure-pipelines.yml` by Roberto Villegas-Diaz -# -# OVERVIEW # # Inside the root directory $(Pipeline.Workspace) will be a file tree like: # /dsBase <- Checked out version of datashield/dsBase +# /dsBase/logs <- Where results of tests and logs are collated # /testStatus <- Checked out version of datashield/testStatus -# /logs <- Where results of tests and logs are collated -# -# As of Jul 2025 this takes ~ 15 mins to run. # +# As of Jul 2025 this takes ~ 9 mins to run. ################################################################################ -name: DataSHIELD GHA test suite +name: DataSHIELD GHA test suite for dsBase on: push: @@ -25,7 +22,7 @@ on: - cron: '0 1 * * *' # Nightly on v6.3.3-dev jobs: - run_devtools_check: + dsBase_test_suite: runs-on: ubuntu-latest timeout-minutes: 120 From 0b8a9c6a8703e387d867f8ab0d9df92493464a3a Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Tue, 22 Jul 2025 16:12:18 +0100 Subject: [PATCH 20/56] Add recursive and verbose flags to cp command --- .github/workflows/dsBase_test_suite.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index d547ab57..5546eeb9 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -148,9 +148,9 @@ jobs: mkdir -p logs/${{ env.projectName }}/${{ env.branchName }} mkdir -p docs/${{ env.projectName }}/${{ env.branchName }}/latest - cp ../dsBase/logs/*.csv logs/${{ env.projectName }}/${{ env.branchName }}/ - cp ../dsBase/logs/*.xml logs/${{ env.projectName }}/${{ env.branchName }}/ - cp ../dsBase/logs/${{ env.datetime }}.txt logs/${{ env.projectName }}/${{ env.branchName }}/ + cp -rv ../dsBase/logs/*.csv logs/${{ env.projectName }}/${{ env.branchName }}/ + cp -rv ../dsBase/logs/*.xml logs/${{ env.projectName }}/${{ env.branchName }}/ + cp -rv ../dsBase/logs/${{ env.datetime }}.txt logs/${{ env.projectName }}/${{ env.branchName }}/ # Run status.py script here if applicable # e.g. source/status.py logs/... logs/... status.html ... From b8c437d5cb5a8f17daaaec7d302ec380a746034e Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Tue, 22 Jul 2025 16:38:08 +0100 Subject: [PATCH 21/56] Simplify copying command for log files --- .github/workflows/dsBase_test_suite.yaml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 5546eeb9..3bfefcee 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -148,8 +148,7 @@ jobs: mkdir -p logs/${{ env.projectName }}/${{ env.branchName }} mkdir -p docs/${{ env.projectName }}/${{ env.branchName }}/latest - cp -rv ../dsBase/logs/*.csv logs/${{ env.projectName }}/${{ env.branchName }}/ - cp -rv ../dsBase/logs/*.xml logs/${{ env.projectName }}/${{ env.branchName }}/ + cp -rv ../dsBase/logs/* logs/${{ env.projectName }}/${{ env.branchName }}/ cp -rv ../dsBase/logs/${{ env.datetime }}.txt logs/${{ env.projectName }}/${{ env.branchName }}/ # Run status.py script here if applicable From a51f6817e9ee8b0fa43cae7d323acf2f81a1608f Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Wed, 23 Jul 2025 09:18:48 +0100 Subject: [PATCH 22/56] Update user.name for git push --- .github/workflows/dsBase_test_suite.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 3bfefcee..edc4fa78 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -140,7 +140,7 @@ jobs: if: github.repository == 'villegar/dsBase' && github.event_name != 'pull_request' run: | git config --global user.email "$GITHUB_ACTOR@users.noreply.github.com" - git config --global user.name "GitHub Actions" + git config --global user.name "$GITHUB_ACTOR" cd testStatus git checkout master git pull From d961ac06f174d7cd7f7bc53c9d10cdb9f3b94bc8 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Wed, 23 Jul 2025 09:21:55 +0100 Subject: [PATCH 23/56] Rename branchName >> BRANCH_NAME --- .github/workflows/dsBase_test_suite.yaml | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index edc4fa78..e9ed8de3 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -36,7 +36,7 @@ jobs: _r_check_system_clock_: 0 datetime: ${{ github.run_id }}-${{ github.run_attempt }} projectName: dsBase - branchName: ${{ github.ref_name }} + BRANCH_NAME: ${{ github.ref_name }} GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} R_KEEP_PKG_SOURCE: yes @@ -131,7 +131,7 @@ jobs: - name: Write versions to file run: | - echo "branch:${{ env.branchName }}" > ${{ env.datetime }}.txt + echo "branch:${{ env.BRANCH_NAME }}" > ${{ env.datetime }}.txt echo "os:$(lsb_release -ds)" >> ${{ env.datetime }}.txt echo "R:$(R --version | head -n1)" >> ${{ env.datetime }}.txt working-directory: dsBase/logs @@ -145,17 +145,17 @@ jobs: git checkout master git pull - mkdir -p logs/${{ env.projectName }}/${{ env.branchName }} - mkdir -p docs/${{ env.projectName }}/${{ env.branchName }}/latest + mkdir -p logs/${{ env.projectName }}/${{ env.BRANCH_NAME }} + mkdir -p docs/${{ env.projectName }}/${{ env.BRANCH_NAME }}/latest - cp -rv ../dsBase/logs/* logs/${{ env.projectName }}/${{ env.branchName }}/ - cp -rv ../dsBase/logs/${{ env.datetime }}.txt logs/${{ env.projectName }}/${{ env.branchName }}/ + cp -rv ../dsBase/logs/* logs/${{ env.projectName }}/${{ env.BRANCH_NAME }}/ + cp -rv ../dsBase/logs/${{ env.datetime }}.txt logs/${{ env.projectName }}/${{ env.BRANCH_NAME }}/ # Run status.py script here if applicable # e.g. source/status.py logs/... logs/... status.html ... git add . - git commit -m "Auto test for ${{ env.projectName }}/${{ env.branchName }} @ ${{ env.datetime }}" + git commit -m "Auto test for ${{ env.projectName }}/${{ env.BRANCH_NAME }} @ ${{ env.datetime }}" git push - name: Dump environment info From 2c11650804968ced0a6f8a4124edffba32497883 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Wed, 23 Jul 2025 09:26:47 +0100 Subject: [PATCH 24/56] Rename environment variables from cammel case to all upper case and add new REPO_OWNER variable --- .github/workflows/dsBase_test_suite.yaml | 33 ++++++++++++------------ 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index e9ed8de3..58c1a2b2 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -26,17 +26,18 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 120 - # These should all be constant, except test_filter. This can be used to test + # These should all be constant, except TEST_FILTER. This can be used to test # subsets of test files in the testthat directory. Options are like: # '*' <- Run all tests. # 'asNumericDS*' <- Run all asNumericDS tests, i.e. all the arg, etc. tests. # '*_smk_*' <- Run all the smoke tests for all functions. env: - test_filter: '*' + TEST_FILTER: '*' _r_check_system_clock_: 0 - datetime: ${{ github.run_id }}-${{ github.run_attempt }} - projectName: dsBase + DATETIME: ${{ github.run_id }}-${{ github.run_attempt }} + PROJECT_NAME: dsBase BRANCH_NAME: ${{ github.ref_name }} + REPO_OWNER: ${{ github.repository_owner }} GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} R_KEEP_PKG_SOURCE: yes @@ -49,8 +50,8 @@ jobs: - name: Checkout testStatus uses: actions/checkout@v4 with: - repository: villegar/testStatus - token: ${{ secrets.DATASHIELD_DEV }} + repository: ${{ env.REPO_OWNER }}/testStatus + token: ${{ secrets.GITHUB_TOKEN }} ref: master path: testStatus @@ -108,7 +109,7 @@ jobs: junit_rep <- JunitReporter$new(file = "test_results.xml"); progress_rep <- ProgressReporter$new(max_failures = 999999); multi_rep <- MultiReporter$new(reporters = list(progress_rep, junit_rep)); - testthat::test_local(filter = "${{ env.test_filter }}", reporter = multi_rep, stop_on_failure = FALSE)'"'"' + testthat::test_local(filter = "${{ env.TEST_FILTER }}", reporter = multi_rep, stop_on_failure = FALSE)'"'"' ) ) ), @@ -131,13 +132,13 @@ jobs: - name: Write versions to file run: | - echo "branch:${{ env.BRANCH_NAME }}" > ${{ env.datetime }}.txt - echo "os:$(lsb_release -ds)" >> ${{ env.datetime }}.txt - echo "R:$(R --version | head -n1)" >> ${{ env.datetime }}.txt + echo "branch:${{ env.BRANCH_NAME }}" > ${{ env.DATETIME }}.txt + echo "os:$(lsb_release -ds)" >> ${{ env.DATETIME }}.txt + echo "R:$(R --version | head -n1)" >> ${{ env.DATETIME }}.txt working-directory: dsBase/logs - name: Commit results to testStatus - if: github.repository == 'villegar/dsBase' && github.event_name != 'pull_request' + # if: github.repository == 'villegar/dsBase' && github.event_name != 'pull_request' run: | git config --global user.email "$GITHUB_ACTOR@users.noreply.github.com" git config --global user.name "$GITHUB_ACTOR" @@ -145,17 +146,17 @@ jobs: git checkout master git pull - mkdir -p logs/${{ env.projectName }}/${{ env.BRANCH_NAME }} - mkdir -p docs/${{ env.projectName }}/${{ env.BRANCH_NAME }}/latest + mkdir -p logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }} + mkdir -p docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/latest - cp -rv ../dsBase/logs/* logs/${{ env.projectName }}/${{ env.BRANCH_NAME }}/ - cp -rv ../dsBase/logs/${{ env.datetime }}.txt logs/${{ env.projectName }}/${{ env.BRANCH_NAME }}/ + cp -rv ../dsBase/logs/* logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/ + cp -rv ../dsBase/logs/${{ env.DATETIME }}.txt logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/ # Run status.py script here if applicable # e.g. source/status.py logs/... logs/... status.html ... git add . - git commit -m "Auto test for ${{ env.projectName }}/${{ env.BRANCH_NAME }} @ ${{ env.datetime }}" + git commit -m "Auto test for ${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }} @ ${{ env.DATETIME }}" git push - name: Dump environment info From b5eada4cc010e572db229ad467b888a8a5ca1c3b Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Wed, 23 Jul 2025 09:47:07 +0100 Subject: [PATCH 25/56] Ammend token secret variable name and structure of files --- .github/workflows/dsBase_test_suite.yaml | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 58c1a2b2..cf2a310e 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -51,7 +51,7 @@ jobs: uses: actions/checkout@v4 with: repository: ${{ env.REPO_OWNER }}/testStatus - token: ${{ secrets.GITHUB_TOKEN }} + token: ${{ secrets.DATASHIELD_DEV }} ref: master path: testStatus @@ -146,11 +146,14 @@ jobs: git checkout master git pull - mkdir -p logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }} - mkdir -p docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/latest - - cp -rv ../dsBase/logs/* logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/ - cp -rv ../dsBase/logs/${{ env.DATETIME }}.txt logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/ + mkdir -p logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ + + # Copy logs to new logs directory location + cp -rv ../dsBase/logs/* logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ + cp -rv ../dsBase/logs/${{ env.DATETIME }}.txt logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ + + # Create symbolic link + ln -sf logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/[latest] # Run status.py script here if applicable # e.g. source/status.py logs/... logs/... status.html ... From 9a47b4d46a6c1bb7eaca9163b340a26d1c165ce6 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Wed, 23 Jul 2025 10:01:37 +0100 Subject: [PATCH 26/56] Update location of symbolic link --- .github/workflows/dsBase_test_suite.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index cf2a310e..eaa433d9 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -153,7 +153,7 @@ jobs: cp -rv ../dsBase/logs/${{ env.DATETIME }}.txt logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ # Create symbolic link - ln -sf logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/[latest] + ln -sf logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/[latest] # Run status.py script here if applicable # e.g. source/status.py logs/... logs/... status.html ... From 668041180346aa3af72dafcaac643e6e2c9ed5e2 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Wed, 23 Jul 2025 10:12:41 +0100 Subject: [PATCH 27/56] Test a new way to get GitHub authentication token --- .github/workflows/dsBase_test_suite.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index eaa433d9..f28b6fdd 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -51,7 +51,7 @@ jobs: uses: actions/checkout@v4 with: repository: ${{ env.REPO_OWNER }}/testStatus - token: ${{ secrets.DATASHIELD_DEV }} + token: ${{ github.token }} ref: master path: testStatus From 4023d0376e0649cf751fec80aa9d515a2de454db Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Wed, 23 Jul 2025 11:28:31 +0100 Subject: [PATCH 28/56] Revert to secrets.DATASHIELD_DEV for token --- .github/workflows/dsBase_test_suite.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index f28b6fdd..eaa433d9 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -51,7 +51,7 @@ jobs: uses: actions/checkout@v4 with: repository: ${{ env.REPO_OWNER }}/testStatus - token: ${{ github.token }} + token: ${{ secrets.DATASHIELD_DEV }} ref: master path: testStatus From 152de9bf0b215d30bf469df33e910ef437fcd2a4 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Wed, 23 Jul 2025 12:01:34 +0100 Subject: [PATCH 29/56] Change function call for tests: testthat::test_local >> testthat::test_package --- .github/workflows/dsBase_test_suite.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index eaa433d9..2ceb0907 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -109,7 +109,7 @@ jobs: junit_rep <- JunitReporter$new(file = "test_results.xml"); progress_rep <- ProgressReporter$new(max_failures = 999999); multi_rep <- MultiReporter$new(reporters = list(progress_rep, junit_rep)); - testthat::test_local(filter = "${{ env.TEST_FILTER }}", reporter = multi_rep, stop_on_failure = FALSE)'"'"' + testthat::test_package("${{ env.PROJECT_NAME }}", filter = "${{ env.TEST_FILTER }}", reporter = multi_rep, stop_on_failure = FALSE)'"'"' ) ) ), From 678940b201abf468720a20494c277065f1c96609 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Wed, 23 Jul 2025 12:08:05 +0100 Subject: [PATCH 30/56] Add GHA workflow for deployment of package documentation website generated with pkgdown --- .github/workflows/pkgdown.yaml | 49 ++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 .github/workflows/pkgdown.yaml diff --git a/.github/workflows/pkgdown.yaml b/.github/workflows/pkgdown.yaml new file mode 100644 index 00000000..bfc9f4db --- /dev/null +++ b/.github/workflows/pkgdown.yaml @@ -0,0 +1,49 @@ +# Workflow derived from https://github.com/r-lib/actions/tree/v2/examples +# Need help debugging build failures? Start at https://github.com/r-lib/actions#where-to-find-help +on: + push: + branches: [main, master] + pull_request: + release: + types: [published] + workflow_dispatch: + +name: pkgdown.yaml + +permissions: read-all + +jobs: + pkgdown: + runs-on: ubuntu-latest + # Only restrict concurrency for non-PR jobs + concurrency: + group: pkgdown-${{ github.event_name != 'pull_request' || github.run_id }} + env: + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + permissions: + contents: write + steps: + - uses: actions/checkout@v4 + + - uses: r-lib/actions/setup-pandoc@v2 + + - uses: r-lib/actions/setup-r@v2 + with: + use-public-rspm: true + + - uses: r-lib/actions/setup-r-dependencies@v2 + with: + extra-packages: any::pkgdown, local::. + needs: website + + - name: Build site + run: pkgdown::build_site_github_pages(new_process = FALSE, install = FALSE) + shell: Rscript {0} + + - name: Deploy to GitHub pages 🚀 + if: github.event_name != 'pull_request' + uses: JamesIves/github-pages-deploy-action@v4.5.0 + with: + clean: false + branch: gh-pages + folder: docs From 8297d11d0e4e4cef0ea548c321fb5c619201e776 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Wed, 23 Jul 2025 12:19:16 +0100 Subject: [PATCH 31/56] Update run tests step to remove direct calls to library --- .github/workflows/dsBase_test_suite.yaml | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 2ceb0907..18e97edc 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -96,26 +96,25 @@ jobs: mkdir -p logs R -q -e "library('devtools'); devtools::reload();" R -q -e ' - library(covr); write.csv( - coverage_to_list( + covr::coverage_to_list( covr::package_coverage( type = c("none"), - code = c( - '"'"'library(testthat); + code = c(' output_file <- file("test_console_output.txt"); sink(output_file); sink(output_file, type = "message"); - junit_rep <- JunitReporter$new(file = "test_results.xml"); - progress_rep <- ProgressReporter$new(max_failures = 999999); - multi_rep <- MultiReporter$new(reporters = list(progress_rep, junit_rep)); - testthat::test_package("${{ env.PROJECT_NAME }}", filter = "${{ env.TEST_FILTER }}", reporter = multi_rep, stop_on_failure = FALSE)'"'"' + junit_rep <- testthat::JunitReporter$new(file = "test_results.xml"); + progress_rep <- testthat::ProgressReporter$new(max_failures = 999999); + multi_rep <- testthat::MultiReporter$new(reporters = list(progress_rep, junit_rep)); + testthat::test_package("${{ env.PROJECT_NAME }}", filter = "${{ env.TEST_FILTER }}", reporter = multi_rep, stop_on_failure = FALSE)' ) ) ), "coveragelist.csv" )' + ls * mv coveragelist.csv logs/ mv test_* logs/ grep -q " FAIL 0 " logs/test_console_output.txt From d1e27c38e3191864c1c9b150bb3812269be7a055 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Wed, 23 Jul 2025 12:28:07 +0100 Subject: [PATCH 32/56] Fix quotation marks --- .github/workflows/dsBase_test_suite.yaml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 18e97edc..9bfaea6e 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -100,14 +100,14 @@ jobs: covr::coverage_to_list( covr::package_coverage( type = c("none"), - code = c(' - output_file <- file("test_console_output.txt"); + code = c(" + output_file <- file('test_console_output.txt'); sink(output_file); - sink(output_file, type = "message"); - junit_rep <- testthat::JunitReporter$new(file = "test_results.xml"); + sink(output_file, type = 'message'); + junit_rep <- testthat::JunitReporter$new(file = 'test_results.xml'); progress_rep <- testthat::ProgressReporter$new(max_failures = 999999); multi_rep <- testthat::MultiReporter$new(reporters = list(progress_rep, junit_rep)); - testthat::test_package("${{ env.PROJECT_NAME }}", filter = "${{ env.TEST_FILTER }}", reporter = multi_rep, stop_on_failure = FALSE)' + testthat::test_package('${{ env.PROJECT_NAME }}', filter = '${{ env.TEST_FILTER }}', reporter = multi_rep, stop_on_failure = FALSE)'" ) ) ), From d755066a951bcd9bbd3ff96a973109186274a5c4 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Wed, 23 Jul 2025 12:36:20 +0100 Subject: [PATCH 33/56] Revert changes with single/double quotes surrounding code snip --- .github/workflows/dsBase_test_suite.yaml | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 9bfaea6e..26264c83 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -94,20 +94,20 @@ jobs: - name: Run tests with coverage & JUnit report run: | mkdir -p logs - R -q -e "library('devtools'); devtools::reload();" + R -q -e "devtools::reload();" R -q -e ' write.csv( covr::coverage_to_list( covr::package_coverage( type = c("none"), - code = c(" - output_file <- file('test_console_output.txt'); + code = c('"'"' + output_file <- file("test_console_output.txt"); sink(output_file); - sink(output_file, type = 'message'); - junit_rep <- testthat::JunitReporter$new(file = 'test_results.xml'); + sink(output_file, type = "message"); + junit_rep <- testthat::JunitReporter$new(file = "test_results.xml"); progress_rep <- testthat::ProgressReporter$new(max_failures = 999999); multi_rep <- testthat::MultiReporter$new(reporters = list(progress_rep, junit_rep)); - testthat::test_package('${{ env.PROJECT_NAME }}', filter = '${{ env.TEST_FILTER }}', reporter = multi_rep, stop_on_failure = FALSE)'" + testthat::test_package("${{ env.PROJECT_NAME }}", filter = "${{ env.TEST_FILTER }}", reporter = multi_rep, stop_on_failure = FALSE)'"'"' ) ) ), From e41608d42417998720a314d7bd32f6b4b2c0a522 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Wed, 23 Jul 2025 12:40:29 +0100 Subject: [PATCH 34/56] Regenerate documentation with pkgdown::build_site() --- docs/authors.html | 54 ++++++++++++++++-- docs/index.html | 57 ++++++++++++++----- docs/pkgdown.yml | 4 +- docs/reference/BooleDS.html | 7 ++- docs/reference/asFactorDS2.html | 2 +- .../checkPermissivePrivacyControlLevel.html | 10 +++- docs/reference/covDS.html | 2 +- docs/reference/glmDS1.html | 8 ++- docs/reference/glmDS2.html | 8 ++- docs/reference/heatmapPlotDS.html | 2 +- docs/reference/histogramDS1.html | 6 +- docs/reference/histogramDS2.html | 2 +- docs/reference/lexisDS1.html | 8 ++- docs/reference/lexisDS2.html | 8 ++- docs/reference/lexisDS3.html | 4 ++ docs/reference/listDisclosureSettingsDS.html | 4 ++ docs/reference/matrixMultDS.html | 2 +- docs/reference/meanSdGpDS.html | 12 ++-- docs/reference/scatterPlotDS.html | 2 +- docs/reference/subsetDS.html | 14 ++--- docs/reference/table1DDS.html | 2 +- docs/reference/table2DDS.html | 2 +- docs/reference/testObjExistsDS.html | 10 +++- 23 files changed, 172 insertions(+), 58 deletions(-) diff --git a/docs/authors.html b/docs/authors.html index cc06044e..70570f2a 100644 --- a/docs/authors.html +++ b/docs/authors.html @@ -88,16 +88,62 @@

Citation

-

Burton P, Wilson R, Butters O, Ryser-Welch P, Westerberg A, Abarrategui L, Villegas-Diaz R, Avraam D, Marcon Y, Wheater S (2025). -dsBase: DataSHIELD Server Site Base Functions. +

Burton P, Wilson R, Butters O, Ryser-Welch P, Westerberg A, Abarrategui L, Villegas-Diaz R, Avraam D, Marcon Y, Wheater S (????). +dsBase: 'DataSHIELD' Server Site Base Functions. R package version 6.3.3. 

@Manual{,
-  title = {dsBase: DataSHIELD Server Site Base Functions},
+  title = {dsBase: 'DataSHIELD' Server Site Base Functions},
   author = {Paul Burton and Rebecca Wilson and Olly Butters and Patricia Ryser-Welch and Alex Westerberg and Leire Abarrategui and Roberto Villegas-Diaz and Demetris Avraam and Yannick Marcon and Stuart Wheater},
-  year = {2025},
   note = {R package version 6.3.3},
 }
+

Gaye A, Marcon Y, Isaeva J, LaFlamme P, Turner A, Jones E, Minion J, Boyd A, Newby C, Nuotio M, Wilson R, Butters O, Murtagh B, Demir I, Doiron D, Giepmans L, Wallace S, Budin-Ljøsne I, Oliver Schmidt C, Boffetta P, Boniol M, Bota M, Carter K, deKlerk N, Dibben C, Francis R, Hiekkalinna T, Hveem K, Kvaløy K, Millar S, Perry I, Peters A, Phillips C, Popham F, Raab G, Reischl E, Sheehan N, Waldenberger M, Perola M, van den Heuvel E, Macleod J, Knoppers B, Stolk R, Fortier I, Harris J, Woffenbuttel B, Murtagh M, Ferretti V, Burton P (2014). +“DataSHIELD: taking the analysis to the data, not the data to the analysis.” +International Journal of Epidemiology, 43(6), 1929–1944. +doi:10.1093/ije/dyu188. +

+ 
@Article{,
+  title = {{DataSHIELD: taking the analysis to the data, not the data to the analysis}},
+  author = {Amadou Gaye and Yannick Marcon and Julia Isaeva and Philippe {LaFlamme} and Andrew Turner and Elinor M Jones and Joel Minion and Andrew W Boyd and Christopher J Newby and Marja-Liisa Nuotio and Rebecca Wilson and Oliver Butters and Barnaby Murtagh and Ipek Demir and Dany Doiron and Lisette Giepmans and Susan E Wallace and Isabelle Budin-Lj{\o}sne and Carsten {{Oliver Schmidt}} and Paolo Boffetta and Mathieu Boniol and Maria Bota and Kim W Carter and Nick {deKlerk} and Chris Dibben and Richard W Francis and Tero Hiekkalinna and Kristian Hveem and Kirsti Kval{\o}y and Sean Millar and Ivan J Perry and Annette Peters and Catherine M Phillips and Frank Popham and Gillian Raab and Eva Reischl and Nuala Sheehan and Melanie Waldenberger and Markus Perola and Edwin {{van den Heuvel}} and John Macleod and Bartha M Knoppers and Ronald P Stolk and Isabel Fortier and Jennifer R Harris and Bruce H R Woffenbuttel and Madeleine J Murtagh and Vincent Ferretti and Paul R Burton},
+  journal = {International Journal of Epidemiology},
+  year = {2014},
+  volume = {43},
+  number = {6},
+  pages = {1929--1944},
+  doi = {10.1093/ije/dyu188},
+}
+

Wilson R, W. Butters O, Avraam D, Baker J, Tedds J, Turner A, Murtagh M, R. Burton P (2017). +“DataSHIELD – New Directions and Dimensions.” +Data Science Journal, 16(21), 1–21. +doi:10.5334/dsj-2017-021. +

+ 
@Article{,
+  title = {{DataSHIELD – New Directions and Dimensions}},
+  author = {Rebecca C. Wilson and Oliver {W. Butters} and Demetris Avraam and James Baker and Jonathan A. Tedds and Andrew Turner and Madeleine Murtagh and Paul {R. Burton}},
+  journal = {Data Science Journal},
+  year = {2017},
+  volume = {16},
+  number = {21},
+  pages = {1--21},
+  doi = {10.5334/dsj-2017-021},
+}
+

Avraam D, Wilson R, Aguirre Chan N, Banerjee S, Bishop T, Butters O, Cadman T, Cederkvist L, Duijts L, Escribà Montagut X, Garner H, Gonçalves G, González J, Haakma S, Hartlev M, Hasenauer J, Huth M, Hyde E, Jaddoe V, Marcon Y, Mayrhofer M, Molnar-Gabor F, Morgan A, Murtagh M, Nestor M, Nybo Andersen A, Parker S, Pinot de Moira A, Schwarz F, Strandberg-Larsen K, Morris AvSwertz, Welten M, Wheater S, Burton P (2024). +“DataSHIELD: mitigating disclosure risk in a multi-site federated analysis platform.” +Bioinformatics Advances, 5(1), 1–21. +doi:10.1093/bioadv/vbaf046. +

+ 
@Article{,
+  title = {{DataSHIELD: mitigating disclosure risk in a multi-site federated analysis platform}},
+  author = {Demetris Avraam and Rebecca C Wilson and Noemi {{Aguirre Chan}} and Soumya Banerjee and Tom R P Bishop and Olly Butters and Tim Cadman and Luise Cederkvist and Liesbeth Duijts and Xavier {{Escrib{\a`a} Montagut}} and Hugh Garner and Gon{\c c}alo {Gon{\c c}alves} and Juan R Gonz{\a'a}lez and Sido Haakma and Mette Hartlev and Jan Hasenauer and Manuel Huth and Eleanor Hyde and Vincent W V Jaddoe and Yannick Marcon and Michaela Th Mayrhofer and Fruzsina Molnar-Gabor and Andrei Scott Morgan and Madeleine Murtagh and Marc Nestor and Anne-Marie {{Nybo Andersen}} and Simon Parker and Angela {{Pinot de Moira}} and Florian Schwarz and Katrine Strandberg-Larsen and {Morris AvSwertz} and Marieke Welten and Stuart Wheater and Paul Burton},
+  journal = {Bioinformatics Advances},
+  year = {2024},
+  volume = {5},
+  number = {1},
+  pages = {1--21},
+  doi = {10.1093/bioadv/vbaf046},
+  editor = {Thomas Lengauer},
+  publisher = {Oxford University Press (OUP)},
+}
diff --git a/docs/index.html b/docs/index.html index 54bbe6e5..ca9ac5a7 100644 --- a/docs/index.html +++ b/docs/index.html @@ -12,7 +12,7 @@ - +

+
+install.packages("remotes")
+remotes::install_github("datashield/dsBase", "<BRANCH>")
+
+# Install v6.4.0 with the following
+remotes::install_github("datashield/dsBase", "v6.4.0-dev")
+

For a full list of development branches, checkout https://github.com/datashield/dsBase/branches

+ +
+

About +

DataSHIELD is a software package which allows you to do non-disclosive federated analysis on sensitive data. Our website (https://www.datashield.org) has in depth descriptions of what it is, how it works and how to install it. A key point to highlight is that DataSHIELD has a client-server infrastructure, so the dsBase package (https://github.com/datashield/dsBase) needs to be used in conjuction with the dsBaseClient package (https://github.com/datashield/dsBaseClient) - trying to use one without the other makes no sense.

Detailed instructions on how to install DataSHIELD are at https://wiki.datashield.org/. The code here is organised as:

@@ -72,31 +86,46 @@

About

- + - + - + - +
Location What is it?
obiba CRAN Where you probably should install DataSHIELD from.
releases Stable releases.
master branch Mostly in sync with the latest release, changes rarely.
+
+
+

References +

+

[1] Burton P, Wilson R, Butters O, Ryser-Welch P, Westerberg A, Abarrategui L, Villegas-Diaz R, Avraam D, Marcon Y, Bishop T, Gaye A, Escribà Montagut X, Wheater S (2025). dsBase: ‘DataSHIELD’ Server Side Base Functions. R package version 6.3.3. https://doi.org/10.32614/CRAN.package.dsBase.

+

[2] Gaye A, Marcon Y, Isaeva J, LaFlamme P, Turner A, Jones E, Minion J, Boyd A, Newby C, Nuotio M, Wilson R, Butters O, Murtagh B, Demir I, Doiron D, Giepmans L, Wallace S, Budin-Ljøsne I, Oliver Schmidt C, Boffetta P, Boniol M, Bota M, Carter K, deKlerk N, Dibben C, Francis R, Hiekkalinna T, Hveem K, Kvaløy K, Millar S, Perry I, Peters A, Phillips C, Popham F, Raab G, Reischl E, Sheehan N, Waldenberger M, Perola M, van den Heuvel E, Macleod J, Knoppers B, Stolk R, Fortier I, Harris J, Woffenbuttel B, Murtagh M, Ferretti V, Burton P (2014). “DataSHIELD: taking the analysis to the data, not the data to the analysis.” International Journal of Epidemiology, 43(6), 1929-1944. https://doi.org/10.1093/ije/dyu188.

+

[3] Wilson R, W. Butters O, Avraam D, Baker J, Tedds J, Turner A, Murtagh M, R. Burton P (2017). “DataSHIELD – New Directions and Dimensions.” Data Science Journal, 16(21), 1-21. https://doi.org/10.5334/dsj-2017-021.

+

[4] Avraam D, Wilson R, Aguirre Chan N, Banerjee S, Bishop T, Butters O, Cadman T, Cederkvist L, Duijts L, Escribà Montagut X, Garner H, Gonçalves G, González J, Haakma S, Hartlev M, Hasenauer J, Huth M, Hyde E, Jaddoe V, Marcon Y, Mayrhofer M, Molnar-Gabor F, Morgan A, Murtagh M, Nestor M, Nybo Andersen A, Parker S, Pinot de Moira A, Schwarz F, Strandberg-Larsen K, Morris AvSwertz, Welten M, Wheater S, Burton P (2024). “DataSHIELD: mitigating disclosure risk in a multi-site federated analysis platform.” Bioinformatics Advances, 5(1), 1-21. https://doi.org/10.1093/bioadv/vbaf046.

-

This is the first serverside aggregate function called by ds.glm

+

This is the first server-side aggregate function called by ds.glm

@@ -75,6 +75,10 @@

Arguments

analysed under the specified model

+
+

Value

+

List with values from GLM model.

+

Details

It is an diff --git a/docs/reference/glmDS2.html b/docs/reference/glmDS2.html index 2cefd28f..0f9df3f4 100644 --- a/docs/reference/glmDS2.html +++ b/docs/reference/glmDS2.html @@ -1,5 +1,5 @@ -glmDS2 called by ds.glm — glmDS2 • dsBaseglmDS2 called by ds.glm — glmDS2 • dsBase @@ -40,7 +40,7 @@

glmDS2 called by ds.glm

-

This is the second serverside aggregate function called by ds.glm.

+

This is the second server-side aggregate function called by ds.glm.

@@ -79,6 +79,10 @@

Arguments

the data to be analysed under the specified model same

+
+

Value

+

List with values from GLM model

+

Details

It is an aggregate function that uses the model structure and starting diff --git a/docs/reference/heatmapPlotDS.html b/docs/reference/heatmapPlotDS.html index eb3715fa..cf9941b6 100644 --- a/docs/reference/heatmapPlotDS.html +++ b/docs/reference/heatmapPlotDS.html @@ -60,7 +60,7 @@

Arguments

k
-

the number of the nearest neghbours for which their centroid is calculated if the +

the number of the nearest neighbours for which their centroid is calculated if the method.indicator is equal to 1 (i.e. deterministic method).

diff --git a/docs/reference/histogramDS1.html b/docs/reference/histogramDS1.html index 59888e26..2a83be22 100644 --- a/docs/reference/histogramDS1.html +++ b/docs/reference/histogramDS1.html @@ -2,7 +2,7 @@ returns the minimum and the maximum of the input numeric vector — histogramDS1 • dsBaselexisDS1 — lexisDS1 • dsBaselexisDS1 — lexisDS1 • dsBase @@ -40,7 +40,7 @@

lexisDS1

-

The first serverside function called by ds.lexis.

+

The first server-side function called by ds.lexis.

@@ -55,6 +55,10 @@

Arguments

a character string specifying the variable holding the time that each individual is censored or fails

+
+

Value

+

List with `max.time`

+

Details

This is an aggregate function. diff --git a/docs/reference/lexisDS2.html b/docs/reference/lexisDS2.html index 39b99d50..89ebeabf 100644 --- a/docs/reference/lexisDS2.html +++ b/docs/reference/lexisDS2.html @@ -78,7 +78,7 @@

Arguments

idCol

a clientside generated character string specifying the variable -holding the IDs of indivuals in the data set to be expanded

+holding the IDs of individuals in the data set to be expanded

entryCol
@@ -103,6 +103,10 @@

Arguments

'data' argument is set the full data.frame will be expanded and carried forward

+
+

Value

+

List with `expanded.table`

+

Details

This is the assign @@ -111,7 +115,7 @@

Details

regression. lexisDS2 also carries out a series of disclosure checks and if the arguments or data fail any of those tests, -creation of the exapanded dataframe is blocked and an appropriate serverside error +creation of the expanded dataframe is blocked and an appropriate serverside error message is stored. For more details see the extensive header for ds.lexis.

diff --git a/docs/reference/lexisDS3.html b/docs/reference/lexisDS3.html index 023ff8e4..dc237cda 100644 --- a/docs/reference/lexisDS3.html +++ b/docs/reference/lexisDS3.html @@ -47,6 +47,10 @@

@title lexisDS3

lexisDS3()
+
+

Value

+

Data frame with `messageobj` object

+

Details

This is an assign function that simplifies the diff --git a/docs/reference/listDisclosureSettingsDS.html b/docs/reference/listDisclosureSettingsDS.html index 10134413..549c94e3 100644 --- a/docs/reference/listDisclosureSettingsDS.html +++ b/docs/reference/listDisclosureSettingsDS.html @@ -49,6 +49,10 @@

listDisclosureSettingsDS

listDisclosureSettingsDS()
+
+

Value

+

List with DataSHIELD disclosure settings

+

Details

For more details see the extensive header for ds.listDisclosureSettings

diff --git a/docs/reference/matrixMultDS.html b/docs/reference/matrixMultDS.html index 6af3ff82..c782eaac 100644 --- a/docs/reference/matrixMultDS.html +++ b/docs/reference/matrixMultDS.html @@ -69,7 +69,7 @@

Value

Details

Undertakes standard matrix multiplication where with input matrices A and B with -dimensions A: mxn and B: nxp the output C has dimensions mxp and each elemnt C[i,j] has +dimensions A: mxn and B: nxp the output C has dimensions mxp and each element C[i,j] has value equal to the dot product of row i of A and column j of B where the dot product is obtained as sum(A[i,1]*B[1,j] + A[i,2]*B[2,j] + .... + A[i,n]*B[n,j]). This calculation is only valid if the number of columns of A is the same as the number of rows of B

diff --git a/docs/reference/meanSdGpDS.html b/docs/reference/meanSdGpDS.html index d60d33d7..7e902552 100644 --- a/docs/reference/meanSdGpDS.html +++ b/docs/reference/meanSdGpDS.html @@ -1,5 +1,5 @@ -MeanSdGpDS — meanSdGpDS • dsBaseMeanSdGpDS — meanSdGpDS • dsBase @@ -40,7 +40,7 @@

MeanSdGpDS

-

Serverside function called by ds.meanSdGp

+

Server-side function called by ds.meanSdGp

@@ -52,15 +52,19 @@

Arguments

X
-

a clientside supplied character string identifying the variable for which +

a client-side supplied character string identifying the variable for which means/SDs are to be calculated

INDEX
-

a clientside supplied character string identifying the factor across +

a client-side supplied character string identifying the factor across which means/SDs are to be calculated

+
+

Value

+

List with results from the group statistics

+

Details

Computes the mean and standard deviation across groups defined by one diff --git a/docs/reference/scatterPlotDS.html b/docs/reference/scatterPlotDS.html index b1298f5d..2e3d5407 100644 --- a/docs/reference/scatterPlotDS.html +++ b/docs/reference/scatterPlotDS.html @@ -68,7 +68,7 @@

Arguments

k
-

the number of the nearest neghbours for which their centroid is calculated if the +

the number of the nearest neighbours for which their centroid is calculated if the deterministic method is selected.

diff --git a/docs/reference/subsetDS.html b/docs/reference/subsetDS.html index 8bc2f861..19e73aab 100644 --- a/docs/reference/subsetDS.html +++ b/docs/reference/subsetDS.html @@ -1,7 +1,7 @@ Generates a valid subset of a table or a vector — subsetDS • dsBase @@ -43,8 +43,8 @@

Generates a valid subset of a table or a vector

The function uses the R classical subsetting with squared brackets '[]' and allows also to -subset using a logical oprator and a threshold. The object to subset from must be a vector (factor, numeric -or charcater) or a table (data.frame or matrix).

+subset using a logical operator and a threshold. The object to subset from must be a vector (factor, numeric +or character) or a table (data.frame or matrix).

@@ -92,18 +92,18 @@

Arguments

varname

a character, if the input data is a table, if this parameter is provided along with the 'logical' and 'threshold' -parameters, a subtable is based the threshold applied to the speicified variable. This parameter is however ignored if the parameter +parameters, a subtable is based the threshold applied to the specified variable. This parameter is however ignored if the parameter 'rows' and/or 'cols' are provided.

Value

-

a subset of the vector, matric or dataframe as specified is stored on the server side

+

a subset of the vector, matrix or dataframe as specified is stored on the server side

Details

If the input data is a table: The user specifies the rows and/or columns to include in the subset if the input -object is a table; the columns can be refered to by their names. The name of a vector (i.e. a variable) can also be provided +object is a table; the columns can be referred to by their names. The name of a vector (i.e. a variable) can also be provided with a logical operator and a threshold (see example 3). If the input data is a vector: when the parameters 'rows', 'logical' and 'threshold' are all provided the last two are ignored ( 'rows' has precedence over the other two parameters then). diff --git a/docs/reference/table1DDS.html b/docs/reference/table1DDS.html index 6796312e..82d157f7 100644 --- a/docs/reference/table1DDS.html +++ b/docs/reference/table1DDS.html @@ -65,7 +65,7 @@

Value

Details

It generates a 1-dimensional tables where valid (non-disclosive) 1-dimensional tables are defined -as data from sources where no table cells have counts between 1 and the set threshold. When the ouput table +as data from sources where no table cells have counts between 1 and the set threshold. When the output table is invalid all cells but the total count are replaced by missing values. Only the total count is visible on the table returned to the client site. A message is also returned with the 1-dimensional; the message says "invalid table - invalid counts present" if the table is invalid and 'valid table' otherwise.

diff --git a/docs/reference/table2DDS.html b/docs/reference/table2DDS.html index 501a2273..0b5f74e1 100644 --- a/docs/reference/table2DDS.html +++ b/docs/reference/table2DDS.html @@ -69,7 +69,7 @@

Value

Details

It generates 2-dimensional contingency tables where valid (non-disclosive) tables are defined -as those where none of their cells have counts between 1 and the set threshold "nfilter.tab". When the ouput table +as those where none of their cells have counts between 1 and the set threshold "nfilter.tab". When the output table is invalid all cells except the total counts are replaced by missing values. Only the total counts are visible on the table returned to the client side. A message is also returned with the 2-dimensional table; the message says "invalid table - invalid counts present" if the table is invalid and 'valid table' otherwise.

diff --git a/docs/reference/testObjExistsDS.html b/docs/reference/testObjExistsDS.html index 09cb4b5f..a0246beb 100644 --- a/docs/reference/testObjExistsDS.html +++ b/docs/reference/testObjExistsDS.html @@ -1,5 +1,5 @@ -testObjExistsDS — testObjExistsDS • dsBasetestObjExistsDS — testObjExistsDS • dsBase @@ -40,7 +40,7 @@

testObjExistsDS

-

The serverside function called by ds.testObjExists

+

The server-side function called by ds.testObjExists

@@ -52,10 +52,14 @@

Arguments

test.obj.name
-

a clientside provided character string specifying the variable +

a client-side provided character string specifying the variable whose presence is to be tested in each data source

+
+

Value

+

List with `test.obj.exists` and `test.obj.class`

+

Details

Tests whether a given object exists in From fb2780ac3e1471c2b6de10329982104ae7e84b19 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Wed, 23 Jul 2025 12:44:36 +0100 Subject: [PATCH 35/56] Remove superfluous text --- docs/authors.html | 4 ---- docs/pkgdown.yml | 6 +----- 2 files changed, 1 insertion(+), 9 deletions(-) diff --git a/docs/authors.html b/docs/authors.html index fe73b1d0..70570f2a 100644 --- a/docs/authors.html +++ b/docs/authors.html @@ -88,11 +88,7 @@

Citation

-<<<<<<< HEAD

Burton P, Wilson R, Butters O, Ryser-Welch P, Westerberg A, Abarrategui L, Villegas-Diaz R, Avraam D, Marcon Y, Wheater S (????). -======= -

Burton P, Wilson R, Butters O, Ryser-Welch P, Westerberg A, Abarrategui L, Villegas-Diaz R, Avraam D, Marcon Y, Wheater S (2025). ->>>>>>> refs/remotes/origin/v6.3.3-dev dsBase: 'DataSHIELD' Server Site Base Functions. R package version 6.3.3.

diff --git a/docs/pkgdown.yml b/docs/pkgdown.yml index 7f059d1c..dbbcf4e6 100644 --- a/docs/pkgdown.yml +++ b/docs/pkgdown.yml @@ -2,8 +2,4 @@ pandoc: '3.4' pkgdown: 2.1.3 pkgdown_sha: ~ articles: {} -<<<<<<< HEAD -last_built: 2025-07-23T11:39Z -======= -last_built: 2025-07-16T10:31Z ->>>>>>> refs/remotes/origin/v6.3.3-dev +last_built: 2025-07-23T11:43Z From db494825a6cee80683dbcb7bbbe39f5dd1bf96d6 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Wed, 23 Jul 2025 12:53:05 +0100 Subject: [PATCH 36/56] Fix path for reporter output --- .github/workflows/dsBase_test_suite.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 26264c83..f4d0487d 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -104,7 +104,7 @@ jobs: output_file <- file("test_console_output.txt"); sink(output_file); sink(output_file, type = "message"); - junit_rep <- testthat::JunitReporter$new(file = "test_results.xml"); + junit_rep <- testthat::JunitReporter$new(file = file.path(getwd(), "test_results.xml")); progress_rep <- testthat::ProgressReporter$new(max_failures = 999999); multi_rep <- testthat::MultiReporter$new(reporters = list(progress_rep, junit_rep)); testthat::test_package("${{ env.PROJECT_NAME }}", filter = "${{ env.TEST_FILTER }}", reporter = multi_rep, stop_on_failure = FALSE)'"'"' From 1a6a74183ee31a140852a348cdbc6bd161af34ec Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Wed, 23 Jul 2025 12:58:16 +0100 Subject: [PATCH 37/56] Update symbolic link --- .github/workflows/dsBase_test_suite.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index f4d0487d..1df06485 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -152,7 +152,7 @@ jobs: cp -rv ../dsBase/logs/${{ env.DATETIME }}.txt logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ # Create symbolic link - ln -sf logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/[latest] + ln -sf ${{ env.DATETIME }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/.LATEST # Run status.py script here if applicable # e.g. source/status.py logs/... logs/... status.html ... From 771a91c38b3d34988c0b54a6868c268d8d93b770 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 08:48:31 +0100 Subject: [PATCH 38/56] Add additional dependencies to run the parser for test results --- .github/workflows/dsBase_test_suite.yaml | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 1df06485..ba3ddb35 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -68,6 +68,15 @@ jobs: extra-packages: | any::rcmdcheck cran::devtools + cran::git2r + cran::RCurl + cran::readr + cran::magrittr + cran::xml2 + cran::purrr + cran::dplyr + cran::stringr + cran::tidyr needs: check - name: Check man files up-to-date @@ -135,6 +144,10 @@ jobs: echo "os:$(lsb_release -ds)" >> ${{ env.DATETIME }}.txt echo "R:$(R --version | head -n1)" >> ${{ env.DATETIME }}.txt working-directory: dsBase/logs + + - name: Parse results from testthat and covr + run: | + Rscript --verbose --vanilla testStatus/source/parse_test_report.R dsBase/logs/ - name: Commit results to testStatus # if: github.repository == 'villegar/dsBase' && github.event_name != 'pull_request' From e8fed4d1a5727e183982a8e8a375ab6f67977bc7 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 08:57:13 +0100 Subject: [PATCH 39/56] Change working directory to prevent error message when calling git2r::remote_url() --- .github/workflows/dsBase_test_suite.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index ba3ddb35..fe58c120 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -147,7 +147,8 @@ jobs: - name: Parse results from testthat and covr run: | - Rscript --verbose --vanilla testStatus/source/parse_test_report.R dsBase/logs/ + Rscript --verbose --vanilla ../testStatus/source/parse_test_report.R logs/ + working-directory: dsBase - name: Commit results to testStatus # if: github.repository == 'villegar/dsBase' && github.event_name != 'pull_request' From c61416bd8a2c781e407ef9ec1dc8e4cef5adc87f Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 11:39:09 +0100 Subject: [PATCH 40/56] Add additional packages to render test report in HTML format --- .github/workflows/dsBase_test_suite.yaml | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index fe58c120..de510a7a 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -77,6 +77,9 @@ jobs: cran::dplyr cran::stringr cran::tidyr + cran::quarto + cran::knitr + cran::kableExtra needs: check - name: Check man files up-to-date @@ -160,14 +163,17 @@ jobs: git pull mkdir -p logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ + mkdir -p docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ # Copy logs to new logs directory location cp -rv ../dsBase/logs/* logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ cp -rv ../dsBase/logs/${{ env.DATETIME }}.txt logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ - # Create symbolic link + # Create symbolic links ln -sf ${{ env.DATETIME }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/.LATEST - + ln -sf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/latest + + R -e "quarto::render('source/test_report.qmd', execute_params = list(input_dir = '"logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}"'), output_file = '"docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/"index.html')" # Run status.py script here if applicable # e.g. source/status.py logs/... logs/... status.html ... From 1c46394802627c0ea332407cfc5f57f19a8aa16c Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 11:48:46 +0100 Subject: [PATCH 41/56] Correct function call render >> quarto_render --- .github/workflows/dsBase_test_suite.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index de510a7a..39c9f7c9 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -173,7 +173,7 @@ jobs: ln -sf ${{ env.DATETIME }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/.LATEST ln -sf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/latest - R -e "quarto::render('source/test_report.qmd', execute_params = list(input_dir = '"logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}"'), output_file = '"docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/"index.html')" + R -e "quarto::quarto_render('source/test_report.qmd', execute_params = list(input_dir = '"logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}"'), output_file = '"docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/"index.html')" # Run status.py script here if applicable # e.g. source/status.py logs/... logs/... status.html ... From 36c31600d71bb160ca92b2243abd0e54fe01fd23 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 12:03:06 +0100 Subject: [PATCH 42/56] Remove additional quotation marks --- .github/workflows/dsBase_test_suite.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 39c9f7c9..fa0da83f 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -173,7 +173,7 @@ jobs: ln -sf ${{ env.DATETIME }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/.LATEST ln -sf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/latest - R -e "quarto::quarto_render('source/test_report.qmd', execute_params = list(input_dir = '"logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}"'), output_file = '"docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/"index.html')" + R -e "quarto::quarto_render('source/test_report.qmd', execute_params = list(input_dir = 'logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}'), output_file = 'docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/index.html')" # Run status.py script here if applicable # e.g. source/status.py logs/... logs/... status.html ... From dbc689c9afca02de19cb7381363ece700c846fec Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 12:21:02 +0100 Subject: [PATCH 43/56] Update command to render quarto report --- .github/workflows/dsBase_test_suite.yaml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index fa0da83f..d131dd18 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -173,9 +173,8 @@ jobs: ln -sf ${{ env.DATETIME }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/.LATEST ln -sf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/latest - R -e "quarto::quarto_render('source/test_report.qmd', execute_params = list(input_dir = 'logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}'), output_file = 'docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/index.html')" - # Run status.py script here if applicable - # e.g. source/status.py logs/... logs/... status.html ... + R -e 'input_dir <- file.path("logs", Sys.getenv("env.PROJECT_NAME"), Sys.getenv("env.BRANCH_NAME")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir), output_file = paste0(Sys.getenv("env.DATETIME"), "html"))' + mv source/${{ env.DATETIME }}.html docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/index.html git add . git commit -m "Auto test for ${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }} @ ${{ env.DATETIME }}" From 23a22bf54165f1f92aba951f98b4383a93f65dda Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 12:42:35 +0100 Subject: [PATCH 44/56] Fix location of input for quarto_render command --- .github/workflows/dsBase_test_suite.yaml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index d131dd18..c6329d99 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -126,7 +126,6 @@ jobs: "coveragelist.csv" )' - ls * mv coveragelist.csv logs/ mv test_* logs/ grep -q " FAIL 0 " logs/test_console_output.txt @@ -173,7 +172,7 @@ jobs: ln -sf ${{ env.DATETIME }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/.LATEST ln -sf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/latest - R -e 'input_dir <- file.path("logs", Sys.getenv("env.PROJECT_NAME"), Sys.getenv("env.BRANCH_NAME")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir), output_file = paste0(Sys.getenv("env.DATETIME"), "html"))' + R -e 'input_dir <- file.path("../logs", Sys.getenv("env.PROJECT_NAME"), Sys.getenv("env.BRANCH_NAME")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir), output_file = paste0(Sys.getenv("env.DATETIME"), ".html"))' mv source/${{ env.DATETIME }}.html docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/index.html git add . From f9b4ca7c06593ea1bea5b0b95b2c40a9abd15976 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 12:53:36 +0100 Subject: [PATCH 45/56] Add line for debugging --- .github/workflows/dsBase_test_suite.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index c6329d99..92e287a8 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -172,6 +172,7 @@ jobs: ln -sf ${{ env.DATETIME }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/.LATEST ln -sf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/latest + R -e 'input_dir <- file.path("../logs", Sys.getenv("env.PROJECT_NAME"), Sys.getenv("env.BRANCH_NAME"));print(input_dir);list.files(input_dir);' R -e 'input_dir <- file.path("../logs", Sys.getenv("env.PROJECT_NAME"), Sys.getenv("env.BRANCH_NAME")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir), output_file = paste0(Sys.getenv("env.DATETIME"), ".html"))' mv source/${{ env.DATETIME }}.html docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/index.html From b6f86dff8b7a2b96715a22517869476b7f2f2a62 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 13:05:36 +0100 Subject: [PATCH 46/56] Set up local environment variables --- .github/workflows/dsBase_test_suite.yaml | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 92e287a8..ea60f679 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -172,13 +172,18 @@ jobs: ln -sf ${{ env.DATETIME }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/.LATEST ln -sf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/latest - R -e 'input_dir <- file.path("../logs", Sys.getenv("env.PROJECT_NAME"), Sys.getenv("env.BRANCH_NAME"));print(input_dir);list.files(input_dir);' - R -e 'input_dir <- file.path("../logs", Sys.getenv("env.PROJECT_NAME"), Sys.getenv("env.BRANCH_NAME")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir), output_file = paste0(Sys.getenv("env.DATETIME"), ".html"))' + R -e 'input_dir <- file.path("../logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME"));print(input_dir);list.files(input_dir);' + R -e 'input_dir <- file.path("../logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir), output_file = paste0(Sys.getenv("DATETIME"), ".html"))' mv source/${{ env.DATETIME }}.html docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/index.html git add . git commit -m "Auto test for ${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }} @ ${{ env.DATETIME }}" git push + + env: + PROJECT_NAME: ${{ env.PROJECT_NAME }} + BRANCH_NAME: ${{ env.BRANCH_NAME }} + DATETIME: ${{ env.DATETIME }} - name: Dump environment info run: | From 510422a995a33f3e90b072045a29f49896e59331 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 13:18:04 +0100 Subject: [PATCH 47/56] Add missing portion of path --- .github/workflows/dsBase_test_suite.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index ea60f679..8d057ec1 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -172,8 +172,8 @@ jobs: ln -sf ${{ env.DATETIME }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/.LATEST ln -sf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/latest - R -e 'input_dir <- file.path("../logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME"));print(input_dir);list.files(input_dir);' - R -e 'input_dir <- file.path("../logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir), output_file = paste0(Sys.getenv("DATETIME"), ".html"))' + R -e 'input_dir <- file.path("../logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME"), Sys.getenv("DATETIME"));print(input_dir);list.files(input_dir);' + R -e 'input_dir <- file.path("../logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME"), Sys.getenv("DATETIME")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir), output_file = paste0(Sys.getenv("DATETIME"), ".html"))' mv source/${{ env.DATETIME }}.html docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/index.html git add . From 2c26a0621c34123376f11741665aba53e0668404 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 13:29:14 +0100 Subject: [PATCH 48/56] Rename DATETIME to WORKFLOW_ID and update path to logs: ../logs >> logs --- .github/workflows/dsBase_test_suite.yaml | 30 ++++++++++++------------ 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 8d057ec1..325f5812 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -34,7 +34,7 @@ jobs: env: TEST_FILTER: '*' _r_check_system_clock_: 0 - DATETIME: ${{ github.run_id }}-${{ github.run_attempt }} + WORKFLOW_ID: ${{ github.run_id }}-${{ github.run_attempt }} PROJECT_NAME: dsBase BRANCH_NAME: ${{ github.ref_name }} REPO_OWNER: ${{ github.repository_owner }} @@ -142,9 +142,9 @@ jobs: - name: Write versions to file run: | - echo "branch:${{ env.BRANCH_NAME }}" > ${{ env.DATETIME }}.txt - echo "os:$(lsb_release -ds)" >> ${{ env.DATETIME }}.txt - echo "R:$(R --version | head -n1)" >> ${{ env.DATETIME }}.txt + echo "branch:${{ env.BRANCH_NAME }}" > ${{ env.WORKFLOW_ID }}.txt + echo "os:$(lsb_release -ds)" >> ${{ env.WORKFLOW_ID }}.txt + echo "R:$(R --version | head -n1)" >> ${{ env.WORKFLOW_ID }}.txt working-directory: dsBase/logs - name: Parse results from testthat and covr @@ -161,29 +161,29 @@ jobs: git checkout master git pull - mkdir -p logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ - mkdir -p docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ + mkdir -p logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/ + mkdir -p docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/ # Copy logs to new logs directory location - cp -rv ../dsBase/logs/* logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ - cp -rv ../dsBase/logs/${{ env.DATETIME }}.txt logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ + cp -rv ../dsBase/logs/* logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/ + cp -rv ../dsBase/logs/${{ env.WORKFLOW_ID }}.txt logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/ # Create symbolic links - ln -sf ${{ env.DATETIME }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/.LATEST - ln -sf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/ docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/latest + ln -sf ${{ env.WORKFLOW_ID }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/.LATEST + ln -sf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/ docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/latest - R -e 'input_dir <- file.path("../logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME"), Sys.getenv("DATETIME"));print(input_dir);list.files(input_dir);' - R -e 'input_dir <- file.path("../logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME"), Sys.getenv("DATETIME")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir), output_file = paste0(Sys.getenv("DATETIME"), ".html"))' - mv source/${{ env.DATETIME }}.html docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.DATETIME }}/index.html + R -e 'input_dir <- file.path("logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME"), Sys.getenv("WORKFLOW_ID"));print(input_dir);list.files(input_dir);' + R -e 'input_dir <- file.path("logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME"), Sys.getenv("WORKFLOW_ID")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir), output_file = paste0(Sys.getenv("WORKFLOW_ID"), ".html"))' + mv source/${{ env.WORKFLOW_ID }}.html docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/index.html git add . - git commit -m "Auto test for ${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }} @ ${{ env.DATETIME }}" + git commit -m "Auto test for ${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }} @ ${{ env.WORKFLOW_ID }}" git push env: PROJECT_NAME: ${{ env.PROJECT_NAME }} BRANCH_NAME: ${{ env.BRANCH_NAME }} - DATETIME: ${{ env.DATETIME }} + WORKFLOW_ID: ${{ env.WORKFLOW_ID }} - name: Dump environment info run: | From fdfca18ccc4ca0fed65d9b051c4fe5496dace2e7 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 13:39:53 +0100 Subject: [PATCH 49/56] Revert path to match the working directory structure of the quarto_render command --- .github/workflows/dsBase_test_suite.yaml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 325f5812..260a2463 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -172,8 +172,7 @@ jobs: ln -sf ${{ env.WORKFLOW_ID }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/.LATEST ln -sf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/ docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/latest - R -e 'input_dir <- file.path("logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME"), Sys.getenv("WORKFLOW_ID"));print(input_dir);list.files(input_dir);' - R -e 'input_dir <- file.path("logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME"), Sys.getenv("WORKFLOW_ID")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir), output_file = paste0(Sys.getenv("WORKFLOW_ID"), ".html"))' + R -e 'input_dir <- file.path("../logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME"), Sys.getenv("WORKFLOW_ID")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir), output_file = paste0(Sys.getenv("WORKFLOW_ID"), ".html"))' mv source/${{ env.WORKFLOW_ID }}.html docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/index.html git add . From 4078d70f0d91d08fe39a9ee6ebb5fd4ce2e9557c Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 14:03:01 +0100 Subject: [PATCH 50/56] Add missing dependencies and change output file for quarto_render --- .github/workflows/dsBase_test_suite.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 260a2463..81a68144 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -80,6 +80,8 @@ jobs: cran::quarto cran::knitr cran::kableExtra + cran::rmarkdown + cran::downlit needs: check - name: Check man files up-to-date @@ -172,8 +174,8 @@ jobs: ln -sf ${{ env.WORKFLOW_ID }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/.LATEST ln -sf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/ docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/latest - R -e 'input_dir <- file.path("../logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME"), Sys.getenv("WORKFLOW_ID")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir), output_file = paste0(Sys.getenv("WORKFLOW_ID"), ".html"))' - mv source/${{ env.WORKFLOW_ID }}.html docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/index.html + R -e 'input_dir <- file.path("../logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME"), Sys.getenv("WORKFLOW_ID")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir))' + mv source/index.html docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/index.html git add . git commit -m "Auto test for ${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }} @ ${{ env.WORKFLOW_ID }}" From 5e779b8a31b72f0cee46cf3897e6a5e83be1d79d Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 14:14:51 +0100 Subject: [PATCH 51/56] Correct name of output file from quarto_render: index.html >> test_report.html --- .github/workflows/dsBase_test_suite.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 81a68144..af86fa91 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -175,7 +175,7 @@ jobs: ln -sf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/ docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/latest R -e 'input_dir <- file.path("../logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME"), Sys.getenv("WORKFLOW_ID")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir))' - mv source/index.html docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/index.html + mv source/test_report.html docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/index.html git add . git commit -m "Auto test for ${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }} @ ${{ env.WORKFLOW_ID }}" From b025cf1f6cee9d4f229d95946ac23dd75d0a0b28 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 15:02:09 +0100 Subject: [PATCH 52/56] Replace symbolic link with copy of the latest render of the test report (GHA failed to deploy when having a symbolic link) --- .github/workflows/dsBase_test_suite.yaml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index af86fa91..70be3043 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -165,6 +165,9 @@ jobs: mkdir -p logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/ mkdir -p docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/ + mkdir -p docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/latest}/ + # clear the latest directory + rm -rf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/latest}/* # Copy logs to new logs directory location cp -rv ../dsBase/logs/* logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/ @@ -172,10 +175,11 @@ jobs: # Create symbolic links ln -sf ${{ env.WORKFLOW_ID }}/ logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/.LATEST - ln -sf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/ docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/latest + # ln -sf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/ docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/latest R -e 'input_dir <- file.path("../logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME"), Sys.getenv("WORKFLOW_ID")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir))' mv source/test_report.html docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/index.html + cp -r docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }} docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/latest git add . git commit -m "Auto test for ${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }} @ ${{ env.WORKFLOW_ID }}" From 260419fcbd0820a73df707177f0c7437ced8631c Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 22:44:04 +0100 Subject: [PATCH 53/56] Test removing unused token --- .github/workflows/dsBase_test_suite.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 70be3043..e8f1fae3 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -51,7 +51,6 @@ jobs: uses: actions/checkout@v4 with: repository: ${{ env.REPO_OWNER }}/testStatus - token: ${{ secrets.DATASHIELD_DEV }} ref: master path: testStatus From f8a414133cf1452ff20a88f2389298f2ba1c5e27 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Thu, 24 Jul 2025 23:02:54 +0100 Subject: [PATCH 54/56] Update token name --- .github/workflows/dsBase_test_suite.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index e8f1fae3..83c6e9c5 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -38,7 +38,6 @@ jobs: PROJECT_NAME: dsBase BRANCH_NAME: ${{ github.ref_name }} REPO_OWNER: ${{ github.repository_owner }} - GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} R_KEEP_PKG_SOURCE: yes steps: @@ -51,6 +50,7 @@ jobs: uses: actions/checkout@v4 with: repository: ${{ env.REPO_OWNER }}/testStatus + token: ${{ secrets.GH_TOKEN }} ref: master path: testStatus From f6c657c7fbe9c9ccdff80470777e3e403afc56d3 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Fri, 25 Jul 2025 09:50:24 +0100 Subject: [PATCH 55/56] Fix path to latest documentation directory --- .github/workflows/dsBase_test_suite.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 83c6e9c5..7cc693a2 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -164,9 +164,9 @@ jobs: mkdir -p logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/ mkdir -p docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/ - mkdir -p docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/latest}/ + mkdir -p docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/latest/ # clear the latest directory - rm -rf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/latest}/* + rm -rf docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/latest/* # Copy logs to new logs directory location cp -rv ../dsBase/logs/* logs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/ @@ -178,7 +178,7 @@ jobs: R -e 'input_dir <- file.path("../logs", Sys.getenv("PROJECT_NAME"), Sys.getenv("BRANCH_NAME"), Sys.getenv("WORKFLOW_ID")); quarto::quarto_render("source/test_report.qmd", execute_params = list(input_dir = input_dir))' mv source/test_report.html docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/index.html - cp -r docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }} docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/latest + cp -r docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/${{ env.WORKFLOW_ID }}/* docs/${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }}/latest git add . git commit -m "Auto test for ${{ env.PROJECT_NAME }}/${{ env.BRANCH_NAME }} @ ${{ env.WORKFLOW_ID }}" From 32b9639c7d5a040aae24dbaf6eed7047863d93b4 Mon Sep 17 00:00:00 2001 From: Roberto Villegas-Diaz Date: Fri, 25 Jul 2025 10:31:28 +0100 Subject: [PATCH 56/56] Update name of workflow --- .github/workflows/dsBase_test_suite.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/dsBase_test_suite.yaml b/.github/workflows/dsBase_test_suite.yaml index 7cc693a2..e951ca3a 100644 --- a/.github/workflows/dsBase_test_suite.yaml +++ b/.github/workflows/dsBase_test_suite.yaml @@ -9,7 +9,7 @@ # # As of Jul 2025 this takes ~ 9 mins to run. ################################################################################ -name: DataSHIELD GHA test suite for dsBase +name: dsBase tests' suite on: push: