Add a flag to `cdmFromCon` to skip the cdm validation

[javier-gracia-tabuenca-tuni]

Is your feature request related to a problem? Please describe.
cdmFromCon runs a very useful series of validations on the cdm
this is very useful for the first time we see a CDM but it takes to much time for every time we connect to the same cdm
(at least in bigQuery it takes more than a minute )

Describe the solution you'd like
Add parameter 'skipValidation = FALSEtocdmFromCon`

Describe alternatives you've considered

Additional context

[ablack3]

Hi @javier-gracia-tabuenca-tuni,

Have you tried using the .softValidation argument? cdm <- cdmFromCon(con, cdmSchema, writeSchema, .softValidation = T)

We could use this argument to skip more validation checks if needed.

The idea with .softValidation is to skip validation checks, but it does add more risk.

[javier-gracia-tabuenca-tuni]

Thanks @ablack3,

setting .softValidation = T doesn't lower the time much.

According to the documentation it only skips one test ?
https://darwin-eu.github.io/CDMConnector/reference/cdmFromCon.html#arg--softvalidation

[ablack3]

My guess is that the time is being spent creating the references to the cdm tables and not the tests. The checking of table names and column names should be very quick. However I have noticed that creating the table references can take time on some database systems. Let's try to identify where time is being spent using rprofvis.

Will you try running something like this?

profvis:::profvis({
  library(CDMConnector)
  con <- DBI::dbConnect(duckdb::duckdb(), eunomiaDir())
  cdm <- cdmFromCon(con, "main", "main")
})

And share the output?

If it is indeed the creation of the table references that is taking so long there isn't much we can do about it because that is a necessary step to create the CDMObject. I did think that Bigquery cached queries so that it should be faster the 2nd time the same query runs but I could be wrong.

[ablack3]

Hi @javier-gracia-tabuenca-tuni, I think what is taking so much time is not the validations but the creation of the cdm reference tables. To create those we need to query every table. Unfortunately I don't think this can be sped up. .softValidation = T should skip any time consuming validation steps (check that there are no overlapping observation periods). The column name checks are very fast. I think you're study with a 1 minute setup time. The queries that get sent are "select * from {table} limit 10". I'm not sure if bigquery could cache those so they return faster.