Skip to content

Commit

Permalink
Merge pull request #1008 from dondi/beta
Browse files Browse the repository at this point in the history
v6.0.4
ahmad00m authored Dec 7, 2022
2 parents 4105a20 + ca79174 commit 846e9ab
Showing 32 changed files with 1,711 additions and 788 deletions.
2 changes: 2 additions & 0 deletions .eslintrc.yml
Original file line number Diff line number Diff line change
@@ -44,6 +44,8 @@ rules:
- error
brace-style:
- error
- 1tbs
- allowSingleLine: true
comma-spacing:
- error
max-len:
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# dotenv environment variables file
.env
.env.test

lib-cov
*.seed
*.log
@@ -12,13 +16,16 @@ lib-cov
documents/developer_documents/testing_script_generator/GRNsightTestingDocument.pdf
web-client/public/js/grnsight.min.js


pids
logs
results
/.idea

database/network-database/script-results
database/network-database/source-files
database/expression-database/script-results
database/expression-database/source-files

npm-debug.log
node_modules
88 changes: 87 additions & 1 deletion database/README.md
Original file line number Diff line number Diff line change
@@ -1 +1,87 @@
Here are the files pertaining to both the network and expression databases. Look within the README.md files of both folders for information pertinent to the schema that you intend to be using.
# GRNsight Database
Here are the files pertaining to both the network and expression databases. Look within the README.md files of both folders for information pertinent to the schema that you intend to be using.
## Setting up a local postgres GRNsight Database
1. Installing PostgreSQL on your computer
- MacOS and Windows can follow these [instructions](https://dondi.lmu.build/share/db/postgresql-setup-day.pdf) on how to install postgreSQL.
- Step 1 tells you how to install postgreSQL on your local machine, initialize a database, and how to start and stop running your database instance.
- If your terminal emits a message that looks like `initdb --locale=C -E UTF-8 location-of-cluster` from Step 1B, then your installer has initialized a database for you.
- Additionally, your installer may start the server for you upon installation. To start the server yourself run `pg_ctl start -D location-of-cluster`. To stop the server run `pg_ctl stop -D location-of-cluster`.
- Linux users
- The MacOS and Windows instructions will _probably_ not work for you. You can try at your own risk to check.
- Linux users can try these [instructions](https://www.geeksforgeeks.org/install-postgresql-on-linux/) and that should work for you (...maybe...). If it doesn't try googling instructions with your specific operating system. Sorry!
2. Loading data to your database
1. Adding the Schemas to your database.
1. Go into your database using the following command:

```
psql postgresql://localhost/postgres
```
From there, create the schemas using the following commands:
```
CREATE SCHEMA spring2022_network;
```
```
CREATE SCHEMA fall2021;
```
Once they are created you can exit your database using the command `\q`.
2. Once your schema's are created, you can add the table specifications using the following commands:
```
psql postgresql://localhost/postgres -f <path to GRNsight/database/network-database>/schema.sql
```
```
psql postgresql://localhost/postgres -f <path to GRNsight/database/expression-database>/schema.sql
```
Your database is now ready to accept expression and network data!
2. Loading the GRNsight Network Data to your local database
1. GRNsight generates Network Data from SGD through YeastMine. In order to run the script that generates these Network files, you must pip3 install the dependencies used. If you get an error saying that a module doesn't exist, just run `pip3 install <Module Name>` and it should fix the error. If the error persists and is found in a specific file on your machine, you might have to manually go into that file and alter the naming conventions of the dependencies that are used. _Note: So far this issue has only occured on Ubuntu 22.04.1, so you might be lucky and not have to do it!_
```
pip3 install pandas requests intermine tzlocal
```
Once the dependencies have been installed, you can run
```
python3 <path to GRNsight/database/network-database/scripts>/generate_network.py
```
This will take a while to get all of the network data and generate all of the files. This will create a folder full of the processed files in `database/network-database/script-results`.
2. Load the processed files into your database.
```
python3 <path to GRNsight/database/network-database/scripts>/loader.py | psql postgresql://localhost/postgres
```
This should output a bunch of COPY print statements to your terminal. Once complete your database is now loaded with the network data.
3. Loading the GRNsight Expression Data to your local database
1. Create a directory (aka folder) in the database/expression-database folder called `source-files`.
```
mkdir <path to GRNsight/database/expression-database>/source-files
```
2. Download the _"Expression 2020"_ folder from Box located in `GRNsight > GRNsight Expression > Expression 2020` to your newly created `source-files` folder
3. Run the pre-processing script on the data. This will create a folder full of the processed files in `database/expression-database/script-results`.
```
python3 <path to GRNsight/database/expression-database/scripts>/preprocessing.py
```
4. Load the processed files into your database.
```
python3 <path to GRNsight/database/expression-database/scripts>/loader.py | psql postgresql://localhost/postgres
```
This should output a bunch of COPY print statements to your terminal. Once complete your database is now loaded with the expression data.
60 changes: 60 additions & 0 deletions database/expression-database/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Expression Database

All files pertaining the expression database live within this directory.

## The basics

#### Schema

All network data is stored within the fall2021 schema on our Postgres database.

The schema is located within this directory at the top level in the file `schema.sql`. It defines the tables located within the fall2021 schema.

Usage:
To load to local database
```
psql postgresql://localhost/postgres -f schema.sql
```
To load to production database
```
psql <address to database> -f schema.sql
```

### Scripts

All scripts live within the subdirectory `scripts`, located in the top-level of the network database directory.

Any source files required to run the scripts live within the subdirectory `source-files`, located in the top-level of the network database directory. As source files may be large, you must create this directory yourself and add any source files you need to use there.

All generated results of the scripts live in the subdirectory `script-results`, located in the top-level of the network database directory. Currently, all scripts that generate code create the directory if it does not currently exist. When adding a new script that generates resulting code, best practice is to create the script-results directory and any subdirectories if it does not exist, in order to prevent errors and snafus for recently cloned repositories.

Within the scripts directory, there are the following files:

- `preprocessing.py`
- `loader.py`

#### Data Preprocessor(s)
*Note: Data Preprocessing is always specific to each dataset that you obtain. `preprocessing.py` is capable of preprocessing the specific Expression data files located in `source-files/Expression 2020`. Because these files are too large to be stored on github, access the direct source files on BOX and move them into this directory. If more data sources are to be added in the database, create a new directory in source-files for it, note it in this `README.md` file and create a new preprocessing script for that data source (if required). Please document the changes in this section so that future developers may use your work to recreate the database if ever required.*

* The script (`preprocessing.py`) is used to preprocess the data in `source-files/Expression 2020`. It parses through each file to construct the processed loader files, so that they are ready to load using `loader.py`. Please read through the code, as there are instructions on what to add within the comments. Good luck!
* The resulting processed loader files are located in `script-results/processed-expression` and the resulting processed loader files are located within `script-results/processed-loader-files`

Usage:
```
python3 preprocessing.py
```
#### Database Loader
This script (`loader.py`) is to be used to load your preprocessed expression data into the database.
This program generates direct SQL statements from the source files generated by the data preprocessor in order to populate a relational database with those files’ data
Usage:
To load to local database
```
python3 loader.py | psql postgresql://localhost/postgres
```
To load to production database
```
python3 loader.py | psql <path to database>
```
71 changes: 71 additions & 0 deletions database/expression-database/schema.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
CREATE TABLE fall2021.ref (
pubmed_id VARCHAR,
authors VARCHAR,
publication_year VARCHAR,
title VARCHAR,
doi VARCHAR,
ncbi_geo_id VARCHAR,
PRIMARY KEY(ncbi_geo_id, pubmed_id)
);

CREATE TABLE fall2021.gene (
gene_id VARCHAR, -- systematic like name
display_gene_id VARCHAR, -- standard like name
species VARCHAR,
taxon_id VARCHAR,
PRIMARY KEY(gene_id, taxon_id)
);

CREATE TABLE fall2021.expression_metadata (
ncbi_geo_id VARCHAR,
pubmed_id VARCHAR,
FOREIGN KEY (ncbi_geo_id, pubmed_id) REFERENCES fall2021.ref(ncbi_geo_id, pubmed_id),
control_yeast_strain VARCHAR,
treatment_yeast_strain VARCHAR,
control VARCHAR,
treatment VARCHAR,
concentration_value FLOAT,
concentration_unit VARCHAR,
time_value FLOAT,
time_unit VARCHAR,
number_of_replicates INT,
expression_table VARCHAR,
display_expression_table VARCHAR,
PRIMARY KEY(ncbi_geo_id, pubmed_id, time_value)
);
CREATE TABLE fall2021.expression (
gene_id VARCHAR,
taxon_id VARCHAR,
FOREIGN KEY (gene_id, taxon_id) REFERENCES fall2021.gene(gene_id, taxon_id),
-- ncbi_geo_id VARCHAR,
-- pubmed_id VARCHAR,
sort_index INT,
sample_id VARCHAR,
expression FLOAT,
time_point FLOAT,
dataset VARCHAR,
PRIMARY KEY(gene_id, sample_id)
-- FOREIGN KEY (ncbi_geo_id, pubmed_id, time_point) REFERENCES fall2021.expression_metadata(ncbi_geo_id, pubmed_id, time_value)
);
CREATE TABLE fall2021.degradation_rate (
gene_id VARCHAR,
taxon_id VARCHAR,
FOREIGN KEY (gene_id, taxon_id) REFERENCES fall2021.gene(gene_id, taxon_id),
ncbi_geo_id VARCHAR,
pubmed_id VARCHAR,
FOREIGN KEY (ncbi_geo_id, pubmed_id) REFERENCES fall2021.ref(ncbi_geo_id, pubmed_id),
PRIMARY KEY(gene_id, ncbi_geo_id, pubmed_id),
degradation_rate FLOAT
);

CREATE TABLE fall2021.production_rate (
gene_id VARCHAR,
taxon_id VARCHAR,
FOREIGN KEY (gene_id, taxon_id) REFERENCES fall2021.gene(gene_id, taxon_id),
ncbi_geo_id VARCHAR,
pubmed_id VARCHAR,
FOREIGN KEY (ncbi_geo_id, pubmed_id) REFERENCES fall2021.ref(ncbi_geo_id, pubmed_id),
PRIMARY KEY(gene_id, ncbi_geo_id, pubmed_id),
production_rate FLOAT
-- FOREIGN KEY (gene_id, ncbi_geo_id, pubmed_id) REFERENCES fall2021.degradation_rate(gene_id, ncbi_geo_id, pubmed_id) -- not sure if we want to link the generated production rate to it's original degradation rate
);
186 changes: 186 additions & 0 deletions database/expression-database/scripts/loader.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,186 @@
import csv
import re
# Usage
# python3 loader.py | psql postgresql://localhost/postgres
"""
This program generates direct SQL statements from the source files in order
to populate a relational database with those files’ data.
By taking the approach of emitting SQL statements directly, we bypass the need to import
some kind of database library for the loading process, instead passing the statements
directly into a database command line utility such as `psql`.
"""

"""
Stolen from https://www.kite.com/python/answers/how-to-check-if-a-string-is-a-valid-float-in-python
"""
def check_float(potential_float):
try:
float(potential_float)
return True
except ValueError:
return False
"""
Inspired by https://www.kite.com/python/answers/how-to-check-if-a-string-is-a-valid-float-in-python
"""
def check_int(potential_int):
try:
int(potential_int)
return True
except ValueError:
return False
"""
Created out of necessity
"""
def convert_float(potential_float):
return float("".join(potential_float.split()).replace(" ", "")) if "".join(potential_float.split()).replace(" ", "") else -0.000000000001
"""
Created out of necessity
"""
def convert_int(potential_int):
return int("".join(potential_int.split()).replace(" ", "")) if check_int("".join(potential_int.split()).replace(" ", "")) else -1111111


"""
This program Loads Refs into the database
"""
def LOAD_REFS():
print('COPY fall2021.ref (pubmed_id, authors, publication_year, title, doi, ncbi_geo_id) FROM stdin;')
REFS_SOURCE = '../script-results/processed-expression/refs.csv'
with open(REFS_SOURCE, 'r+') as f:
reader = csv.reader(f)
row_num = 0
for row in reader:
if row_num != 0:
r= ','.join(row).split('\t')
pubmed_id = r[0]
authors = r[1]
publication_year = r[2]
title = r[3]
doi = r[4]
ncbi_geo_id = r[5]
print(f'{pubmed_id}\t{authors}\t{publication_year}\t{title}\t{doi}\t{ncbi_geo_id}')
row_num += 1
print('\\.')

"""
This program Loads ID Mapping into the database
"""
def LOAD_GENES():
print('COPY fall2021.gene (gene_id, display_gene_id, species, taxon_id) FROM stdin;')
GENE_SOURCE = '../script-results/processed-expression/genes.csv'
with open(GENE_SOURCE, 'r+') as f:
reader = csv.reader(f)
row_num = 0
for row in reader:
if row_num != 0:
r= ','.join(row).split('\t')
gene_id = r[0]
display_gene_id= r[1]
species = r[2]
taxon_id = r[3]
print(f'{gene_id}\t{display_gene_id}\t{species}\t{taxon_id}')
row_num += 1
print('\\.')

"""
This program Loads Expression Metadata into the database
"""
def LOAD_EXPRESSION_METADATA():
print('COPY fall2021.expression_metadata (ncbi_geo_id, pubmed_id, control_yeast_strain, treatment_yeast_strain, control, treatment, concentration_value, concentration_unit, time_value, time_unit, number_of_replicates, expression_table) FROM stdin;')
EXPRESSION_METADATA_SOURCE = '../script-results/processed-expression/expression-metadata.csv'
with open(EXPRESSION_METADATA_SOURCE, 'r+') as f:
reader = csv.reader(f)
row_num = 0
for row in reader:
if row_num != 0:
r= ','.join(row).split('\t')
ncbi_geo_id = r[0]
pubmed_id =r[1]
control_yeast_strain = r[2]
treatment_yeast_strain = r[3]
control = r[4]
treatment = r[5]
concentration_value = float(r[6])
concentration_unit = r[7]
time_value = float(r[8])
time_unit = r[9]
number_of_replicates = int(r[10])
expression_table = r[11]

print(f'{ncbi_geo_id}\t{pubmed_id}\t{control_yeast_strain}\t{treatment_yeast_strain}\t{control}\t{treatment}\t{concentration_value}\t{concentration_unit}\t{time_value}\t{time_unit}\t{number_of_replicates}\t{expression_table}')
row_num += 1
print('\\.')

"""
This program Loads Expression Data into the database
"""
def LOAD_EXPRESSION_DATA():
print('COPY fall2021.expression (gene_id, taxon_id, sort_index, sample_id, expression, time_point, dataset) FROM stdin;')
EXPRESSION_DATA_SOURCE = '../script-results/processed-expression/expression-data.csv'
with open(EXPRESSION_DATA_SOURCE, 'r+') as f:
reader = csv.reader(f)
row_num = 0
for row in reader:
if row_num != 0:
r= ','.join(row).split('\t')
gene_id = r[0]
taxon_id = r[1]
sort_index = int(r[2])
sample_id = r[3]
expression = float(r[4]) if r[4] != "" else "NaN"

time_point = float(r[5])
data_set = r[6]
print(f'{gene_id}\t{taxon_id}\t{sort_index}\t{sample_id}\t{expression}\t{time_point}\t{data_set}')
row_num += 1
print('\\.')

"""
This program Loads Production Rates into the database
"""
def LOAD_PRODUCTION_RATES():
print('COPY fall2021.production_rate (gene_id, taxon_id, ncbi_geo_id, pubmed_id, production_rate) FROM stdin;')
PRODUCTION_RATES_SOURCE = '../script-results/processed-expression/production-rates.csv'
with open(PRODUCTION_RATES_SOURCE, 'r+') as f:
reader = csv.reader(f)
row_num = 0
for row in reader:
if row_num != 0:
r= ','.join(row).split('\t')
gene_id = r[0]
taxon_id = r[1]
ncbi_geo_id = r[2]
pubmed_id = r[3]
production_rate = float(r[4]) if r[4] != "" else "NaN"
print(f'{gene_id}\t{taxon_id}\t{ncbi_geo_id}\t{pubmed_id}\t{production_rate}')
row_num += 1
print('\\.')

"""
This program Loads Degradation Rates into the database
"""
def LOAD_DEGRADATION_RATES():
print('COPY fall2021.degradation_rate (gene_id, taxon_id, ncbi_geo_id, pubmed_id, degradation_rate) FROM stdin;')
DEGRADATION_RATES_SOURCE = '../script-results/processed-expression/degradation-rates.csv'
with open(DEGRADATION_RATES_SOURCE, 'r+') as f:
reader = csv.reader(f)
row_num = 0
for row in reader:
if row_num != 0:
r= ','.join(row).split('\t')
gene_id = r[0]
taxon_id = r[1]
ncbi_geo_id = r[2]
pubmed_id = r[3]
degradation_rate = float(r[4]) if r[4] != "" else "NaN"
print(f'{gene_id}\t{taxon_id}\t{ncbi_geo_id}\t{pubmed_id}\t{degradation_rate}')
row_num += 1
print('\\.')

LOAD_REFS()
LOAD_GENES()
LOAD_EXPRESSION_METADATA()
LOAD_EXPRESSION_DATA()
LOAD_PRODUCTION_RATES()
LOAD_DEGRADATION_RATES()
197 changes: 197 additions & 0 deletions database/expression-database/scripts/preprocessing.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
import csv
import re
import sys
import os

# Need to manually add Dahlquist data to Expression metadata and refs


species = "Saccharomyces cerevisiae"
taxon_id = "559292"

# Gene Id Generation and Expression Data Generation

# Create folder paths
if not os.path.exists('../script-results'):
os.makedirs('../script-results')

if not os.path.exists('../script-results/processed-expression/'):
os.makedirs('../script-results/processed-expression')

# For simplicity, we assume that the program runs in the expression-database-folder.
EXPRESSION_DATA_SOURCE = '../source-files/Expression 2020/ExpressionData.csv'
EXPRESSION_DATA_DESTINATION = '../script-results/processed-expression/expression-data.csv'
EXPRESSION_SHEET_DESTINATION = '../script-results/processed-expression/expression-sheet.csv'
GENES_DESTINATION = '../script-results/processed-expression/genes.csv'

genes = {}
expression_data = []
expression_sheets = {}
print(f'Processing file {EXPRESSION_DATA_SOURCE}')
with open(EXPRESSION_DATA_SOURCE, 'r+', encoding="UTF-8") as f:
i = 0
replicate_count = 0
prev_dataset = ""
reader = csv.reader(f)
for row in reader:
if i != 0:
col_num = 0
display_gene_id = row[2].replace('\t','')
gene_id = row[1].replace('\t','')
sort_index = row[0]
sample_id = row[4]
expression = row[5]
time_points = row[6]
dataset = row[7]
# update the objects
if gene_id not in genes:
genes.update({gene_id : [display_gene_id, species, taxon_id]})
expression_data.append([gene_id, taxon_id, sort_index, sample_id, expression, time_points, dataset])
i+=1
print(f'Creating {EXPRESSION_DATA_DESTINATION}\n')
expression_data_file = open(EXPRESSION_DATA_DESTINATION, 'w')
expression_data_file.write(f'Gene ID\tTaxon ID\tSort Index\tSample ID\tExpression\tTime Points\tDataset\n')
for d in expression_data:
result = '{}\t{}\t{}\t{}\t{}\t{}\t{}'.format(d[0], d[1], d[2], d[3], d[4], d[5], d[6])
expression_data_file.write(f'{result}\n')
expression_data_file.close()

# Expression Metadata
EXPRESSION_METADATA_SOURCE = '../source-files/Expression 2020/ExpressionMetadata.csv'
EXPRESSION_METADATA_DESTINATION = '../script-results/processed-expression/expression-metadata.csv'
# Add Dalquist Data Here
expression_metadata = [
# [1, 'GSE83656', '', 'control_yeast_strain', 'treatment_yeast_strain', 'control', 'treatment', 'concentration_value', 'concentration_unit', 'time_value', 'time_unit', 'number_of_replicates,', 'expression_table'],
# [3, 'GSE83656', '', 'control_yeast_strain', 'treatment_yeast_strain', 'control', 'treatment', 'concentration_value', 'concentration_unit', 'time_value', 'time_unit', 'number_of_replicates,', 'expression_table'],
# [2, 'GSE83656', '', 'control_yeast_strain', 'treatment_yeast_strain', 'control', 'treatment', 'concentration_value', 'concentration_unit', 'time_value', 'time_unit', 'number_of_replicates,', 'expression_table'],
# [4, 'GSE83656', '', 'control_yeast_strain', 'treatment_yeast_strain', 'control', 'treatment', 'concentration_value', 'concentration_unit', 'time_value', 'time_unit', 'number_of_replicates,', 'expression_table'],
]

pubmed_to_geo_conversion = {
'12269742': 'GSE9336',
'17327492': 'GSE6129',
'23039231': 'GSE24712'
}

print(f'Processing file {EXPRESSION_METADATA_SOURCE}')
with open(EXPRESSION_METADATA_SOURCE, 'r+', encoding="UTF-8") as f:
i = 0
reader = csv.reader(f)
for row in reader:
if i != 0:
# replicate_index = row[0][-1]
pubmed_id = row[1]
geo_id = pubmed_to_geo_conversion[pubmed_id]
control_yeast_strain = row[2]
treatment_yeast_strain = row[3]
control = row[4]
treatment = row[5]
concentration_value = row[6]
concentration_unit = row[7]
time_value = row[8]
time_unit = row[9]
number_of_replicates = row[10]
expression_table = row[11]

expression_metadata.append([geo_id, pubmed_id, control_yeast_strain, treatment_yeast_strain, control, treatment, concentration_value, concentration_unit, time_value, time_unit, number_of_replicates, expression_table])
# next row
i+= 1

print(f'Creating {EXPRESSION_METADATA_DESTINATION}\n')
expression_metadata_file = open(EXPRESSION_METADATA_DESTINATION, 'w')
expression_metadata_file.write(f'NCBI GEO ID\tPubmed ID\tControl Yeast Strain\tTreatment Yeast Strain\tControl\tTreatment\tConcentration Value\tConcentration Unit\tTime Value\tTime Units\tNumber of Replicates\tExpression Table\n')
for m in expression_metadata:
expression_metadata_file.write(f'{m[0]}\t{m[1]}\t{m[2]}\t{m[3]}\t{m[4]}\t{m[5]}\t{m[6]}\t{m[7]}\t{m[8]}\t{m[9]}\t{m[10]}\t{m[11]}\n')
expression_metadata_file.close()


# Refs csv file generation (She is smol so we write her ourselves)
refs = [
# [pubmed_id, authors, publication_year, title, doi, ncbi_geo_id]
['12269742', 'Kitagawa E., Takahashi J., Momose Y., Iwahashi H.', '2002', 'Effects of the Pesticide Thiuram: Genome-wide Screening of Indicator Genes by Yeast DNA Microarray', '10.1021/es015705v', 'GSE9336'],
['17327492', 'Thorsen, M., Lagniel, G., Kristiansson, E., Junot, C., Nerman, O., Labarre, J., & Tamás, M. J.', '2007', 'Quantitative transcriptome, proteome, and sulfur metabolite profiling of the Saccharomyces cerevisiae response to arsenite.', '10.1152/physiolgenomics.00236.2006', 'GSE6129'],
['23039231', 'Barreto, L., Canadell, D., Valverde‐Saubí, D., Casamayor, A., & Ariño, J.', '2012', 'The short‐term response of yeast to potassium starvation', '10.1111/j.1462-2920.2012.02887.x', 'GSE24712'],
['', 'Dahlquist KD, Abdulla H, Arnell AJ, Arsan C, Baker JM, Carson RM, Citti WT, De Las Casas SE, Ellis LG, Entzminger KC, Entzminger SD, Fitzpatrick BG, Flores SP, Harmon NS, Hennessy KP, Herman AF, Hong MV, King HL, Kubeck LN, La-Anyane OM, Land DL, Leon Guerrero MJ, Liu EM, Luu MD, McGee KP, Mejia MR, Melone SN, Pepe NT, Rodriguez KR, Rohacz NA, Rovetti RJ, Sakhon OS, Sampana JT, Sherbina K, Terada LH, Vega AJ, Wavrin AJ, Wyllie KW, Zapata BB',
'2018', 'Global transcriptional response of wild type and transcription factor deletion strains of Saccharomyces cerevisiae to the environmental stress of cold shock and subsequent recovery',
'', 'GSE83656'],
['25161313', 'Neymotin, B., Athanasiadou R., and Gresham D.', '2014', ' Determination of in vivo RNA kinetics using RATE-seq. RNA, 20, 1645-1652.', '10.1261/rna.045104.114', '']
]

REFS_DESTINATION = '../script-results/processed-expression/refs.csv'
print(f'Creating {REFS_DESTINATION}\n')
refs_file = open(REFS_DESTINATION, 'w')
refs_file.write(f'Pubmed ID\tAuthors\tPublication Year\tTitle\tDOI\tNCBI GEO ID\n')
for r in refs:
result = '{}\t{}\t{}\t{}\t{}\t{}'.format(r[0], r[1], r[2], r[3], r[4], r[5])
refs_file.write(f'{result}\n')
refs_file.close()

# Degradation Rates
DEGRADATION_RATES_SOURCE = '../source-files/Expression 2020/DegradationRates.csv'
DEGRADATION_RATES_DESTINATION = '../script-results/processed-expression/degradation-rates.csv'

degradation_rates = []

print(f'Processing file {DEGRADATION_RATES_SOURCE}')
with open(DEGRADATION_RATES_SOURCE, 'r+', encoding="UTF-8") as f:
i = 0
reader = csv.reader(f)
for row in reader:
if i != 0:
gene_id = row[0]
display_gene_id = row[1]
degradation_rate = row[2]
pubmed_id = "25161313"
geo_id = ""
degradation_rates.append([gene_id, taxon_id, geo_id, pubmed_id, degradation_rate])
if gene_id not in genes:
genes.update({gene_id : [display_gene_id, species, taxon_id]})
i+= 1

print(f'Creating {DEGRADATION_RATES_DESTINATION}\n')
degradation_rates_file = open(DEGRADATION_RATES_DESTINATION, 'w')
degradation_rates_file.write(f'Gene ID\tTaxon ID\tNCBI GEO ID\tPubmed ID\tDegradation Rate\n')
for r in degradation_rates:
result = '{}\t{}\t{}\t{}\t{}'.format(r[0], r[1], r[2], r[3], r[4])
degradation_rates_file.write(f'{result}\n')
degradation_rates_file.close()

# Production Rates
PRODUCTION_RATES_SOURCE = '../source-files/Expression 2020/ProductionRates.csv'
PRODUCTION_RATES_DESTINATION = '../script-results/processed-expression/production-rates.csv'

production_rates = []

print(f'Processing file {PRODUCTION_RATES_SOURCE}')
with open(PRODUCTION_RATES_SOURCE, 'r+', encoding="UTF-8") as f:
i = 0
reader = csv.reader(f)
for row in reader:
if i != 0:
gene_id = row[0]
display_gene_id = row[1]
production_rate = row[2]
pubmed_id = "25161313"
geo_id = ""
production_rates.append([gene_id, taxon_id, geo_id, pubmed_id, production_rate])
if gene_id not in genes:
genes.update({gene_id : [display_gene_id, species, taxon_id]})
# next row
i+= 1

print(f'Creating {PRODUCTION_RATES_DESTINATION}\n')
production_rates_file = open(PRODUCTION_RATES_DESTINATION, 'w')
production_rates_file.write(f'Gene ID\tTaxon ID\tNCBI GEO ID\tPubmed ID\tProduction Rate\n')
for r in production_rates:
result = '{}\t{}\t{}\t{}\t{}'.format(r[0], r[1], r[2], r[3], r[4])
production_rates_file.write(f'{result}\n')
production_rates_file.close()


print(f'Creating {GENES_DESTINATION}\n')
genes_file = open(GENES_DESTINATION, 'w')
genes_file.write(f'Gene ID\tDisplay Gene ID\tSpecies\tTaxon ID\n')
for g in genes:
result = '{}\t{}\t{}\t{}'.format(g, genes[g][0], genes[g][1], genes[g][2],)
genes_file.write(f'{result}\n')
genes_file.close()
50 changes: 47 additions & 3 deletions database/network-database/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Network Database (Schema)
# Network Database

All files pertaining the network database live within this directory.

@@ -10,6 +10,16 @@ All network data is stored within the spring2022_network schema on our Postgres

The schema is located within this directory at the top level in the file `schema.sql`. It defines the tables located within the spring2022_network schema.

Usage:
To load to local database
```
psql postgresql://localhost/postgres -f schema.sql
```
To load to production database
```
psql <address to database> -f schema.sql
```

### Scripts

All scripts live within the subdirectory `scripts`, located in the top-level of the network database directory.
@@ -21,24 +31,58 @@ All generated results of the scripts live in the subdirectory `script-results`,
Within the scripts directory, there are the following files:

- `generate_network.py`
- `generate_sgd_network_from_yeastract_network.py`
- `loader.py`
- `filter_genes.py`
- `generate_sgd_network_from_yeastract_network.py`

#### Network Generator (and data preprocessor)

This script (`generate_network.py`) is a two-for-one. It first uses the yeastmine service from the SGD database to query for all regulator genes relating to Saccharomyces cerevisiae. From there it gets all all of the targets for each regulator gene. We then construct two networks from these connections (a regulator by regulator matrix as well as a regulator by target matrix). We also construct the processed loader files, so that they are ready to load using `loader.py`.

The resulting network matrices are located in `script-results/networks` and the resulting processed loader files are located within `script-results/processed-loader-files`

Make sure to have all dependencies installed beforehand or you will recieve errors. (pip3 install intermine, tzlocal, etc. [see file for all imports])
Make sure to have all dependencies installed beforehand or you will recieve errors. (pip3 install intermine, tzlocal, etc. [see file for all imports]

Usage:
```
python3 generate_network.py
```
#### Database Loader

This script (`loader.py`) is to be used to load your preprocessed genes into the database.

This program generates direct SQL statements from the source files generated by the network generator in order to populate a relational database with those files’ data

Usage:
To load to local database
```
python3 loader.py | psql postgresql://localhost/postgres
```
To load to production database
```
python3 loader.py | psql <address to database>
```

#### Filter Genes (beta functionality, not tested)

This script (`filter_genes.py`) is to be used when updating a pre-existing database. It requires you to generate a new network from yeastmine using the script.`generate_network.py`. Once you generate the network, the script will access the database get all of the genes stored within. From there it will generate a csv file of all genes in the new network that are missing from your database, and all genes that have updated their display name (standard like name). You should change the database host to your localhost if you are running your own instance of postgresql and not the production database. Once finished, you can load the updated genes list using `loader.py`. *Note:* You will have to change the `GENE_SOURCE` to be the output file of the missing genes.

**Never save the password to your database in filter_genes.py! If you want, you can set up an environment variable where you store sensitive information, but for convience you will have to enter the password yourself.**

Usage:
```
python3 filter_genes.py
```


#### Generate an SGD network from a Yeastract network

This script takes a network (assumed to have data from Yeastract, but it can be any given network) and gives you a network with data queried from Yeastmine (SGD). It takes the regulators and targets from a given network file, then queries Yeastmine in order to get the regulatory connections between the genes. From there, it creates a new network using the data obtained from Yeastmine.

To use, create a folder called `source-files` within the `network-database` folder. Add your network(s) to the newly created directory. Then go into the script and change the `YEASTRACT_NETWORK` to be the path of the network you would like to convert. Run the script and your SGD network (using the same genes) will be output in the `/script-results/yeastract-to-sgd-networks` directory. Change the name of the output files by editing the `SGD_MATRIX_EXCEL` and `SGD_MATRIX` variables

Usage:
```
python3 generate_sgd_network_from_yeastract_network.py
```

5 changes: 3 additions & 2 deletions database/network-database/schema.sql
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
CREATE TABLE spring2022_network.source (
time_stamp TIMESTAMP,
time_stamp TIMESTAMP WITH TIME ZONE,
source VARCHAR,
source_display_name VARCHAR,
PRIMARY KEY(time_stamp, source)
);

@@ -16,7 +17,7 @@ CREATE TABLE spring2022_network.network (
regulator_gene_id VARCHAR,
target_gene_id VARCHAR,
taxon_id VARCHAR,
time_stamp TIMESTAMP,
time_stamp TIMESTAMP WITH TIME ZONE,
source VARCHAR,
FOREIGN KEY (regulator_gene_id, taxon_id) REFERENCES spring2022_network.gene(gene_id, taxon_id),
FOREIGN KEY (target_gene_id, taxon_id) REFERENCES spring2022_network.gene(gene_id, taxon_id),
22 changes: 4 additions & 18 deletions database/network-database/scripts/generate_network.py
Original file line number Diff line number Diff line change
@@ -8,8 +8,6 @@
import sys
import os
import datetime
import pytz
import tzlocal

# Get Network Data from Yeastmine

@@ -97,7 +95,7 @@ def create_regulator_to_target_row(target, all_regulators):

# Files to be generated

# Create Networks
# Generate Networks

REGULATORS_TO_TARGETS_MATRIX = '../script-results/networks/regulators_to_targets.csv'
REGULATORS_TO_REGULATORS_MATRIX = '../script-results/networks/regulators_to_regulators.csv'
@@ -142,25 +140,13 @@ def create_regulator_to_target_row(target, all_regulators):
# Source Table

SOURCE_DESTINATION = '../script-results/processed-loader-files/source.csv'
dt = datetime.datetime.now()
timestamp = datetime.datetime.now(datetime.timezone.utc)

year = dt.year
month = f'{dt.month}'
if len(month) == 1:
month = "0" + month
day = f'{dt.day}'
if len(day) == 1:
day = "0" + day
hour = dt.hour
minute = dt.minute
second = dt.second


timestamp = f'{year}-{month}-{day} {hour}:{minute}:{second}'
source = "YeastMine - Saccharomyces Genome Database"
display_name = "Yeastmine - SGD"

source_file = open(SOURCE_DESTINATION, 'w')
headers = f'Timestamp\tSource\n{timestamp}\t{source}'
headers = f'Timestamp\tSource\tDisplay Name\n{timestamp}\t{source}\t{display_name}'
source_file.write(f'{headers}\n')
source_file.close()

Original file line number Diff line number Diff line change
@@ -94,7 +94,7 @@ def create_regulator_to_target_row(target, all_regulators):

# Files to be generated

# Create Networks
# Generate Networks

SGD_MATRIX = '../script-results/yeastract-to-sgd-networks/SGD_Regulation_matrix_profile2.csv'
SGD_MATRIX_EXCEL = '../script-results/yeastract-to-sgd-networks/SGD_Regulation_matrix_profile2.xlsx'
9 changes: 5 additions & 4 deletions database/network-database/scripts/loader.py
Original file line number Diff line number Diff line change
@@ -15,7 +15,7 @@
This function Loads Network Data Sources into the database
"""
def LOAD_SOURCES():
print('COPY spring2022_network.source (time_stamp, source) FROM stdin;')
print('COPY spring2022_network.source (time_stamp, source, display_name) FROM stdin;')
NETWORK_DATA_SOURCE = '../script-results/processed-loader-files/source.csv'
with open(NETWORK_DATA_SOURCE, 'r+') as f:
reader = csv.reader(f)
@@ -25,7 +25,8 @@ def LOAD_SOURCES():
r= ','.join(row).split('\t')
time_stamp = r[0]
source = r[1]
print(f'{time_stamp}\t{source}')
display_name = r[2]
print(f'{time_stamp}\t{source}\t{display_name}')
row_num += 1
print('\\.')

@@ -57,8 +58,8 @@ def LOAD_GENES():
"""
def LOAD_NETWORK():
print('COPY spring2022_network.network (regulator_gene_id, target_gene_id, taxon_id, time_stamp, source) FROM stdin;')
GENE_SOURCE = '../script-results/processed-loader-files/network.csv'
with open(GENE_SOURCE, 'r+') as f:
NETWORK_SOURCE = '../script-results/processed-loader-files/network.csv'
with open(NETWORK_SOURCE, 'r+') as f:
reader = csv.reader(f)
row_num = 0
for row in reader:
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "grnsight",
"version": "6.0.0",
"version": "6.0.4",
"description": "Web app and service for visualizing models of gene regulatory networks",
"directories": {
"test": "test"
4 changes: 2 additions & 2 deletions server/controllers/additional-sheet-parser.js
Original file line number Diff line number Diff line change
@@ -306,7 +306,7 @@ module.exports = function (workbookFile) {
errors: [],
warnings: []
}, // optimization_parameters only
test: {}, // 2-column data
twoColumnSheets: {}, // 2-column data
meta2: {} // optimation_diagnostics only //temporary until where it goes is decided
};
workbookFile.forEach(function (sheet) {
@@ -315,7 +315,7 @@ module.exports = function (workbookFile) {
// above line creates an object from the optimization paramerters sheet
// these are part of the "meta" property
} else if (TWO_COL_SHEET_NAMES.includes(sheet.name)) {
output["test"][sheet.name] = parseTwoColumnSheet(sheet);
output.twoColumnSheets[sheet.name] = parseTwoColumnSheet(sheet);
} else if (sheet.name === "optimization_diagnostics") {
output.meta2 = parseOptimizationDiagnosticsSheet(sheet);
}
5 changes: 3 additions & 2 deletions server/controllers/custom-workbook-controller.js
Original file line number Diff line number Diff line change
@@ -15,7 +15,7 @@ const createCustomWorkbook = (genesString, linksString) => {
let genes = genesString.split(",").map(gene => {
return {name: gene};
});
let links = linksString.split(",").map( link => {
let links = linksString === "" ? [] : linksString.split(",").map( link => {
link = link.split("->");
return {
source: parseInt(link[0]),
@@ -47,7 +47,8 @@ const createCustomWorkbook = (genesString, linksString) => {
taxon_id: 559292
}
},
test: {
meta2: {},
twoColumnSheets: {
},
expression: {
}
8 changes: 4 additions & 4 deletions server/controllers/demo-workbooks.js
Original file line number Diff line number Diff line change
@@ -532,7 +532,7 @@ var demoWorkbook1 = function (path, res, app) {
taxon_id: 559292
}
},
test: {
twoColumnSheets: {
production_rates: {
data: {
ACE2: 0.2236,
@@ -2695,7 +2695,7 @@ var demoWorkbook2 = function (path, res, app) {
taxon_id: 559292
}
},
test: {
twoColumnSheets: {
production_rates: {
data: {
ACE2: 0.2236,
@@ -4622,7 +4622,7 @@ var demoWorkbook3 = function (path, res, app) {
taxon_id: 559292
}
},
test: {},
twoColumnSheets: {},
expression: {
wt_log2_expression: {
errors: [],
@@ -5683,7 +5683,7 @@ var demoWorkbook4 = function (path, res, app) {
taxon_id: 559292
}
},
test: {},
twoColumnSheets: {},
expression: {
wt_log2_expression: {
errors: [],
81 changes: 29 additions & 52 deletions server/controllers/exporters/xlsx.js
Original file line number Diff line number Diff line change
@@ -87,94 +87,71 @@ const isExpressionSheet = (sheetName) => {
const buildExpressionSheets = function (expressions) {
const builtExpressionSheets = [];
Object.keys(expressions).forEach((expression) => {
let expressionName = expression;
if (!isExpressionSheet(expression)) {
expressionName = expression + "_expression";
if (expressions[expression] !== null) {
let expressionName = expression;
if (!isExpressionSheet(expression)) {
expressionName = expression + "_expression";
}
const builtSheet = { name: expressionName, data: [] };
Object.keys(expressions[expression]["data"]).forEach((key) => {
const expressionData = expressions[expression]["data"][key];
builtSheet["data"].push([key, ...expressionData]);
});
builtExpressionSheets.push(builtSheet);
}
const builtSheet = { name: expressionName, data: [] };
Object.keys(expressions[expression]["data"]).forEach((key) => {
const expressionData = expressions[expression]["data"][key];
builtSheet["data"].push([key, ...expressionData]);
});
builtExpressionSheets.push(builtSheet);
});
return builtExpressionSheets;
};

const buildXlsxSheet = function (workbook) {
const resultSheet = [];
const exportNetworkType = workbook.exportNetworkType;

Object.keys(workbook).forEach((key) => {
switch (key) {
case "network":
if (Object.keys(workbook.network).length > 0) {
resultSheet.push(
{
"name": "network",
"data": buildNetworkSheet(workbook.network.genes, workbook.network.links)
}
);
}
break;
case "networkOptimizedWeights":
if (exportNetworkType === "weighted") {
if (Object.keys(workbook.networkOptimizedWeights).length > 0) {
Object.keys(workbook.exportSheets).forEach((type) => {
switch (type) {
case "networks":
for (let network in workbook.exportSheets.networks) {
if (Object.keys(workbook.exportSheets.networks[network]).length > 0) {
resultSheet.push(
{
"name": "network_optimized_weights",
"data": buildNetworkSheet(workbook.networkOptimizedWeights.genes,
workbook.networkOptimizedWeights.links)
"name": network,
"data": buildNetworkSheet(workbook.exportSheets.networks[network].genes,
workbook.exportSheets.networks[network].links)
}
);
}
}
break;
case "networkWeights":
if (Object.keys(workbook.networkWeights).length > 0) {
case "optimization_parameters":
if (workbook.exportSheets[type] !== null && Object.keys(workbook.exportSheets[type]).length > 0) {
resultSheet.push(
{
"name": "network_weights",
"data": buildNetworkSheet(workbook.networkWeights.genes, workbook.networkWeights.links)
"name": type,
"data": buildMetaSheet(workbook.exportSheets[type])
}
);
}
break;
case "meta":
if (Object.keys(workbook.meta).length > 0) {
resultSheet.push(
{
"name": "optimization_parameters",
"data": buildMetaSheet(workbook.meta)
}
);
}
break;
case "meta2":
case "optimization_diagnostics":
// Optimization Diagnostics sheet not properly implemented yet.
if (Object.keys(workbook.meta2).length > 0) {
if (Object.keys(workbook.exportSheets[type]).length > 0) {
resultSheet.push(
{
"name": "optimization_diagnostics",
"data": buildMeta2Sheet(workbook.meta2)
"data": buildMeta2Sheet(workbook.exportSheets[type])
}
);
}
break;
case "test":
resultSheet.push(...buildTestSheets(workbook[key]));
case "two_column_sheets":
resultSheet.push(...buildTestSheets(workbook.exportSheets[type]));
break;
case "expression":
// resultSheet.push(...buildExpressionSheets(workbook[key]));
break;
case "exportExpression":
resultSheet.push(...buildExpressionSheets(workbook[key]));
resultSheet.push(...buildExpressionSheets(workbook.exportSheets.expression));
break;
default:
break;
}
});

return resultSheet;
};

16 changes: 8 additions & 8 deletions server/controllers/spreadsheet-controller.js
Original file line number Diff line number Diff line change
@@ -158,19 +158,19 @@ var crossSheetInteractions = function (workbookFile) {
}
}

if (additionalData && additionalData.test) {
// Add errors and warnings from test sheets
for (let sheet in additionalData.test) {
additionalData.test[sheet].errors.forEach(data => workbook.errors.push(data));
if (additionalData && additionalData.twoColumnSheets) {
// Add errors and warnings from two column sheets
for (let sheet in additionalData.twoColumnSheets) {
additionalData.twoColumnSheets[sheet].errors.forEach(data => workbook.errors.push(data));
}

for (let sheet in additionalData.test) {
additionalData.test[sheet].warnings.forEach(data => workbook.warnings.push(data));
for (let sheet in additionalData.twoColumnSheets) {
additionalData.twoColumnSheets[sheet].warnings.forEach(data => workbook.warnings.push(data));
}
}

if (additionalData && additionalData.meta2) {
// Add errors and warnings from test sheets
// Add errors and warnings from two column sheets
if (additionalData.meta2.errors !== undefined) {
additionalData.meta2.errors.forEach(data => workbook.errors.push(data));
}
@@ -266,7 +266,7 @@ var crossSheetInteractions = function (workbookFile) {
workbook.networkOptimizedWeights = networks.networkOptimizedWeights;
workbook.networkWeights = networks.networkWeights;
workbook.meta = additionalData.meta;
workbook.test = additionalData.test;
workbook.twoColumnSheets = additionalData.twoColumnSheets;
workbook.meta2 = additionalData.meta2;
workbook.expression = expressionData.expression;
return workbook;
141 changes: 75 additions & 66 deletions server/dals/expression-dal.js
Original file line number Diff line number Diff line change
@@ -17,76 +17,48 @@ var sequelize = new Sequelize(
}
);

const expressionTimepointsSources = [
{
key: "Barreto_2012_wt",
value: [10, 10, 20, 20, 20, 20, 40, 40, 40, 40, 60, 60, 60, 60, 120, 120, 120, 120]
},

{
key: "Dahlquist_2018_dcin5",
value: [15, 15, 15, 15, 30, 30, 30, 30, 60, 60, 60, 60, 90, 90, 90, 90, 120, 120, 120, 120]
},

{
key: "Dahlquist_2018_dgln3",
value: [15, 15, 15, 15, 30, 30, 30, 30, 60, 60, 60, 60, 90, 90, 90, 90, 120, 120, 120, 120]
},

{
key: "Dahlquist_2018_dhap4",
value: [15, 15, 15, 15, 30, 30, 30, 30, 60, 60, 60, 60, 90, 90, 90, 120, 120, 120]
},

{
key: "Dahlquist_2018_dzap1",
value: [15, 15, 15, 15, 30, 30, 30, 30, 60, 60, 60, 60, 90, 90, 90, 90, 120, 120, 120, 120]
},

{
key: "Dahlquist_2018_wt",
value: [15, 15, 15, 15, 30, 30, 30, 30, 30, 60, 60, 60, 60, 90, 90, 90, 90, 90, 120, 120, 120, 120, 120]
},

{
key: "Kitagawa_2002_wt",
value: [15, 15, 15, 30, 30, 30, 120, 120, 120]
},

{
key: "Thorsen_2007_wt",
value: [15, 15, 15, 30, 30, 30, 60, 60, 60, 60, 60, 60, 1080, 1080, 1080]
}
];

const expressionTimepointsByDataset = {};
expressionTimepointsSources.forEach(source => expressionTimepointsByDataset[source.key] = source.value);

let buildExpressionTimepointsQuery = function (selection) {
let timepoints = "";
selection.forEach(x => timepoints += ("fall2021.expression.time_point=" + x + " OR "));
return timepoints.substring(0, timepoints.length - 4);
};

let buildExpressionGenesQuery = function (geneString) {
const buildExpressionGenesQuery = function (geneString) {
let genes = "";
let geneList = geneString.split(",");
geneList.forEach(x => genes += ( `(fall2021.gene.display_gene_id =\'${x}\') OR `));
return genes.substring(0, genes.length - 4);
};

let buildExpressionQuery = function (dataset, timepoints, genes) {
return timepoints ?
`SELECT * FROM fall2021.expression, fall2021.gene WHERE fall2021.expression.dataset='${dataset}' AND
(${buildExpressionTimepointsQuery(timepoints)}) AND
const buildExpressionProductionDegradationRatesQuery = function (rateType, genes) {
return `
SELECT gene.display_gene_id, ${rateType} FROM fall2021.${rateType}, fall2021.gene WHERE
((${buildExpressionGenesQuery(genes)})
AND fall2021.gene.gene_id = fall2021.expression.gene_id) ORDER BY sort_index;`
: `SELECT * FROM fall2021.expression, fall2021.gene WHERE fall2021.expression.dataset='${dataset}'
AND fall2021.gene.gene_id = fall2021.${rateType}.gene_id) ORDER BY display_gene_id;`;
};

const buildExpressionTimepointsFromDatasetQuery = function (dataset) {
return `
SELECT DISTINCT time_point, sample_id FROM fall2021.expression WHERE
dataset = '${dataset}' ORDER BY time_point ASC;`;
};

const buildExpressionDataQuery = function (dataset, genes) {
return `SELECT * FROM fall2021.expression, fall2021.gene WHERE fall2021.expression.dataset='${dataset}'
AND ((${buildExpressionGenesQuery(genes)})
AND fall2021.gene.gene_id = fall2021.expression.gene_id) ORDER BY sort_index;`;
};

let listExpressionGeneData = function (gene, totalOutput) {


const buildExpressionQuery = function (query) {
const expressionQueries = {
"DegradationRates": () => buildExpressionProductionDegradationRatesQuery("degradation_rate", query.genes),
"ProductionRates" : () => buildExpressionProductionDegradationRatesQuery("production_rate", query.genes),
"ExpressionData" : () => buildExpressionDataQuery(query.dataset, query.genes),
"ExpressionDatasets" : () => "SELECT DISTINCT dataset FROM fall2021.expression ORDER BY dataset ASC;",
"ExpressionTimePoints": () => buildExpressionTimepointsFromDatasetQuery(query.dataset)
};
if (Object.keys(expressionQueries).includes(query.type)) {
return expressionQueries[query.type]();
}
};

const listExpressionGeneData = function (gene, totalOutput) {
let listOfData = [];
totalOutput.forEach(function (x) {
if (x.display_gene_id === gene) {
@@ -96,7 +68,7 @@ let listExpressionGeneData = function (gene, totalOutput) {
return listOfData;
};

let convertExpressionToJSON = function (totalOutput, dataset, timePoints, allGenes) {
const convertExpressionToJSON = function (totalOutput, dataset, timePoints, allGenes) {
let JSONOutput = {
timePoints,
data: {
@@ -107,16 +79,53 @@ let convertExpressionToJSON = function (totalOutput, dataset, timePoints, allGen
return JSONOutput;
};

const ProductionDegradationRateToJSON = (totalOutput, rateType) => {
const JSONOutput = {
};
for (let gene of totalOutput) {
JSONOutput[gene.display_gene_id] = gene[rateType];
}
return JSONOutput;
};

const DatasetToJSON = (totalOutput) => {
const JSONOutput = {
expressionDatasets : []
};
for (let dataset of totalOutput) {
JSONOutput.expressionDatasets.push(dataset.dataset);
}
return JSONOutput;
};

const TimePointsToJSON = (totalOutput, dataset) => {
const JSONOutput = {};
JSONOutput[dataset] = [];
for (let timePoint of totalOutput) {
JSONOutput[dataset].push(timePoint.time_point);
}
return JSONOutput;
};

module.exports = {
queryExpressionDatabase: function (req, res) {
return sequelize.query(buildExpressionQuery(req.query.dataset, req.query.timepoints, req.query.genes),
return sequelize.query(buildExpressionQuery(req.query),
{ type: sequelize.QueryTypes.SELECT })
.then(function (stdname) {
let dataset = req.query.dataset;
let geneList = req.query.genes.split(",");
let response = convertExpressionToJSON(
stdname, dataset, expressionTimepointsByDataset[dataset], geneList);
return res.send(response);
const convertToJSON = {
"DegradationRates" : () => ProductionDegradationRateToJSON(stdname, "degradation_rate"),
"ProductionRates" : () => ProductionDegradationRateToJSON(stdname, "production_rate"),
"ExpressionData" : () => convertExpressionToJSON(
stdname,
req.query.dataset,
req.query.timepoints.split(",").map(x => Number(x)),
req.query.genes.split(",")),
"ExpressionDatasets": () => DatasetToJSON(stdname),
"ExpressionTimePoints": () => TimePointsToJSON(stdname, req.query.dataset)
};
const type = req.query.type;
return (Object.keys(convertToJSON).includes(type)) ? res.send(convertToJSON[type]()) :
res.send(500, { errors: "Something went wrong."});
});
}
};
31 changes: 18 additions & 13 deletions server/dals/network-dal.js
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
/* eslint-disable max-len */
const Sequelize = require("sequelize");
require("dotenv").config();
var env = process.env.NODE_ENV || "development";
@@ -38,21 +39,21 @@ const buildNetworkGenesQuery = function (geneString) {

};

const buildCreateNetworkQuery = function (genes, source, timestamp) {
const buildGenerateNetworkQuery = function (genes, source, timestamp) {
return `SELECT DISTINCT regulator_gene_id, target_gene_id FROM
spring2022_network.network WHERE
time_stamp='${timestamp}' AND source='${source}' AND
${buildNetworkGenesQuery(genes)} ORDER BY regulator_gene_id DESC;`;
};

const buildQueryByType = function (queryType, query) {
switch (queryType) {
case "NetworkSource":
return buildNetworkSourceQuery();
case "NetworkGeneFromSource":
return buildNetworkGeneFromSourceQuery(query.gene, query.source, query.timestamp);
case "CreateNetwork":
return buildCreateNetworkQuery(query.genes, query.source, query.timestamp);
const networkQueries = {
"NetworkSource": () => buildNetworkSourceQuery(),
"NetworkGeneFromSource": () => buildNetworkGeneFromSourceQuery(query.gene, query.source, query.timestamp),
"GenerateNetwork": () => buildGenerateNetworkQuery(query.genes, query.source, query.timestamp)
};
if (Object.keys(networkQueries).includes(query.type)) {
return networkQueries[query.type]();
}
};

@@ -62,16 +63,17 @@ const convertResponseToJSON = function (queryType, query, totalOutput) {
case "NetworkSource":
JSONOutput.sources = {};
totalOutput.forEach(function (x) {
let timestamp = x.time_stamp;
let source = x.source;
JSONOutput.sources[`${source} : ${timestamp}`] = {timestamp, source};
const timestamp = x.time_stamp;
const source = x.source;
const displayName = x.display_name;
JSONOutput.sources[`${displayName} : ${timestamp.toISOString().split("T")[0]}`] = {timestamp, source};
});
return JSONOutput;
case "NetworkGeneFromSource":
JSONOutput.displayGeneId = totalOutput.length > 0 ? totalOutput[0].display_gene_id : null;
JSONOutput.geneId = totalOutput.length > 0 ? totalOutput[0].gene_id : null;
return JSONOutput;
case "CreateNetwork":
case "GenerateNetwork":
JSONOutput.links = {};
for (let connection of totalOutput) {
if (JSONOutput.links[connection.regulator_gene_id] === undefined) {
@@ -81,15 +83,18 @@ const convertResponseToJSON = function (queryType, query, totalOutput) {
}
}
return JSONOutput;
default:
return JSONOutput;
}

};

module.exports = {
buildNetworkSourceQuery: buildNetworkSourceQuery,
queryNetworkDatabase: function (req, res) {
sequelize.query(buildQueryByType(req.query.type, req.query), { type: sequelize.QueryTypes.SELECT })
.then(function (stdname) {
let response = convertResponseToJSON(req.query.type, req.query, stdname);
const response = convertResponseToJSON(req.query.type, req.query, stdname);
return res.send(response);
});
}
2 changes: 1 addition & 1 deletion test/additional-sheet-parser-tests.js
Original file line number Diff line number Diff line change
@@ -103,7 +103,7 @@ describe("additional-sheet-parser", function () {
var workbook = xlsx.parse(__dirname + "/../test-files/spreadsheet-controller-test-files/" +
"2_column_data_format_test.xlsx");
var data = parseAdditionalSheets(workbook);
assert(data["test"]["degradation_rates"], {
assert(data.twoColumnSheets["degradation_rates"], {
"ACE2": 0.1118,
"ASH1": 0.2166,
"CIN5": 0.1005,
15 changes: 13 additions & 2 deletions test/export-tests.js
Original file line number Diff line number Diff line change
@@ -648,6 +648,16 @@ const inputWorkbook = {
}
};

inputWorkbook.exportSheets = {
networks: {
"network": inputWorkbook.network,
"network_weights": inputWorkbook.networkWeights
},
"optimization_parameters": inputWorkbook.meta,
"two_column_sheets": inputWorkbook.test,
expression: inputWorkbook.exportExpression
};

describe("Export to spreadsheet", function () {
it("should export a workbook to a spreadsheet object properly", function () {
const expectedSheet = [
@@ -783,8 +793,9 @@ describe("Export to spreadsheet", function () {

it("should export a workbook exactly as the import",
function () {
test.importFileSameAsExportFile(
"test-files/additional-sheet-test-files/optimization-diagnostics-default.xlsx");
// Commented out temporarily while reworking the export of the optimization diagnostics sheet
// test.importFileSameAsExportFile(
// "test-files/additional-sheet-test-files/optimization-diagnostics-default.xlsx");
test.importFileSameAsExportFile(
"test-files/expression-data-test-sheets/expression_sheet_missing_data_ok_export_exact.xlsx");
test.importFileSameAsExportFile(
75 changes: 56 additions & 19 deletions test/test.js
Original file line number Diff line number Diff line change
@@ -526,8 +526,8 @@ var twoColumnIdError = function (input, frequency) {
var sheet = xlsx.parse(input);
var workbook = parseAdditionalSheet(sheet);
var twoColumnIdErrorCount = 0;
for (let page in workbook.test) {
twoColumnIdErrorCount += workbook.test[page].errors.filter(function (x) {
for (let page in workbook.twoColumnSheets) {
twoColumnIdErrorCount += workbook.twoColumnSheets[page].errors.filter(function (x) {
return x.errorCode === "MISLABELED_ID_CELL";
}).length;
}
@@ -538,8 +538,8 @@ var additionalSheetIncorrectColumnHeaderError = function (input, frequency) {
var sheet = xlsx.parse(input);
var workbook = parseAdditionalSheet(sheet);
var additionalSheetIncorrectColumnHeaderErrorCount = 0;
for (let page in workbook.test) {
additionalSheetIncorrectColumnHeaderErrorCount += workbook.test[page].errors.filter(
for (let page in workbook.twoColumnSheets) {
additionalSheetIncorrectColumnHeaderErrorCount += workbook.twoColumnSheets[page].errors.filter(
(x) => x.errorCode === "INCORRECT_COLUMN_HEADER").length;
}
additionalSheetIncorrectColumnHeaderErrorCount += workbook.meta.errors.filter(
@@ -555,8 +555,8 @@ var additionalSheetMissingColumnHeaderError = function (input, frequency) {
var sheet = xlsx.parse(input);
var workbook = parseAdditionalSheet(sheet);
var additionalSheetMissingColumnHeaderErrorCount = 0;
for (let page in workbook.test) {
additionalSheetMissingColumnHeaderErrorCount += workbook.test[page].errors.filter(
for (let page in workbook.twoColumnSheets) {
additionalSheetMissingColumnHeaderErrorCount += workbook.twoColumnSheets[page].errors.filter(
(x) => x.errorCode === "MISSING_COLUMN_HEADER").length;
}
additionalSheetMissingColumnHeaderErrorCount += workbook.meta.errors.filter(
@@ -572,8 +572,8 @@ var twoColumnInvalidGeneTypeError = function (input, frequency) {
var sheet = xlsx.parse(input);
var workbook = parseAdditionalSheet(sheet);
var twoColumnInvalidGeneTypeErrorCount = 0;
for (let page in workbook.test) {
twoColumnInvalidGeneTypeErrorCount += workbook.test[page].errors.filter(function (x) {
for (let page in workbook.twoColumnSheets) {
twoColumnInvalidGeneTypeErrorCount += workbook.twoColumnSheets[page].errors.filter(function (x) {
return x.errorCode === "INVALID_GENE_TYPE";
}).length;
}
@@ -584,8 +584,8 @@ var twoColumnInvalidValueError = function (input, frequency) {
var sheet = xlsx.parse(input);
var workbook = parseAdditionalSheet(sheet);
var twoColumnInvalidValueErrorCount = 0;
for (let page in workbook.test) {
twoColumnInvalidValueErrorCount += workbook.test[page].errors.filter(function (x) {
for (let page in workbook.twoColumnSheets) {
twoColumnInvalidValueErrorCount += workbook.twoColumnSheets[page].errors.filter(function (x) {
return x.errorCode === "INVALID_VALUE";
}).length;
}
@@ -596,8 +596,8 @@ var twoColumnInvalidGeneLengthError = function (input, frequency) {
var sheet = xlsx.parse(input);
var workbook = parseAdditionalSheet(sheet);
var twoColumnInvalidGeneLengthErrorCount = 0;
for (let page in workbook.test) {
twoColumnInvalidGeneLengthErrorCount += workbook.test[page].errors.filter(function (x) {
for (let page in workbook.twoColumnSheets) {
twoColumnInvalidGeneLengthErrorCount += workbook.twoColumnSheets[page].errors.filter(function (x) {
return x.errorCode === "INVALID_GENE_LENGTH";
}).length;
}
@@ -608,8 +608,8 @@ var twoColumnSpecialCharacterError = function (input, frequency) {
var sheet = xlsx.parse(input);
var workbook = parseAdditionalSheet(sheet);
var twoColumnSpecialCharacterErrorCount = 0;
for (let page in workbook.test) {
twoColumnSpecialCharacterErrorCount += workbook.test[page].errors.filter(function (x) {
for (let page in workbook.twoColumnSheets) {
twoColumnSpecialCharacterErrorCount += workbook.twoColumnSheets[page].errors.filter(function (x) {
return x.errorCode === "INVALID_CHARACTER";
}).length;
}
@@ -622,8 +622,8 @@ var additionalSheetExtraneousDataWarning = function (input, frequency) {
var sheet = xlsx.parse(input);
var workbook = parseAdditionalSheet(sheet);
var additionalSheetExtraneousDataWarningCount = 0;
for (let page in workbook.test) {
additionalSheetExtraneousDataWarningCount += workbook.test[page].warnings.filter(function (x) {
for (let page in workbook.twoColumnSheets) {
additionalSheetExtraneousDataWarningCount += workbook.twoColumnSheets[page].warnings.filter(function (x) {
return x.warningCode === "EXTRANEOUS_DATA";
}).length;
}
@@ -716,7 +716,26 @@ var invalidMSEDataWarning = function (input, frequency) {
var importExportReImportNoErrorsOrWarnings = function (input) {
var sheet = xlsx.parse(input);
var inputWorkbook = spreadsheetController.crossSheetInteractions(sheet);
inputWorkbook["exportExpression"] = inputWorkbook.expression;
inputWorkbook.exportSheets = {
"optimization_parameters": inputWorkbook.meta,
expression: inputWorkbook.expression,
networks: {}
};
if (inputWorkbook.network) {
inputWorkbook.exportSheets.networks["network"] = inputWorkbook.network;
}
if (inputWorkbook.networkOptimizedWeights) {
inputWorkbook.exportSheets.networks["network_optimized_weights"] = inputWorkbook.networkOptimizedWeights;
}
if (inputWorkbook.networkWeights) {
inputWorkbook.exportSheets.networks["network_weights"] = inputWorkbook.networkWeights;
}
if (inputWorkbook.twoColumnSheets) {
inputWorkbook.exportSheets["two_column_sheets"] = inputWorkbook.twoColumnSheets;
}
if (inputWorkbook.meta2) {
inputWorkbook.exportSheets["optimization_diagnostics"] = inputWorkbook.meta2;
}
var exportedWorkbook = exportController.grnsightToXlsx(inputWorkbook);
var sheet2 = xlsx.parse(exportedWorkbook);
var reImportedWorkbook = spreadsheetController.crossSheetInteractions(sheet2);
@@ -726,8 +745,26 @@ var importExportReImportNoErrorsOrWarnings = function (input) {
var importFileSameAsExportFile = function (input) {
var sheet = xlsx.parse(input);
var inputWorkbook = spreadsheetController.crossSheetInteractions(sheet);
inputWorkbook["exportExpression"] = inputWorkbook.expression;
inputWorkbook["exportNetworkType"] = inputWorkbook.sheetType;
inputWorkbook.exportSheets = {
"optimization_parameters": inputWorkbook.meta,
expression: inputWorkbook.expression,
networks: {}
};
if (inputWorkbook.network) {
inputWorkbook.exportSheets.networks["network"] = inputWorkbook.network;
}
if (inputWorkbook.networkOptimizedWeights) {
inputWorkbook.exportSheets.networks["network_optimized_weights"] = inputWorkbook.networkOptimizedWeights;
}
if (inputWorkbook.networkWeights) {
inputWorkbook.exportSheets.networks["network_weights"] = inputWorkbook.networkWeights;
}
if (inputWorkbook.twoColumnSheets) {
inputWorkbook.exportSheets["two_column_sheets"] = inputWorkbook.twoColumnSheets;
}
if (inputWorkbook.meta2) {
inputWorkbook.exportSheets["optimization_diagnostics"] = inputWorkbook.meta2;
}
var exportedWorkbook = exportController.grnsightToXlsx(inputWorkbook);
var sheet2 = xlsx.parse(exportedWorkbook);
sheet.sort((a, b) => (a.name > b.name) ? 1 : -1);
172 changes: 68 additions & 104 deletions web-client/public/js/api/grnsight-api.js
Original file line number Diff line number Diff line change
@@ -1,124 +1,88 @@
import {responseCustomWorkbookData} from "../setup-load-and-import-handlers";

// Expression DB Access Functions
const buildExpressionTimepointsString = function (selection) {
let timepoints = "";
selection.timepoints.forEach(x => timepoints += (x + ","));
return timepoints.substring(0, timepoints.length - 1);
};
const buildExpressionGeneQuery = function (workbookGenes) {
let genes = "";
workbookGenes.forEach(x => genes += (x.name + ","));
return genes.substring(0, genes.length - 1);
import { responseCustomWorkbookData } from "../setup-load-and-import-handlers";
// General DB Access Functions
const buildQueryURL = function(path, parameters) {
const searchParams = new URLSearchParams("");
for (let p in parameters) {
searchParams.append(p, parameters[p]);
}
return `${path}?${searchParams.toString()}`;
};

const buildExpressionURL = function (selection, genes) {
const baseQuery = `expressiondb?dataset=${selection.dataset}&genes=${buildExpressionGeneQuery(genes)}`;
return selection.timepoints ?
`${baseQuery}&timepoints=${buildExpressionTimepointsString(selection)}` :
baseQuery;
const responseData = (database, formData, queryURL) => {
return new Promise(function(resolve) {
const uploadRoute = queryURL;
const fullUrl = [$(".service-root").val(), uploadRoute].join("/");
(formData
? $.ajax({
url: fullUrl,
data: formData,
processData: false,
contentType: false,
type: "GET",
crossDomain: true,
})
: $.getJSON(fullUrl)
)
.done((data) => {
resolve(data);
})
.error(function() {
console.log(
`Error in accessing ${database} database. Result may just be loading.`
);
});
});
};

const responseExpressionData = (formData, queryURL) => {
return new Promise(function (resolve) {
const uploadRoute = queryURL;
const fullUrl = [ $(".service-root").val(), uploadRoute ].join("/");
(formData ?
$.ajax({
url: fullUrl,
data: formData,
processData: false,
contentType: false,
type: "GET",
crossDomain: true
}) :
$.getJSON(fullUrl)
).done((expressionData) => {
resolve(expressionData);
}).error(console.log("Error in accessing expression database. Result may just be loading."));
});
};
// Expression DB Access Functions

const queryExpressionDatabase = (query) => {
let queryURL = buildExpressionURL({dataset: query.dataset}, query.genes);
return responseExpressionData("", queryURL);
const queryURL = buildQueryURL("expressiondb", query);
return responseData("expression", "", queryURL);
};

// Network DB Access Functions

const buildNetworkGenesQuery = (genes) => {
let result = "";
for (let gene in genes) {
result += `${gene},`;
}
return result.substring(0, result.length - 1);
const queryNetworkDatabase = (query) => {
const queryURL = buildQueryURL("networkdb", query);
return responseData("network", "", queryURL);
};

const buildNetworkURL = function (queryType, queryInfo) {
let baseQuery = `networkdb?type=${queryType}`;
if (queryInfo !== null) {
for (let header in queryInfo) {
if (header === "genes") {
baseQuery += `&${header}=${buildNetworkGenesQuery(queryInfo[header])}`;
} else {
baseQuery += `&${header}=${queryInfo[header]}`;
}
}
}
return baseQuery;
};
// Upload Custom Workbook Functions

const responseNetworkData = (formData, queryURL) => {
return new Promise(function (resolve) {
const uploadRoute = queryURL;
const fullUrl = [ $(".service-root").val(), uploadRoute ].join("/");
(formData ?
$.ajax({
url: fullUrl,
data: formData,
processData: false,
contentType: false,
type: "GET",
crossDomain: true
}) :
$.getJSON(fullUrl)
).done((networkData) => {
resolve(networkData);
}).error(console.log("Error in accessing network database. Result may just be loading."));
});
const uploadCustomWorkbook = (workbook, grnState) => {
const queryURL = buildQueryURL("upload-custom-workbook", workbook);
return responseCustomWorkbookData(grnState, queryURL, workbook.name);
};

const queryNetworkDatabase = (query) => {
let queryURL = buildNetworkURL(query.type, query.info);
return responseNetworkData("", queryURL);
};
const constructFullUrl = (queryURL) =>
[$(".service-root").val(), queryURL].join("/");

// Upload Custom Workbook Functions
const buildCustomWorkbookURL = (name, genes, links) => {
let baseQuery = `upload-custom-workbook?name=${name}`;
let genesString = "";
let linksString = "";
let genesByIndex = {};
let i = 0;
for (let gene in genes) {
genesString += `${genes[gene]},`;
genesByIndex[gene] = i;
i++;
}
for (let regulator in links) {
for (let target of links[regulator]) {
linksString += `${genesByIndex[regulator]}->${genesByIndex[target]},`;
}
}
baseQuery += `&genes=${genesString.substring(0, genesString.length - 1)}`;
baseQuery += `&links=${linksString.substring(0, linksString.length - 1)}`;
return baseQuery;
};
const getWorkbookFromForm = (formData, queryURL) => {
const fullUrl = constructFullUrl(queryURL);

const uploadCustomWorkbook = (workbook, grnState) => {
let queryURL = buildCustomWorkbookURL(workbook.name, workbook.genes, workbook.links);
return responseCustomWorkbookData(grnState, queryURL, workbook.name);
// The presence of formData is taken to indicate a POST.
return formData
? $.ajax({
url: fullUrl,
data: formData,
processData: false,
contentType: false,
type: "POST",
crossDomain: true,
})
: $.getJSON(fullUrl);
};

const getWorkbookFromUrl = (queryURL) => {
const fullUrl = constructFullUrl(queryURL);
return $.getJSON(fullUrl);
};

export {queryExpressionDatabase, queryNetworkDatabase, uploadCustomWorkbook};
export {
queryExpressionDatabase,
queryNetworkDatabase,
uploadCustomWorkbook,
getWorkbookFromForm,
getWorkbookFromUrl,
};
2 changes: 1 addition & 1 deletion web-client/public/js/constants.js
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ export const GREY_EDGES_DASHED_MENU = "#grey-edges-dashed-menu";
export const GREY_EDGES_DASHED_SIDEBAR = "#dashedGrayLineButton";

export const CREATE_NETWORK_CLASS = ".create-network";
export const CREATE_NETWORK_MODAL = "#createNetworkModal";
export const CREATE_NETWORK_MODAL = "#generateNetworkModal";

export const UNWEIGHTED_DEMO_ID = ".unweighted";
export const UNWEIGHTED_DEMO_PATH = "demo/unweighted";
Original file line number Diff line number Diff line change
@@ -1,15 +1,24 @@
/* eslint-disable max-len */
import {CREATE_NETWORK_CLASS, CREATE_NETWORK_MODAL} from "./constants";
import { queryNetworkDatabase, uploadCustomWorkbook } from "./api/grnsight-api";
import { grnState } from "./grnstate";

export const createNetwork = function () {
export const generateNetwork = function () {
const GENE_EXCEPTIONS = {
"DUR1,2" : "DUR12",
"IMP2'" : "IMP21",
"ARG5,6" : "ARG56",
"ADE5,7" : "ADE57",
"MF(ALPHA)1" : "YPL187W",
"MF(ALPHA)2" : "YGL089C"
};
const createHTMLforForm = (sources) => {
let result = `
<div id=\'createNetworkFormContainer\' '>
<h2 id=\'createNetwork\'>Create Network</h2>
<div class=\'form-group\'>
<label for=\'network-source\' id=\'network-source-label\'>Network Source</label>
<select class=\'network-dropdown btn btn-default\' id=\'network-source\'>
<div id=\'generateNetworkFormContainer\' '>
<h2 id=\'generateNetwork\'>Generate Network</h2>
<div class=\'form-group\'>
<label for=\'network-source\' id=\'network-source-label\'>Network Source</label>
<select class=\'network-dropdown btn btn-default\' id=\'network-source\'>
`;
if (sources.length !== 1) {
result += "<option value=\'none\' selected=\'true\' disabled>Select Network Source</option>";
@@ -24,10 +33,10 @@ export const createNetwork = function () {
`;
}
result += `</select>
<p>Warning: changing network source will remove all current genes in network</p>
<p>Warning: changing network source will clear the list of genes below.</p>
</div>
<div class=\'form-group\' id=\'getNetworkGenesForm\'>
<form id=\'getNetworkGenesForm\'>
<form id=\'getNetworkGenesForm\' class=\'NetworkGenesForm\' >
<label for=\'network-search-bar\' id=\'network-source-label\'>Select genes</label>
<input type=\'text\' id=\'network-search-bar\' name=\'network-search-bar\'></input>
<button id=\'enter-search\' type=\'submit\' class=\'search-button btn btn-default\'>
@@ -37,10 +46,7 @@ export const createNetwork = function () {
</div>
<div id=\'selected-genes-container\'>
<div id=\'selected-genes\'>
<p>Added genes go here! Click on a gene to remove it</p>
</div>
</div>
<button id=\'submit-network\' class=\'btn btn-default\'>Create Network</input>
<p>Added genes go here! Click on a gene to remove it.</p>
</div>
`;
return result;
@@ -78,28 +84,38 @@ export const createNetwork = function () {
}
};

const validGene = function (gene) {
if (/^[A-Z0-9_-]{1,12}$/.test(gene)) {
return gene;
}
if (Object.keys(GENE_EXCEPTIONS).includes(gene)) {
return GENE_EXCEPTIONS[gene];
}
return "";
};
const addGene = function () {
let gene = `${$("#network-search-bar").val()}`.toUpperCase();
const searchGene = `${$("#network-search-bar").val()}`.toUpperCase();
$("#network-search-bar").val("");
if (!(/^[A-Z0-9_-]{1,12}$/.test(gene))) {
alert(`Gene: ${gene} is not to GRNsight specifications. Genes must be 12 characters or less,
const gene = validGene(searchGene);
if (gene === "") {
alert(`Gene: ${searchGene} is not to GRNsight specifications. Genes must be 12 characters or less,
containing "-", "_", and alpha-numeric characters only`);
} else {
let source = grnState.customWorkbook.source;
let headers = {
type:"NetworkGeneFromSource",
info: {
gene: gene,
source:grnState.customWorkbook.sources[source].source,
timestamp:grnState.customWorkbook.sources[source].timestamp.substring(0, 19).replace("T", " ")
}
gene: gene,
source:grnState.customWorkbook.sources[source].source,
timestamp:grnState.customWorkbook.sources[source].timestamp
};
queryNetworkDatabase(headers).then(function (response) {
if (response.geneId !== null && response.displayGeneId !== null) {
grnState.customWorkbook.genes[response.geneId] = response.displayGeneId;
displayCurrentGenes();
} else {
alert(`Gene: ${gene} was not found in this database. Please check for any typos and try again.`);
alert(
`Gene: ${searchGene} was not found in this database. Please check for any typos and try again.`
);
}
}).catch(function (error) {
console.log(error.stack);
@@ -110,17 +126,31 @@ containing "-", "_", and alpha-numeric characters only`);
}
};

const displayCreateNetworkModal = function () {
$("#createNetworkFormContainer").remove();
const createHTMLforModalButtons = () => {
return `
<div id=\'generateNetworkFooter\' class=\'modal-footer-div\'>
<div>
<input type=\'button\' class=\'btn btn-default\' id=\'submit-network\' value=\'Generate Network\'/>
<input type=\'button\' class=\'btn btn-default\' data-dismiss=\'modal\' value=\'Cancel\' />
</div>
</div>
`;
};

const displayGenerateNetworkModal = function () {
$("#generateNetworkFormContainer").remove();
$("#generateNetworkFooter").remove();
$("#generateNetworkFooter-container").append(createHTMLforModalButtons());
grnState.customWorkbook = {
genes : {},
source : null
source : null,
sources : null
};
// get sources from database
queryNetworkDatabase({type:"NetworkSource", info:null}).then(function (response) {
$("#createNetworkQuestions-container").append(createHTMLforForm(Object.keys(response.sources)));
queryNetworkDatabase({type:"NetworkSource"}).then(function (response) {
$("#generateNetworkQuestions-container").append(createHTMLforForm(Object.keys(response.sources)));
grnState.customWorkbook.sources = response.sources;
grnState.customWorkbook.source = Object.keys(response.sources).length === 1 ?
grnState.customWorkbook.source = Object.keys(response.sources).length >= 1 ?
Object.keys(response.sources)[0] : null;
}).catch(function (error) {
console.log(error.stack);
@@ -133,7 +163,7 @@ containing "-", "_", and alpha-numeric characters only`);
$("body").on("click", CREATE_NETWORK_CLASS, function (event) {
event.preventDefault();
event.stopPropagation();
displayCreateNetworkModal();
displayGenerateNetworkModal();
});

$("body").on("change", "#network-source", function (event) {
@@ -150,28 +180,25 @@ containing "-", "_", and alpha-numeric characters only`);
alert(`GRNsight is only capable of handling 75 genes at most. Your proposed network contains
${genesAmount} genes. Please remove some genes from your proposed network.`);
} else {

let source = grnState.customWorkbook.source;
let headers = {
type:"CreateNetwork",
info: {
genes: grnState.customWorkbook.genes,
source:grnState.customWorkbook.sources[source].source,
timestamp:grnState.customWorkbook.sources[source].timestamp.substring(0, 19).replace("T", " ")
}
const genes = Object.keys(grnState.customWorkbook.genes).map(g => grnState.customWorkbook.genes[g]);
const source = grnState.customWorkbook.source;
const headers = {
type:"GenerateNetwork",
genes: genes.join(","),
source:grnState.customWorkbook.sources[source].source,
timestamp:grnState.customWorkbook.sources[source].timestamp
};
queryNetworkDatabase(headers).then(function (response) {
grnState.customWorkbook.links = response.links;
let genes = grnState.customWorkbook.genes;
let links = grnState.customWorkbook.links;
let genesAmount = Object.keys(genes).length;
let edgesAmount = Object.keys(links).length;
const links = Object.entries(grnState.customWorkbook.links);
const genesAmount = genes.length;
const edgesAmount = links.flatMap( (entry) => entry[1].map((target) => [entry[0], target])).length;
if (edgesAmount > 100) {
alert(`GRNsight is only capable of handling 100 edges at most. Your proposed network contains
${edgesAmount} regulatory connections. Please remove some genes from your proposed network.`);
} else {
let name = `Custom Workbook: UnweightedGRN(${genesAmount} genes, ${edgesAmount} edges)`;
let workbook = {name, genes, links};
const name = `Custom Workbook: UnweightedGRN(${genesAmount} genes, ${edgesAmount} edges)`;
const workbook = {name, genes, links : links.map( l => `${l[0]}->${l[1]}`).join(",")};
uploadCustomWorkbook(workbook, grnState);
$(CREATE_NETWORK_MODAL).modal("hide");
}
4 changes: 2 additions & 2 deletions web-client/public/js/grnsight.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { displayStatistics } from "./graph-statistics"; // eslint-disable-line no-unused-vars
import { upload } from "./upload";
import { createNetwork } from "./createNetwork";
import { generateNetwork } from "./generateNetwork";

import { grnState } from "./grnstate";
import { updateApp } from "./update-app";
@@ -10,4 +10,4 @@ setupHandlers(grnState);
updateApp(grnState);

upload();
createNetwork();
generateNetwork();
350 changes: 181 additions & 169 deletions web-client/public/js/setup-load-and-import-handlers.js
Original file line number Diff line number Diff line change
@@ -1,200 +1,212 @@
import { updateApp } from "./update-app";

import {
DEMO_INFORMATION,
UNWEIGHTED_DEMO_PATH,
WEIGHTED_DEMO_PATH,
SCHADE_INPUT_PATH,
SCHADE_OUTPUT_PATH,
WEIGHTED_DEMO_NAME,
UNWEIGHTED_DEMO_NAME,
SCHADE_INPUT_NAME,
SCHADE_OUTPUT_NAME,
DEMO_INFORMATION,
UNWEIGHTED_DEMO_PATH,
WEIGHTED_DEMO_PATH,
SCHADE_INPUT_PATH,
SCHADE_OUTPUT_PATH,
WEIGHTED_DEMO_NAME,
UNWEIGHTED_DEMO_NAME,
SCHADE_INPUT_NAME,
SCHADE_OUTPUT_NAME,
} from "./constants";
import { getWorkbookFromForm, getWorkbookFromUrl } from "./api/grnsight-api";

const demoFiles = [UNWEIGHTED_DEMO_PATH, WEIGHTED_DEMO_PATH, SCHADE_INPUT_PATH, SCHADE_OUTPUT_PATH];
const demoFiles = [
UNWEIGHTED_DEMO_PATH,
WEIGHTED_DEMO_PATH,
SCHADE_INPUT_PATH,
SCHADE_OUTPUT_PATH,
];

const submittedFilename = $upload => {
let path = $upload.val();
let fakePathCheck = path.search("\\\\") + 1;
const submittedFilename = ($upload) => {
let path = $upload.val();
let fakePathCheck = path.search("\\\\") + 1;

while (fakePathCheck) {
path = path.substring(fakePathCheck);
fakePathCheck = path.search("\\\\") + 1;
}
while (fakePathCheck) {
path = path.substring(fakePathCheck);
fakePathCheck = path.search("\\\\") + 1;
}

return path;
return path;
};

const createFileForm = $upload => {
const formData = new FormData();
formData.append("file", $upload[0].files[0]);
return formData;
const createFileForm = ($upload) => {
const formData = new FormData();
formData.append("file", $upload[0].files[0]);
return formData;
};

const uploadEpilogue = event => {
if (window.ga) {
window.ga("send", "pageview", {
page: "/GRNsight/upload",
sessionControl: "start"
});
}
const uploadEpilogue = (event) => {
if (window.ga) {
window.ga("send", "pageview", {
page: "/GRNsight/upload",
sessionControl: "start",
});
}

$("a.upload > input[type=file]").val("");
event.preventDefault();
$("a.upload > input[type=file]").val("");
event.preventDefault();
};
const disableUpload = state => {
$(".upload").attr("disabled", state);
$(".upload-sif").attr("disabled", state);
$(".upload-graphml").attr("disabled", state);
const disableUpload = (state) => {
$(".upload").attr("disabled", state);
$(".upload-sif").attr("disabled", state);
$(".upload-graphml").attr("disabled", state);
};

const uploadHandler = (uploader) => {
return function (event) { // Must be `function` due to use of `this`.
const $upload = $(this);
const filename = submittedFilename($upload); // TODO: remove before master release (beta@4.0.6)
if ($upload[0].files[0].size < 2000000) {
// disable upload button to prevent multiple uploads
disableUpload(true);
const formData = createFileForm($upload);
uploader(filename, formData);
uploadEpilogue(event);
} else {
let errorString = "The file uploaded is too large. Please try again with a file smaller than 1 MB.";
$("#error").html(errorString);
$("#errorModal").modal("show");
}
};
};

const workbookErrorDisplayer = xhr => {
// re-enable upload button
disableUpload(false);
// Deleted status, error for argument because it was never used
const err = JSON.parse(xhr.responseText);
let errorString = "Your graph failed to load.<br><br>";

if (!err.errors) { // will be falsy if an error was thrown before the workbook was generated
errorString += err;
return function(event) {
// Must be `function` due to use of `this`.
const $upload = $(this);
const filename = submittedFilename($upload); // TODO: remove before master release (beta@4.0.6)
if ($upload[0].files[0].size < 2000000) {
// disable upload button to prevent multiple uploads
disableUpload(true);
const formData = createFileForm($upload);
uploader(filename, formData);
uploadEpilogue(event);
} else {
errorString = err.errors.reduce(
(currentErrorString, currentError) =>
`${currentErrorString}${currentError.possibleCause} ${currentError.suggestedFix}<br><br>`,

errorString
);
let errorString =
"The file uploaded is too large. Please try again with a file smaller than 1 MB.";
$("#error").html(errorString);
$("#errorModal").modal("show");
}

$("#error").html(errorString);
$("#errorModal").modal("show");
};
};

let reloader = () => { };


const returnUploadRoute = filename => {
if (demoFiles.indexOf(filename) !== -1) {
return filename;
} else if (filename.includes(".xlsx")) {
return "upload";
} else if (filename.includes(".sif")) {
return "upload-sif";
} else if (filename.includes(".graphml")) {
return "upload-graphml";
}
const workbookErrorDisplayer = (xhr) => {
// re-enable upload button
disableUpload(false);
// Deleted status, error for argument because it was never used
const err = JSON.parse(xhr.responseText);
let errorString = "Your graph failed to load.<br><br>";

if (!err.errors) {
// will be falsy if an error was thrown before the workbook was generated
errorString += err;
} else {
errorString = err.errors.reduce(
(currentErrorString, currentError) =>
`${currentErrorString}${currentError.possibleCause} ${currentError.suggestedFix}<br><br>`,

errorString
);
}

$("#error").html(errorString);
$("#errorModal").modal("show");
};

export const setupLoadAndImportHandlers = grnState => {
const loadGrn = (name, formData) => {
const uploadRoute = returnUploadRoute(name);
const fullUrl = [ $(".service-root").val(), uploadRoute ].join("/");
// The presence of formData is taken to indicate a POST.
(formData ?
$.ajax({
url: fullUrl,
data: formData,
processData: false,
contentType: false,
type: "POST",
crossDomain: true
}) :
$.getJSON(fullUrl)
).done((workbook, textStatus, jqXhr) => {
grnState.name = name || jqXhr.getResponseHeader("X-GRNsight-Filename");
if (demoFiles.indexOf(name) > -1) {
switch (name) {
case WEIGHTED_DEMO_PATH:
grnState.name = WEIGHTED_DEMO_NAME;
break;
case UNWEIGHTED_DEMO_PATH:
grnState.name = UNWEIGHTED_DEMO_NAME;
break;
case SCHADE_INPUT_PATH:
grnState.name = SCHADE_INPUT_NAME;
break;
case SCHADE_OUTPUT_PATH:
grnState.name = SCHADE_OUTPUT_NAME;
}
}
grnState.workbook = workbook;
if (uploadRoute !== "upload") {
grnState.annotateLinks();
}
reloader = () => loadGrn(name, formData);
// re-enable upload button
disableUpload(false);
updateApp(grnState);
// displayStatistics(workbook);
}).error(workbookErrorDisplayer);
};
/*
* Thanks to http://stackoverflow.com/questions/6974684/how-to-send-formdata-objects-with-ajax-requests-in-jquery
* for helping to resolve this.
*/

// $(".upload").change(uploadHandler(loadGrn));
$("body").on("change", ".upload", uploadHandler(loadGrn));
const loadDemo = (url, value) => {
$("#demoSourceDropdown option[value='" + value.substring(1) + "']").prop("selected", true);
loadGrn(url);
reloader = () => loadGrn(url);

$("a.upload > input[type=file]").val("");
};

const initializeDemoFile = (demoClass, demoPath, demoName) => {
// Deleted parameter `event`
$(demoClass).on("click", () => {
loadDemo(demoPath, demoClass, demoName);
});

$("#demoSourceDropdown").on("change", () => {
loadDemo(demoPath, demoClass, demoName);
});
};

DEMO_INFORMATION.forEach(demoInfo => initializeDemoFile.apply(null, demoInfo));

$("body").on("click", ".reload", function () {
// Deleted `event` parameter but need `function` because of `this`.
if (!$(this).parent().hasClass("disabled")) {
if ($.isFunction(reloader)) {
reloader();
}
}
});
let reloader = () => {};

const returnUploadRoute = (filename) => {
if (demoFiles.indexOf(filename) !== -1) {
return filename;
} else if (filename.includes(".xlsx")) {
return "upload";
} else if (filename.includes(".sif")) {
return "upload-sif";
} else if (filename.includes(".graphml")) {
return "upload-graphml";
}
};

export const responseCustomWorkbookData = (grnState, queryURL, name) => {
const uploadRoute = queryURL;
const fullUrl = [ $(".service-root").val(), uploadRoute ].join("/");
$.getJSON(fullUrl).done((workbook) => {
grnState.name = name;
export const setupLoadAndImportHandlers = (grnState) => {
const loadGrn = (name, formData) => {
const uploadRoute = returnUploadRoute(name);
// The presence of formData is taken to indicate a POST.
getWorkbookFromForm(formData, uploadRoute)
.done((workbook, textStatus, jqXhr) => {
grnState.name = name || jqXhr.getResponseHeader("X-GRNsight-Filename");
if (demoFiles.indexOf(name) > -1) {
switch (name) {
case WEIGHTED_DEMO_PATH:
grnState.name = WEIGHTED_DEMO_NAME;
break;
case UNWEIGHTED_DEMO_PATH:
grnState.name = UNWEIGHTED_DEMO_NAME;
break;
case SCHADE_INPUT_PATH:
grnState.name = SCHADE_INPUT_NAME;
break;
case SCHADE_OUTPUT_PATH:
grnState.name = SCHADE_OUTPUT_NAME;
}
}
grnState.workbook = workbook;
grnState.annotateLinks();
grnState.workbook.expressionNames = Object.keys(workbook.expression);
if (uploadRoute !== "upload") {
grnState.annotateLinks();
}
reloader = () => loadGrn(name, formData);
// re-enable upload button
disableUpload(false);
updateApp(grnState);
reloader = () => responseCustomWorkbookData(grnState, queryURL, name);
// displayStatistics(workbook);
})
.error(workbookErrorDisplayer);
};
/*
* Thanks to http://stackoverflow.com/questions/6974684/how-to-send-formdata-objects-with-ajax-requests-in-jquery
* for helping to resolve this.
*/

// $(".upload").change(uploadHandler(loadGrn));
$("body").on("change", ".upload", uploadHandler(loadGrn));
const loadDemo = (url, value) => {
$("#demoSourceDropdown option[value='" + value.substring(1) + "']").prop(
"selected",
true
);
loadGrn(url);
reloader = () => loadGrn(url);

$("a.upload > input[type=file]").val("");
};

const initializeDemoFile = (demoClass, demoPath, demoName) => {
// Deleted parameter `event`
$(demoClass).on("click", () => {
loadDemo(demoPath, demoClass, demoName);
});
$("#demoSourceDropdown").on("change", () => {
const selected = `.${$("#demoSourceDropdown").val()}`;
if (selected === demoClass) {
loadDemo(demoPath, demoClass, demoName);
}
});
};

DEMO_INFORMATION.forEach((demoInfo) =>
initializeDemoFile.apply(null, demoInfo)
);

$("body").on("click", ".reload", function() {
// Deleted `event` parameter but need `function` because of `this`.
if (
!$(this)
.parent()
.hasClass("disabled")
) {
if ($.isFunction(reloader)) {
reloader();
}
}
});
};

export const responseCustomWorkbookData = (grnState, queryURL, name) => {
const uploadRoute = queryURL;
getWorkbookFromUrl(uploadRoute).done((workbook) => {
grnState.name = name;
grnState.workbook = workbook;
// Reset the node coloring dataset selection
grnState.nodeColoring.topDataset = undefined;
grnState.nodeColoring.bottomDataset = undefined;
grnState.annotateLinks();
disableUpload(false);
updateApp(grnState);
reloader = () => responseCustomWorkbookData(grnState, queryURL, name);
});
};
60 changes: 40 additions & 20 deletions web-client/public/js/update-app.js
Original file line number Diff line number Diff line change
@@ -111,6 +111,15 @@ import { queryExpressionDatabase } from "./api/grnsight-api.js";

// In this transitory state, updateApp might get called before things are completely set up, so for now
// we define this wrapper function that guards against uninitialized values.

queryExpressionDatabase({type:"ExpressionDatasets"}).then(function (response) {
grnState.database = response;
}).catch(function (error) {
console.log(error.stack);
console.log(error.name);
console.log(error.message);
});

const refreshApp = () => {
if (uploadState && uploadState.currentWorkbook) {
drawGraph(uploadState.currentWorkbook);
@@ -369,20 +378,33 @@ const enableNodeColoringUI = function () {
$(LOG_FOLD_CHANGE_MAX_VALUE_SIDEBAR_BUTTON).removeClass("hidden");
$(LOG_FOLD_CHANGE_MAX_VALUE_HEADER).removeClass("hidden");
};

const loadExpressionDatabase = function (isTopDataset) {
const dataset = isTopDataset ? grnState.nodeColoring.topDataset : grnState.nodeColoring.bottomDataset;
startLoadingIcon();
queryExpressionDatabase({
dataset: isTopDataset ? grnState.nodeColoring.topDataset : grnState.nodeColoring.bottomDataset,
genes : grnState.workbook.genes
}).then(function (response) {
if (isTopDataset) {
grnState.workbook.expression[grnState.nodeColoring.topDataset] = response;
} else {
grnState.workbook.expression[grnState.nodeColoring.bottomDataset] = response;
}
enableNodeColoringUI();
stopLoadingIcon();
updaters.renderNodeColoring();
type: "ExpressionTimePoints",
dataset
}).then(function (timepointsResponse) {
queryExpressionDatabase({
type:"ExpressionData",
dataset,
genes : grnState.workbook.genes.map(x => {return x.name;}).join(","),
timepoints: timepointsResponse[dataset]
}).then(function (response) {
if (isTopDataset) {
grnState.workbook.expression[grnState.nodeColoring.topDataset] = response;
} else {
grnState.workbook.expression[grnState.nodeColoring.bottomDataset] = response;
}
enableNodeColoringUI();
stopLoadingIcon();
updaters.renderNodeColoring();
}).catch(function (error) {
console.log(error.stack);
console.log(error.name);
console.log(error.message);
});
}).catch(function (error) {
console.log(error.stack);
console.log(error.name);
@@ -599,10 +621,6 @@ const clearDropdownMenus = () => {
$(BOTTOM_DATASET_SELECTION_SIDEBAR).html("");
};

const expressionDBDatasets = ["Barreto_2012_wt", "Dahlquist_2018_dcin5",
"Dahlquist_2018_dgln3", "Dahlquist_2018_dhap4", "Dahlquist_2018_dzap1",
"Dahlquist_2018_wt", "Kitagawa_2002_wt", "Thorsen_2007_wt"];

const resetDatasetDropdownMenus = (workbook) => {
clearDropdownMenus();
$(".dataset-option").remove(); // clear all menu dataset options
@@ -625,7 +643,8 @@ const resetDatasetDropdownMenus = (workbook) => {
}

// Add expression database options
expressionDBDatasets.forEach(option => grnState.nodeColoring.nodeColoringOptions.push({value: [option]}));
grnState.database.expressionDatasets.forEach( option =>
grnState.nodeColoring.nodeColoringOptions.push({value: [option]}));

$(BOTTOM_DATASET_SELECTION_SIDEBAR).append($("<option>")
.attr("value", "Same as Top Dataset").text("Same as Top Dataset"));
@@ -697,6 +716,7 @@ if (!grnState.genePageData.identified) {

export const updateApp = grnState => {
if (grnState.newWorkbook) {
grnState.nodeColoring.nodeColoringEnabled = true;
grnState.normalizationMax = max(grnState.workbook.positiveWeights.concat(grnState.workbook.negativeWeights));
displayworkbook(grnState.workbook, grnState.name);
expandLayoutSidebar();
@@ -793,12 +813,12 @@ export const updateApp = grnState => {
$(NODE_COLORING_TOGGLE_SIDEBAR).prop("checked", true);
$(LOG_FOLD_CHANGE_MAX_VALUE_CLASS).val(DEFAULT_MAX_LOG_FOLD_CHANGE);
$(NODE_COLORING_SIDEBAR_BODY).removeClass("hidden");
if (expressionDBDatasets.includes(grnState.nodeColoring.topDataset) &&
if (grnState.database.expressionDatasets.includes(grnState.nodeColoring.topDataset) &&
grnState.workbook.expression[grnState.nodeColoring.topDataset] === undefined) {
if ($(NODE_COLORING_TOGGLE_SIDEBAR).prop("checked")) {
loadExpressionDatabase(true);
}
} else if (expressionDBDatasets.includes(grnState.nodeColoring.bottomDataset) &&
} else if (grnState.database.expressionDatasets.includes(grnState.nodeColoring.bottomDataset) &&
!grnState.nodeColoring.bottomDataSameAsTop &&
grnState.workbook.expression[grnState.nodeColoring.bottomDataset] === undefined) {
if (!grnState.nodeColoring.bottomDataSameAsTop) {
@@ -817,9 +837,9 @@ export const updateApp = grnState => {
}
grnState.nodeColoring.showMenu = true;
grnState.nodeColoring.topDataset = grnState.nodeColoring.topDataset ?
grnState.nodeColoring.topDataset : "Barreto_2012_wt";
grnState.nodeColoring.topDataset : "Dahlquist_2018_wt";
grnState.nodeColoring.bottomDataset = grnState.nodeColoring.bottomDataset ?
grnState.nodeColoring.bottomDataset : "Barreto_2012_wt";
grnState.nodeColoring.bottomDataset : "Dahlquist_2018_wt";
$(LOG_FOLD_CHANGE_MAX_VALUE_CLASS).addClass("hidden");
$(LOG_FOLD_CHANGE_MAX_VALUE_SIDEBAR_BUTTON).addClass("hidden");
$(LOG_FOLD_CHANGE_MAX_VALUE_HEADER).addClass("hidden");
Loading

0 comments on commit 846e9ab

Please sign in to comment.