Skip to content

Commit cca1e1d

Browse files
carsonhhcarsonhh
committed
first commit
0 parents  commit cca1e1d

File tree

187 files changed

+5050
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

187 files changed

+5050
-0
lines changed

‎.gitignore

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Dependencies
2+
/node_modules
3+
4+
# Production
5+
/build
6+
7+
# Generated files
8+
.docusaurus
9+
.cache-loader
10+
11+
# Misc
12+
.DS_Store
13+
.env.local
14+
.env.development.local
15+
.env.test.local
16+
.env.production.local
17+
18+
npm-debug.log*
19+
yarn-debug.log*
20+
yarn-error.log*

‎README.md

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Core-Docs
2+
3+
This website is built using [Docusaurus 2](https://docusaurus.io/), a modern static website generator.
4+
5+
### Installation
6+
7+
```
8+
$ yarn
9+
```
10+
11+
### Local Development
12+
13+
```
14+
$ yarn start
15+
```
16+
17+
This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.
18+
19+
### Build
20+
21+
```
22+
$ yarn build
23+
```
24+
25+
This command generates static content into the `build` directory and can be served using any static contents hosting service.
26+
27+
### Deployment
28+
29+
```
30+
$ GIT_USER=<Your GitHub username> USE_SSH=true yarn deploy
31+
```
32+
33+
If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the `gh-pages` branch.

‎babel.config.js

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
module.exports = {
2+
presets: [require.resolve('@docusaurus/core/lib/babel/preset')],
3+
};

‎blog/2019-05-28-first-blog-post.md

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
slug: first-blog-post
3+
title: First Blog Post
4+
authors:
5+
name: Gao Wei
6+
title: Docusaurus Core Team
7+
url: https://github.com/wgao19
8+
image_url: https://github.com/wgao19.png
9+
tags: [hola, docusaurus]
10+
---
11+
12+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet

‎blog/2019-05-29-long-blog-post.md

+44
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
slug: long-blog-post
3+
title: Long Blog Post
4+
authors: endi
5+
tags: [hello, docusaurus]
6+
---
7+
8+
This is the summary of a very long blog post,
9+
10+
Use a `<!--` `truncate` `-->` comment to limit blog post size in the list view.
11+
12+
<!--truncate-->
13+
14+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
15+
16+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
17+
18+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
19+
20+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
21+
22+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
23+
24+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
25+
26+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
27+
28+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
29+
30+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
31+
32+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
33+
34+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
35+
36+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
37+
38+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
39+
40+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
41+
42+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
43+
44+
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet

‎blog/2021-08-01-mdx-blog-post.mdx

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
slug: mdx-blog-post
3+
title: MDX Blog Post
4+
authors: [slorber]
5+
tags: [docusaurus]
6+
---
7+
8+
Blog posts support [Docusaurus Markdown features](https://docusaurus.io/docs/markdown-features), such as [MDX](https://mdxjs.com/).
9+
10+
:::tip
11+
12+
Use the power of React to create interactive blog posts.
13+
14+
```js
15+
<button onClick={() => alert('button clicked!')}>Click me!</button>
16+
```
17+
18+
<button onClick={() => alert('button clicked!')}>Click me!</button>
19+
20+
:::

‎blog/2021-08-26-welcome/docusaurus-plushie-banner.jpeg

93.9 KB
Loading

‎blog/2021-08-26-welcome/index.md

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
---
2+
slug: welcome
3+
title: Welcome
4+
authors: [slorber, yangshun]
5+
tags: [facebook, hello, docusaurus]
6+
---
7+
8+
[Docusaurus blogging features](https://docusaurus.io/docs/blog) are powered by the [blog plugin](https://docusaurus.io/docs/api/plugins/@docusaurus/plugin-content-blog).
9+
10+
Simply add Markdown files (or folders) to the `blog` directory.
11+
12+
Regular blog authors can be added to `authors.yml`.
13+
14+
The blog post date can be extracted from filenames, such as:
15+
16+
- `2019-05-30-welcome.md`
17+
- `2019-05-30-welcome/index.md`
18+
19+
A blog post folder can be convenient to co-locate blog post images:
20+
21+
![Docusaurus Plushie](./docusaurus-plushie-banner.jpeg)
22+
23+
The blog supports tags as well!
24+
25+
**And if you don't want a blog**: just delete this directory, and use `blog: false` in your Docusaurus config.

‎blog/authors.yml

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
endi:
2+
name: Endilie Yacop Sucipto
3+
title: Maintainer of Docusaurus
4+
url: https://github.com/endiliey
5+
image_url: https://github.com/endiliey.png
6+
7+
yangshun:
8+
name: Yangshun Tay
9+
title: Front End Engineer @ Facebook
10+
url: https://github.com/yangshun
11+
image_url: https://github.com/yangshun.png
12+
13+
slorber:
14+
name: Sébastien Lorber
15+
title: Docusaurus maintainer
16+
url: https://sebastienlorber.com
17+
image_url: https://github.com/slorber.png

‎docs/CHPC_Guide/Data_Retirement_Policy.md

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
---
2+
id: Data_Retirement_Policy
3+
toplevel: true
4+
title: Data Retirement Policy
5+
---
6+
7+
New projects are guaranteed one year of data storage as part of recharge
8+
center billing. This includes a primary copy of all data and analyses on
9+
UCGD owned Lustre storage and a backup of raw primary data files (CRAM
10+
files) on UCGD owned CEPH storage. If projects require more than one
11+
year of storage and backup, then arrangements can be made through the
12+
recharge center for additional time. For projects that directly fund the
13+
purchase of UCGD storage hardware, data storage and backup is guaranteed
14+
for the warrantied life of the hardware (generally five years from the
15+
time of purchase).
16+
17+
## Notification
18+
19+
A general reminder of data retirement policies will be sent out to all
20+
UCGD collaborators annually at least 24 hours before major data moves.
21+
PI's will only be contacted individually when projects are at risk of
22+
being permanently deleted.
23+
24+
## Actual Implementation
25+
26+
It is important to note that there is no guarantee that data will be
27+
archived beyond what was billed for by the recharge center. However, in
28+
practice we try to maintain project data for \~3 years on Lustre and \~5
29+
years on CEPH archive space depending on the actual storage needs of
30+
UCGD.
31+
32+
At the end of each year, we move (but not delete) projects on Lustre
33+
that are three years old or older to a holding space. When Lustre
34+
approaches it's storage limits (\~90% full), all projects in the holding
35+
space are synchronized to CEPH archive storage and the Lustre copy is
36+
deleted. This means that for data older than three years, the CEPH
37+
archive copy may be the only copy (there is no backup). Additionally,
38+
when CEPH storage approaches it's limits (\~90% full), project owners
39+
will be individually notified that their data may soon be deleted. Data
40+
will then be moved from CEPH archive space into general scratch space at
41+
CHPC (this will immediately relieve our storage limitations on CEPH).
42+
CHPC automatically deletes files older than 60 days from scratch space.
43+
We will e-mail the PI at 60 days, 30 days, and 2 days to remind them
44+
that their data will permanently disappear without action on their part
45+
(i.e. copying data elsewhere, purchasing additional CEPH space, etc).
46+
47+
# Data Access
48+
49+
Project data at UCGD is kept on high performance Lustre storage and is
50+
backed up to CEPH object storage at the University of Utah Center for
51+
High Performance Computing (CHPC). Project files can be accessed over
52+
the web using Mosaic, or they can be accessed via a Linux terminal by
53+
logging into the Redwood HIPAA protected environment at CHPC. Data that
54+
is archived on CEPH object storage cannot be directly accessed by UCGD
55+
collaborators, but it can be restored to Lustre space on request.

‎docs/CHPC_Guide/Data_Structure_and_Policies.md

+108
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
---
2+
id: Data_Structure_and_Policies
3+
toplevel: true
4+
title: Data Structure and Policies
5+
---
6+
7+
## UCGD Directory Structure
8+
9+
- $COMMON is a non-lustre directory for shared applications and data
10+
11+
/uufs/chpc.utah.edu/common/PE/proj_UCGD/
12+
or
13+
/uufs/chpc.utah.edu/common/PE/proj_UCGDCollab/
14+
├──apps
15+
├──modulefiles
16+
├──data
17+
├──lustre_ACLs
18+
└──shell_scripts
19+
20+
apps - Installed applications and modules.<br/>
21+
modulefiles - Configuration files for application in apps (used by lmod).<br/>
22+
data - Shared datasets used by apps (i.e. GATK Bundles)<br/>
23+
lustre_ACLs - ACL scripts applied to lustre<br/>
24+
shell_scripts - Template scripts for jobs, prelaunch/postlaunch scripts, and environmental scripts (bashrc, bash_profile, etc).
25+
26+
27+
- Recharge Center Lustre
28+
29+
/scratch/ucgd/lustre/
30+
├──common
31+
│ └──data #synced with …/proj_UCGD/common/data
32+
├──Purgatory
33+
├──UCGD_Analysis
34+
├──UCGD_Processing
35+
├──UCGD_Datahub
36+
│ └──Repository
37+
└──work
38+
└──proj_UCGD
39+
└──u0123456
40+
41+
common/data - Fast IO sync of $COMMON/data<br/>
42+
Purgatory - Holding place for data waiting to be deleted (see if anyone complains before emptying the trash)<br/>
43+
UCGD_Analysis - Project directories for UCGD analysts<br/>
44+
UCGD_Processing - Working directory for data download and pipeline processing<br/>
45+
UCGD_Datahub - Final storage location of data. Served up by Gnomex.
46+
47+
48+
- UCGD legacy Lustre
49+
50+
/scratch/ucgd/lustre-work/
51+
├──marth
52+
│ └──u0123456
53+
├──quinlan
54+
│ └──u0123456
55+
└──yandell
56+
└──u0123456
57+
58+
marth, quinlan, yandell - Each lab in the UCGD gets a group work directory with a hard quota of 600TB and 200 million files (200TB and 50 million file user limit)<br/>
59+
temp - An unlimited work location for temporary files and data. Will be cleaned aggressively.
60+
61+
62+
- Isilon (general environment)
63+
64+
/uufs/chpc.utah.edu/common/home/
65+
├── marth-ucgdstor
66+
├── quinlan-ucgdstor
67+
└── yandell-ucgdstor
68+
69+
marth, quinlan, yandell - Each lab in the UCGD gets a group work directory with 100TB of storage.
70+
71+
72+
- UCGD Serial (general environment)
73+
74+
/scratch/ucgd/serial/
75+
76+
Space is currently being used by the UCGD-SRC group as a temporary storage space as we finalize other storage options. 175 TB of storage.
77+
78+
Contact Carson Holt if you have questions about this space.
79+
80+
81+
## CEPH Storage
82+
83+
The CEPH object storage is used to archive data in UCGD_Datahub under
84+
PolishedBams directories. AS long as result files are lossless, they
85+
function as an archive of the original Primary_Data files. You can
86+
access CEPH archives using rclone.
87+
88+
See documentation on how to setup and use Rclone:
89+
[CHPC Documentation on Rclone](https://www.chpc.utah.edu/documentation/software/rclone.php)
90+
91+
92+
## UCGD_Datahub Repository Policies
93+
94+
1. No softlinks in Primary_Data or Project_Setup unless it's to another
95+
Primary_Data or PolishedBams file directory.
96+
2. No softlinks in PolishedBams unless it's to another PolishedBams
97+
file.
98+
3. No Primary or Polished data should be inside ExternalData unless
99+
it's a softlink.
100+
4. PolishedBams and not Primary_Data is what gets backed up
101+
(Primary_Data is considered a temporary directory).
102+
5. 3 months after Billing, Primary_Data and Project_Setup get deleted
103+
after meeting the following criteria.
104+
1. All files in PolishedBams have been lossless validated
105+
2. All PolishedBam files have been backed up to CEPH storage
106+
3. Immutable bit is set on PolishedBams
107+
6. PolishedBams are always CRAM and not BAM.
108+
7. 3 years after billing project is retired and all data removed.

0 commit comments

Comments
 (0)