- 
-
-
-- -**Export image stack to a specific dataset** - -1. In the OMERO webclient, create a new dataset under your personal workshop project. **Make note of the dataset ID.** - -2. File > Open Samples > T1 Head (2.4M, 16-bit) - -3. File > Export > OMERO... - -**Question:** What happens when you repeatedly upload the same image to Omero? - ---- - -### Uploading Results with Fiji - -1. In the OMERO.web client, make note of a dataset ID where you would like to upload a new image and associated results file to. You make pick the dataset ID from the previous exercise. - -2. In Fiji, go to `File` > `Open Samples` > `Blobs (25K)` - -3. Let's apply some basic filtering and simple image segmentation to identify all cellular outlines in the image: - - a. Go to `Process` > `Filter` > `Median`. In the popup dialog enter a `Radius` of `3.0` and click `OK`. This will smooth out some of the image's intrinsic noise without degrading the object outlines. - - b. Go to `Image` > `Adjust Threshold`. In the popup dialog choose the `Default` thresholding algorithm, uncheck the `Dark Background` box and click `Apply`. The image should have been converted to a binary mask with white objects ona black background. -
- 
-
-
- c. Go to `Analyze` > `Set Measurements...`. In the popup dialog specify the parameters as shown in this screenshot. Click `OK`.
-
- d. Go to `Analyze` > `Analyze Particles` and set up the parameters as shown. Click `OK`.
- 
- 
-
- e. These steps should create a `Results` and a `Summary` table.
-
-3. Go to `File` > `Export` > `OMERO...`. Enter the dataset ID you chose under step 1 and click `OK`.
-
-{{< figure src="/courses/fiji-image-processing/fiji-omero/fiji-omero-blobs-export.png" >}}
-
-4. Click on the `Results` table window and go to `File` > `Save As`. Save the file as `Results.csv` on your computer. Repeat the same for the `Summary` table.
-
-5. **Upload the results files:** Go to the OMERO web client, open the Dataset folder that you had chosen for the image upload. On the right side of the webpage, expand the `Attachments` pane and click on the `+` icon. In the popup dialog, click `Choose File` and select the saved csv files. Click `Accept`.
-
-Now you have analyzed the image and uploaded the image as well as all results to the OMERO dataset.
-
-
----
-
-# Scripting
-
-Fiji provides convenient programming wrappers for the Fiji/ImageJ and OMERO functions that allow you to develop your scripts in a variety of programming languages:
-
-* ImageJ macro language: simple, slow, not very versatile
-* Jython: Python syntax with a few limitations, easy to learn, very versatile
-* BeanShell: Syntax similar to Java, versatile
-* Several others…
-
-Fiji provides a richer programming environment than ImageJ and it is recommended to use Fiji instead of ImageJ for any script development. Our [Fiji/ImageJ: Script development for Image Processing](/tutorials/fiji-scripting/) tutorial provides a more general introduction to this topic.
-
-### Example Scripts
-To follow along, you can download the Jython scripts presented in this tutorial through **[this link](/scripts/fiji/fiji-omero-scripts.zip)**.
-
-
-### The Script Editor {#script-editor-id}
-
-We'll be using the built-in **Script Editor** in Fiji to run our scripts. To start the script editor in Fiji go to menu `File` > `New` > `Script…`.
-
-{{< figure src="/courses/fiji-image-processing/fiji-omero/fiji-script-editor.png" >}}
-
-* The top pane provides the editor. Multiple scripts can be open at the same time and will show up as separate tabs.
-* The bottom pane shows output (e.g. produced by print statements) and any errors encountered during script execution.
-
-**Script editor menus:**
-
-+ **File:** Load, save scripts
-+ **Edit:** Similar to word processor functionality (Copy, Paste, etc.)
-+ **Language:** Choose language your script is written in with syntax highlighting
-+ **Templates:** Example scripts
-+ **Tools:** Access to source code of a class in the ImageJ/Fiji package
-+ **Tabs:** Navigation aid in script
-
-- -### The Macro Recorder {#macro-recorder-id} -The Macro Recorder logs all commands entered through the Fiji graphical user interface (GUI). It is useful to convert these GUI actions into script commands. - - -* In the Fiji menu, go to `Plugins` > `Macros…` > `Record`. -+ In the `Record` drop-down menu, select `BeanShell`. -+ Clicking the `Create` button copies the code to a new script in the [Script Editor](#script-editor-id). - -
- -### The Console Window {#console-id} -In the Fiji menu, go to `Window` > `Console`. - -+ The Console window shows the output and logging information produced by running plugins and scripts. - -
- ---- - -### Connecting to OMERO -In order to get full access to OMERO's programming interface, we will now use a more advanced approach to establish an authenticated connection with the OMERO database. We need instances of three classes: `LoginCredientials`, `SimpleLogger`, and `Gateway`. The central one for the configuration is `LoginCredentials` which has to be initialized with user specific credentials and database host information. - -Our script would not be very useful or secure if we had to hardcode these values. Fortunately we can use the [SciJava@Parameter](https://imagej.net/Script_Parameters) annotation to prompt the script user for the relevant information: -```python -#@ String (label="Omero User") username -#@ String (label="Omero Password", style="password") password -#@ String (label="Omero Server", value="omero.hpc.virginia.edu") server -#@ Integer (label="Omero Port", value=4064) port -``` - -These four lines at the top of our scripts are sufficient to create a dialog window that prompts the user for information that will be populated in the `username`, `password`, `host`, and `port` variables. With these variables in place we can now establish a connection to the OMERO database server. - -```python -cred = LoginCredentials() -if group_id != -1: - cred.setGroupID(group_id) -cred.getServer().setHostname(server) -cred.getServer().setPort(port) -cred.getUser().setUsername(username) -cred.getUser().setPassword(password) -simpleLogger = SimpleLogger() -gateway = Gateway(simpleLogger) -e = gateway.connect(cred) -``` - -The return value of the `connect` method is stored as a boolean value in the variable `e`. If `e==True`, the connection was established; if `e==False`, the connection failed. We can reuse this code block for most of our OMERO scripts. - -It is very important to close the connection to the database at the end of your script, like this: -```python -gateway.disconnect() -``` - ---- - -### Getting the Basic OMERO Dataset Info - -OMERO organizes users in groups. Each user can be a member of multiple groups. Images are organized in Projects and Datasets, or in Screens and Plates. The following script, `Omero_Info.py`, connects a user to a remote OMERO instance and shows a list of: - -* the groups that the user belongs to and the associated group ID. This ID is important when you want to access images stored for a particular group; -* the projects and datasets for a particular group (specified via unique group ID); -* and a list of images, organized by project and dataset, that the user has access to in a particular group. - -The following script, `Omero_info.py` establishes a connection to the OMERO database and outputs your OMERO group memberships, as well as a list of all of your projects, datasets, and images. The code contains separate functions to connect to the database, retrieve information from the database, and parse the data into a set of tables. If you're just starting with programming, you may find it helpful to work through our [Fiji Scripting](/tutorials/fiji-scripting/) and other tutorials on our [learning portal](/categories/). - -(Click on the black triangle next to **View** to take a look at the script.) - -
-View
-
-{{< highlight python "linenos=table,linenostart=1" >}}
-#@ String (label="Omero User") username
-#@ String (label="Omero Password", style="password") password
-#@ String (label="Omero Server", value="omero.hpc.virginia.edu") server
-#@ Integer (label="Omero Port", value=4064) port
-#@ Integer (label="Omero Group ID", min=-1, value=-1) group_id
-
-
-# Basic Java and ImageJ dependencies
-from ij.measure import ResultsTable
-from java.lang import Long
-from java.lang import String
-from java.util import ArrayList
-
-# Omero dependencies
-import omero
-from omero.gateway import Gateway
-from omero.gateway import LoginCredentials
-from omero.gateway import SecurityContext
-from omero.gateway.exception import DSAccessException
-from omero.gateway.exception import DSOutOfServiceException
-from omero.gateway.facility import BrowseFacility
-from omero.log import SimpleLogger
-
-
-def connect(group_id, username, password, server, port):
- """Omero Connect with credentials and simpleLogger"""
- cred = LoginCredentials()
- if group_id != -1:
- cred.setGroupID(group_id)
- cred.getServer().setHostname(server)
- cred.getServer().setPort(port)
- cred.getUser().setUsername(username)
- cred.getUser().setPassword(password)
- simpleLogger = SimpleLogger()
- gateway = Gateway(simpleLogger)
- e = gateway.connect(cred)
- return gateway
-
-
-def get_groups(gateway):
- """Retrieves the groups for the user"""
- currentGroupId = gateway.getLoggedInUser().getGroupId()
- ctx = SecurityContext(currentGroupId)
- adminService = gateway.getAdminService(ctx, True)
- uid = adminService.lookupExperimenter(username)
- groups = []
- for g in sorted(adminService.getMemberOfGroupIds(uid)):
- groupname = str(adminService.getGroup(g).getName().getValue())
- groups.append({
- 'Id': g,
- 'Name': groupname,
- })
- if g == currentGroupId:
- currentGroup = groupname
- return groups, currentGroup
-
-
-def get_projects_datasets(gateway):
- """Retrieves the projects and datasets for the user"""
- results = []
- proj_dict = {}
- ds_dict = {}
- groupid = gateway.getLoggedInUser().getGroupId()
- ctx = SecurityContext(groupid)
- containerService = gateway.getPojosService(ctx)
-
- # Read datasets in all projects
- projects = containerService.loadContainerHierarchy("Project", None, None) # allowed: 'Project", "Dataset", "Screen", "Plate"
- for p in projects: # omero.model.ProjectI
- p_id = p.getId().getValue()
- p_name = p.getName().getValue()
- proj_dict[p_id] = p_name
- for d in p.linkedDatasetList():
- ds_id = d.getId().getValue()
- ds_name = d.getName().getValue()
- results.append({
- 'Project Id': p_id,
- 'Project Name': p_name,
- 'Dataset Id': ds_id,
- 'Dataset Name': ds_name,
- 'Group Id': groupid,
- })
- ds_dict[ds_id] = ds_name
-
- # read datasets not linked to any project
- ds_in_proj = [p['Dataset Id'] for p in results]
- ds = containerService.loadContainerHierarchy("Dataset", None, None)
- for d in ds: # omero.model.ProjectI
- ds_id = d.getId().getValue()
- ds_name = d.getName().getValue()
- if ds_id not in ds_in_proj:
- ds_dict[ds_id] = ds_name
- results.append({
- 'Project Id': '--',
- 'Project Name': '--',
- 'Dataset Id': ds_id,
- 'Dataset Name': ds_name,
- 'Group Id': groupid,
- })
- return results, proj_dict, ds_dict
-
-
-def get_images(gateway, datasets, orphaned=True):
- """Return all image ids and image names for provided dataset ids"""
- browse = gateway.getFacility(BrowseFacility)
- experimenter = gateway.getLoggedInUser()
- ctx = SecurityContext(experimenter.getGroupId())
- images = []
- for dataset_id in datasets:
- ids = ArrayList(1)
- ids.add(Long(dataset_id))
- j = browse.getImagesForDatasets(ctx, ids).iterator()
- while j.hasNext():
- image = j.next()
- images.append({
- 'Image Id': String.valueOf(image.getId()),
- 'Image Name': image.getName(),
- 'Dataset Id': dataset_id,
- 'Dataset Name': datasets[dataset_id],
- })
- if orphaned:
- orphans = browse.getOrphanedImages(ctx, ctx.getExperimenter()) # need to pass user id (long)
- for image in orphans:
- images.append({
- 'Image Id': String.valueOf(image.getId()),
- 'Image Name': image.getName(),
- 'Dataset Id': -1,
- 'Dataset Name': '',
- })
- return images
-
-
-def show_as_table(title, data, order=[]):
- """Helper function to display group and data information as a ResultsTable"""
- table = ResultsTable()
- for d in data:
- table.incrementCounter()
- order = [k for k in order]
- order.extend([k for k in d.keys() if not d in order])
- for k in order:
- table.addValue(k, d[k])
- table.show(title)
-
-
-# Main code
-gateway = connect(group_id, username, password, server, port)
-
-groups, current_group = get_groups(gateway)
-show_as_table("My Groups", groups, order=['Id', 'Name'])
-
-all_data,_,datasets = get_projects_datasets(gateway)
-show_as_table("Projects and Datasets - Group: %s" % current_group, all_data, order=['Group Id', 'Dataset Id', 'Dataset Name', 'Project Name', 'Project Id'])
-
-gateway.disconnect()
-{{< /highlight >}}
-
-
-
----
-
-### Downloading Images from the OMERO database
-
-Let's try to download images from the database through a script. The OMERO plugin provides simple download (aka import to Fiji) functionality to achieve this.
-
-1. In the OMERO web interface, click on any image in the `Fiji Omero Workshop` project or your `xxx_workshop` project/dataset and note the Image ID displayed in the sidebar on the right side of the webpage. **Image retrieval relies on these unique image identifiers**.
-
-2. Go back to the Fiji Script Editor and open the `Omero_Image_Download.py` script.
-
-3. Run the script. A dialog window will open; enter these values:
-
- * **Omero User:** Your computing ID
- * **Omero Password:** Your OMERO Password
- * **Omero Server:** omero.hpc.virginia.edu
- * **Omero Port:** 4064
- * **Omero Group ID:** Enter `53` as ID for the `omero-demo` group, or use `-1` to use your default group
- * **Image ID:** Enter the ID for an image that is part of your `xxx_workshop` dataset, or use `11980` from the example files.
-
-The script consists of the these core blocks:
-
-* Lines 1-6 define user input to connect to OMERO.
-* Lines 12-20 define a `command` variable that specifies OMERO connection and image parameters.
-* Line 21 executes the OMERO importer plugin that retrieves the image.
-
-{{< highlight python "linenos=table,linenostart=1" >}}
-# @ String (label="Omero User") user
-# @ String (label="Omero Password", style="password") pwd
-# @ String (label="Omero Server", value="omero.hpc.virginia.edu") server
-# @ Integer (label="Omero Port", value=4064) port
-# @ Integer (label="Omero Group ID", min=-1, value=53) omero_group_id
-# @ Integer (label="Image ID", value=2014) image_id
-
-from ij import IJ
-from loci.plugins.in import ImporterOptions
-
-# Main code
-command="location=[OMERO] open=[omero:"
-command+="server=%s\n" % server
-command+="user=%s\n" % user
-command+="port=%s\n" % port
-command+="pass=%s\n" % pwd
-if omero_group_id > -1:
- command+="groupID=%s\n" % omero_group_id
-command+="iid=%s] " % image_id
-command+="windowless=true view=\'%s\' " % ImporterOptions.VIEW_HYPERSTACK
-IJ.runPlugIn("loci.plugins.LociImporter", command)
-{{< / highlight >}}
-
----
-
-### Uploading Images to the OMERO database
-
-Let's try to upload an image from Fiji to OMERO.
-
-1. Go back to Fiji and then to `File` > `Open Samples` > `Blobs`.
-
-2. Go back to the Fiji Script Editor and open the `Omero_Image_Upload.py file`.
-
- {{< highlight python "linenos=table,linenostart=1" >}}
- from ij import IJ
-
- imp = IJ.getImage()
- IJ.run(imp, "OMERO... ", "")
- {{< /highlight >}}
-
-3. Run the script. The **Export to OMERO** dialog window will open. Enter the following values:
-
- * **Server**: omero.hpc.virginia.edu
-
- * **Port:** 4064
-
- * **User:** Your computing ID
-
- * **Password:** Your OMERO password
-
- * **OMERO Dataset ID:** Enter the ___ID___ for the `xxx_workshop` dataset that you created in the OMERO web interface.
-
- * Check the **Upload new image** box. Leave the other boxes unchecked.
-
- Click `OK`.
-
- If you see an error, make sure you entered the correct password and Dataset ID. **Note: you have to use your own project/dataset.**
-
-4. Go to the OMERO website and refresh the page. Double-click on your `xxx_workshop` dataset icon to expand it. You should see the blobs.gif image.
-
----
-
-### Creating Key:Value Annotations
-
-{{< figure src="/courses/fiji-image-processing/fiji-omero/fiji-omero-keyvalue.png" >}}
-OMERO allows you to link other pieces of information to your Project, Dataset, Screen, Plate or Image objects. This additional information is displayed on the right side in the OMERO web client, labeled under the `General` tab as `Image Details`, `Tags`, `Key-Value Pairs`, `Tables`, `Attachments`, `Comments`, and `Ratings`. In addition, there is the `Acquisition` tab that provides metadata information that was automatically extracted from the image file headers during import.
-
-For the remainder of this workshop, we will focus on `Key-Value` pairs and `Attachments`. The key-value pairs are implemented as a dictionary (or HashMaps) that can be used to annotate individual images or whole datasets/project or plates/screens with additional information. Such information may include experimental conditions etc..
-
-Let's look at an example:
-
-1. In the OMERO webclient, expand the `Fiji Omero Workshop` project folder and the `Sample Data` dataset folder inside it.
-
-2. Click on the `blobs.gif` image. In the general tab, you will see three entries under the `Key-Value` group. (You may have to click on the triangle next to the label to expand the tab and see it).
-
-The values displayed are not particular meaningful, but they illustrate the concept. You can create and modify annotations interactively through the OMERO client. In addition, you can manipulate key-value pairs (as well as other annotation categories) through Fiji scripts.
-
-View Omero_Info.py script
-
-{{< highlight python "linenos=table,linenostart=1" >}}
-#@ String (label="Omero User") username
-#@ String (label="Omero Password", style="password") password
-#@ String (label="Omero Server", value="omero.hpc.virginia.edu") server
-#@ Integer (label="Omero Port", value=4064) port
-#@ Integer (label="Omero Group ID", min=-1, value=-1) group_id
-
-
-# Basic Java and ImageJ dependencies
-from ij.measure import ResultsTable
-from java.lang import Long
-from java.lang import String
-from java.util import ArrayList
-
-# Omero dependencies
-import omero
-from omero.gateway import Gateway
-from omero.gateway import LoginCredentials
-from omero.gateway import SecurityContext
-from omero.gateway.exception import DSAccessException
-from omero.gateway.exception import DSOutOfServiceException
-from omero.gateway.facility import BrowseFacility
-from omero.log import SimpleLogger
-
-
-def connect(group_id, username, password, server, port):
- """Omero Connect with credentials and simpleLogger"""
- cred = LoginCredentials()
- if group_id != -1:
- cred.setGroupID(group_id)
- cred.getServer().setHostname(server)
- cred.getServer().setPort(port)
- cred.getUser().setUsername(username)
- cred.getUser().setPassword(password)
- simpleLogger = SimpleLogger()
- gateway = Gateway(simpleLogger)
- e = gateway.connect(cred)
- return gateway
-
-
-def get_groups(gateway):
- """Retrieves the groups for the user"""
- currentGroupId = gateway.getLoggedInUser().getGroupId()
- ctx = SecurityContext(currentGroupId)
- adminService = gateway.getAdminService(ctx, True)
- uid = adminService.lookupExperimenter(username)
- groups = []
- for g in sorted(adminService.getMemberOfGroupIds(uid)):
- groupname = str(adminService.getGroup(g).getName().getValue())
- groups.append({
- 'Id': g,
- 'Name': groupname,
- })
- if g == currentGroupId:
- currentGroup = groupname
- return groups, currentGroup
-
-
-def get_projects_datasets(gateway):
- """Retrieves the projects and datasets for the user"""
- results = []
- proj_dict = {}
- ds_dict = {}
- groupid = gateway.getLoggedInUser().getGroupId()
- ctx = SecurityContext(groupid)
- containerService = gateway.getPojosService(ctx)
-
- # Read datasets in all projects
- projects = containerService.loadContainerHierarchy("Project", None, None) # allowed: 'Project", "Dataset", "Screen", "Plate"
- for p in projects: # omero.model.ProjectI
- p_id = p.getId().getValue()
- p_name = p.getName().getValue()
- proj_dict[p_id] = p_name
- for d in p.linkedDatasetList():
- ds_id = d.getId().getValue()
- ds_name = d.getName().getValue()
- results.append({
- 'Project Id': p_id,
- 'Project Name': p_name,
- 'Dataset Id': ds_id,
- 'Dataset Name': ds_name,
- 'Group Id': groupid,
- })
- ds_dict[ds_id] = ds_name
-
- # read datasets not linked to any project
- ds_in_proj = [p['Dataset Id'] for p in results]
- ds = containerService.loadContainerHierarchy("Dataset", None, None)
- for d in ds: # omero.model.ProjectI
- ds_id = d.getId().getValue()
- ds_name = d.getName().getValue()
- if ds_id not in ds_in_proj:
- ds_dict[ds_id] = ds_name
- results.append({
- 'Project Id': '--',
- 'Project Name': '--',
- 'Dataset Id': ds_id,
- 'Dataset Name': ds_name,
- 'Group Id': groupid,
- })
- return results, proj_dict, ds_dict
-
-
-def get_images(gateway, datasets, orphaned=True):
- """Return all image ids and image names for provided dataset ids"""
- browse = gateway.getFacility(BrowseFacility)
- experimenter = gateway.getLoggedInUser()
- ctx = SecurityContext(experimenter.getGroupId())
- images = []
- for dataset_id in datasets:
- ids = ArrayList(1)
- ids.add(Long(dataset_id))
- j = browse.getImagesForDatasets(ctx, ids).iterator()
- while j.hasNext():
- image = j.next()
- images.append({
- 'Image Id': String.valueOf(image.getId()),
- 'Image Name': image.getName(),
- 'Dataset Id': dataset_id,
- 'Dataset Name': datasets[dataset_id],
- })
- if orphaned:
- orphans = browse.getOrphanedImages(ctx, ctx.getExperimenter()) # need to pass user id (long)
- for image in orphans:
- images.append({
- 'Image Id': String.valueOf(image.getId()),
- 'Image Name': image.getName(),
- 'Dataset Id': -1,
- 'Dataset Name': '
-View
-
-{{< highlight python "linenos=table" >}}
-#@ String (label="Omero User") username
-#@ String (label="Omero Password", style="password") password
-#@ String (label="Omero Server", value="omero.hpc.virginia.edu") server
-#@ Integer (label="Omero Port", value=4064) port
-#@ Integer (label="Omero Group ID", min=-1, value=-1) group_id
-#@ String (label="Target", value="Image", choices = ["Image", "Dataset", "Project"]) target_type
-#@ Integer (label="Target ID", min=-1, value=-1) target_id
-
-# Basic Java and ImageJ dependencies
-from ij.measure import ResultsTable
-from java.lang import Double
-from java.util import ArrayList
-from ij import IJ
-from ij.plugin.frame import RoiManager
-from ij.measure import ResultsTable
-
-# Omero dependencies
-import omero
-from omero.log import SimpleLogger
-from omero.gateway import Gateway
-from omero.gateway import LoginCredentials
-from omero.gateway import SecurityContext
-from omero.gateway.model import ExperimenterData;
-
-from omero.gateway.facility import DataManagerFacility
-from omero.gateway.model import MapAnnotationData
-from omero.gateway.model import ProjectData
-from omero.gateway.model import DatasetData
-from omero.gateway.model import ImageData
-from omero.model import NamedValue
-from omero.model import ProjectDatasetLinkI
-from omero.model import ProjectI
-from omero.model import DatasetI
-from omero.model import ImageI
-
-
-def connect(group_id, username, password, server, port):
- """Omero Connect with credentials and simpleLogger"""
- cred = LoginCredentials()
- if group_id != -1:
- cred.setGroupID(group_id)
- cred.getServer().setHostname(server)
- cred.getServer().setPort(port)
- cred.getUser().setUsername(username)
- cred.getUser().setPassword(password)
- simpleLogger = SimpleLogger()
- gateway = Gateway(simpleLogger)
- e = gateway.connect(cred)
- return gateway
-
-
-def create_map_annotation(ctx, annotation, target_id, target_type="Project"):
- """Creates a map annotation, uploads it to Omero, and links it to target object"""
- # populate new MapAnnotationData object with dictionary
- result = ArrayList()
- for item in annotation:
- # add key:value pairs; both need to be strings
- result.add(NamedValue(str(item), str(annotation[item])))
- data = MapAnnotationData()
- data.setContent(result);
- data.setDescription("Demo Example");
-
- # use the following namespace if you want the annotation to be editable in the webclient and insight
- data.setNameSpace(MapAnnotationData.NS_CLIENT_CREATED);
- dm = gateway.getFacility(DataManagerFacility);
- target_obj = None
-
- # use the appropriate target DataObject and attach the MapAnnotationData object to it
- if target_type == "Project":
- target_obj = ProjectData(ProjectI(target_id, False))
- elif target_type == "Dataset":
- target_obj = DatasetData(DatasetI(target_id, False))
- elif target_type == "Image":
- target_obj = ImageData(ImageI(target_id, False))
- result = dm.attachAnnotation(ctx, data, target_obj)
- return result
-
-# Main code
-gateway = connect(group_id, username, password, server, port)
-currentGroupId = gateway.getLoggedInUser().getGroupId()
-ctx = SecurityContext(currentGroupId)
-
-# create a dictionary with key:value pairs
-annotation = {'Temperature': 25.3, 'Sample': 'control', 'Object count': 34}
-
-result = create_map_annotation(ctx, annotation, target_id, target_type=target_type)
-print "Annotation %s exported to Omero." % annotation
-
-gateway.disconnect()
-{{< /highlight >}}
-
-
-
----
-
-### Batch Processing and Results Tables for OMERO Datasets
-
-The previous examples demonstrated how to export local images to OMERO, or how to import OMERO images to a local workstation. As the final exercise, let's explore how an entire dataset comprised of many images can be downloaded from the remote OMERO instance, processed and analyzed locally, followed by an upload of the processed images and created results files back to the OMERO database.
-
-The example script, `Omero_Batch_Processing.py`, consists of five key functions:
-
-* **connect:** Establishes a connection to the OMERO server with specific user credentials. It returns an instance of the OMERO `Gateway` class that is used later to upload processed images to the same OMERO server instance.
-* **get_image_ids:** Gets a list of unique image IDs for a given dataset managed by the remote OMERO instance.
-* **open_image:** Downloads the image associated with an image ID and shows it in Fiji.
-* **process:** Applies a custom image processing routine to a given image. In this case a basic segmentation and counting of cells.
-* **create_map_annotation:** Uploads the cell count value to OMERO and links it to the original image.
-* **upload_csv_to_omero:** Converts an ImageJ ResultsTable into a csv file, uploads that csv file and links it ot the original image objects.
-* **upload_image:** Uploads an Image to a specific dataset managed by the remote OMERO instance.
-
-**Remember that the gateway connection needs to be closed at the end of the script**.
-
-{{< figure src="/courses/fiji-image-processing/fiji-omero/fiji-omero-batchprocessing.png" >}}
-
-To test this and see the process in action we will process a set of four images that has been deposited in the OMERO database. The setup is as follows:
-
-1. Go to the OMERO webclient and make note of your `Project ID`, or you cna create a new project if you prefer. Again you need the `ID`.
-
-2. In the Fiji Script Editor, open the `Omero_Batch_Processing.py` script and execute it.
-
-3. In the popup window, specify the parameters as follows:
-
- a. Replace the `mst3k` with your own credentials.
-
- b. **Omero Input Dataset ID:** `265`
-
- c. **Omero Output Dataset Name:** Enter name to your liking
-
- d. **Omero Output Project ID:** Enter the `ID` that you looked up as step 1. The script will create a new dataset (with the name you chose) and place all the processed images in there.
-
-4. Click `OK`. Watch the console output for logging messages.
-
-5. After the script ru has completed, go to the OMERO webclient and open the Project that you had chosen to collect the output. Look for the `binary segmentation masks`, the attached `Results.csv` files and the new `Key-Value Pairs` annotations for each image.
-
-View Omero_Map_Annotation.py script
-
-{{< highlight python "linenos=table" >}}
-#@ String (label="Omero User") username
-#@ String (label="Omero Password", style="password") password
-#@ String (label="Omero Server", value="omero.hpc.virginia.edu") server
-#@ Integer (label="Omero Port", value=4064) port
-#@ Integer (label="Omero Group ID", min=-1, value=-1) group_id
-#@ String (label="Target", value="Image", choices = ["Image", "Dataset", "Project"]) target_type
-#@ Integer (label="Target ID", min=-1, value=-1) target_id
-
-# Basic Java and ImageJ dependencies
-from ij.measure import ResultsTable
-from java.lang import Double
-from java.util import ArrayList
-from ij import IJ
-from ij.plugin.frame import RoiManager
-from ij.measure import ResultsTable
-
-# Omero dependencies
-import omero
-from omero.log import SimpleLogger
-from omero.gateway import Gateway
-from omero.gateway import LoginCredentials
-from omero.gateway import SecurityContext
-from omero.gateway.model import ExperimenterData;
-
-from omero.gateway.facility import DataManagerFacility
-from omero.gateway.model import MapAnnotationData
-from omero.gateway.model import ProjectData
-from omero.gateway.model import DatasetData
-from omero.gateway.model import ImageData
-from omero.model import NamedValue
-from omero.model import ProjectDatasetLinkI
-from omero.model import ProjectI
-from omero.model import DatasetI
-from omero.model import ImageI
-
-
-def connect(group_id, username, password, server, port):
- """Omero Connect with credentials and simpleLogger"""
- cred = LoginCredentials()
- if group_id != -1:
- cred.setGroupID(group_id)
- cred.getServer().setHostname(server)
- cred.getServer().setPort(port)
- cred.getUser().setUsername(username)
- cred.getUser().setPassword(password)
- simpleLogger = SimpleLogger()
- gateway = Gateway(simpleLogger)
- e = gateway.connect(cred)
- return gateway
-
-
-def create_map_annotation(ctx, annotation, target_id, target_type="Project"):
- """Creates a map annotation, uploads it to Omero, and links it to target object"""
- # populate new MapAnnotationData object with dictionary
- result = ArrayList()
- for item in annotation:
- # add key:value pairs; both need to be strings
- result.add(NamedValue(str(item), str(annotation[item])))
- data = MapAnnotationData()
- data.setContent(result);
- data.setDescription("Demo Example");
-
- # use the following namespace if you want the annotation to be editable in the webclient and insight
- data.setNameSpace(MapAnnotationData.NS_CLIENT_CREATED);
- dm = gateway.getFacility(DataManagerFacility);
- target_obj = None
-
- # use the appropriate target DataObject and attach the MapAnnotationData object to it
- if target_type == "Project":
- target_obj = ProjectData(ProjectI(target_id, False))
- elif target_type == "Dataset":
- target_obj = DatasetData(DatasetI(target_id, False))
- elif target_type == "Image":
- target_obj = ImageData(ImageI(target_id, False))
- result = dm.attachAnnotation(ctx, data, target_obj)
- return result
-
-# Main code
-gateway = connect(group_id, username, password, server, port)
-currentGroupId = gateway.getLoggedInUser().getGroupId()
-ctx = SecurityContext(currentGroupId)
-
-# create a dictionary with key:value pairs
-annotation = {'Temperature': 25.3, 'Sample': 'control', 'Object count': 34}
-
-result = create_map_annotation(ctx, annotation, target_id, target_type=target_type)
-print "Annotation %s exported to Omero." % annotation
-
-gateway.disconnect()
-{{< /highlight >}}
-
-
-View
-
-{{< highlight python "linenos=table" >}}
-#@ String (label="Omero User") username
-#@ String (label="Omero Password", style="password") password
-#@ String (label="Omero Server", value="omero.hpc.virginia.edu") server
-#@ Integer (label="Omero Port", value=4064) server_port
-#@ Integer (label="Omero Group ID", min=-1, value=-1) omero_group_id
-#@ Integer (label="Omero Input Dataset ID", min=-1, value=-1) dataset_id
-#@ String (label="Omero Output Dataset Name", value="Processed Images") target_ds_name
-#@ Integer (label="Omero Output Project ID", min=-1, value=-1) project_id
-
-
-import os
-import tempfile
-
-
-from java.lang import Long
-from java.lang import String
-from java.lang.Long import longValue
-from java.util import ArrayList
-from jarray import array
-from java.lang.reflect import Array
-import java
-from ij import IJ,ImagePlus
-from ij.measure import ResultsTable
-import loci.common
-from loci.formats.in import DefaultMetadataOptions
-from loci.formats.in import MetadataLevel
-from loci.plugins.in import ImporterOptions
-
-from loci.plugins.in import ImporterOptions
-
-# Omero Dependencies
-import omero
-from omero.rtypes import rstring
-from omero.gateway import Gateway
-from omero.gateway import LoginCredentials
-from omero.gateway import SecurityContext
-from omero.gateway.facility import BrowseFacility
-from omero.gateway.facility import DataManagerFacility
-from omero.log import Logger
-from omero.log import SimpleLogger
-from omero.gateway.model import MapAnnotationData
-from omero.gateway.model import ProjectData
-from omero.gateway.model import DatasetData
-from omero.gateway.model import ImageData
-from omero.gateway.model import FileAnnotationData
-from omero.model import FileAnnotationI
-from omero.model import OriginalFileI
-from omero.model import Pixels
-from omero.model import NamedValue
-from omero.model import ProjectDatasetLinkI
-from omero.model import ProjectI
-from omero.model import DatasetI
-from omero.model import ImageI
-from omero.model import ChecksumAlgorithmI
-from omero.model.enums import ChecksumAlgorithmSHA1160
-
-from ome.formats.importer import ImportConfig
-from ome.formats.importer import OMEROWrapper
-from ome.formats.importer import ImportLibrary
-from ome.formats.importer import ImportCandidates
-from ome.formats.importer.cli import ErrorHandler
-from ome.formats.importer.cli import LoggingImportMonitor
-from omero.rtypes import rlong
-
-
-def connect(group_id, username, password, host, port):
- '''Omero Connect with credentials and simpleLogger'''
- cred = LoginCredentials()
- if group_id != -1:
- cred.setGroupID(group_id)
- cred.getServer().setHostname(host)
- cred.getServer().setPort(port)
- cred.getUser().setUsername(username)
- cred.getUser().setPassword(password)
- simpleLogger = SimpleLogger()
- gateway = Gateway(simpleLogger)
- gateway.connect(cred)
- group_id = cred.getGroupID()
- return gateway
-
-
-def open_image(username, password, host, server_port, group_id, image_id):
- command="location=[OMERO] open=[omero:"
- command+="server=%s\n" % server
- command+="user=%s\n" % username
- command+="port=%s\n" % server_port
- command+="pass=%s\n" % password
- if group_id > -1:
- command+="groupID=%s\n" % group_id
- command+="iid=%s] " % image_id
- command+="windowless=true "
- command+="splitWindows=false "
- command+="color_mode=Default view=[%s] stack_order=Default" % ImporterOptions.VIEW_HYPERSTACK
- print "Opening image: id", image_id
- IJ.runPlugIn("loci.plugins.LociImporter", command)
- imp = IJ.getImage()
- return imp
-
-
-def upload_image(gateway, server, dataset_id, filepath):
- user = gateway.getLoggedInUser()
- ctx = SecurityContext(user.getGroupId())
- sessionKey = gateway.getSessionId(user)
-
- config = ImportConfig()
- config.email.set("")
- config.sendFiles.set('true')
- config.sendReport.set('false')
- config.contOnError.set('false')
- config.debug.set('false')
- config.hostname.set(server)
- config.sessionKey.set(sessionKey)
- config.targetClass.set("omero.model.Dataset")
- config.targetId.set(dataset_id)
- loci.common.DebugTools.enableLogging("DEBUG")
-
- store = config.createStore()
- reader = OMEROWrapper(config)
- library = ImportLibrary(store,reader)
- errorHandler = ErrorHandler(config)
-
- library.addObserver(LoggingImportMonitor())
- candidates = ImportCandidates (reader, filepath, errorHandler)
- reader.setMetadataOptions(DefaultMetadataOptions(MetadataLevel.ALL))
- success = library.importCandidates(config, candidates)
- return success
-
-
-def get_image_ids(gateway, dataset_id):
- """Return all image ids for given dataset"""
- browse = gateway.getFacility(BrowseFacility)
- experimenter = gateway.getLoggedInUser()
- ctx = SecurityContext(experimenter.getGroupId())
- images = []
- ids = ArrayList(1)
- ids.add(Long(dataset_id))
- j = browse.getImagesForDatasets(ctx, ids).iterator()
- while j.hasNext():
- image = j.next()
- images.append({
- 'Image Id': String.valueOf(image.getId()),
- 'Image Name': image.getName(),
- 'Dataset Id': dataset_id,
- })
- return images
-
-
-def create_map_annotation(ctx, annotation, target_id, target_type="Project"):
- # populate new MapAnnotationData object with dictionary
- result = ArrayList()
- for item in annotation:
- # add key:value pairs; both need to be strings
- result.add(NamedValue(str(item), str(annotation[item])))
- data = MapAnnotationData()
- data.setContent(result);
- data.setDescription("Demo Example");
-
- #Use the following namespace if you want the annotation to be editable in the webclient and insight
- data.setNameSpace(MapAnnotationData.NS_CLIENT_CREATED);
- dm = gateway.getFacility(DataManagerFacility);
- target_obj = None
-
- # use the appropriate target DataObject and attach the MapAnnotationData object to it
- if target_type == "Project":
- target_obj = ProjectData(ProjectI(target_id, False))
- elif target_type == "Dataset":
- target_obj = DatasetData(DatasetI(target_id, False))
- elif target_type == "Image":
- target_obj = ImageData(ImageI(Long(target_id), False))
- result = dm.attachAnnotation(ctx, data, target_obj);
- return result
-
-
-def upload_csv_to_omero(ctx, file, tablename, target_id, target_type="Project"):
- """Upload the CSV file and attach it to the specified object"""
- print file
- print file.name
- svc = gateway.getFacility(DataManagerFacility)
- file_size = os.path.getsize(file.name)
- original_file = OriginalFileI()
- original_file.setName(rstring(tablename))
- original_file.setPath(rstring(file.name))
- original_file.setSize(rlong(file_size))
-
- checksum_algorithm = ChecksumAlgorithmI()
- checksum_algorithm.setValue(rstring(ChecksumAlgorithmSHA1160.value))
- original_file.setHasher(checksum_algorithm)
- original_file.setMimetype(rstring("text/csv"))
- original_file = svc.saveAndReturnObject(ctx, original_file)
- store = gateway.getRawFileService(ctx)
-
- # Open file and read stream
- store.setFileId(original_file.getId().getValue())
- print original_file.getId().getValue()
- try:
- store.setFileId(original_file.getId().getValue())
- with open(file.name, 'rb') as stream:
- buf = 10000
- for pos in range(0, long(file_size), buf):
- block = None
- if file_size-pos < buf:
- block_size = file_size-pos
- else:
- block_size = buf
- stream.seek(pos)
- block = stream.read(block_size)
- store.write(block, pos, block_size)
-
- original_file = store.save()
- finally:
- store.close()
-
- # create the file annotation
- namespace = "training.demo"
- fa = FileAnnotationI()
- fa.setFile(original_file)
- fa.setNs(rstring(namespace))
-
- if target_type == "Project":
- target_obj = ProjectData(ProjectI(target_id, False))
- elif target_type == "Dataset":
- target_obj = DatasetData(DatasetI(target_id, False))
- elif target_type == "Image":
- target_obj = ImageData(ImageI(target_id, False))
-
- svc.attachAnnotation(ctx, FileAnnotationData(fa), target_obj)
-
-
-def process_file(imp):
- """Run segmentation"""
- print "Processing", imp.getTitle()
- title = imp.getTitle().split('.')[:-1]
- title = '.'.join(title) + "_mask.ome.tiff"
- nimp = ImagePlus(title, imp.getStack().getProcessor(1))
- IJ.run(nimp, "Median...", "radius=3")
- IJ.run(nimp, "Auto Local Threshold", "method=Bernsen radius=15 parameter_1=0 parameter_2=0 white")
- IJ.run(nimp, "Watershed", "")
-
- IJ.run("Set Measurements...", "area mean standard centroid decimal=3")
- IJ.run(nimp, "Analyze Particles...", "size=50-Infinity summary exclude clear add")
- rt = ResultsTable.getResultsTable()
- rt.show("Results")
-
- imp.close()
- return nimp, rt
-
-
-def create_new_dataset(ctx, project_id, ds_name, ):
- dataset_obj = omero.model.DatasetI()
- dataset_obj.setName(rstring(ds_name))
- dataset_obj = gateway.getUpdateService(ctx).saveAndReturnObject(dataset_obj)
- dataset_id = dataset_obj.getId().getValue()
-
- dm = gateway.getFacility(DataManagerFacility)
- link = ProjectDatasetLinkI();
- link.setChild(dataset_obj);
- link.setParent(ProjectI(project_id, False));
- r = dm.saveAndReturnObject(ctx, link);
- return dataset_id
-
-
-# Main code
-gateway = connect(omero_group_id, username, password, server, server_port)
-currentGroupId = gateway.getLoggedInUser().getGroupId()
-ctx = SecurityContext(currentGroupId)
-
-image_info = get_image_ids(gateway, dataset_id)
-tmp_dir = tempfile.gettempdir()
-print tmp_dir
-
-target_ds_id = create_new_dataset(ctx, project_id, target_ds_name)
-for info in image_info:
- imp = open_image(username, password, server, server_port, omero_group_id, info['Image Id'])
- processed_imp, rt = process_file(imp)
-
- # Save processed image locally in omero_tmp dir
- imgfile = tempfile.TemporaryFile(mode='wb', prefix='img_', suffix='.tiff', dir=tmp_dir)
-
- #filepath = os.path.join(tmp_dir, processed_imp.getTitle())
- options = "save=" + imgfile.name + " export compression=Uncompressed"
- IJ.run(processed_imp, "Bio-Formats Exporter", options)
- # ignore changes & close
- processed_imp.changes=False
- processed_imp.close()
-
- # uploaad image to a target dataset
- upload_image(gateway, server, target_ds_id, [imgfile.name])
-
- # create annotation
- annotation = {
- "Cell count": rt.size()
- }
- create_map_annotation(ctx, annotation, info['Image Id'], target_type="Image")
-
- # export ResultsTable to csv file and link to image object
- file = tempfile.TemporaryFile(mode='wb', prefix='results_', suffix='.csv', dir=tmp_dir)
- rt.saveAs(file.name)
- #upload_csv_to_omero(ctx, file, "Results.csv", long(info['Image Id']), "Image")
-
-# done, clean up
-shutil.rmtree(tmp_dir)
-gateway.disconnect()
-print "Done.\n"
-
-{{< /highlight >}}
-
-
----
-
-# Resources {#resources-id}
-
-**OMERO**
-
-* OMERO: https://www.openmicroscopy.org/omero/
-* OMERO User Support: https://help.openmicroscopy.org
-* UVA Research Computing: https://www.rc.virginia.edu
-* OMERO at the University of Virginia: https://www.rc.virginia.edu/userinfo/omero/overview/
-
-**Fiji Scripting**
-
-* RC tutorial [Fiji/ImageJ: Script development for Image Processing](/tutorials/fiji-scripting/)
-* Tutorial: https://syn.mrc-lmb.cam.ac.uk/acardona/fiji-tutorial/
-* Tips for Developers: https://imagej.net/Tips_for_developers
-* API: https://imagej.nih.gov/ij/developer/api/
-* SciJava: https://javadoc.scijava.org/Fiji/
-
-**General Python Programming**
-
-* [Introduction to Programming in Python](/courses/python_introduction/)
-* [Programming in Python for Scientists and Engineers](/courses/programming_python_scientists_engineers/)
diff --git a/content/courses/fiji-image-processing/fiji-omero/leaf.png b/content/courses/fiji-image-processing/fiji-omero/leaf.png
deleted file mode 100644
index 993480bd..00000000
Binary files a/content/courses/fiji-image-processing/fiji-omero/leaf.png and /dev/null differ
diff --git a/content/courses/fiji-image-processing/introduction/3d-brain-rendering.gif b/content/courses/fiji-image-processing/introduction/3d-brain-rendering.gif
deleted file mode 100644
index cb6b975e..00000000
Binary files a/content/courses/fiji-image-processing/introduction/3d-brain-rendering.gif and /dev/null differ
diff --git a/content/courses/fiji-image-processing/introduction/index.md b/content/courses/fiji-image-processing/introduction/index.md
deleted file mode 100644
index 1e4302a5..00000000
--- a/content/courses/fiji-image-processing/introduction/index.md
+++ /dev/null
@@ -1,1054 +0,0 @@
----
-title: Getting Started
-draft: false
-date: 2023-05-30T00:00:00-05:00
-type: book
-toc: true
-weight: 15
-url_dataset: data/intro-fiji-images.zip
----
-
-{{< figure src="/img/fiji.png" >}}
-
-{{% link-button href="/data/intro-fiji-images.zip" icon="fa fa-download" aria-label="Download Data" title="Download Data" %}}Data{{% /link-button %}}
-
-## Fiji Installation
-
-Fiji is available on Windows, Mac, and Linux platforms and can be downloaded directly
-from the Fiji website: [https://fiji.sc/](https://fiji.sc/). Fiji is a portable application,
-meaning it doesn't require an installer. To use Fiji, simply download and unzip the application,
-and then you are able to start it.
-
-* **macOS:** The Fiji application should be installed in the user’s home directory rather than the default Applications folder.
-* **Windows 7 & 10:** The Fiji application should be installed in the user’s home directory rather than the default C:\Program Files directory.
-* **Linux:** The Fiji application should be installed in a directory where the user has read, execution, and write permissions, e.g. the user’s home directory.
-
-## Introduction
-
-### Image Processing & Analysis
-
-* Image Filtering (removal of noise and artifacts)
-* Image Enhancement (highlight features of interest)
-* Image Segmentation (object recognition)
-* Quantitative Image Feature Description (object measurements)
-
-**Image Processing Applications**
-
-* Photography & cinematography
-* Digitization of non-digital documents -> text mining
-* Natural sciences, engineering
-* Forensics, e.g. fingerprint matching
-* Quality control, e.g. in semiconductor fabrication
-* Satellite surveillance, e.g. weather forecasting, terrain usage analysis
-* Autonomously operating machinery (cars, etc.)
-* Automatic cataloging of pictures on the web
-
-### The Digital Representation of an Image
-
-* 2D images are 2D arrays of numbers organized in rows and columns
-* The intersection of row and column is a pixel
-* Each pixel is represented as an intensity value (e.g. between 0-255)
-
-{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-1.png" >}}
-
-View Omero_Processing_Nuclei.py script
-
-{{< highlight python "linenos=table" >}}
-#@ String (label="Omero User") username
-#@ String (label="Omero Password", style="password") password
-#@ String (label="Omero Server", value="omero.hpc.virginia.edu") server
-#@ Integer (label="Omero Port", value=4064) server_port
-#@ Integer (label="Omero Group ID", min=-1, value=-1) omero_group_id
-#@ Integer (label="Omero Input Dataset ID", min=-1, value=-1) dataset_id
-#@ String (label="Omero Output Dataset Name", value="Processed Images") target_ds_name
-#@ Integer (label="Omero Output Project ID", min=-1, value=-1) project_id
-
-
-import os
-import tempfile
-
-
-from java.lang import Long
-from java.lang import String
-from java.lang.Long import longValue
-from java.util import ArrayList
-from jarray import array
-from java.lang.reflect import Array
-import java
-from ij import IJ,ImagePlus
-from ij.measure import ResultsTable
-import loci.common
-from loci.formats.in import DefaultMetadataOptions
-from loci.formats.in import MetadataLevel
-from loci.plugins.in import ImporterOptions
-
-from loci.plugins.in import ImporterOptions
-
-# Omero Dependencies
-import omero
-from omero.rtypes import rstring
-from omero.gateway import Gateway
-from omero.gateway import LoginCredentials
-from omero.gateway import SecurityContext
-from omero.gateway.facility import BrowseFacility
-from omero.gateway.facility import DataManagerFacility
-from omero.log import Logger
-from omero.log import SimpleLogger
-from omero.gateway.model import MapAnnotationData
-from omero.gateway.model import ProjectData
-from omero.gateway.model import DatasetData
-from omero.gateway.model import ImageData
-from omero.gateway.model import FileAnnotationData
-from omero.model import FileAnnotationI
-from omero.model import OriginalFileI
-from omero.model import Pixels
-from omero.model import NamedValue
-from omero.model import ProjectDatasetLinkI
-from omero.model import ProjectI
-from omero.model import DatasetI
-from omero.model import ImageI
-from omero.model import ChecksumAlgorithmI
-from omero.model.enums import ChecksumAlgorithmSHA1160
-
-from ome.formats.importer import ImportConfig
-from ome.formats.importer import OMEROWrapper
-from ome.formats.importer import ImportLibrary
-from ome.formats.importer import ImportCandidates
-from ome.formats.importer.cli import ErrorHandler
-from ome.formats.importer.cli import LoggingImportMonitor
-from omero.rtypes import rlong
-
-
-def connect(group_id, username, password, host, port):
- '''Omero Connect with credentials and simpleLogger'''
- cred = LoginCredentials()
- if group_id != -1:
- cred.setGroupID(group_id)
- cred.getServer().setHostname(host)
- cred.getServer().setPort(port)
- cred.getUser().setUsername(username)
- cred.getUser().setPassword(password)
- simpleLogger = SimpleLogger()
- gateway = Gateway(simpleLogger)
- gateway.connect(cred)
- group_id = cred.getGroupID()
- return gateway
-
-
-def open_image(username, password, host, server_port, group_id, image_id):
- command="location=[OMERO] open=[omero:"
- command+="server=%s\n" % server
- command+="user=%s\n" % username
- command+="port=%s\n" % server_port
- command+="pass=%s\n" % password
- if group_id > -1:
- command+="groupID=%s\n" % group_id
- command+="iid=%s] " % image_id
- command+="windowless=true "
- command+="splitWindows=false "
- command+="color_mode=Default view=[%s] stack_order=Default" % ImporterOptions.VIEW_HYPERSTACK
- print "Opening image: id", image_id
- IJ.runPlugIn("loci.plugins.LociImporter", command)
- imp = IJ.getImage()
- return imp
-
-
-def upload_image(gateway, server, dataset_id, filepath):
- user = gateway.getLoggedInUser()
- ctx = SecurityContext(user.getGroupId())
- sessionKey = gateway.getSessionId(user)
-
- config = ImportConfig()
- config.email.set("")
- config.sendFiles.set('true')
- config.sendReport.set('false')
- config.contOnError.set('false')
- config.debug.set('false')
- config.hostname.set(server)
- config.sessionKey.set(sessionKey)
- config.targetClass.set("omero.model.Dataset")
- config.targetId.set(dataset_id)
- loci.common.DebugTools.enableLogging("DEBUG")
-
- store = config.createStore()
- reader = OMEROWrapper(config)
- library = ImportLibrary(store,reader)
- errorHandler = ErrorHandler(config)
-
- library.addObserver(LoggingImportMonitor())
- candidates = ImportCandidates (reader, filepath, errorHandler)
- reader.setMetadataOptions(DefaultMetadataOptions(MetadataLevel.ALL))
- success = library.importCandidates(config, candidates)
- return success
-
-
-def get_image_ids(gateway, dataset_id):
- """Return all image ids for given dataset"""
- browse = gateway.getFacility(BrowseFacility)
- experimenter = gateway.getLoggedInUser()
- ctx = SecurityContext(experimenter.getGroupId())
- images = []
- ids = ArrayList(1)
- ids.add(Long(dataset_id))
- j = browse.getImagesForDatasets(ctx, ids).iterator()
- while j.hasNext():
- image = j.next()
- images.append({
- 'Image Id': String.valueOf(image.getId()),
- 'Image Name': image.getName(),
- 'Dataset Id': dataset_id,
- })
- return images
-
-
-def create_map_annotation(ctx, annotation, target_id, target_type="Project"):
- # populate new MapAnnotationData object with dictionary
- result = ArrayList()
- for item in annotation:
- # add key:value pairs; both need to be strings
- result.add(NamedValue(str(item), str(annotation[item])))
- data = MapAnnotationData()
- data.setContent(result);
- data.setDescription("Demo Example");
-
- #Use the following namespace if you want the annotation to be editable in the webclient and insight
- data.setNameSpace(MapAnnotationData.NS_CLIENT_CREATED);
- dm = gateway.getFacility(DataManagerFacility);
- target_obj = None
-
- # use the appropriate target DataObject and attach the MapAnnotationData object to it
- if target_type == "Project":
- target_obj = ProjectData(ProjectI(target_id, False))
- elif target_type == "Dataset":
- target_obj = DatasetData(DatasetI(target_id, False))
- elif target_type == "Image":
- target_obj = ImageData(ImageI(Long(target_id), False))
- result = dm.attachAnnotation(ctx, data, target_obj);
- return result
-
-
-def upload_csv_to_omero(ctx, file, tablename, target_id, target_type="Project"):
- """Upload the CSV file and attach it to the specified object"""
- print file
- print file.name
- svc = gateway.getFacility(DataManagerFacility)
- file_size = os.path.getsize(file.name)
- original_file = OriginalFileI()
- original_file.setName(rstring(tablename))
- original_file.setPath(rstring(file.name))
- original_file.setSize(rlong(file_size))
-
- checksum_algorithm = ChecksumAlgorithmI()
- checksum_algorithm.setValue(rstring(ChecksumAlgorithmSHA1160.value))
- original_file.setHasher(checksum_algorithm)
- original_file.setMimetype(rstring("text/csv"))
- original_file = svc.saveAndReturnObject(ctx, original_file)
- store = gateway.getRawFileService(ctx)
-
- # Open file and read stream
- store.setFileId(original_file.getId().getValue())
- print original_file.getId().getValue()
- try:
- store.setFileId(original_file.getId().getValue())
- with open(file.name, 'rb') as stream:
- buf = 10000
- for pos in range(0, long(file_size), buf):
- block = None
- if file_size-pos < buf:
- block_size = file_size-pos
- else:
- block_size = buf
- stream.seek(pos)
- block = stream.read(block_size)
- store.write(block, pos, block_size)
-
- original_file = store.save()
- finally:
- store.close()
-
- # create the file annotation
- namespace = "training.demo"
- fa = FileAnnotationI()
- fa.setFile(original_file)
- fa.setNs(rstring(namespace))
-
- if target_type == "Project":
- target_obj = ProjectData(ProjectI(target_id, False))
- elif target_type == "Dataset":
- target_obj = DatasetData(DatasetI(target_id, False))
- elif target_type == "Image":
- target_obj = ImageData(ImageI(target_id, False))
-
- svc.attachAnnotation(ctx, FileAnnotationData(fa), target_obj)
-
-
-def process_file(imp):
- """Run segmentation"""
- print "Processing", imp.getTitle()
- title = imp.getTitle().split('.')[:-1]
- title = '.'.join(title) + "_mask.ome.tiff"
- nimp = ImagePlus(title, imp.getStack().getProcessor(1))
- IJ.run(nimp, "Median...", "radius=3")
- IJ.run(nimp, "Auto Local Threshold", "method=Bernsen radius=15 parameter_1=0 parameter_2=0 white")
- IJ.run(nimp, "Watershed", "")
-
- IJ.run("Set Measurements...", "area mean standard centroid decimal=3")
- IJ.run(nimp, "Analyze Particles...", "size=50-Infinity summary exclude clear add")
- rt = ResultsTable.getResultsTable()
- rt.show("Results")
-
- imp.close()
- return nimp, rt
-
-
-def create_new_dataset(ctx, project_id, ds_name, ):
- dataset_obj = omero.model.DatasetI()
- dataset_obj.setName(rstring(ds_name))
- dataset_obj = gateway.getUpdateService(ctx).saveAndReturnObject(dataset_obj)
- dataset_id = dataset_obj.getId().getValue()
-
- dm = gateway.getFacility(DataManagerFacility)
- link = ProjectDatasetLinkI();
- link.setChild(dataset_obj);
- link.setParent(ProjectI(project_id, False));
- r = dm.saveAndReturnObject(ctx, link);
- return dataset_id
-
-
-# Main code
-gateway = connect(omero_group_id, username, password, server, server_port)
-currentGroupId = gateway.getLoggedInUser().getGroupId()
-ctx = SecurityContext(currentGroupId)
-
-image_info = get_image_ids(gateway, dataset_id)
-tmp_dir = tempfile.gettempdir()
-print tmp_dir
-
-target_ds_id = create_new_dataset(ctx, project_id, target_ds_name)
-for info in image_info:
- imp = open_image(username, password, server, server_port, omero_group_id, info['Image Id'])
- processed_imp, rt = process_file(imp)
-
- # Save processed image locally in omero_tmp dir
- imgfile = tempfile.TemporaryFile(mode='wb', prefix='img_', suffix='.tiff', dir=tmp_dir)
-
- #filepath = os.path.join(tmp_dir, processed_imp.getTitle())
- options = "save=" + imgfile.name + " export compression=Uncompressed"
- IJ.run(processed_imp, "Bio-Formats Exporter", options)
- # ignore changes & close
- processed_imp.changes=False
- processed_imp.close()
-
- # uploaad image to a target dataset
- upload_image(gateway, server, target_ds_id, [imgfile.name])
-
- # create annotation
- annotation = {
- "Cell count": rt.size()
- }
- create_map_annotation(ctx, annotation, info['Image Id'], target_type="Image")
-
- # export ResultsTable to csv file and link to image object
- file = tempfile.TemporaryFile(mode='wb', prefix='results_', suffix='.csv', dir=tmp_dir)
- rt.saveAs(file.name)
- #upload_csv_to_omero(ctx, file, "Results.csv", long(info['Image Id']), "Image")
-
-# done, clean up
-shutil.rmtree(tmp_dir)
-gateway.disconnect()
-print "Done.\n"
-
-{{< /highlight >}}
-- -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-2.png" >}} - -Fiji can handle image files in various file formats and pixel encodings, e.g. color images (RGB, HSB, etc.), grayscale images, and binary images. - -***Color images*** use 8-bit encoding for each channel (e.g. RGB: 256 intensity values for red, green, and blue). - -Pixels in ***grayscale images*** may be encoded as 8-bit integer (256 intensity levels), 16-bit integer (65536 intensity levels), or 32-bit float (decimal number) values. - -***Binary images*** are a special type of 8-bit grayscale images that only contain the pixel values 0 (black) or 255 (white). They are used for masking and segmentation of object areas of interest in an image. - -## Image Processing & Analysis - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-3.png" >}} - -Image processing and analysis procedures often share a common workflow as shown here. - -The original image (or raw data) that serves as input for the image processing pipeline may contain background noise that may need to be removed by applying specific image filters. - -The cleaned-up image may then be processed to enhance certain features, e.g. highlight object edges. - -The enhanced image can then be converted into binary image masks that define groups of pixels as objects. Based on these masks, objects of interest can be classified and analyzed for size, geometry, pixel intensities, etc.. - -# Basic Operations in ImageJ/Fiji - -* [Handling Image Files](#handling-image-files) -* [Working with Image Channels](#working-with-image-channels) -* [Image Montage and Cropping](#image-montage-and-cropping) -* [Image Scale Bars](#image-scale-bars) - - -## Handling of Image Files - -**Images and Image Stacks in Fiji** - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-4.png" >}} - -## Working with Image Channels - -Fiji has built-in tools to manipulate multichannel composite and RGB images. For this example, open the 5-dimensional (x, y, z, color, time) image stack. - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-5.png" >}} - -1 Go to `File` > `Open Samples` > `Mitosis (26MB, 5D stack)`. An image window should open as shown here. - -2 Open the Channels Tool dialog with `Image` > `Color` > `Channels Tool…`. - -3 In the Channels Tool dialog window, change from `Composite` to `Color`. Move the channel slider and note how the display updates based on the selected channel. - -4 Switch back to `Composite` mode. Uncheck selected channels and note how the channel overlay changes based on the selected images. - -5 Change to `Grayscale`. Although the color information has been converted to gray, the **pixel intensity values remain unchanged**. - -6 Change back to `Color`, Click `More >>`, select a new color. - -7 Change to `Composite` mode. - -
- - -**Splitting Channels** - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-6.png" >}} - -1) Go to `File` > `Open Samples` > `Mitosis (26MB, 5D stack)`. - -2) Go to `Image` > `Color` > `Split Channels`. - -Individual image channels are displayed in separate windows. This operation only works on multichannel image stacks or RGB images. - -
- - -**Merging Channels** - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-7.png" >}} - -Images displayed in separate windows can be ***merged*** into multichannel composite images. The input ***images need to be of the same type*** (either 8-bit or 16-bit pixel encoding). You can merge up to 7 different images into a single multichannel image. - -1) Go to `File` > `Open Samples` > `Fluorescent Cells`. - -2) Go to `Image` > `Color` > `Split Channels`. - -3) Go to `Image` > `Color` > `Merge Channels`. - -4) Set the green channel to `None` and leave the other default settings. Click `OK`. - -The resulting image should contain a merge of the red and blue channels without the green channel component. - -
- - -**Changing Image Color Assignment** - -The color representation in 8-bit and 16-bit images is defined by a color Lookup Table (LUT). - -For **single channel images**, you can change the LUT by running these commands: - -1) `Image` > `Lookup Tables`. - -For a **multichannel image**, you can change the LUT by running these commands: - -1) `Image` > `Color` > `Channels Tools`. - -2) In the **Channels Tools** window, switch to `Color` mode and select a channel of interest. - -3) Click `More >>`, select a new color. - -
- -### Image Montage and Cropping - -**Making an Image Montage** -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-8.png" >}} - -Images can be laid out in a grid-like fashion, aka a ***montage***. Given the regular arrangement of the images in a grid layout, all image tiles used for the montage have to be of the same x-y dimension and same image type (e.g. all 8-bit, 8-bit, 32-bit, or RGB). - -The easiest way to create a montage is to create an image stack that contains all the images for the individual image tiles and then use the Fiji Montage tool. Here is an example: - -1) Close All Image Windows: `File` > `Close All`. - -2) `File` > `Open Samples` > `Fluorescent Cells`. - -3) `Image` > `Color` > `Split Channels`. - -4) Close `Blue` Image. - -5) Select `Red` Image. - -6) Go to `Image` > `Lookup Tables` > `Grays`. - -7) Select `Green` Image. - -8) Go to `Image` > `Lookup Tables` > `Grays`. - -9) `Image` > `Color` > `Merge Channels` (green and red channel); select `Keep Sources`. - -10) For each of the three images: - a) Click on grayscale image window to make it the active window, - b) then go to `Image` > `Type` > `RGB Color`. - -11) Go to `Image` > `Stacks` > `Images to Stack`. - -12) Go to `Image` > `Stacks` > `Make Montage`.(Columns: 3, Scale factor: 1.0) - -
- -**Image Size: Cropping** - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-select-tools.png" >}} - -Images can be cropped around regions-of-interest (ROIs) that can be drawn with the selection tools in the Fiji Toolbar. - -1) Go to `File` > `Open Samples` and select any image. - -2) Click **Rectangular selection icon**, then draw rectangular box in image. - -3) Go to `Image` > `Crop`. This crops image to size of rectangular selection box. - -Note that the image will be cropped for all z-focal planes, all channels, and all timepoints. - - - - -
- - -### Image Scale Bars - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-10.png" >}} - -Many scientific instruments used for image acquisition store additional metadata in the image files that contain calibrations defining the size of a pixel (or 3D voxel) for that image or image stack in physical units (e.g. mm, nm, etc.). These values define the scale of an image. - -Taking a look at the image window right below the image title indicates whether the image scale (e.g. pixel/voxel size) has been set or not. - -* If the ***scale is not set***, the image dimensions are displayed as ***NNN x NNN pixels***, see top left panel. -* If the ***scale is set***, the image dimensions are displayed as ***NNN x NNN
- -**Defining an Image Scale** - -You can set or modify an image's scale following these steps: - -1) Go to `Analyze` > `Set Scale...`. - -2) In the top box of the dialog window, enter the number of pixels that correspond to a known physical distance. - -3) In the second box, enter the physical distance represented by the pixel distance. - -4) Enter the physical units label, e.g. ***mm***, ***nm***, etc.. The label ***um*** is interpreted as micrometer. -* If `Global` is checked, the new scale (or scale removal) operation is **applied to all open images**. - -5) Click `OK`. - -
- -**Adding a Scale Bar** - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-11.png" >}} - -For presentation purposes it is frequently desired to display a scale bar in the image. This can be done following these steps: - -1) Go to `Analyze` > `Tools` > `Scale Bar...`. - -2) Define positioning of the scale bar and its size. Check the `Overlay` box and click `OK`. - -2) Go to `Image` > `Overlay` > `Show Overlay/Hide Overlay` to toggle between showing and hiding the scale bar. - -* The **Overlay** ***will not affect the pixel values*** of the image region under the scale bar. It is **non-destructive**. - -* Saving the image in TIF format retains the non-destructive overlay. - -* Saving the image in JPG format burns the overlay into the image. - -
- -### Useful ImageJ/Fiji Keyboard Shortcuts - -| Shortcut | Operation | -| -------- | -------- | -| **Ctrl + I** | Show image file info | -| **Ctrl + Shift + D** | Duplicate current image | -| **Ctrl + Shift + X** | Crop image | -| **Ctrl + Shift + I** | Invert image | -| **Ctrl + A** | Select All (entire image) | -| **Ctrl + Shift + A** | Deselect marked region | -| **Ctrl + M** | Measure (in selected region) | -| **Ctrl + T** | Add selection to ROI Manager | - -\*On a Mac, use the Command key instead of the Ctrl key. - -
- -# Image Filtering & Enhancement - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-12.png" >}} - -
- -### Applying Filters: Noise Removal - -Image noise is an undesirable by-product of image capture that adds spurious and extraneous information. - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-13.png" >}} - -
- -### How Do Filters Work? - -Fiji provides several standard filters that can be applied to images. Here we are just exploring a few of them. - -1) Open image file **noisy_image.tif** from the [Tutorial Example Files](#example-files). - -2) Create three copies of the image file by pressing `Ctrl` + `Shift` + `D` (three times). - -3) Go to `Process` > `Filters` and experiment with the `Mean`, `Median`, `Gaussian`, and `Unsharp Mask` filters on the four images. - -4) Experiment with settings and click `Preview`. - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-14.png" >}} -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-15.png" >}} - -
- -**Comparing Average (Mean) and Median Filters** - -* **Mean**: Replaces pixel value with average value in kernel region. - -* **Median**: Replaces pixel value with median value in kernel region. - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-16.png" >}} - -
- -**Other Commonly Used Filters** - -| **Filter** | **Effect** | -| ------ | ------ | -| Convolve | Convolution with **custom** filter | -| Gaussian Blur | **Gaussian** filter, reasonable edge preservation | -| Median | **Median** of neighbors, non-linear, good for salt-and-pepper noise reduction | -| Mean | **Average** of neighbors, smoothing | -| Minimum | **Smallest** value of neighbors, smoothing | -| Maximum | **Largest** value of neighbors, smoothing | -| Variance | **Variance** of neighbors, indicator of image textures, highlight edges | - -
- -**Removing Periodic Noise (Advanced)** - -Periodic noise as shown by the banding pattern observed in this image cannot be removed with the kernel-based standard filters described above. However applying ***Fast-Fourier Transform (FFT)*** analysis can reveal the periodicity of such noise represented by the high intensity spots in the FFT image (top middle panel). By masking these hotspots (bottom middle panel) and applying the inverse FFT one can retrieve the original image without the periodic noise (right panel). - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-17.png" >}} - - -### Applying Filters: Edge Detection - -Edge detection is commonly used to define the outline of objects of interest. It is implemented by applying special convolution kernels. - -**Note that edge detection algorithms can be very sensitive to noise. So it is generally advised to remove noise with other filters first (e.g. median filter which preserves edges quite well) before applying the edge detection filter (see images below).** - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-18.png" >}} - - -# Image Segmentation - -* Image Segmentation is the process that groups individual image pixels that represent specific objects. - -* It often involves the application of a variety of image pixel filters. - -* It requires binary (black and white) image masks and object shape descriptors (*morphometric* parameters). - -
- -### Example: Blob Analysis - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-19.png" >}} - -To illustrate typical steps in image segmentation and analysis, we are going to process the ***blobs-shaded.tif*** image that is part of the [Tutorial Example Files] (#example-files). - -**Analysis Goals:** - -* Separate dark spots as individual objects from background. - -* Reject small objects (debris) by size exclusion. - -* Count total number of objects and compute average object size. - -* Measure size (2D surface area) of each object. - -
- -### Separating "Blobs" from Background - -In order to separate the dark spots from the background we will be using intensity thresholding. The idea is that all pixels with an intensity below (or equal) the threshold go in one bin, all other pixels go into another bin. Pixels in one bin are considered to belong to objects of interest while those in the other bin are considered background. This is the principal method to create binary images. - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-20.png" >}} - - -1) Open image file ***blobs-shaded.tif***. - -2) In menu, go to `Image` > `Adjust` > `Threshold…`. - -3) Click on image window, then click on `Auto` button in **Threshold** window. If you lost track of the Threshold window, goto `Windows` and select it from the list. - -4) In the Threshold dialog window, uncheck the `Dark Background` box. In the drop-down boxes select `Default` and `Red`. - -5) Move the sliders and observe the change of the red pattern overlaying the image. Click `Reset` to remove the overlay. ***Don’t click Apply yet!*** - -* What is the problem? - -
- -### Correction of Uneven Background Signal - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-21.png" >}} - -**A Background Correction Image** - -1) Go to `Image` > `Duplicate`, in the `Title` box enter: **Bg**. - -2) Go to `Process` > `Filters` > `Gaussian Blur…`. - -* Check the `Preview` box. - -* Try different `Sigma Radius` values: e.g. **1**, **20**, **100**. - -3) Finally, set `Sigma Radius` to **60**. - -4) Click `OK`. - -
- -**Performing the Background Correction** - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-22.png" >}} - -5) Go to `Process` > `Image Calculator…`. - -* From the `Image1` drop-down, select the ***blobs-shaded.tif*** image. - -* Select the ***Divide*** Operation. - -* From the `Image2` drop-down, select the ***Bg*** image. - -* Check the `Create new window` box. - -* Check the `32-bit (float) result` box. - -* Click `OK`. - -6) Click on the ***resulting*** window that contains the corrected image. - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-23.png" >}} - -7) Go to `Image` > `Rename...` and enter ***Corr*** as a new name for the corrected image. - -8) Go to `Image` > `Type` > `8-bit` to change the image from 32-bit float to 8-bit integer encoding. - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-24.png" >}} - -
- - -### Creating a Binary Object Mask - -9) Go to `Process` > `Binary` > `Options`. Check the `Black background` box and click `OK`. This controls the display mode of ***thresholded images*** and also the mode that the ***Particle Analyzer*** uses to identify objects (see below). **In our example objects of interest will be displayed white on black background.** - -10) Go to `Image` > `Adjust Threshold`. - -* In the Threshold dialog window, choose the ***Default*** algorithm. - -* Choose the ***Red*** overlay. - -* Deselect `Dark Background` and deselect `Stack histogram`. - -* Click on the ***Corr*** image window and click `Auto` in the Threshold window. Note the uniform detection of dark spots indicated by the red overlay. Note the bimodal distribution of the pixel intensity values in the thresholding histogram. Compare this thresholding to the one obtained for the original image ***blobs-shaded.tif***. - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-25.png" >}} -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-27.png" >}} -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-28.png" >}} - -
- -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-29.png" >}} - -11) In the Threshold dialog window, click `Apply`. - -* If you see a message like: **“Convert to 8-bit mask or set background pixels to NaN”**, it means that you are running the thresholding on a 32-bit image. Convert the ***Corr*** image to 8-bit first (see step 8). - -* If you see black objects on white background, check `Process` > `Binary` > `Options` (step 9). - -* Redo the thresholding on **“Corr”** image. - -12) Go to `Image` > `Rename` and enter ***Mask*** as a new image title. - -The result after steps 1-12 should be a binary mask image that shows white bobs on a black background. - -
- -### Splitting Joined Objects - -The previous steps produced a binary mask, but closer inspection reveals that some blob objects are fused. These appear as peanut-shaped objects. We will apply a watershed algorithm to separate these fused objects. - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-32.png" >}} - -13) Click on the ***Mask*** image window and go to `Process` > `Binary` > `Watershed`. - -14) Save the image obtained after the watershed operation as **blobs-watershed.tif**. - -* The watershed algorithm only works on binary images, e.g. the black and white “blob” mask. - -* It only works for objects that show significant opposing convex-to-concave curvature (neck regions). - - -
- - -### Analyze Particles - -Now we are ready to identify and extract information from each blob. - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-34.png" >}} -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-33.png" >}} - -15) Go to `Analyze` > `Analyze Particles...`. - -* In the **Analyze Particles** dialog, set up the analysis parameters as shown. - -* Click `OK`. - -The option **Show** `Outlines` leads to the creation of a new image that contains the outline and object ID (red numbers). - -The option `Add to Manager` sends the definition for each identified particle, i.e. dark blob, to the **Region-of-Interest Manager**, (ROI Manager for short). - -
- -### The ROI Manager - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-35.png" >}} - -Regions-of-interest (ROIs) define groups of pixels in an image. ROIs can have different shapes, e.g. line, rectangular, ellipsoid, polygonal, text, or irregular. An ROI can be used to - -* Measure pixel values in a specific image region. -* Change pixel values in a specific image region. -* Mark image regions in non-destructive overlays (without changing pixel values). - -When we use the Particle Analyzer with the `Add to Manager` option, the ROI for each blob is sent to the Roi Manager. - -**ROI Properties** - -Via the `Properties` button, the ROI Manager provides functions to change properties of selected ROIs, e.g. name, Z-position, outline color & width, and fill color. Multiple ROIs can be selected by holding the `Ctrl` (`command` on a Mac) or `Shift` keys while clicking on the ROIs. - -**ROI Import & Export** - -The ROI Manager can save selected ROIs to a file or open ROIs from a file. Typically, the ROI files have a .roi or .zip extension. The **Open** & **Save** functions are available after clicking the `More>>` button in the ROI Manager. - -* `Save`: Export All Selected ROIs to File - -* `Open`: Import ROIs from ROI file - -* This only affects **selected ROI**. - -* If **no ROIs are selected**, new properties are be applied to all ROIs in Manager. - -
- -# Image Object Measurements - -### Set Measurements - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-37.png" >}} - -Fiji can measure a variety of parameters for each region-of-interest, or an entire image. The specific measurements to be calculated by the Particle Analyzer can be set in the **Set Measurements** dialog. - -1) Go to `Analyze` > `Set Measurements...`. - -* Adjust settings as shown in screenshot. Click `OK`. - -* Settings affect future measurements, not existing ones. - -2) For quick measurements, draw a selection on the image and press `Ctrl` + `M`. - -* Measurements occur for **current selected region in the current image**. - -**Example:** - -1) Draw a selection on image - -2) Use `Ctrl` + `M`. - -3) Change settings in **Set Measurements** window. - -4) Use `Ctrl` + `M`. - -5) Select an ROI in the ROI Manager. Click `Measure` or use `Ctrl` + `M` to create measurements for the currently selected ROI. - -
- -### Analyze Particles with Custom Parameters - -The Particle Analyzer allows filtering of identified ROIs by size and circularity (range 0.0-1.0). A perfect circle has a circularity of 1.0. Let's filter the identified blobs and discard any ROI that has an area <150 square pixels. - -1) Go to `Edit` > `Selection` > `Select None` (`Ctrl` + `Shift` + `A`). - -2) Go to `Analyze` > `Analyze Particles...`. - -3) In the **Analyze Particles** dialog window, set up the analysis parameters as shown in the screenshot. - -4) Click `OK`. - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-38.png" >}} - -Based on our selection, the Particle Analyzer creates two results tables, titled **Results** and **Summary**. The type of measurements computed (e.g. **Mean**, **Median**, etc.) depend on the settings in the **Set Measurements** dialog. - -* Each line in the **Results** table reflects measurements for a single ROI. - -* The **Summary** table, as the name implies, provides summary statistics for all ROIs. - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-39.png" >}} - -
- -### Measuring Pixel Intensities - -Let's go back and modify the measurement selection. - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-40.png" >}} - -1) Go to `Analyze` > `Set Measurements...`. - -2) In the `Set Measurements` dialog window, set up the analysis parameters as shown in the screenshot. - -3) Click `OK`. - -4) Select all ROIs: - -* Go to `Window` > `ROI Manager`. -* Click on first ROI, **hold Shift** key, -* Scroll down, click last ROI in list. - -5) Measure ROIs in the **Mask** image - -* Click on black/white **mask** image -* In **ROI Manager** window, click `Measure`. This adds measurements for all selected ROIs to the **Results** table. -* Click on the **Results** window, go to `File` > `Rename`, and enter **Mask-Results** as a new name. -* If you don’t rename the window, the Results will be overwritten by new measurements. - -6) Measure ROIs in the **original** image - -* Click on **blobs-shaded.tif** image -* In **ROI Manager** window, click `Measure`. This adds measurements for all selected ROIs to the **Results** table. - -
- -Now we can compare the intensity measurements for the corresponding ROIs in the **Mask** and the **blobs-shaded.tif** images. - -* What do you observe? - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-41.png" >}} - -### Additional ROI Manager Functions - -* ROIs that are listed in the ROI Manager window can be saved to a file (single ROI or multiple ROIs). -* Conversely, ROIs can be loaded from file into the ROI Manager and (re-)applied to an image. -* Hand-drawn ROIs created with the ROI tools can be added to the ROI-Manager. -* ROIs can be renamed and colored. -* ROIs can be combined to create ROIs with complex shapes. - -Go to the [Exercises](#exercises) section for ideas to explore ROI-Manager functionality. - -
- -# 3D Image Reconstruction - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-42.png" >}} - -### Isolating the Brain Segmentation - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-43.png" >}} - -### Creating the Brain Segmentation - -1) Apply gaussian filter (0.5). - -2) Apply local thresholding (Grey Median, Otsu). - -3) Run particle analysis (1000-infinity particle size). - -4) Manual clean up segmentation mask as needed. - -5) Save brain segmentation mask as TIF image stack. - -These steps have been executed for you. The resulting brain segmentation mask is saved as **brain-mask.tif** in the [Tutorials Example Files](#example-files). - -### Creating the Composite 3D Rendering - -1) Go to `File` > `Open Samples` > `T1-Head`. - -2) Go to `Image` > `Type` > `8-bit`. - -3) Go to `File` > `Open` and select **brain-mask.tif** [see Tutorials Example Files](#example-files). - -4) Go to `Process` > `Image Calculator`: - -* Select **t1-head.tif** as Image 1. -* Select the **AND** operation. -* Select **brain-mask.tif** as Image 2. -* Check the `Create new window` box. - -This will create a new grayscale window with the brain proper that corresponds to the mask. - -5) Go to `Image` > `Color` > `Merge Channels`: t1-head.tif (gray), brain (magenta) - -6) Go to `Plugins` > `3D Viewer`. - -* Select `Display as Volume`. Leave the other default settings. - -8) In 3D Viewer, go to `Edit` > `Change Transparency`: Change skull transparency to 75%. - -9) Go to `View` > `Create 360 degree animation` to view a rotating animation of the rendered head volume. - -{{< figure src="/courses/fiji-image-processing/introduction/3d-brain-rendering.gif" >}} - -* Better: Export surfaces in Fiji to .stl files. - -* Process .stl surfaces for high quality rendering in other software, e.g. Paraview, Blender, etc.. - -
- -# Fiji/ImageJ Tutorials and Resources - -[https://imagej.nih.gov/ij/docs/examples/](https://imagej.nih.gov/ij/docs/examples/) - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-44.png" >}} - -
- -# Exercises - -### 1. Saving files in different file formats -Fiji supports many different file formats. - -1) Use **File** > **Open Samples** to open an image of your choosing - -2) Go to **File** > **Save As** and pick a file format to export the current image to. - -Note that some image file formats use compression algorithms that may reduce the pixel resolution or -dynamic range of an image’s intensity values. The TIF file format is very versatile because it can store -image data without resolution loss, it can handle multi-image files (i.e. image stacks or multi-page), -store image annotations as file internal metadata, and can be read by many programs. -File import/export functionality is further extended by the Bio-Formats plugin (Plugins > Bio-Formats). -See [https://docs.openmicroscopy.org/bio-formats/6.0.0/](https://docs.openmicroscopy.org/bio-formats/6.0.0/) - -
- -### 2. Image Channels - -**2.1 Splitting and Merging Channels** - -1) Go to **File** > **Open Samples** > **Mitosis** (26MB, 5D stack). - -2) Go to **Image** > **Color** > **Split Channels.** - -3) Go to **Image** > **Color** > **Merge**, merge the two channels but assign new colors to each channel. - -What happens if you assign one of the images to multiple different colors? - -
- -**2.2 Manipulating Color Lookup Tables** - -Color lookup tables (LUTs) can be used on 8-bit grayscale, 16-bit grayscale, and 8-bit Color images. Note that 8- -bit color is not the same as RGB color. LUTs for 16-bit grayscale will be resampled to 8-bit (256 colors). - -1) Go to **File** > **Open Samples** > **Clown** - -2) Go to **Image** > **Zoom** > **In**. - -3) Go to **Image** > **Color** > **Edit LUT**. - -Did this work? Look at the image information in the clown.jpg window. Alternatively, go to **Image > Show -information** and look at the **Bits per pixel** entry. What is the image format? - -4) Click on the clown.jpg image and convert the image to 8-bit color (**Image > Type > 8-bit Color**) Keep 256 -colors (the default). Click **OK**. - -5) Go to **Image > Color > Edit LUT**. The window shows the LUT color palette - -6) Double-click on a LUT tile and edit its RGB values to pure blue. Click **OK** and repeat this for other LUT -entries, change the color representation as you wish. - -What do you observe? - -
- -### 3. Image Montage - -For this exercise we are creating a montage of RGB images, each image representing the center focal plane at a -single timepoint within an x-y-color-t timelapse series. Experiment with creating different image tile sizes and -border widths. - -1) **Close** all images (**File** > **Close All**). - -2) Go to **File** > **Open Samples** > **Mitosis** (26MB, 5D stack). - -3) Go to **Image** > **Type** > **RGB Color**. Keep all slices and frames selected. - -4) Go to **Image** > **Duplicate**. Check the **Duplicate Hyperstack** box, enter **Slices (z): 3** (central focal plane), -enter **Frames (t): 1-51**. Click **OK**. - -5) Click on the newly created stack. It should contain images for a single focal plane over multiple -timepoints (frames). - -6) Save this image stack as a TIF file to your hard drive so you can create new montage layouts without -having to go through steps 1-5 again. - -7) Go to **Image** > **Stacks** > **Make Montage**. In the dialog box set the following values: - -* **Columns**: 10 -* **Rows**: 6 -* **Scale factor**: 0.25 -* **Border width**: 5 -* Check **Use foreground color** - -8) Click **OK**. - -Select the image stack created in step 5 (or reopen the stack saved in step 6) and create new montages with -modified montage settings. To change the foreground color for the border, double-click on the **Color Picker** in -the Fiji toolbar. - -{{< figure src="/courses/fiji-image-processing/introduction/intro-fiji-45.png" >}} - -
- -### 4. Image Scale Bars - -Scale bars can be added to any image. Before a scale bar can be drawn, the pixel size has to be defined. - -1) Go to **File** > **Open Sample** and select an image. - -2) Repeat step 1 and open a second image. - -3) Go to **Analyze** > **Set Scale**. - -4) If the image does not have a scale bar, the entries for **Distance in pixels** and **Known distance** will be set -to zero values and **Scale** label will show
- -### 5. Custom Image Filters - -To apply a filter to images, a convolution kernel has to be created. The built-in filters have standardized -convolution kernels. You can create your own convolution kernels to create custom filters. In this example we -create a simple edge detector, called the Sobel operator. - -1) Open the noisy.tif image. - -2) Apply a filter to remove the salt-and-pepper noise, e.g. with the median filter. - -3) Duplicate the image. - -4) Go to **Process** > **Filters** > **Convolve**. - -5) In the **textfield** of the dialog box, enter these numbers (**each number separated by a white space**). - --1 0 1 - --2 0 2 - --1 0 1 - -6) Check the **Preview** box. Experiment with changing the kernel values; make sure to keep the square -matrix format. - -7) Click **OK**. -What edges are highlighted in the image? Can you modify the kernel to detect the missing vertical edges? -8) Create another kernel with these values and apply it to the copied image obtained in step 3: - --1 -2 -1 - -0 0 0 - -1 2 1 - -What edges are being detected now? - -9) Go to **Process** > **Image Calculator**. - -10) Select the output image from step 7 as Image 1. - -11) Change **Operation** to **Max**. - -12) Select the output image from step 8 as Image 2. - -13) Check the **Create new window** box. - -14) Check the **32-bit (float) result** box. - -15) Click **OK**. - -The resulting image should show a combination of edges detected in steps 7 and 8. Can you create four kernels -to detect all vertical and horizontal edges? - -
- -### 6. Image Segmentation & Object Measurements - -For this exercise we assume that we already have a binary mask (thresholded) image, and we will use the -Particle Analyzer for image segmentation and object measurements. - -1) Open the **blobs-watershed.tif** image. This should be a binary image. - -2) Go to **Analyze** > **Set Measurements** and change the selection of measurement options. - -3) Go to **Analyze** > **Analyze Particles** and modify the **Size (pixel^2)**, **Circularity** (1.0 refers to perfect circular -shaped objects), and **Show** options. - -4) Click **OK**. - -5) Repeat steps 2-4 and experiment with different settings. - -Depending on your selection you will see a results table with data for each detected object (if **Display results** is -checked), a summary table for all results (if **Summarize** is checked), and the ROI-Manager (if **Add to Manager** is -checked) with a list of ROIs corresponding to the individual objects. - -6) Go to **File** > **Open Samples** > **Blobs**. - -7) Click on the **blobs-watershed.tif** image. - -8) Go to **Analyze** > **Set Measurements**, check the Min & max gray value and the Mean gray value boxes, -and change the Redirect to option to blobs.gif. This means that object detection is performed on the -blobs-watershed.tif binary mask, but pixel intensity measurements will be performed on the blobs.gif -image due to the redirect. - -9) Go to **Analyze** > **Analyze Particles**. Click **OK**. - -Repeat the measurement results with and without redirect and compare the changes in the results for **Min & -max gray value** and the **Mean gray value.** - -
- -### 7. ROI Manager - -The ROI Manager is very useful to handle multiple ROIs. It allows saving/loading ROIs and also to -combine simple shaped ROIs into more complex ones. - -1) Open an image of your choice. - -2) Use any of the selection tools from the Fiji toolbar to draw a selection area onto -the image. - -3) Go to **Selection** > **Edit** > **Add to Manager**. This will add the current selection to the ROI -Manager. - -4) Draw a new region of interest and go to **Selection** > **Edit** > **Add to Manager**. - -
- -**7.1 Saving ROIs to file** - -The ROI Manager should show two ROIs. - -5) In the ROI Manager, click on one of the ROIs. - -6) In the ROI Manager, click **More>>** and then **Save**. In the dialog box enter a name for the ROI file and -click Save. - -7) In the ROI Manager, hold the Shift key and select both ROIs. - -8) Click **More>>** and then **Save**. In the dialog box enter a name for the ROI file and click **Save**. This will save -multiple ROI definitions into a single file. - -**Notes:** - -* For a single ROI, the default ROI file extension is **.roi**. For multiple ROIs, the default file extension is **.zip.** - -* If none of the ROIs are selected, the operations in the ROI Manager are applied to all ROIs. - -
- -**7.2 Opening ROIs from file** -1) In the ROI Manager, select all ROIs and click **Delete**. - -2) Click on **More>>** and then click **Open**. Select the ROI file saved under exercise 7.1 step 8 (multiple ROIs). - -3) Click on one of the ROIs in the ROI Manager. This should create corresponding selection in the current -image. - -**Note:** You can open another image and reload saved ROIs to apply them to the new current image. - -
- -**7.3 Changing ROI display properties** -Each ROI has multiple properties, e.g. position, color, etc. - -1) In the ROI Manager, select the first ROI (or reload from file). - -2) Click on Properties. - -3) Change the **Name** to ROI\_1, the **Stroke color** to blue, and the **width** to 2. Click **OK**. - -4) Select the second ROI. - -5) Click on Properties. - -6) Change the **Name** to ROI\_2, the **Stroke color** to red, the **width** to 1, and the **Fill color** to cyan. Click **OK**. - -7) Click **OK**. - -If you have multiple ROIs selected OR none at all, the properties will be applied to all ROIs. - -
- -**7.4 Combining ROIs** - -Simple ROI shapes can be combined into more complex ones. - -1) In the current image, draw a new selection that partially overlaps the first ROI. - -2) Add the selection to the ROI Manager. It should show as the last one in the ROI Manager list. - -3) In the ROI Manager, hold the **Ctrl** (Windows) or **Command** (Mac) key and click on the first and -third ROI. - -4) A context menu should pop up; if not, click on **More >>**. In the ROI Manager’s context menu, -select **OR (Combine)**. The menu will disappear. In the ROI Manager, click on **Add [t]** button. A -new ROI will appear at the bottom of the ROI Manager list. - -5) Select the last ROI in the list (the one you just created). It encompasses the combined area of -the first and third ROI. - -6) Repeat steps 3 and 4 but choose different ROI combination methods, e.g. **And, XOR**. The **And** -operation produces a new ROI that contains only the overlapping region of the selected ROIs, -while the **XOR** operation creates a combined ROI that excludes any overlap between the -selected ROIs. diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-1.png b/content/courses/fiji-image-processing/introduction/intro-fiji-1.png deleted file mode 100644 index 9ded27e0..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-1.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-10.png b/content/courses/fiji-image-processing/introduction/intro-fiji-10.png deleted file mode 100644 index 877d6497..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-10.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-11.png b/content/courses/fiji-image-processing/introduction/intro-fiji-11.png deleted file mode 100644 index 9734b88b..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-11.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-12.png b/content/courses/fiji-image-processing/introduction/intro-fiji-12.png deleted file mode 100644 index 8973c457..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-12.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-13.png b/content/courses/fiji-image-processing/introduction/intro-fiji-13.png deleted file mode 100644 index 53bea540..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-13.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-14.png b/content/courses/fiji-image-processing/introduction/intro-fiji-14.png deleted file mode 100644 index 1b8db39c..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-14.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-15.png b/content/courses/fiji-image-processing/introduction/intro-fiji-15.png deleted file mode 100644 index 5297e0ab..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-15.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-16.png b/content/courses/fiji-image-processing/introduction/intro-fiji-16.png deleted file mode 100644 index e6cdadd8..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-16.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-17.png b/content/courses/fiji-image-processing/introduction/intro-fiji-17.png deleted file mode 100644 index cc0986bc..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-17.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-18.png b/content/courses/fiji-image-processing/introduction/intro-fiji-18.png deleted file mode 100644 index 8264585c..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-18.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-19.png b/content/courses/fiji-image-processing/introduction/intro-fiji-19.png deleted file mode 100644 index 25471549..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-19.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-2.png b/content/courses/fiji-image-processing/introduction/intro-fiji-2.png deleted file mode 100644 index b526ef85..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-2.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-20.png b/content/courses/fiji-image-processing/introduction/intro-fiji-20.png deleted file mode 100644 index dfda24e6..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-20.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-21.png b/content/courses/fiji-image-processing/introduction/intro-fiji-21.png deleted file mode 100644 index 734baeca..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-21.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-22.png b/content/courses/fiji-image-processing/introduction/intro-fiji-22.png deleted file mode 100644 index 0b782558..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-22.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-23.png b/content/courses/fiji-image-processing/introduction/intro-fiji-23.png deleted file mode 100644 index aaa5356a..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-23.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-24.png b/content/courses/fiji-image-processing/introduction/intro-fiji-24.png deleted file mode 100644 index 9e130f81..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-24.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-25.png b/content/courses/fiji-image-processing/introduction/intro-fiji-25.png deleted file mode 100644 index c39c384c..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-25.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-26.png b/content/courses/fiji-image-processing/introduction/intro-fiji-26.png deleted file mode 100644 index 2127b112..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-26.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-27.png b/content/courses/fiji-image-processing/introduction/intro-fiji-27.png deleted file mode 100644 index 4bbb076c..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-27.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-28.png b/content/courses/fiji-image-processing/introduction/intro-fiji-28.png deleted file mode 100644 index 7c9c964c..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-28.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-29.png b/content/courses/fiji-image-processing/introduction/intro-fiji-29.png deleted file mode 100644 index b7cf9533..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-29.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-3.png b/content/courses/fiji-image-processing/introduction/intro-fiji-3.png deleted file mode 100644 index ae218c6f..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-3.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-30.png b/content/courses/fiji-image-processing/introduction/intro-fiji-30.png deleted file mode 100644 index 429e5b53..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-30.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-31.png b/content/courses/fiji-image-processing/introduction/intro-fiji-31.png deleted file mode 100644 index 97cf69ea..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-31.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-32.png b/content/courses/fiji-image-processing/introduction/intro-fiji-32.png deleted file mode 100644 index 8460874e..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-32.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-33.png b/content/courses/fiji-image-processing/introduction/intro-fiji-33.png deleted file mode 100644 index e197768f..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-33.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-34.png b/content/courses/fiji-image-processing/introduction/intro-fiji-34.png deleted file mode 100644 index 714a8a03..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-34.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-35.png b/content/courses/fiji-image-processing/introduction/intro-fiji-35.png deleted file mode 100644 index eb2e3b62..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-35.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-36.png b/content/courses/fiji-image-processing/introduction/intro-fiji-36.png deleted file mode 100644 index 95e57bf5..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-36.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-37.png b/content/courses/fiji-image-processing/introduction/intro-fiji-37.png deleted file mode 100644 index 18472c48..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-37.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-38.png b/content/courses/fiji-image-processing/introduction/intro-fiji-38.png deleted file mode 100644 index 9221e150..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-38.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-39.png b/content/courses/fiji-image-processing/introduction/intro-fiji-39.png deleted file mode 100644 index cef7c079..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-39.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-4.png b/content/courses/fiji-image-processing/introduction/intro-fiji-4.png deleted file mode 100644 index 58788c0a..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-4.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-40.png b/content/courses/fiji-image-processing/introduction/intro-fiji-40.png deleted file mode 100644 index 66bb7197..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-40.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-41.png b/content/courses/fiji-image-processing/introduction/intro-fiji-41.png deleted file mode 100644 index ff0503fd..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-41.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-42.png b/content/courses/fiji-image-processing/introduction/intro-fiji-42.png deleted file mode 100644 index f0ce847e..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-42.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-43.png b/content/courses/fiji-image-processing/introduction/intro-fiji-43.png deleted file mode 100644 index 24d45952..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-43.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-44.png b/content/courses/fiji-image-processing/introduction/intro-fiji-44.png deleted file mode 100644 index a3ae499c..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-44.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-45.png b/content/courses/fiji-image-processing/introduction/intro-fiji-45.png deleted file mode 100644 index 89357835..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-45.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-5.png b/content/courses/fiji-image-processing/introduction/intro-fiji-5.png deleted file mode 100644 index 4e4ad567..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-5.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-6.png b/content/courses/fiji-image-processing/introduction/intro-fiji-6.png deleted file mode 100644 index 6ad20e2f..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-6.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-7.png b/content/courses/fiji-image-processing/introduction/intro-fiji-7.png deleted file mode 100644 index 01ebffb9..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-7.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-8.png b/content/courses/fiji-image-processing/introduction/intro-fiji-8.png deleted file mode 100644 index 7f316ec3..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-8.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-9.png b/content/courses/fiji-image-processing/introduction/intro-fiji-9.png deleted file mode 100644 index 7d2ae80c..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-9.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/introduction/intro-fiji-select-tools.png b/content/courses/fiji-image-processing/introduction/intro-fiji-select-tools.png deleted file mode 100644 index f779bb0d..00000000 Binary files a/content/courses/fiji-image-processing/introduction/intro-fiji-select-tools.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/scripting/fiji-greeting-script.png b/content/courses/fiji-image-processing/scripting/fiji-greeting-script.png deleted file mode 100644 index 701c151a..00000000 Binary files a/content/courses/fiji-image-processing/scripting/fiji-greeting-script.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/scripting/fiji-macro-recorder.png b/content/courses/fiji-image-processing/scripting/fiji-macro-recorder.png deleted file mode 100644 index c44d9c99..00000000 Binary files a/content/courses/fiji-image-processing/scripting/fiji-macro-recorder.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/scripting/fiji-newimages.png b/content/courses/fiji-image-processing/scripting/fiji-newimages.png deleted file mode 100644 index 432e1ab8..00000000 Binary files a/content/courses/fiji-image-processing/scripting/fiji-newimages.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/scripting/fiji-particle-analysis-results.png b/content/courses/fiji-image-processing/scripting/fiji-particle-analysis-results.png deleted file mode 100644 index 0110e62c..00000000 Binary files a/content/courses/fiji-image-processing/scripting/fiji-particle-analysis-results.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/scripting/fiji-particle-analysis.png b/content/courses/fiji-image-processing/scripting/fiji-particle-analysis.png deleted file mode 100644 index 6aa9d2d2..00000000 Binary files a/content/courses/fiji-image-processing/scripting/fiji-particle-analysis.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/scripting/fiji-script-editor.png b/content/courses/fiji-image-processing/scripting/fiji-script-editor.png deleted file mode 100644 index 39a2fdfe..00000000 Binary files a/content/courses/fiji-image-processing/scripting/fiji-script-editor.png and /dev/null differ diff --git a/content/courses/fiji-image-processing/scripting/index.md b/content/courses/fiji-image-processing/scripting/index.md deleted file mode 100644 index 5d49b0ad..00000000 --- a/content/courses/fiji-image-processing/scripting/index.md +++ /dev/null @@ -1,1193 +0,0 @@ ---- -title: "Script development for Image Processing with Fiji" -type: book -toc: true -date: 2021-10-14T00:00:00-05:00 -weight: 100 -url_code: /scripts/fiji/fiji-example-scripts.zip ---- - -{{< figure library="true" src="fiji.png" >}} - -{{% link-button href="/scripts/fiji/fiji-example-scripts.zip" icon="fa fa-download" title="Download Scripts" aria-label="Download Scripts" %}}Scripts{{% /link-button %}} - -This chapter is an introduction to the scripting interface of the [Fiji](https://fiji.sc) application, an open-source and enhanced version of the popular ImageJ program used for scientific image processing. Here you will learn how to write scripts for automated execution of image processing pipelines and batch processing of multiple image files in Fiji. - -Step-by-step instructions are provided for developing scripts in Jython, an implementation of the popular Python language for Java platforms. Example scripts are provided for the Jython and BeanShell languages. - - -## Introduction & Fiji Programming Tools {#intro-id} - -### Installation {#installation-id} -Fiji is a stand-alone application that can be downloaded from the [Fiji website](https://fiji.sc/#download). It is available for Mac OSX, Windows, and Linux platforms. - -+ **macOS** The Fiji application should be installed in the user's home directory rather than the default `Applications` folder. -+ **Windows 7 & 10:** The Fiji application should be installed in the user's home directory rather than the default `C:\Program Files` directory. -+ **Linux:** The Fiji application should be installed in a directory where the user has read, execution, and write permissions, e.g. the user's `home` directory. - -### Download the Example Scripts - -Go to [this tutorials landing page](/tutorials/fiji-scripting/) and click on the `Code` button. - -### The Application Programming Interface (API) {#api-id} -Fiji and ImageJ are written in Java. The application programming interface (API) defines the various Java packages, classes, methods, and constants that a programmer can use for development of custom Fiji & ImageJ scripts. The API is well documented on these public sites: - -+ ImageJ (https://imagej.nih.gov/ij/developer/api/) -+ Fiji (https://javadoc.scijava.org/Fiji/) - -Fiji provides convenient programming wrappers for these Java classes so you can write your scripts in: - -+ ImageJ macro language: simple, slow, not very versatile -+ Jython: Python syntax with a few limitations, easy to learn, very versatile -+ BeanShell: Syntax similar to Java, versatile -+ Several others… - -**Fiji provides a richer programming environment than ImageJ and it is recommended to use Fiji instead of ImageJ for any script development.** - -Step-by-step instructions in this tutorial are provided for developing scripts in Jython. The download section contains scripts written in the Jython and BeanShell languages. - -
- - -### The Script Editor {#script-editor-id} -To start the script editor in Fiji go to menu `File` > `New` > `Script…`. - -{{< figure src="fiji-script-editor.png" >}} - -* The top pane provides the editor. Multiple scripts can be open at the same time and will show up as separate tabs. -* The bottom pane shows output (e.g. produced by print statements) and any errors encountered during script execution. - -**Script editor menus:** - -+ **File:** Load, save scripts -+ **Edit:** Similar to word processor functionality (Copy, Paste, etc.) -+ **Language:** Choose language your script is written in with syntax highlighting -+ **Templates:** Example scripts -+ **Tools:** Access to source code of a class in the ImageJ/Fiji package -+ **Tabs:** Navigation aid in script - -
- -### The Macro Recorder {#macro-recorder-id} -In the Fiji menu, go to `Plugins` > `Macros…` > `Record`. - -{{< figure src="fiji-macro-recorder.png" >}} - -+ The Macro Recorder logs all commands entered through the Fiji graphical user interface (GUI). It is useful to convert these GUI actions into script commands. -+ In the `Record` drop-down menu, select `BeanShell`. -+ Clicking the `Create` button copies the code to a new script in the [Script Editor](#script-editor-id). -+ **Keep the Recorder window open throughout this tutorial.** - -### The Console Window {#console-id} -In the Fiji menu, go to `Window` > `Console`. - -+ The Console window shows the output and logging information produced by running plugins and scripts. - -
- - -## Jython {#jython-id} - -Jython is an implementation of the Python 2.7 programming language designed to run on the Java platform. - -+ Jython/Python are interpreter languages. The code is evaluated line-by-line when the script is run (executed). This is different from compiled languages. -+ Some Python packages (like numpy) are not available in Jython. -+ Jython scripts can use ImageJ/Fiji packages and classes via specific `import` statements. -+ In Python many problems can be solved with less code than in other languages. Still the code is easy to read. - -### Variables {#variables-id} -A variable is used to store values or references to other objects/functions in the computer's memory. - -* Values are assigned to variables in statements with an `=`. -* The identifier to the left of `=` defines the variable name, the expression to the right of the `=`defines the value to be assigned to the variable. -* Variable names have to start with a letter and cannot contain any special characters, e.g. `?`, `,`, etc.. Variable names are case-sensitive, so `my_name` is not the same as `My_name` or `my_Name`. - -Let's create five variables, `x`, `y`, `my_name`, `is_it_Monday`, and `my_list`. - -``` -x = 10 -y = 2.3 -my_name = "Santa Claus" -is_it_Monday = False -my_list = [2,3,5,7,11] -``` - -* `x` is assigned an integer value of `10`. -* `y` is a float variable, i.e. a number with decimals. -* `my_name` is assigned a sequence of characters, called a `string`. Strings are defined by enclosing single `'` or double `"` quotes. -* `is_it_Monday` is assigned a boolean value. A boolean can be either `True` or `False`. -* `my_list` contains a sequence of data elements, in this case integers. In Python/Jython, you can create lists containing elements with mixed types, even nested lists, e.g. `[1.0, 'abc', 1.2, "hello"]`. - -
- -### Conditionals: if Statements {#conditionals-id} - -Often different blocks of code need to be executed depending on a specific condition. This is achieved by using `if`, `elif`, `else` code blocks. - -``` -if
- -### Code Loops {#loops-id} - -Like most languages, Python and Jython offer loop constructs to repeatedly execute the same code block. - -**For loops**: - -`for` loops are executed a fixed number of iterations. It is possible to exit early but this must be added to the code. - -``` -for
- - -**While loops**: - -`while` loops do not have a predetermined number of iterations. They terminate when some condition is evaluated as `False`. - -``` -while
- -## Script Example {#jython-scripts-id} - -Jython scripts are simple text files that use specific syntax and structure. **Jython scripts (or scripts in any other programing language) should not be edited in word processing programs like Microsoft Word which add invisible formatting characters that mess with programming language syntax.** - -``` -from ij import IJ - -def print_info(imp): - """This function prints basic image info.""" - print "\nTitle:", imp.getTitle() # print title - - # output image dimensions - print "Width x Height:", imp.getWidth(), "x", imp.getHeight() - -# Main code block -imp = IJ.getImage() -print_info(imp) -``` - -* `import` statements define code that your script needs to work but that is organized in packages/modules outside your current script. An overview of the standard Fiji/ImageJ packages can be found [here][imagej-api]. -* Indentation levels define code blocks, e.g. conditionals, code loops, functions. -* The `def` keyword defines the beginning of a function, followed by the function's name and argument list. The content of the function is defined by the indented code block following the `def` function declaration. -* Empty lines are ignored by the interpreter but can improve readability of the code. -* The `#` defines a comment. All characters following a `#` on that line are ignored by the code interpreter. -* Text between a pair of triple quotes `"""` defines a special comment type, called doc string, that is used to annotate an entire program or function. - -
- -# Fiji Scripts in Jython {#fiji-id} - -## Images in Fiji: ImagePlus, ImageStack, ImageProcessor {#image-handling-id} - -+ An [ImagePlus](https://imagej.nih.gov/ij/developer/api/ij/ij/ImagePlus.html) object contains an [ImageProcessor](https://imagej.nih.gov/ij/developer/api/ij/ij/process/ImageProcessor.html) (2D image) or an [ImageStack](https://imagej.nih.gov/ij/developer/api/ij/ImageStack.html) (3D, 4D or 5D image). It is used to show the image in a window and also includes metadata (e.g. spatial calibration, the directory/file where it was read from, acquisition info, etc.). -+ The [ImageProcessor](https://imagej.nih.gov/ij/developer/api/ij/ij/process/ImageProcessor.html) contains the pixel data (8-bit, 16-bit, 32-bit float, or RGB) of the 2D image and some basic methods to manipulate it. -+ An [ImageStack](https://imagej.nih.gov/ij/developer/api/ij/ImageStack.html) is a group of [ImageProcessors](https://imagej.nih.gov/ij/developer/api/ij/ij/process/ImageProcessor.html) of the same type and size. ImageStack allows the implementation of multi-dimensional images, e.g. with `x`, `y`, `z`, `time`, and `channel` axes. - -
- -## Get Current Image {#current-image-id} -In **Fiji**, go to `File` > `Open Samples` > `Blobs`. - -``` -from ij import IJ # find the IJ class - -imp = IJ.getImage() # get active Image -print "\nTitle:", imp.getTitle() # output image title - -# output image dimensions -print "Width x Height:", imp.getWidth(), "x", imp.getHeight() -``` - -* The IJ.getImage() command retrieves the image from the currently active image window in Fiji. We assign the returned [ImagePlus](https://imagej.nih.gov/ij/developer/api/ij/ij/ImagePlus.html) object to the variable `imp`. -* The [ImagePlus](https://imagej.nih.gov/ij/developer/api/ij/ij/ImagePlus.html) class provides numerous methods that can be accessed via the `imp` object instance, e.g. `imp.getTitle()`, `imp.getWidth()`, `imp.getHeight()`, etc. -* The `print` statement outputs Strings and values in the [Script Editors](#script-editor-id) console pane. - -**Create the script:** - -In the **Fiji Script Editor**, go through these steps: - -1. Create a new file (`File` > `New`). -2. Go to `Language` and select `Python`. -3. Copy the script above into the script editor. -4. Go to `File` > `Save as...` and save the file as `Get_Image.py`. -5. Click the `Run` button. - -**Output:** -``` -Started Get_Image.py at Tue Jun 04 00:50:07 EDT 2019 - -Title: blobs.gif -Width x Height: 256 x 254 -``` - -
- -## Image Dimensions {#image-dimensions-id} - -Fiji and ImageJ can handle multidimensional images, e.g. volumetric images or time-lapse series, up to five dimensions. Each image ([ImagePlus](https://imagej.nih.gov/ij/developer/api/ij/ij/ImagePlus.html) object) has at least two dimensions, i.e. width (x-axis) and height (y-axis). In addition, ImagePlus object can have multiple color channels, z-focal planes (called slices), and/or timepoints (called frames). Lastly, images can have different pixel encodings, e.g. 8-bit, 16-bit, 32-bit, RGB that define an image's dynamic range (e.g. number of distinct intensity values per pixel) and color representation. - -The [ImagePlus](https://imagej.nih.gov/ij/developer/api/ij/ij/ImagePlus.html) class provides convenient methods to get this information. - -``` -from ij import IJ - -imp = IJ.getImage() -width = imp.getWidth() # get number of pixels along x-axis -height = imp.getHeight() # get number of pixels along y-axis -channel_no = imp.getNChannels() # get number of channels -slice_no = imp.getNSlices() # get number of slices -frame_no = imp.getNFrames() # get number of frames - -bitdepth = imp.getBitDepth() # bits per pixel -bpp = imp.getBytesPerPixel() # bytes per pixel (there are 8 bits in a byte) -``` - -
- -## Image Calibration {#image-calibration-id} - -Many image formats allow inclusion of image annotations (metadata) in addition to the pixel data. An important piece of information is the spatial pixel calibration, e.g. the definition of pixel size in real-world physical units. For example, a pixel may correspond to 200 nanometer in x-y dimension. - -Let's assume that we have a variable `imp` as reference to an [ImagePlus](https://imagej.nih.gov/ij/developer/api/ij/ij/ImagePlus.html) object. We can get and set an image's calibration with the following [ImagePlus](https://imagej.nih.gov/ij/developer/api/ij/ij/ImagePlus.html) class methods: - -**Get copy of image calibration** -``` -calibration = imp.getCalibration().copy() -``` - -**Set image calibration** -``` -new_imp.setCalibration(calibration) -``` - -
- -## Creating a New Image {#new-image-id} - -New images can be created by initializing an [ImagePlus](https://imagej.nih.gov/ij/developer/api/ij/ij/ImagePlus.html) object with an [ImageProcessor](https://imagej.nih.gov/ij/developer/api/ij/ij/process/ImageProcessor.html) object instance. The following ImageProcessor subclasses can be used, depending on bit-depth and color mode of the desired image. - -| ImageProcessor subclass | Mode | bits/pixel | -| --- | --- | --- | -| [ByteProcessor][byteprocessor] | grayscale | 8-bit int | -| [ShortProcessor][shortprocessor] | grayscale | 16-bit int | -| [FloatProcessor][floatprocessor] | grayscale |32-bit float | -| [ColorProcessor][colorprocessor] | color | 8-bit int per channel | - -
- -**Example** -``` -from ij.process import ByteProcessor, ShortProcessor, \ - FloatProcessor, ColorProcessor -from ij import ImagePlus - -width = 200 # in pixel -height = 100 # in pixel -bp = ByteProcessor(width, height) # create ImageProcessor -imp_bp = ImagePlus("New 8-bit image", bp) # create ImagePlus with specific title and ImageProcessor object -imp_bp.show() # show image window - -sp = ShortProcessor(width, height) -imp_sp = ImagePlus("New 16-bit image", sp) -imp_sp.show() - -fp = FloatProcessor(width, height) -imp_fp = ImagePlus("New 32-bit image", fp) -imp_fp.show() - -cp = ColorProcessor(width, height) -imp_cp = ImagePlus("New RGB image", cp) -imp_cp.show() -``` - -This script creates four new images, each with 200 x 100 pixels but with different pixel encodings. Take a look at each image window's header and note the differences. - -{{< figure src="fiji-newimages.png" >}} - -* Pixel encoding (8-bit, 16-bit, 32-bit, RGB). -* Memory footprint in kilobytes (20K, 39K, 78K, 78K). - -
- -## Duplicating an Image {#image-duplication-id} - -The [Duplicator](https://imagej.nih.gov/ij/developer/api/ij/ij/plugin/Duplicator.html) class provides a convenient way to create an exact copy of an existing [ImagePlus](https://imagej.nih.gov/ij/developer/api/ij/ij/ImagePlus.html) object. - -``` -from ij import IJ -from ij.plugin import Duplicator - -imp = IJ.getImage() -copied_imp = Duplicator().run(imp) -copied_imp.show() -``` - -Note that the duplicated image will be named `DUP_
- -## Opening and Saving Image Files {#image-io-id} - -**Open Images** - -Fiji can read many different image file formats. This versatility is now provided via the integrated [Bio-Formats plugin][bioformats]. - -| Class/Method | Behavior | -| --- | --- | -| IJ.open() | interactive dialog | -| IJ.openImage() | non-interactive, using default image display | -| loci.formats.ImageReader | non-interactive, configurable image display | - -
- -**Save Images** - -Fiji can also save image files in various common formats, including TIF, OME-TIF, BMP, PNG, JPEG, AVI, etc.). - -| Class/Method | Behavior | -| --- | --- | -| ij.io.FileSaver | interactive dialog | -| IJ.saveAs () | non-interactive, simple image writer (TIF, JPEG, GIF, PNG, etc.) | -| loci.formats.ImageWriter | non-interactive, advanced with many formats | - -
- -### Interactive Image Opening and Saving (with Dialog) -``` -from ij import IJ -from ij.io import FileSaver -``` - -### Open file (interactive dialog) -``` -imp = IJ.open() -imp = IJ.getImage() -imp.setTitle("copy.tif") -``` - -### Save file in original format (interactive dialog) -``` -fs = FileSaver(imp) -fs.save() # could also use fs.saveAsTiff(), fs.saveAsPng(), etc. -``` - -
- -### Non-interactive Image Opening and Saving (without Dialog)** -``` -from ij import IJ -import os -from os import path - -# Open file from URL or storage device (non-interactive) -filesource = "http://imagej.nih.gov/ij/images/blobs.gif" -imp = IJ.openImage(filesource) -imp.show() - -# Create new output directories and filename -homedir = path.expanduser("~") # home directory -outputdir = path.join(homedir, "workshop", "images") # full dir path -print outputdir -if not path.isdir(outputdir): - os.makedirs(outputdir) - -# Save file as .tif (non-interactive) -outputfile = os.path.join(outputdir, "blobs-copy.tif") -IJ.saveAs(imp, "tiff", outputfile) -print "Saved file:", outputfile -``` - -Check the output in the console window. The image should have been saved to the `workshop/images` subdirectory in your `home` directory. -**If you cannot find your saved file, try this:** - -* Click on the `blobs-copy.tif` image window, -* Go to `Image` > `Show Info…`. -* View path entry in `Info for blobs-copy.tif` window. - -**Note that file paths use `\` on Windows but `/` on Linux and OSX platforms. Avoid hardcoding the `\` or `/` and use [os.path][ospath] functions to create platform appropriate path names.** - -
- -## IJ.run {#ij-run-id} - -The `IJ.run` command can be used to execute many of the commands available through the Fiji graphical user interface. - -Basic syntax: `IJ.run(image, command, option)` - -* **image** : reference to [ImagePlus](https://imagej.nih.gov/ij/developer/api/ij/ij/ImagePlus.html) object -* **command** : String -* **option** : String, in some cases just `""` (empty String) - -The [Macro Recorder](#macro-recorder-id) is an excellent tool to convert a function accessible via the Fiji menu into an `IJ.run` statement. Let's try it. - -**Record** - -1. Ensure that the [Macro Recorder](#macro-recorder-id) window is open. If not, go to `Plugins` > `Macro` > `Recorder`. -2. In the Recorder window’s `Record` drop-down menu, select `BeanShell`. -3. In the Recorder window select and remove any code. -4. In Fiji, go to `File` > `Open Samples` > `Boats` -5. Go to `Process` > `Filters` > `Gaussian Blur`, a dialog will pop up: - * Enter a `Sigma value` of `5`. - * Click `Ok`. -6. Go to `File` > `Save As…` > `Tiff`. -7. Browse to your `workshop/images` directory (if it does not exist yet, create it), use `blurry_boats.tif` as file name and click `Save`. -8. In the Recorder window, click `Create`. This should bring up the [Script Editor](#script-editor-id) window with a new script file containing the code copied from the Recorder. **If you see more than three lines of code, remove everything that starts before `imp = IJ.openImage(….`.** - -**Convert to Jython script** - -1. In the [Script Editor](#script-editor-id) , go to `Language`, and select `Python`. -2. Insert an `from ij import IJ` statement at the top of the script. -3. Remove semicolons (`;`) at the end of the last three lines so your script looks like this (ignore the file path; yours will be different): -``` -from ij import IJ - -imp = IJ.openImage("http://imagej.nih.gov/ij/images/boats.gif") -IJ.run(imp, "Gaussian Blur...", "sigma=5") -IJ.saveAs(imp, "Tiff", "/Users/mst3k/blurry_boats.tif") -``` -4. Save your script as `IJ_Run.py`. Close all image windows. -5. Run the script. - -Check your `workshop/images` directory for the `blurry_boats.tif` file. - -
- -## Working with ImageStack {#imagestack-id} - -In Fiji, go to `File` > `Open Samples` > `Mitosis`. This image has the following dimensions: - -* 2 Channels -* 5 Focal Planes (Slices, Z) -* 51 Timepoints (Frames) - -The multidimensional image (represented by an [ImagePlus](https://imagej.nih.gov/ij/developer/api/ij/ij/ImagePlus.html) object `imp`) contains an [ImageStack](https://imagej.nih.gov/ij/developer/api/ij/ij/ImageStack.html) object to manage the different image planes. An ImageStack contains a group of [ImageProcessors](https://imagej.nih.gov/ij/developer/api/ij/ij/process/ImageProcessor.html) of the same type and size. - -The ImageStack object of an ImagePlus object (`imp`) can be retrieved like so: -``` -stack = imp.getStack() -``` - -The following code provides a reference to the [ImageProcessor](https://imagej.nih.gov/ij/developer/api/ij/ij/process/ImageProcessor.html) for a particular channel, slice, and frame in the `imp` [ImagePlus](https://imagej.nih.gov/ij/developer/api/ij/ij/ImagePlus.html) object: -``` -index = imp.getStackIndex(channel_no, slice_no, frame_no) -stack = imp.getStack() -ip = stack.getProcessor(index) -``` - -Conversely, we can replace an ImageProcessor in an ImageStack (referenced by the `stack` variable) like so: -``` -stack.setProcessor(new_ip, index) # provide new ImageProcessor object, and stack position to place it -``` - -**Important:** The new ImageProcessor object has to be of the same type, width, and height as all the other ImageProcessors in the stack. - -
- -## Regions-of-Interest (ROIs) {#rois-id} - -[ROIs](https://imagej.nih.gov/ij/developer/api/ij/ij/gui/Roi.html) define groups of pixels in an image. An ROI can be used to: - -* Measure pixel values -* Change pixel values -* Mark image regions in non-destructive overlays (without changing pixel values) - -ROI Types (classes): - -* Roi -* Line -* OvalRoi -* PolygonRoi -* ShapeRoi -* TextRoi -* ImageRoi - -ROIs can be managed directly or through the [RoiManager](#roi-manager-id) class. - -### ROI: Cropping an Image {#roi-crop-id} - -In Fiji, go to `File` > `Open` > `Samples` > `Clown` - -``` -from ij.gui import Roi -from ij import IJ -from ij.plugin import Duplicator - -# get current image -imp = IJ.getImage() - -# create rectangular ROI centered in image -rel_size = 0.5 # 0.0 < rel_size < 1.0 -width = int(rel_size * imp.getWidth()) -height = int(rel_size * imp.getHeight()) -top_left_x = int(0.5 * (imp.getWidth() - width)) -top_left_y = int(0.5 * (imp.getHeight()- height)) -roi = Roi(top_left_x, top_left_y, width, height) - -# copy image, set ROI on copied image, crop, set title and show image -cropped_imp = Duplicator().run(imp) -cropped_imp.setRoi(roi) -IJ.run(cropped_imp, "Crop", "") # modifies passed ImagePlus object! -cropped_imp.setTitle('Cropped') -cropped_imp.show() -``` - -Run the script in the [Script Editor](#script-editor-id). - -### Fill ROI with a Given Value {#roi-fill-id} - -``` -from ij import IJ, ImagePlus -from ij.process import FloatProcessor -from array import zeros -from random import random -from ij.gui import Roi, OvalRoi - -# Create a new ImagePlus filled with noise -width = 1024 -height = 1024 -pixels = zeros('f', width * height) -for i in xrange(len(pixels)): - pixels[i] = random() -fp = FloatProcessor(width, height, pixels) -imp = ImagePlus("Random", fp) - -# Fill a rectangular region of interest with a value of 2.0 -roi = Roi(40, 100, 400, 300) -fp.setRoi(roi) -fp.setValue(2.0) -fp.fill() - -# Fill a oval region of interest with a value of -2.0 -oroi = OvalRoi(500, 300, 150, 550) -fp.setRoi(oroi) -fp.setValue(-2.0) -fp.fill(oroi.getMask()) # Attention! Required for non-rectangular ROIs - -imp.show() -``` - -### Changing ROI Properties {#roi-properties-id} - -The appearance of an [ROI](https://imagej.nih.gov/ij/developer/api/ij/ij/gui/Roi.html) object (`roi`) can be changed. - -**Border [Color][color]** -``` -from java.awt import Color -roi.setStrokeColor(Color.RED) -``` - -**Border Width** -``` -roi.setStrokeWidth(2.0) -``` - -**Fill [Color][color]** -``` -from java.awt import Color -roi.setFillColor(Color.YELLOW) -``` - -### Set ROI Position {#roi-set-id} - -The positioning of an ROI object (`roi`) can be set for a specific channel, slice, frame in a given image (`imp`) with multi-dimensional [ImageStack](https://imagej.nih.gov/ij/developer/api/ij/ij/ImageStack.html). - -``` -index = imp.getStackIndex(channelNo, sliceNo, frameNo) -roi.setPosition(index) -``` - -**Note:** If index is 0 (default), the Roi applies to all image planes in the stack. - -
- -## Particle Analysis {#particle-analysis-id} - -1. In the [Macro Recorder](#macro-recorder-id) window’s `Record` drop-down menu, select `BeanShell`. Remove any code. -2. In Fiji, go to `File` > `Open Samples` > `Blobs` -3. Go to `Process` > `Filters` > `Median…` - * Set `Radius` : 2 - * Click `Ok`. -4. Go to `Process` > `Binary` > `Options…` - * Check `Black background` box. - * Click `OK`. -5. Go to `Image` > `Adjust Threshold…` - * Select `Default` and `Red` in drop-down menu - * Uncheck `Dark background` box. - * Click `Apply`. -6. Go to `Process` > `Binary` > `Watershed` -7. Go to `Analyze` > `Set Measurements` - * Select options as shown. - * Click `Ok`. -8. Go to `Analyze` > `Analyze Particles…` - * Select options as shown. - * Click `Ok`. - -{{< figure src="fiji-particle-analysis.png" >}} - -Convert the Macro recordings into a Python script, remember to do the following in the [Script Editor](#script-editor-id): - -* In the Script Editor, go to `Language`, and select `Python`. -* Insert an `import` statement at the top the script -* Remove semicolons (`;`) at the end of the lines. - -The script should look like this: -``` -from ij import IJ - -imp = IJ.openImage("http://imagej.nih.gov/ij/images/blobs.gif") -IJ.run(imp, "Median...", "radius=2") -IJ.run(imp, "Options...", "iterations=1 count=1 black") -IJ.setAutoThreshold(imp, "Default") -IJ.run(imp, "Convert to Mask", "") -IJ.run(imp, "Watershed", "") -IJ.run(imp, "Set Measurements...", "area mean min centroid integrated display redirect=None decimal=3") -IJ.run(imp, "Analyze Particles...", "size=0-Infinity display exclude clear summarize add") -imp.show() -``` - -Save the script as `Blob_Analysis.py`, and run the script. - -You should see the following results tables. - -{{< figure src="fiji-particle-analysis-results.png" >}} - -
- - -## RoiManager {#roi-manager-id} - -The [RoiManager](https://imagej.nih.gov/ij/developer/api/ij/ij/plugin/frame/RoiManager.html) class implements the `Analyze` > `Tools` > `ROI Manager` function. The Particle Analyzer can also use the Roi Manager to store ROIs identified during the analysis, see [Particle Analysis](#particle-analysis-id). - -**Using the RoiManager class in scripts:** - -Get reference to system default instance -``` -from ij.plugin.frame import RoiManager -rm = RoiManager.getInstance2() -``` - -Get number of ROIs managed by RoiManager -``` -count = rm.getCount() -``` - -Get all ROIs as a list -``` -rois = rm.getAllRoisAsArray() -``` - -Get specific ROI at specific index position -``` -roi = rm.getRoi(index) -``` - -Remove all ROIs -``` -rm.reset() -``` - -Add ROI -``` -roi = OvalRoi(100, 150, 50, 50) -rm.addRoi(roi) -``` - -Run operation on ROI: combine, select, save, open, etc. -``` -rm.runCommand(command_string) -``` - -
- -## ImageStatistics {#imagestats-id} - -The [ImageStatistics](https://imagej.nih.gov/ij/developer/api/ij/ij/process/ImageStatistics.html) class provides access to pixel-based statistics, including the histogram, of an entire image or ROI selection in an image. - -**Common statements:** - -Define measurement options: bitwise OR combination of constants -``` -from ij.process import ImageStatistics as IS -options = IS.MEAN | IS.AREA | IS.STD_DEV # many others -``` -
-Measure entire image (imp) -``` -imp.setRoi(null) # remove any ROI from image -stats = imp.getStatistics() -``` -
-Measure particular ROI (roi) in image (imp) -``` -imp.setRoi(roi) -stats = imp.getStatistics() -``` -
-Get histogram for image (imp) as a list of intensity counts: -``` -stats = imp.getStatistics() -histo_values = stats.getHistogram() -``` - -
- -## ResultsTable {#resultstable-id} - -The [ResultsTable](https://imagej.nih.gov/ij/developer/api/ij/ij/measure/ResultsTable.html) class is used for storing measurement results and strings as columns of values. - -**Common statements:** - -Get default table used by `Analyze` > `Measure` -``` -from ij.measure import ResultsTable -rt = ResultsTable.getResultsTable() -``` - -Create new empty table -``` -rt = ResultsTable() -``` - -Show table with title. The title `Results` is reserved for default table for `Analyze` > `Measure`. -``` -rt.show("My Results") # passed argument becomes table title -``` - -Get number of table rows -``` -rt.getCount() -``` - -Get list of float values in column (by column header) -``` -col_index = rt.getColumnIndex("Area") -rt.getColumn(col_index) -``` - -Add value to specific column in last row -``` -rt.addValue("Area", 23.0) -``` - -Delete column -``` -rt.deleteColumn("Area") -``` - -Delete row (by row index) -``` -rt.deleteRow(row_index) -``` - -
- -## Custom Dialog Windows: Collecting User Input {#dialogs-id} - -Custom language agnostic dialogs can be created using the [SciJava@Parameter][scijava-params] annotation. - -* Script parameters are a feature of ImageJ2; they will work in Fiji but will not work in plain ImageJ1. -* Basic Syntax: - * Parameter declarations begin with `#@`. Each such line contains a single parameter declaration or script directive and nothing else. - * `#@ Type variableName` will declare an input of the indicated type, assigned to the specified name. The use of a space between #@ and Type is encouraged, but not required. - * `#@output Type outputName` will declare the variable of the specified name as an output parameter with the given type. The Type parameter is optional, as the output will be treated as Object be default. (For the output directive and other script directives, no space is allowed between `#@` and the directive.) - -| Data type | Widget type | Available styles | -| --- | --- | --- | -| boolean / Boolean | checkbox | | -| byte / short / int / long | numeric field | slider / spinner / scroll bar | -| Byte / Short / Integer / Long | numeric field | slider / spinner / scroll bar | -| Float | numeric field | slider / spinner / scroll bar | -| BigInteger / BigDecimal | numeric field | slider / spinner / scroll bar | -| char / Character / String |text field | text field / text area / password | -| Dataset | ImagePlus | (>=2 images) triggers a dropdown list | -| ColorRGB | color chooser | | -| Date | date chooser | | -| File | file chooser | open / save / file / directory / extensions | - -Source: https://imagej.net/Script_Parameters - -**Example:** -``` -# @String(label="Please enter your name",description="Name field") name -# @OUTPUT String greeting - -greeting = "Hello, " + name + "!" -``` - -When you run the script, a dialog will pop up asking for your input. Type in a name (any character sequence) and click `OK`. The output message will be shown in a separate window. - -{{< figure src="fiji-greeting-script.png" >}} - -
- -## Batch Processing of Image Files {#batch-processing-id} - -Often we want to apply the same image processing method to a set of images. - -* It is convenient to organize such image files into a single directory or group of directories. -* We already created a series of images and saved them in the workshop/images directory. Let’s produce a simple batch processing script that applies a Gaussian filter to all images in that directory. - -**Simple Template:** -``` -# @ File (label="Input directory", style="directory") inputdir -# @ File (label="Output directory", style="directory") outputdir - -from ij import IJ -import os -from os import path - -def process_file(f): - print "Processing", f - imp = IJ.openImage(f) - IJ.run(imp, "Gaussian Blur...", "sigma=2"); - return imp - -def save_as_tif(directory, image): - title = image.getTitle().split(".")[0] + "-blurred.tif" - outputfile = path.join(outputdir, title) - IJ.saveAs(image, "TIFF", outputfile) - print "Saved", outputfile - -# Main code -inputdir = str(inputdir) -outputdir = str(outputdir) -if not path.isdir(inputdir): - print inputdir, " is does not exist or is not a directory." -else: - if not path.isdir(outputdir): - os.makedirs(outputdir) - filenames = os.listdir(inputdir) - tif_files = [f for f in filenames if f.split(".")[-1] == "tif"] - for tif_file in tif_files: - fullpath = path.join(inputdir, tif_file) - imp = process_file(fullpath) - save_as_tif(outputdir, imp) - print "Done.\n" -``` - -
- -# Installing Scripts as Plugins {#install-plugins-id} - -Once you have completed script development, you can install the script in Fiji. It will show up as a selectable entry in the `Plugins` menu. - -* **Note:** Your script name needs to include at least one underscore (`_`). Example: `Basic_Crop.py` - -There are two installation options: - -* **Option A) Manual process:** - 1. Copy the script to the `plugins` directory in your Fiji install tree. You can create a subdirectory in `plugins` (e.g. plugins/workshop) and place your script there if you like to group related scripts. -* **Option B) Fiji installer:** - 1. In Fiji, go to `Plugins` > `Install…`. - 2. In the first pop-up dialog browse to and select the script. Click `Open`. - 3. In the second dialog window, select the "plugins" directory (or a subdirectory within plugins) to where the script should be installed. Click `Save`. - -After installing the script as plugin, restart Fiji. - -Go to `File` > `Plugins` and verify that the script is available. The `_` and file extension are stripped from the plugin name. Example: `Basic_Crop.py` is installed as `Basic Crop` - -
- -# Exercises {#exercises-id} - -## Beginner {#beginner-ex-id} - -**Project 1:** Modify the [Get Current Image example script](#current-image-id) to print the number of channels, number of focal planes (z), and number of timepoints. Open images of your choice from `Fiji` > `Open Samples` and compare the output when running the script. - -**Project 2:** Open the [Macro Recorder](#macro-recorder-id), delete any code in the recorder pane. Record the following steps and convert the macro recording into Jython script. [Hint: IJ.run](#ij-run-id) - -1. Go to `File` > `Open Samples` > `Fluorescent Cells`. -2. Split the channels (Image > Color > Split Channels). -3. Merge the channels again as `Composite` image, but swap the red and green channel. - -**Project 3:** Implement the [crop ROI example script](#roi-crop-id) with functions. - -**Project 4:** Modify the processing function in [Batch Processing example script](#batch-processing-id) to apply a different image filter. [Hint: IJ.run](#ij-run-id) - -
- -## Intermediate {#intermediate-ex-id} - -**Project 5:** Modify the [crop ROI example script](#roi-crop-id) to have it create multiple cropped images of various sizes in a code loop. [Hint: For loops](#loops-id) - -**Project 6:** Modify the processing function in [Batch Processing example script](#batch-processing-id) to prompt user for `sigma` value to be used by the Gaussian filter. [Hint: Custom Dialog Windows](#dialogs-id). The custom sigma value should be passed from the main code block as an argument to the `process` function. [Hint: Functions](#functions-id) - -**Project 7:** Let's try to improve the [particle analysis](#particle-analysis-id) by implementing the following features: - -* Duplicate the original image: [Duplicator](https://imagej.nih.gov/ij/developer/api/ij/ij/plugin/Duplicator.html) class -* Access to ROIs identified by Particle Analyzer: [RoiManager](https://imagej.nih.gov/ij/developer/api/ij/ij/plugin/frame/RoiManager.html) class -* Measuring area, pixel intensities, etc. for each ROI: [ImageStatistics](https://imagej.nih.gov/ij/developer/api/ij/ij/process/ImageStatistics.html) class -* Creating a customized results table and save it: [ResultsTable](https://imagej.nih.gov/ij/developer/api/ij/ij/measure/ResultsTable.html) class -* Custom Dialog for User Input to specify particle size: [Custom User Dialogs](#dialogs-id) - -
- -``` -# @ Float (label="Min particle size", value=50) min_size -# @ Float (label="Max particle size", value=200) max_size -# @ Boolean (label="Save ROIs", value=false) save_rois -# @ File (label="Output directory", style="directory") outputdir - -from ij import IJ -from ij.plugin import Duplicator -from ij.plugin.frame import RoiManager -from ij.measure import ResultsTable -from ij.process import ImageStatistics as IS -from os import path - -# open image and create copy -original = IJ.openImage("http://imagej.nih.gov/ij/images/blobs.gif") -original.show() -imp = Duplicator().run(original) -imp.setTitle("Mask") # rename the copy - -IJ.run(imp, "Median...", "radius=2"); -IJ.run(imp, "Options...", "iterations=1 count=1 black") -IJ.setAutoThreshold(imp, "Default") -IJ.run(imp, "Convert to Mask", "") -IJ.run(imp, "Watershed", "") # break up particle clumps -IJ.run(imp, "Set Measurements...", "area mean min centroid integrated display redirect=None decimal=3") -# hardcoded: "size=50-Infinity display exclude clear summarize add" -moptions = "size=" \ - + str(min_size) + "-" + str(max_size) \ - + " display exclude clear summarize add" -IJ.run(imp, "Analyze Particles...", moptions) -imp.show() - -# get default instance of RoiManager(used by ParticleAnalyzer) -rm = RoiManager.getInstance2() - -# get list of all ROIs -rois = rm.getRoisAsArray() - -# set measurement options -options = IS.MEAN | IS.AREA | IS.CENTROID - -# create Results Table -rt = ResultsTable() - -# iterate over all ROIs -for roi in rois: - original.setRoi(roi) - stats = original.getStatistics(options) # measure - rt.incrementCounter() # advance row counter - - # add values to various columns - rt.addValue("Label", original.getTitle()) - rt.addValue("Mean", stats.mean) - rt.addValue("Area", stats.area) - rt.addValue("Centroid X", stats.xCentroid) - rt.addValue("Centroid Y", stats.yCentroid) - -# show custom table -rt.show("Blob Results") - -# save contents of ResultsTable object as .csv file -outputdir = str(outputdir) -resultsfile = path.join(outputdir, "blobs.csv") -print "Results file:", resultsfile -rt.saveAs(resultsfile) - -if save_rois: - roifile = path.join(outputdir, "blobs-rois.zip") - rm.deselect() # ensure all ROIs rather than selected are saved - rm.runCommand("Save", roifile) -``` - -
- -## Expert {#expert-ex-id} - -**Project 8:** Modify the processing function in [Batch Processing example script](#batch-processing-id) to apply a median filter to a circular shaped ROI in the center of the image. The ROI diameter should be half of the width or height (whichever is smaller) of the image. The radius of the median filter should be requested from the user. [Hint: Setting ROI](#roi-set-id), [Hint: Recording filter functions](#ij-run-id), [Hint: Custom Dialog Windows](#dialogs-id) - -**Project 9:** Let’s create a script that performs the following operations on a multidimensional [ImageStack](https://imagej.nih.gov/ij/developer/api/ij/ij/ImageStack.html): - -* Extract the central focal image planes of the last channel for all timepoints. -* Save each extracted image plane as a separate .tif image file in a designated directory. -* The pixel size calibration of the original image should be retained in each saved image plane. - - -``` -from ij import IJ -from ij import ImagePlus -import os -from os import path - -imp = IJ.openImage("http://imagej.nih.gov/ij/images/Spindly-GFP.zip") -channel_no = imp.getNChannels() # last channel -slice_no = 1 + imp.getNSlices() // 2 # integer division to get center slice -stack = imp.getStack() # get ImageStack object -calibration = imp.getCalibration().copy() # get calibration - -# create output dir workshop/images in home directory if it does not exist -outputdir = path.join(path.expanduser("~"), "workshop", "images") -print outputdir -if not path.isdir(outputdir): - os.makedirs(outputdir) - -# iterate over all frames and save central slice of last channel -for frame_no in range(1, imp.getNFrames() + 1): - stack_index = imp.getStackIndex(channel_no, slice_no, frame_no) - ip = stack.getProcessor(stack_index) - - # remove extension from image title and add frame no as suffix - title = imp.getTitle().split(".")[0] + "-" + str(frame_no) - single_imp = ImagePlus(title, ip) - single_imp.setCalibration(calibration) - - # create file name with absolute path (incl directory name) and save - outputfile = path.join(outputdir, title) - IJ.saveAs(single_imp, "Tiff", outputfile) -``` - -**Project 10:** Modify Project 9 to include the following features: - -* Apply either a Gaussian or Median filter of a chosen radius to the extracted image plane. -* Downsample the extracted image plane by a certain factor (see next instruction). Pay to attention to where to place that command in relation to the `setCalibration()` command. -* Include a custom dialog that - * lets the user choose the filter mode ("Gaussian Blur..." or "Median...") and filter radius (sigma for Gaussian), - * lets the user specify the downsampling fraction (0 < fraction <= 1.0). 1.0 would imply no downsampling. Hint: `Image` > `Adjust` > `Size`. - -# Resources {#resources-id} - -**Fiji Scripting** - -* Tutorial: https://syn.mrc-lmb.cam.ac.uk/acardona/fiji-tutorial/ -* Tips for Developers: https://imagej.net/Tips_for_developers -* API: https://imagej.nih.gov/ij/developer/api/ -* SciJava: https://javadoc.scijava.org/Fiji/ -* Advanced: https://imagej.net/tutorials/imagej2-python - -**General Scripting** - -* Python: https://learning.rc.virginia.edu/courses/programming_python_scientists_engineers/ - - -[imageplus]: https://imagej.nih.gov/ij/developer/api/ij/ij/ImagePlus.html -[imageprocessor]: https://imagej.nih.gov/ij/developer/api/ij/ij/process/ImageProcessor.html -[imagestack]: https://imagej.nih.gov/ij/developer/api/ij/ij/ImageStack.html -[byteprocessor]: https://imagej.nih.gov/ij/developer/api/ij/ij/process/ByteProcessor.html -[shortprocessor]: https://imagej.nih.gov/ij/developer/api/ij/ij/process/ShortProcessor.html -[floatprocessor]: https://imagej.nih.gov/ij/developer/api/ij/ij/process/FloatProcessor.html -[colorprocessor]: https://imagej.nih.gov/ij/developer/api/ij/ij/process/ColorProcessor.html -[ospath]: https://docs.python.org/2/library/os.path.html -[imagej-api]: https://imagej.nih.gov/ij/developer/api/index.html -[bioformats]: https://www.openmicroscopy.org/bio-formats/ -[roi]: https://imagej.nih.gov/ij/developer/api/ij/ij/gui/Roi.html -[color]: https://docs.oracle.com/javase/7/docs/api/java/awt/Color.html -[roi-manager]: https://imagej.nih.gov/ij/developer/api/ij/ij/plugin/frame/RoiManager.html -[imagestats]: https://imagej.nih.gov/ij/developer/api/ij/ij/process/ImageStatistics.html -[resultstable]: https://imagej.nih.gov/ij/developer/api/ij/ij/measure/ResultsTable.html -[duplicator]: https://imagej.nih.gov/ij/developer/api/ij/ij/plugin/Duplicator.html -[scijava-params]: https://imagej.net/Script_Parameters diff --git a/content/courses/fortran-introduction/_index.md b/content/courses/fortran-introduction/_index.md index fab5d796..f6a94271 100644 --- a/content/courses/fortran-introduction/_index.md +++ b/content/courses/fortran-introduction/_index.md @@ -2,7 +2,7 @@ date : "2021-04-05T00:00:00-05:00" title : "Programming in Modern Fortran" summary: "This short course is an introduction to programming in modern Fortran. Experience programming in some other language is helpful but not required." -authors: [kah] +authors: [ kah ] categories: ["Programming","Fortran"] tags: [Programming, Compilers, Fortran] toc: true diff --git a/content/courses/fortran-introduction/advanced_arrays.md b/content/courses/fortran-introduction/advanced_arrays.md index 333f31a3..42f66556 100644 --- a/content/courses/fortran-introduction/advanced_arrays.md +++ b/content/courses/fortran-introduction/advanced_arrays.md @@ -41,7 +41,7 @@ INTEGER, DIMENSION(20) :: C C=N(:,i) !ith column of N ``` -The upper bound of the range is always included. If the first bound is omitted, it starts from 1. If the second bound is absent, the slice is extracted to the end of the range. A single colon `:` represents the full range along a dimension. +The value of the upper bound of the range is always included in the range. If the first bound is omitted, it starts from 1. If the second bound is absent, the slice is extracted to the end of the range. A single colon `:` represents the full range along a dimension. ## Allocatable Arrays diff --git a/content/courses/fortran-introduction/building.md b/content/courses/fortran-introduction/building.md index 6071511c..34ba1b70 100644 --- a/content/courses/fortran-introduction/building.md +++ b/content/courses/fortran-introduction/building.md @@ -37,9 +37,9 @@ An Integrated Development Environment (IDE) combines an editor and a way to comp A well-known IDE for Microsoft Windows is Visual Studio. This is available through the Microsoft Store; it is not free for individuals. macOS uses Xcode as its native IDE. Xcode includes some compilers, particularly for Swift, but it can manage several other languages. Available at the App Store and free. -A full-featured cross-platform IDE is [Eclipse] (http://www.eclipse.org/). Free. +A full-featured cross-platform IDE is [Eclipse](http://www.eclipse.org/). Free. -A lighter-weight IDE for Windows and Linux is [Code::Blocks] (http://www.codeblocks.org/). Free. +A lighter-weight IDE for Windows and Linux is [Code::Blocks](http://www.codeblocks.org/). Free. Windows programmers using Intel's oneAPI distribution must also install [Visual Studio](https://visualstudio.microsoft.com/). diff --git a/content/courses/fortran-introduction/intrinsic_modules.md b/content/courses/fortran-introduction/intrinsic_modules.md index ed68071e..4482241f 100644 --- a/content/courses/fortran-introduction/intrinsic_modules.md +++ b/content/courses/fortran-introduction/intrinsic_modules.md @@ -79,7 +79,7 @@ CALL IEEE_SET_HALTING_MODE(flag,halt) ``` `Flag` is a variable of TYPE(IEEE_FLAG_TYPE), also defined in the module. It can be IEEE_DIVIDE_BY_ZERO, IEEE_INEXACT, IEEE_INVALID, IEEE_OVERFLOW, or IEEE_UNDERFLOW. The `HALTING` argument is LOGICAL. If HALTING is set to `.true.` the program will stop on occurrence of the specified MODE. Since this is often desired for INVALID (NaN), DIVIDE_BY_ZERO, and OVERFLOW, a predefined array is available. -{{< code-download file="courses/fortran-introduction/codes/ieee_exc.f90" lang="fortran" >}} +{{< code-download file="/courses/fortran-introduction/codes/ieee_exc.f90" lang="fortran" >}} ### IEEE Arithmetic @@ -88,7 +88,7 @@ Several useful procedures and included. IEEE_SELECTED_REAL_KIND chooses only KIN For more elegant handling of errors, the IEEE_IS_NAN, IEEE_IS_FINITE, and some others can be used to test a result and handle it in some manner other than halting the execution. -{{< code-download file="courses/fortran-introduction/codes/ieee_arith.f90" lang="fortran" >}} +{{< code-download file="/courses/fortran-introduction/codes/ieee_arith.f90" lang="fortran" >}} ## ISO_C_BINDING diff --git a/content/courses/fortran-introduction/modules.md b/content/courses/fortran-introduction/modules.md index bf545c66..e3413b71 100644 --- a/content/courses/fortran-introduction/modules.md +++ b/content/courses/fortran-introduction/modules.md @@ -97,7 +97,7 @@ A module must be compiled _before_ any other file that uses it. This can create Fortran allows the module and the file to have either the same or a different name, but the name of the module is the name that must appear in the use statement. 2. Fill out the subroutine `mysub` to set b to 11., then set x to the sum of corresponding elements of a and b. Hint: you can use x=a(:)+b(:size(a)) to avoid a loop. 3. Write a main program `main.f90` that uses `mymod`, initializes `A` allocatable, allocates it to 1000, sets its values to `i+3` in a loop, then passes it to `mysub`. Print the value of `x` that is returned. -4. Create a Makefile. If you wish you may copy the example Makefile from the earlier [chapter](/courses/fortran_introdution/make). Make the appropriate changes to the program name, the names of the source files, and the names of the object files. Make the dependency line at the end +4. Create a Makefile. If you wish you may copy the example Makefile from the earlier [chapter](/courses/fortran-introduction/make/). Make the appropriate changes to the program name, the names of the source files, and the names of the object files. Make the dependency line at the end ```make main.o:main.o mymod.o ``` diff --git a/content/courses/fortran-introduction/scope.md b/content/courses/fortran-introduction/scope.md index 8356f1a1..0fe6854a 100644 --- a/content/courses/fortran-introduction/scope.md +++ b/content/courses/fortran-introduction/scope.md @@ -102,6 +102,6 @@ where `file.h` can be any name; Fortran does not have a rule about file extensio COMMON is a frequent source of memory errors. COMMON makes interface control difficult to impossible. -The recommended first step in updating [old code](/courses/fortran-introduction/updating_old_code) is to replace all COMMON with modules and then gradually to move the variables into the appropriate parameter lists. +The recommended first step in updating [old code](/courses/fortran-introduction/update_old_code) is to replace all COMMON with modules and then gradually to move the variables into the appropriate parameter lists. COMMON is a vestige of the early computer era, when main memory was measuring in megabytes or even kilobytes, and every byte was precious. With COMMON the global variables would occupy a single spot in memory, would not be copied, and could be reused. In particularly old code it is not unusual for arrays in COMMON to be reshaped implicitly, since COMMON inherently represents a large linear block of memory. Sometimes the EQUIVALENCE statement is still seen, by which types could be converted implicitly as long as they occupied the same number of bytes. diff --git a/content/courses/fortran-introduction/setting_up.md b/content/courses/fortran-introduction/setting_up.md index 6f052edf..b06d27e8 100644 --- a/content/courses/fortran-introduction/setting_up.md +++ b/content/courses/fortran-introduction/setting_up.md @@ -40,7 +40,7 @@ Download the Mac version from Intel. The NVIDIA HPC SDK is not available for Macs. -Geany can be installed from its [homepage](www.geany.org). Other options, such as VSCode, can be installed similarly. +Geany can be installed from its [homepage](https://www.geany.org). Other options, such as VSCode, can be installed similarly. ### Windows diff --git a/content/courses/fortran-introduction/subprogram_arrays.md b/content/courses/fortran-introduction/subprogram_arrays.md index dce03dac..475010e9 100644 --- a/content/courses/fortran-introduction/subprogram_arrays.md +++ b/content/courses/fortran-introduction/subprogram_arrays.md @@ -66,7 +66,7 @@ real, dimension(:) :: A ! in sub Though the dimensions need not be known for assumed-shape arrays, the rank must match. **Example** -{{< code-download file="courses/fortran-introduction/codes/pass_arrays.f90" lang="fortran" >}} +{{< code-download file="/courses/fortran-introduction/codes/pass_arrays.f90" lang="fortran" >}} ### Allocating Arrays in a Subprogram diff --git a/content/courses/matlab-machine-learning/80823v00_machine_learning_section3_ebook_v05.pdf b/content/courses/matlab-machine-learning/80823v00_machine_learning_section3_ebook_v05.pdf deleted file mode 100644 index 4b1110da..00000000 Binary files a/content/courses/matlab-machine-learning/80823v00_machine_learning_section3_ebook_v05.pdf and /dev/null differ diff --git a/content/courses/matlab-machine-learning/88174_92991v00_machine_learning_section1_ebook.pdf b/content/courses/matlab-machine-learning/88174_92991v00_machine_learning_section1_ebook.pdf deleted file mode 100644 index b51ab03b..00000000 Binary files a/content/courses/matlab-machine-learning/88174_92991v00_machine_learning_section1_ebook.pdf and /dev/null differ diff --git a/content/courses/matlab-machine-learning/88655_93014v00_machine_learning_section2_ebook.pdf b/content/courses/matlab-machine-learning/88655_93014v00_machine_learning_section2_ebook.pdf deleted file mode 100644 index cd486eb5..00000000 Binary files a/content/courses/matlab-machine-learning/88655_93014v00_machine_learning_section2_ebook.pdf and /dev/null differ diff --git a/content/courses/matlab-machine-learning/90221_80827v00_machine_learning_section4_ebook_v03.pdf b/content/courses/matlab-machine-learning/90221_80827v00_machine_learning_section4_ebook_v03.pdf deleted file mode 100644 index 84db1569..00000000 Binary files a/content/courses/matlab-machine-learning/90221_80827v00_machine_learning_section4_ebook_v03.pdf and /dev/null differ diff --git a/content/courses/matlab-machine-learning/_index.md b/content/courses/matlab-machine-learning/_index.md deleted file mode 100644 index 0cb9a9db..00000000 --- a/content/courses/matlab-machine-learning/_index.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -title: "Machine Learning with Matlab" -summary: Machine learning in Matlab, based on Mathworks online tutorials. -authors: [cag] -date: 2019-10-16T21:13:14-05:00 -categories: ["MATLAB","Machine Learning"] -tags: ["MATLAB","Machine_Learning"] -toc: true -type: docs - -menu: - matlab-machine-learning: - name: Overview - weight: 1 ---- - -# Overview - -This short course focuses on data analytics and machine learning techniques in MATLAB® using functionality within Statistics and Machine Learning Toolbox™ and Neural Network Toolbox™. The course demonstrates the use of unsupervised learning to discover features in large data sets and supervised learning to build predictive models. Examples and exercises highlight techniques for visualization and evaluation of results. Topics include: - - - Organizing and preprocessing data - - Clustering data - - Creating classification and regression models - - Interpreting and evaluating models - - Simplifying data sets - - Using ensembles to improve model performance - - -The material for this course uses selected topics from the online self-paced course from the MathWorks in the link below. Participants should have a Mathworks account in order to access the links in this document. - -**[Matlab Academy: Machine Learning with Matlab](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlml)** - diff --git a/content/courses/matlab-machine-learning/bayes1.png b/content/courses/matlab-machine-learning/bayes1.png deleted file mode 100644 index be8385cd..00000000 Binary files a/content/courses/matlab-machine-learning/bayes1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/bayes2.png b/content/courses/matlab-machine-learning/bayes2.png deleted file mode 100644 index 2a33cc6a..00000000 Binary files a/content/courses/matlab-machine-learning/bayes2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/class1.png b/content/courses/matlab-machine-learning/class1.png deleted file mode 100644 index 4e255507..00000000 Binary files a/content/courses/matlab-machine-learning/class1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/class2.png b/content/courses/matlab-machine-learning/class2.png deleted file mode 100644 index c726bbbe..00000000 Binary files a/content/courses/matlab-machine-learning/class2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/class3.png b/content/courses/matlab-machine-learning/class3.png deleted file mode 100644 index c8fe48eb..00000000 Binary files a/content/courses/matlab-machine-learning/class3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/class4.png b/content/courses/matlab-machine-learning/class4.png deleted file mode 100644 index 69e95ef3..00000000 Binary files a/content/courses/matlab-machine-learning/class4.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/class5.png b/content/courses/matlab-machine-learning/class5.png deleted file mode 100644 index b11d1b06..00000000 Binary files a/content/courses/matlab-machine-learning/class5.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/class6.png b/content/courses/matlab-machine-learning/class6.png deleted file mode 100644 index 2ea3dd5a..00000000 Binary files a/content/courses/matlab-machine-learning/class6.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/class7.png b/content/courses/matlab-machine-learning/class7.png deleted file mode 100644 index 56f023fd..00000000 Binary files a/content/courses/matlab-machine-learning/class7.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/classify1.png b/content/courses/matlab-machine-learning/classify1.png deleted file mode 100644 index 3491f82c..00000000 Binary files a/content/courses/matlab-machine-learning/classify1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/classify2.png b/content/courses/matlab-machine-learning/classify2.png deleted file mode 100644 index bbe1f5fa..00000000 Binary files a/content/courses/matlab-machine-learning/classify2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/classify3.png b/content/courses/matlab-machine-learning/classify3.png deleted file mode 100644 index c7cd9c97..00000000 Binary files a/content/courses/matlab-machine-learning/classify3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/classify4.png b/content/courses/matlab-machine-learning/classify4.png deleted file mode 100644 index 9bb614d6..00000000 Binary files a/content/courses/matlab-machine-learning/classify4.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/confuse1.png b/content/courses/matlab-machine-learning/confuse1.png deleted file mode 100644 index a670fcf2..00000000 Binary files a/content/courses/matlab-machine-learning/confuse1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/confuse2.png b/content/courses/matlab-machine-learning/confuse2.png deleted file mode 100644 index 85a27220..00000000 Binary files a/content/courses/matlab-machine-learning/confuse2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/confuse3.png b/content/courses/matlab-machine-learning/confuse3.png deleted file mode 100644 index 4a47653e..00000000 Binary files a/content/courses/matlab-machine-learning/confuse3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/confuse4.png b/content/courses/matlab-machine-learning/confuse4.png deleted file mode 100644 index 53f91ef0..00000000 Binary files a/content/courses/matlab-machine-learning/confuse4.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/crossval1.png b/content/courses/matlab-machine-learning/crossval1.png deleted file mode 100644 index ac3c7116..00000000 Binary files a/content/courses/matlab-machine-learning/crossval1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/crossval2.png b/content/courses/matlab-machine-learning/crossval2.png deleted file mode 100644 index bb044d00..00000000 Binary files a/content/courses/matlab-machine-learning/crossval2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/crossval3.png b/content/courses/matlab-machine-learning/crossval3.png deleted file mode 100644 index 5733cefc..00000000 Binary files a/content/courses/matlab-machine-learning/crossval3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/crossval4.png b/content/courses/matlab-machine-learning/crossval4.png deleted file mode 100644 index 87b9404e..00000000 Binary files a/content/courses/matlab-machine-learning/crossval4.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/crossval5.png b/content/courses/matlab-machine-learning/crossval5.png deleted file mode 100644 index d54e165e..00000000 Binary files a/content/courses/matlab-machine-learning/crossval5.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/discrim1.png b/content/courses/matlab-machine-learning/discrim1.png deleted file mode 100644 index 82a6acda..00000000 Binary files a/content/courses/matlab-machine-learning/discrim1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/discrim2.png b/content/courses/matlab-machine-learning/discrim2.png deleted file mode 100644 index 69fbb6c9..00000000 Binary files a/content/courses/matlab-machine-learning/discrim2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/discrim3.png b/content/courses/matlab-machine-learning/discrim3.png deleted file mode 100644 index 168719e0..00000000 Binary files a/content/courses/matlab-machine-learning/discrim3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/ensembl1.png b/content/courses/matlab-machine-learning/ensembl1.png deleted file mode 100644 index 731d963f..00000000 Binary files a/content/courses/matlab-machine-learning/ensembl1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/ensembl2.png b/content/courses/matlab-machine-learning/ensembl2.png deleted file mode 100644 index 50bd296e..00000000 Binary files a/content/courses/matlab-machine-learning/ensembl2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/ensembl3.png b/content/courses/matlab-machine-learning/ensembl3.png deleted file mode 100644 index 39bc66fd..00000000 Binary files a/content/courses/matlab-machine-learning/ensembl3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/ensembl4.png b/content/courses/matlab-machine-learning/ensembl4.png deleted file mode 100644 index ea2bdbba..00000000 Binary files a/content/courses/matlab-machine-learning/ensembl4.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/eval1.png b/content/courses/matlab-machine-learning/eval1.png deleted file mode 100644 index c8fa44dd..00000000 Binary files a/content/courses/matlab-machine-learning/eval1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/eval2.png b/content/courses/matlab-machine-learning/eval2.png deleted file mode 100644 index 82362872..00000000 Binary files a/content/courses/matlab-machine-learning/eval2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/eval3.png b/content/courses/matlab-machine-learning/eval3.png deleted file mode 100644 index ae364762..00000000 Binary files a/content/courses/matlab-machine-learning/eval3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/fit1.png b/content/courses/matlab-machine-learning/fit1.png deleted file mode 100644 index 99872832..00000000 Binary files a/content/courses/matlab-machine-learning/fit1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/fit2.png b/content/courses/matlab-machine-learning/fit2.png deleted file mode 100644 index e8c870af..00000000 Binary files a/content/courses/matlab-machine-learning/fit2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/fit3.png b/content/courses/matlab-machine-learning/fit3.png deleted file mode 100644 index b5d5d32d..00000000 Binary files a/content/courses/matlab-machine-learning/fit3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/fit4.png b/content/courses/matlab-machine-learning/fit4.png deleted file mode 100644 index c8cb070d..00000000 Binary files a/content/courses/matlab-machine-learning/fit4.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/fit5.png b/content/courses/matlab-machine-learning/fit5.png deleted file mode 100644 index 5568466e..00000000 Binary files a/content/courses/matlab-machine-learning/fit5.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/forward1.png b/content/courses/matlab-machine-learning/forward1.png deleted file mode 100644 index 01a5586a..00000000 Binary files a/content/courses/matlab-machine-learning/forward1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/forward2.png b/content/courses/matlab-machine-learning/forward2.png deleted file mode 100644 index 16b18a7b..00000000 Binary files a/content/courses/matlab-machine-learning/forward2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/forward3.png b/content/courses/matlab-machine-learning/forward3.png deleted file mode 100644 index fe288c89..00000000 Binary files a/content/courses/matlab-machine-learning/forward3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/forward4.png b/content/courses/matlab-machine-learning/forward4.png deleted file mode 100644 index 96563fd4..00000000 Binary files a/content/courses/matlab-machine-learning/forward4.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/forward5.png b/content/courses/matlab-machine-learning/forward5.png deleted file mode 100644 index ad0d73fe..00000000 Binary files a/content/courses/matlab-machine-learning/forward5.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/forward6.png b/content/courses/matlab-machine-learning/forward6.png deleted file mode 100644 index d21e8426..00000000 Binary files a/content/courses/matlab-machine-learning/forward6.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/forward7.png b/content/courses/matlab-machine-learning/forward7.png deleted file mode 100644 index 1d9400a9..00000000 Binary files a/content/courses/matlab-machine-learning/forward7.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/forward8.png b/content/courses/matlab-machine-learning/forward8.png deleted file mode 100644 index 76a9500d..00000000 Binary files a/content/courses/matlab-machine-learning/forward8.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/forward9.png b/content/courses/matlab-machine-learning/forward9.png deleted file mode 100644 index d0325aa6..00000000 Binary files a/content/courses/matlab-machine-learning/forward9.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/fuel1.png b/content/courses/matlab-machine-learning/fuel1.png deleted file mode 100644 index 039f12c7..00000000 Binary files a/content/courses/matlab-machine-learning/fuel1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/fuel2.png b/content/courses/matlab-machine-learning/fuel2.png deleted file mode 100644 index 876aac30..00000000 Binary files a/content/courses/matlab-machine-learning/fuel2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/fuel3.png b/content/courses/matlab-machine-learning/fuel3.png deleted file mode 100644 index 42c3d009..00000000 Binary files a/content/courses/matlab-machine-learning/fuel3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/fuel4.png b/content/courses/matlab-machine-learning/fuel4.png deleted file mode 100644 index fd83e770..00000000 Binary files a/content/courses/matlab-machine-learning/fuel4.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/gauss1.png b/content/courses/matlab-machine-learning/gauss1.png deleted file mode 100644 index d269d699..00000000 Binary files a/content/courses/matlab-machine-learning/gauss1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/gauss2.png b/content/courses/matlab-machine-learning/gauss2.png deleted file mode 100644 index 12c0ac32..00000000 Binary files a/content/courses/matlab-machine-learning/gauss2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/gauss3.png b/content/courses/matlab-machine-learning/gauss3.png deleted file mode 100644 index fe33e74c..00000000 Binary files a/content/courses/matlab-machine-learning/gauss3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/gaussreg1.png b/content/courses/matlab-machine-learning/gaussreg1.png deleted file mode 100644 index ee40ebe1..00000000 Binary files a/content/courses/matlab-machine-learning/gaussreg1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/gaussreg2.png b/content/courses/matlab-machine-learning/gaussreg2.png deleted file mode 100644 index 43eef3fc..00000000 Binary files a/content/courses/matlab-machine-learning/gaussreg2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/gaussreg3.png b/content/courses/matlab-machine-learning/gaussreg3.png deleted file mode 100644 index f9b3ee75..00000000 Binary files a/content/courses/matlab-machine-learning/gaussreg3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/heir1.png b/content/courses/matlab-machine-learning/heir1.png deleted file mode 100644 index 6195d806..00000000 Binary files a/content/courses/matlab-machine-learning/heir1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/heir2.png b/content/courses/matlab-machine-learning/heir2.png deleted file mode 100644 index d700e204..00000000 Binary files a/content/courses/matlab-machine-learning/heir2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/heir3.png b/content/courses/matlab-machine-learning/heir3.png deleted file mode 100644 index b91e2699..00000000 Binary files a/content/courses/matlab-machine-learning/heir3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/heir4.png b/content/courses/matlab-machine-learning/heir4.png deleted file mode 100644 index 7862b380..00000000 Binary files a/content/courses/matlab-machine-learning/heir4.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/interpret1.png b/content/courses/matlab-machine-learning/interpret1.png deleted file mode 100644 index f39e9865..00000000 Binary files a/content/courses/matlab-machine-learning/interpret1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/interpret10.png b/content/courses/matlab-machine-learning/interpret10.png deleted file mode 100644 index 45455bc7..00000000 Binary files a/content/courses/matlab-machine-learning/interpret10.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/interpret2.png b/content/courses/matlab-machine-learning/interpret2.png deleted file mode 100644 index 58819e7f..00000000 Binary files a/content/courses/matlab-machine-learning/interpret2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/interpret3.png b/content/courses/matlab-machine-learning/interpret3.png deleted file mode 100644 index 6ea1ec8c..00000000 Binary files a/content/courses/matlab-machine-learning/interpret3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/interpret4.png b/content/courses/matlab-machine-learning/interpret4.png deleted file mode 100644 index f38c6543..00000000 Binary files a/content/courses/matlab-machine-learning/interpret4.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/interpret5.png b/content/courses/matlab-machine-learning/interpret5.png deleted file mode 100644 index 26faaa66..00000000 Binary files a/content/courses/matlab-machine-learning/interpret5.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/interpret6.png b/content/courses/matlab-machine-learning/interpret6.png deleted file mode 100644 index 3b643931..00000000 Binary files a/content/courses/matlab-machine-learning/interpret6.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/interpret7.png b/content/courses/matlab-machine-learning/interpret7.png deleted file mode 100644 index 1bcf6ffe..00000000 Binary files a/content/courses/matlab-machine-learning/interpret7.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/interpret8.png b/content/courses/matlab-machine-learning/interpret8.png deleted file mode 100644 index f17b453b..00000000 Binary files a/content/courses/matlab-machine-learning/interpret8.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/interpret9.png b/content/courses/matlab-machine-learning/interpret9.png deleted file mode 100644 index 33fb8f31..00000000 Binary files a/content/courses/matlab-machine-learning/interpret9.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/kmeans1.png b/content/courses/matlab-machine-learning/kmeans1.png deleted file mode 100644 index 4fb53458..00000000 Binary files a/content/courses/matlab-machine-learning/kmeans1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/kmeans2.png b/content/courses/matlab-machine-learning/kmeans2.png deleted file mode 100644 index ecddbf4b..00000000 Binary files a/content/courses/matlab-machine-learning/kmeans2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/kmeans3.png b/content/courses/matlab-machine-learning/kmeans3.png deleted file mode 100644 index d337d611..00000000 Binary files a/content/courses/matlab-machine-learning/kmeans3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/kmeans4.png b/content/courses/matlab-machine-learning/kmeans4.png deleted file mode 100644 index e9ca0eee..00000000 Binary files a/content/courses/matlab-machine-learning/kmeans4.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/kmeans5.png b/content/courses/matlab-machine-learning/kmeans5.png deleted file mode 100644 index a18585b8..00000000 Binary files a/content/courses/matlab-machine-learning/kmeans5.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/kmeans6.png b/content/courses/matlab-machine-learning/kmeans6.png deleted file mode 100644 index 6ac69d4d..00000000 Binary files a/content/courses/matlab-machine-learning/kmeans6.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/kmeans7.png b/content/courses/matlab-machine-learning/kmeans7.png deleted file mode 100644 index 324e5f4f..00000000 Binary files a/content/courses/matlab-machine-learning/kmeans7.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/lasso1.png b/content/courses/matlab-machine-learning/lasso1.png deleted file mode 100644 index 26e21a9a..00000000 Binary files a/content/courses/matlab-machine-learning/lasso1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/lasso2.png b/content/courses/matlab-machine-learning/lasso2.png deleted file mode 100644 index 22b07f43..00000000 Binary files a/content/courses/matlab-machine-learning/lasso2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/lasso3.png b/content/courses/matlab-machine-learning/lasso3.png deleted file mode 100644 index a7d1b18a..00000000 Binary files a/content/courses/matlab-machine-learning/lasso3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/linear1.png b/content/courses/matlab-machine-learning/linear1.png deleted file mode 100644 index 34f65385..00000000 Binary files a/content/courses/matlab-machine-learning/linear1.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/linear2.png b/content/courses/matlab-machine-learning/linear2.png deleted file mode 100644 index 574c2d92..00000000 Binary files a/content/courses/matlab-machine-learning/linear2.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/linear3.png b/content/courses/matlab-machine-learning/linear3.png deleted file mode 100644 index 08d055e6..00000000 Binary files a/content/courses/matlab-machine-learning/linear3.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/linear4.png b/content/courses/matlab-machine-learning/linear4.png deleted file mode 100644 index aac3575b..00000000 Binary files a/content/courses/matlab-machine-learning/linear4.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/linear5.png b/content/courses/matlab-machine-learning/linear5.png deleted file mode 100644 index e616ebe4..00000000 Binary files a/content/courses/matlab-machine-learning/linear5.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/linear6.png b/content/courses/matlab-machine-learning/linear6.png deleted file mode 100644 index e5d1b580..00000000 Binary files a/content/courses/matlab-machine-learning/linear6.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/linear7.png b/content/courses/matlab-machine-learning/linear7.png deleted file mode 100644 index 596b8f5e..00000000 Binary files a/content/courses/matlab-machine-learning/linear7.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/linear8.png b/content/courses/matlab-machine-learning/linear8.png deleted file mode 100644 index 268fa55e..00000000 Binary files a/content/courses/matlab-machine-learning/linear8.png and /dev/null differ diff --git a/content/courses/matlab-machine-learning/mathlab_ml.md b/content/courses/matlab-machine-learning/mathlab_ml.md deleted file mode 100644 index 8f69c3ac..00000000 --- a/content/courses/matlab-machine-learning/mathlab_ml.md +++ /dev/null @@ -1,552 +0,0 @@ ---- -title: Introduction to Machine Learning with Matlab -toc: true -type: docs -date: 2019-10-16T21:13:14-05:00 - -menu: - matlab-machine-learning: - parent: Machine Learning with Matlab - weight: 2 - ---- - -## Course Overview - -**Video: Machine Learning with Matlab** - -## Course Example - Basketball Player Statistics - -
-
-
-## Getting Starting with Data
-
-**Exercise: Importing Data**
-
-**Exercise: Using Logical Indexing**
-
-**Exercise: Creating Categorical Data**
-
-**Exercise: Grouping Data**
-
-**Exercise: Merging and Visualizing Data**
-
-**Exercise: Merging and Visualizing Data**
-
-**Exercise: Normalizing Data**
-
-**Exercise: Basketball Statistics Script**
-
-
-
-
-
-# Finding Natural Patterns in the Data
-
-## Clustering Basketball Players
-
-**Video: Clustering Basketball Players**
-
-## Low Dimensional Visualization
-
-
-
-
-
-To effectively visualize the data containing more than three predictors, you can use the dimensionality reduction techniques such as multidimensional scaling and principal component analysis (PCA).
-
-
-
-
-**Exercise: Classical Multidimensional Scaling**
-
-**Exercise: Nonclassical Multidimensional Scaling**
-
-
-
-
-
-
-
-
-
-
-
-**Exercise: Basketball Players**
-
-## k-Means Clustering
-
-
-
-**Video: What is k-Means Clustering**
-
-
-
-
-
-
-
-
-**Exercise: k-Means Clustering**
-
-**Exercise: Options for k-Means Clustering**
-
-**Exercise: Basketball Players**
-
-
-## Gaussian Mixture Models
-
-
-
-**Video: What are Gaussian Mixture Models**
-
-
-
-
-**Exercise: GMM Clustering**
-
-**Exercise: Basketball Players**
-
-## Interpreting the Clusters
-
-
-
-
-
-
-
-
-**Exercise: parallelcoords**
-
-**Exercise: cross-tabulation**
-
-
-
-
-
-
-**Exercise: Silhouette Plots**
-
-**Exercise: Basketball Players**
-
-## Hierarchical Clustering
-
-
-
-**Video: Hierarchical Clustering**
-
-
-
-
-
-**Exercise: Determine Hierarchical Structure**
-
-**Exercise: Divide Hierarchical Tree into Clusters**
-
-**Exercise: Basketball Players**
-
-# Building Classification Models
-
-## Course Example: Heart Disease
-
-**Video: Heart Disease Classification**
-
-## Preparing Data
-
-
-
-**Video: Training and Validation Data**
-
-
-
-
-
-
-
-
-**Exercise: Making Training and Test Sets**
-
-**Exercise: Heart Health**
-
-## Fitting and Predicting
-
-
-
-
-
-
-
-
-
-
-
-**Exercise: Fitting and Predicting**
-
-**Exercise: Using a Classification Variable**
-
-## Evaluating the Classification
-
-
-
-
-
-**Exercise: Prediction and Resubstitution Loss**
-
-
-
-
-
-
-**Exercise: Confusion and Cost Matrix**
-
-**Exercise: Heart Health**
-
-# Classification Methods
-
-## Course Example: Different Methods to Classify Heart Patients
-
-
-
-
-
-
-**Video: Classification Learner App**
-
-**Exercise: Classification Learner App**
-
-## Nearest Neighbor Classification
-
-
-
-**Video: What is k-NN?**
-
-
-
-**Exercise: Using Nearest Neighbor Classification with Tables**
-
-
-
-**Exercise: Heart Health**
-
-## Classification Trees
-
-
-
-**Video: What is a Classification Tree?**
-
-
-
-**Exercise: Using Classification Trees**
-
-**Exercise: Heart Health – Numeric Data**
-
-**Exercise: Heart Health – Numeric and Categorical Data**
-
-## Naive Bayes Classification
-
-
-
-**Video: What is Naive Bayes?**
-
-
-
-**Exercise: Using Naive Bayes Classification**
-
-**Exercise: Heart Health – Numeric Data**
-
-**Exercise: Heart Health – Numeric and Categorical Data**
-
-## Discriminant Analysis
-
-
-
-**Video: What is Discriminant Analysis??**
-
-
-
-
-**Exercise: Using Discriminant Analysis**
-
-**Exercise: Heart Health**
-
-## Support Vector Machines
-
-
-
-**Video: What are Support Vector Machines?**
-
-
-
-**Exercise: Using Support Vector Machine Classification**
-
-**Exercise: Concentric Data**
-
-**Exercise: Heart Health – Numeric Data**
-
-**Exercise: Heart Health – Numeric and Categorical Data**
-
-## Multiclass Support Vector Machines
-
-
-
-
-
-
-
-**Exercise: Using Multiclass Support Vector Machine Classification**
-
-**Exercise: Heart Health – Numeric Data**
-
-**Exercise: Heart Health – Numeric and Categorical Data**
-
-
-# Improving Predictive Models
-
-## Methods for Improving Predictive Models
-
-
-
-
-
-
-
-
-## Cross Validation
-
-
-
-
-**Video: What is Cross Validation?**
-
-
-
-
-
-**Exercise: Cross Validation**
-
-**Exercise: Heart Health**
-
-## Reducing Predictors - Feature Transformation
-
-
-
-
-
-
-
-
-
-
-
-**Exercise: PCA**
-
-**Exercise: Heart Health**
-
-## Reducing Predictors - Feature Selection
-
-
-
-
-
-**Exercise: Built-in Feature Selection**
-
-**Exercise: Heart Health – Built-in Feature Selection**
-
-
-
-
-
-
-
-
-**Exercise: Sequential Feature Selection**
-
-**Exercise: Heart Health – Sequential Feature Selection**
-
-
-
-
-**Exercise: Creating Dummy Variables**
-
-**Exercise: Heart Health – Feature Selection with Categorical Data**
-
-## Ensemble Learning
-
-
-
-
-
-
-**Exercise: Creating Ensembles**
-
-**Exercise: Using Templates**
-
-**Exercise: Heart Health**
-
-# Building Regression Models
-
-## Course Example - Fuel Economy
-
-
-
-
-
-
-## Linear Models
-
-
-
-**Video: What is Linear Regression?**
-
-
-
-
-
-
-**Exercise: Fitting a Line**
-
-**Exercise: Fitting a Polynomial**
-
-**Exercise: Multivariable Linear Regression**
-
-
-
-
-
-**Exercise: Multivariable Linear Regression with Numeric Arrays**
-
-**Exercise: Fuel Economy**
-
-## SVMS and Trees
-
-
-
-**Exercise: Using Tree and SVM Models**
-
-**Exercise: Choosing a Regression Model**
-
-**Exercise: Fuel Economy – Tree**
-
-**Exercise: Fuel Economy – SVM**
-
-## Gaussian Process Regression
-
-
-
-**Video: What is Gaussian Process Regression?**
-
-
-
-
-**Exercise: Using GPR**
-
-**Exercise: Using GPR with Outliers**
-
-**Exercise: Fuel Economy - GPR**
-
-## Regularized Linear Models
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-**Exercise: Ridge Regression**
-
-
-
-
-
-**Exercise: Lasso Regression**
-
-**Exercise: Fuel Economy – Ridge Regression**
-
-**Exercise: Fuel Economy – Lasso Regression**
-
-## Stepwise Fitting
-
-
-
-
-
-
-**Exercise: Stepwise Feature Selection**
-
-**Exercise: Fuel Economy**
-
-# Creating Neural Networks
-
-## Overview of Neural Networks
-
-**Video: What are Neural Networks?**
-
-## Self-Organizing Maps
-
-
-
-
-
-
-**Video: What are Self-Organizing Maps?**
-
-**Video: Interactively Creating SOMs**
-
-
-
-
-
-**Exercise: Interactively Creating SOMs**
-
-**Exercise: Using Commands to Create SOMs **
-
-## Feed-Forward Networks
-
-
-
-
-
-
-
-
-
-
-
-**Video: What are Feed-Forward Networks?**
-
-**Video: Interactively Creating Feed-Forward Networks**
-
-**Exercise: Interactively Creating Feed-Forward Classification Networks**
-
-# Additional Resources
-
-
-**Videos: Introduction to Machine Learning**
-
-- **E-book: Introducing Machine Learning**
-
-- **E-book: Getting Started Machine Learning**
-
-- **E-book: Applying Unsupervised Learning**
-
-- **E-book: Applying Supervised Learning**
-
-**Videos: Applied Machine Learning**
-
-**Machine Learning Quick Start Guide**
-
-**Machine Learning Workflow E-Book**
-
-**Statistics and Machine Learning Toolbox**
-
-**MathWorks Resources**
diff --git a/content/courses/matlab-machine-learning/multi1.png b/content/courses/matlab-machine-learning/multi1.png
deleted file mode 100644
index 41eecf56..00000000
Binary files a/content/courses/matlab-machine-learning/multi1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/multi2.png b/content/courses/matlab-machine-learning/multi2.png
deleted file mode 100644
index 75bafd42..00000000
Binary files a/content/courses/matlab-machine-learning/multi2.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/multi3.png b/content/courses/matlab-machine-learning/multi3.png
deleted file mode 100644
index 98534b95..00000000
Binary files a/content/courses/matlab-machine-learning/multi3.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/multi4.png b/content/courses/matlab-machine-learning/multi4.png
deleted file mode 100644
index d2575997..00000000
Binary files a/content/courses/matlab-machine-learning/multi4.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/multi5.png b/content/courses/matlab-machine-learning/multi5.png
deleted file mode 100644
index 0fc83b94..00000000
Binary files a/content/courses/matlab-machine-learning/multi5.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/nearest1.png b/content/courses/matlab-machine-learning/nearest1.png
deleted file mode 100644
index 0cf4f616..00000000
Binary files a/content/courses/matlab-machine-learning/nearest1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/nearest2.png b/content/courses/matlab-machine-learning/nearest2.png
deleted file mode 100644
index efc8b17d..00000000
Binary files a/content/courses/matlab-machine-learning/nearest2.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/nearest3.png b/content/courses/matlab-machine-learning/nearest3.png
deleted file mode 100644
index e46b0151..00000000
Binary files a/content/courses/matlab-machine-learning/nearest3.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/neural1.png b/content/courses/matlab-machine-learning/neural1.png
deleted file mode 100644
index 70268246..00000000
Binary files a/content/courses/matlab-machine-learning/neural1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/neural2.png b/content/courses/matlab-machine-learning/neural2.png
deleted file mode 100644
index c14d7ee2..00000000
Binary files a/content/courses/matlab-machine-learning/neural2.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/neural3.png b/content/courses/matlab-machine-learning/neural3.png
deleted file mode 100644
index 262bf217..00000000
Binary files a/content/courses/matlab-machine-learning/neural3.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/neural4.png b/content/courses/matlab-machine-learning/neural4.png
deleted file mode 100644
index 32a03b12..00000000
Binary files a/content/courses/matlab-machine-learning/neural4.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/neural5.png b/content/courses/matlab-machine-learning/neural5.png
deleted file mode 100644
index 3707a1a5..00000000
Binary files a/content/courses/matlab-machine-learning/neural5.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/neural6.png b/content/courses/matlab-machine-learning/neural6.png
deleted file mode 100644
index f7a93404..00000000
Binary files a/content/courses/matlab-machine-learning/neural6.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/overview1.png b/content/courses/matlab-machine-learning/overview1.png
deleted file mode 100644
index 426252c7..00000000
Binary files a/content/courses/matlab-machine-learning/overview1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/overview2.png b/content/courses/matlab-machine-learning/overview2.png
deleted file mode 100644
index d1ad970c..00000000
Binary files a/content/courses/matlab-machine-learning/overview2.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/overview3.png b/content/courses/matlab-machine-learning/overview3.png
deleted file mode 100644
index 007e8e5f..00000000
Binary files a/content/courses/matlab-machine-learning/overview3.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/overview4.png b/content/courses/matlab-machine-learning/overview4.png
deleted file mode 100644
index a86d384a..00000000
Binary files a/content/courses/matlab-machine-learning/overview4.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/pca1.png b/content/courses/matlab-machine-learning/pca1.png
deleted file mode 100644
index 8158ec1d..00000000
Binary files a/content/courses/matlab-machine-learning/pca1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/pca2.png b/content/courses/matlab-machine-learning/pca2.png
deleted file mode 100644
index 40b4ea9a..00000000
Binary files a/content/courses/matlab-machine-learning/pca2.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/pca3.png b/content/courses/matlab-machine-learning/pca3.png
deleted file mode 100644
index 6d604891..00000000
Binary files a/content/courses/matlab-machine-learning/pca3.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/pca4.png b/content/courses/matlab-machine-learning/pca4.png
deleted file mode 100644
index 84c7734d..00000000
Binary files a/content/courses/matlab-machine-learning/pca4.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/pca5.png b/content/courses/matlab-machine-learning/pca5.png
deleted file mode 100644
index bffc75f2..00000000
Binary files a/content/courses/matlab-machine-learning/pca5.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/pca6.png b/content/courses/matlab-machine-learning/pca6.png
deleted file mode 100644
index 7a986a7e..00000000
Binary files a/content/courses/matlab-machine-learning/pca6.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/pca7.png b/content/courses/matlab-machine-learning/pca7.png
deleted file mode 100644
index a915c042..00000000
Binary files a/content/courses/matlab-machine-learning/pca7.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/pca8.png b/content/courses/matlab-machine-learning/pca8.png
deleted file mode 100644
index 24819150..00000000
Binary files a/content/courses/matlab-machine-learning/pca8.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/pred1.png b/content/courses/matlab-machine-learning/pred1.png
deleted file mode 100644
index e035b6da..00000000
Binary files a/content/courses/matlab-machine-learning/pred1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/pred2.png b/content/courses/matlab-machine-learning/pred2.png
deleted file mode 100644
index 4b59257c..00000000
Binary files a/content/courses/matlab-machine-learning/pred2.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/pred3.png b/content/courses/matlab-machine-learning/pred3.png
deleted file mode 100644
index e90d360e..00000000
Binary files a/content/courses/matlab-machine-learning/pred3.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/pred4.png b/content/courses/matlab-machine-learning/pred4.png
deleted file mode 100644
index 9ea70bac..00000000
Binary files a/content/courses/matlab-machine-learning/pred4.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/predmdl1.png b/content/courses/matlab-machine-learning/predmdl1.png
deleted file mode 100644
index 5a11aa0b..00000000
Binary files a/content/courses/matlab-machine-learning/predmdl1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/predmdl2.png b/content/courses/matlab-machine-learning/predmdl2.png
deleted file mode 100644
index 9b5ed27c..00000000
Binary files a/content/courses/matlab-machine-learning/predmdl2.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/predmdl3.png b/content/courses/matlab-machine-learning/predmdl3.png
deleted file mode 100644
index cd5c4730..00000000
Binary files a/content/courses/matlab-machine-learning/predmdl3.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/predmdl4.png b/content/courses/matlab-machine-learning/predmdl4.png
deleted file mode 100644
index 6b6697dd..00000000
Binary files a/content/courses/matlab-machine-learning/predmdl4.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/predmdl5.png b/content/courses/matlab-machine-learning/predmdl5.png
deleted file mode 100644
index fd96e1c4..00000000
Binary files a/content/courses/matlab-machine-learning/predmdl5.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/predmdl6.png b/content/courses/matlab-machine-learning/predmdl6.png
deleted file mode 100644
index f64a09b5..00000000
Binary files a/content/courses/matlab-machine-learning/predmdl6.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpred1.png b/content/courses/matlab-machine-learning/redpred1.png
deleted file mode 100644
index 1963d789..00000000
Binary files a/content/courses/matlab-machine-learning/redpred1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpred2.png b/content/courses/matlab-machine-learning/redpred2.png
deleted file mode 100644
index 5dbe4e6f..00000000
Binary files a/content/courses/matlab-machine-learning/redpred2.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpred3.png b/content/courses/matlab-machine-learning/redpred3.png
deleted file mode 100644
index 86fe7a82..00000000
Binary files a/content/courses/matlab-machine-learning/redpred3.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpred4.png b/content/courses/matlab-machine-learning/redpred4.png
deleted file mode 100644
index 4487fbcf..00000000
Binary files a/content/courses/matlab-machine-learning/redpred4.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpred5.png b/content/courses/matlab-machine-learning/redpred5.png
deleted file mode 100644
index d46bcb47..00000000
Binary files a/content/courses/matlab-machine-learning/redpred5.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpred6.png b/content/courses/matlab-machine-learning/redpred6.png
deleted file mode 100644
index f564c499..00000000
Binary files a/content/courses/matlab-machine-learning/redpred6.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpred7.png b/content/courses/matlab-machine-learning/redpred7.png
deleted file mode 100644
index d4a73729..00000000
Binary files a/content/courses/matlab-machine-learning/redpred7.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpred8.png b/content/courses/matlab-machine-learning/redpred8.png
deleted file mode 100644
index 6c1a72d0..00000000
Binary files a/content/courses/matlab-machine-learning/redpred8.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpred9.png b/content/courses/matlab-machine-learning/redpred9.png
deleted file mode 100644
index 60cb3995..00000000
Binary files a/content/courses/matlab-machine-learning/redpred9.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpredf1.png b/content/courses/matlab-machine-learning/redpredf1.png
deleted file mode 100644
index deec587c..00000000
Binary files a/content/courses/matlab-machine-learning/redpredf1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpredf10.png b/content/courses/matlab-machine-learning/redpredf10.png
deleted file mode 100644
index c6ce6b7d..00000000
Binary files a/content/courses/matlab-machine-learning/redpredf10.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpredf11.png b/content/courses/matlab-machine-learning/redpredf11.png
deleted file mode 100644
index 336ce082..00000000
Binary files a/content/courses/matlab-machine-learning/redpredf11.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpredf2.png b/content/courses/matlab-machine-learning/redpredf2.png
deleted file mode 100644
index be358251..00000000
Binary files a/content/courses/matlab-machine-learning/redpredf2.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpredf3.png b/content/courses/matlab-machine-learning/redpredf3.png
deleted file mode 100644
index d4235ec7..00000000
Binary files a/content/courses/matlab-machine-learning/redpredf3.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpredf4.png b/content/courses/matlab-machine-learning/redpredf4.png
deleted file mode 100644
index 4be8f863..00000000
Binary files a/content/courses/matlab-machine-learning/redpredf4.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpredf5.png b/content/courses/matlab-machine-learning/redpredf5.png
deleted file mode 100644
index fe773853..00000000
Binary files a/content/courses/matlab-machine-learning/redpredf5.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpredf6.png b/content/courses/matlab-machine-learning/redpredf6.png
deleted file mode 100644
index 68a0cf9c..00000000
Binary files a/content/courses/matlab-machine-learning/redpredf6.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpredf7.png b/content/courses/matlab-machine-learning/redpredf7.png
deleted file mode 100644
index 7a6d7054..00000000
Binary files a/content/courses/matlab-machine-learning/redpredf7.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpredf8.png b/content/courses/matlab-machine-learning/redpredf8.png
deleted file mode 100644
index fae3e926..00000000
Binary files a/content/courses/matlab-machine-learning/redpredf8.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/redpredf9.png b/content/courses/matlab-machine-learning/redpredf9.png
deleted file mode 100644
index 6d009026..00000000
Binary files a/content/courses/matlab-machine-learning/redpredf9.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/reglinregr1.png b/content/courses/matlab-machine-learning/reglinregr1.png
deleted file mode 100644
index 1e6c5d35..00000000
Binary files a/content/courses/matlab-machine-learning/reglinregr1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/reglinregr10.png b/content/courses/matlab-machine-learning/reglinregr10.png
deleted file mode 100644
index 94e97133..00000000
Binary files a/content/courses/matlab-machine-learning/reglinregr10.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/reglinregr11.png b/content/courses/matlab-machine-learning/reglinregr11.png
deleted file mode 100644
index 2cbb5fba..00000000
Binary files a/content/courses/matlab-machine-learning/reglinregr11.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/reglinregr12.png b/content/courses/matlab-machine-learning/reglinregr12.png
deleted file mode 100644
index acb0ff74..00000000
Binary files a/content/courses/matlab-machine-learning/reglinregr12.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/reglinregr13.png b/content/courses/matlab-machine-learning/reglinregr13.png
deleted file mode 100644
index 3c84b62b..00000000
Binary files a/content/courses/matlab-machine-learning/reglinregr13.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/reglinregr14.png b/content/courses/matlab-machine-learning/reglinregr14.png
deleted file mode 100644
index 4d186f7a..00000000
Binary files a/content/courses/matlab-machine-learning/reglinregr14.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/reglinregr2.png b/content/courses/matlab-machine-learning/reglinregr2.png
deleted file mode 100644
index 6332e330..00000000
Binary files a/content/courses/matlab-machine-learning/reglinregr2.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/reglinregr3.png b/content/courses/matlab-machine-learning/reglinregr3.png
deleted file mode 100644
index caade6d1..00000000
Binary files a/content/courses/matlab-machine-learning/reglinregr3.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/reglinregr4.png b/content/courses/matlab-machine-learning/reglinregr4.png
deleted file mode 100644
index d4f64a0f..00000000
Binary files a/content/courses/matlab-machine-learning/reglinregr4.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/reglinregr5.png b/content/courses/matlab-machine-learning/reglinregr5.png
deleted file mode 100644
index c0deb95a..00000000
Binary files a/content/courses/matlab-machine-learning/reglinregr5.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/reglinregr6.png b/content/courses/matlab-machine-learning/reglinregr6.png
deleted file mode 100644
index 3ca3db06..00000000
Binary files a/content/courses/matlab-machine-learning/reglinregr6.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/reglinregr7.png b/content/courses/matlab-machine-learning/reglinregr7.png
deleted file mode 100644
index 47e6dbbe..00000000
Binary files a/content/courses/matlab-machine-learning/reglinregr7.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/reglinregr8.png b/content/courses/matlab-machine-learning/reglinregr8.png
deleted file mode 100644
index 2ab4f246..00000000
Binary files a/content/courses/matlab-machine-learning/reglinregr8.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/reglinregr9.png b/content/courses/matlab-machine-learning/reglinregr9.png
deleted file mode 100644
index 0d52f7cb..00000000
Binary files a/content/courses/matlab-machine-learning/reglinregr9.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/regresssvm1.png b/content/courses/matlab-machine-learning/regresssvm1.png
deleted file mode 100644
index cd871c36..00000000
Binary files a/content/courses/matlab-machine-learning/regresssvm1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/scaling1.png b/content/courses/matlab-machine-learning/scaling1.png
deleted file mode 100644
index 577b6ada..00000000
Binary files a/content/courses/matlab-machine-learning/scaling1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/scaling2.png b/content/courses/matlab-machine-learning/scaling2.png
deleted file mode 100644
index dc616c5e..00000000
Binary files a/content/courses/matlab-machine-learning/scaling2.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/som1.png b/content/courses/matlab-machine-learning/som1.png
deleted file mode 100644
index 98b2a751..00000000
Binary files a/content/courses/matlab-machine-learning/som1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/som2.png b/content/courses/matlab-machine-learning/som2.png
deleted file mode 100644
index 04d459b6..00000000
Binary files a/content/courses/matlab-machine-learning/som2.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/som3.png b/content/courses/matlab-machine-learning/som3.png
deleted file mode 100644
index d1f7e745..00000000
Binary files a/content/courses/matlab-machine-learning/som3.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/steplin1.png b/content/courses/matlab-machine-learning/steplin1.png
deleted file mode 100644
index f448300e..00000000
Binary files a/content/courses/matlab-machine-learning/steplin1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/steplin2.png b/content/courses/matlab-machine-learning/steplin2.png
deleted file mode 100644
index ce7a26ea..00000000
Binary files a/content/courses/matlab-machine-learning/steplin2.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/steplin3.png b/content/courses/matlab-machine-learning/steplin3.png
deleted file mode 100644
index b88f49a4..00000000
Binary files a/content/courses/matlab-machine-learning/steplin3.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/steplin4.png b/content/courses/matlab-machine-learning/steplin4.png
deleted file mode 100644
index 4102be6a..00000000
Binary files a/content/courses/matlab-machine-learning/steplin4.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/svm1.png b/content/courses/matlab-machine-learning/svm1.png
deleted file mode 100644
index 656b1837..00000000
Binary files a/content/courses/matlab-machine-learning/svm1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/svm2.png b/content/courses/matlab-machine-learning/svm2.png
deleted file mode 100644
index 828780c9..00000000
Binary files a/content/courses/matlab-machine-learning/svm2.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/tree1.png b/content/courses/matlab-machine-learning/tree1.png
deleted file mode 100644
index f0fd41e8..00000000
Binary files a/content/courses/matlab-machine-learning/tree1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/tree2.png b/content/courses/matlab-machine-learning/tree2.png
deleted file mode 100644
index 1110f128..00000000
Binary files a/content/courses/matlab-machine-learning/tree2.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/visualize1.png b/content/courses/matlab-machine-learning/visualize1.png
deleted file mode 100644
index 77d0e07e..00000000
Binary files a/content/courses/matlab-machine-learning/visualize1.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/visualize2.png b/content/courses/matlab-machine-learning/visualize2.png
deleted file mode 100644
index bc1f39d1..00000000
Binary files a/content/courses/matlab-machine-learning/visualize2.png and /dev/null differ
diff --git a/content/courses/matlab-machine-learning/visualize3.png b/content/courses/matlab-machine-learning/visualize3.png
deleted file mode 100644
index a51fb5e3..00000000
Binary files a/content/courses/matlab-machine-learning/visualize3.png and /dev/null differ
diff --git a/content/courses/matlab-programming/_index.md b/content/courses/matlab-programming/_index.md
index 0527b713..1e4bc9e9 100644
--- a/content/courses/matlab-programming/_index.md
+++ b/content/courses/matlab-programming/_index.md
@@ -1,7 +1,7 @@
---
title : "Programming in MATLAB"
summary : "An introduction to programming in MATLAB"
-authors : [uvarc]
+authors : [teh]
date : "2019-06-23T08:37:46-05:00"
categories : ["MATLAB","Programming"]
tags : ["MATLAB","Programming"]
@@ -21,9 +21,9 @@ Participants must have a Mathworks account in order to access the links to the o
**Matlab Academy:Fundamentals of Matlab**
+**Video of Fundamentals of Matlab**
-**Video: Fundamentals of Matlab**
-**Quick Reference Guide (used for presentation)**
+**Quick Reference Guide**
diff --git a/content/courses/matlab-programming/getting_started.md b/content/courses/matlab-programming/getting_started.md
index fbbb16ed..89f1ecbd 100644
--- a/content/courses/matlab-programming/getting_started.md
+++ b/content/courses/matlab-programming/getting_started.md
@@ -19,7 +19,8 @@ menu:
### Getting Data into Matlab
-**Video: Import Tool**
+**Video: Import Tool**
+
**Exercise: Using the Variable Editor**
@@ -27,7 +28,7 @@ menu:
### Obtaining Help
-**Video: Using MATLAB's Documentation**
+**Video: Using MATLAB's Documentation**
**Exercise: Open and Use Function Documentation**
diff --git a/content/courses/matlab-programming/techniques.md b/content/courses/matlab-programming/techniques.md
index af3490a3..388b3a1e 100644
--- a/content/courses/matlab-programming/techniques.md
+++ b/content/courses/matlab-programming/techniques.md
@@ -10,18 +10,24 @@ menu:
parent: Effective MATLAB Programming
---
-# Matlab Programming Review
+## Matlab Programming Review
The section is a review of the basic programming constructs presented in the
**Matlab Fundamentals** tutorial.
-## Logical operations
+### Logical operations
+
+If expression, statements, end evaluates an expression, and executes a group of statements when the expression is true. An expression is true when its result is nonempty and contains only nonzero elements (logical or real numeric). Otherwise, the expression is false.
+
+The elseif and else blocks are optional. The statements execute only if previous expressions in the if...end block are false. An if block can include multiple elseif blocks.
+
+This information is derived from the documentation linked below.
**[Documentation: if,elseif,else](https://www.mathworks.com/help/matlab/ref/if.html)** **[Exercise: Logical Operations](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlbe#chapter=8&lesson=2§ion=2)** -## Decision Branching +### Decision Branching {{< figure src="/courses/matlab-programming/branching1.png" >}}
@@ -32,7 +38,7 @@ The section is a review of the basic programming constructs presented in the **[Exercise: Using switch-case](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlbe#chapter=14&lesson=3§ion=10)** -## For Loops +### For Loops **[Video: For Loops](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlbe#chapter=14&lesson=5§ion=2)** @@ -40,9 +46,10 @@ The section is a review of the basic programming constructs presented in the {{< figure src="/courses/matlab-programming/forLoop1.png" >}}
+ **[Exercise: Looping Through a Vector](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlbe#chapter=14&lesson=5§ion=6)** -## While Loops +### While Loops To use a for loop, you need to know in advance how many iterations are required. If you want to execute a block of code repeatedly until a result is achieved, you can use a while-loop.
@@ -55,14 +62,16 @@ To use a for loop, you need to know in advance how many iterations are required.
**[Exercise: Finding eps](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlbe#chapter=14&lesson=6§ion=7)** -## Creating and Calling Functions +### Creating and Calling Functions
+ **[Slide: Creating and Calling Functions](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlbe#chapter=15&lesson=2§ion=1)**
{{< figure src="/courses/matlab-programming/function1.png" >}}
+ **[Exercise: Create and Call a Function](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlbe#chapter=15&lesson=2§ion=4)** **[Slide: Creating Functions Files](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlbe#chapter=15&lesson=3§ion=2)** @@ -79,30 +88,35 @@ To use a for loop, you need to know in advance how many iterations are required. {{< figure src="/courses/matlab-programming/function3.png" >}}
-# Storing Heterogeneous Data -## Matlab Data Types +## Storing Heterogeneous Data + +### Matlab Data Types
+ **[Slide: MATLAB Data Types](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=2&lesson=1§ion=1)** **[Exercise: Creating Variables of a Specific Data Type](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=2&lesson=1§ion=3)** -## Table Basics +### Table Basics
+ **[Documentation: Tables](https://www.mathworks.com/help/matlab/ref/table.html)** **[Exercise: Create a Table from Workspace Variables](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=2&lesson=2§ion=1)** -## Extracting Data from a Table +### Extracting Data from a Table
+ **[Exercise: Extracting Data from a Table: Dot Notation](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=2&lesson=5§ion=1)** **[Exercise: Extracting Data from a Table: Curly Braces](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=2&lesson=5§ion=2)** -## Cell Array Basics +### Cell Array Basics
+ **[Documentation: Cell Arrays](https://www.mathworks.com/help/matlab/ref/cell.html)** **[Exercise: Creating and Concatenating Cell Arrays](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=2&lesson=6§ion=1)** @@ -110,9 +124,10 @@ To use a for loop, you need to know in advance how many iterations are required. **[Exercise: Cell Array Extraction](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=2&lesson=7§ion=1)** -## Structures +### Structures
+ **[Documentation: Structures](https://www.mathworks.com/help/matlab/ref/struct.html)** **[Exercise: Create a Structure and Access Fields](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=2&lesson=8§ion=1)** @@ -121,11 +136,12 @@ To use a for loop, you need to know in advance how many iterations are required. -# Structuring Heterogeneous Data +## Structuring Heterogeneous Data -## Structuring Data Considerations +### Structuring Data Considerations
+ **[Slide: Structuring Data Considerations](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=3&lesson=1§ion=1)** **[Exercise: Cell Array](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=3&lesson=1§ion=3)** @@ -134,8 +150,9 @@ To use a for loop, you need to know in advance how many iterations are required. **[Exercise Table](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=3&lesson=1§ion=8)** -## Extracting Multiple Elements from Cell and Structure arrays +### Extracting Multiple Elements from Cell and Structure arrays
+ **[Slide: Gathering Output](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=3&lesson=2§ion=2)** **[Exercise: Multiple Elements from a Cell Array](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=3&lesson=2§ion=4)** @@ -143,8 +160,9 @@ To use a for loop, you need to know in advance how many iterations are required. **[Exercise: Multiple Elements from a Structure Array](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=3&lesson=2§ion=3)**
-## Function Handles +### Function Handles
+ **[Using Function Handles](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=3&lesson=3§ion=1)** **[Exercise: Optimization](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=3&lesson=3§ion=4)** @@ -155,6 +173,7 @@ To use a for loop, you need to know in advance how many iterations are required.
+ **[Video: Applying Scalar Functions to Arrays](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=3&lesson=4§ion=2)** **[Exercise: Applying Functions to Groups](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=3&lesson=4§ion=3)** @@ -162,16 +181,18 @@ To use a for loop, you need to know in advance how many iterations are required. **[Exercise: cellfun Basics](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=3&lesson=4§ion=8)** -## Converting Data Types +### Converting Data Types
+ **[Slide: Converting Data Types](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=3&lesson=5§ion=1)** **[Exercise: Numeric to Cell Array](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=3&lesson=5§ion=2)** **[Exercise: Converting Strings](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=3&lesson=5§ion=5)** -# Managing Data Efficiently +## Managing Data Efficiently
+ **[Slide: Datatypes and Memory](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=4&lesson=1§ion=1)** **[Quiz 1](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=4&lesson=1§ion=2)** @@ -184,76 +205,86 @@ To use a for loop, you need to know in advance how many iterations are required. **[Exercise: Preallocation Experiment](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=4&lesson=2§ion=5)** -## Vectorization +### Vectorization
{{< figure src="/courses/matlab-programming/vectorization1.png" >}} {{< figure src="/courses/matlab-programming/vectorization2.png" >}} -## Copy-on-write with Function Parameters +### Copy-on-write with Function Parameters
+ **[Slide: Copy-on-write Behavior](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=4&lesson=6§ion=1)** -## In-place Optimizations +### In-place Optimizations
+ **[Slide: In-place Optimizations](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=4&lesson=7§ion=1)** -## Nested Functions +### Nested Functions {{< figure src="/courses/matlab-programming/nestedFunction1.png" >}}
+ **[Slide: Nested Functions](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=4&lesson=8§ion=2)** **[Exercise: Create a Nested Functions](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=4&lesson=8§ion=5)** -# Creating Flexible functions +## Creating Flexible functions
+ **[Video: Creating Flexible Function Interfaces](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=5&lesson=1§ion=1)** -## Creating Multiple Interfaces with Wrapper Functions +### Creating Multiple Interfaces with Wrapper Functions
+ **[Slide: Separating the Interface from the Algorithm](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=5&lesson=2§ion=1)** **[Exercise: Create a Wrapper Function with a Fixed Interface](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=5&lesson=2§ion=3)** -## Setting Default Input Values +### Setting Default Input Values
+ **[Slide: Setting Default Input Values](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=5&lesson=5§ion=1)** -## Missing Input Arguments +### Missing Input Arguments
+ **[Slide: Missing Input Arguments](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=5&lesson=5§ion=2)** **[Slide: Empty Input Arguments](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=5&lesson=5§ion=6)** **[Exercise: Skipping the Input](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=5&lesson=5§ion=7)** -## Allowing Any Number of Inputs +### Allowing Any Number of Inputs
+ **[Slide: Functions with a Variable Number of Inputs](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=5&lesson=6§ion=1)** **[Slide: Variable Length Input Argument List](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=5&lesson=6§ion=2)** **[Slide: Passing Argument Lists to Another Function](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=5&lesson=6§ion=6)** -## Allowing a Variable Number of outputs +### Allowing a Variable Number of outputs
+ **[Slide: Defining Different Behaviors Based on Outputs](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=5&lesson=8§ion=1)** **[Exercise: Matrix or Vector Output](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=5&lesson=8§ion=2)** -## Changing the Function Interface with Anonymous Functions +### Changing the Function Interface with Anonymous Functions
+ **[Slide: Modifying Function Interfaces](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=5&lesson=9§ion=1)** **[Slide: Wrapping Functions with Anonymous Functions](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=5&lesson=9§ion=2)** @@ -264,77 +295,86 @@ To use a for loop, you need to know in advance how many iterations are required. **[Exercise: Change Function Interface with an Anonymous Function](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=5&lesson=9§ion=5)** -# Creating Robust Applications +## Creating Robust Applications -## Restricting Access Using Private Functions +### Restricting Access Using Private Functions {{< figure src="/courses/matlab-programming/privateFunction1.png" >}}
+ **[Slide: Making Functions Private](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=6&lesson=2§ion=2)**
{{< figure src="/courses/matlab-programming/privateFunction2.png" >}} -## Writing Local Functions +### Writing Local Functions {{< figure src="/courses/matlab-programming/localFunction1.png" >}} {{< figure src="/courses/matlab-programming/localFunction2.png" >}} {{< figure src="/courses/matlab-programming/localFunction3.png" >}} -## Comparison of Functions +### Comparison of Functions {{< figure src="/courses/matlab-programming/compareFunction1.png" >}}
+ **[Exercise: Create Local Functions](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=6&lesson=3§ion=4)** -## Validating Function Inputs +### Validating Function Inputs {{< figure src="/courses/matlab-programming/validateInput1.png" >}} {{< figure src="/courses/matlab-programming/validateInput2.png" >}}
+ **[Quiz](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=6&lesson=5§ion=4)** -# Verifying Application Behavior +## Verifying Application Behavior -## Why Use a Testing Framework? +### Why Use a Testing Framework?
+ **[Video: Why Use a Testing Framework](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=7&lesson=1§ion=1)** -## What is a Test +### What is a Test {{< figure src="/courses/matlab-programming/test1.png" >}}
+ **[Slide: Elements of a Test](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=7&lesson=2§ion=1)** -### "is" Functions +#### "is" Functions {{< figure src="/courses/matlab-programming/isFunction.png" >}}
+ **[Exercise: isequal Versus ==](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=7&lesson=2§ion=3)** -### Test Response +#### Test Response {{< figure src="/courses/matlab-programming/assertFunction.png" >}}
+ **[Exercise: assert](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=7&lesson=2§ion=5)** -## Writing and Running a Test Script +### Writing and Running a Test Script {{< figure src="/courses/matlab-programming/testScript1.png" >}} -### Writing a Test Script +#### Writing a Test Script {{< figure src="/courses/matlab-programming/testScript2.png" >}}
+ **[Exercise: Write a Test Script](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=7&lesson=3§ion=3)** -### Running a Test Script +#### Running a Test Script
+ **[Slide: Running a Test Script](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=7&lesson=3§ion=4)** **[Exercise: Run a Test Script](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=7&lesson=3§ion=5)** @@ -342,48 +382,55 @@ To use a for loop, you need to know in advance how many iterations are required. **[Exercise: Fix Broken Tests](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=7&lesson=3§ion=8)** -# Utilizing Development Tools +## Utilizing Development Tools -## Developing and Maintaining Code +### Developing and Maintaining Code {{< figure src="/courses/matlab-programming/devTools1.png" >}} -## Folder Reports +### Folder Reports
+ **[Slide: Folder Reports](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=8&lesson=2§ion=2)** -## Errors and Debugging + +### Errors and Debugging
+ **[Video: Different Kinds of Errors](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=8&lesson=3§ion=1)** -## Code Analyzer +### Code Analyzer {{< figure src="/courses/matlab-programming/codeAnalyzer1.png" >}}
+ **[Slide: Suppressing and Fixing Code Analyzer Warnings](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=8&lesson=4§ion=2)** **[Exercise: Remove Code Analyzer Warnings](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=8&lesson=4§ion=6)** -## Debugging Runtime Errors +### Debugging Runtime Errors {{< figure src="/courses/matlab-programming/debugRun1.png" >}}
+ **[Slide: Debugging Run-Time Errors](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=8&lesson=5§ion=3)** -## Measuring Performance +### Measuring Performance {{< figure src="/courses/matlab-programming/performance1.png" >}}
+ **[Slide: Tic and Toc](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=8&lesson=6§ion=2)** -## Finding Bottlenecks +### Finding Bottlenecks {{< figure src="/courses/matlab-programming/bottleneck1.png" >}}
+ **[Video: The MATLAB Profiler](https://matlabacademy.mathworks.com/R2019a/portal.html?course=mlpr#chapter=8&lesson=7§ion=2)** diff --git a/content/courses/opencv/clown-copy.png b/content/courses/opencv/clown-copy.png deleted file mode 100644 index b3cec0bf..00000000 Binary files a/content/courses/opencv/clown-copy.png and /dev/null differ diff --git a/content/courses/opencv/clown-denoised.png b/content/courses/opencv/clown-denoised.png deleted file mode 100644 index 52f345de..00000000 Binary files a/content/courses/opencv/clown-denoised.png and /dev/null differ diff --git a/content/courses/opencv/clown-fixed.png b/content/courses/opencv/clown-fixed.png deleted file mode 100644 index 10c3dcaa..00000000 Binary files a/content/courses/opencv/clown-fixed.png and /dev/null differ diff --git a/content/courses/opencv/clown-merged.png b/content/courses/opencv/clown-merged.png deleted file mode 100644 index be26b91f..00000000 Binary files a/content/courses/opencv/clown-merged.png and /dev/null differ diff --git a/content/courses/opencv/clown-noisy-denoised.png b/content/courses/opencv/clown-noisy-denoised.png deleted file mode 100644 index 8da7badd..00000000 Binary files a/content/courses/opencv/clown-noisy-denoised.png and /dev/null differ diff --git a/content/courses/opencv/clown-noisy.png b/content/courses/opencv/clown-noisy.png deleted file mode 100644 index 2e299c7c..00000000 Binary files a/content/courses/opencv/clown-noisy.png and /dev/null differ diff --git a/content/courses/opencv/clown-proportional.png b/content/courses/opencv/clown-proportional.png deleted file mode 100644 index 789d533f..00000000 Binary files a/content/courses/opencv/clown-proportional.png and /dev/null differ diff --git a/content/courses/opencv/clown-resized.png b/content/courses/opencv/clown-resized.png deleted file mode 100644 index 6c3baf84..00000000 Binary files a/content/courses/opencv/clown-resized.png and /dev/null differ diff --git a/content/courses/opencv/clown-split.png b/content/courses/opencv/clown-split.png deleted file mode 100644 index c6d5821d..00000000 Binary files a/content/courses/opencv/clown-split.png and /dev/null differ diff --git a/content/courses/opencv/clown.png b/content/courses/opencv/clown.png deleted file mode 100644 index 230821d9..00000000 Binary files a/content/courses/opencv/clown.png and /dev/null differ diff --git a/content/courses/opencv/data/opencv-examples.zip b/content/courses/opencv/data/opencv-examples.zip deleted file mode 100644 index 1e47754a..00000000 Binary files a/content/courses/opencv/data/opencv-examples.zip and /dev/null differ diff --git a/content/courses/opencv/fluorescent-cells.png b/content/courses/opencv/fluorescent-cells.png deleted file mode 100644 index 3e1718cd..00000000 Binary files a/content/courses/opencv/fluorescent-cells.png and /dev/null differ diff --git a/content/courses/opencv/image-segmented.png b/content/courses/opencv/image-segmented.png deleted file mode 100644 index 538eeb8a..00000000 Binary files a/content/courses/opencv/image-segmented.png and /dev/null differ diff --git a/content/courses/opencv/index.md b/content/courses/opencv/index.md deleted file mode 100644 index 5e7b0022..00000000 --- a/content/courses/opencv/index.md +++ /dev/null @@ -1,630 +0,0 @@ ---- -title: "Scientific Image Processing with Python OpenCV" -summary: "An introduction to scientific image processing with the Python OpenCV package. Topics include splitting and merging of color channels, morphological filters, image thresholding and segmentation." -author: [khs] -categories: ["Image Processing","Python"] - -highlight_style: "github" -date: 2022-10-27T00:00:00-05:00 -toc: true -type: article -draft: false ---- -
-
-
-
-# Introduction
-
-From the [OpenCV project documentation](https://docs.opencv.org/master/d1/dfb/intro.html):
-
-> OpenCV (Open Source Computer Vision Library: http://opencv.org) is an open-source library that includes several hundreds of computer vision algorithms.
-
-This workshop assumes a working knowledge of the Python programming language and basic understanding of image processing concepts.
-
-Introductions to Python can be found [here](/courses/programming_python_scientists_engineers/python-interpreter/) and [here](/courses/python_introduction/).
-
----
-
-# Getting Started
-
-**Python code examples**
-
-The Python scripts and data files for this workshop can be [downloaded from here](/notes/opencv/data/opencv-examples.zip). On your computer, unzip the downloaded folder and use it as working directory for this workshop.
-
-**Python programming environment**
-
-The Anaconda environment from [Anaconda Inc.](https://anaconda.com/) is widely used because it bundles a Python interpreter, most of the popular packages, and development environments. It is cross-platform and freely available. There are two somewhat incompatible versions of Python; version 2.7 is deprecated but still fairly widely used. Version 3 is the supported version.
-
-**Note: We are using Python 3 for this workshop.**
-
-## Option 1: Using the UVA HPC platform
-
-If you have a Rivanna account, you can work through this tutorial using an [Open OnDemand](https://www.rc.virginia.edu/userinfo/rivanna/ood/overview/) Desktop session.
-
-1. Go to https://rivanna-portal.hpc.virginia.edu.
-
-2. Log in with your UVA credentials.
-
-3. Go to `Interactive Apps` > `Desktop`
-
-4. On the next screen, specify resources as shown in this screenshot:
-
- 
-
- >**Note:** Workshop participants may specify `rivanna-training` in the `Allocation (SUs)` field. Alternatively, you may use any other Rivanna allocation that you are a member of.
-
-5. Click `Launch` at the bottom of the screen. Your desktop session will be queued up -- this may take a few minutes until the requested resources become available.
-
-## Option 2 - Use your own computer
-
-1. Visit the [Anaconda download website](https://www.anaconda.com/products/individual#Downloads) and download the installer for Python 3 for your operating system (Windows, Mac OSX, or Linux). We recommend to use the graphical installer for ease of use.
-
-2. Launch the downloaded installer, follow the onscreen prompts and install the Anaconda distribution on your local hard drive.
-
-The [Anaconda Documentation](https://docs.anaconda.com/anaconda/user-guide/getting-started/) provides an introduction to the Anaconda environment and bundled applications. For the purpose of this workshop we focus on the `Anaconda Navigator` and `Spyder`.
-
-# Using Anaconda
-
-## Navigator
-
-Once you have installed Anaconda, start the Navigator application:
-* [Instructions for Windows](https://docs.anaconda.com/anaconda/user-guide/getting-started/#open-nav-win)
-* [Instructions for Mac](https://docs.anaconda.com/anaconda/user-guide/getting-started/#open-nav-mac)
-* [Instructions for Linux](https://docs.anaconda.com/anaconda/user-guide/getting-started/#open-nav-lin)
-
-You should see a workspace similar to the screenshot, with several options for working environments, some of which are not installed. We will use `Spyder` which should already be installed. If not, click the button to install the package.
-
-
-
-## Spyder
-
-Now we will switch to Spyder. Spyder is an Integrated Development Environment, or IDE, aimed at Python. It is well suited for developing longer, more modular programs.
-
-1. To start it, return to the `Anaconda Navigator` and click on the `Spyder` tile. It may take a while to open (watch the lower left of the Navigator).
-2. Once it starts, you will see a layout with an editor pane on the left, an explorer pane at the top right, and an iPython console on the lower right. This arrangement can be customized but we will use the default for our examples. Type code into the editor. The explorer window can show files, variable values, and other useful information. The iPython console is a frontend to the Python interpreter itself. It is comparable to a cell in JupyterLab.
-
-
-
-## Installation of OpenCV
-
-It is recommended to install the `opencv-python` package from PyPI using the `pip install` command.
-
-**On your own computer:**
-Start the `Anaconda Prompt` command line tool following the instructions for your operating system.
-* Start Anaconda Prompt on [Windows](https://docs.anaconda.com/anaconda/user-guide/getting-started/#open-prompt-win)
-* Start Anaconda Prompt on [Mac](https://docs.anaconda.com/anaconda/user-guide/getting-started/#open-prompt-mac), or open a terminal window.
-* [Linux:](https://docs.anaconda.com/anaconda/user-guide/getting-started/#open-prompt-lin) Just open a terminal window.
-
-At the prompt, type the following command and press enter/return:
-```bash
-pip install opencv-python matplotlib scikit-image pandas
-```
-This command will install the latest `opencv-python` package version in your current Anaconda Python environment. The `matplotlib` package is used for plotting and image display. It is part of the Anaconda default packages. The `scikit-image` and `pandas` packages are useful for additional image analysis and data wrangling, respectively.
-
-**On Rivanna (UVA's HPC platform):**
-
-[Rivanna](https://www.rc.virginia.edu/userinfo/rivanna/overview/) offers several Anaconda distributions with different Python versions. Before you use Python you need to load one of the **Anaconda** software modules and then run the `pip install` command in a terminal.
-
-```bash
-module load anaconda
-pip install --user opencv-python matplotlib scikit-image pandas
-```
-> **Note:** You have to use the `--user` flag which instructs the interpreter to install the package in your home directory. Alternatively, create your own custom [Conda environment](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) first and run the `pip install opencv-python matplotlib pandas` command in that environment (without the `--user` flag)
-
-To confirm successful package installation, start the **Spyder IDE** by typing the following command in the terminal:
-```bash
-spyder &
-```
-In the **Spyder IDE**, go to the `IPython console` pane, type the following command and press `enter/return`:
-
-```python:
-import cv2
-print (cv2.__version__)
-```
-
-If the package is installed correctly, the output will show the openCV version number.
-
-## Example scripts and images
-
-Download the example scripts and images from [this link](/notes/opencv/data/opencv-examples.zip). Unzip the downloaded file and start your Python IDE, e.g. Spyder.
-
-If you are on Rivanna, run the following command to copy the examples to your home directory:
-```bash
-cp -R /share/resources/tutorials/opencv-examples ~/
-```
-
----
-
-# Basic Operations
-
-## Loading Images
-
-The `imread` function is used to read images from files. Images are represented as a multidimensional [NumPy](https://numpy.org) arrays. Learn more about NumPy arrays [here](/courses/python_introduction/numpy_ndarrays/). The multidimensional properties are stored in an image's `shape` attribute, e.g. number of rows (height) x number of columns (width) x number of channels (depth).
-
-
-```python:
-import cv2
-
-# load the input image and show its dimensions
-image = cv2.imread("clown.png")
-(h, w, d) = image.shape
-print('width={}, height={}, depth={}'.format(w, h, d))
-```
-
-**Output:**
-```
-width=320, height=200, depth=3
-```
-
-## Displaying Images
-
-We can use an openCV function to display the image to our screen.
-
-```python:
-# open with OpenCV and press a key on our keyboard to continue execution
-cv2.imshow('Image', image)
-cv2.waitKey(0)
-cv2.destroyAllWindows()
-```
-
-
-
-The `cv2.imshow()` method displays the image on our screen. The `cv2.waitKey()` function waits for a key to be pressed. This is important otherwise our image would display and immediately disappear before we even see the image. The call of `destroyAllWindows()` should be placed at the end of any script that uses the `imshow` function.
-
->**Note:** Before you run the code through the Spyder IDE, go to `Run` > `Run configuration per file` and select `Execute in dedicated console` first. Then, when you run the code uyou need to actually click the image window opened by OpenCV and press a key on your keyboard to advance the script. OpenCV cannot monitor your terminal for input so if you press a key in the terminal OpenCV will not notice.
-
-Alternatively, we can use the `matplotlib` package to display an image.
-
-```python:
-import matplotlib.pyplot as plt
-
-plt.imshow(cv2.cvtColor(image,cv2.COLOR_BGR2RGB))
-```
-
-> Note that OpenCV stores channels of an RGB image in Blue, Green, Red order. We use the `cv2.cvtColor(image,cv2.COLOR_BGR2RGB)` function to convert from BGR --> RGB channel ordering for display purposes.
-
-
-## Saving Images
-
-We can use the `imwrite()` function to save images. For example:
-
-```python:
-filename = 'clown-copy.png'
-cv2.imwrite(filename, image)
-```
-
-## Accessing Image Pixels
-
-Since an image's underlying pixel information is stored in multidimensional numpy arrays, we can use common numpy operations to slice and dice image regions, including the images channels.
-
-We can use the following code to extract the red, green and blue intensity values of a specific image pixel at position x=100 and y=50.
-
-```
-(b, g, r) = image[100, 50]
-print("red={}, green={}, blue={}".format(r, g, b))
-````
-
-**Output:**
-```
-red=184, green=35, blue=15
-```
-
-> Remember that OpenCV stores the channels of an RGB image in Blue, Green, Red order.
-
-## Slicing and Cropping
-
-It is also very easy to extract a rectangular region of interest from an image and storing it as a cropped copy. Let's extract the pixels for **30<=y<130** and **140<=x<240** from our original image. The resulting cropped image has a width and height of **100x100** pixels.
-
-```python:
-roi = image[30:130,140:240]
-plt.imshow(cv2.cvtColor(roi,cv2.COLOR_BGR2RGB))
-```
-
-## Resizing
-
-It is very easy to resize images. It just takes a single line of code. In this case we are resizing the input image to 500x500 (width x height) pixels.
-
-```python:
-resized = cv2.resize(image,(500,500))
-```
-
-Note that we are _forcing_ the resized image into a square 500x500 pixel format. To avoid distortion of the resized image, we can calculate the height/width `aspect` ratio of the original image and use it to calculate the `new_height = new_width * aspect` ratio (or `new_width = new_height / aspect` ratio).
-
-```python:
-# resize width while preserving height proportions
-height = image.shape[0]
-width = image.shape[1]
-aspect = height/width
-new_width = 640
-new_height = int(new_width * aspect)
-resized2 = cv2.resize(image,(new_width,new_height))
-print (image.shape)
-print (resized2.shape)
-```
-
-```python:
-# display the two resized images
-_,ax = plt.subplots(1,2)
-ax[0].imshow(cv2.cvtColor(resized, cv2.COLOR_BGR2RGB))
-ax[0].axis('off')
-ax[1].imshow(cv2.cvtColor(resized2, cv2.COLOR_BGR2RGB))
-ax[1].axis('off')
-```
-
-
-
-
-
-## Splitting and Merging of Color Channels
-
-The `split()` function provides a convenient way to split multi-channel images (e.g. RGB) into its channel components.
-
-```python:
-# Split color channels
-(B, G, R) = cv2.split(image)
-# create 2x2 grid for displaying images
-_, axarr = plt.subplots(2,2)
-axarr[0,0].imshow(R, cmap='gray')
-axarr[0,0].axis('off')
-axarr[0,0].set_title('red')
-
-axarr[0,1].imshow(G, cmap='gray')
-axarr[0,1].axis('off')
-axarr[0,1].set_title('green')
-
-axarr[1,0].imshow(B, cmap='gray')
-axarr[1,0].axis('off')
-axarr[1,0].set_title('blue')
-
-axarr[1,1].imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
-axarr[1,1].axis('off')
-axarr[1,1].set_title('RGB')
-```
-
-
-
-Let's take the blue and green channel only and merge them back into a new RGB image, effectively masking the red channel. For this we'll define a new numpy array with the same width and height as the original image and a depth of 1 (single channel), all pixels filled with zero values. Since the individual channels of an RGB image are 8-bit numpy arrays, we choose the numpy `uint8` data type.
-
-```python:
-import numpy as np
-zeros = np.zeros((image.shape[0], image.shape[1]), dtype=np.uint8)
-# alternative:
-# zeros = np.zeros_like(B)
-
-print (B.shape, zeros.shape)
-merged = cv2.merge([B, G, zeros])
-_,ax = plt.subplots(1,1)
-ax.imshow(cv2.cvtColor(merged, cv2.COLOR_BGR2RGB))
-ax.axis('off')
-```
-
-
-
-# Exercises
-
-1. In the clown.png image, inspect the pixel value for x=300, y=25.
-2. Crop the clown.png image to a centered rectangle with half the width and half the height of the original.
-3. Extract the green channel, apply the values to the red channel and merge the original blue, original green and new red channel into a new BGR image. Then display this image as an RGB image using matplotlib.
-
----
-
-# Filters
-
-## Denoising
-
-OpenCV provides four convenient built-in [denoising tools](https://docs.opencv.org/3.4/d5/d69/tutorial_py_non_local_means.html):
-
-1. `cv2.fastNlMeansDenoising()` - works with a single grayscale images
-2. `cv2.fastNlMeansDenoisingColored()` - works with a color image.
-3. `cv2.fastNlMeansDenoisingMulti()` - works with image sequence captured in short period of time (grayscale images)
-4. `cv2.fastNlMeansDenoisingColoredMulti()` - same as above, but for color images.
-
-Common arguments are:
-* `h`: parameter deciding filter strength. Higher h value removes noise better, but removes details of image also. (10 may be a good starting point)
-* `hForColorComponents`: same as h, but for color images only. (normally same as h)
-* `templateWindowSize`: should be odd. (recommended 7)
-* `searchWindowSize`: should be odd. (recommended 21)
-
-Let's try this with a noisy version of the clown image. This is a color RGB image and so we'll try the `cv2.fastNlMeansDenoisingColored()` filter. Here is the noisy input image `clown-noisy.png`.
-
-
-
-The `denoising.py` script demonstrates how it works.
-
-```python:
-import cv2
-from matplotlib import pyplot as plt
-
-noisy = cv2.imread('clown-noisy.png')
-
-# define denoising parameters
-h = 15
-hColor = 15
-templateWindowSize = 7
-searchWindowSize = 21
-
-# denoise and save
-denoised = cv2.fastNlMeansDenoisingColored(noisy,None,h,hColor,templateWindowSize,searchWindowSize)
-cv2.imwrite('clown-denoised.png', denoised)
-
-# display
-plt.subplot(121),plt.imshow(cv2.cvtColor(noisy, cv2.COLOR_BGR2RGB), interpolation=None)
-plt.subplot(122),plt.imshow(cv2.cvtColor(denoised, cv2.COLOR_BGR2RGB), interpolation=None)
-plt.show()
-```
-
-
-
-Additional useful filters for smoothing images:
- * `GaussianBlur` - blurs an image using a Gaussian filter
- * `medianBlur` - blurs an image using a median filter
-
-There are many other image smoothing filters [described here](https://docs.opencv.org/4.5.3/dc/dd3/tutorial_gausian_median_blur_bilateral_filter.html).
-
-## Morphological Filters
-
-Morphological filters are used for smoothing, edge detection or extraction of other features. The principal inputs are an image and a structuring element also called a kernel.
-
-The two most basic operations are dilation and erosion on binary images (pixels have value 1 or 0; or 255 and 0). The kernel slides through the image pixel by pixel (as in 2D convolution).
-
-* During _dilation_, a pixel in the original image (either 1 or 0) will be considered 1 **if at least one pixel** under the kernel is 1. The dilation operation is implemented as `cv2.dilate(image,kernel,iterations = n)`.
-* During _erosion_, a pixel in the original image (either 1 or 0) will be considered 1 only **if all the pixels** under the kernel is 1, otherwise it is eroded (set to zero). The erosion operation is implemented as `cv2.erode(image,kernel,iterations = n)`.
-
-The `erode-dilate.py` script provides an example:
-
-```python:
-import cv2
-import numpy as np
-import matplotlib.pyplot as plt
-
-image = cv2.imread('morph-input.png',0)
-# create square shaped 7x7 pixel kernel
-kernel = np.ones((7,7),np.uint8)
-
-# dilate, erode and save results
-dilated = cv2.dilate(image,kernel,iterations = 1)
-eroded = cv2.erode(image,kernel,iterations = 1)
-cv2.imwrite('morph-dilated.png', dilated)
-cv2.imwrite('morph-eroded.png', eroded)
-
-# display results
-_,ax = plt.subplots(1,3)
-ax[0].imshow(image, cmap='gray')
-ax[0].axis('off')
-ax[1].imshow(dilated, cmap='gray')
-ax[1].axis('off')
-ax[2].imshow(eroded, cmap='gray')
-ax[2].axis('off')
-```
-
-Original | Dilation | Erosion
-:-------------------:|:----------------------:|:----------------------:
- |  |  |
-
-> By increasing the kernel size or number of iterations we can dilate or erode more of the original object.
-
-Elementary morphological filters may be chained together to define composite operations.
-
-**Opening** is just another name of erosion followed by dilation. It is useful in removing noise.
-
-```python:
-opening = cv2.morphologyEx(img, cv2.MORPH_OPEN, kernel)
-```
-
-**Closing** is reverse of opening, i.e. dilation followed by erosion. It is useful in closing small holes inside the foreground objects, or small black points on the object.
-
-```
-closing = cv2.morphologyEx(img, cv2.MORPH_CLOSE, kernel)
-```
-
-Original | Opening | Closing
-:-------------------:|:----------------------:|:----------------------:
- |  |  |
-
-> Note how the opening operation removed the small dot in the top right corner. In contrast the closing operation retained that small object and also filled in the black hole in the object on the left side of the input image.
-
-**Morphological Gradients** can be calculated as the difference between dilation and erosion of an image. It is used to reveal object's edges.
-
-```python:
-kernel = np.ones((2,2),np.uint8)
-gradient = cv2.morphologyEx(image, cv2.MORPH_GRADIENT, kernel)
-```
-
-Original | Gradient (edges)
-:-------------------:|:----------------------:
- | 
-
-# Exercises
-
-1. Experiment with different kernel sizes and number of iterations. What do you observe?
-
----
-
-# Segmentation & Quantification
-
-Image Segmentation is the process that groups related pixels to define higher level objects. The following techniques are commonly used to accomplish this task:
-
-1. Thresholding - conversion of input image into binary image
-2. Edge detection - see above
-3. Region based - expand/shrink/merge object boundaries from object seed points
-4. Clustering - use statistical analysis of proximity to group pixels as objects
-5. Watershed - separates touching objects
-6. Artificial Neural Networks - train object recognition from examples
-
-Let's try to identify and measure the area of the nuclei in this image with fluorescent labeled cells. This is the `fluorescent-cells.png` image in the `examples` folder. We will explore the use of morphology filters, thresholding and watershed to accomplish this.
-
-The complete code is in the `segmentation.py` script.
-
-
-
-## Preprocessing
-
-First, we load the image and extract the blue channel which contains the labeling of the nuclei. Since OpenCV reads RGB images in BGR order, the blue channel is at index position 0 of the third image axis.
-
-```python:
-import cv2
-
-image = cv2.imread('fluorescent-cells.png')
-nuclei = image[:,:,0] # get blue channel
-```
-
-
-
-To eliminate noise, we apply a Gaussian filter with 3x3 kernel, then apply the Otsu thresholding algorithm. The thresholding converts the grayscale intensity image into a black and white binary image. The function returns two values, we store them in `ret` (the applied threshold value) and `thresh` (the thresholded black & white binary image). White pixels represent nuclei; black pixel represent background.
-
-```python:
-# apply Gaussian filter to smoothen image, then apply Otsu threshold
-blurred = cv2.GaussianBlur(nuclei, (3, 3), 0)
-ret, thresh = cv2.threshold(blurred,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)
-```
-
-Next, we'll apply an opening operation to exclude small non-nuclear particles in the binary image. Furthermore we use scikit's `clear_border()` function to exclude objects (nuclei) touching the edge of the image.
-
-```python:
-# fill small holes
-import numpy as np
-from skimage.segmentation import clear_border
-
-kernel = np.ones((3,3),np.uint8)
-opening = cv2.morphologyEx(thresh,cv2.MORPH_OPEN,kernel,iterations=7)
-
-# remove image border touching objects
-opening = clear_border(opening)
-```
-
-The resulting image looks like this.
-
-
-
-A tricky issue is that some of the nuclei masks are touching each other. We need to find a way to break up these clumps. We do this in several steps. First, we will dilate the binary nuclei mask. The black areas in the resulting image represent pixels that certainly do not contain any nuclear components. We call it the `sure_bg`.
-
-```python:
-# sure background area
-sure_bg = cv2.dilate(opening,kernel,iterations=10)
-```
-
-
-
-The nuclei are all fully contained inside the white pixel area. The next step is to find estimates for the center of each nucleus. Some of the white regions may contain more than one nucleus and we need to separate the joined ones. We calculate the distance transform to do this.
-
-> The result of the distance transform is a graylevel image that looks similar to the input image, except that the graylevel intensities of points inside foreground (white) regions are changed to show the distance to the closest boundary from each point.
-
-We can use the intensity peaks in the distance transform map (a grayscale image) as proxies and seeds for the individual nuclei. We isolate the peaks by applying a simple threshold. The result is the `sure_fg` image.
-
-```python:
-# calculate distance transform to establish sure foreground
-dist_transform = cv2.distanceTransform(opening,cv2.DIST_L2,5)
-ret, sure_fg = cv2.threshold(dist_transform,0.6*dist_transform.max(),255,0)
-```
-
-Distance Transform | Sure Foreground (nuclei seeds)
-:-------------------------------:|:------------------------:
-|  |  |
-
-
-By subtracting the sure foreground regions from the sure background regions we can identify the regions of unknown association, i.e. the pixels that we have not assigned to be either nuclei or background.
-
-```python:
-# sure_fg is float32, convert to uint8 and find unknown region
-sure_fg = np.uint8(sure_fg)
-unknown = cv2.subtract(sure_bg,sure_fg)
-```
-
-Sure Background | Sure Foreground (nuclei seeds) | Unknown
-:----------------------:|:-------------------------------:|:-----------------------:
- |  |  |
-
-
-## Watershed
-
-Now we can come back to the sure foreground and create markers to label individual nuclei. First, we use OpenCV's `connectedComponents` function to create different color labels for the set of regions in the `sure_fg` image. We store the label information in the `markers` image and add 1 to each color value. The pixels that are part of the `unknown` region are set to zero in the `markers` image. This is critical for the following `watershed` step that separates connected nuclei regions based on the set of markers.
-
-```python:
-# label markers
-ret, markers = cv2.connectedComponents(sure_fg)
-
-# add one to all labels so that sure background is not 0, but 1
-markers = markers + 1
-
-# mark the region of unknown with zero
-markers[unknown==255] = 0
-markers = cv2.watershed(image,markers)
-```
-
-Lastly, we overlay a yellow outline to the original image for all identified nuclei.
-
-```
-image[markers == -1] = [0,255,255]
-```
-
-The resulting `markers` (pseudo-colored) and input images with segmentation overlay look like this:
-
-Markers | Segmentation
-:----------------------:|:-------------------------:
- |  |
-
-
-## Measure
-
-With the markers in hand, it is very easy to extract pixel and object information for each identified object. We use the `scikit-image` package (`skimage`) for the data extraction and `pandas` for storing the data in csv format.
-
-```
-from skimage import measure, color
-import pandas as pd
-
-# compute image properties and return them as a pandas-compatible table
-p = ['label', 'area', 'equivalent_diameter', 'mean_intensity', 'perimeter']
-props = measure.regionprops_table(markers, nuclei, properties=p)
-df = pd.DataFrame(props)
-
-# print data to screen and save
-print (df)
-df.to_csv('nuclei-data.csv')
-```
-
-**Output:**
-```
- label area equivalent_diameter mean_intensity perimeter
-0 1 204775 510.614951 72.078891 6343.194406
-1 2 1906 49.262507 218.294334 190.716775
-2 3 1038 36.354128 204.438343 148.568542
-3 4 2194 52.853454 156.269827 215.858910
-4 5 2014 50.638962 199.993545 177.432504
-5 6 1461 43.130070 185.911020 168.610173
-6 7 2219 53.153726 170.962596 212.817280
-7 8 1837 48.362600 230.387044 184.024387
-8 9 1032 36.248906 228.920543 135.769553
-9 10 2433 55.657810 189.083436 218.781746
-10 11 1374 41.826202 214.344978 167.396970
-11 12 1632 45.584284 191.976716 196.024387
-12 13 1205 39.169550 245.765145 141.639610
-13 14 2508 56.509157 153.325359 229.894444
-14 15 2086 51.536178 195.962608 244.929978
-15 16 1526 44.079060 243.675623 163.124892
-16 17 1929 49.558845 217.509072 174.124892
-17 18 1284 40.433149 165.881620 150.710678
-18 19 2191 52.817306 174.357827 190.331998
-19 20 2218 53.141747 170.529306 210.260931
-20 21 2209 53.033821 164.460842 203.858910
-21 22 2370 54.932483 193.639241 206.296465
-22 23 1426 42.610323 249.032959 157.296465
-23 24 2056 51.164250 194.098735 181.396970
-```
-
-
-# Exercises
-
-1. Experiment with different kernel sizes during the preprocessing step.
-2. Experiment with different iteration numbers for the opening and dilation operation.
-3. Experiment with different thresholding values for isolating the nuclei seeds from the distance transform image.
-4. Change the overlay color from yellow to magenta. Tip: magenta corresponds to BGR (255,255,0).
-
-How do these changes affect the segmentation?
-
----
-
-# Resources
-
-* [Introduction to OpenCV](https://docs.opencv.org/master/d1/dfb/intro.html)
-* [OpenCV Python Tutorial](https://opencv24-python-tutorials.readthedocs.io/en/latest/index.html)
diff --git a/content/courses/opencv/morph-closed.png b/content/courses/opencv/morph-closed.png
deleted file mode 100644
index 708bf1b5..00000000
Binary files a/content/courses/opencv/morph-closed.png and /dev/null differ
diff --git a/content/courses/opencv/morph-dilated.png b/content/courses/opencv/morph-dilated.png
deleted file mode 100644
index bb05e790..00000000
Binary files a/content/courses/opencv/morph-dilated.png and /dev/null differ
diff --git a/content/courses/opencv/morph-eroded.png b/content/courses/opencv/morph-eroded.png
deleted file mode 100644
index 40932265..00000000
Binary files a/content/courses/opencv/morph-eroded.png and /dev/null differ
diff --git a/content/courses/opencv/morph-gradient.png b/content/courses/opencv/morph-gradient.png
deleted file mode 100644
index 0440b884..00000000
Binary files a/content/courses/opencv/morph-gradient.png and /dev/null differ
diff --git a/content/courses/opencv/morph-input.png b/content/courses/opencv/morph-input.png
deleted file mode 100644
index 1000d086..00000000
Binary files a/content/courses/opencv/morph-input.png and /dev/null differ
diff --git a/content/courses/opencv/morph-opened.png b/content/courses/opencv/morph-opened.png
deleted file mode 100644
index 0b5c6a26..00000000
Binary files a/content/courses/opencv/morph-opened.png and /dev/null differ
diff --git a/content/courses/opencv/nuclei-data.csv b/content/courses/opencv/nuclei-data.csv
deleted file mode 100644
index dab83b49..00000000
--- a/content/courses/opencv/nuclei-data.csv
+++ /dev/null
@@ -1,25 +0,0 @@
-,label,area,equivalent_diameter,mean_intensity,perimeter
-0,1,200744,505.56423841913056,70.64225082692384,6884.700069236609
-1,2,1946,49.77674310413073,216.78365878725592,203.44469683442247
-2,3,1038,36.3541283410165,204.4383429672447,148.5685424949238
-3,4,2204,52.97376668310739,156.11252268602541,215.85891039679558
-4,5,1475,43.33622420659609,185.08745762711865,165.7462120245875
-5,6,2006,50.53828773057845,200.27367896311065,179.0893577750951
-6,7,1472,43.292131038448076,139.11820652173913,176.9888527117068
-7,8,2300,54.1151637980601,167.7104347826087,228.5452018978108
-8,9,1862,48.69057436811434,229.56015037593986,187.09545442950497
-9,10,6680,92.22385894567027,218.30344311377246,582.7949852598665
-10,11,942,34.63223427878316,145.3524416135881,143.4030662740799
-11,12,1466,43.203809699860365,214.03410641200546,167.74011537017762
-12,13,1591,45.00804501057165,192.74670018856065,193.5391052434009
-13,14,2438,55.71497114837561,154.04347826086956,234.65180361560903
-14,15,2133,52.11352942298288,196.41537740271917,233.23759005323592
-15,16,1930,49.57168870775802,217.52487046632123,173.78174593052023
-16,17,1291,40.5432146265327,165.46862896979087,149.29646455628165
-17,18,2192,52.829358145442924,174.30839416058393,189.7462120245875
-18,19,2213,53.08181527132352,164.34342521464075,203.2731239591687
-19,20,2225,53.225538860924054,170.32898876404494,209.33199846221441
-20,21,2380,55.04825261958537,193.39033613445378,203.46803743153546
-21,22,477,24.64417299969047,151.90356394129978,112.26093065034893
-22,23,1471,43.27742333255787,246.66281441196466,165.9177848998413
-23,24,2067,51.30093701841694,193.7527818093856,182.56854249492378
diff --git a/content/courses/opencv/nuclei-dist_transform.png b/content/courses/opencv/nuclei-dist_transform.png
deleted file mode 100644
index 981dea83..00000000
Binary files a/content/courses/opencv/nuclei-dist_transform.png and /dev/null differ
diff --git a/content/courses/opencv/nuclei-markers.png b/content/courses/opencv/nuclei-markers.png
deleted file mode 100644
index b8ec2959..00000000
Binary files a/content/courses/opencv/nuclei-markers.png and /dev/null differ
diff --git a/content/courses/opencv/nuclei-opening.png b/content/courses/opencv/nuclei-opening.png
deleted file mode 100644
index 890b552f..00000000
Binary files a/content/courses/opencv/nuclei-opening.png and /dev/null differ
diff --git a/content/courses/opencv/nuclei-segmented.png b/content/courses/opencv/nuclei-segmented.png
deleted file mode 100644
index 4eb74e8e..00000000
Binary files a/content/courses/opencv/nuclei-segmented.png and /dev/null differ
diff --git a/content/courses/opencv/nuclei-sure_bg.png b/content/courses/opencv/nuclei-sure_bg.png
deleted file mode 100644
index c5dda31b..00000000
Binary files a/content/courses/opencv/nuclei-sure_bg.png and /dev/null differ
diff --git a/content/courses/opencv/nuclei-sure_fg.png b/content/courses/opencv/nuclei-sure_fg.png
deleted file mode 100644
index 82060310..00000000
Binary files a/content/courses/opencv/nuclei-sure_fg.png and /dev/null differ
diff --git a/content/courses/opencv/nuclei-unknown.png b/content/courses/opencv/nuclei-unknown.png
deleted file mode 100644
index 960db365..00000000
Binary files a/content/courses/opencv/nuclei-unknown.png and /dev/null differ
diff --git a/content/courses/opencv/nuclei.png b/content/courses/opencv/nuclei.png
deleted file mode 100644
index 01c2cd58..00000000
Binary files a/content/courses/opencv/nuclei.png and /dev/null differ
diff --git a/content/courses/opencv/ood-resources.png b/content/courses/opencv/ood-resources.png
deleted file mode 100644
index b0a5a958..00000000
Binary files a/content/courses/opencv/ood-resources.png and /dev/null differ
diff --git a/content/courses/opencv/opencv_logo.png b/content/courses/opencv/opencv_logo.png
deleted file mode 100644
index 719466cc..00000000
Binary files a/content/courses/opencv/opencv_logo.png and /dev/null differ
diff --git a/content/courses/opencv/pdf/notes.pdf b/content/courses/opencv/pdf/notes.pdf
deleted file mode 100644
index fbefcd6a..00000000
Binary files a/content/courses/opencv/pdf/notes.pdf and /dev/null differ
diff --git a/content/courses/opencv/segmentation.py b/content/courses/opencv/segmentation.py
deleted file mode 100755
index 91ca4ab0..00000000
--- a/content/courses/opencv/segmentation.py
+++ /dev/null
@@ -1,73 +0,0 @@
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
-"""
-Created on Wed Nov 3 22:20:19 2021
-
-@author: khs3z
-"""
-import cv2
-import matplotlib.pyplot as plt
-import numpy as np
-from skimage.segmentation import clear_border
-from skimage import measure
-import pandas as pd
-
-image = cv2.imread('fluorescent-cells.png')
-nuclei = np.array(image[:,:,0]) # get blue channel
-
-# apply Gaussian filter to smoothen image, then apply Otsu threshold
-blurred = cv2.GaussianBlur(nuclei, (3, 3), 0)
-ret, thresh = cv2.threshold(blurred,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)
-
-# remove small particles
-kernel = np.ones((3,3),np.uint8)
-opening = cv2.morphologyEx(thresh,cv2.MORPH_OPEN,kernel, iterations=7)
-
-# remove image border touching objects
-opening = clear_border(opening)
-
-# sure background area
-sure_bg = cv2.dilate(opening,kernel,iterations=10)
-
-# find sure foreground area
-dist_transform = cv2.distanceTransform(opening,cv2.DIST_L2,5)
-ret, sure_fg = cv2.threshold(dist_transform,0.6*dist_transform.max(),255,0)
-
-# sure_fg is float32, convert to uint8 and find unknown region
-sure_fg = np.uint8(sure_fg)
-unknown = cv2.subtract(sure_bg,sure_fg)
-
-# label markers
-ret, markers = cv2.connectedComponents(sure_fg)
-
-# add one to all labels so that sure background is not 0, but 1
-markers = markers + 1
-
-# mark the region of unknown with zero
-markers[unknown==255] = 0
-markers = cv2.watershed(image,markers)
-image[markers == -1] = [0,255,255]
-
-
-# compute image properties and return them as a pandas-compatible table
-p = ['label', 'area', 'equivalent_diameter', 'mean_intensity', 'perimeter']
-props = measure.regionprops_table(markers, nuclei, properties=p)
-df = pd.DataFrame(props)
-
-# print data to screen and save
-print (df)
-df.to_csv('nuclei-data.csv')
-
-# save output images
-cv2.imwrite('nuclei.png', nuclei)
-cv2.imwrite('nuclei-opening.png', opening)
-cv2.imwrite('nuclei-sure_bg.png', sure_bg)
-# cv2.imwrite('nuclei-dist_transform.png', dist_transform)
-cv2.imwrite('nuclei-sure_fg.png', sure_fg)
-cv2.imwrite('nuclei-unknown.png', unknown)
-#cv2.imwrite('nuclei-markers.png', markers)
-cv2.imwrite('image-segmented.png', image)
-
-for i,img in enumerate([image,nuclei,opening,sure_bg,sure_fg,dist_transform,unknown,markers,image]):
- plt.subplot(331+i).imshow(img, cmap='jet')
-
diff --git a/content/courses/parallel-computing-introduction/codes/cexe b/content/courses/parallel-computing-introduction/codes/cexe
deleted file mode 100755
index 2b6b102f..00000000
Binary files a/content/courses/parallel-computing-introduction/codes/cexe and /dev/null differ
diff --git a/content/courses/parallel-computing-introduction/codes/manager_worker_stub.f90 b/content/courses/parallel-computing-introduction/codes/manager_worker_stub.f90
index edcd25d9..a8748f50 100644
--- a/content/courses/parallel-computing-introduction/codes/manager_worker_stub.f90
+++ b/content/courses/parallel-computing-introduction/codes/manager_worker_stub.f90
@@ -96,6 +96,7 @@ end module random
program manager_worker
use random
use mpi
+implicit none
integer :: i, n
diff --git a/content/courses/parallel-computing-introduction/codes/mpi1.f90 b/content/courses/parallel-computing-introduction/codes/mpi1.f90
index fdd292f0..da3862fe 100644
--- a/content/courses/parallel-computing-introduction/codes/mpi1.f90
+++ b/content/courses/parallel-computing-introduction/codes/mpi1.f90
@@ -1,5 +1,6 @@
program hello
use mpi
+implicit none
integer :: myrank, nprocs
integer :: err
diff --git a/content/courses/parallel-computing-introduction/codes/mpi_cart_topo.cxx b/content/courses/parallel-computing-introduction/codes/mpi_cart_topo.cxx
index 0919a5ee..5a570a14 100644
--- a/content/courses/parallel-computing-introduction/codes/mpi_cart_topo.cxx
+++ b/content/courses/parallel-computing-introduction/codes/mpi_cart_topo.cxx
@@ -1,4 +1,4 @@
-
+#include We will introduce the R statistical computing environment as well as RStudio. We will cover very basic functionality, including variables, functions, importing data, and the fundamentals of inspecting data frame objects. The course assumes little to no experience with statistical computing or R.
- -```{r, echo=FALSE, message=FALSE, eval=TRUE, warning=FALSE} -library(knitr) -opts_chunk$set(message=FALSE, warning=FALSE, eval=TRUE, echo=TRUE, results="show", cache=FALSE) -opts_knit$set(root.dir = "../../static/data/") -options(digits=3) -options(max.print=200) -.ex <- 1 # Track ex numbers w/ hidden var. Increment each ex: `r .ex``r .ex=.ex+1` -``` - -## RStudio - -Let's start by learning about RStudio. **R** is the underlying statistical computing environment, but using R alone is no fun. **RStudio** is a graphical integrated development environment that makes using R much easier. - -- Panes in RStudio. There are four panes, and their orientation is configurable under "Tools -- Global Options." You don't have to do it this way, but I usually set up my window to have: - - Editor in the top left - - Console top right - - Environment/history on the bottom left - - Plots/help on the bottom right. -- Projects: first, start a new project in a new folder somewhere easy to remember. When we start reading in data it'll be important that the _code and the data are in the same place._ Creating a project creates a Rproj file that opens R running _in that folder_. This way, when you want to read in dataset _whatever.txt_, you just tell it the filename rather than a full path. This is critical for reproducibility, and we'll talk about that more later. -- Code that you type into the console is code that R executes. From here forward we will use the editor window to write a script that we can save to a file and run it again whenever we want to. We usually give it a `.R` extension, but it's just a plain text file. If you want to send commands from your editor to the console, use `CMD`+`Enter` (`Ctrl`+`Enter` on Windows). -- Anything after a `#` sign is a comment. Use them liberally to *comment your code*. - -## Basic operations - -R can be used as a glorified calculator. Try typing this in directly into the console. Make sure you're typing into the editor, not the console, and save your script. Use the run button, or press `CMD`+`Enter` (`Ctrl`+`Enter` on Windows). - -```{r calculator} -2+2 -5*4 -2^3 -6/7 -``` - -R Knows order of operations and scientific notation. - -```{r calculator2} -2+3*4/(5+3)*15/2^2+3*4^2 -5e4 -``` - -However, to do useful and interesting things, we need to assign _values_ to _objects_. To create objects, we need to give it a name followed by the assignment operator `<-` and the value we want to give it: - -```{r assignment} -weight_kg <- 55 -``` - -`<-` is the assignment operator. It assigns values on the right to objects on the left, it is like an arrow that points from the value to the object. Mostly similar to `=` but not always. Learn to use `<-` as it is good programming practice. Using `=` in place of `<-` can lead to issues down the line. The keyboard shortcut for inserting the `<-` operator is `Alt-dash`. - -Objects can be given any name such as `x`, `current_temperature`, or `subject_id`. You want your object names to be explicit and not too long. They cannot start with a number (`2x` is not valid but `x2` is). R is case sensitive (e.g., `weight_kg` is different from `Weight_kg`). There are some names that cannot be used because they represent the names of fundamental functions in R (e.g., `if`, `else`, `for`, see [here](https://stat.ethz.ch/R-manual/R-devel/library/base/html/Reserved.html) for a complete list). In general, even if it's allowed, it's best to not use other function names, which we'll get into shortly (e.g., `c`, `T`, `mean`, `data`, `df`, `weights`). In doubt check the help to see if the name is already in use. It's also best to avoid dots (`.`) within a variable name as in `my.dataset`. It is also recommended to use nouns for variable names, and verbs for function names. - -When assigning a value to an object, R does not print anything. You can force to print the value by typing the name: - -```{r printAssignment} -weight_kg -``` - -Now that R has `weight_kg` in memory, we can do arithmetic with it. For instance, we may want to convert this weight in pounds (weight in pounds is 2.2 times the weight in kg). - -```{r modAssignment} -2.2 * weight_kg -``` - -We can also change a variable's value by assigning it a new one: - -```{r newAssignment} -weight_kg <- 57.5 -2.2 * weight_kg -``` - -This means that assigning a value to one variable does not change the values of other variables. For example, let's store the animal's weight in pounds in a variable. - -```{r calculationWithVar} -weight_lb <- 2.2 * weight_kg -``` - -and then change `weight_kg` to 100. - -```{r modAssignment2} -weight_kg <- 100 -``` - -What do you think is the current content of the object `weight_lb`? 126.5 or 220? - -You can see what objects (variables) are stored by viewing the Environment tab in Rstudio. You can also use the `ls()` function. You can remove objects (variables) with the `rm()` function. You can do this one at a time or remove several objects at once. You can also use the little broom button in your environment pane to remove everything from your environment. - -```{r rm, eval=FALSE} -ls() -rm(weight_lb, weight_kg) -ls() -weight_lb # oops! you should get an error because weight_lb no longer exists! -``` - ----- - -**EXERCISE `r .ex``r .ex=.ex+1`** - -What are the values after each statement in the following? - -```{r ex1} -mass <- 50 # mass? -age <- 30 # age? -mass <- mass * 2 # mass? -age <- age - 10 # age? -mass_index <- mass/age # massIndex? -``` - ----- - -## Functions - -R has built-in functions. - -```{r fns} -# Notice that this is a comment. -# Anything behind a # is "commented out" and is not run. -sqrt(144) -log(1000) -``` - -Get help by typing a question mark in front of the function's name, or `help(functionname)`: - -``` -help(log) -?log -``` - -Note syntax highlighting when typing this into the editor. Also note how we pass *arguments* to functions. The `base=` part inside the parentheses is called an argument, and most functions use arguments. Arguments modify the behavior of the function. Functions some input (e.g., some data, an object) and other options to change what the function will return, or how to treat the data provided. Finally, see how you can *nest* one function inside another (here taking the square root of the log-base-10 of 1000). - -```{r log} -log(1000) -log(1000, base=10) -log(1000, 10) -sqrt(log(1000, base=10)) -``` - ----- - -**EXERCISE `r .ex``r .ex=.ex+1`** - -See `?abs` and calculate the square root of the log-base-10 of the absolute value of `-4*(2550-50)`. Answer should be `2`. - ----- - -## Data Frames - -There are _lots_ of different basic data structures in R. If you take any kind of longer introduction to R you'll probably learn about arrays, lists, matrices, etc. We are going to skip straight to the data structure you'll probably use most -- the **data frame**. We use data frames to store heterogeneous tabular data in R: tabular, meaning that individuals or observations are typically represented in rows, while variables or features are represented as columns; heterogeneous, meaning that columns/features/variables can be different classes (on variable, e.g. age, can be numeric, while another, e.g., cause of death, can be text). - -**Recommended reading:** Review the [_Introduction_ (10.1)](http://r4ds.had.co.nz/tibbles.html#introduction-4) and [_Tibbles vs. data.frame_ (10.3)](http://r4ds.had.co.nz/tibbles.html#tibbles-vs.data.frame) sections of the [**_R for Data Science_ book**](http://r4ds.had.co.nz/tibbles.html). We will initially be using the `read_*` functions from the [**readr** package](http://readr.tidyverse.org/). These functions load data into a _tibble_ instead of R's traditional data.frame. Tibbles are data frames, but they tweak some older behaviors to make life a little easier. These sections explain the few key small differences between traditional data.frames and tibbles. - -## Our data - -The data we're going to look at is cleaned up version of a gene expression dataset from [Brauer et al. Coordination of Growth Rate, Cell Cycle, Stress Response, and Metabolic Activity in Yeast (2008) _Mol Biol Cell_ 19:352-367](http://www.ncbi.nlm.nih.gov/pubmed/17959824). This data is from a gene expression microarray, and in this paper the authors are examining the relationship between growth rate and gene expression in yeast cultures limited by one of six different nutrients (glucose, leucine, ammonium, sulfate, phosphate, uracil). If you give yeast a rich media loaded with nutrients except restrict the supply of a _single_ nutrient, you can control the growth rate to any rate you choose. By starving yeast of specific nutrients you can find genes that: - -1. **Raise or lower their expression in response to growth rate**. Growth-rate dependent expression patterns can tell us a lot about cell cycle control, and how the cell responds to stress. The authors found that expression of >25% of all yeast genes is linearly correlated with growth rate, independent of the limiting nutrient. They also found that the subset of negatively growth-correlated genes is enriched for peroxisomal functions, and positively correlated genes mainly encode ribosomal functions. -2. **Respond differently when different nutrients are being limited**. If you see particular genes that respond very differently when a nutrient is sharply restricted, these genes might be involved in the transport or metabolism of that specific nutrient. - -You can download the cleaned up version of the data at [the link above](data.html). The file is called [**brauer2007_tidy.csv**](data/brauer2007_tidy.csv). Later on we'll actually start with the original raw data (minimally processed) and manipulate it so that we can make it more amenable for analysis. - -## Reading in data - -### dplyr and readr - -There are some built-in functions for reading in data in text files. These functions are _read-dot-something_ -- for example, `read.csv()` reads in comma-delimited text data; `read.delim()` reads in tab-delimited text, etc. We're going to read in data a little differently here using the [readr](http://readr.tidyverse.org/) package. When you load the readr package, you'll have access to very similar looking functions, named _read-underscore-something_ -- e.g., `read_csv()`. You have to have the readr package installed to access these functions. Compared to the base functions, they're _much_ faster, they're good at guessing the types of data in the columns, they don't do some of the other silly things that the base functions do. We're going to use another package later on called [dplyr](https://cran.r-project.org/web/packages/dplyr/index.html), and if you have the dplyr package loaded as well, and you read in the data with readr, the data will display nicely. - -First let's load those packages. - -```{r loadpkgs} -library(readr) -library(dplyr) -``` - -If you see a warning that looks like this: `Error in library(packageName) : there is no package called 'packageName'`, then you don't have the package installed correctly. - -### `read_csv()` - -Now, let's actually load the data. You can get help for the import function with `?read_csv`. When we load data we assign it to a variable just like any other, and we can choose a name for that data. Since we're going to be referring to this data a lot, let's give it a short easy name to type. I'm going to call it `ydat`. Once we've loaded it we can type the name of the object itself (`ydat`) to see it printed to the screen. - -```{r loaddata} -ydat <- read_csv(file="brauer2007_tidy.csv") -ydat -``` - -Take a look at that output. The nice thing about loading dplyr and reading in data with readr is that data frames are displayed in a much more friendly way. This dataset has nearly 200,000 rows and 7 columns. When you import data this way and try to display the object in the console, instead of trying to display all 200,000 rows, you'll only see about 10 by default. Also, if you have so many columns that the data would wrap off the edge of your screen, those columns will not be displayed, but you'll see at the bottom of the output which, if any, columns were hidden from view. If you want to see the whole dataset, there are two ways to do this. First, you can click on the name of the data.frame in the **Environment** panel in RStudio. Or you could use the `View()` function (_with a capital V_). - -```{r view, eval=FALSE} -View(ydat) -``` - -## Inspecting data.frame objects - -### Built-in functions - -There are several built-in functions that are useful for working with data frames. - -* Content: - * `head()`: shows the first few rows - * `tail()`: shows the last few rows -* Size: - * `dim()`: returns a 2-element vector with the number of rows in the first element, and the number of columns as the second element (the dimensions of the object) - * `nrow()`: returns the number of rows - * `ncol()`: returns the number of columns -* Summary: - * `colnames()` (or just `names()`): returns the column names - * `str()`: structure of the object and information about the class, length and content of each column - * `summary()`: works differently depending on what kind of object you pass to it. Passing a data frame to the `summary()` function prints out useful summary statistics about numeric column (min, max, median, mean, etc.) - -```{r data_frame_functions} -head(ydat) -tail(ydat) -dim(ydat) -names(ydat) -str(ydat) -summary(ydat) -``` - -### Other packages - -The `glimpse()` function is available once you load the **dplyr** library, and it's like `str()` but its display is a little better. - -```{r} -glimpse(ydat) -``` - -The **skimr** package has a nice function, skim, that provides summary statistics the user can skim quickly to understand your data. You can install it with `install.packages("skimr")` if you don't have it already. - -```{r, eval = FALSE} -library(skimr) -skim(ydat) -``` - - -## Accessing variables & subsetting data frames - -We can access individual variables within a data frame using the `$` operator, e.g., `mydataframe$specificVariable`. Let's print out all the gene names in the data. Then let's calculate the average expression across all conditions, all genes (using the built-in `mean()` function). - -```{r} -# display all gene symbols -ydat$symbol - -#mean expression -mean(ydat$expression) -``` - -Now that's not too interesting. This is the average gene expression across all genes, across all conditions. The data is actually scaled/centered around zero: - -```{r histogram_expression_values, echo=F} -library(ggplot2) -ggplot(ydat, aes(expression)) + - geom_histogram(bins=100)+ - xlab("Expression") + - ggtitle("Histogram of expression values") + - theme_bw() -``` - -We might be interested in the average expression of genes with a particular biological function, and how that changes over different growth rates restricted by particular nutrients. This is the kind of thing we're going to do in the next section. - ----- - -**EXERCISE `r .ex``r .ex=.ex+1`** - -1. What's the standard deviation expression (hint: get help on the `sd` function with `?sd`). -1. What's the range of rate represented in the data? (hint: `range()`). - ----- - -## BONUS: Preview to advanced manipulation - -What if we wanted to show the mean expression, standard deviation, and correlation between growth rate and expression, separately for each limiting nutrient, separately for each gene, for all genes involved in the leucine biosynthesis pathway? - -```{r, results='hide'} -ydat %>% - filter(bp=="leucine biosynthesis") %>% - group_by(nutrient, symbol) %>% - summarize(mean=mean(expression), sd=sd(expression), r=cor(rate, expression)) -``` - -```{r, echo=FALSE} -ydat %>% - filter(bp=="leucine biosynthesis") %>% - group_by(nutrient, symbol) %>% - summarize(mean=mean(expression), sd=sd(expression), r=cor(rate, expression)) %>% - mutate_each(funs(round(., 2)), mean:r) -``` - -```{r makedata, include=F, eval=F} -# Here's how I made the data -library(readr) -library(dplyr) -library(tidyr) -library(ggplot2) -theme_set(theme_bw()) - -# read original data -(ydat <- read_delim("data/brauer2007_orig.txt", delim="\t")) -# change variable WS||WS to :: -(ydat <- ydat %>% mutate(NAME=gsub("\\s*\\|\\|\\s*", "::", NAME, perl=TRUE))) -# separate on :: into gene symbol, go-bp, go-mf, syssymbol, and somenumber -(ydat <- ydat %>% separate(NAME, c("symbol", "bp", "mf", "systematic_symbol", "somenumber"), sep = "::")) -# replace missing with NA -(ydat[ydat==""] <- NA) -# remove things where syssymbol is missing -(ydat <- ydat %>% filter(!is.na(systematic_symbol) & systematic_symbol!="")) -# write out go terms to join to later -ydat %>% select(systematic_symbol, bp, mf) %>% distinct %>% write_csv("data/brauer2007_syssymbol2go.csv") -# unite the symbol column again -(ydat <- ydat %>% unite(NAME, symbol, systematic_symbol, somenumber, sep="::") %>% select(-bp, -mf)) -# write out the messy data that you'll use for data cleaning class -ydat %>% write_csv("data/brauer2007_messy.csv") -rm(ydat) - -(ydat <- read_csv("data/brauer2007_messy.csv")) -(syssymbol2go <- read_csv("data/brauer2007_syssymbol2go.csv")) - -ydat <- ydat %>% - separate(NAME, c("symbol", "systematic_symbol", "somenumber"), sep = "::") %>% - select(-somenumber, -GID, -YORF, -GWEIGHT) %>% - gather(sample, expression, G0.05:U0.3) %>% - separate(sample, c("nutrient", "rate"), sep = 1) %>% - filter(!is.na(expression), systematic_symbol != "") %>% - inner_join(syssymbol2go, by="systematic_symbol") %>% - mutate(nutrient = plyr::mapvalues(nutrient, - from=c("G", "L", "P", "S", "N", "U"), - to=c("Glucose", "Leucine", "Phosphate", "Sulfate", "Ammonium", "Uracil"))) - -ydat %>% write_csv("data/brauer2007_tidy.csv") -``` - - -```{r, include=FALSE, eval = TRUE} -# find packages attached during for this Rmd session and their versions -package <- installed.packages()[names(sessionInfo()$otherPkgs), "Package"] -version <- installed.packages()[names(sessionInfo()$otherPkgs), "Version"] - -thesepkgs <- data.frame(package, version) - -# if no non-base packages installed skip -if(nrow(thesepkgs) == 0) invisible() - -# find package csv file -fp <- "../../packages.csv" - -# if it exists find read in contents ... combine with attached pkgs and dedupe -if(file.exists(fp)) { - - pkgs <- read.csv(fp) - - pkgs <- - rbind(pkgs, thesepkgs) - - pkgs <- pkgs[!duplicated(pkgs),] - -} else{ - - pkgs <- thesepkgs - -} - -# write out new package.csv file -write.table(pkgs, - file = fp, - sep = ",", - row.names = FALSE) -``` diff --git a/content/courses/r-shiny/_index.md b/content/courses/r-shiny/_index.md deleted file mode 100644 index 860c051f..00000000 --- a/content/courses/r-shiny/_index.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -# Course title, summary, and position. -title: Introduction to Shiny -summary: "An introduction to creating Web apps with R and Shiny." -authors: [cag] -categories: ["R","Shiny", "App Development"] -tags: ["R","Shiny","App_Development"] - -date: "2020-11-17T00:00:00Z" -draft: false # Is this a draft? true/false -toc: true # Show table of contents? true/false -type: book # Do not modify. - -weight: 1 ---- - -Shiny is a package that lets you build interactive Web apps using the R language. diff --git a/content/courses/r-shiny/customization/_index.md b/content/courses/r-shiny/customization/_index.md deleted file mode 100755 index 82085a4f..00000000 --- a/content/courses/r-shiny/customization/_index.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -# Course title, summary, and position. -title: "Customizing Your Shiny App" - -# Page metadata. -date: "2023-05-01T00:00:00" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. - -# Add menu entry to sidebar. -weight: 500 -menu: - r-shiny: - name: Customization - ---- - -Learn how to "prettify" our apps and prepare them for publication! - -{{< figure src="/courses/r-shiny/customization/img/shiny-logo.png" caption="Shiny R package logo">}} diff --git a/content/courses/r-shiny/customization/img/blank-dashboard.png b/content/courses/r-shiny/customization/img/blank-dashboard.png deleted file mode 100755 index 4bcde280..00000000 Binary files a/content/courses/r-shiny/customization/img/blank-dashboard.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/bootstrap.png b/content/courses/r-shiny/customization/img/bootstrap.png deleted file mode 100755 index 7c569f3d..00000000 Binary files a/content/courses/r-shiny/customization/img/bootstrap.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/bootswatch.png b/content/courses/r-shiny/customization/img/bootswatch.png deleted file mode 100755 index ad15694c..00000000 Binary files a/content/courses/r-shiny/customization/img/bootswatch.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/boxes-code.png b/content/courses/r-shiny/customization/img/boxes-code.png deleted file mode 100755 index 9a9442bf..00000000 Binary files a/content/courses/r-shiny/customization/img/boxes-code.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/boxes.png b/content/courses/r-shiny/customization/img/boxes.png deleted file mode 100755 index c6be2f8c..00000000 Binary files a/content/courses/r-shiny/customization/img/boxes.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/brand.png b/content/courses/r-shiny/customization/img/brand.png deleted file mode 100755 index fb8e5668..00000000 Binary files a/content/courses/r-shiny/customization/img/brand.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/bslib.gif b/content/courses/r-shiny/customization/img/bslib.gif deleted file mode 100755 index 2b8cef6d..00000000 Binary files a/content/courses/r-shiny/customization/img/bslib.gif and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/dashboard-skin.png b/content/courses/r-shiny/customization/img/dashboard-skin.png deleted file mode 100755 index 0fe497ce..00000000 Binary files a/content/courses/r-shiny/customization/img/dashboard-skin.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/dashboard.png b/content/courses/r-shiny/customization/img/dashboard.png deleted file mode 100755 index b0b9378d..00000000 Binary files a/content/courses/r-shiny/customization/img/dashboard.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/defaults.png b/content/courses/r-shiny/customization/img/defaults.png deleted file mode 100755 index 0d791a17..00000000 Binary files a/content/courses/r-shiny/customization/img/defaults.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/menu-items-code.png b/content/courses/r-shiny/customization/img/menu-items-code.png deleted file mode 100755 index 523c290f..00000000 Binary files a/content/courses/r-shiny/customization/img/menu-items-code.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/menu-items.png b/content/courses/r-shiny/customization/img/menu-items.png deleted file mode 100755 index 0905513a..00000000 Binary files a/content/courses/r-shiny/customization/img/menu-items.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/plot.png b/content/courses/r-shiny/customization/img/plot.png deleted file mode 100755 index 3b5576f7..00000000 Binary files a/content/courses/r-shiny/customization/img/plot.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/rc-website.png b/content/courses/r-shiny/customization/img/rc-website.png deleted file mode 100755 index 177a8079..00000000 Binary files a/content/courses/r-shiny/customization/img/rc-website.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/sass-compiler.png b/content/courses/r-shiny/customization/img/sass-compiler.png deleted file mode 100755 index c9d41b71..00000000 Binary files a/content/courses/r-shiny/customization/img/sass-compiler.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/sass.png b/content/courses/r-shiny/customization/img/sass.png deleted file mode 100755 index 6d1c43ad..00000000 Binary files a/content/courses/r-shiny/customization/img/sass.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/shiny-logo.png b/content/courses/r-shiny/customization/img/shiny-logo.png deleted file mode 100755 index 4451d094..00000000 Binary files a/content/courses/r-shiny/customization/img/shiny-logo.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/shinytheme-code.png b/content/courses/r-shiny/customization/img/shinytheme-code.png deleted file mode 100755 index 30cc3059..00000000 Binary files a/content/courses/r-shiny/customization/img/shinytheme-code.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/shinythemes.png b/content/courses/r-shiny/customization/img/shinythemes.png deleted file mode 100755 index 19d52ef2..00000000 Binary files a/content/courses/r-shiny/customization/img/shinythemes.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/img/uva-website.png b/content/courses/r-shiny/customization/img/uva-website.png deleted file mode 100755 index 802b7892..00000000 Binary files a/content/courses/r-shiny/customization/img/uva-website.png and /dev/null differ diff --git a/content/courses/r-shiny/customization/intro.md b/content/courses/r-shiny/customization/intro.md deleted file mode 100755 index 0d0da8cd..00000000 --- a/content/courses/r-shiny/customization/intro.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: Four Approaches to Styling Shiny -date: "2023-05-01T00:00:00" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 506 - -menu: - r-shiny: - parent: Customization - ---- - -1. Accepting the Shiny Defaults - -2. Shiny Themes - -3. Shiny Dashboards - -4. Bootstraplib diff --git a/content/courses/r-shiny/customization/method1-bootstrap.md b/content/courses/r-shiny/customization/method1-bootstrap.md deleted file mode 100755 index 64b29840..00000000 --- a/content/courses/r-shiny/customization/method1-bootstrap.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -title: "Built with Bootstrap" -date: "2023-05-01T00:00:00" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 520 - -menu: - r-shiny: - parent: Customization - ---- - -- Shiny's default UI uses Bootstrap - -- Bootstrap is a CSS framework developed by Twitter - -- Easy to customize if you're comfortable with HTML and CSS - -## Millions of websites use Bootstrap - -{{< figure src="/courses/r-shiny/customization/img/bootstrap.png">}} -{{< figure src="/courses/r-shiny/customization/img/rc-website.png">}} diff --git a/content/courses/r-shiny/customization/method1.md b/content/courses/r-shiny/customization/method1.md deleted file mode 100755 index e104a8a2..00000000 --- a/content/courses/r-shiny/customization/method1.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -title: "Method 1: Accepting the Shiny Defaults" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 510 - -menu: - r-shiny: - parent: Customization - ---- - -{{< figure src="/courses/r-shiny/customization/img/defaults.png">}} diff --git a/content/courses/r-shiny/customization/method2-bootswatch.md b/content/courses/r-shiny/customization/method2-bootswatch.md deleted file mode 100755 index 9c684954..00000000 --- a/content/courses/r-shiny/customization/method2-bootswatch.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -title: "Bootswatch" -date: "2023-05-01T00:00:00" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 720 - -menu: - r-shiny: - parent: Customization - ---- - -All Shiny Themes come from [bootswatch.com](https://bootswatch.com) - -{{< figure src="/courses/r-shiny/customization/img/bootswatch.png">}} diff --git a/content/courses/r-shiny/customization/method2-nonshinythemes.md b/content/courses/r-shiny/customization/method2-nonshinythemes.md deleted file mode 100755 index 88077b99..00000000 --- a/content/courses/r-shiny/customization/method2-nonshinythemes.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: "Using Non-shinythemes Themes" -date: "2023-05-01T00:00:00" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 730 - -menu: - r-shiny: - parent: Customization - ---- - -1. Download the CSS files - -2. Add .css file to www folder in app directory - -3. Set `theme = "myfile.css"` - -Try this with the "Quartz" theme! diff --git a/content/courses/r-shiny/customization/method2-shinythemes.md b/content/courses/r-shiny/customization/method2-shinythemes.md deleted file mode 100755 index 2d582f36..00000000 --- a/content/courses/r-shiny/customization/method2-shinythemes.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: "shinythemes" -date: "2023-05-01T00:00:00" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 710 - -menu: - r-shiny: - parent: Customization - ---- - -1. `install.packages("shinythemes")` - -2. Add `library(shinythemes)` to `app.R` or `ui.R` - -3. Add the `theme` argument to your `fluidPage()`, `navbarPage()`, etc. Set `theme` with `shinytheme("- -Unfortunately doesn't work with Shiny Dashboards -- use fresh instead! - -[https://dreamrs.github.io/fresh/articles/vars-shinydashboard.html](https://dreamrs.github.io/fresh/articles/vars-shinydashboard.html) - diff --git a/content/courses/r-shiny/customization/method4-preview.md b/content/courses/r-shiny/customization/method4-preview.md deleted file mode 100755 index 355c89bf..00000000 --- a/content/courses/r-shiny/customization/method4-preview.md +++ /dev/null @@ -1,55 +0,0 @@ ---- -title: "Preview Your App" -date: "2023-05-01T00:00:00" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 450 - -menu: - r-shiny: - parent: Customization - ---- - -1. Add `bs_themer()` to your server function - -``` -library(shiny) -library(bslib) # step 1 - -... - -server <- function(input, output){ - bs_themer() - - ##cool reactive logic## -} - -``` - -{{< figure src="/courses/r-shiny/customization/img/bslib.gif" caption="Adjust colors and fonts">}} - -
- -When you make changes in the Preview tool, code for updating your theme is printed to the console. Copy the updates to your Shiny app. - -``` -my_theme <- bs_theme() - -# This line is copied from the console (may need to update theme name) -my_theme <- bs_theme_update(my_theme, bg = "#4B3E23", fg = "#000000") - -``` - -
- -## How do I find out Bootstrap variable names? - -- List of Bootstrap sass variables: - - [https://rstudio.github.io/bslib/articles/bs4-variables.html](https://rstudio.github.io/bslib/articles/bs4-variables.html) - -- bslib commands: - [https://rstudio.github.io/bslib/reference/index.html](https://rstudio.github.io/bslib/reference/index.html) - diff --git a/content/courses/r-shiny/customization/method4-sass.md b/content/courses/r-shiny/customization/method4-sass.md deleted file mode 100755 index f8b8dde9..00000000 --- a/content/courses/r-shiny/customization/method4-sass.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -title: "Sass" -date: "2023-05-01T00:00:00" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 920 - -menu: - r-shiny: - parent: Customization - ---- - -## Modifying CSS - -- Requires you to know at least some CSS - -- Each color value appears in multiple places - -
- -## Syntactically Awesome Style Sheets - -CSS with Superpowers - -{{< figure src="/courses/r-shiny/customization/img/sass.png" width="30%">}} - -
- -Sass allows you to change colors and fonts using variables - -{{< figure src="/courses/r-shiny/customization/img/sass-compiler.png">}} diff --git a/content/courses/r-shiny/customization/method4-start.md b/content/courses/r-shiny/customization/method4-start.md deleted file mode 100755 index 9e62ab5e..00000000 --- a/content/courses/r-shiny/customization/method4-start.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: "Start with Regular Shiny Code" -date: "2023-05-01T00:00:00" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 940 - -menu: - r-shiny: - parent: Customization - ---- - -1. Load bslib -2. Instantiate a `bs_theme` object -3. Set `theme` to your `bs_theme` object in the UI - -``` -library(shiny) -library(bslib) # step 1 - -my_theme <- bs_theme() # step 2 - -ui <- fluidPage( - theme = my_theme, # step 3 - ... -) diff --git a/content/courses/r-shiny/customization/method4-theming.md b/content/courses/r-shiny/customization/method4-theming.md deleted file mode 100755 index 062c710b..00000000 --- a/content/courses/r-shiny/customization/method4-theming.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -title: "Theming Plots" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 960 - -menu: - r-shiny: - parent: Customization - ---- - -The `thematic` package enables automatic theming of plots - -``` -library(thematic) - -thematic_on() -onStop(thematic_off) -``` - -{{< figure src="/courses/r-shiny/customization/img/plot.png" caption="Great for dark mode!">}} diff --git a/content/courses/r-shiny/customization/method4.md b/content/courses/r-shiny/customization/method4.md deleted file mode 100755 index 7308a4f5..00000000 --- a/content/courses/r-shiny/customization/method4.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -title: "Common Shiny Question" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 400 - -menu: - r-shiny: - parent: Customization - ---- - -How do I customize my app so the font and colors match my university's? - -{{< figure src="/courses/r-shiny/customization/img/uva-website.png" >}} diff --git a/content/courses/r-shiny/introduction/_index.md b/content/courses/r-shiny/introduction/_index.md deleted file mode 100755 index 9dc88d63..00000000 --- a/content/courses/r-shiny/introduction/_index.md +++ /dev/null @@ -1,23 +0,0 @@ ---- -# Course title, summary, and position. -authors: [cag] -categories: ["R","Shiny", "App Development"] -tags: ["R","Shiny","App_Development"] - -# Page metadata. -date: "2023-05-01T00:00:00" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. - -# Add menu entry to sidebar. -weight: 50 - -menu: - r-shiny: - name: Introduction to Shiny - ---- - -Learn how to create your own web apps in R using Shiny! In this workshop, we will go over Shiny basics and will code an app together from scratch. -{{< figure src="/courses/r-shiny/introduction/img/shiny-logo.png" caption="Shiny R package logo">}} diff --git a/content/courses/r-shiny/introduction/img/conductor.png b/content/courses/r-shiny/introduction/img/conductor.png deleted file mode 100755 index c1937129..00000000 Binary files a/content/courses/r-shiny/introduction/img/conductor.png and /dev/null differ diff --git a/content/courses/r-shiny/introduction/img/deployment-server.jpg b/content/courses/r-shiny/introduction/img/deployment-server.jpg deleted file mode 100755 index f082dc66..00000000 Binary files a/content/courses/r-shiny/introduction/img/deployment-server.jpg and /dev/null differ diff --git a/content/courses/r-shiny/introduction/img/development-server.jpg b/content/courses/r-shiny/introduction/img/development-server.jpg deleted file mode 100755 index f861aa04..00000000 Binary files a/content/courses/r-shiny/introduction/img/development-server.jpg and /dev/null differ diff --git a/content/courses/r-shiny/introduction/img/dog.jpeg b/content/courses/r-shiny/introduction/img/dog.jpeg deleted file mode 100755 index 08dd41ae..00000000 Binary files a/content/courses/r-shiny/introduction/img/dog.jpeg and /dev/null differ diff --git a/content/courses/r-shiny/introduction/img/fluid-wide.png b/content/courses/r-shiny/introduction/img/fluid-wide.png deleted file mode 100755 index 551c6672..00000000 Binary files a/content/courses/r-shiny/introduction/img/fluid-wide.png and /dev/null differ diff --git a/content/courses/r-shiny/introduction/img/input.png b/content/courses/r-shiny/introduction/img/input.png deleted file mode 100755 index e9e85880..00000000 Binary files a/content/courses/r-shiny/introduction/img/input.png and /dev/null differ diff --git a/content/courses/r-shiny/introduction/img/observer.png b/content/courses/r-shiny/introduction/img/observer.png deleted file mode 100755 index faffac85..00000000 Binary files a/content/courses/r-shiny/introduction/img/observer.png and /dev/null differ diff --git a/content/courses/r-shiny/introduction/img/old-faithful.png b/content/courses/r-shiny/introduction/img/old-faithful.png deleted file mode 100755 index 9deb764a..00000000 Binary files a/content/courses/r-shiny/introduction/img/old-faithful.png and /dev/null differ diff --git a/content/courses/r-shiny/introduction/img/output-widgets.png b/content/courses/r-shiny/introduction/img/output-widgets.png deleted file mode 100755 index fd23a662..00000000 Binary files a/content/courses/r-shiny/introduction/img/output-widgets.png and /dev/null differ diff --git a/content/courses/r-shiny/introduction/img/reactivity.png b/content/courses/r-shiny/introduction/img/reactivity.png deleted file mode 100755 index 3dc05301..00000000 Binary files a/content/courses/r-shiny/introduction/img/reactivity.png and /dev/null differ diff --git a/content/courses/r-shiny/introduction/img/shiny-logo.png b/content/courses/r-shiny/introduction/img/shiny-logo.png deleted file mode 100755 index 4451d094..00000000 Binary files a/content/courses/r-shiny/introduction/img/shiny-logo.png and /dev/null differ diff --git a/content/courses/r-shiny/introduction/intro-computer-setup.md b/content/courses/r-shiny/introduction/intro-computer-setup.md deleted file mode 100755 index 91481b86..00000000 --- a/content/courses/r-shiny/introduction/intro-computer-setup.md +++ /dev/null @@ -1,64 +0,0 @@ ---- -title: Computer Setup -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: true # Show table of contents? true/false -type: docs # Do not modify. -weight: 55 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -## What You'll Need - -1. **Development** server (just your computer!) - -2. **Deployment** server (shinyapps.io, UVA Microservices) - -## Development Server - -This is where you will write and test the code for your Shiny app. You can interact with your app, but no one else can. - -{{< figure src="/courses/r-shiny/introduction/img/development-server.jpg" >}} - -You will spend most of your time on your development server. - -### Set Up Your Development Server - -**Option 1**: Clone or Download the GitHub Repository - -1. Run `git clone https://github.com/uvarc/shiny-workshop` in the terminal -2. -OR- Go to https://github.com/uvarc/shiny-workshop and click green "Code" button, then click "Download ZIP" - -
- -**Option 2**: Create a New R Project (need Git installed for this option) - -1. **File** > **New Project** > **Version Control** > **Git** -2. Repository URL: https://github.com/uvarc/shiny-workshop -3. **Create Project** - -After downloading the code, run `packages-for-workshop.R` - -
- -## Deployment Server - -Once you're ready to share your app, it's time to move it to a deployment server (i.e. deploy your app). - -{{< figure src="/courses/r-shiny/introduction/img/deployment-server.jpg" >}} - -Once deployed on the deployment server, your development server is no longer serving your app. This means any changes you make locally will need to be pushed to the deployment server before they're visible to the world. - - -### Set Up Your Deployment Server - -Create a free account on shinyapps.io. - -Can host 5 apps on shinyapps.io for free. - -You can also host your apps on UVA servers: -https://www.rc.virginia.edu/userinfo/microservices/ diff --git a/content/courses/r-shiny/introduction/intro-creating-new-app.md b/content/courses/r-shiny/introduction/intro-creating-new-app.md deleted file mode 100755 index 892c3ef5..00000000 --- a/content/courses/r-shiny/introduction/intro-creating-new-app.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -title: Creating a New Shiny App -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: true # Show table of contents? true/false -type: docs # Do not modify. -weight: 65 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -How do you create a new blank Shiny app? - -## Old Faithful - -Using **New File** -> **Shiny Web App...** creates a living, breathing Shiny App - -{{< figure src="/courses/r-shiny/introduction/img/old-faithful.png" >}} - -## A Better Way to Do It - -1. Open a new R script file. - -2. Start typing `shinyapp` and press **Tab** to autocomplete. **This will expand into a "snippet" of code--the skeleton of a Shiny App.** - -3. Save the file in the `sandbox` folder and run the app. - -It's still a working Shiny app--it just doesn't do anything. Starting from the snippet is less error-prone than creating a new project and deleting the guts. - -Stop an app by clicking the STOP button in the console. diff --git a/content/courses/r-shiny/introduction/intro-workshop-files.md b/content/courses/r-shiny/introduction/intro-workshop-files.md deleted file mode 100755 index 1383bb9d..00000000 --- a/content/courses/r-shiny/introduction/intro-workshop-files.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -title: Workshop Files -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 60 - -menu: - r-shiny: - parent: Introduction to Shiny - - ---- - -## File Organization - -Project Directory - - - **`slides/`**: contains all the RMarkdown files/slides for this workshop - - Knit `MAIN.Rmd` to see the main slideshow - - Each section is a separate RMarkdown file (feel free to take notes in these!) - - - **`projects/`**: contains all the code we'll be working with today - - Each app will live in a separate directory - - - **`sandbox/`**: place to put code snippets or quick demos diff --git a/content/courses/r-shiny/introduction/overview-metaphor.md b/content/courses/r-shiny/introduction/overview-metaphor.md deleted file mode 100755 index dc8e6616..00000000 --- a/content/courses/r-shiny/introduction/overview-metaphor.md +++ /dev/null @@ -1,26 +0,0 @@ ---- -title: A Metaphor -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 120 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -## Shiny Bridges Two Worlds of R Programming - -In Harry Potter, the Knight Bus connected the non-magical, or "Muggle", world to the magical Wizarding World. Similarly, Shiny connects regular R code to the magical world of "reactives" (we will talk more about these later). - -| 1. **Muggle World** | 2. **Wizarding World** | -| --- | --- | -| - Regular R code | - Reactives | -| - Functions, packages, etc | - Functions w/o arguments | -| | - Values that can't be changed with **`<-`** | -| | - Packaged in a `server` function | - -The Wizarding world can reach out to the Muggle world, but not the other way around. diff --git a/content/courses/r-shiny/introduction/overview-project0.md b/content/courses/r-shiny/introduction/overview-project0.md deleted file mode 100755 index 2e6c010a..00000000 --- a/content/courses/r-shiny/introduction/overview-project0.md +++ /dev/null @@ -1,91 +0,0 @@ ---- -title: "Project 0: Our First App" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: true # Show table of contents? true/false -type: docs # Do not modify. -weight: 130 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -## The Main File, app.R - -Go to **projects** -> **project0-first-app** and open **app.R**. - -- This file contains all the components of the app: the user interface and reactive logic. - -- The UI and reactive logic can be written in separate .R files that are sourced in `app.R` or all put in the same file. - -- In this case the UI is in `user_interface.R` and the reactive logic is in `reactive_logic.R`. - -
- -## shinyApp() - -`app.R` is just a regular R script - -- Adding the call to `shinyApp()` at the end of the script changes the "Run" button to "Run App" - -## Starting in the Muggle World - -Let's start with `muggle.R` (our run-of-the-mill R code) - -- We load the libraries we need - -- We create a function `muggle_plot` that takes variable names from the `diamonds` dataset as inputs and generates a scatterplot - -- Test it out if you want by uncommenting and running the last line of the script - -## User Interface - -Open `user_interface.R` - -- The function `tagList` takes HTML functions as inputs and creates a list of HTML components - -- Try running the first line, `h2("A very basic Shiny app")` in the console (make sure you have `shiny` loaded) - -- Functions like `h2`, `p`, and `actionButton` are wrappers for HTML code (essentially strings) - -- The first argument in `actionButton`, `plotOutput`, and `textOutput` are IDs. We will use these IDs in the next part - - -## Reactive Logic - -Open `reactive_logic.R` - -- We will connect the UI to the Muggle code with **reactive logic** - -- We are assigning a function to `reactive_logic` with three arguments: `input`, `output`, and `session`. The arguments are always the same, but `reactive_logic` is usually called `server`. - -- Functions `renderPlot` and `renderText` are assigned to `output$IDname`. They correspond to the UI functions `plotOutput` and `textOutput` - - -## Invoking the App - -`shinyApp(user_interface, reactive_logic)` - -To run our app, we use the command `shinyApp`. The first argument is the UI, and the second argument is our reactive logic. - - -## Deploying the App - -1. Put all the files your app needs in a single directory (the name of the directory will be the name of the app) - -2. Make sure there is a file called `app.R` with the call to `shinyApp()` - -3. Run `rsconnect::deployApp("/path/to/app/directory")`. You may need to connect your shinyapps.io account to deploy an app for the first time. - - -## Your Turn: Modify the App - -Try the following on your own! - -1. Hide the button message until the button has been clicked 3 times. - -2. Add `selectInput` dropdown menus for the X and Y variables. - -3. (Superstar) Connect the `selectInput` menus to the plot (we haven't covered this yet) diff --git a/content/courses/r-shiny/introduction/overview.md b/content/courses/r-shiny/introduction/overview.md deleted file mode 100755 index ad7b4c99..00000000 --- a/content/courses/r-shiny/introduction/overview.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: What Is a Shiny App? -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 110 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -**Shiny is an R package for developing interactive web applications, or apps.** Shiny apps are just webpages! - - -Webpages are made up of two main components. - -1. **HTML/CSS:** what your app looks like (the **form**) - -2. **JavaScript:** what your app does (the **function**) - -The difference between Shiny apps and regular webpages: **Shiny apps are powered by an R session.** - -
- -## Creating a Shiny app - -To create a Shiny app, we need: - -1. Regular R stuff: writing code to manipulate and analyze data, visualize with graphs, etc... - -
- -2. To create a user interface - -3. To connect **1** to **2** with **reactive logic** diff --git a/content/courses/r-shiny/introduction/project3-step1.md b/content/courses/r-shiny/introduction/project3-step1.md deleted file mode 100755 index 42e1bd51..00000000 --- a/content/courses/r-shiny/introduction/project3-step1.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -title: "Step 1: Reorganize the Code" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 410 - -menu: - r-shiny: - parent: Introduction to Shiny ---- - -Start with `app-0.R`. It's a working app-see for yourself! - -Even though the app has very little functionality, the script is already crowded. - -1. Pull out the UI and put it into a file `ui.R`. Do the same for `server`. - -2. Source these into `app-0.R`. - -3. Make sure the app still works! diff --git a/content/courses/r-shiny/introduction/project3-step2.md b/content/courses/r-shiny/introduction/project3-step2.md deleted file mode 100755 index 9be77766..00000000 --- a/content/courses/r-shiny/introduction/project3-step2.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: "Step 2: Tabify the app" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 420 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -Having all the outputs on one page can be messy. - -1. Put each output widget on its own tab with an informative label - -2. Put the input widgets in the same tab as the table display (head of data frame) - -3. Make sure the input widget/table display tab is displayed when you start the app diff --git a/content/courses/r-shiny/introduction/project3-step3.md b/content/courses/r-shiny/introduction/project3-step3.md deleted file mode 100755 index 8328f627..00000000 --- a/content/courses/r-shiny/introduction/project3-step3.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: "Step 3: `response` and `explanatory`" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 430 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -Write some reactive logic so that the appropriate choices appear for the `response` and `explanatory` input widgets - -Hint: use `updateSelectInput()`. The variable names will be returned by `names(Raw_data())`. Should you use a `eventReactive()` or `observeEvent()` to accomplish this? - diff --git a/content/courses/r-shiny/introduction/project3-step4.md b/content/courses/r-shiny/introduction/project3-step4.md deleted file mode 100755 index 95216bfe..00000000 --- a/content/courses/r-shiny/introduction/project3-step4.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: "Step 4: Display the data" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 440 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -Write reactive logic to: - -1. Display the head of the selected data frame in a table (only the response and explanatory variables) - -2. Plot response versus explanatory in a scatterplot - -You'll be using the `render` functions to create the displays. Is there a way you can create a common object for that can be used by both of the `render` functions? Hint: use `reactive()` diff --git a/content/courses/r-shiny/introduction/project3-step5.md b/content/courses/r-shiny/introduction/project3-step5.md deleted file mode 100755 index eccc7315..00000000 --- a/content/courses/r-shiny/introduction/project3-step5.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: "Step 5: Show a summary of the model" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 450 - -menu: - r-shiny-intro: - parent: "Project 3: Plotting a Data Frame" ---- - -Display a summary of the linear model `response ~ explanatory`. - -Hints: -- You can use `paste()` to construct the formula - -- Try `renderPrint` \ No newline at end of file diff --git a/content/courses/r-shiny/introduction/project3.md b/content/courses/r-shiny/introduction/project3.md deleted file mode 100755 index c9e269c5..00000000 --- a/content/courses/r-shiny/introduction/project3.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -title: "Project: Creating a DataFrame" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 400 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - - -It's time to try coding your own Shiny app. With this app, a user will be able to select a data frame from a list of choices, and then select a response and explanatory variable. The app will have 3 types of functionality: - -1. Plot a scatterplot of the data frame (response vs explanatory) - -2. Show the head of the data frame - -3. Print out a regression report diff --git a/content/courses/r-shiny/introduction/reactives-connections-states.md b/content/courses/r-shiny/introduction/reactives-connections-states.md deleted file mode 100755 index 17e3a912..00000000 --- a/content/courses/r-shiny/introduction/reactives-connections-states.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -title: "Project 2: Reactives" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 340 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -## Reactives have connections - -* Inputs have outward connections -* Observers have inward connections -* Conductors have both inward and outward connections - -## Reactive States - -***Reactives have 2 kinds of states*** - -1. **Their value**, e.g. numerical/text value, plot, image -2. **Their validity**--validated, invalidated - -
- -- Reactives are invalidated when their inputs change value. They become valid again when they recalculate or re-execute based on their input's new value. diff --git a/content/courses/r-shiny/introduction/reactives-diagram.md b/content/courses/r-shiny/introduction/reactives-diagram.md deleted file mode 100755 index 52528d3a..00000000 --- a/content/courses/r-shiny/introduction/reactives-diagram.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -title: "Reactivity Diagram" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 370 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -{{< figure src="/courses/r-shiny/introduction/img/reactivity.png" >}} diff --git a/content/courses/r-shiny/introduction/reactives-example.md b/content/courses/r-shiny/introduction/reactives-example.md deleted file mode 100755 index 6d4c9aa3..00000000 --- a/content/courses/r-shiny/introduction/reactives-example.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -title: "Reactives: An Example" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 320 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -``` -server <- function(input, output) { - output$distPlot <- renderPlot({ - hist(rnorm(input$obs)) - }) -} -``` - -In this example, `input$obs` is a reactive source, and `output$distPlot` is a reactive endpoint. - -- Clicking or typing into input widgets will set some sort of value, or **reactive source**. - -- A **reactive endpoint** is an object that appears in the app, like a plot, text, or values. Reactive endpoints re-execute when their reactive inputs change. diff --git a/content/courses/r-shiny/introduction/reactives-kb.md b/content/courses/r-shiny/introduction/reactives-kb.md deleted file mode 100755 index 55029fa5..00000000 --- a/content/courses/r-shiny/introduction/reactives-kb.md +++ /dev/null @@ -1,26 +0,0 @@ ---- -title: "3 Types of KB Passengers" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 330 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -{{< table >}} -| inputs | conductors | observers | -| --- | --- | --- | -| {{< figure src="/courses/r-shiny/introduction/img/input.png" >}} | {{< figure src="/courses/r-shiny/introduction/img/conductor.png" >}} | {{< figure src="/courses/r-shiny/introduction/img/observer.png" >}} | -| actionButton() | reactive() | observe() | -| selectInput() | eventReactive() | observeEvent() | -| reactiveValues() | renderText() | textOutput() | -{{< /table >}} - -- Inputs are reactive sources. Reactive endpoints are called observers because they observe changes to inputs. - -- Conductors act like both sources and endpoints. They depend on inputs, but they also provide inputs to other reactives. diff --git a/content/courses/r-shiny/introduction/reactives-modularizing.md b/content/courses/r-shiny/introduction/reactives-modularizing.md deleted file mode 100755 index aace2c7c..00000000 --- a/content/courses/r-shiny/introduction/reactives-modularizing.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: "Modularizing Reactions" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 360 - -menu: - r-shiny: - parent: Introduction to Shiny ---- - -## Adding an expression - -This script still calculates the hypotenuse but splits the calculation into two parts - -1. `C2 <- reactive(input$A^2 + input$B^2, label = "C2")` - -2. `output$C <- renderText(sqrt(C2()))` - -- Run `pythagorean-2.R` and change the input values. - -- Stop the app - -- Run `reactlogShow()` - -- Reactives are lazy and only update when they need to diff --git a/content/courses/r-shiny/introduction/reactives-reactlogshow.md b/content/courses/r-shiny/introduction/reactives-reactlogshow.md deleted file mode 100755 index cc5a7625..00000000 --- a/content/courses/r-shiny/introduction/reactives-reactlogshow.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: "reactlogShow()" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 350 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -## Project 2: Pythagorean Theorem - -This script calculates the hypotenuse of a right triangle using -``` -output$C <- renderText({ - sqrt(input$A^2 + input$B^2) - }) -``` - -- Run `pythagorean-1.R` and change the input values a few times. - -- Stop the app - -- Run `reactlogShow()` in the console (may need to install first) - -A log will open in your browser - - -## reactlogShow() chart - -- Click the forward arrow in the log window - -- When `input$A` or `input$B`'s value changed, it became invalidated. - -- This then invalidates `output$C` - -- `output$C` becomes validated once it recalculates diff --git a/content/courses/r-shiny/introduction/reactives-vs-observers.md b/content/courses/r-shiny/introduction/reactives-vs-observers.md deleted file mode 100755 index 37c19303..00000000 --- a/content/courses/r-shiny/introduction/reactives-vs-observers.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -title: "Reactives vs. Observers" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 380 - -menu: - r-shiny: - parent: Introduction to Shiny ---- - -**Reactives**: calculate or cache a value, meant to be used as a variable, has a result - -**Observers**: for side effects--runs some code when a dependency changes, doesn't yield a result, isn't used as input for other expressions, not assigned to a variable (e.g., updating a menu based on a selection) - -
- -{{< figure src="/courses/r-shiny/introduction/img/dog.jpeg" >}} - -> Imagine a delivery man rings your doorbell to deliver a package. You head to the front door to receive it. At the same time, your pet dog starts wildly barking at the door. You open the door and take the package from the delivery man. - -In this example: - -- The delivery man ringing the doorbell is a reactive input. - -- You going to the front door is a conductor. - -- Your dog barking is an observer. The dog is reacting to the doorbell ringing, but the barking has nothing to do with package delivery. - -- You taking the package is a reactive endpoint (not a side effect). diff --git a/content/courses/r-shiny/introduction/reactives.md b/content/courses/r-shiny/introduction/reactives.md deleted file mode 100755 index a0400974..00000000 --- a/content/courses/r-shiny/introduction/reactives.md +++ /dev/null @@ -1,26 +0,0 @@ ---- -title: "Project: Reactivity" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 310 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -We've already seen some examples with our `render` functions, but what exactly is **reactivity**? - -## More Harry Potter Comparisons - -Remember, the Knight Bus is our connection between regular R code (Muggle World) and the world of reactives (Wizarding World) - -``` -knight_bus <- function(input, output, session) { - reactive code here! -} -``` - diff --git a/content/courses/r-shiny/introduction/ui-frameworks.md b/content/courses/r-shiny/introduction/ui-frameworks.md deleted file mode 100755 index 9a9cc11d..00000000 --- a/content/courses/r-shiny/introduction/ui-frameworks.md +++ /dev/null @@ -1,68 +0,0 @@ ---- -title: "Frameworks and Layouts" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: true # Show table of contents? true/false -type: docs # Do not modify. -weight: 205 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -- Replace `UI_starting.R` with `UI_fluid_page.R` in the `app.R`. - -- Run the app - -- Pretty ugly, right? Now try replacing `tagList` with `fluidPage` - -
- -## fluidPage() - -A fluid page layout consists of rows which in turn contain columns - -- Rows ensure that items appear on the same line as long as the browser is wide enough (hence the fluid) - -- Columns define how much horizontal space elements should occupy. Horizontal space is defined by a 12-unit wide grid - -- Adds some Bootstrap styling (framework for designing websites--developed by Twitter) - -
- -## fluidRow() and column() - - -Changing `tagList` to `fluidPage` didn't do much - -Let's add some `fluidRows()` and `column()` functions to create this: - -{{< figure src="/courses/r-shiny/introduction/img/fluid-wide.png" >}} - -The solution is in `UI_language_soln.R` - -## `fluidPage()` is responsive - -- Try adjusting the size of your browser window. - -- The text adjusts so that it all fits within a single window--no need to scroll left and right! - -## Other Layouts - -Check out the Shiny cheatsheet to see other types of layouts - -https://shiny.rstudio.com/images/shiny-cheatsheet.pdf - -- `tabsetPanel()` + `tabPanel()` - -- `sidebarLayout()` + `sidebarPanel()`/`mainPanel()` - -- `splitLayout()` - -- `wellPanel()` - -- `navbarPage()` - -- `navlistPanel()` diff --git a/content/courses/r-shiny/introduction/ui-widgets.md b/content/courses/r-shiny/introduction/ui-widgets.md deleted file mode 100755 index 1a8d813f..00000000 --- a/content/courses/r-shiny/introduction/ui-widgets.md +++ /dev/null @@ -1,65 +0,0 @@ ---- -title: "Widgets" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: true # Show table of contents? true/false -type: docs # Do not modify. -weight: 210 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -- Go to `projects/project1-ui` and take a look at `UI_starting.R` and `Knight_bus.R`. - -- Run `app.R`. - -- **Widgets** are the different buttons and fields we see on a webpage. - -- We only see the inputs from `UI_starting.R` because nothing is connected to the outputs. - -
- -## Input Widgets - -### The Shiny Widget Gallery - -- You can try out and play with Shiny input widgets on RStudio's website. - -- The website shows you how the values change when you modify the input widgets. The site will also show you the code you need to include the widget in your own app. - -https://shiny.rstudio.com/gallery/widget-gallery.html - -
- -## Output Widgets - -- Output widgets are similar to input widgets. However, output widgets require a `render` function to be visible in the app. - -- Each output widget has its own corresponding `render` function. - -{{< figure src="/courses/r-shiny/introduction/img/output-widgets.png" >}} - -
- -## Playing with Widgets - -Add some input widgets to `UI_starting.R` - -- actionLink - -- checkboxInput - -- radioButtons - -- textInput - -Make sure the app still works with your changes! - -
- -## Connecting Inputs to Outputs - -Add some reactive logic to `Knight_bus.R` so that the text output `felix` (`output$felix`) displays the selected choice from `annie` (`input$annie`) diff --git a/content/courses/r-shiny/introduction/ui.md b/content/courses/r-shiny/introduction/ui.md deleted file mode 100755 index 51059f62..00000000 --- a/content/courses/r-shiny/introduction/ui.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: "Building a User Interface" -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 200 - -menu: - r-shiny: - parent: Introduction to Shiny - ---- - -There are 3 steps to building a UI: - -1. Instantiating input and output widgets - -2. Frameworks and layouts - -3. Organizing widgets within the framework diff --git a/content/home/hero.md b/content/home/hero.md index ba58f8a9..ad38de9c 100644 --- a/content/home/hero.md +++ b/content/home/hero.md @@ -25,9 +25,9 @@ title = "UVA Research Computing Learning Portal" # Background image. # image = "learning-bg.jpg" # Name of image in `static/img/`. -# image = "scott_stadium_afc_aerial_ss_01.png" - image = "rotunda_lawn_aerial_old_cabell_fall_ss_01.jpg" - image_darken = 0.1 # Darken the image? Range 0-1 where 0 is transparent and 1 is opaque. + image = "scott_stadium_afc_aerial_ss_01.png" +# image = "rotunda_lawn_aerial_old_cabell_fall_ss_01.jpg" + image_darken = 0 # Darken the image? Range 0-1 where 0 is transparent and 1 is opaque. # image_size = "cover" # Options are `cover` (default), `contain`, or `actual` size. # image_position = "center" # Options include `left`, `center` (default), or `right`. # image_parallax = true # Use a fun parallax-like fixed background effect? true/false @@ -46,4 +46,4 @@ title = "UVA Research Computing Learning Portal" +++ -Research Computing provides computational, scientific programming, and data analysis workshops, many of which are offered in partnership with the UVA Library [Research Data Services](https://data.library.virginia.edu/training) and UVA [Health Sciences Library](https://cal.hsl.virginia.edu/calendar/data) websites. This site compiles RC workshop and tutorial materials as well as interactive exercises for students to review or study at their own pace. +Research Computing provides computational, scientific programming, and data analysis tutorials, many of which are offered as live sessions in partnership with the UVA Library [Research Data Services](https://data.library.virginia.edu/training) and UVA [Health Sciences Library](https://cal.hsl.virginia.edu/calendar/data) websites. This site compiles RC's tutorial materials, many of which contain interactive exercises for students to review or study at their own pace. diff --git a/content/home/people.md b/content/home/people.md new file mode 100644 index 00000000..a0f3c62c --- /dev/null +++ b/content/home/people.md @@ -0,0 +1,56 @@ ++++ +# A "Meet the Team" section created with the People widget. +# This section displays people from `content/authors/` which belong to the `user_groups` below. + +type = "widget_page" +widget = "people" # See https://sourcethemes.com/academic/docs/page-builder/ +headless = true # This file represents a page section. +active = true # Activate this widget? true/false + +weight=300 + +title = "Our Authors" +subtitle = "" + +[content] + # Choose which groups/teams of users to display. + # Edit `user_groups` in each user's profile to add them to one or more of these groups. + user_groups = ["staff", + "DAC", + "RC", + "student",] + +[design] + # Show user's social networking links? (true/false) + show_social = false + + # Show user's interests? (true/false) + show_interests = true + +[design.background] + # Apply a background color, gradient, or image. + # Uncomment (by removing `#`) an option to apply it. + # Choose a light or dark text color by setting `text_color_light`. + # Any HTML color name or Hex value is valid. + + # Background color. + # color = "navy" + + # Background gradient. + # gradient_start = "DeepSkyBlue" + # gradient_end = "SkyBlue" + + # Background image. + # image = "background.jpg" # Name of image in `static/img/`. + # image_darken = 0.6 # Darken the image? Range 0-1 where 0 is transparent and 1 is opaque. + + # Text color (true=light or false=dark). + # text_color_light = true + +[advanced] + # Custom CSS. + css_style = "" + + # CSS class. + css_class = "" ++++ diff --git a/content/home/slider.md b/content/home/slider.md index 6ddc3be9..29a384e2 100644 --- a/content/home/slider.md +++ b/content/home/slider.md @@ -2,8 +2,8 @@ # Slider widget. widget = "slider" # See https://sourcethemes.com/academic/docs/page-builder/ headless = true # This file represents a page section. -active = false # Activate this widget? true/false -weight = 1 # Order that this section will appear. +active = true # Activate this widget? true/false +weight = 250 # Order that this section will appear. # Slide interval. # Use `false` to disable animation or enter a time in ms, e.g. `5000` (5s). @@ -12,43 +12,47 @@ interval = false # Slide height (optional). # E.g. `500px` for 500 pixels or `calc(100vh - 70px)` for full screen. height = "" - # Slides. # Duplicate an `[[item]]` block to add more slides. [[item]] - title = "Hello" - content = "I am center aligned :smile:" + title = "Contact Us" align = "center" # Choose `center`, `left`, or `right`. # Overlay a color or image (optional). # Deactivate an option by commenting out the line, prefixing it with `#`. - overlay_color = "#666" # An HTML color value. - overlay_img = "headers/bubbles-wide.jpg" # Image path relative to your `static/img/` folder. + overlay_color = "#232D4B" # An HTML color value. + overlay_img = "grounds_early_fall.jpg" # Image path relative to your `static/img/` folder. overlay_filter = 0.5 # Darken the image. Value in range 0-1. # Call to action button (optional). # Activate the button by specifying a URL and button label below. # Deactivate by commenting out parameters, prefixing lines with `#`. - cta_label = "Get Academic" - cta_url = "https://sourcethemes.com/academic/" + cta_label = "Get Help" + cta_url = "https://rc.virginia.edu/support" cta_icon_pack = "fas" - cta_icon = "graduation-cap" + #cta_icon = "graduation-cap" [[item]] - title = "Left" - content = "I am left aligned :smile:" - align = "left" + title = "Upcoming Presentations" + align = "center" - overlay_color = "#555" # An HTML color value. - overlay_img = "" # Image path relative to your `static/img/` folder. + overlay_color = "#232D4B" # An HTML color value. + overlay_img = "learning-bg.jpg" # Image path relative to your `static/img/` folder. overlay_filter = 0.5 # Darken the image. Value in range 0-1. + cta_label = "Next Events" + cta_url = "https://rc.virginia.edu/education/workshops/" + [[item]] - title = "Right" - content = "I am right aligned :smile:" - align = "right" + title = "Visit Office Hours" + #content = "I am left aligned :smile:" + align = "center" - overlay_color = "#333" # An HTML color value. - overlay_img = "" # Image path relative to your `static/img/` folder. + overlay_color = "#232D4B" # An HTML color value. + overlay_img = "brooks_hall_spring_ss_02.jpg" overlay_filter = 0.5 # Darken the image. Value in range 0-1. + + cta_label = "Office Hours" + cta_url = "https://rc.virginia.edu/support/#office-hours" + cta_icon_pack = "fas" +++ diff --git a/content/landing/_index.md b/content/landing/_index.md index 8ec49902..47ff9b43 100644 --- a/content/landing/_index.md +++ b/content/landing/_index.md @@ -7,9 +7,9 @@ subtitle = "" Some tutorials are offered as in-person and/or Zoom workshops regularly. Find the notes or slides and download materials [on the Tutorials page](/tutorials). -For a list of upcoming workshops, please see [the Workshops page](https://www.rc.virginia.edu/education/workshops/) at the UVARC Website. +For a list of upcoming tutorials, please see [the Workshops page](https://www.rc.virginia.edu/education/workshops/) at the UVARC Website. -## [Short courses](/courses) are longer and more in-depth than tutorials or workshops. +## [Programming courses](/courses) are longer and more in-depth than tutorials or workshops and teach more fundamental programming topics.
diff --git a/content/notes/biopython/AnacondaNavigator.png b/content/notes/biopython/AnacondaNavigator.png deleted file mode 100644 index bc82df75..00000000 Binary files a/content/notes/biopython/AnacondaNavigator.png and /dev/null differ diff --git a/content/notes/biopython/Spyder.png b/content/notes/biopython/Spyder.png deleted file mode 100644 index 3611b56c..00000000 Binary files a/content/notes/biopython/Spyder.png and /dev/null differ diff --git a/content/notes/biopython/anaconda-navigator.png b/content/notes/biopython/anaconda-navigator.png deleted file mode 100644 index 5a5c6668..00000000 Binary files a/content/notes/biopython/anaconda-navigator.png and /dev/null differ diff --git a/content/notes/biopython/anaconda-spyder.png b/content/notes/biopython/anaconda-spyder.png deleted file mode 100644 index 6d7fc105..00000000 Binary files a/content/notes/biopython/anaconda-spyder.png and /dev/null differ diff --git a/content/notes/biopython/data/Pax6-multispec-protein.aln b/content/notes/biopython/data/Pax6-multispec-protein.aln deleted file mode 100755 index 03e63ebc..00000000 --- a/content/notes/biopython/data/Pax6-multispec-protein.aln +++ /dev/null @@ -1,152 +0,0 @@ -CLUSTAL O(1.2.4) multiple sequence alignment - - -NP_524628.2 MFTLQPTPTAIGTVVPPWSAGTLIERLPSLEDMAHKGHSGVNQLGGVFVGGRPLPDSTRQ 60 -NP_524638.3 ---MMLTTEHIMHGHPH-----SSVGQSTLFGCSTAGHSGINQLGGVYVNGRPLPDSTRQ 52 -NP_001006763.1 ---------------------------------MQNSHSGVNQLGGVFVNGRPLPDSTRQ 27 -NP_001355831.1 ------------------------------------------------------------ 0 -AAH36957.1 ---------------------------------MQNSHSGVNQLGGVFVNGRPLPDSTRQ 27 -XP_029701655.1 --------------------------------MMQNSHSGVNQLGGVFVNGRPLPDSTRQ 28 -NP_571379.1 ----MPQKEY-Y----N-----RATWESGVASMMQNSHSGVNQLGGVFVNGRPLPDSTRQ 46 -NP_571716.1 ----MPQKEY-H----N-----QPTWESGVASMMQNSHSGVNQLGGVFVNGRPLPDSTRQ 46 - - -NP_524628.2 KIVELAHSGARPCDISRILQ--------------VSNGCVSKILGRYYETGSIRPRAIGG 106 -NP_524638.3 KIVELAHSGARPCDISRILQ--------------VSNGCVSKILGRYYETGSIKPRAIGG 98 -NP_001006763.1 KIVELAHSGARPCDISRILQ--------------VSNGCVSKILGRYYETGSIRPRAIGG 73 -NP_001355831.1 ------------------------------------------------------------ 0 -AAH36957.1 KIVELAHSGARPCDISRILQ--------------VSNGCVSKILGRYYETGSIRPRAIGG 73 -XP_029701655.1 KIVELAHSGARPCDISRILQTHD--AVQVLDSEKVSNGCVSKILGRYYETGSIRPRAIGG 86 -NP_571379.1 KIVELAHSGARPCDISRILQTHADAKVQVLDNENVSNGCVSKILGRYYETGSIRPRAIGG 106 -NP_571716.1 KIVELAHSGARPCDISRILQ--------------VSNGCVSKILGRYYETGSIRPRAIGG 92 - - -NP_524628.2 SKPRVATAEVVSKISQYKRECPSIFAWEIRDRLLQENVCTNDNIPSVSSINRVLRNLAAQ 166 -NP_524638.3 SKPRVATTPVVQKIADYKRECPSIFAWEIRDRLLSEQVCNSDNIPSVSSINRVLRNLASQ 158 -NP_001006763.1 SKPRVATPEVVSKIAQYKRECPSIFAWEIRDRLLSEGVCTNDNIPSVSSINRVLRNLASE 133 -NP_001355831.1 ------------------------------------------------------------ 0 -AAH36957.1 SKPRVATPEVVSKIAQYKRECPSIFAWEIRDRLLSEGVCTNDNIPSVSSINRVLRNLASE 133 -XP_029701655.1 SKPRVATPEVVAKIAQYKRECPSIFAWEIRDRLLSEGICTNDNIPSVSSINRVLRNLASE 146 -NP_571379.1 SKPRVATPEVVGKIAQYKRECPSIFAWEIRDRLLSEGVCTNDNIPSVSSINRVLRNLASE 166 -NP_571716.1 SKPRVATPEVVGKIAQYKRECPSIFAWEIRDRLLSEGVCTNDNIPSVSSINRVLRNLASE 152 - - -NP_524628.2 KEQQSTGSGSSSTSAGNSISAKVSVSIGGNVSNVASGSRGTLSSSTDLMQTATPLNSSES 226 -NP_524638.3 KEQQAQQ----------------------------------------------------- 165 -NP_001006763.1 KQQM-------------------------------------------------------- 137 -NP_001355831.1 ---M-------------------------------------------------------- 1 -AAH36957.1 KQQM-------------------------------------------------------- 137 -XP_029701655.1 KQQM-------------------------------------------------------- 150 -NP_571379.1 KQQM-------------------------------------------------------- 170 -NP_571716.1 KQQM-------------------------------------------------------- 156 - - -NP_524628.2 GGASNSGEGSEQEAIYEKLRLLNTQHAAGPGPLEPARAAPLVGQSPNHLGTRSSHPQLVH 286 -NP_524638.3 ----------QNESVYEKLRMFNGQTG--------------------------------- 182 -NP_001006763.1 ----------GADGMYDKLRMLNGQTG--------------------------------- 154 -NP_001355831.1 ----------GADGMYDKLRMLNGQTG--------------------------------- 18 -AAH36957.1 ----------GADGMYDKLRMLNGQTG--------------------------------- 154 -XP_029701655.1 ----------GADGMYDKLRMLNGQTG--------------------------------- 167 -NP_571379.1 ----------GADGMYEKLRMLNGQTG--------------------------------- 187 -NP_571716.1 ----------GADGMYDKLRMLNGQSG--------------------------------- 173 - :.:*:***::* * . - -NP_524628.2 GNHQALQQHQQQSWPPRHYSGSWYPTSLSEIPISSAPNIASVTAYASGPSLAHSLSPPND 346 -NP_524638.3 ------------GWA-------WYPSNTTTAHLTLPPAASVVT-------------SPAN 210 -NP_001006763.1 ------------TWGT---RPGWYPGTSVPG----------------------------- 170 -NP_001355831.1 ------------SWGT---RPGWYPGTSVPG----------------------------- 34 -AAH36957.1 ------------SWGT---RPGWYPGTSVPG----------------------------- 170 -XP_029701655.1 ------------TWGT---RPGWYPGTSVPG----------------------------- 183 -NP_571379.1 ------------TWGT---RPGWYPGTSVPG----------------------------- 203 -NP_571716.1 ------------TWGT---RPGWYPGTSVPG----------------------------- 189 - * *** . - -NP_524628.2 IESLASIGHQRNCPVATEDIHLKKELDGHQSDETGSGEGENSNGGASNIGNTEDDQARLI 406 -NP_524638.3 LS---GQADRDDVQK--RELQFSVEVS-----HTNSHDSTSDGNSEHNSSGDEDSQMRLR 260 -NP_001006763.1 ------QPAQDGCQP--QEGGG---------------GGENTNSISSNGEDSDEAQMRLQ 207 -NP_001355831.1 ------QPTQDGCQQ--QEGG-----------------GENTNSISSNGEDSDEAQMRLQ 69 -AAH36957.1 ------QPTQDGCQQ--QEGG-----------------GENTNSISSNGEDSDEAQMRLQ 205 -XP_029701655.1 ------QPNQDGCQQ--QDG-----------------AGENTNSISSNGEDSEETQLRLQ 218 -NP_571379.1 ------QPNQDGCQQ--SDG-----------------GGENTNSISSNGEDSDETQMRLQ 238 -NP_571716.1 ------QPNQDGCQQ--QDN-----------------GGENTNSISSNGEDSDETQMRLQ 224 - : . : . . .. * . :: * ** - -NP_524628.2 LKRKLQRNRTSFTNDQIDSLEKEFERTHYPDVFARERLAGKIGLPEARIQVWFSNRRAKW 466 -NP_524638.3 LKRKLQRNRTSFSNEQIDSLEKEFERTHYPDVFARERLADKIGLPEARIQVWFSNRRAKW 320 -NP_001006763.1 LKRKLQRNRTSFTQEQIEALEKEFERTHYPDVFARERLAAKIDLPEARIQVWFSNRRAKW 267 -NP_001355831.1 LKRKLQRNRTSFTQEQIEALEKEFERTHYPDVFARERLAAKIDLPEARIQVWFSNRRAKW 129 -AAH36957.1 LKRKLQRNRTSFTQEQIEALEKEFERTHYPDVFARERLAAKIDLPEARIQVWFSNRRAKW 265 -XP_029701655.1 LKRKLQRNRTSFTQEQIENLEKEFERTHYPDVFARERLAAKIDLPEARIQVWFSNRRAKW 278 -NP_571379.1 LKRKLQRNRTSFTQEQIEALEKEFERTHYPDVFARERLAAKIDLPEARIQVWFSNRRAKW 298 -NP_571716.1 LKRKLQRNRTSFTQEQIEALEKEFERTHYPDVFARERLAAKIDLPEARIQVWFSNRRAKW 284 - ************:::**: ******************** **.***************** - -NP_524628.2 RREEKLRNQRRTPNSTGASATSSSTSATASLTDSPNSLSACSSLLSGSAGGPSVSTINGL 526 -NP_524638.3 RREEKMRTQRRSADTVDGSGRTSTAN-------NPSGTTASSSVATSNNST--------- 364 -NP_001006763.1 RREEKLRNQRRQASNTPSHIP------------------ISSSFSTSVY----------- 298 -NP_001355831.1 RREEKLRNQRRQASNTPSHIP------------------ISSSFSTSVY----------- 160 -AAH36957.1 RREEKLRNQRRQASNTPSHIP------------------ISSSFSTSVY----------- 296 -XP_029701655.1 RREEKLRNQRRQA-NSSSHIP------------------ISSSFSTSVY----------- 308 -NP_571379.1 RREEKLRNQRRQASNSSSHIP------------------ISSSFSTSVY----------- 329 -NP_571716.1 RREEKLRNQRRQASNSSSHIP------------------ISSSFNTSVY----------- 315 - *****:*.*** . . .**. :. - -NP_524628.2 SSPSTLSTNVNAPTLGAGIDSSESPTPIPHIRPSCTSDNDNGRQSEDCRRVCSPCPLGVG 586 -NP_524638.3 --PGIVNSAINVAER---------T--SSALVSNSLPEASN-----------GPTVLGGE 400 -NP_001006763.1 -------QPIPQPTT---------P--VSSFTS-------------------G-SMLGRT 320 -NP_001355831.1 -------QPIPQPTT---------P--VSSFTS-------------------G-SMLGRT 182 -AAH36957.1 -------QPIPQPTT---------P--VSSFTS-------------------G-SMLGRT 318 -XP_029701655.1 -------QAIPQPTT---------P--V-SFTS-------------------G-SMLGRP 329 -NP_571379.1 -------QPIPQPTT---------P--V-SFTS-------------------G-SMLGRS 350 -NP_571716.1 -------QAIPQPTT---------P--V-SFTT-------------------G-SMLGRP 336 - : : . ** - -NP_524628.2 GHQNTHHIQSNGHAQGHALVPAISPRLNFNSGSFGAMYSNMHHTALSMSDSYGAVTPIPS 646 -NP_524638.3 A-NTTH-TSS-----ESPPLQPAAPRLPLNSG-FNTMYSSIPQPIATMAENYNS------ 446 -NP_001006763.1 DTALTN-SYS-----ALPPM-----------------------PSFTMGNNL-------- 343 -NP_001355831.1 DTALTN-TYS-----ALPPM-----------------------PSFTMANNL-------- 205 -AAH36957.1 DTALTN-TYS-----ALPPM-----------------------PSFTMANNL-------- 341 -XP_029701655.1 DSALTN-TYS-----ALPPM-----------------------PSFTMANNL-------- 352 -NP_571379.1 DTALTN-TYS-----ALPPM-----------------------PSFTMANNL-------- 373 -NP_571716.1 DTALTN-TYT-----GLPPM-----------------------PSFTMANNL-------- 359 - *: : : :*.:. - -NP_524628.2 FNHSAVGPLAPPSPIPQQGDLTPSSLYPCHMTLRPP----PMAPAHHHIVPGD----GGR 698 -NP_524638.3 ----SLGSMTPS-------CLQQRDAYPYMFHDPLS-LGSPYVSAHHRNTACNPSAAHQQ 494 -NP_001006763.1 -------PMQPP-------VPSQTSSYSCMLPTSPSVNGRSY--------------DTYT 375 -NP_001355831.1 -------PMQPP-------VPSQTSSYSCMLPTSPSVNGRSY--------------DTYT 237 -AAH36957.1 -------PMQPP-------VPSQTSSYSCMLPTSPSVNGRSY--------------DTYT 373 -XP_029701655.1 -------PMQP----------SQTSSYSCMLPTSPPVNGRSY--------------DTYT 381 -NP_571379.1 -------PMQP----------SQTSSYSCMLPTSPSVNGRSY--------------DTYT 402 -NP_571716.1 -------PMQP----------SQTSSYSCMLPSSPSVNGRSF--------------DTYT 388 - : * . * : - -NP_524628.2 PAGVG-------LGSGQSANLGASCSGSGYEVLSAYALPPPPMASSSAADSSFSAASSAS 751 -NP_524638.3 PPQHGVYTNSSPM---PSSNTGVISAGVSV---------PVQISTQNVSDLT-------G 535 -NP_001006763.1 PPHMQTHMNSQPMGTSGTASTGLISPGVSV---------PVQVPGS---EPD-------M 416 -NP_001355831.1 PPHMQTHMNSQPMGTSGTTSTGLISPGVSV---------PVQVPGS---EPD-------M 278 -AAH36957.1 PPHMQTHMNSQPMGTSGTTSTGLISPGVSV---------PVQVPGS---EPD-------M 414 -XP_029701655.1 PPHMQAHMNSQSMTTSGTTSTGLISPGVSV---------PVQVPGS---EPD-------M 422 -NP_571379.1 PPHMQAHMNSQSMAASGTTSTGLISPGVSV---------PVQVPGS---EPD-------M 443 -NP_571716.1 PPHMQAHMNSQTMATSGTASTGLISPGVSV---------PVQVPGT---EPD-------M 429 - * : ::. * . * . * : : - -NP_524628.2 ANVTPHHTIAQESCPSPCSSASHFGVAHSSGFSSDPISPAVSSYAHMSYNYASSANTMTP 811 -NP_524638.3 SNYWPRLQ---------------------------------------------------- 543 -NP_001006763.1 SQYWPRLQ---------------------------------------------------- 424 -NP_001355831.1 SQYWPRLQ---------------------------------------------------- 286 -AAH36957.1 SQYWPRLQ---------------------------------------------------- 422 -XP_029701655.1 SQYWSRLQ---------------------------------------------------- 430 -NP_571379.1 SQYWPRLQ---------------------------------------------------- 451 -NP_571716.1 SQYWSRLQ---------------------------------------------------- 437 - :: : - -NP_524628.2 SSASGTSAHVAPGKQQFFASCFYSPWV 838 -NP_524638.3 --------------------------- 543 -NP_001006763.1 --------------------------- 424 -NP_001355831.1 --------------------------- 286 -AAH36957.1 --------------------------- 422 -XP_029701655.1 --------------------------- 430 -NP_571379.1 --------------------------- 451 -NP_571716.1 --------------------------- 437 - \ No newline at end of file diff --git a/content/notes/biopython/data/biopython-workshop.zip b/content/notes/biopython/data/biopython-workshop.zip deleted file mode 100644 index 61ae7b37..00000000 Binary files a/content/notes/biopython/data/biopython-workshop.zip and /dev/null differ diff --git a/content/notes/biopython/index.md b/content/notes/biopython/index.md deleted file mode 100644 index d509cf25..00000000 --- a/content/notes/biopython/index.md +++ /dev/null @@ -1,906 +0,0 @@ ---- -title: Biopython -author: khs -highlight_style: "github" -date: "2020-10-01T00:00:00" -toc: true -type: article -draft: false ---- - -# Introduction - -From the [official Biopython project website](https://biopython.org): - -> Biopython is a set of freely available tools for biological computation written in Python by an international team of developers.
It is a distributed collaborative effort to develop Python libraries and applications which address the needs of current and future work in bioinformatics. The source code is made available under the Biopython License, which is extremely liberal and compatible with almost every license in the world. - -This workshop assumes a working knowledge of the Python programming language and basic understanding of the concepts of online DNA and Protein sequence repositories. - -Introductions to Python can be found [here](/courses/programming_python_scientists_engineers/python-interpreter/) and [here](/courses/python_introduction/). - ---- - -# Getting Started - -**Python code examples** - -The Python scripts and data files for this workshop can be [downloaded from here](data/biopython-workshop.zip). On your computer, unzip the downloaded folder and use it as working directory for this workshop. - -**Python programming environment** - -The Anaconda environment from [Anaconda Inc.](https://anaconda.com/) is widely used because it bundles a Python interpreter, most of the popular packages, and development environments. It is cross-platform and freely available. There are two somewhat incompatible versions of Python; version 2.7 is deprecated but still fairly widely used. Version 3 is the supported version. - -**Note: The latest Biopython package version (1.77+) requires Python 3.** - -1. Visit the [Anaconda download website](https://www.anaconda.com/products/individual#Downloads) and download the installer for Python 3 for your operating system (Windows, Mac OSX, or Linux). We recommend to use the graphical installer for ease of use. - -2. Launch the downloaded installer, follow the onscreen prompts and install the Anaconda distribution on your local hard drive. - -The [Anaconda Documentation](https://docs.anaconda.com/anaconda/user-guide/getting-started/) provides an introduction to the Anaconda environment and bundled applications. For the purpose of this workshop we focus on the `Anaconda Navigator` and `Spyder`. - -## Navigator - -Once you have installed Anaconda, start the Navigator application: -* [Instructions for Windows](https://docs.anaconda.com/anaconda/user-guide/getting-started/#open-nav-win) -* [Instructions for Mac](https://docs.anaconda.com/anaconda/user-guide/getting-started/#open-nav-mac) -* [Instructions for Linux](https://docs.anaconda.com/anaconda/user-guide/getting-started/#open-nav-lin) - -You should see a workspace similar to the screenshot, with several options for working environments, some of which are not installed. We will use `Spyder` which should already be installed. If not, click the button to install the package. - - - -## Spyder - -Now we will switch to Spyder. Spyder is an Integrated Development Environment, or IDE, aimed at Python. It is well suited for developing longer, more modular programs. - -1. To start it, return to the `Anaconda Navigator` and click on the `Spyder` tile. It may take a while to open (watch the lower left of the Navigator). -2. Once it starts, you will see a layout with an editor pane on the left, an explorer pane at the top right, and an iPython console on the lower right. This arrangement can be customized but we will use the default for our examples. Type code into the editor. The explorer window can show files, variable values, and other useful information. The iPython console is a frontend to the Python interpreter itself. It is comparable to a cell in JupyterLab. - - - -## Installation of the Biopython package - -It is recommended to install the `biopython` package from PyPI using the `pip install` command. Detailed instructions are available [here](https://biopython.org/wiki/Download). - -**On your own computer:** -Start the `Anaconda Prompt` command line tool following the instructions for your operating system. -* Start Anaconda Prompt on [Windows](https://docs.anaconda.com/anaconda/user-guide/getting-started/#open-prompt-win) -* Start Anaconda Prompt on [Mac](https://docs.anaconda.com/anaconda/user-guide/getting-started/#open-prompt-mac), or open a terminal window. -* [Linux:](https://docs.anaconda.com/anaconda/user-guide/getting-started/#open-prompt-lin) Just open a terminal window. - -At the prompt, type the following command and press enter/return: -```bash -pip install biopython -``` -This command will install the latest biopython package version in your current Anaconda Python environment. - -**On Rivanna (UVA's HPC platform):** - -[Rivanna](https://www.rc.virginia.edu/userinfo/rivanna/overview/) offers several Anaconda distributions with different Python versions. Before you use Python you need to load one of the `anaconda` software modules and then run the `pip install` command. - -```bash -module load anaconda -pip install --user biopython -``` -**Note:** You have to use the `--user` flag which instructs the interpreter to install the package in your home directory. Alternatively, create your own custom [Conda environment](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) first and run the `pip install biopython` command in that environment (without `--user` flag) - -## Testing the Biopython Installation - -Start the Spyder IDE (see [here](#spyder)). In the `IPython console` pane, type the following command and press `enter/return`: - -``` -import Bio -print (Bio.__version__) -``` - -If the package is installed correctly, the output will show the biopython version number. - ---- - -# Bio Subpackages and Classes - -The `Bio` package provides a large number of subpackages providing specific functionality. The Biopython website provides a [full list of all subpackages](https://biopython.org/docs/1.78/api/Bio.html#subpackages). - -The following table shows an excerpt of that list relevant for this workshop. - -| Subpackages/Classes | Purpose | -| - | - | -| [Bio.Entrez](https://biopython.org/docs/1.75/api/Bio.Entrez.html) | Functions to retrieve Entrez records and associated data | -| [Bio.ExPASy](https://biopython.org/docs/1.78/api/Bio.ExPASy.html) | Tools to access data hosted on the ExPASy protein databases | -| [Bio.SwissProt](https://biopython.org/docs/1.78/api/Bio.SwissProt.html) | Tools to work with the sprotXX.dat file from SwissProt | -| [Bio.Seq](https://biopython.org/docs/1.78/api/Bio.Seq.html?highlight=seq#module-Bio.Seq) | Sequence datastructure (immutable=read-only) | -| [Bio.MutableSeq](https://biopython.org/docs/1.78/api/Bio.Seq.html?highlight=mutableseq#Bio.Seq.MutableSeq) | Sequence datastructure (mutable=modifiable | -| [Bio.SeqRecord](https://biopython.org/docs/1.78/api/Bio.SeqRecord.html?highlight=seqrecord#module-Bio.SeqRecord) | Datastucture for Seq object plus enriched information | -| [Bio.SeqIO](https://biopython.org/docs/1.78/api/Bio.SeqIO.html?highlight=seqio#module-Bio.SeqIO) | Read/write sequences (various file formats )| -| [Bio.AlignIO](https://biopython.org/docs/1.78/api/Bio.AlignIO.html) | A new multiple sequence Alignment Input/Output interface for BioPython 1.46 and later | -| [Bio.Align.MultipleSeqAlignment](https://biopython.org/docs/1.78/api/Bio.Align.html?highlight=mult#Bio.Align.MultipleSeqAlignment) | Tools for Code for working with sequence alignments | - ---- - -# Online Datasets and Databases - -The `Bio` module provides several classes to process output and datasets provided by the following web services and tools: -* FASTA -* Blast output – both from standalone and WWW Blast -* Clustalw -* GenBank -* PubMed and Medline -* ExPASy files, like Enzyme and Prosite -* SCOP, including ‘dom’ and ‘lin’ files -* UniGene -* SwissProt - -In this workshop we will explore options to download nucleotide and protein sequences from [Entrez](https://www.ncbi.nlm.nih.gov/Web/Search/entrezfs.html) and [ExPASy](https://www.expasy.org). - -
- -## Accessing NCBI's Entrez Databases - -The `Bio.Entrez` submodule provides access to the Entrez databases. When you use this module you need to know the String descriptor of the database you want to query (aka its name). A list of valid database names is provided in [column three of this table](https://www.ncbi.nlm.nih.gov/books/NBK25497/table/chapter2.T._entrez_unique_identifiers_ui/?report=objectonly). - -**Note:** Please review the [General Introduction to the E-utilities](https://www.ncbi.nlm.nih.gov/books/NBK25497/) for accessing the Entrez Application Programming Interface Program. The E-utilities limit the frequency of API calls and your IP address may be blocked if you continuously exceed the limit. - -**Basic Steps:** - -1. Provide an email address. This is required! `Entrez.email = "your@somewhere"`. -2. Use an [E-utility](https://www.ncbi.nlm.nih.gov/books/NBK25497/) to get a __handle__ for the data of interest, e.g. `handle = Entrez.esearch(...)`. -3. Use __handle__ to read or parse data with `handle.read()` or `handle.parse()`. -4. Close the __handle__ with `handle.close()`. - -Also read ["What the heck is a handle?"](http://biopython.org/DIST/docs/tutorial/Tutorial.html#sec416) - -**Find Database Records:** - -Let's find the **protein** records associated with the **human Pax6 gene** and download the associated sequences in [FASTA](https://en.wikipedia.org/wiki/FASTA_format) format. - -To search the database we use the `Entrez.esearch` function. We need to specify the database via the `db`, argument and specify a search `term` (provided as a list of Strings). - -The following code is available in the __entrez-fasta.py__ file. - -```python -from Bio import Entrez - -Entrez.email = "YOU@SOMEWHERE.com" # your email address is required -handle = Entrez.esearch(db="protein", term = ["Homo sapiens[Orgn] AND pax6[Gene]"], usehistory="y") -record = Entrez.read(handle) -handle.close() -# iterate over items -for k,v in record.items(): - print (k,v) -``` - -
-
-Output
- -```bash -Count 88 -RetMax 20 -RetStart 0 -QueryKey 1 -WebEnv MCID_5f87b7433f6876111801786c -IdList ['1587062735', '1587062729', '1587062727', '1587062721', '1587062719', '1587062717', '1587062715', '1587062713', '1587062710', '1587062708', '1587062706', '1587062703', '1587062701', '1587062699', '1587062697', '1587062695', '1587062690', '1587062688', '1587062686', '1587062684'] -TranslationSet [{'From': 'Homo sapiens[Orgn]', 'To': '"Homo sapiens"[Organism]'}] -TranslationStack [{'Term': '"Homo sapiens"[Organism]', 'Field': 'Organism', 'Count': '1423829', 'Explode': 'Y'}, {'Term': 'pax6[Gene]', 'Field': 'Gene', 'Count': '2621', 'Explode': 'N'}, 'AND'] -QueryTranslation "Homo sapiens"[Organism] AND pax6[Gene] -``` -- -The search results are returned as a [dictionary](https://docs.python.org/3/tutorial/datastructures.html#dictionaries) and we can retrieve the list of unique IDs that match our query via `record["IdList"]`. - -**Note:** The `IdList` returned by `esearch` is limited to the top 20 hits by default (defined by `retmax`). There are two workarounds: -1. Use the `retmax=
-
-Output:
- -```bash -Count 106 -RetMax 5 -RetStart 0 -QueryKey 1 -WebEnv MCID_5f87bb652dd862297e1727ea -IdList ['1844161464', '1844139642', '1844139635', '1844139631', '1844139629'] -TranslationSet [{'From': 'Homo sapiens[Orgn]', 'To': '"Homo sapiens"[Organism]'}] -TranslationStack [{'Term': '"Homo sapiens"[Organism]', 'Field': 'Organism', 'Count': '27682287', 'Explode': 'Y'}, {'Term': 'pax6[Gene]', 'Field': 'Gene', 'Count': '3232', 'Explode': 'N'}, 'AND'] -QueryTranslation "Homo sapiens"[Organism] AND pax6[Gene] -1844161464 -Saving to file: HsPax6-1844161464-nucleotide.gb -1844139642 -Saving to file: HsPax6-1844139642-nucleotide.gb -1844139635 -Saving to file: HsPax6-1844139635-nucleotide.gb -1844139631 -Saving to file: HsPax6-1844139631-nucleotide.gb -1844139629 -Saving to file: HsPax6-1844139629-nucleotide.gb -Done -``` -- -Note that the `record['IdList']` may not represent all the records. Remember that the `record['WebEnv']` and `record['QueryKey']` entries provide access to the search history on the remote server. So we can use these instead of the `record['IdList']` to get all records. -``` -# Alternative: fetch all records using search history -handle = Entrez.efetch(db="protein", rettype="fasta", retmode="text", webenv=record["WebEnv"], query_key=record["QueryKey"]) -``` - -**Exercise:** Find and download the top 10 FASTA EST nucleotide sequences for the mouse (Mus Musculus) TP53 tumor suppressor. **Hint:** look up the EST database descriptor in [this table](https://www.ncbi.nlm.nih.gov/books/NBK25497/table/chapter2.T._entrez_unique_identifiers_ui/?report=objectonly). - -
-
-
-Solution:
- -```python -handle = Entrez.esearch(db="nucest", term = ["Mus musculus[Orgn] AND tp53[Gene]"], retmax=10, usehistory="y") -record = Entrez.read(handle) -handle.close() -for k,v in record.items(): - print (k,v) - -# iterate over ids in list -for seq_id in record["IdList"]: - # get entry from Entrez - print (seq_id) - handle = Entrez.efetch(db="nucest", id=seq_id, rettype="fasta", retmode="text") - result = handle.read() - handle.close() - # save - filename = f"MmP53-{seq_id}-est.fasta" - print ("Saving to file:", filename) - with open(filename, 'w') as fastafile: - # append fasta entry - fastafile.write(result.rstrip()+"\n") -``` -- -## Retrieve Protein Records from the ExPASy Database - -Here are a few examples demonstrating how to access the ExPASy databases Swissport and Prosite. The [Biopython documentation](https://biopython.readthedocs.io/en/latest/chapter_uniprot.html) provides more details. - -**Swiss-Prot** - -```python -from Bio import ExPASy -from Bio import SwissProt - -# get single protein record -accession_no = "O23729" -handle = ExPASy.get_sprot_raw(accession_no) -record = SwissProt.read(handle) -print (record.entry_name) -print (record.sequence_length) -print (record.data_class) -print (record.accessions) -print (record.organism) -# print first 10 aa -print (record.sequence[:10]) # string -``` - -
-
-
-Output:
- -```bash -CHS3_BROFI -394 -Reviewed -['O23729'] -Bromheadia finlaysoniana (Orchid). -MAPAMEEIRQ -``` -- -The return type of `SwissProt.read()` is a [Bio.SwissProt.Record](https://biopython.org/docs/1.75/api/Bio.SwissProt.html) object. In the above example we're printing only a subset of its fields. The `record.sequence` field is a string, but it can easily be converted into a [Bio.Seq](#sequence-objects) object. - -**Tip:** Use `dir(record)` to get a list of all record attribute names. - -**Prosite** - -```python -from Bio import ExPASy -from Bio.ExPASy import Prosite - -handle = ExPASy.get_prosite_raw("PS00001") -record = Prosite.read(handle) -print (record.name) -print (record.type) # e.g. PATTERN, MATRIX, or RULE -print (record.pattern) -print (record.rules) -print (record.matrix) -``` - -
-
-Output:
- -```bash -ASN_GLYCOSYLATION -PATTERN -N-{P}-[ST]-{P}. -[] -[] -``` -- -The return type of `Prosite.read()` is a [Bio.ExPASy.Prosite.Record](https://biopython.org/docs/1.75/api/Bio.ExPASy.Prosite.html) object. -**Note:** Use the `Bio.ExPASy.Prosite.parse()` function to parse files containing multiple records. - -**Prosite Documentation** -``` -from Bio import ExPASy -from Bio.ExPASy import Prodoc - -handle = ExPASy.get_prosite_raw("PDOC00001") -record = Prodoc.read(handle) -``` - -**Exercise:** Retrieve the SwissProt records for proteins with the following IDs: "O23729", "O23730", "O23731". Try to use list comprehension to create a list containing the records for all retrieved proteins. -
-
-Solution
- -```python -from Bio import ExPASy,SwissProt - -accession_nos = ["O23729", "O23730", "O23731"] -handles = [ExPASy.get_sprot_raw(a) for a in accession_nos] -records = [SwissProt.read(h) for h in handles] -for record in records: - print(record.entry_name) - print(",".join(record.accessions)) - print(record.keywords) - print(repr(record.organism)) - print(record.sequence[:20],"\n") -``` -- -Learn more about [SwissProt](http://biopython.org/DIST/docs/tutorial/Tutorial.html#sec%3Aexpasy_swissprot). - - -**ScanProsite** - -We can query the Prosite database with protein sequences or motifs to find proteins with corresponding matches, see [ScanProsite](https://prosite.expasy.org/scanprosite/scanprosite_doc.html) for details. - -_Option 1_: Submit protein sequence (use the `seq=` keyword argument) -* UniProtKB accessions e.g. P98073 -* Identifiers e.g. ENTK_HUMAN -* PDB identifiers e.g. 4DGJ -* Sequences in FASTA format. - -_Option 2_: Submit motif sequence (use the `sig=` keyword argument) -* PROSITE accession e.g. PS50240 -* Identifier e.g. TRYPSIN_DOM -* Your own pattern e.g. P-x(2)-G-E-S-G(2)-[AS]. -* Combinations of motifs can also be used. - - -```python -from Bio.ExPASy import ScanProsite - -uniprot_id = "P26367" # human Pax-6 -handle = ScanProsite.scan(seq=uniprot_id) -results = ScanProsite.read(handle) -``` -By executing `handle.read()`, you can obtain the search results in raw XML format. Here we use `Bio.ExPASy.ScanProsite.read` to parse the raw XML into a `Bio.ExPASy.ScanProsite.Record` object which represents a specialized list. - -We can now access the found matches like this: -``` -print ("Number of matches:", results.n_match) -for r in results: - print (r) -``` - -
-
-Output:
- -```bash -Number of matches: 4 -{'sequence_ac': 'P26367', 'sequence_id': 'PAX6_HUMAN', 'sequence_db': 'sp', 'start': 4, 'stop': 130, 'signature_ac': 'PS51057', 'signature_id': 'PAIRED_2', 'score': '64.941', 'level': '0'} -{'sequence_ac': 'P26367', 'sequence_id': 'PAX6_HUMAN', 'sequence_db': 'sp', 'start': 38, 'stop': 54, 'signature_ac': 'PS00034', 'signature_id': 'PAIRED_1', 'level_tag': '(0)'} -{'sequence_ac': 'P26367', 'sequence_id': 'PAX6_HUMAN', 'sequence_db': 'sp', 'start': 208, 'stop': 268, 'signature_ac': 'PS50071', 'signature_id': 'HOMEOBOX_2', 'score': '20.164', 'level': '0'} -{'sequence_ac': 'P26367', 'sequence_id': 'PAX6_HUMAN', 'sequence_db': 'sp', 'start': 243, 'stop': 266, 'signature_ac': 'PS00027', 'signature_id': 'HOMEOBOX_1', 'level_tag': '(0)'} -``` -- -You see that each item `r` represents a dictionary describing a specific match. - ---- - -# Working with Sequence Files - - -## Sequence Objects - -The `Seq` object is similar to a string object augmented with methods for nucleotide sequence operations including -* `find()`, `count()` -* `complement()`, `reverse_complement()` -* `transcribe()`, `back_transcribe()` -* `translate()` - -The following code examples are in the __seq.py__ script. - -```python -from Bio.Seq import Seq - -my_dna = Seq("ATGAGTACACTATAGA") -print (my_dna) -# find position of first subsequence -print (my_dna.find("TAC")) -print (my_dna.find("AC")) -print (my_dna.find("CTC")) - -# count -print ("A count:", my_dna.count("A")) -print ("AC count:", my_dna.count("AC")) -print ("AA count:", Seq("AAAA").count("AA")) # non-overlapping -``` - -
-
-Output:
- -```bash -ATGAGTACACTATAGA -5 -6 --1 -A count: 7 -AC count: 2 -AA count: 2 -``` -- -Note the return of `-1` if no sequence match was found. - -```python -# complement and reverse complement -compl = my_dna.complement() -rev_compl = my_dna.reverse_complement() -print ("original: \t", my_dna) -print ("complement:\t", compl) -print ("rev complement:\t", rev_compl) - -# transcription -my_rna = my_dna.transcribe() -print ("RNA:", my_rna) - -# translation -my_peptide = my_dna.translate() -print ("Peptide:", my_peptide) -``` - -
-
-Output:
- -```bash -original: ATGAGTACACTATAGA -complement: TACTCATGTGATATCT -rev complement: TCTATAGTGTACTCAT -RNA: AUGAGUACACUAUAGA -Peptide: MSTL* -``` -- -Like Strings, Seq objects are immutable; this means that the sequence is read-only and cannot be modified in place. However, you can convert a `Seq` object into a `MutableSeq` object that allows you to manipulate the sequence after object initialization. - -```python -my_dna[2] = 'A' # error, immutable Seq object -mutable_dna = my_dna.tomutable() -mutable_dna[2] = 'A' -print (my_dna) -print (mutable_dna) -``` - -
-
-Output:
- -```bash -ATGAGTACACTATAGA -ATAAGTACACTATAGA -``` -- -Note that the sequence is zero-indexed: the first nucleotide has index 0, the second has index 1, and so forth. So in this example we're changing the third nucleotide (index 2, G->A). - -**Exercise:** Create a `Seq` object with a DNA nucleotide sequence of your choice. Find the first putative start codon (ATG), replace each "C" with a "G", and transcribe and translate the original as well as the modified sequence. **Hint:** As an intermediary step, convert `Seq` object to a string and use a string method for replacement. - -
-
-Solution:
- -```python -from Bio.Seq import Seq - -seq = Seq("CTGACTGGATGACCATTGGGCAACTACCCATGACTAGTTAAGTAATTTTTAAAAA") -atg_pos = seq.find("ATG") - -cds = seq[atg_pos:] -peptide = cds.translate(to_stop=True) - -mod_seq = Seq(str(seq).replace("C","G")) -mod_cds = mod_seq[atg_pos:] -mod_peptide = mod_cds.translate(to_stop=True) - -print ("DNA (original):", seq) -print ("DNA (modified):", mod_seq) -print ("Peptide (original):", peptide) -print ("Peptide (modified):", mod_peptide) -``` -- -## Handling Sequence Records - -The [SeqRecord](https://biopython.org/wiki/SeqRecord) class provides the following fields: -* `.seq`: a sequence (as a [Seq](#seq) object) -* `.id`: the identifier, e.g. an accession number (String) -* `.name`: can be just the accession number or the locus name (String) -* `.description`: self-explanatory (String) -* `.annotations`: dictionary of additional often unstructured info (optional) -* `.letter_annotations`: often used for quality scores or secondary structure info -* `.features`: list of [SeqFeature]() objects; more structured than annotations, e.g. gene position in a genome, or domain position in a protein -* `.dbxref`: list of database cross-references - - -So it is used to wrap around a [Seq](#seq) object with richer information. We can manually create a `SeqRecord` object like this: -```python -from Bio.Seq import Seq -from Bio.SeqRecord import SeqRecord - -record = SeqRecord( - Seq("MKQHKAMIVALIVICITAVVAALVTRKDLCEVHIRTGQTEVAVF"), - id="YP_025292.1", - name="HokC", - description="toxic membrane protein, small") -print(record) -``` - -The above code example is in the __seqrecord.py__ script. - -
-
-Output:
- -```bash -ID: YP_025292.1 -Name: HokC -Description: toxic membrane protein, small -Number of features: 0 -Seq('MKQHKAMIVALIVICITAVVAALVTRKDLCEVHIRTGQTEVAVF') -``` -- -## Sequence File Operations - -The [Bio.SeqIO](https://biopython.org/wiki/SeqIO) class provides simple tools to read and write a variety of sequence file formats (including multiple sequence alignments). It operates exclusively on [SeqRecord](#seqrecord) objects. - -**Read Fasta File** -```python -from Bio import SeqIO - -file = open('HsPax6-protein.fasta') -fastarecords = SeqIO.parse(file, "fasta") -# create a list of SeqRecords -fastalist = [entry for entry in fastarecords] -# iterate over fasta entries -for entry in fastalist: - print (f"ID={entry.id}") - print (f"Name={entry.name}") - print (f"Description={entry.description}") - print (f"Seq length={len(entry.seq)}") - print (f"Features={entry.features}") # empty for Fasta format - print (f"Sequence={entry.seq}\n") -``` - -**Exercise:** Filter the list of records to only include sequences with less than 300 amino acids. - -
-
-Solution
- -```python -# filter list of records -sublist = [e for e in fastalist if len(e.seq) < 300] -print (f"Total number of sequences: {len(fastalist)}") -print (f"Number of sequences (<300 aa): {len(sublist)}") -``` -- -**Convert Genbank to Fasta File** - -```python -from Bio import SeqIO - -gb_file = 'HsPax6-208879460-nucleotide.gb' -with open(gb_file) as f: - gb_generator = SeqIO.parse(f, format='gb') - for entry in gb_generator: - with open(f'Hs-pax6-{entry.id}-nucleotide.fasta', 'w') as fileout: - SeqIO.write(entry, fileout, 'fasta') -``` - -The later versions of Biopython also include a `Bio.SeqIO.convert()` function. -``` -# convert GenBank to Fasta -count = Bio.SeqIO.convert("my_file.gb", "genbank", "my_file.fasta", "fasta") -``` -
- -## AlignIO: Reading Sequence Alignment Files - -The [Bio.AlignIO](https://biopython.org/wiki/AlignIO) class provides functions to handle paired or multiple sequence alignment files. It does not perform the alignment but provides tools to read/write alignment files and manipulate alignment objects. `Bio.AlignIO` uses the same set of functions for input and output as in `Bio.SeqIO`, and the same names are supported for the file formats. - -The key functions are: -* `Bio.AlignIO.read()`: For a file that contains one and only one alignment. The return type is a `Bio.Align.MultipleSeqAlignment` object. -* `Bio.AlignIO.parse()`: A more general function when the file may contain multiple separate alignments. The return type is a generator that can be converted into a list of `Bio.Align.MultipleSeqAlignment` objects. - -**Example:** - -Let's create a Fasta file with Pax6 orthologs from human, mouse, xenopus, pufferfish, zebrafish (2), and fruitfly. The following code example is in the __createPax6_fasta.py__ script. - -```python -from Bio import Entrez - -# get human, mouse, xenopus, pufferfish, zebrafish 1, zebrafish 2, Drosophila 1, Drosophila 2 -ids = ['1587062735','AAH36957.1','NP_001006763.1','XP_029701655.1','NP_571379.1','NP_571716.1','NP_524628','NP_524638'] -handle = Entrez.efetch(db="protein", rettype="fasta", retmode="text", id=ids) -result = handle.read() -handle.close() -fastaseq = result.replace("\n\n","\n") -with open('Pax6-multispec-protein.fasta', 'w') as f: - f.write(fastaseq) -``` - -This will create the `Pax6-multispec-protein.fasta` Fasta file with 8 sequences. The alignment was performed using [Clustal Omega](https://www.ebi.ac.uk/Tools/msa/clustalo/) and you can [download the Pax6-multispec-protein.aln](data/Pax6-multispec-protein.aln) alignment file and move it to your Python script folder that you use for this workshop. - - -**Alternatively, create the alignment yourself:** -1. Visit the [Clustal Omega website](https://www.ebi.ac.uk/Tools/msa/clustalo/) and upload the `Pax6-multispec-protein.fasta` file as input. -2. Under **Step 1**, click the **Choose File** button and upload the `Pax6-multispec-protein.fasta` file as input. -3. Under **Step 3**, click **Submit**. -4. When the alignment is done, click the **Alignments** tab, select the entire alignment output in the window and paste it into a text editor. **Do not use Microsoft Word for this but programs like `Text Edit`, `Notepad++`, `Emacs` or `vim` instead.** -5. Save the alignment in the text editor as `Pax6-multispec-protein.aln` in your Python script folder that you use for this workshop. - -The following code examples are in the __alignio-parse_clustal.py__ script. - -**Parse the alignment file** -``` -from Bio import AlignIO - -inputfile = open("Pax6-multispec-protein.aln", "r") -# assuming single alignment in file; use AlignIO.parse for multiple alignments -alignment = AlignIO.read(inputfile, "clustal") -inputfile.close() -print ("Alignment length:", alignment.get_alignment_length()) -print (alignment,"\n") -``` - -
-
-Output:
- -```bash -Alignment with 8 rows and 867 columns -MFTLQPTPTAIGTVVPPWSAGTLIERLPSLEDMAHKGHSGVNQL...PWV NP_524628.2 ----MMLTTEHIMHGHPH-----SSVGQSTLFGCSTAGHSGINQL...--- NP_524638.3 ----------------------------------MQNSHSGVNQL...--- NP_001006763.1 ---------------------------------------------...--- NP_001355831.1 ----------------------------------MQNSHSGVNQL...--- AAH36957.1 ---------------------------------MMQNSHSGVNQL...--- XP_029701655.1 -----MPQKEY-Y----N-----RATWESGVASMMQNSHSGVNQL...--- NP_571379.1 -----MPQKEY-H----N-----QPTWESGVASMMQNSHSGVNQL...--- NP_571716.1 -``` -- -**Update identifier** -```python -species = ['H.sapiens', 'M.musculus', 'X.tropicalis', 'T.rubripes', 'D.rerio', 'D.rerio', 'D.melanogaster', 'D.melanogaster'] -for idx,line in enumerate(alignment): - line.id = f"{species[idx]}:{line.id}" -print (alignment) -``` - -
-
-Output:
- -```bash -Alignment with 8 rows and 867 columns -MFTLQPTPTAIGTVVPPWSAGTLIERLPSLEDMAHKGHSGVNQL...PWV H.sapiens:NP_524628.2 ----MMLTTEHIMHGHPH-----SSVGQSTLFGCSTAGHSGINQL...--- M.musculus:NP_524638.3 ----------------------------------MQNSHSGVNQL...--- X.tropicalis:NP_001006763.1 ---------------------------------------------...--- T.rubripes:NP_001355831.1 ----------------------------------MQNSHSGVNQL...--- D.rerio:AAH36957.1 ---------------------------------MMQNSHSGVNQL...--- D.rerio:XP_029701655.1 -----MPQKEY-Y----N-----RATWESGVASMMQNSHSGVNQL...--- D.melanogaster:NP_571379.1 -----MPQKEY-H----N-----QPTWESGVASMMQNSHSGVNQL...--- D.melanogaster:NP_571716.1 -``` -- -**Slicing and joining** -```python -# slice: first axis defines line, second axis defines column index (zero-indexed) -# get lines 1-6, first 50 columns -subset = alignment[:6,:50] -print (subset) -``` - -
-
-Output:
- -```bash -Alignment with 6 rows and 50 columns -MFTLQPTPTAIGTVVPPWSAGTLIERLPSLEDMAHKGHSGVNQLGGVFVG H.sapiens:NP_524628.2 ----MMLTTEHIMHGHPH-----SSVGQSTLFGCSTAGHSGINQLGGVYVN M.musculus:NP_524638.3 ----------------------------------MQNSHSGVNQLGGVFVN X.tropicalis:NP_001006763.1 --------------------------------------------------- T.rubripes:NP_001355831.1 ----------------------------------MQNSHSGVNQLGGVFVN D.rerio:AAH36957.1 ---------------------------------MMQNSHSGVNQLGGVFVN D.rerio:XP_029701655.1 -``` -- -Let's join two alignment blocks: -```python -edited = alignment[:,:50] + alignment[:,500:] -print (edited) -``` - -
-
-Output:
- -```bash -Alignment with 8 rows and 417 columns -MFTLQPTPTAIGTVVPPWSAGTLIERLPSLEDMAHKGHSGVNQL...PWV H.sapiens:NP_524628.2 ----MMLTTEHIMHGHPH-----SSVGQSTLFGCSTAGHSGINQL...--- M.musculus:NP_524638.3 ----------------------------------MQNSHSGVNQL...--- X.tropicalis:NP_001006763.1 ---------------------------------------------...--- T.rubripes:NP_001355831.1 ----------------------------------MQNSHSGVNQL...--- D.rerio:AAH36957.1 ---------------------------------MMQNSHSGVNQL...--- D.rerio:XP_029701655.1 -----MPQKEY-Y----N-----RATWESGVASMMQNSHSGVNQL...--- D.melanogaster:NP_571379.1 -----MPQKEY-H----N-----QPTWESGVASMMQNSHSGVNQL...--- D.melanogaster:NP_571716.1 -``` -- -**Exporting to other alignment file formats** -```python -# save as Stockholm -with open("Pax6-multispec-protein.sth", "w") as outputfile: - AlignIO.write(alignment, outputfile, "stockholm") - -# get alignment as formatted string -print ("Formatted Alignment:") -print (format(alignment, "clustal")) -``` - -
-
-Output:
- -```bash -Formatted Alignment: -CLUSTAL X (1.81) multiple sequence alignment - -H.sapiens:NP_524628.2 MFTLQPTPTAIGTVVPPWSAGTLIERLPSLEDMAHKGHSGVNQLGGVFVG -M.musculus:NP_524638.3 ---MMLTTEHIMHGHPH-----SSVGQSTLFGCSTAGHSGINQLGGVYVN -X.tropicalis:NP_001006763.1 ---------------------------------MQNSHSGVNQLGGVFVN -T.rubripes:NP_001355831.1 -------------------------------------------------- -D.rerio:AAH36957.1 ---------------------------------MQNSHSGVNQLGGVFVN -D.rerio:XP_029701655.1 --------------------------------MMQNSHSGVNQLGGVFVN -D.melanogaster:NP_571379.1 ----MPQKEY-Y----N-----RATWESGVASMMQNSHSGVNQLGGVFVN -D.melanogaster:NP_571716.1 ----MPQKEY-H----N-----QPTWESGVASMMQNSHSGVNQLGGVFVN -``` -- -**Exercise:** -Find the first alignment block that shows no gaps across all 8 aligned sequences. -1. Print the block. -2. Save the block as a new clustal formatted text file. -3. From that block, extract the D. rerio (zebrafish) sequences and print the two sequences - -
-
-Solution:
- -```python -from Bio import AlignIO - -inputfile = open("Pax6-multispec-protein.aln", "r") -# assuming single alignment in file; use AlignIO.parse for multiple alignments -alignment = AlignIO.read(inputfile, "clustal") -inputfile.close() - -length = alignment.get_alignment_length() -# create boolean list to indicate if all lines in column are wiithout gap -pattern = [all([a.seq[i] != "-" for a in alignment]) for i in range(length)] -print ("No gap in column:") -print (pattern,"\n") -# create list of column indices without gaps -full_indices = [i for i in range(length) if pattern[i]] -print ("Full column indices:") -print (full_indices,"\n") - -# scan consecutive columns for completeness -firstblock_i = [] -j=0 -start=0 -for i in full_indices[start:]: - if j==0 or i==full_indices[start]+j: - firstblock_i.append(i) - else: - break - j+=1 - -# slice alignment to show first block without gaps -block1 = alignment[:,firstblock_i[0]:firstblock_i[-1]+1] -print (block1) - -# save -with open("Pax6-multispec-block1.aln", "w") as outputfile: - AlignIO.write(block1, outputfile, "clustal") - -# get zebrafish sequence lines -zebrafish_lines = block1[4:6,:] -print (zebrafish_lines) -``` -- -# Resources - -* [Biopython Tutorial and Cookbook](http://biopython.org/DIST/docs/tutorial/Tutorial.html) -* [UVA Research Computing](https://www.rc.virginia.edu) diff --git a/content/courses/containers-for-hpc/_index.md b/content/notes/containers-for-hpc/_index.md similarity index 87% rename from content/courses/containers-for-hpc/_index.md rename to content/notes/containers-for-hpc/_index.md index 61015558..c838cc65 100644 --- a/content/courses/containers-for-hpc/_index.md +++ b/content/notes/containers-for-hpc/_index.md @@ -5,14 +5,18 @@ authors: [rs] categories: ["Containers","HPC"] tags: [Containers, HPC] toc: true -type: book +type: docs date: "2025-05-15T00:00:00" weight: 1 +menu: + containers-for-hpc: + name: Software Containers for HPC + --- ## Overview -This short course is an introduction to using and building software containers, structured as follows: +This tutorial is an introduction to using and building software containers, structured as follows: - Introduction to Software Containers - Using Containers on HPC [Apptainer] diff --git a/content/courses/containers-for-hpc/building-apptainer.md b/content/notes/containers-for-hpc/building-apptainer.md similarity index 99% rename from content/courses/containers-for-hpc/building-apptainer.md rename to content/notes/containers-for-hpc/building-apptainer.md index 73c396c6..0670a480 100644 --- a/content/courses/containers-for-hpc/building-apptainer.md +++ b/content/notes/containers-for-hpc/building-apptainer.md @@ -2,8 +2,11 @@ date: "2025-06-04T00:00:00" title: "Building Containers on HPC [Apptainer]" toc: true -type: book +type: docs weight: 4 +menu: + containers-for-hpc: + parent: Software Containers for HPC --- diff --git a/content/courses/containers-for-hpc/building-docker.md b/content/notes/containers-for-hpc/building-docker.md similarity index 98% rename from content/courses/containers-for-hpc/building-docker.md rename to content/notes/containers-for-hpc/building-docker.md index 274944d4..ed3554d5 100644 --- a/content/courses/containers-for-hpc/building-docker.md +++ b/content/notes/containers-for-hpc/building-docker.md @@ -2,9 +2,13 @@ date : "2024-06-03T00:00:00" title: Appendix 1 - Building Containers [Docker] toc: true -type: book +type: docs weight: 10 +menu: + containers-for-hpc: + parent: Software Containers for HPC + --- _Docker, Dockerfile, Docker Hub_ @@ -347,7 +351,7 @@ Hints: - If you encounter a Docker statement that you have not used before, first check the official documentation for best practices. - A comprehensive list of dependencies may be lacking. Some developers may not specify any at all. You will have to rely on a combination of experience, error message, and web search. (Most likely all of the above.) - Especially for Python packages, versions may be too permissive or too restrictive such that, in either case, future installation of the application will fail. (I have encountered both.) Tweak the versions until it works. -- The next step is "multi-stage build" which is covered in the [Minimal Containers](/workshops/minimal-containers) workshop. There you will learn how to distinguish between buildtime versus runtime dependencies and separate them out. +- The next step is "multi-stage build" which is covered in the [Minimal Containers](../minimal) workshop. There you will learn how to distinguish between buildtime versus runtime dependencies and separate them out. ## Clean Up diff --git a/content/courses/containers-for-hpc/intro.md b/content/notes/containers-for-hpc/intro.md similarity index 92% rename from content/courses/containers-for-hpc/intro.md rename to content/notes/containers-for-hpc/intro.md index 090c902c..78348f66 100755 --- a/content/courses/containers-for-hpc/intro.md +++ b/content/notes/containers-for-hpc/intro.md @@ -1,10 +1,14 @@ --- title: Introduction to Software Containers toc: true -type: book +type: docs date: "2025-05-15T00:00:00" weight: 2 +menu: + containers-for-hpc: + parent: Software Containers for HPC + --- "A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another." (
-4. Challenge: Verify that the two containers have the same performance on Rivanna's GPU node. Follow the [tutorial example](https://lightgbm.readthedocs.io/en/latest/GPU-Tutorial.html#dataset-preparation). Run the same job without using GPU. How much faster is it with GPU? +4. Challenge: Verify that the two containers have the same performance on a GPU node. Follow the [tutorial example](https://lightgbm.readthedocs.io/en/latest/GPU-Tutorial.html#dataset-preparation). Run the same job without using GPU. How much faster is it with GPU? --- @@ -443,18 +447,6 @@ We submitted a [pull request](https://github.com/microsoft/LightGBM/pull/3408) t --- -## Example: TensorFlow distroless - -TensorFlow is a popular platform for machine learning. It is an open source project by Google. - -The TF 2.3 container that you used in the previous workshop is actually based on distroless, which is why you were not able to run `ls` inside the container. - -- [Dockerfile](https://github.com/uvarc/rivanna-docker/blob/master/tensorflow/2.3.0/Dockerfile.distroless) -- 18% image size reduction -- [PR](https://github.com/tensorflow/build/pull/13) approved and [merged](https://github.com/tensorflow/build) - ---- - ## Dynamic vs Static Linking The above procedure, while impressive, may be tedious for the average user. All the examples so far are based on [dynamic linking](https://en.wikipedia.org/wiki/Dynamic_linker), where the shared libraries of an executable are stored separately. If you are compiling code from source, you may choose to build a static binary (e.g. `-static` in GCC) so that all the necessary libraries are built into the binary. @@ -687,13 +679,11 @@ LibTorch is the C++ frontend of PyTorch. This exercise is based on the ["Writing 1. Challenge: The above image cannot make use of GPU. Build an image for GPU. Hints: - You do not need a physical GPU to build an image for GPU. - - Pick a `nvidia/cuda` base image. Read their overview page on Docker Hub to decide which flavor to use. + - Use `nvidia/cuda` as the base image. Read their overview page on Docker Hub to decide which flavor to use. - Choose a CUDA version to get the download link for LibTorch on the PyTorch webpage.
-1. Challenge: Can you build `dcgan` on Rivanna without using a container? Why (not)? - 1. Challenge: Can you build a static binary of `dcgan`? Why (not)? --- diff --git a/content/courses/containers-for-hpc/using.md b/content/notes/containers-for-hpc/using.md similarity index 98% rename from content/courses/containers-for-hpc/using.md rename to content/notes/containers-for-hpc/using.md index 9cdd2dcf..fcc6e3bb 100755 --- a/content/courses/containers-for-hpc/using.md +++ b/content/notes/containers-for-hpc/using.md @@ -1,10 +1,14 @@ --- title: Using Containers on HPC [Apptainer] toc: true -type: book +type: docs date: "2025-05-13T00:00:00" weight: 3 +menu: + containers-for-hpc: + parent: Software Containers for HPC + --- Log on to our HPC cluster @@ -14,7 +18,7 @@ Log on to our HPC cluster - Make sure you have a few GBs of free space - Run `allocations` - Check if you have `hpc_training` -- Request an interative job +- Request an interactive job {{< code-snippet >}}ijob -A hpc_training -p interactive -c 1 -t 2:0:0{{< /code-snippet >}} - Run `module load apptainer` diff --git a/content/notes/containers/_index.md b/content/notes/containers/_index.md deleted file mode 100755 index 61b61002..00000000 --- a/content/notes/containers/_index.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -# Course title, summary, and position. -title: "Introduction to Building and Deploying Containers" -authors: [cag] - -# Page metadata. -date: "2023-05-01T00:00:00" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. - -# Add menu entry to sidebar. -menu: - containers: - name: Containers - weight: 5 ---- - -You've developed an app and written the paper. Now it's time to deploy the app so the world (and the reviewers) can see how awesome it is. This is Part 1 of a two-part workshop that will cover how to deploy web apps for publication. In Part 1 we will go over how to containerize our apps with Docker and maintain them with GitHub. - -
diff --git a/content/notes/containers/basics-dockerfile.md b/content/notes/containers/basics-dockerfile.md
deleted file mode 100755
index 157c6b04..00000000
--- a/content/notes/containers/basics-dockerfile.md
+++ /dev/null
@@ -1,17 +0,0 @@
----
-title: Building a Docker Image
-date: "2023-05-01T00:00:00"
-draft: false # Is this a draft? true/false
-toc: false # Show table of contents? true/false
-type: docs # Do not modify.
-weight: 130
-menu:
- containers:
- parent: Container Basics
----
-
-In order to serve our own webapp in a container, we will need to build an image for it. There are many "blank" images like the nginx container we pulled earlier that we can use as a starting point. This is called a base image.
-
-To create our image, we will first write a Dockerfile. A Dockerfile is a text file containing the commands we will use to build the environment we need for our app. These are very similar to commands we would use to install libraries and packages that we need locally.
-
-Once complete, we will use the `docker build` and `docker push` commands to build our image and upload it to DockerHub.
diff --git a/content/notes/containers/basics-example.md b/content/notes/containers/basics-example.md
deleted file mode 100755
index 2483ed91..00000000
--- a/content/notes/containers/basics-example.md
+++ /dev/null
@@ -1,78 +0,0 @@
----
-title: A Quick Example
-date: "2023-05-01T00:00:00"
-draft: false # Is this a draft? true/false
-toc: false # Show table of contents? true/false
-type: docs # Do not modify.
-weight: 110
-menu:
- containers:
- parent: Container Basics
----
-
-Cowsay is a Linux game that prints ASCII art of a cow and a speech bubble containing a string input by the user. The Whalesay image modifies that game by replacing the cow with a whale.
-
-Let's try pulling the whalesay image from DockerHub and running a container on our machine.
-
-## 1. Pull the Image
-
-```
-> docker pull docker/whalesay
-
-Using default tag: latest
-latest: Pulling from docker/whalesay
-Image docker.io/docker/whalesay:latest uses outdated schema1 manifest format. Please upgrade to a schema2 image for better future compatibility. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/
-e190868d63f8: Pull complete
-909cd34c6fd7: Pull complete
-0b9bfabab7c1: Pull complete
-a3ed95caeb02: Pull complete
-00bf65475aba: Pull complete
-c57b6bcc83e3: Pull complete
-8978f6879e2f: Pull complete
-8eed3712d2cf: Pull complete
-Digest: sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b
-Status: Downloaded newer image for docker/whalesay:latest
-docker.io/docker/whalesay:latest
-```
-
-You'll notice that there are several lines with a hash/SHA ID followed by "Pull complete". These correspond to the different layers of the image, or the different components that will make up the compute environment when we run the container. We will talk more about layers when we cover building our own images.
-
-## 2. List all pulled images
-
-By listing all images, we are confirming that we successfully pulled the whalesay image.
-```
-> docker images
-
-REPOSITORY TAG IMAGE ID CREATED SIZE
-docker/whalesay latest 6b362a9f73eb 8 years ago 247MB
-```
-
-## 3. Run the Container
-```
-> docker run docker/whalesay
-```
-
-Just running whalesay looks like it didn't do anything since we didn't provide a command. What happened behind-the-scenes is that Docker spun up an instance of the whalesay image, ran an empty command, and then exited once that was complete.
-
-Let's try running a container again, this time adding the cowsay command at the end.
-
-```
-> docker run docker/whalesay cowsay "hi there"
-
- __________
-< hi there >
- ----------
- \
- \
- \
- ## .
- ## ## ## ==
- ## ## ## ## ===
- /""""""""""""""""___/ ===
- ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~
- \______ o __/
- \ \ __/
- \____\______/
-
-```
-
diff --git a/content/notes/containers/basics-webapps.md b/content/notes/containers/basics-webapps.md
deleted file mode 100755
index c345bcbe..00000000
--- a/content/notes/containers/basics-webapps.md
+++ /dev/null
@@ -1,31 +0,0 @@
----
-title: Running WebApps in a Container
-date: "2023-05-01T00:00:00"
-draft: false # Is this a draft? true/false
-toc: false # Show table of contents? true/false
-type: docs # Do not modify.
-weight: 120
-menu:
- containers:
- parent: Container Basics
----
-
-Just like with the whalesay/cowsay program, we can run web apps in a container. Let's try with this static website example.
-
-## 1. Pull the nginx image
-
-```
-> docker pull nginx
-```
-
-## 2. Run the nginx container
-
-We can use -p to specify a custom port to connect to our container. In this case we are using port 8080.
-
-```
-> docker run -p 8080:80 nginx
-```
-
-We can see that the container is running when we go to [http://localhost:8080](http://localhost:8080) in the browser.
-
-When you're done, you can use Ctrl + C to stop the container.
\ No newline at end of file
diff --git a/content/notes/containers/basics.md b/content/notes/containers/basics.md
deleted file mode 100755
index 591b4746..00000000
--- a/content/notes/containers/basics.md
+++ /dev/null
@@ -1,30 +0,0 @@
----
-title: Container Basics
-date: "2023-05-01T00:00:00"
-draft: false # Is this a draft? true/false
-toc: false # Show table of contents? true/false
-type: docs # Do not modify.
-weight: 100
-
-menu:
- containers:
- name: Container Basics
----
-
-To run and build containers, you will need Docker Desktop installed on your local machine. Instructions and installation files can be found here: https://docs.docker.com/engine/install/.
-
-## Terminology
-
-**Image**: The layers of libraries, code, and configuration that make up the environment that you need to run your application.
-
-**Container**: A running instance of an image. You can have many containers of a single image run simultaneously.
-
-**DockerHub**: An online registry for Docker images (similar to GitHub)
-
-## Commonly Used Docker Commands
-
-- **docker pull**: Fetches an image from a container registry to your local machine
-
-- **docker images**: List all locally available images (kind of like ls)
-
-- **docker run**: Run a container based on a particular image
diff --git a/content/notes/containers/github-actions.md b/content/notes/containers/github-actions.md
deleted file mode 100755
index faf26a73..00000000
--- a/content/notes/containers/github-actions.md
+++ /dev/null
@@ -1,80 +0,0 @@
----
-title: GitHub Actions
-date: "2023-05-01T00:00:00Z"
-draft: false # Is this a draft? true/false
-toc: false # Show table of contents? true/false
-type: docs # Do not modify.
-weight: 340
-menu:
- containers:
- parent: Version Control
----
-
-GitHub Actions allow us to incorporate **Continuous Integration/Continuous Deployment** in our repository. We can automatically rebuild and redeploy our app whenever any changes are committed to the repo.
-
-To add a workflow to your repo:
-
-1. Create a `.github/workflows` directory.
-
-2. Add the following .YAML file to the new directory, replacing the **IMAGE_NAME** and **SVC_NAME** with your image's name and the name that you want for your app on the Kubernetes cluster, respectively.
-
-```
-name: Container Build CICD
-
-on:
- push:
- branches:
- - 'main'
-
-env:
- # The preferred container registry
- REGISTRY: ghcr.io
- # The base org/image-name for the container
- IMAGE_NAME: uvarc/chickweight
- # A service name if mapping this to a k8s deployment
- SVC_NAME: chickWeight
-
-jobs:
- build:
- runs-on: ubuntu-latest
- steps:
- - uses: actions/checkout@v2
- - name: Set ENV
- run: echo "IMAGE_TAG=${GITHUB_REF#refs/*/}" >> $GITHUB_ENV
- -
- name: Set up QEMU
- uses: docker/setup-qemu-action@v1
- -
- name: Set up Docker Buildx
- uses: docker/setup-buildx-action@v1
- -
- # GHCR require a GitHub username and a Personal Access Token with the right permissions.
- # These can be stored as repository secrets in the repo settings.
- name: Login to GHCR
- uses: docker/login-action@v1
- with:
- registry: ghcr.io
- username: ${{ secrets.GHCR_USERNAME }}
- password: ${{ secrets.GHCR_PAT }}
- -
- name: Build and push
- id: docker_build
- uses: docker/build-push-action@v2
- with:
- push: ${{ github.event_name != 'pull_request' }}
- tags: ghcr.io/${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
- labels: ${{ steps.meta.outputs.labels }}
- -
- name: Image digest
- run: echo ${{ steps.docker_build.outputs.digest }}
-
- # Now update another repo so that ArgoCD can deploy the new version.
- # Note that the dispatch call simply curls a POST payload to another repository with JSON that you define.
- - name: Remote Dispatch
- run: |
- curl -X POST https://api.github.com/repos/uvarc/uvarc-services/dispatches \
- -H 'Accept: application/vnd.github.everest-preview+json' \
- -H "Authorization: token ${{ secrets.GHCR_PAT }}" \
- --data '{"event_type": "${{ env.IMAGE_NAME }} update to ${{ env.IMAGE_TAG }}", "client_payload": { "service": "${{ env.SVC_NAME }}", "version": "${{ env.IMAGE_TAG }}" }}'
-
-```
\ No newline at end of file
diff --git a/content/notes/containers/github-commands.md b/content/notes/containers/github-commands.md
deleted file mode 100755
index c76802ae..00000000
--- a/content/notes/containers/github-commands.md
+++ /dev/null
@@ -1,23 +0,0 @@
----
-title: Common Git Commands
-date: "2023-05-01T00:00:00Z"
-draft: false # Is this a draft? true/false
-toc: false # Show table of contents? true/false
-type: docs # Do not modify.
-weight: 320
-menu:
- containers:
- parent: Version Control
----
-
-As we make changes to our Shiny app or Dockerfile, we will need to push the updates to our GitHub repository. Typically this is done with a `git add` + `git commit` + `git push`.
-
-1. `git add - -If we are working with collaborators, another useful command is `git pull`. This allows us to update our local repository with any updates our friends have pushed. \ No newline at end of file diff --git a/content/notes/containers/github-newrepo.md b/content/notes/containers/github-newrepo.md deleted file mode 100755 index b7672c65..00000000 --- a/content/notes/containers/github-newrepo.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: Creating a New GitHub Repository -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 310 -menu: - containers: - parent: Version Control ---- - -There are several ways to create a new GitHub repository. Here is one way I like to do it: - -1. Create a new repository with the "New repository" button. - -2. Fill out the "Repository name". - -3. Select "Add a README file". - -4. Click "Create repository". - -5. In your local terminal, clone the new repository with the following command: - - ``` - git clone https://github.com/
-
\ No newline at end of file
diff --git a/content/notes/containers/overview-services.md b/content/notes/containers/overview-services.md
deleted file mode 100755
index 28a6e202..00000000
--- a/content/notes/containers/overview-services.md
+++ /dev/null
@@ -1,19 +0,0 @@
----
-title: Container Services
-date: "2023-05-01T00:00:00Z"
-draft: false # Is this a draft? true/false
-toc: false # Show table of contents? true/false
-type: docs # Do not modify.
-weight: 40
-menu:
- containers:
- parent: Containers
----
-
-Container-based architecture, also known as "microservices," is an approach to designing and running applications as a distributed set of components or layers. Such applications are typically run within containers, made popular in the last few years by Docker.
-
-Containers are portable, efficient, reusable, and contain code and any dependencies in a single package. Containerized services typically run a single process, rather than an entire stack within the same environment. This allows developers to replace, scale, or troubleshoot portions of their entire application at a time.
-
-
-
-Research Computing runs microservices in a clustered orchestration environment that automates the deployment and management of many containers easy and scalable. This cluster has >1000 cores and ~1TB of memory allocated to running containerized services. It also has over 300TB of cluster storage and can attach to project and value storage.
\ No newline at end of file
diff --git a/content/notes/containers/overview-vms.md b/content/notes/containers/overview-vms.md
deleted file mode 100755
index 11c06332..00000000
--- a/content/notes/containers/overview-vms.md
+++ /dev/null
@@ -1,17 +0,0 @@
----
-title: Containers vs VMs
-date: "2023-05-01T00:00:00Z"
-draft: false # Is this a draft? true/false
-toc: false # Show table of contents? true/false
-type: docs # Do not modify.
-weight: 30
-menu:
- containers:
- parent: Containers
----
-
-You may be familiar with the virtual machines (VMs), which accomplish the same goal as containers: to create a reproducible and shareable compute environment. The big difference between VMs and containers is that VMs provide their own guest OS, whereas containers don't require an OS and run on the Docker Engine and share the host's OS kernel.
-
-The size of Docker images is usually on the order of tens of MBs, while VMs can be several tens of GBs large.
-
-
\ No newline at end of file
diff --git a/content/notes/containers/overview.md b/content/notes/containers/overview.md
deleted file mode 100755
index 9f5920b3..00000000
--- a/content/notes/containers/overview.md
+++ /dev/null
@@ -1,15 +0,0 @@
----
-title: What Are Containers?
-date: "2023-05-01T00:00:00Z"
-draft: false # Is this a draft? true/false
-toc: false # Show table of contents? true/false
-type: docs # Do not modify.
-weight: 10
-menu:
- containers:
- parent: Containers
----
-
-{{}}
-
-Shipping containers are used to transport cargo around the world. In computing, containers allow you to transport and share entire filesystems, processes, scripts, and more!
\ No newline at end of file
diff --git a/content/notes/containers/shinyapp-debugging.md b/content/notes/containers/shinyapp-debugging.md
deleted file mode 100755
index bd82c406..00000000
--- a/content/notes/containers/shinyapp-debugging.md
+++ /dev/null
@@ -1,53 +0,0 @@
----
-title: Running and Debugging
-date: "2023-05-01T00:00:00Z"
-draft: false # Is this a draft? true/false
-toc: false # Show table of contents? true/false
-type: docs # Do not modify.
-weight: 230
-menu:
- containers:
- parent: Serving a ShinyApp
----
-
-## Running the Container
-
-Before we do anything else with our new Docker image, we want to make sure it runs correctly locally.
-
-To run the container, we will use the following command.
-
-```
-docker run -p 8080:80 /:
-```
-
-With `-p 8080:80`, we are mapping port 80 of the container to port 8080 of our local machine. When we want to preview our app, we will go to [localhost:8080](localhost:8080) in our web browser. You don't have to use 8080, you can use another number if you want.
-
-{{
}}
-
-
- -## Debugging - -What if there's something wrong with the image for our Shiny app? How will we know? If there's an error, we'll see this message: - -{{
- -## Debugging - -What if there's something wrong with the image for our Shiny app? How will we know? If there's an error, we'll see this message: - -{{
}}
-
-How do we go about debugging this? We'll need to take a peek into our app. We will use `docker run` like before, but this time we will run the container interactively.
-
-```
-docker run -it -p 8888:80 /: /bin/bash
-```
-
-- `-it` allows us to run the container interactively
-
-- With `/bin/bash`, we are entering the container in a bash shell.
-
-- We use port 8888 instead of 8080, because we need to restart Docker Desktop in order to use 8080 again.
-
-
- -When we try going to [localhost:8888](localhost:8888), the app isn't running yet. We had a command at the end of our Dockerfile to start up Shiny server and run our app, but we just replaced that with `/bin/bash`. So within our new container, let's run `/usr/bin/shiny-server.sh`. - -Now we see that error message again when we reload the page in the browser. Hit **Ctrl + C** to stop running the app. - -Navigate to `/var/log/shiny-server`. Here you will see log files--use the `cat` command to take a look at them. In this example, we are missing the `fresh` package from our Docker image. We need to add that to our Dockerfile and rebuild the image. diff --git a/content/notes/containers/shinyapp-dockerbuild.md b/content/notes/containers/shinyapp-dockerbuild.md deleted file mode 100755 index cc2127fc..00000000 --- a/content/notes/containers/shinyapp-dockerbuild.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -title: Building the App Image -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 220 -menu: - containers: - parent: Serving a ShinyApp ---- - -Now that we've written the Dockerfile, it's time to build our image! To do that, we will use the `docker build` command. - -In the same directory that your Dockerfile is in, you can run the following: - -``` -docker build -t/: .
-```
-
-- `-t` lets us specify the tag name for this image.
-
-- `dockerhub-username` is pretty self-explanatory.
-
-- `image-name` is the name for our image. It will also be the name of the image repository in DockerHub.
-
-- `tag` can be a version number or other identifying label to help you differentiate this image from previous iterations. If you leave this blank, the tag name will default to `latest`.
-
-
-I will be calling this image `cagancayco/chickweight:latest`. Simply replace `cagancayco` with your own username.
\ No newline at end of file
diff --git a/content/notes/containers/shinyapp-dockerfile.md b/content/notes/containers/shinyapp-dockerfile.md
deleted file mode 100755
index 173e4b97..00000000
--- a/content/notes/containers/shinyapp-dockerfile.md
+++ /dev/null
@@ -1,114 +0,0 @@
----
-title: Writing the Dockerfile
-date: "2023-05-01T00:00:00Z"
-draft: false # Is this a draft? true/false
-toc: false # Show table of contents? true/false
-type: docs # Do not modify.
-weight: 210
-menu:
- containers:
- parent: Serving a ShinyApp
----
-
-Below is the Dockerfile that we will use to build the container for our Chick Weight app. Each line is a command for building our environment and corresponds to a different layer of our image. We will cover each section below.
-
-
-## 1. Choose a base image
-```
-# Install R version 4.1.2
-FROM r-base:4.1.2
-```
-
-In this section we are specifying that we are starting with the r-base Docker image. The r-base container uses Ubuntu and already has R installed, so we don't need to worry about installing that ourselves. There are many other base containers out there that you can use depending on what kind of app you're developing.
-
-## 2. Install Ubuntu packages and libraries
-
-The following packages and libraries will allow us to install Shiny server and various R packages that we need for our app. This list will cover most of your bases for most Shiny apps. If you find you need additional libraries, you can just add them to this list.
-
-How do you know if you're missing a library? You'll get an error message, and we will cover how to debug in a later section.
-```dockerfile
-# Install Ubuntu packages
-RUN apt-get update && apt-get install -y \
- sudo \
- gdebi-core \
- pandoc \
- libcurl4-gnutls-dev \
- libcairo2-dev/unstable \
- libxt-dev \
- libssl-dev \
- libxml2-dev \
- libnlopt-dev \
- libudunits2-dev \
- libgeos-dev \
- libfreetype6-dev \
- libpng-dev \
- libtiff5-dev \
- libjpeg-dev \
- libgdal-dev \
- git
-```
-
-## 3. Install Shiny server
-
-This just installs Shiny server in your image. If you're not developing a Shiny app, no need to include it. If you *are* developing a Shiny app, no need to change it!
-
-```dockerfile
-# Install Shiny server
-RUN wget --no-verbose https://s3.amazonaws.com/rstudio-shiny-server-os-build/ubuntu-12.04/x86_64/VERSION -O "version.txt" && \
- VERSION=$(cat version.txt) && \
- wget "https://s3.amazonaws.com/rstudio-shiny-server-os-build/ubuntu-12.04/x86_64/shiny-server-$VERSION-amd64.deb" -O ss-latest.deb && \
- gdebi -n ss-latest.deb && \
- rm -f version.txt ss-latest.deb
-```
-
-## 4. Install R Packages
-
-Here we are installing all the packages that we need for our Shiny app. Again, if you're not developing a Shiny app, you can skip this part.
-
-```dockerfile
-##### Install R packages that are required ######
-## CRAN packages
-RUN R -e "install.packages(c('shiny','shinydashboard','dplyr','ggplot2','fresh'))"
-```
-
-## 5. Copy configuration files to the Docker image
-
-These are just some files that make our Shiny app run. These will be in the directory that your app is in. These will be the same for all Shiny apps.
-```dockerfile
-# Copy configuration files into the Docker image
-COPY shiny-server.conf /etc/shiny-server/shiny-server.conf
-COPY shiny-server.sh /usr/bin/shiny-server.sh
-RUN rm -rf /srv/shiny-server/*
-```
-
-## 6. Copy your code to your app
-
-Ideally your code will be maintained within a GitHub repository (we will cover how to do this in a later section). Here we are cloning the GitHub repo and copying the contents to the shiny-server folder.
-```dockerfile
-# Get the app code
-RUN git clone https://github.com/uvarc/chickweight.git
-COPY chickweight/* /srv/shiny-server/
-RUN rm -rf chickweight
-```
-
-## 7. Some R and Shiny Stuff
-
-This is just some stuff for setting R paths, etc.
-```dockerfile
-# Make the ShinyApp available at port 80
-ENV R_HOME=/usr/lib/R
-ENV PATH=/usr/lib/R:/usr/lib/R/bin:$PATH
-EXPOSE 80
-WORKDIR /srv/shiny-server
-RUN chown shiny.shiny /usr/bin/shiny-server.sh && chmod 755 /usr/bin/shiny-server.sh
-```
-
-## 8. Run the Shiny app!
-
-The CMD just tells the container what to run once it starts. This line starts up the Shiny server and app.
-```dockerfile
-# Run the server setup script
-CMD ["/usr/bin/shiny-server.sh"]
-```
-
-Once we're done writing the Dockerfile, we save it as "Dockerfile" (no file extension).
\ No newline at end of file
diff --git a/content/notes/containers/shinyapp-dockerpush.md b/content/notes/containers/shinyapp-dockerpush.md
deleted file mode 100755
index 4c1637b2..00000000
--- a/content/notes/containers/shinyapp-dockerpush.md
+++ /dev/null
@@ -1,55 +0,0 @@
----
-title: Pushing our Image to a Container Registry
-date: "2023-05-01T00:00:00Z"
-draft: false # Is this a draft? true/false
-toc: false # Show table of contents? true/false
-type: docs # Do not modify.
-weight: 240
-menu:
- containers:
- parent: Serving a ShinyApp
----
-
-Once our container runs successfully, we can push it to a container registry so that others can pull the container. "Others" includes our Kubernetes cluster at UVA.
-
-We have a few options for container registries. The two main options are DockerHub and GitHub Container Registry (ghcr.io).
-
-For this workshop, we're going to use GHCR. The main benefit of using GHCR over DockerHub is that it is less restrictive with how many times your image can be pulled in an hour.
-
-## Re-tagging our image
-
-To use GHCR, we need to prepend our image name with `ghcr.io/`. That means we will need to re-tag our image. We can do that with the `docker tag` command.
-
-```
-docker tag cagancayco/chickweight:latest ghcr.io/uvarc/chickweight
-```
-
-## Allowing Docker to push to GHCR
-
-If this is your first time using GHCR, it is likely that you need to authorize your local Docker installation to push to GitHub.
-
-1. Go to GitHub -> Settings -> Developer Settings -> Personal Access Tokens, or click [here](https://github.com/settings/tokens).
-
-2. Generate new token with read/write permissions for packages.
-
-3. Copy the token to the clipboard.
-
-4. In your terminal, run
-
- ```
- docker login ghcr.io -u -p
- ```
-
-Now we can use `docker push` to push our container to GitHub!
-
-```
-docker push ghcr.io/uvarc/chickweight
-```
-
-## Set the package to public
-
-By default, GHCR packages (or images) are set to private. We need them to be public in order for the Kubernetes cluster to see it. To change the visibility, do the following:
-
-1. Go to Packages -> chickweight -> Package Settings
-
-2. Click "change visibility" and follow the on-screen instructions.
\ No newline at end of file
diff --git a/content/notes/containers/shinyapp.md b/content/notes/containers/shinyapp.md
deleted file mode 100755
index 6e2c2050..00000000
--- a/content/notes/containers/shinyapp.md
+++ /dev/null
@@ -1,24 +0,0 @@
----
-title: Serving a ShinyApp
-date: "2023-05-01T00:00:00Z"
-draft: false # Is this a draft? true/false
-toc: false # Show table of contents? true/false
-type: docs # Do not modify.
-weight: 200
-menu:
- containers:
- name: Serving a ShinyApp
----
-
-In this section of the workshop, we will take a completed ShinyApp and create a Docker image for it by writing a Dockerfile. We will then run the container to make sure that the ShinyApp is working as expected. Once we know the container is working, we will create a GitHub repository for our Shiny code and Dockerfile.
-
-To complete this section on your own, you will need:
-
-1. Docker Desktop installed on your machine
- The installers and installation instructions can be found here: https://docs.docker.com/engine/install/
-
-2. A GitHub account
- If you don't have one already you can make one here: https://github.com/
-
-3. A copy of the Shiny code
- You can clone a copy of the code to your local machine using `git clone https://github.com/uvarc/chickweight`
\ No newline at end of file
diff --git a/content/notes/databases-intro/graphdb-property.png b/content/notes/databases-intro/graphdb-property.png
deleted file mode 100644
index 7d5b6f6e..00000000
Binary files a/content/notes/databases-intro/graphdb-property.png and /dev/null differ
diff --git a/content/notes/databases-intro/index.md b/content/notes/databases-intro/index.md
deleted file mode 100644
index 367bd506..00000000
--- a/content/notes/databases-intro/index.md
+++ /dev/null
@@ -1,257 +0,0 @@
----
-
-title: "Introduction to Databases"
-type: article
-date: 2023-01-20T00:00:00-05:00
-draft: false
-
----
-
-
-In the case of document NoSQL databases, the “value” portion of the entry can get much larger. - -Here is an example of an entry in JSON. Note that the entire entry (or “document”) breaks down into a hierarchy of data: fields and their values, and dictionaries of multiple values, - -``` -{ - "success": { - "total": 1 - }, - "contents": { - "quotes": [ - { - "quote": "Remove the temptation to settle for anything short of what you deserve.", - "length": "71", - "author": "Lorii Myers", - "tags": [ - "expectation", - "inspire", - "perfection" - ], - "category": "inspire", - "date": "2017-09-08", - "permalink": "https://theysaidso.com/quote/ZWrV624xU_q6_KYYlrQpYgeF/lorii-myers-remove-the-temptation-to-settle-for-anything-short-of-what-you-deser", - "title": "Inspiring Quote of the day", - "background": "https://theysaidso.com/img/bgs/man_on_the_mountain.jpg", - "id": "ZWrV624xU_q6_KYYlrQpYgeF" - } - ], - "copyright": "2017-19 theysaidso.com" - } -} -``` - -Also consider that subsequent entries into this table may or may not contain a background image, or the same number of tags, or the precise data structure of this -entry. NoSQL evolved out of the need to quickly collect varied data at very high rates and so it does not suffer from impedance mismatch. Rather, it suffers from -its difficulty to aggregate or join. - -## 2. Node-Arc / Graph Databases - -Graph, or Node-arc databases are entirely different, in that they try to store and represent connectivity between nodes in a constellation, and their relationships. So a “query” of a graph database might inform you about the networks of other nodes related to the node you are interested in, and the types and strengths of those relationships, among other uses. Some examples of Graph DBs are: - -* Neo4j -* TinkerPop -* Infinite - - - - ---- - -# Using Databases in Your Research - -We are frequently asked by researchers how to incorporate databases into their work. Here are four suggestions: - -
- -**Note:** Research Computing may be able to provide support for your database needs. Please schedule a consultation request on our website by filling out [this form](https://www.rc.virginia.edu/form/support-request/?category=Consultation). - - -# Other Resources - -Here is a great overview of databases and their histories: - -{{< youtube qI_g07C_Q5I >}} - -Martin Fowler - NoSQL - YouTube diff --git a/content/notes/databases-intro/reldb.png b/content/notes/databases-intro/reldb.png deleted file mode 100644 index 56d5e2da..00000000 Binary files a/content/notes/databases-intro/reldb.png and /dev/null differ diff --git a/content/notes/rivanna-alphafold/23-alphafold-db.md b/content/notes/hpc-alphafold/23-alphafold-db.md similarity index 57% rename from content/notes/rivanna-alphafold/23-alphafold-db.md rename to content/notes/hpc-alphafold/23-alphafold-db.md index e05f36d5..1aaf13fa 100644 --- a/content/notes/rivanna-alphafold/23-alphafold-db.md +++ b/content/notes/hpc-alphafold/23-alphafold-db.md @@ -4,13 +4,13 @@ date: 2025-05-20T00:23:54Z type: docs weight: 2300 menu: - rivanna-alphafold: + hpc-alphafold: name: Protein Structure Database --- -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_57.png width=75% height=75% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_57.png width=75% height=75% >}} -The [AlphaFold Protein Structure Database](www.alphafold.ebi.ac.uk) has 200 million structure predictions. Even so, running the monomer version is still necessary in special use-cases such as engineered proteins, chimeric proteins, novel proteins translated from newly recognized Open Reading Frames (ORFs), and more. For protein complexes, it is best to use the multimer version. +The [AlphaFold Protein Structure Database](https://www.alphafold.ebi.ac.uk) has 200 million structure predictions. Even so, running the monomer version is still necessary in special use-cases such as engineered proteins, chimeric proteins, novel proteins translated from newly recognized Open Reading Frames (ORFs), and more. For protein complexes, it is best to use the multimer version. DeepMind and EMBL’s European Bioinformatics Institute (EMBL-EBI) have partnered to create AlphaFold DB to make these predictions freely available to the scientific community. The latest database release contains over 200 million entries, providing broad coverage of UniProt (the standard repository of protein sequences and annotations). AlphaFold DB provides individual downloads for the human proteome, the proteomes of 47 key research and global health organisms, and the manually curated Swiss-Prot subset of UniProt. diff --git a/content/notes/rivanna-alphafold/24-limitations-alphafold-db.md b/content/notes/hpc-alphafold/24-limitations-alphafold-db.md similarity index 99% rename from content/notes/rivanna-alphafold/24-limitations-alphafold-db.md rename to content/notes/hpc-alphafold/24-limitations-alphafold-db.md index 98d3b8aa..ba9ad1fe 100644 --- a/content/notes/rivanna-alphafold/24-limitations-alphafold-db.md +++ b/content/notes/hpc-alphafold/24-limitations-alphafold-db.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 2350 menu: - rivanna-alphafold: + hpc-alphafold: parent: Protein Structure Database --- The AlphaFold DB uses a monomeric model similar to the CASP14 version. As a result, many of the same limitations are expected: diff --git a/content/notes/rivanna-alphafold/25-alphafold-alternatives.md b/content/notes/hpc-alphafold/25-alphafold-alternatives.md similarity index 96% rename from content/notes/rivanna-alphafold/25-alphafold-alternatives.md rename to content/notes/hpc-alphafold/25-alphafold-alternatives.md index a7231d96..2fe8defc 100644 --- a/content/notes/rivanna-alphafold/25-alphafold-alternatives.md +++ b/content/notes/hpc-alphafold/25-alphafold-alternatives.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 2400 menu: - rivanna-alphafold: + hpc-alphafold: --- diff --git a/content/notes/rivanna-alphafold/_index.md b/content/notes/hpc-alphafold/_index.md similarity index 80% rename from content/notes/rivanna-alphafold/_index.md rename to content/notes/hpc-alphafold/_index.md index f1bbafd7..39b0531a 100644 --- a/content/notes/rivanna-alphafold/_index.md +++ b/content/notes/hpc-alphafold/_index.md @@ -1,12 +1,12 @@ --- -title: AlphaFold on Rivanna +title: AlphaFold on HPC date: 2025-05-20T00:23:54Z authors: [mab] type: docs weight: 1 menu: - rivanna-alphafold: + hpc-alphafold: --- This workshop will cover: diff --git a/content/notes/rivanna-alphafold/alphafold/4-alphafold-intro.md b/content/notes/hpc-alphafold/alphafold/4-alphafold-intro.md similarity index 98% rename from content/notes/rivanna-alphafold/alphafold/4-alphafold-intro.md rename to content/notes/hpc-alphafold/alphafold/4-alphafold-intro.md index 44e45027..f78f4340 100644 --- a/content/notes/rivanna-alphafold/alphafold/4-alphafold-intro.md +++ b/content/notes/hpc-alphafold/alphafold/4-alphafold-intro.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 400 menu: - rivanna-alphafold: + hpc-alphafold: name: AlphaFold --- diff --git a/content/notes/rivanna-alphafold/alphafold/5-prediction-pipeline.md b/content/notes/hpc-alphafold/alphafold/5-prediction-pipeline.md similarity index 84% rename from content/notes/rivanna-alphafold/alphafold/5-prediction-pipeline.md rename to content/notes/hpc-alphafold/alphafold/5-prediction-pipeline.md index e7835787..dfeb2122 100644 --- a/content/notes/rivanna-alphafold/alphafold/5-prediction-pipeline.md +++ b/content/notes/hpc-alphafold/alphafold/5-prediction-pipeline.md @@ -4,11 +4,11 @@ date: 2025-05-20T00:23:54Z type: docs weight: 450 menu: - rivanna-alphafold: + hpc-alphafold: parent: AlphaFold --- -{{< figure src=/notes/rivanna-alphafold/img/pipeline-steps.png caption="Prediction pipeline" width=70% height=70% >}} +{{< figure src=/notes/hpc-alphafold/img/pipeline-steps.png caption="Prediction pipeline" width=70% height=70% >}} 1. **User Input (FASTA)** The user provides a FASTA file: one sequence for monomer predictions or multiple sequences for multimer predictions. @@ -45,6 +45,6 @@ menu: 5. **Ranking** The five models are ranked based on a confidence score, and the highest-ranking structures are output as **PDB files**. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_10.png width=90% height=90% caption="AlphaFold model architecture." >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_10.png width=90% height=90% caption="AlphaFold model architecture." >}} -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_12.png width=70% height=70% caption="Each of AlphaFold’s five model predictions is run in parallel, relaxed independently, and ranked by confidence." >}} \ No newline at end of file +{{< figure src=/notes/hpc-alphafold/img/Alphafold_12.png width=70% height=70% caption="Each of AlphaFold’s five model predictions is run in parallel, relaxed independently, and ranked by confidence." >}} \ No newline at end of file diff --git a/content/notes/rivanna-alphafold/alphafold/6-outputs-alphafold.md b/content/notes/hpc-alphafold/alphafold/6-outputs-alphafold.md similarity index 97% rename from content/notes/rivanna-alphafold/alphafold/6-outputs-alphafold.md rename to content/notes/hpc-alphafold/alphafold/6-outputs-alphafold.md index f5fdf4ca..7bb42be6 100644 --- a/content/notes/rivanna-alphafold/alphafold/6-outputs-alphafold.md +++ b/content/notes/hpc-alphafold/alphafold/6-outputs-alphafold.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 550 menu: - rivanna-alphafold: + hpc-alphafold: parent: AlphaFold --- diff --git a/content/notes/rivanna-alphafold/alphafold/7-confidence-metrics.md b/content/notes/hpc-alphafold/alphafold/7-confidence-metrics.md similarity index 97% rename from content/notes/rivanna-alphafold/alphafold/7-confidence-metrics.md rename to content/notes/hpc-alphafold/alphafold/7-confidence-metrics.md index db397d8e..770e5d90 100644 --- a/content/notes/rivanna-alphafold/alphafold/7-confidence-metrics.md +++ b/content/notes/hpc-alphafold/alphafold/7-confidence-metrics.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 580 menu: - rivanna-alphafold: + hpc-alphafold: parent: AlphaFold --- diff --git a/content/notes/rivanna-alphafold/img/Alphafold_10.png b/content/notes/hpc-alphafold/img/Alphafold_10.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_10.png rename to content/notes/hpc-alphafold/img/Alphafold_10.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_12.png b/content/notes/hpc-alphafold/img/Alphafold_12.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_12.png rename to content/notes/hpc-alphafold/img/Alphafold_12.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_17.png b/content/notes/hpc-alphafold/img/Alphafold_17.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_17.png rename to content/notes/hpc-alphafold/img/Alphafold_17.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_18.png b/content/notes/hpc-alphafold/img/Alphafold_18.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_18.png rename to content/notes/hpc-alphafold/img/Alphafold_18.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_2.jpg b/content/notes/hpc-alphafold/img/Alphafold_2.jpg similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_2.jpg rename to content/notes/hpc-alphafold/img/Alphafold_2.jpg diff --git a/content/notes/rivanna-alphafold/img/Alphafold_20.jpg b/content/notes/hpc-alphafold/img/Alphafold_20.jpg similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_20.jpg rename to content/notes/hpc-alphafold/img/Alphafold_20.jpg diff --git a/content/notes/rivanna-alphafold/img/Alphafold_21.png b/content/notes/hpc-alphafold/img/Alphafold_21.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_21.png rename to content/notes/hpc-alphafold/img/Alphafold_21.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_22.png b/content/notes/hpc-alphafold/img/Alphafold_22.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_22.png rename to content/notes/hpc-alphafold/img/Alphafold_22.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_23.png b/content/notes/hpc-alphafold/img/Alphafold_23.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_23.png rename to content/notes/hpc-alphafold/img/Alphafold_23.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_24.png b/content/notes/hpc-alphafold/img/Alphafold_24.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_24.png rename to content/notes/hpc-alphafold/img/Alphafold_24.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_27.png b/content/notes/hpc-alphafold/img/Alphafold_27.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_27.png rename to content/notes/hpc-alphafold/img/Alphafold_27.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_28.png b/content/notes/hpc-alphafold/img/Alphafold_28.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_28.png rename to content/notes/hpc-alphafold/img/Alphafold_28.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_29.png b/content/notes/hpc-alphafold/img/Alphafold_29.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_29.png rename to content/notes/hpc-alphafold/img/Alphafold_29.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_3.png b/content/notes/hpc-alphafold/img/Alphafold_3.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_3.png rename to content/notes/hpc-alphafold/img/Alphafold_3.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_30.png b/content/notes/hpc-alphafold/img/Alphafold_30.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_30.png rename to content/notes/hpc-alphafold/img/Alphafold_30.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_31.png b/content/notes/hpc-alphafold/img/Alphafold_31.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_31.png rename to content/notes/hpc-alphafold/img/Alphafold_31.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_32.png b/content/notes/hpc-alphafold/img/Alphafold_32.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_32.png rename to content/notes/hpc-alphafold/img/Alphafold_32.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_33.png b/content/notes/hpc-alphafold/img/Alphafold_33.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_33.png rename to content/notes/hpc-alphafold/img/Alphafold_33.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_34.png b/content/notes/hpc-alphafold/img/Alphafold_34.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_34.png rename to content/notes/hpc-alphafold/img/Alphafold_34.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_35.png b/content/notes/hpc-alphafold/img/Alphafold_35.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_35.png rename to content/notes/hpc-alphafold/img/Alphafold_35.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_36.jpg b/content/notes/hpc-alphafold/img/Alphafold_36.jpg similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_36.jpg rename to content/notes/hpc-alphafold/img/Alphafold_36.jpg diff --git a/content/notes/rivanna-alphafold/img/Alphafold_4.png b/content/notes/hpc-alphafold/img/Alphafold_4.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_4.png rename to content/notes/hpc-alphafold/img/Alphafold_4.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_40.png b/content/notes/hpc-alphafold/img/Alphafold_40.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_40.png rename to content/notes/hpc-alphafold/img/Alphafold_40.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_41.jpg b/content/notes/hpc-alphafold/img/Alphafold_41.jpg similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_41.jpg rename to content/notes/hpc-alphafold/img/Alphafold_41.jpg diff --git a/content/notes/rivanna-alphafold/img/Alphafold_42.png b/content/notes/hpc-alphafold/img/Alphafold_42.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_42.png rename to content/notes/hpc-alphafold/img/Alphafold_42.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_43.png b/content/notes/hpc-alphafold/img/Alphafold_43.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_43.png rename to content/notes/hpc-alphafold/img/Alphafold_43.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_44.png b/content/notes/hpc-alphafold/img/Alphafold_44.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_44.png rename to content/notes/hpc-alphafold/img/Alphafold_44.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_45.png b/content/notes/hpc-alphafold/img/Alphafold_45.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_45.png rename to content/notes/hpc-alphafold/img/Alphafold_45.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_47.png b/content/notes/hpc-alphafold/img/Alphafold_47.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_47.png rename to content/notes/hpc-alphafold/img/Alphafold_47.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_48.png b/content/notes/hpc-alphafold/img/Alphafold_48.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_48.png rename to content/notes/hpc-alphafold/img/Alphafold_48.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_5.png b/content/notes/hpc-alphafold/img/Alphafold_5.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_5.png rename to content/notes/hpc-alphafold/img/Alphafold_5.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_50.jpg b/content/notes/hpc-alphafold/img/Alphafold_50.jpg similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_50.jpg rename to content/notes/hpc-alphafold/img/Alphafold_50.jpg diff --git a/content/notes/rivanna-alphafold/img/Alphafold_51.png b/content/notes/hpc-alphafold/img/Alphafold_51.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_51.png rename to content/notes/hpc-alphafold/img/Alphafold_51.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_53.png b/content/notes/hpc-alphafold/img/Alphafold_53.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_53.png rename to content/notes/hpc-alphafold/img/Alphafold_53.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_54.png b/content/notes/hpc-alphafold/img/Alphafold_54.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_54.png rename to content/notes/hpc-alphafold/img/Alphafold_54.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_57.png b/content/notes/hpc-alphafold/img/Alphafold_57.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_57.png rename to content/notes/hpc-alphafold/img/Alphafold_57.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_58.png b/content/notes/hpc-alphafold/img/Alphafold_58.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_58.png rename to content/notes/hpc-alphafold/img/Alphafold_58.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_7.png b/content/notes/hpc-alphafold/img/Alphafold_7.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_7.png rename to content/notes/hpc-alphafold/img/Alphafold_7.png diff --git a/content/notes/rivanna-alphafold/img/background-colors.png b/content/notes/hpc-alphafold/img/background-colors.png similarity index 100% rename from content/notes/rivanna-alphafold/img/background-colors.png rename to content/notes/hpc-alphafold/img/background-colors.png diff --git a/content/notes/rivanna-alphafold/img/molecule-file-browser.png b/content/notes/hpc-alphafold/img/molecule-file-browser.png similarity index 100% rename from content/notes/rivanna-alphafold/img/molecule-file-browser.png rename to content/notes/hpc-alphafold/img/molecule-file-browser.png diff --git a/content/notes/rivanna-alphafold/img/pipeline-steps.png b/content/notes/hpc-alphafold/img/pipeline-steps.png similarity index 100% rename from content/notes/rivanna-alphafold/img/pipeline-steps.png rename to content/notes/hpc-alphafold/img/pipeline-steps.png diff --git a/content/notes/rivanna-alphafold/img/rivanna-steps.png b/content/notes/hpc-alphafold/img/rivanna-steps.png similarity index 100% rename from content/notes/rivanna-alphafold/img/rivanna-steps.png rename to content/notes/hpc-alphafold/img/rivanna-steps.png diff --git a/content/notes/rivanna-alphafold/img/vmd-export.png b/content/notes/hpc-alphafold/img/vmd-export.png similarity index 100% rename from content/notes/rivanna-alphafold/img/vmd-export.png rename to content/notes/hpc-alphafold/img/vmd-export.png diff --git a/content/notes/rivanna-alphafold/img/vmd-part-3.png b/content/notes/hpc-alphafold/img/vmd-part-3.png similarity index 100% rename from content/notes/rivanna-alphafold/img/vmd-part-3.png rename to content/notes/hpc-alphafold/img/vmd-part-3.png diff --git a/content/notes/rivanna-alphafold/img/vmd-part-4.png b/content/notes/hpc-alphafold/img/vmd-part-4.png similarity index 100% rename from content/notes/rivanna-alphafold/img/vmd-part-4.png rename to content/notes/hpc-alphafold/img/vmd-part-4.png diff --git a/content/notes/rivanna-alphafold/intro/0-intro.md b/content/notes/hpc-alphafold/intro/0-intro.md similarity index 84% rename from content/notes/rivanna-alphafold/intro/0-intro.md rename to content/notes/hpc-alphafold/intro/0-intro.md index 6fd6a513..8a4fc9bc 100644 --- a/content/notes/rivanna-alphafold/intro/0-intro.md +++ b/content/notes/hpc-alphafold/intro/0-intro.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 200 menu: - rivanna-alphafold: + hpc-alphafold: --- This section will cover: @@ -19,4 +19,4 @@ This section will cover: * Pipeline and Architecture -* Confidence Metrics \ No newline at end of file +* Confidence Metrics diff --git a/content/notes/rivanna-alphafold/intro/1-gpu-vs-cpu.md b/content/notes/hpc-alphafold/intro/1-gpu-vs-cpu.md similarity index 89% rename from content/notes/rivanna-alphafold/intro/1-gpu-vs-cpu.md rename to content/notes/hpc-alphafold/intro/1-gpu-vs-cpu.md index 72c34c9d..83ed031a 100644 --- a/content/notes/rivanna-alphafold/intro/1-gpu-vs-cpu.md +++ b/content/notes/hpc-alphafold/intro/1-gpu-vs-cpu.md @@ -4,11 +4,11 @@ date: 2025-05-20T00:23:54Z type: docs weight: 210 menu: - rivanna-alphafold: + hpc-alphafold: parent: Introduction --- -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_2.jpg width=70% height=70% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_2.jpg width=70% height=70% >}} - The **CPU (Central Processing Unit)** is the general-purpose processor responsible for handling most computing tasks in a system. - The **GPU (Graphics Processing Unit)** is now widely used for accelerating scientific and high-performance computing tasks. @@ -28,9 +28,9 @@ menu: **Metaphor: GPU vs CPU** -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_3.png width=50% height=50% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_3.png width=50% height=50% >}} -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_4.png width=50% height=50% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_4.png width=50% height=50% >}} **CPUs are like scooters**: simple, efficient, and great for quickly handling one task or a few tasks at a time. diff --git a/content/notes/rivanna-alphafold/intro/2-protein-folding-problem.md b/content/notes/hpc-alphafold/intro/2-protein-folding-problem.md similarity index 95% rename from content/notes/rivanna-alphafold/intro/2-protein-folding-problem.md rename to content/notes/hpc-alphafold/intro/2-protein-folding-problem.md index b8c0c013..5b657131 100644 --- a/content/notes/rivanna-alphafold/intro/2-protein-folding-problem.md +++ b/content/notes/hpc-alphafold/intro/2-protein-folding-problem.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 250 menu: - rivanna-alphafold: + hpc-alphafold: parent: Introduction --- @@ -15,4 +15,4 @@ As of 2025, only about 174,000 protein structures have been solved and deposited This massive gap between known sequences and solved structures is known as the **protein folding problem**. -**AlphaFold** addresses this problem using AI to predict protein structures from amino acid sequences — at scale and with high accuracy. \ No newline at end of file +**AlphaFold** addresses this problem using AI to predict protein structures from amino acid sequences — at scale and with high accuracy. diff --git a/content/notes/rivanna-alphafold/intro/3-casp-competitions.md b/content/notes/hpc-alphafold/intro/3-casp-competitions.md similarity index 73% rename from content/notes/rivanna-alphafold/intro/3-casp-competitions.md rename to content/notes/hpc-alphafold/intro/3-casp-competitions.md index fa7a6a08..9045dbfc 100644 --- a/content/notes/rivanna-alphafold/intro/3-casp-competitions.md +++ b/content/notes/hpc-alphafold/intro/3-casp-competitions.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 300 menu: - rivanna-alphafold: + hpc-alphafold: parent: Introduction --- @@ -14,12 +14,12 @@ Participants are challenged to predict structures of proteins whose experimental CASP uses the **Global Distance Test - Total Score** (GDT-TS) as a performance metric, which measures what percentage of α-carbons are within a threshold distance (in angstroms) of the experimental structure. The higher the GDT-TS score, the more accurate the model's prediction is. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_5.png width=25% height=25% caption="Example of a predicted protein structure (orange) compared to the experimentally solved structure (black). CASP evaluates prediction accuracy using these kinds of comparisons." >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_5.png width=25% height=25% caption="Example of a predicted protein structure (orange) compared to the experimentally solved structure (black). CASP evaluates prediction accuracy using these kinds of comparisons." >}} AlphaFold’s strong performance in **CASP13** and **CASP14** proved its ability to predict protein structures with near-experimental accuracy. This established AlphaFold as a game changer in the world of structural biology and computational science. [Learn More](https://www.nature.com/articles/d41586-020-03348-4) -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_7.png width=60% height=60% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_7.png width=60% height=60% >}} diff --git a/content/notes/rivanna-alphafold/out/14-outputs.md b/content/notes/hpc-alphafold/out/14-outputs.md similarity index 80% rename from content/notes/rivanna-alphafold/out/14-outputs.md rename to content/notes/hpc-alphafold/out/14-outputs.md index 2718fde4..0b415011 100644 --- a/content/notes/rivanna-alphafold/out/14-outputs.md +++ b/content/notes/hpc-alphafold/out/14-outputs.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1650 menu: - rivanna-alphafold: + hpc-alphafold: name: Outputs --- @@ -16,6 +16,6 @@ After running AlphaFold, your output directory will include a mix of files from > Files labeled `ranked_0` represent the **best-ranked model** based on AlphaFold's internal confidence scoring. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_24.png width=40% height=40% caption="Example output directory" >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_24.png width=40% height=40% caption="Example output directory" >}} diff --git a/content/notes/rivanna-alphafold/out/15-sequence-coverage-plot.md b/content/notes/hpc-alphafold/out/15-sequence-coverage-plot.md similarity index 69% rename from content/notes/rivanna-alphafold/out/15-sequence-coverage-plot.md rename to content/notes/hpc-alphafold/out/15-sequence-coverage-plot.md index 889d4477..fa5e8dc2 100644 --- a/content/notes/rivanna-alphafold/out/15-sequence-coverage-plot.md +++ b/content/notes/hpc-alphafold/out/15-sequence-coverage-plot.md @@ -4,13 +4,13 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1700 menu: - rivanna-alphafold: + hpc-alphafold: parent: Outputs --- Sequence coverage plots (`*_sequence_coverage.pdf`) show the alignment depth along the length of the protein. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_27.png width=85% height=85% caption="Sequence coverage plot for CB1. Darker colors indicate higher identity to the query sequence.">}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_27.png width=85% height=85% caption="Sequence coverage plot for CB1. Darker colors indicate higher identity to the query sequence.">}} This example has a depth of ~13,000 sequences, with varying levels of homology to the query. The CB1 protein is a 7-transmembrane (7TM) helical protein with less-structured tails on both termini — which is also reflected in its 3D structure predictions. diff --git a/content/notes/rivanna-alphafold/out/15a-postprocessing.md b/content/notes/hpc-alphafold/out/15a-postprocessing.md similarity index 98% rename from content/notes/rivanna-alphafold/out/15a-postprocessing.md rename to content/notes/hpc-alphafold/out/15a-postprocessing.md index 77a7cd13..f4c63e66 100644 --- a/content/notes/rivanna-alphafold/out/15a-postprocessing.md +++ b/content/notes/hpc-alphafold/out/15a-postprocessing.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1740 menu: - rivanna-alphafold: + hpc-alphafold: parent: Outputs --- diff --git a/content/notes/rivanna-alphafold/out/16-plddt-plot.md b/content/notes/hpc-alphafold/out/16-plddt-plot.md similarity index 82% rename from content/notes/rivanna-alphafold/out/16-plddt-plot.md rename to content/notes/hpc-alphafold/out/16-plddt-plot.md index 7caabbdb..aa3e4392 100644 --- a/content/notes/rivanna-alphafold/out/16-plddt-plot.md +++ b/content/notes/hpc-alphafold/out/16-plddt-plot.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1750 menu: - rivanna-alphafold: + hpc-alphafold: parent: Outputs --- @@ -14,9 +14,9 @@ This output plot shows the **per-residue confidence scores** (pLDDT) for the top For example, CB1 is a 7-transmembrane (7TM) domain protein with less-structured tails on both termini which is reflected in the plot. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_29.png width=80% height=80% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_29.png width=80% height=80% >}} -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_30.png width=30% height=30% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_30.png width=30% height=30% >}} ### Key Details diff --git a/content/notes/rivanna-alphafold/out/17-pae-plot.md b/content/notes/hpc-alphafold/out/17-pae-plot.md similarity index 89% rename from content/notes/rivanna-alphafold/out/17-pae-plot.md rename to content/notes/hpc-alphafold/out/17-pae-plot.md index 576ba088..26cac732 100644 --- a/content/notes/rivanna-alphafold/out/17-pae-plot.md +++ b/content/notes/hpc-alphafold/out/17-pae-plot.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1800 menu: - rivanna-alphafold: + hpc-alphafold: parent: Outputs --- @@ -18,7 +18,7 @@ The color scale indicates _expected distance error_ in angstroms. Purple is 'go For example, the CB1 protein is a 7-transmembrane domain GPCR with less-structured tails on both termini, which appear as regions of higher predicted error. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_31.png width=60% height=60% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_31.png width=60% height=60% >}} Source: Arnold, M. J. (2021) AlphaPickle doi.org/10.5281/zenodo.5708709 diff --git a/content/notes/rivanna-alphafold/out/18-csv-outputs.md b/content/notes/hpc-alphafold/out/18-csv-outputs.md similarity index 65% rename from content/notes/rivanna-alphafold/out/18-csv-outputs.md rename to content/notes/hpc-alphafold/out/18-csv-outputs.md index 7a2ef5c0..ea1fe27b 100644 --- a/content/notes/rivanna-alphafold/out/18-csv-outputs.md +++ b/content/notes/hpc-alphafold/out/18-csv-outputs.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1850 menu: - rivanna-alphafold: + hpc-alphafold: parent: Outputs --- @@ -12,13 +12,13 @@ AlphaPickle generates CSV files that can be used for downstream analysis and plo **`PAE.csv`** -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_32.png width=70% height=70% caption="Example PAE.csv file contents." >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_32.png width=70% height=70% caption="Example PAE.csv file contents." >}} `PAE.csv` is an `n_residue` x `n_residue` matrix that is used for generating PAE heatmaps. **`pLDDT.csv`** -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_33.png width=20% height=20% caption="Example pLDDT.csv file contents." >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_33.png width=20% height=20% caption="Example pLDDT.csv file contents." >}} `pLDDT.csv` is an `n_residue`-long list which is helpful for generating pLDDT plots. diff --git a/content/notes/rivanna-alphafold/out/19-pdb-structure.md b/content/notes/hpc-alphafold/out/19-pdb-structure.md similarity index 80% rename from content/notes/rivanna-alphafold/out/19-pdb-structure.md rename to content/notes/hpc-alphafold/out/19-pdb-structure.md index 2d153a2d..df2660f2 100644 --- a/content/notes/rivanna-alphafold/out/19-pdb-structure.md +++ b/content/notes/hpc-alphafold/out/19-pdb-structure.md @@ -4,18 +4,18 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1900 menu: - rivanna-alphafold: + hpc-alphafold: parent: Outputs --- -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_34.png width=40% height=40% caption="Predicted 3D structure of CB1." >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_34.png width=40% height=40% caption="Predicted 3D structure of CB1." >}} The above image shows an example of a 3D structure from a `ranked_0.pdb` file. The structure is colored by `pLDDT`, where dark blue represents high confidence and red represents low confidence. CB1 is a 7-transmembrane domain (7TM) septa-helical protein with less-structured tails on both termini. This is reflected in the 3D structure, where disordered regions ("ribbons") are low-confidence and helices are high-confidence. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_35.png width=40% height=40% caption="Schematic diagram of CB1.">}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_35.png width=40% height=40% caption="Schematic diagram of CB1.">}} This schematic complements the 3D model by showing the membrane topology of CB1. The arrangement of helices and loops explains the structured and unstructured regions seen in the AlphaFold prediction. diff --git a/content/notes/rivanna-alphafold/running/10-working-directory-alphafold.md b/content/notes/hpc-alphafold/running/10-working-directory-alphafold.md similarity index 90% rename from content/notes/rivanna-alphafold/running/10-working-directory-alphafold.md rename to content/notes/hpc-alphafold/running/10-working-directory-alphafold.md index dd5ac5f8..ef1bbf47 100644 --- a/content/notes/rivanna-alphafold/running/10-working-directory-alphafold.md +++ b/content/notes/hpc-alphafold/running/10-working-directory-alphafold.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 800 menu: - rivanna-alphafold: + hpc-alphafold: parent: Running AlphaFold On Rivanna --- @@ -48,4 +48,4 @@ cd alphafold ls -lh ``` -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_21.png width=90% height=90% caption="Example of AlphaFold working directory contents">}} \ No newline at end of file +{{< figure src=/notes/hpc-alphafold/img/Alphafold_21.png width=90% height=90% caption="Example of AlphaFold working directory contents">}} \ No newline at end of file diff --git a/content/notes/rivanna-alphafold/running/11-input-fasta.md b/content/notes/hpc-alphafold/running/11-input-fasta.md similarity index 71% rename from content/notes/rivanna-alphafold/running/11-input-fasta.md rename to content/notes/hpc-alphafold/running/11-input-fasta.md index 9d62731a..0758cc46 100644 --- a/content/notes/rivanna-alphafold/running/11-input-fasta.md +++ b/content/notes/hpc-alphafold/running/11-input-fasta.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1100 menu: - rivanna-alphafold: + hpc-alphafold: parent: Running AlphaFold On Rivanna --- @@ -16,6 +16,6 @@ As an example, you can download a sequence from UniProt, such as [P21554 (CNR1)] Next, get your FASTA file onto Rivanna in your working directory. You can do this using OOD, Globus, or `scp` (command line tool), or you can make the file directly on Rivanna with a text editor like `nano` or `vim`. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_22.png width=40% height=40% caption="Schematic diagram of the expected structure for CNR1">}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_22.png width=40% height=40% caption="Schematic diagram of the expected structure for CNR1">}} -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_23.png width=90% height=90% caption="Text representation of a FASTA file for CNR1">}} \ No newline at end of file +{{< figure src=/notes/hpc-alphafold/img/Alphafold_23.png width=90% height=90% caption="Text representation of a FASTA file for CNR1">}} \ No newline at end of file diff --git a/content/notes/rivanna-alphafold/running/12-submit-slurm.md b/content/notes/hpc-alphafold/running/12-submit-slurm.md similarity index 94% rename from content/notes/rivanna-alphafold/running/12-submit-slurm.md rename to content/notes/hpc-alphafold/running/12-submit-slurm.md index 31b713f6..c6625a31 100644 --- a/content/notes/rivanna-alphafold/running/12-submit-slurm.md +++ b/content/notes/hpc-alphafold/running/12-submit-slurm.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1150 menu: - rivanna-alphafold: + hpc-alphafold: parent: Running AlphaFold On Rivanna --- diff --git a/content/notes/rivanna-alphafold/running/7-loggingon.md b/content/notes/hpc-alphafold/running/7-loggingon.md similarity index 68% rename from content/notes/rivanna-alphafold/running/7-loggingon.md rename to content/notes/hpc-alphafold/running/7-loggingon.md index ec864a60..4c92ae91 100644 --- a/content/notes/rivanna-alphafold/running/7-loggingon.md +++ b/content/notes/hpc-alphafold/running/7-loggingon.md @@ -4,22 +4,22 @@ date: 2025-05-20T00:23:54Z type: docs weight: 605 menu: - rivanna-alphafold: + hpc-alphafold: parent: Running AlphaFold On Rivanna --- First, log in to the [Open OnDemand portal](ood.hpc.virginia.edu) and navigate to 'Desktop'. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_17.png width=80% height=80% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_17.png width=80% height=80% >}} From there, you can launch a Desktop session on Rivanna. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_18.png width=60% height=60% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_18.png width=60% height=60% >}} Choose your desired resources (cores, memory, time, etc.). Note: You must have a valid allocation to launch a job. -The `rivanna-training` allocation is no longer supported for AlphaFold jobs. +The `hpc-training` allocation is no longer supported for AlphaFold jobs. If you are part of a lab or course, use the allocation provided to you (e.g., `yourgroup-allocation`). If you're unsure what to use, contact your PI or the RC support team for help. @@ -29,7 +29,7 @@ Once your session is ready, click the blue "Launch Desktop" button. You’ll be placed in a Linux environment like this: -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_20.jpg width=70% height=70% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_20.jpg width=70% height=70% >}} Use the MATE Terminal to run commands or load software modules. diff --git a/content/notes/rivanna-alphafold/running/7a-prep-wd.md b/content/notes/hpc-alphafold/running/7a-prep-wd.md similarity index 98% rename from content/notes/rivanna-alphafold/running/7a-prep-wd.md rename to content/notes/hpc-alphafold/running/7a-prep-wd.md index 16fc11ea..860794da 100644 --- a/content/notes/rivanna-alphafold/running/7a-prep-wd.md +++ b/content/notes/hpc-alphafold/running/7a-prep-wd.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 610 menu: - rivanna-alphafold: + hpc-alphafold: parent: Running AlphaFold On Rivanna --- diff --git a/content/notes/rivanna-alphafold/running/7b-prep-conda-env.md b/content/notes/hpc-alphafold/running/7b-prep-conda-env.md similarity index 96% rename from content/notes/rivanna-alphafold/running/7b-prep-conda-env.md rename to content/notes/hpc-alphafold/running/7b-prep-conda-env.md index 96f3095e..8ccd7865 100644 --- a/content/notes/rivanna-alphafold/running/7b-prep-conda-env.md +++ b/content/notes/hpc-alphafold/running/7b-prep-conda-env.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 620 menu: - rivanna-alphafold: + hpc-alphafold: parent: Running AlphaFold On Rivanna --- diff --git a/content/notes/rivanna-alphafold/running/8-running-on-rivanna.md b/content/notes/hpc-alphafold/running/8-running-on-rivanna.md similarity index 97% rename from content/notes/rivanna-alphafold/running/8-running-on-rivanna.md rename to content/notes/hpc-alphafold/running/8-running-on-rivanna.md index bd7a00e1..f5032edb 100644 --- a/content/notes/rivanna-alphafold/running/8-running-on-rivanna.md +++ b/content/notes/hpc-alphafold/running/8-running-on-rivanna.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 600 menu: - rivanna-alphafold: + hpc-alphafold: name: Running AlphaFold On Rivanna --- diff --git a/content/notes/hpc-alphafold/vmd/20-visualizing-structures.md b/content/notes/hpc-alphafold/vmd/20-visualizing-structures.md new file mode 100644 index 00000000..e1db76a1 --- /dev/null +++ b/content/notes/hpc-alphafold/vmd/20-visualizing-structures.md @@ -0,0 +1,31 @@ +--- +title: Visualizing 3D Structures +date: 2025-05-20T00:23:54Z +type: docs +weight: 1950 +menu: + hpc-alphafold: + name: Visualizing 3D Structures +--- + +To view your predicted structure generated by AlphaFold in 3D, you can use **Visual Molecular Dynamics (VMD)**. +{{< figure src=/notes/hpc-alphafold/img/Alphafold_40.png width=60% height=60% >}} + +Steps: +* Log onto Open On Demand ([https://hpc-portal.hpc.virginia.edu/](https://hpc-portal.hpc.virginia.edu/)) +* Choose the ‘Desktop’ interactive app and fill out the webform with your required resources. +* Click launch. +* Load VMD in the terminal: +```bash +module load vmd +vmd +``` +The VMD windows will launch. From there, you can open your `.pdb` file from the AlphaFold output directory to view the 3D model (colored by `pLDDT`). + +{{< figure src=/notes/hpc-alphafold/img/Alphafold_41.jpg width=65% height=65% caption="Loading your .pdb file (a)." >}} +{{< figure src=/notes/hpc-alphafold/img/molecule-file-browser.png width=65% height=65% caption="Loading your .pdb file (b)." >}} +{{< figure src=/notes/hpc-alphafold/img/vmd-part-3.png width=60% height=60% caption="Loading your .pdb file (c)." >}} +{{< figure src=/notes/hpc-alphafold/img/vmd-part-4.png width=50% height=50% caption="Loading your .pdb file (d)." >}} + + + diff --git a/content/notes/rivanna-alphafold/vmd/21-modify-vmd.md b/content/notes/hpc-alphafold/vmd/21-modify-vmd.md similarity index 57% rename from content/notes/rivanna-alphafold/vmd/21-modify-vmd.md rename to content/notes/hpc-alphafold/vmd/21-modify-vmd.md index 8b4c16fa..49a06253 100644 --- a/content/notes/rivanna-alphafold/vmd/21-modify-vmd.md +++ b/content/notes/hpc-alphafold/vmd/21-modify-vmd.md @@ -4,13 +4,13 @@ date: 2025-05-20T00:23:54Z type: docs weight: 2150 menu: - rivanna-alphafold: + hpc-alphafold: parent: Visualizing 3D Structures --- Once your PDB file is loaded in VMD, you can change the visual style using the Graphical Representations menu. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_45.png width=50% height=50% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_45.png width=50% height=50% >}} ### Representation settings @@ -19,10 +19,10 @@ Once your PDB file is loaded in VMD, you can change the visual style using the G **Coloring Method:** Set to `Beta` to visualize `pLDDT` confidence values (stored in the beta column). -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_47.png width=30% height=30% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_47.png width=30% height=30% >}} ### Result -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_48.png width=70% height=70% caption="Resulting visualization colored by pLDDT via 'beta' slot." >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_48.png width=70% height=70% caption="Resulting visualization colored by pLDDT via 'beta' slot." >}} diff --git a/content/notes/rivanna-alphafold/vmd/22-background-color-vmd.md b/content/notes/hpc-alphafold/vmd/22-background-color-vmd.md similarity index 78% rename from content/notes/rivanna-alphafold/vmd/22-background-color-vmd.md rename to content/notes/hpc-alphafold/vmd/22-background-color-vmd.md index c4e44791..32a63351 100644 --- a/content/notes/rivanna-alphafold/vmd/22-background-color-vmd.md +++ b/content/notes/hpc-alphafold/vmd/22-background-color-vmd.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 2200 menu: - rivanna-alphafold: + hpc-alphafold: parent: Visualizing 3D Structures --- @@ -18,8 +18,8 @@ To change the background color: 4. In the *Names* list, select **Background**. 5. Choose a new color from the *Colors* list or modify the RGB sliders in the *Color Definitions* tab. -{{< figure src=/notes/rivanna-alphafold/img/background-colors.png width=75% height=75% >}} -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_54.png width=65% height=65% >}} +{{< figure src=/notes/hpc-alphafold/img/background-colors.png width=75% height=75% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_54.png width=65% height=65% >}} >This isn’t a full VMD tutorial, but it is beneficial to play around with it as needed to help you in your research. diff --git a/content/notes/rivanna-alphafold/vmd/22-exporting-vmd.md b/content/notes/hpc-alphafold/vmd/22-exporting-vmd.md similarity index 79% rename from content/notes/rivanna-alphafold/vmd/22-exporting-vmd.md rename to content/notes/hpc-alphafold/vmd/22-exporting-vmd.md index 1b9a3105..7fb469cf 100644 --- a/content/notes/rivanna-alphafold/vmd/22-exporting-vmd.md +++ b/content/notes/hpc-alphafold/vmd/22-exporting-vmd.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 2250 menu: - rivanna-alphafold: + hpc-alphafold: parent: Visualizing 3D Structures --- @@ -13,17 +13,17 @@ The simplest way to produce raster image files in VMD is to use the `Snapshot` f To use the snapshot feature, simply open the Render form and choose the snapshot option. VMD will capture the contents of the graphics window, and attempt to save the resulting image to the filename given in the Render form. You may find that it is important not to have other windows or cursors in front of the VMD graphics display when using snapshot, since the resulting images may include obscuring windows or cursors. This is a platform-dependent behavior, so you will need to determine if your system does this or not. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_50.jpg width=60% height=60% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_50.jpg width=60% height=60% >}} -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_51.png width=60% height=60% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_51.png width=60% height=60% >}} In the pop-up `imagemagick` window, right-click and hit `save` (see below). -{{< figure src=/notes/rivanna-alphafold/img/vmd-export.png width=50% height=50% >}} +{{< figure src=/notes/hpc-alphafold/img/vmd-export.png width=50% height=50% >}} Next, hit `format`, and a new popup appears. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_53.png width=75% height=75% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_53.png width=75% height=75% >}} Select your image format (`TIFF` or `PNG` is recommended) and hit `select`. Type the file name, including the extension, and hit `save`. diff --git a/content/notes/hpc-intro/_index.md b/content/notes/hpc-intro/_index.md index cd7c1827..0fa8a8bd 100644 --- a/content/notes/hpc-intro/_index.md +++ b/content/notes/hpc-intro/_index.md @@ -18,4 +18,4 @@ menu: UVA's primary resource for high\-performance computing provides a platform for computationally-intensive research across a variety of disciplines. -{{< figure src="/notes/hpc-intro/img/rivanna_racks.png" alt="A data center with racks of servers." caption="Some servers in UVA's HPC cluster" >}} +{{< figure src="/notes/hpc-intro/img/rivanna_racks.jpg" alt="A data center with racks of servers." caption="Some servers in UVA's HPC cluster" >}} diff --git a/content/notes/hpc-intro/img/rivanna_racks.jpg b/content/notes/hpc-intro/img/rivanna_racks.jpg new file mode 100644 index 00000000..9ed770b7 Binary files /dev/null and b/content/notes/hpc-intro/img/rivanna_racks.jpg differ diff --git a/content/notes/matlab-parallel-programming/matlab-parallel-programming_1.md b/content/notes/matlab-parallel-programming/matlab-parallel-programming_1.md index cf780df5..94cb87b0 100644 --- a/content/notes/matlab-parallel-programming/matlab-parallel-programming_1.md +++ b/content/notes/matlab-parallel-programming/matlab-parallel-programming_1.md @@ -27,7 +27,7 @@ However, the challenge lies in effectively utilizing this powerful hardware, whi - **Accelerate workflows** with minimal changes to existing code - **Scale applications** seamlessly from desktop to clusters or cloud for more computational power and memory -{{< figure src=/notes/matlab-parallel-programming/img/matlab-gpu-multicore-cpu.png >}} +{{< figure src="/notes/matlab-parallel-programming/img/matlab-gpu-multicore-cpu.png" >}} Let's look at some examples of customer across multiple industries who have been using MathWorks parallel computing tools and why they chose parallel computing @@ -36,13 +36,13 @@ Let's look at some examples of customer across multiple industries who have been --- -| {{< figure src=/notes/matlab-parallel-programming/img/automotive-test-analysis.png width="300px" >}} | {{< figure src=/notes/matlab-parallel-programming/img/discrete-model-fleet-performance.png width="300px" >}} | +| {{< figure src="/notes/matlab-parallel-programming/img/automotive-test-analysis.png" width="300px" >}} | {{< figure src="/notes/matlab-parallel-programming/img/discrete-model-fleet-performance.png" width="300px" >}} | |:-----------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------:| | **Automotive Test Analysis and Visualization** | **Discrete-Event Model of Fleet Performance** | | - 3–4 months of development time saved | - Simulation time reduced from months to hours | | - Validation time sped up 2X | - Simulation time sped up 20X | -| {{< figure src=/notes/matlab-parallel-programming/img/heart-transplant-studies.png width="300px" >}} | {{< figure src=/notes/matlab-parallel-programming/img/derived-market-data.jpg width="300px" >}} | +| {{< figure src="/notes/matlab-parallel-programming/img/heart-transplant-studies.png" width="300px" >}} | {{< figure src="/notes/matlab-parallel-programming/img/derived-market-data.jpg" width="300px" >}} | |:-----------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------:| | **Heart Transplant Studies** | **Calculating Derived Market Data** | | - 4 weeks reduced to 5 days | - Implementation time reduced by months | diff --git a/content/notes/matlab-statistics/additionalRes1.png b/content/notes/matlab-statistics/additionalRes1.png deleted file mode 100644 index 1a05713a..00000000 Binary files a/content/notes/matlab-statistics/additionalRes1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/centrality1.png b/content/notes/matlab-statistics/centrality1.png deleted file mode 100644 index 36adf03e..00000000 Binary files a/content/notes/matlab-statistics/centrality1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/centrality2.png b/content/notes/matlab-statistics/centrality2.png deleted file mode 100644 index 77747144..00000000 Binary files a/content/notes/matlab-statistics/centrality2.png and /dev/null differ diff --git a/content/notes/matlab-statistics/centrality3.png b/content/notes/matlab-statistics/centrality3.png deleted file mode 100644 index a90ac400..00000000 Binary files a/content/notes/matlab-statistics/centrality3.png and /dev/null differ diff --git a/content/notes/matlab-statistics/distribution1.png b/content/notes/matlab-statistics/distribution1.png deleted file mode 100644 index 0f5497e1..00000000 Binary files a/content/notes/matlab-statistics/distribution1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/distribution2.png b/content/notes/matlab-statistics/distribution2.png deleted file mode 100644 index 5fa596e7..00000000 Binary files a/content/notes/matlab-statistics/distribution2.png and /dev/null differ diff --git a/content/notes/matlab-statistics/distribution3.png b/content/notes/matlab-statistics/distribution3.png deleted file mode 100644 index 664d8d5c..00000000 Binary files a/content/notes/matlab-statistics/distribution3.png and /dev/null differ diff --git a/content/notes/matlab-statistics/exploreData1.png b/content/notes/matlab-statistics/exploreData1.png deleted file mode 100644 index 89c5cac5..00000000 Binary files a/content/notes/matlab-statistics/exploreData1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/exploreData2.png b/content/notes/matlab-statistics/exploreData2.png deleted file mode 100644 index fc7ac5a5..00000000 Binary files a/content/notes/matlab-statistics/exploreData2.png and /dev/null differ diff --git a/content/notes/matlab-statistics/index.md b/content/notes/matlab-statistics/index.md deleted file mode 100644 index 64a64301..00000000 --- a/content/notes/matlab-statistics/index.md +++ /dev/null @@ -1,200 +0,0 @@ ---- -date : "2019-06-23T08:37:46-05:00" -title : "Statistical Methods with MATLAB" -type: article -toc : true ---- - -# Overview - -MATLAB is an integrated technical computing environment from the MathWorks that combines array-based numeric computation, advanced graphics and visualization, and a high-level programming language. Separately licensed toolboxes provide additional domain-specific functionality. - -**Matlab Academy: Statistical Methods with Matlab** - -**Documentation: Statistics and Machine Learning Toolbox (help page)** - -**Documentation: Statistics and Machine Learning Toolbox (product page)** -
-
- - -## Course Overview - -**Video: Statistical Methods with Matlab** - -## Exploring Data - -### Visualizing Data Sets -{{< figure src="exploreData1.png" >}} - -{{< figure src="exploreData2.png" >}} - -**Documentation: histogram ** ** boxplot ** ** scatter ** - -
-### Measures of Centrality and Spread -{{< figure src="centrality1.png" >}} -{{< figure src="centrality2.png" >}} -{{< figure src="centrality3.png" >}} - -** Documentation: mean ** ** median ** -** mode ** ** trimmean ** - -
- -{{< figure src="spread1.png" >}} -{{< figure src="spread2.png" >}} -{{< figure src="spread3.png" >}} - -** Documentation: std ** ** iqr ** -** range ** ** var ** - - -
- - - -### Distributions -{{< figure src="distribution3.png" >}} -{{< figure src="spread4.png" >}} - -
-{{< figure src="distribution1.png" >}} -{{< figure src="distribution2.png" >}} - -**Documentation: normpdf ** ** unifpdf ** ** randn ** ** rand ** - -### Summary -{{< figure src="summary1.png" >}} - - -
-{{< figure src="summary2.png" >}} - - -
- -{{< figure src="summary3.png" >}} - - - -
- -## Fitting a Curve to Data - -### Linear Regression -{{< figure src="regression1.png" >}} -{{< figure src="regression2.png" >}} -{{< figure src="regression4.png" >}} -{{< figure src="regression3.png" >}} - -** Documentation: fit ** - -
- -### Evaluating Goodness of Fit -{{< figure src="regression5.png" >}} -{{< figure src="regression6.png" >}} -{{< figure src="regression7.png" >}} - -
- -### Nonlinear Regression -{{< figure src="regression8.png" >}} -{{< figure src="regression9.png" >}} -{{< figure src="regression10.png" >}} - -### Summary -{{< figure src="summary4.png" >}} -{{< figure src="summary5.png" >}} -{{< figure src="summary6.png" >}} - -## Interpolating Data - -### Linear Interpolation -{{< figure src="interp1.png" >}} -{{< figure src="interp2.png" >}} -{{< figure src="interp3.png" >}} -{{< figure src="interp4.png" >}} -{{< figure src="interp5.png" >}} -{{< figure src="interp6.png" >}} - -### Nonlinear Interpolation - -{{< figure src="interp7.png" >}} -{{< figure src="interp8.png" >}} -{{< figure src="interp9.png" >}} - -** Documentation: interp1 ** - -
- -### Summary -{{< figure src="summary7.png" >}} -{{< figure src="summary8.png" >}} - -## Additional Resources - -{{< figure src="additionalRes1.png" >}} - -
-**MATLAB Central** ** MathWorks Support ** - -
- -## Exercises - -### Visualizing Data sets - -**Exercise: Visualize Height and Weight Data** - -### Measure of Centrality and Spread - -**Exercise: Find the Mean and Median** - -**Exercise: Find the Standard Deviation and IQR** - -### Distributions - -**Exercise: Fit and Plot a Normal Distribution** - -**Exercise: Generating Random Numbers** - -### Review: Exploring Data - -**Exercise: Earthquakes** - -### Linear Regression - -**Exercise: Fit a Line to Data** - -**Exercise: Fit a Polynomial to Data** - -### Evaluating the Goodness of Fit - -**Exercise: Evaluate and Improve the Fit** - - -
-
-### Nonlinear Regression - -**Exercise: Fit a Nonlinear Model** - -### Review: Fitting a Curve to Data - -**Exercise: Temperature Fluctuations** - -### Linear Interpolation - -**Exercise: Fill in Missing Data** - -**Exercise: Resample Data** - -### Nonlinear Interpolation - -**Exercise: Resample Data with Different Interpolation Methods** - -### Review: Interpolation - -**Exercise: Stock Prices** diff --git a/content/notes/matlab-statistics/interp1.png b/content/notes/matlab-statistics/interp1.png deleted file mode 100644 index 1b5705de..00000000 Binary files a/content/notes/matlab-statistics/interp1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interp2.png b/content/notes/matlab-statistics/interp2.png deleted file mode 100644 index a5914fd6..00000000 Binary files a/content/notes/matlab-statistics/interp2.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interp3.png b/content/notes/matlab-statistics/interp3.png deleted file mode 100644 index 59a2c0e8..00000000 Binary files a/content/notes/matlab-statistics/interp3.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interp4.png b/content/notes/matlab-statistics/interp4.png deleted file mode 100644 index 201f6a1c..00000000 Binary files a/content/notes/matlab-statistics/interp4.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interp5.png b/content/notes/matlab-statistics/interp5.png deleted file mode 100644 index 98d2c184..00000000 Binary files a/content/notes/matlab-statistics/interp5.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interp6.png b/content/notes/matlab-statistics/interp6.png deleted file mode 100644 index 61ace1d0..00000000 Binary files a/content/notes/matlab-statistics/interp6.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interp7.png b/content/notes/matlab-statistics/interp7.png deleted file mode 100644 index 502cf0ff..00000000 Binary files a/content/notes/matlab-statistics/interp7.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interp8.png b/content/notes/matlab-statistics/interp8.png deleted file mode 100644 index 842c8a97..00000000 Binary files a/content/notes/matlab-statistics/interp8.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interp9.png b/content/notes/matlab-statistics/interp9.png deleted file mode 100644 index d1f0f1dc..00000000 Binary files a/content/notes/matlab-statistics/interp9.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret1.png b/content/notes/matlab-statistics/interpret1.png deleted file mode 100644 index f39e9865..00000000 Binary files a/content/notes/matlab-statistics/interpret1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret10.png b/content/notes/matlab-statistics/interpret10.png deleted file mode 100644 index 45455bc7..00000000 Binary files a/content/notes/matlab-statistics/interpret10.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret2.png b/content/notes/matlab-statistics/interpret2.png deleted file mode 100644 index 58819e7f..00000000 Binary files a/content/notes/matlab-statistics/interpret2.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret3.png b/content/notes/matlab-statistics/interpret3.png deleted file mode 100644 index 6ea1ec8c..00000000 Binary files a/content/notes/matlab-statistics/interpret3.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret4.png b/content/notes/matlab-statistics/interpret4.png deleted file mode 100644 index f38c6543..00000000 Binary files a/content/notes/matlab-statistics/interpret4.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret5.png b/content/notes/matlab-statistics/interpret5.png deleted file mode 100644 index 26faaa66..00000000 Binary files a/content/notes/matlab-statistics/interpret5.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret6.png b/content/notes/matlab-statistics/interpret6.png deleted file mode 100644 index 3b643931..00000000 Binary files a/content/notes/matlab-statistics/interpret6.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret7.png b/content/notes/matlab-statistics/interpret7.png deleted file mode 100644 index 1bcf6ffe..00000000 Binary files a/content/notes/matlab-statistics/interpret7.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret8.png b/content/notes/matlab-statistics/interpret8.png deleted file mode 100644 index f17b453b..00000000 Binary files a/content/notes/matlab-statistics/interpret8.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret9.png b/content/notes/matlab-statistics/interpret9.png deleted file mode 100644 index 33fb8f31..00000000 Binary files a/content/notes/matlab-statistics/interpret9.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression1.png b/content/notes/matlab-statistics/regression1.png deleted file mode 100644 index ddba8446..00000000 Binary files a/content/notes/matlab-statistics/regression1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression10.png b/content/notes/matlab-statistics/regression10.png deleted file mode 100644 index 5cbb9007..00000000 Binary files a/content/notes/matlab-statistics/regression10.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression2.png b/content/notes/matlab-statistics/regression2.png deleted file mode 100644 index 2feaae00..00000000 Binary files a/content/notes/matlab-statistics/regression2.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression3.png b/content/notes/matlab-statistics/regression3.png deleted file mode 100644 index c884b3e9..00000000 Binary files a/content/notes/matlab-statistics/regression3.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression4.png b/content/notes/matlab-statistics/regression4.png deleted file mode 100644 index 6a2af1ad..00000000 Binary files a/content/notes/matlab-statistics/regression4.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression5.png b/content/notes/matlab-statistics/regression5.png deleted file mode 100644 index 82ff00c1..00000000 Binary files a/content/notes/matlab-statistics/regression5.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression6.png b/content/notes/matlab-statistics/regression6.png deleted file mode 100644 index 152706e3..00000000 Binary files a/content/notes/matlab-statistics/regression6.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression7.png b/content/notes/matlab-statistics/regression7.png deleted file mode 100644 index 985743b1..00000000 Binary files a/content/notes/matlab-statistics/regression7.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression8.png b/content/notes/matlab-statistics/regression8.png deleted file mode 100644 index 1e87519b..00000000 Binary files a/content/notes/matlab-statistics/regression8.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression9.png b/content/notes/matlab-statistics/regression9.png deleted file mode 100644 index 44257c9b..00000000 Binary files a/content/notes/matlab-statistics/regression9.png and /dev/null differ diff --git a/content/notes/matlab-statistics/spread1.png b/content/notes/matlab-statistics/spread1.png deleted file mode 100644 index 1b3960ef..00000000 Binary files a/content/notes/matlab-statistics/spread1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/spread2.png b/content/notes/matlab-statistics/spread2.png deleted file mode 100644 index f1fa8750..00000000 Binary files a/content/notes/matlab-statistics/spread2.png and /dev/null differ diff --git a/content/notes/matlab-statistics/spread3.png b/content/notes/matlab-statistics/spread3.png deleted file mode 100644 index 10729062..00000000 Binary files a/content/notes/matlab-statistics/spread3.png and /dev/null differ diff --git a/content/notes/matlab-statistics/spread4.png b/content/notes/matlab-statistics/spread4.png deleted file mode 100644 index b49fda81..00000000 Binary files a/content/notes/matlab-statistics/spread4.png and /dev/null differ diff --git a/content/notes/matlab-statistics/summary1.png b/content/notes/matlab-statistics/summary1.png deleted file mode 100644 index cea0add6..00000000 Binary files a/content/notes/matlab-statistics/summary1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/summary2.png b/content/notes/matlab-statistics/summary2.png deleted file mode 100644 index 70a08ba4..00000000 Binary files a/content/notes/matlab-statistics/summary2.png and /dev/null differ diff --git a/content/notes/matlab-statistics/summary3.png b/content/notes/matlab-statistics/summary3.png deleted file mode 100644 index 67653041..00000000 Binary files a/content/notes/matlab-statistics/summary3.png and /dev/null differ diff --git a/content/notes/matlab-statistics/summary4.png b/content/notes/matlab-statistics/summary4.png deleted file mode 100644 index 3784c5e4..00000000 Binary files a/content/notes/matlab-statistics/summary4.png and /dev/null differ diff --git a/content/notes/matlab-statistics/summary5.png b/content/notes/matlab-statistics/summary5.png deleted file mode 100644 index a8fb2afb..00000000 Binary files a/content/notes/matlab-statistics/summary5.png and /dev/null differ diff --git a/content/notes/matlab-statistics/summary6.png b/content/notes/matlab-statistics/summary6.png deleted file mode 100644 index 5f6f120b..00000000 Binary files a/content/notes/matlab-statistics/summary6.png and /dev/null differ diff --git a/content/notes/matlab-statistics/summary7.png b/content/notes/matlab-statistics/summary7.png deleted file mode 100644 index ec444cbc..00000000 Binary files a/content/notes/matlab-statistics/summary7.png and /dev/null differ diff --git a/content/notes/matlab-statistics/summary8.png b/content/notes/matlab-statistics/summary8.png deleted file mode 100644 index 2e527de7..00000000 Binary files a/content/notes/matlab-statistics/summary8.png and /dev/null differ diff --git a/content/courses/python-high-performance/.f2py_f2cmap b/content/notes/python-high-performance/.f2py_f2cmap similarity index 100% rename from content/courses/python-high-performance/.f2py_f2cmap rename to content/notes/python-high-performance/.f2py_f2cmap diff --git a/content/courses/python-high-performance/AnacondaNavigatorFastX.png b/content/notes/python-high-performance/AnacondaNavigatorFastX.png similarity index 100% rename from content/courses/python-high-performance/AnacondaNavigatorFastX.png rename to content/notes/python-high-performance/AnacondaNavigatorFastX.png diff --git a/content/courses/python-high-performance/DMP.png b/content/notes/python-high-performance/DMP.png similarity index 100% rename from content/courses/python-high-performance/DMP.png rename to content/notes/python-high-performance/DMP.png diff --git a/content/courses/python-high-performance/SMP.png b/content/notes/python-high-performance/SMP.png similarity index 100% rename from content/courses/python-high-performance/SMP.png rename to content/notes/python-high-performance/SMP.png diff --git a/content/courses/python-high-performance/_index.md b/content/notes/python-high-performance/_index.md similarity index 99% rename from content/courses/python-high-performance/_index.md rename to content/notes/python-high-performance/_index.md index 96827934..60806e31 100644 --- a/content/courses/python-high-performance/_index.md +++ b/content/notes/python-high-performance/_index.md @@ -5,10 +5,10 @@ toc: true date: "2020-11-17T00:00:00" tags: ["Programming","Python","HPC"] categories: ["Programming","Python","HPC"] +weight: 1 menu: hp-python: name: High-Performance Python - weight: 1 --- Python, like most interpreted languages, can be very slow. But there are best practices and some programming tricks that can speed it up considerably. This can make the difference between finishing the work in an acceptable time, or being unable to finish a project. diff --git a/content/courses/python-high-performance/codes/.ipynb_checkpoints/Untitled-checkpoint.ipynb b/content/notes/python-high-performance/codes/.ipynb_checkpoints/Untitled-checkpoint.ipynb similarity index 100% rename from content/courses/python-high-performance/codes/.ipynb_checkpoints/Untitled-checkpoint.ipynb rename to content/notes/python-high-performance/codes/.ipynb_checkpoints/Untitled-checkpoint.ipynb diff --git a/content/courses/python-high-performance/codes/.ipynb_checkpoints/dask_delayed-checkpoint.py b/content/notes/python-high-performance/codes/.ipynb_checkpoints/dask_delayed-checkpoint.py similarity index 100% rename from content/courses/python-high-performance/codes/.ipynb_checkpoints/dask_delayed-checkpoint.py rename to content/notes/python-high-performance/codes/.ipynb_checkpoints/dask_delayed-checkpoint.py diff --git a/content/courses/python-high-performance/codes/.virtual_documents/Untitled.ipynb b/content/notes/python-high-performance/codes/.virtual_documents/Untitled.ipynb similarity index 100% rename from content/courses/python-high-performance/codes/.virtual_documents/Untitled.ipynb rename to content/notes/python-high-performance/codes/.virtual_documents/Untitled.ipynb diff --git a/content/courses/python-high-performance/codes/MonteCarlo.py b/content/notes/python-high-performance/codes/MonteCarlo.py similarity index 100% rename from content/courses/python-high-performance/codes/MonteCarlo.py rename to content/notes/python-high-performance/codes/MonteCarlo.py diff --git a/content/courses/python-high-performance/codes/MonteCarloPi.py b/content/notes/python-high-performance/codes/MonteCarloPi.py similarity index 100% rename from content/courses/python-high-performance/codes/MonteCarloPi.py rename to content/notes/python-high-performance/codes/MonteCarloPi.py diff --git a/content/courses/python-high-performance/codes/MonteCarloPiMC.py b/content/notes/python-high-performance/codes/MonteCarloPiMC.py similarity index 100% rename from content/courses/python-high-performance/codes/MonteCarloPiMC.py rename to content/notes/python-high-performance/codes/MonteCarloPiMC.py diff --git a/content/courses/python-high-performance/codes/MonteCarloPiMPI.py b/content/notes/python-high-performance/codes/MonteCarloPiMPI.py similarity index 100% rename from content/courses/python-high-performance/codes/MonteCarloPiMPI.py rename to content/notes/python-high-performance/codes/MonteCarloPiMPI.py diff --git a/content/courses/python-high-performance/codes/Untitled.ipynb b/content/notes/python-high-performance/codes/Untitled.ipynb similarity index 100% rename from content/courses/python-high-performance/codes/Untitled.ipynb rename to content/notes/python-high-performance/codes/Untitled.ipynb diff --git a/content/notes/python-high-performance/codes/__pycache__/dask_mpi.cpython-39.pyc b/content/notes/python-high-performance/codes/__pycache__/dask_mpi.cpython-39.pyc new file mode 100644 index 00000000..341425c0 Binary files /dev/null and b/content/notes/python-high-performance/codes/__pycache__/dask_mpi.cpython-39.pyc differ diff --git a/content/notes/python-high-performance/codes/__pycache__/fibonacci.cpython-39.pyc b/content/notes/python-high-performance/codes/__pycache__/fibonacci.cpython-39.pyc new file mode 100644 index 00000000..baa22610 Binary files /dev/null and b/content/notes/python-high-performance/codes/__pycache__/fibonacci.cpython-39.pyc differ diff --git a/content/courses/python-high-performance/codes/aops.py b/content/notes/python-high-performance/codes/aops.py similarity index 100% rename from content/courses/python-high-performance/codes/aops.py rename to content/notes/python-high-performance/codes/aops.py diff --git a/content/courses/python-high-performance/codes/arith.c b/content/notes/python-high-performance/codes/arith.c similarity index 100% rename from content/courses/python-high-performance/codes/arith.c rename to content/notes/python-high-performance/codes/arith.c diff --git a/content/courses/python-high-performance/codes/arith.h b/content/notes/python-high-performance/codes/arith.h similarity index 100% rename from content/courses/python-high-performance/codes/arith.h rename to content/notes/python-high-performance/codes/arith.h diff --git a/content/courses/python-high-performance/codes/build/temp.linux-x86_64-3.9/integrate_cyf.o b/content/notes/python-high-performance/codes/build/temp.linux-x86_64-3.9/integrate_cyf.o similarity index 100% rename from content/courses/python-high-performance/codes/build/temp.linux-x86_64-3.9/integrate_cyf.o rename to content/notes/python-high-performance/codes/build/temp.linux-x86_64-3.9/integrate_cyf.o diff --git a/content/courses/python-high-performance/codes/build_arith.py b/content/notes/python-high-performance/codes/build_arith.py similarity index 100% rename from content/courses/python-high-performance/codes/build_arith.py rename to content/notes/python-high-performance/codes/build_arith.py diff --git a/content/courses/python-high-performance/codes/cupy_example.py b/content/notes/python-high-performance/codes/cupy_example.py similarity index 100% rename from content/courses/python-high-performance/codes/cupy_example.py rename to content/notes/python-high-performance/codes/cupy_example.py diff --git a/content/courses/python-high-performance/codes/da_mean.png b/content/notes/python-high-performance/codes/da_mean.png similarity index 100% rename from content/courses/python-high-performance/codes/da_mean.png rename to content/notes/python-high-performance/codes/da_mean.png diff --git a/content/courses/python-high-performance/codes/dask_delayed.py b/content/notes/python-high-performance/codes/dask_delayed.py similarity index 100% rename from content/courses/python-high-performance/codes/dask_delayed.py rename to content/notes/python-high-performance/codes/dask_delayed.py diff --git a/content/courses/python-high-performance/codes/dask_df_example.py b/content/notes/python-high-performance/codes/dask_df_example.py similarity index 100% rename from content/courses/python-high-performance/codes/dask_df_example.py rename to content/notes/python-high-performance/codes/dask_df_example.py diff --git a/content/courses/python-high-performance/codes/dask_df_mpi.py b/content/notes/python-high-performance/codes/dask_df_mpi.py similarity index 100% rename from content/courses/python-high-performance/codes/dask_df_mpi.py rename to content/notes/python-high-performance/codes/dask_df_mpi.py diff --git a/content/courses/python-high-performance/codes/dask_dist_example.py b/content/notes/python-high-performance/codes/dask_dist_example.py similarity index 100% rename from content/courses/python-high-performance/codes/dask_dist_example.py rename to content/notes/python-high-performance/codes/dask_dist_example.py diff --git a/content/courses/python-high-performance/codes/dask_mpi_example.py b/content/notes/python-high-performance/codes/dask_mpi_example.py similarity index 100% rename from content/courses/python-high-performance/codes/dask_mpi_example.py rename to content/notes/python-high-performance/codes/dask_mpi_example.py diff --git a/content/courses/python-high-performance/codes/dask_mpi_futures.py b/content/notes/python-high-performance/codes/dask_mpi_futures.py similarity index 100% rename from content/courses/python-high-performance/codes/dask_mpi_futures.py rename to content/notes/python-high-performance/codes/dask_mpi_futures.py diff --git a/content/courses/python-high-performance/codes/dask_mpi_init.py b/content/notes/python-high-performance/codes/dask_mpi_init.py similarity index 100% rename from content/courses/python-high-performance/codes/dask_mpi_init.py rename to content/notes/python-high-performance/codes/dask_mpi_init.py diff --git a/content/courses/python-high-performance/codes/dask_viz.py b/content/notes/python-high-performance/codes/dask_viz.py similarity index 100% rename from content/courses/python-high-performance/codes/dask_viz.py rename to content/notes/python-high-performance/codes/dask_viz.py diff --git a/content/courses/python-high-performance/codes/fibonacci-cProfile.py b/content/notes/python-high-performance/codes/fibonacci-cProfile.py similarity index 100% rename from content/courses/python-high-performance/codes/fibonacci-cProfile.py rename to content/notes/python-high-performance/codes/fibonacci-cProfile.py diff --git a/content/courses/python-high-performance/codes/fibonacci.py b/content/notes/python-high-performance/codes/fibonacci.py similarity index 100% rename from content/courses/python-high-performance/codes/fibonacci.py rename to content/notes/python-high-performance/codes/fibonacci.py diff --git a/content/courses/python-high-performance/codes/fibonacci.py.lprof b/content/notes/python-high-performance/codes/fibonacci.py.lprof similarity index 100% rename from content/courses/python-high-performance/codes/fibonacci.py.lprof rename to content/notes/python-high-performance/codes/fibonacci.py.lprof diff --git a/content/courses/python-high-performance/codes/fibonacci_direct.py b/content/notes/python-high-performance/codes/fibonacci_direct.py similarity index 100% rename from content/courses/python-high-performance/codes/fibonacci_direct.py rename to content/notes/python-high-performance/codes/fibonacci_direct.py diff --git a/content/courses/python-high-performance/codes/fractions.cxx b/content/notes/python-high-performance/codes/fractions.cxx similarity index 100% rename from content/courses/python-high-performance/codes/fractions.cxx rename to content/notes/python-high-performance/codes/fractions.cxx diff --git a/content/courses/python-high-performance/codes/fractions.f90 b/content/notes/python-high-performance/codes/fractions.f90 similarity index 100% rename from content/courses/python-high-performance/codes/fractions.f90 rename to content/notes/python-high-performance/codes/fractions.f90 diff --git a/content/courses/python-high-performance/codes/fractions.h b/content/notes/python-high-performance/codes/fractions.h similarity index 100% rename from content/courses/python-high-performance/codes/fractions.h rename to content/notes/python-high-performance/codes/fractions.h diff --git a/content/courses/python-high-performance/codes/groupby.png b/content/notes/python-high-performance/codes/groupby.png similarity index 100% rename from content/courses/python-high-performance/codes/groupby.png rename to content/notes/python-high-performance/codes/groupby.png diff --git a/content/courses/python-high-performance/codes/integrate.py b/content/notes/python-high-performance/codes/integrate.py similarity index 100% rename from content/courses/python-high-performance/codes/integrate.py rename to content/notes/python-high-performance/codes/integrate.py diff --git a/content/courses/python-high-performance/codes/integrate_cyf.c b/content/notes/python-high-performance/codes/integrate_cyf.c similarity index 100% rename from content/courses/python-high-performance/codes/integrate_cyf.c rename to content/notes/python-high-performance/codes/integrate_cyf.c diff --git a/content/courses/python-high-performance/codes/integrate_cyf.cpython-39-x86_64-linux-gnu.so b/content/notes/python-high-performance/codes/integrate_cyf.cpython-39-x86_64-linux-gnu.so similarity index 100% rename from content/courses/python-high-performance/codes/integrate_cyf.cpython-39-x86_64-linux-gnu.so rename to content/notes/python-high-performance/codes/integrate_cyf.cpython-39-x86_64-linux-gnu.so diff --git a/content/courses/python-high-performance/codes/integrate_cyf.pyx b/content/notes/python-high-performance/codes/integrate_cyf.pyx similarity index 100% rename from content/courses/python-high-performance/codes/integrate_cyf.pyx rename to content/notes/python-high-performance/codes/integrate_cyf.pyx diff --git a/content/courses/python-high-performance/codes/integrate_cyf_test.py b/content/notes/python-high-performance/codes/integrate_cyf_test.py similarity index 100% rename from content/courses/python-high-performance/codes/integrate_cyf_test.py rename to content/notes/python-high-performance/codes/integrate_cyf_test.py diff --git a/content/courses/python-high-performance/codes/loops.py b/content/notes/python-high-performance/codes/loops.py similarity index 100% rename from content/courses/python-high-performance/codes/loops.py rename to content/notes/python-high-performance/codes/loops.py diff --git a/content/courses/python-high-performance/codes/mppool.py b/content/notes/python-high-performance/codes/mppool.py similarity index 100% rename from content/courses/python-high-performance/codes/mppool.py rename to content/notes/python-high-performance/codes/mppool.py diff --git a/content/courses/python-high-performance/codes/mpprocess.py b/content/notes/python-high-performance/codes/mpprocess.py similarity index 100% rename from content/courses/python-high-performance/codes/mpprocess.py rename to content/notes/python-high-performance/codes/mpprocess.py diff --git a/content/courses/python-high-performance/codes/mpstarmap.py b/content/notes/python-high-performance/codes/mpstarmap.py similarity index 100% rename from content/courses/python-high-performance/codes/mpstarmap.py rename to content/notes/python-high-performance/codes/mpstarmap.py diff --git a/content/courses/python-high-performance/codes/numba_cuda_example.py b/content/notes/python-high-performance/codes/numba_cuda_example.py similarity index 100% rename from content/courses/python-high-performance/codes/numba_cuda_example.py rename to content/notes/python-high-performance/codes/numba_cuda_example.py diff --git a/content/courses/python-high-performance/codes/numba_vectorize.py b/content/notes/python-high-performance/codes/numba_vectorize.py similarity index 100% rename from content/courses/python-high-performance/codes/numba_vectorize.py rename to content/notes/python-high-performance/codes/numba_vectorize.py diff --git a/content/courses/python-high-performance/codes/numba_vectorize_todevice.py b/content/notes/python-high-performance/codes/numba_vectorize_todevice.py similarity index 100% rename from content/courses/python-high-performance/codes/numba_vectorize_todevice.py rename to content/notes/python-high-performance/codes/numba_vectorize_todevice.py diff --git a/content/courses/python-high-performance/codes/pimc.sh b/content/notes/python-high-performance/codes/pimc.sh similarity index 100% rename from content/courses/python-high-performance/codes/pimc.sh rename to content/notes/python-high-performance/codes/pimc.sh diff --git a/content/courses/python-high-performance/codes/plot_mp_scaling.py b/content/notes/python-high-performance/codes/plot_mp_scaling.py similarity index 100% rename from content/courses/python-high-performance/codes/plot_mp_scaling.py rename to content/notes/python-high-performance/codes/plot_mp_scaling.py diff --git a/content/courses/python-high-performance/codes/pycuda_example.py b/content/notes/python-high-performance/codes/pycuda_example.py similarity index 100% rename from content/courses/python-high-performance/codes/pycuda_example.py rename to content/notes/python-high-performance/codes/pycuda_example.py diff --git a/content/courses/python-high-performance/codes/pycuda_numba.py b/content/notes/python-high-performance/codes/pycuda_numba.py similarity index 100% rename from content/courses/python-high-performance/codes/pycuda_numba.py rename to content/notes/python-high-performance/codes/pycuda_numba.py diff --git a/content/courses/python-high-performance/codes/pympi.slurm b/content/notes/python-high-performance/codes/pympi.slurm similarity index 100% rename from content/courses/python-high-performance/codes/pympi.slurm rename to content/notes/python-high-performance/codes/pympi.slurm diff --git a/content/courses/python-high-performance/codes/replace_forloop.py b/content/notes/python-high-performance/codes/replace_forloop.py similarity index 100% rename from content/courses/python-high-performance/codes/replace_forloop.py rename to content/notes/python-high-performance/codes/replace_forloop.py diff --git a/content/courses/python-high-performance/codes/replace_forloop_comp.py b/content/notes/python-high-performance/codes/replace_forloop_comp.py similarity index 100% rename from content/courses/python-high-performance/codes/replace_forloop_comp.py rename to content/notes/python-high-performance/codes/replace_forloop_comp.py diff --git a/content/courses/python-high-performance/codes/run_dask_mpi.slurm b/content/notes/python-high-performance/codes/run_dask_mpi.slurm similarity index 100% rename from content/courses/python-high-performance/codes/run_dask_mpi.slurm rename to content/notes/python-high-performance/codes/run_dask_mpi.slurm diff --git a/content/courses/python-high-performance/codes/setup.py b/content/notes/python-high-performance/codes/setup.py similarity index 100% rename from content/courses/python-high-performance/codes/setup.py rename to content/notes/python-high-performance/codes/setup.py diff --git a/content/courses/python-high-performance/codes/tasks.py b/content/notes/python-high-performance/codes/tasks.py similarity index 100% rename from content/courses/python-high-performance/codes/tasks.py rename to content/notes/python-high-performance/codes/tasks.py diff --git a/content/courses/python-high-performance/codes/testdecorator.py b/content/notes/python-high-performance/codes/testdecorator.py similarity index 100% rename from content/courses/python-high-performance/codes/testdecorator.py rename to content/notes/python-high-performance/codes/testdecorator.py diff --git a/content/courses/python-high-performance/codes/wrap_fractions.cxx b/content/notes/python-high-performance/codes/wrap_fractions.cxx similarity index 100% rename from content/courses/python-high-performance/codes/wrap_fractions.cxx rename to content/notes/python-high-performance/codes/wrap_fractions.cxx diff --git a/content/courses/python-high-performance/compiled_code.md b/content/notes/python-high-performance/compiled_code.md similarity index 100% rename from content/courses/python-high-performance/compiled_code.md rename to content/notes/python-high-performance/compiled_code.md diff --git a/content/courses/python-high-performance/data.zip b/content/notes/python-high-performance/data.zip similarity index 100% rename from content/courses/python-high-performance/data.zip rename to content/notes/python-high-performance/data.zip diff --git a/content/courses/python-high-performance/data/0.json b/content/notes/python-high-performance/data/0.json similarity index 100% rename from content/courses/python-high-performance/data/0.json rename to content/notes/python-high-performance/data/0.json diff --git a/content/courses/python-high-performance/data/1.json b/content/notes/python-high-performance/data/1.json similarity index 100% rename from content/courses/python-high-performance/data/1.json rename to content/notes/python-high-performance/data/1.json diff --git a/content/courses/python-high-performance/data/2.json b/content/notes/python-high-performance/data/2.json similarity index 100% rename from content/courses/python-high-performance/data/2.json rename to content/notes/python-high-performance/data/2.json diff --git a/content/courses/python-high-performance/data/3.json b/content/notes/python-high-performance/data/3.json similarity index 100% rename from content/courses/python-high-performance/data/3.json rename to content/notes/python-high-performance/data/3.json diff --git a/content/courses/python-high-performance/data/4.json b/content/notes/python-high-performance/data/4.json similarity index 100% rename from content/courses/python-high-performance/data/4.json rename to content/notes/python-high-performance/data/4.json diff --git a/content/courses/python-high-performance/data/5.json b/content/notes/python-high-performance/data/5.json similarity index 100% rename from content/courses/python-high-performance/data/5.json rename to content/notes/python-high-performance/data/5.json diff --git a/content/courses/python-high-performance/data/6.json b/content/notes/python-high-performance/data/6.json similarity index 100% rename from content/courses/python-high-performance/data/6.json rename to content/notes/python-high-performance/data/6.json diff --git a/content/courses/python-high-performance/data/7.json b/content/notes/python-high-performance/data/7.json similarity index 100% rename from content/courses/python-high-performance/data/7.json rename to content/notes/python-high-performance/data/7.json diff --git a/content/courses/python-high-performance/data/8.json b/content/notes/python-high-performance/data/8.json similarity index 100% rename from content/courses/python-high-performance/data/8.json rename to content/notes/python-high-performance/data/8.json diff --git a/content/courses/python-high-performance/data/9.json b/content/notes/python-high-performance/data/9.json similarity index 100% rename from content/courses/python-high-performance/data/9.json rename to content/notes/python-high-performance/data/9.json diff --git a/content/courses/python-high-performance/distributed-parallelization.md b/content/notes/python-high-performance/distributed-parallelization.md similarity index 100% rename from content/courses/python-high-performance/distributed-parallelization.md rename to content/notes/python-high-performance/distributed-parallelization.md diff --git a/content/courses/python-high-performance/fibonacci-profiler.png b/content/notes/python-high-performance/fibonacci-profiler.png similarity index 100% rename from content/courses/python-high-performance/fibonacci-profiler.png rename to content/notes/python-high-performance/fibonacci-profiler.png diff --git a/content/courses/python-high-performance/gpu_acceleration.md b/content/notes/python-high-performance/gpu_acceleration.md similarity index 100% rename from content/courses/python-high-performance/gpu_acceleration.md rename to content/notes/python-high-performance/gpu_acceleration.md diff --git a/content/courses/python-high-performance/mp-scaling.png b/content/notes/python-high-performance/mp-scaling.png similarity index 100% rename from content/courses/python-high-performance/mp-scaling.png rename to content/notes/python-high-performance/mp-scaling.png diff --git a/content/courses/python-high-performance/mpi-parallelization.md b/content/notes/python-high-performance/mpi-parallelization.md similarity index 100% rename from content/courses/python-high-performance/mpi-parallelization.md rename to content/notes/python-high-performance/mpi-parallelization.md diff --git a/content/courses/python-high-performance/multiprocessing.md b/content/notes/python-high-performance/multiprocessing.md similarity index 100% rename from content/courses/python-high-performance/multiprocessing.md rename to content/notes/python-high-performance/multiprocessing.md diff --git a/content/courses/python-high-performance/profiling-timing.md b/content/notes/python-high-performance/profiling-timing.md similarity index 100% rename from content/courses/python-high-performance/profiling-timing.md rename to content/notes/python-high-performance/profiling-timing.md diff --git a/content/courses/python-high-performance/serial-optimization.md b/content/notes/python-high-performance/serial-optimization.md similarity index 94% rename from content/courses/python-high-performance/serial-optimization.md rename to content/notes/python-high-performance/serial-optimization.md index fa513084..427afd5f 100644 --- a/content/courses/python-high-performance/serial-optimization.md +++ b/content/notes/python-high-performance/serial-optimization.md @@ -11,7 +11,7 @@ menu: We can represent the optimization process with a flowchart: -{{< diagram >}} +{{< diagram alt="Flowchart of the optimization process: profile code, tune the slowest section, check for performance improvement, then repeat or try a different solution." >}} graph TD; A(Profile or time) --> B(Tune the slowest section); B --> C{Performance increase?}; @@ -20,7 +20,7 @@ C -- No --> E(Try a different solution); D --> A {{< /diagram >}} -There are many approaches to speeding up sections, some specific to Python and some more generic. In this section we will consider several possibilities. +There are many many approaches to speeding up sections, some specific to Python and some more generic. In this section we will consider several possibilities. ### Avoid for Loops diff --git a/content/courses/python-high-performance/testdecorator.py.prof b/content/notes/python-high-performance/testdecorator.py.prof similarity index 100% rename from content/courses/python-high-performance/testdecorator.py.prof rename to content/notes/python-high-performance/testdecorator.py.prof diff --git a/content/notes/python-machine-learning/_index.md b/content/notes/python-machine-learning/_index.md index 23dfae59..5091e17b 100644 --- a/content/notes/python-machine-learning/_index.md +++ b/content/notes/python-machine-learning/_index.md @@ -26,4 +26,4 @@ In this tutorial we will be covering the following topics: As mentioned above, example codes will be provided for respective topics. Prior experience with the Python programming language and some familiarity with machine learning concepts are helpful for this tutorial. Please download and unzip the following file to follow along on code activities. -{{< file-download file="notes/python-machine-learning/code/ML_with_Python.zip" text="ML_with_Python.zip" >}} +{{< file-download file="/notes/python-machine-learning/code/ML_with_Python.zip" text="ML_with_Python.zip" >}} diff --git a/content/courses/pytorch-hpc/_index.md b/content/notes/pytorch-hpc/_index.md similarity index 94% rename from content/courses/pytorch-hpc/_index.md rename to content/notes/pytorch-hpc/_index.md index b9501ae8..8b5e00c0 100644 --- a/content/courses/pytorch-hpc/_index.md +++ b/content/notes/pytorch-hpc/_index.md @@ -1,14 +1,15 @@ --- date : "2025-02-26T00:00:00-05:00" title : "Introduction to PyTorch for HPC" -summary: "An introduction to devlopment with PyTorch and HPC." -authors: [abd] +#summary: "An introduction to using PyTorch on an HPC system." categories: ["Deep Learning","Python","HPC","Machine Learning"] tags: [Deep_learning,Machine_learning,Python,HPC] toc: true -type: book +type: docs weight: 1 - +menu: + pytorch-hpc: + name: Pytorch for HPC --- ### Overview diff --git a/content/notes/pytorch-hpc/basics.md b/content/notes/pytorch-hpc/basics.md deleted file mode 100644 index e69de29b..00000000 diff --git a/content/courses/pytorch-hpc/gpu.md b/content/notes/pytorch-hpc/gpu.md similarity index 97% rename from content/courses/pytorch-hpc/gpu.md rename to content/notes/pytorch-hpc/gpu.md index e7472e64..dcc98248 100644 --- a/content/courses/pytorch-hpc/gpu.md +++ b/content/notes/pytorch-hpc/gpu.md @@ -2,8 +2,12 @@ date : "2025-02-26T00:00:00" title: "GPU Acceleration for Deep Learning" toc: true -type: book +type: docs + weight: 15 +menu: + pytorch-hpc: + parent: Pytorch for HPC --- @@ -93,4 +97,4 @@ Alternatively you can use the [SLURM Script Generator](https://www.rc.virginia.e For more information on SLURM visit: https://www.rc.virginia.edu/userinfo/hpc/software/pytorch/, https://www.rc.virginia.edu/userinfo/hpc/slurm/ -For information on multi-GPU use: https://pytorch.org/tutorials/beginner/ddp_series_multigpu.html \ No newline at end of file +For information on multi-GPU use: https://pytorch.org/tutorials/beginner/ddp_series_multigpu.html diff --git a/content/courses/pytorch-hpc/img/leakyrelu.png b/content/notes/pytorch-hpc/img/leakyrelu.png similarity index 100% rename from content/courses/pytorch-hpc/img/leakyrelu.png rename to content/notes/pytorch-hpc/img/leakyrelu.png diff --git a/content/courses/pytorch-hpc/img/nn.png b/content/notes/pytorch-hpc/img/nn.png similarity index 100% rename from content/courses/pytorch-hpc/img/nn.png rename to content/notes/pytorch-hpc/img/nn.png diff --git a/content/courses/pytorch-hpc/img/relu.png b/content/notes/pytorch-hpc/img/relu.png similarity index 100% rename from content/courses/pytorch-hpc/img/relu.png rename to content/notes/pytorch-hpc/img/relu.png diff --git a/content/courses/pytorch-hpc/img/sigmoid.png b/content/notes/pytorch-hpc/img/sigmoid.png similarity index 100% rename from content/courses/pytorch-hpc/img/sigmoid.png rename to content/notes/pytorch-hpc/img/sigmoid.png diff --git a/content/courses/pytorch-hpc/img/tanh.png b/content/notes/pytorch-hpc/img/tanh.png similarity index 100% rename from content/courses/pytorch-hpc/img/tanh.png rename to content/notes/pytorch-hpc/img/tanh.png diff --git a/content/notes/pytorch-hpc/index.md b/content/notes/pytorch-hpc/index.md deleted file mode 100644 index 8eb700bb..00000000 --- a/content/notes/pytorch-hpc/index.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -# Course title, summary, and position. -title: "Introduction to Pytorch for HPC" -authors: [abd] - -# Page metadata. -date: "2025-02-26T00:00:00" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. - -# Add menu entry to sidebar. -menu: - pytorch-hpc: - name: Introduction - weight: 5 ---- - - - - diff --git a/content/courses/pytorch-hpc/optimize.md b/content/notes/pytorch-hpc/optimize.md similarity index 98% rename from content/courses/pytorch-hpc/optimize.md rename to content/notes/pytorch-hpc/optimize.md index 9d2f5843..cc552acf 100644 --- a/content/courses/pytorch-hpc/optimize.md +++ b/content/notes/pytorch-hpc/optimize.md @@ -2,9 +2,11 @@ date : "2025-02-26T00:00:00" title: "Optimizing Neural Networks" toc: true -type: book +type: docs weight: 25 - +menu: + pytorch-hpc: + parent: Pytorch for HPC --- diff --git a/content/courses/pytorch-hpc/overview.md b/content/notes/pytorch-hpc/overview.md similarity index 99% rename from content/courses/pytorch-hpc/overview.md rename to content/notes/pytorch-hpc/overview.md index 60e66692..17fc4af2 100644 --- a/content/courses/pytorch-hpc/overview.md +++ b/content/notes/pytorch-hpc/overview.md @@ -2,9 +2,12 @@ date : "2025-02-26T00:00:00" title: "Introduction to PyTorch" toc: true -type: book +type: docs weight: 5 +menu: + pytorch-hpc: + parent: Pytorch for HPC --- diff --git a/content/courses/pytorch-hpc/project.md b/content/notes/pytorch-hpc/project.md similarity index 97% rename from content/courses/pytorch-hpc/project.md rename to content/notes/pytorch-hpc/project.md index c96fb1a7..54f5c3b1 100644 --- a/content/courses/pytorch-hpc/project.md +++ b/content/notes/pytorch-hpc/project.md @@ -2,11 +2,14 @@ date : "2025-02-26T00:00:00" title: "PyTorch Experimentation and Project Setup" toc: true -type: book +type: docs weight: 30 ---- +menu: + pytorch-hpc: + parent: Pytorch for HPC +--- ### **Structuring a PyTorch Project** A well-organized project structure makes debugging, collaboration, and experimentation easier. @@ -28,7 +31,6 @@ pytorch_project/ ├── slurm_train.sh # SLURM script for HPC training └── README.md # Documentation ``` ---- ### Managing Experiments with Logging Frameworks Tracking experiments is essential for understanding model performance. @@ -85,4 +87,3 @@ train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=32, shuffle valid_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=32, shuffle=False) test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=32, shuffle=False) ``` - diff --git a/content/courses/pytorch-hpc/simple_nn.md b/content/notes/pytorch-hpc/simple_nn.md similarity index 99% rename from content/courses/pytorch-hpc/simple_nn.md rename to content/notes/pytorch-hpc/simple_nn.md index 8f087e6c..418f1598 100644 --- a/content/courses/pytorch-hpc/simple_nn.md +++ b/content/notes/pytorch-hpc/simple_nn.md @@ -2,8 +2,11 @@ date : "2025-02-26T00:00:00" title: "Building a Simple Neural Network" toc: true -type: book +type: docs weight: 10 +menu: + pytorch-hpc: + parent: Pytorch for HPC --- diff --git a/content/courses/pytorch-hpc/transfer.md b/content/notes/pytorch-hpc/transfer.md similarity index 98% rename from content/courses/pytorch-hpc/transfer.md rename to content/notes/pytorch-hpc/transfer.md index 8c6341dc..66e5c052 100644 --- a/content/courses/pytorch-hpc/transfer.md +++ b/content/notes/pytorch-hpc/transfer.md @@ -2,9 +2,13 @@ date : "2025-02-26T00:00:00" title: "Transfer Learning" toc: true -type: book +type: docs weight: 20 +menu: + pytorch-hpc: + parent: Pytorch for HPC + --- ### **What is Transfer Learning?** diff --git a/content/notes/rio-intro/_index.md b/content/notes/rio-intro/_index.md index ad3d2b06..8b0f6f34 100644 --- a/content/notes/rio-intro/_index.md +++ b/content/notes/rio-intro/_index.md @@ -1,9 +1,10 @@ --- title: Intro to High Security HPC (Rio) +authors: [as, pbo, cmd] date: 2025-11-12T03:53:56Z -authors: [as, pbo, Camden Duy] -categories: [HPC] -tags: [HPC] +authors: [as, pbo, cmd] +categories: [HPC Intro] +tags: [HPC_Intro] type: docs weight: 1 diff --git a/content/notes/rivanna-alphafold/vmd/20-visualizing-structures.md b/content/notes/rivanna-alphafold/vmd/20-visualizing-structures.md deleted file mode 100644 index d7f2e746..00000000 --- a/content/notes/rivanna-alphafold/vmd/20-visualizing-structures.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -title: Visualizing 3D Structures -date: 2025-05-20T00:23:54Z -type: docs -weight: 1950 -menu: - rivanna-alphafold: - name: Visualizing 3D Structures ---- - -To view your predicted structure generated by AlphaFold in 3D, you can use **Visual Molecular Dynamics (VMD)**. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_40.png width=60% height=60% >}} - -Steps: -* Log onto Open On Demand ([https://rivanna-portal.hpc.virginia.edu/](https://rivanna-portal.hpc.virginia.edu/)) -* Choose the ‘Desktop’ interactive app and fill out the webform with your required resources. -* Click launch. -* Load VMD in the terminal: -```bash -module load vmd -vmd -``` -The VMD windows will launch. From there, you can open your `.pdb` file from the AlphaFold output directory to view the 3D model (colored by `pLDDT`). - -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_41.jpg width=65% height=65% caption="Loading your .pdb file (a)." >}} -{{< figure src=/notes/rivanna-alphafold/img/molecule-file-browser.png width=65% height=65% caption="Loading your .pdb file (b)." >}} -{{< figure src=/notes/rivanna-alphafold/img/vmd-part-3.png width=60% height=60% caption="Loading your .pdb file (c)." >}} -{{< figure src=/notes/rivanna-alphafold/img/vmd-part-4.png width=50% height=50% caption="Loading your .pdb file (d)." >}} - - - diff --git a/content/notes/unix-tutorial/_index.md b/content/notes/unix-tutorial/_index.md index 5718fd37..1c24b3b8 100644 --- a/content/notes/unix-tutorial/_index.md +++ b/content/notes/unix-tutorial/_index.md @@ -3,11 +3,11 @@ title: "UNIX Tutorials for Beginners" type: docs toc: true weight: 1 -date: 2023-12-11T00:00:00Z +date: "2023-12-11-14T00:00:00" +author: [kah] menu: unix-tutorials: - --- These tutorials are derived from the excellent tutorials from the University of Surrey, UK, with some minor modifications for our site. The originals can be found [here](http://www.ee.surrey.ac.uk/Teaching/Unix/). diff --git a/content/notes/unix-tutorial/unix_tutorial_1.md b/content/notes/unix-tutorial/unix_tutorial_1.md index 07b54115..3c2200ec 100644 --- a/content/notes/unix-tutorial/unix_tutorial_1.md +++ b/content/notes/unix-tutorial/unix_tutorial_1.md @@ -161,7 +161,7 @@ In newer Linux systems, most commands have a `--help` option ```bash % wc --help ``` -However, `man` sends output through a [pager](/notes/unix-tutorials/unix_tutorial_2), whereas --help prints directly to the console. +However, `man` sends output through a [pager](/notes/unix-tutorial/unix_tutorial_2/), whereas --help prints directly to the console. Another useful command, ``` diff --git a/content/tutorials/benchmark-parallel-programs/index.md b/content/tutorials/benchmark-parallel-programs/index.md index b1428f7c..0693abb0 100755 --- a/content/tutorials/benchmark-parallel-programs/index.md +++ b/content/tutorials/benchmark-parallel-programs/index.md @@ -8,5 +8,5 @@ summary: Learn how to benchmark your program for parallel SLURM jobs. notes: benchmark-parallel-programs -weight: 130 +weight: 830 --- diff --git a/content/tutorials/bioinfo-intro/index.md b/content/tutorials/bioinfo-intro/index.md index 7c3bd509..03c55a6a 100644 --- a/content/tutorials/bioinfo-intro/index.md +++ b/content/tutorials/bioinfo-intro/index.md @@ -6,6 +6,8 @@ categories: ["Bioinformatics"] tags: ["Bioinformatics"] summary: "An introduction to bioinformatics concepts, sequencing technologies, common analysis tools, and their use on high-performance computing systems." +weight: 505 + notes: bioinfo-intro ---- \ No newline at end of file +--- diff --git a/content/tutorials/bioinfo-tools-riv/index.md b/content/tutorials/bioinfo-tools-riv/index.md index 8fb185c5..b5a4cb6c 100755 --- a/content/tutorials/bioinfo-tools-riv/index.md +++ b/content/tutorials/bioinfo-tools-riv/index.md @@ -6,8 +6,10 @@ summary: "This tutorial is an introduction to using several popular bioinformati publishDate: "2021-03-02T00:00:00" authors: [gka] -tags: [HPC,Bioinformatics] -categories: ["HPC","Bioinformatics"] +tags: [Bioinformatics] +categories: ["Bioinformatics"] + +weight: 520 # Markdown Slides (optional). # Associate this talk with Markdown slides. @@ -17,7 +19,7 @@ categories: ["HPC","Bioinformatics"] url_slides: notes/bioinfo-tools-riv -weight: 500 +weight: 400 --- diff --git a/content/tutorials/biopython/index.md b/content/tutorials/biopython/index.md deleted file mode 100644 index 794f98d2..00000000 --- a/content/tutorials/biopython/index.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -title: "Biopython" - -summary: "An introduction to using the Biopython package to develop Python scripts for bioinformatics analysis." - -date: 2020-10-01T00:00:00-05:00 -authors: ["khs"] -tags: [Python,Bioinformatics] -categories: ["Python", "Bioinformatics"] -weight: 520 - -notes: biopython ---- -In this workshop participants will learn how to use the biopython package to develop Python scripts for bioinformatics analysis. Participants will learn how to download public datasets from online databases, parse bioinformatics file formats, and implement some sequence analysis workflows. diff --git a/content/tutorials/building-running-c-cpp-fortran/index.md b/content/tutorials/building-running-c-cpp-fortran/index.md index c2857f4e..7e097e2e 100755 --- a/content/tutorials/building-running-c-cpp-fortran/index.md +++ b/content/tutorials/building-running-c-cpp-fortran/index.md @@ -8,6 +8,6 @@ summary: This tutorial will teach you to compile, link, and run programs written notes: "building-running-c-cpp-fortran" -weight: 110 +weight: 800 --- diff --git a/content/tutorials/connecting-to-hpc/index.md b/content/tutorials/connecting-to-hpc/index.md index 57ef1e3e..8b92194b 100644 --- a/content/tutorials/connecting-to-hpc/index.md +++ b/content/tutorials/connecting-to-hpc/index.md @@ -2,14 +2,14 @@ title: "Connecting to HPC" date: 2020-02-28T00:00:00-05:00 authors: [kah,rs] -categories: ["Programming","HPC","Open OnDemand"] -tags: [Programming,HPC,Open_OnDemand] +categories: ["Programming","HPC Intro","Open OnDemand"] +tags: [Programming,HPC_Intro,Open_OnDemand] summary: This tutorial will teach you multiple methods for connecting UVA's HPC system in the web browser. notes: hpc-intro/connecting_to_the_system/connecting_logging_on -weight: 110 +weight: 100 --- diff --git a/content/tutorials/containers-for-hpc/index.md b/content/tutorials/containers-for-hpc/index.md new file mode 100755 index 00000000..74660f0c --- /dev/null +++ b/content/tutorials/containers-for-hpc/index.md @@ -0,0 +1,12 @@ +--- +title: "Building and Using Containers for HPC" +summary: "An introduction to using and building software containers for use on HPC systems." +date: 2020-10-29T00:00:00-05:00 +authors: [rs] +categories: ["HPC","Containers"] +tags: [HPC, Containers] + +notes: containers-for-hpc + +weight: 890 +--- diff --git a/content/tutorials/containers/index.md b/content/tutorials/containers/index.md deleted file mode 100755 index a7dbda15..00000000 --- a/content/tutorials/containers/index.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -title: "Introduction to Building and Deploying Containers" -summary: "" -authors: [cag] -date: 2023-05-12T00:00:00-05:00 -tags: ["Containers","R","Shiny","App_Development"] -categories: ["Containers","Docker", "R", "Shiny", "App Development"] -draft: false -toc: True - -notes: containers - -weight: 250 - ---- -In this workshop we will go over how to containerize our apps with Docker and maintain them with GitHub. diff --git a/content/tutorials/databases-intro/index.md b/content/tutorials/databases-intro/index.md deleted file mode 100644 index dc8f1c6a..00000000 --- a/content/tutorials/databases-intro/index.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -title: Introduction to Databases -summary: "This tutorial is an introduction to relational and NoSQL databases." - -# Schedule page publish date (NOT talk date). -publishDate: "2024-04-01T00:00:00" - -categories: ["Databases"] -tags: ["Databases"] - -notes: databases-intro - -weight: 200 - ---- diff --git a/content/tutorials/deep-learning-distributed/index.md b/content/tutorials/deep-learning-distributed/index.md index c3c03fdc..5502c77a 100644 --- a/content/tutorials/deep-learning-distributed/index.md +++ b/content/tutorials/deep-learning-distributed/index.md @@ -5,7 +5,8 @@ date: "2024-06-06T00:00:00" authors: [as,jmh] categories: ["GPU","Machine Learning","Python","Deep Learning"] tags: ["GPU","Machine_Learning","Python","Deep_Learning"] -weight: 300 + +weight: 320 notes: deep-learning-distributed url_code: notes/deep-learning-distributed/code/distributed_dl.zip diff --git a/content/tutorials/deep-learning-hpc/index.md b/content/tutorials/deep-learning-hpc/index.md index da28d7b3..7ea791b0 100755 --- a/content/tutorials/deep-learning-hpc/index.md +++ b/content/tutorials/deep-learning-hpc/index.md @@ -1,12 +1,14 @@ --- title: "Introduction to Deep Learning Using HPC" -summary: "This tutorial provides an introduction to deep learning using high power computing. This workshop will cover accessing deep learning containers, resource allocation and helpful tools, how to choose a GPU, and deep learning slurm scripts." +summary: "This tutorial provides an introduction to deep learning using high performance computing. This workshop will cover accessing deep learning containers, resource allocation and helpful tools, how to choose a GPU, and deep learning slurm scripts." date: 2024-06-27T21:13:14-05:00 -authors: [uvarc] +authors: [abd] categories: ["GPU","Machine Learning","Deep Learning"] tags: ["GPU","Machine_Learning","Deep_Learning"] +weight: 310 + notes: deep-learning-hpc --- diff --git a/content/tutorials/dl-drug-discovery/index.md b/content/tutorials/dl-drug-discovery/index.md index dfa3c97b..f9561be0 100644 --- a/content/tutorials/dl-drug-discovery/index.md +++ b/content/tutorials/dl-drug-discovery/index.md @@ -2,10 +2,12 @@ title: "Deep Learning in Drug Discovery" date: "2025-05-14T00:00:00" authors: [ppr] -categories: ["HPC","Deep learning"] -tags: ["HPC","Deep learning"] +categories: [Deep learning"] +tags: ["Deep_learning"] summary: "This tutorial introduces the deep learning approach to computational drug discovery." +weight: 590 + notes: dl-drug-discovery ---- \ No newline at end of file +--- diff --git a/content/tutorials/genomics/index.md b/content/tutorials/genomics/index.md index d0ef56dd..c5d86db1 100644 --- a/content/tutorials/genomics/index.md +++ b/content/tutorials/genomics/index.md @@ -5,12 +5,12 @@ summary: "This tutorial introduces tools and resources for performing genome ass date: "2025-07-10T00:00:00" authors: [dat, mab] -tags: ["Bioinformatics", "Data_Analytics", "HPC"] -categories: ["Bioinformatics","Data Analytics", "HPC"] +tags: ["Bioinformatics", "Data_Analytics"] +categories: ["Bioinformatics","Data Analytics"] -notes: genomics +weight: 530 -weight: 100 +notes: genomics --- diff --git a/content/tutorials/git-intro/index.md b/content/tutorials/git-intro/index.md index b5ab887d..32e7cec7 100755 --- a/content/tutorials/git-intro/index.md +++ b/content/tutorials/git-intro/index.md @@ -4,10 +4,12 @@ summary: "Learn to use the git version-control system to manage your software pr Date: "2015-07-23T21:13:14-05:00" -authors: [uvarc] +authors: [kah] tags: [Programming,Git] categories: ["Programming","Git"] +weight: 810 + # Markdown Slides (optional). # Associate this talk with Markdown slides. # Simply enter your slide deck's filename without extension. @@ -16,7 +18,5 @@ categories: ["Programming","Git"] #url_slides: slides/git-intro/git-slides.html notes: git-intro -weight: 100 - --- diff --git a/content/tutorials/globus-data-transfer/index.md b/content/tutorials/globus-data-transfer/index.md index 53560fdd..b7a8505f 100644 --- a/content/tutorials/globus-data-transfer/index.md +++ b/content/tutorials/globus-data-transfer/index.md @@ -8,5 +8,6 @@ date : 2024-02-10T00:00:00-05:00 notes: globus-data-transfer url_video: https://youtu.be/bMcSqbDiFuo -weight: 50 + +weight: 150 --- diff --git a/content/tutorials/gpu-applications-rivanna/index.md b/content/tutorials/gpu-applications-rivanna/index.md index de0cc865..9b2a2189 100755 --- a/content/tutorials/gpu-applications-rivanna/index.md +++ b/content/tutorials/gpu-applications-rivanna/index.md @@ -8,7 +8,7 @@ tags: ["GPU", "HPC"] notes: gpu-applications-rivanna -weight: 160 +weight: 810 --- diff --git a/content/tutorials/hpc-alphafold/index.md b/content/tutorials/hpc-alphafold/index.md new file mode 100644 index 00000000..94cd33ac --- /dev/null +++ b/content/tutorials/hpc-alphafold/index.md @@ -0,0 +1,16 @@ +--- +title: "AlphaFold on HPC" +date: "2025-05-20" +authors: [mab] +categories: ["GPU","HPC","Bioinformatics"] +tags: ["GPU","HPC","Bioinformatics"] +summary: "This tutorial introduces the basics of GPU computing and demonstrates how to run AlphaFold on the HPC cluster to predict protein structures." + +weight: 550 + +notes: hpc-alphafold + +#url_code: + +--- + diff --git a/content/tutorials/hpc-best-practices/index.md b/content/tutorials/hpc-best-practices/index.md index 7ffa640c..44bb71fd 100644 --- a/content/tutorials/hpc-best-practices/index.md +++ b/content/tutorials/hpc-best-practices/index.md @@ -6,6 +6,8 @@ authors: [teh] tags: ["HPC","Unix"] summary: "This tutorial introduces various techniques and strategies to help users more efficiently use UVA's HPC System." +weight: 250 + notes: hpc-best-practices --- diff --git a/content/tutorials/hpc-from-terminal/index.md b/content/tutorials/hpc-from-terminal/index.md index 447d01d4..3e722f14 100644 --- a/content/tutorials/hpc-from-terminal/index.md +++ b/content/tutorials/hpc-from-terminal/index.md @@ -1,16 +1,16 @@ --- title: Using UVA's HPC System from the Terminal -summary: "This tutorial is an introduction to using the Unix command line on UVA's HPC System." +summary: "This tutorial is an introduction to using the Unix command line on UVA's HPC System, including several commands specific to this environment." # Schedule page publish date (NOT talk date). publishDate: "2023-01-04T00:00:00" authors: [as,gka,kah] -tags: ["HPC","Unix"] -categories: ["HPC","Unix"] +tags: ["HPC_Intro","Unix"] +categories: ["HPC Intro","Unix"] notes: hpc-from-terminal url_slides: slides/hpc-from-terminal/HPC_From_Terminal.pptx -weight: 20 +weight: 210 --- diff --git a/content/tutorials/hpc-intro/index.md b/content/tutorials/hpc-intro/index.md index 567d5d97..d30ab5ec 100755 --- a/content/tutorials/hpc-intro/index.md +++ b/content/tutorials/hpc-intro/index.md @@ -6,8 +6,8 @@ summary: "This tutorial is an introduction to using UVA's high-performance compu publishDate: "2021-04-13T00:00:00" authors: [jmh,kah,pbo] -tags: [HPC] -categories: ["HPC"] +tags: [HPC_Intro] +categories: ["HPC Intro"] # Markdown Slides (optional). # Associate this talk with Markdown slides. @@ -19,4 +19,4 @@ notes: hpc-intro url_video: https://www.youtube.com/playlist?list=PLT4bryHgBcRP7N-hB9u6EWs6tq_2nMoRO weight: 10 ---- \ No newline at end of file +--- diff --git a/content/tutorials/interactive-apps-ood/index.md b/content/tutorials/interactive-apps-ood/index.md index b073a0bc..a2b268d2 100644 --- a/content/tutorials/interactive-apps-ood/index.md +++ b/content/tutorials/interactive-apps-ood/index.md @@ -1,15 +1,14 @@ --- title: "Interactive Apps with Open OnDemand" date: 2020-02-28T00:00:00-05:00 -authors: [kah,rs] -categories: ["HPC","Open OnDemand"] -tags: [HPC,Open_OnDemand] -summary: This tutorial will introduce several interactive apps built-into Open OnDemand. - +authors: [kah, rs] +categories: ["HPC Intro","Open OnDemand"] +tags: [HPC_Intro, Open_OnDemand] +summary: This tutorial will introduce several interactive apps built in to Open OnDemand. notes: hpc-intro/interactive_apps/interactive -weight: 110 +weight: 120 --- diff --git a/content/tutorials/llms-hpc/index.md b/content/tutorials/llms-hpc/index.md index 9e3765db..b95e3a68 100644 --- a/content/tutorials/llms-hpc/index.md +++ b/content/tutorials/llms-hpc/index.md @@ -6,6 +6,8 @@ categories: ["HPC", "Machine learning"] tags: ["HPC"] summary: "This tutorial is an introduction to running large language models on UVA's HPC system." +weight: 350 + notes: llms-hpc --- diff --git a/content/tutorials/matlab-data-science/index.md b/content/tutorials/matlab-data-science/index.md index 1bd3b993..a029d823 100755 --- a/content/tutorials/matlab-data-science/index.md +++ b/content/tutorials/matlab-data-science/index.md @@ -6,7 +6,8 @@ date: 2020-09-08T21:13:14-05:00 authors: [teh] categories: ["MATLAB","Machine Learning","Data Analytics","Deep Learning"] tags: ["MATLAB","Machine_Learning","Data_Analytics","Deep_Learning"] -weight: 240 + +weight: 650 notes: matlab-data-science diff --git a/content/tutorials/matlab-data-viz/index.md b/content/tutorials/matlab-data-viz/index.md index e14030f1..4b17df77 100755 --- a/content/tutorials/matlab-data-viz/index.md +++ b/content/tutorials/matlab-data-viz/index.md @@ -5,7 +5,8 @@ authors: [cag] date: 2019-10-16T21:13:14-05:00 categories: ["MATLAB","Data Analytics"] tags: ["MATLAB","Data_Analytics"] -weight: 250 + +weight: 620 notes: matlab-data-visualization diff --git a/content/tutorials/matlab-deep-learning/index.md b/content/tutorials/matlab-deep-learning/index.md index fbf4acdc..c953da3d 100755 --- a/content/tutorials/matlab-deep-learning/index.md +++ b/content/tutorials/matlab-deep-learning/index.md @@ -5,7 +5,8 @@ date: 2019-10-16T21:13:14-05:00 authors: [cag] categories: ["MATLAB","Machine Learning","Deep Learning"] tags: ["MATLAB","Machine_Learning","Deep_Learning"] -weight: 350 + +weight: 660 notes: matlab-deep-learning diff --git a/content/tutorials/matlab-image-processing/index.md b/content/tutorials/matlab-image-processing/index.md index 4ad28b5c..54ed6fd0 100644 --- a/content/tutorials/matlab-image-processing/index.md +++ b/content/tutorials/matlab-image-processing/index.md @@ -6,7 +6,8 @@ date: 2020-03-03T21:13:14-05:00 categories: ["Image Processing","MATLAB"] tags: ["Image_Processing","MATLAB"] draft: false -weight: 220 + +weight: 610 notes: matlab-image-processing diff --git a/content/tutorials/matlab-optimization/index.md b/content/tutorials/matlab-optimization/index.md index 270920bc..e25e64cb 100644 --- a/content/tutorials/matlab-optimization/index.md +++ b/content/tutorials/matlab-optimization/index.md @@ -5,7 +5,9 @@ tags : [ "MATLAB" ] title : "Optimization Techniques in MATLAB" summary : "This workshop introduces applied optimization in the MATLAB environment, focusing on using the Optimization Toolbox™ and the Global Optimization Toolbox™." authors : [teh] -weight: 260 + +weight: 630 + notes: matlab-optimization --- diff --git a/content/tutorials/matlab-parallel/index.md b/content/tutorials/matlab-parallel/index.md index dc0b3698..c6aeb3b0 100755 --- a/content/tutorials/matlab-parallel/index.md +++ b/content/tutorials/matlab-parallel/index.md @@ -3,9 +3,10 @@ title: "Parallel Computing with MATLAB" summary: "In this hands-on workshop, you will be introduced to parallel and distributed computing in MATLAB™ for speeding up your application and offloading work." authors: [teh] date: "2019-06-23T08:37:46-05:00" -categories: [ "HPC", "Parallel Computing", "MATLAB" ] +categories: [ "HPC", "Parallel Programming", "MATLAB" ] tags: [ "HPC", "Parallel_Programming", "MATLAB" ] -weight: 280 + +weight: 680 url_slides: slides/matlab-parallel/Parallel_Computing_with_MATLAB.pptx url_code: scripts/matlab-parallel/Parallel-Computing-Workshop.zip diff --git a/content/tutorials/matlab-statistics/index.md b/content/tutorials/matlab-statistics/index.md deleted file mode 100644 index 9ddf1b06..00000000 --- a/content/tutorials/matlab-statistics/index.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -title : "Statistical Methods with MATLAB" -summary : "Using statistical methods with MATLAB™" -date : "2019-06-23T08:37:46-05:00" -authors : [uvarc] -categories: ["Matlab","Data Analytics"] -tags: ["Matlab","Data_Analytics"] -weight: 230 - -notes: matlab-statistics ---- - -This workshop provides hands-on experience for performing statistical data analysis with MATLAB® and the Statistics and Machine Learning Toolbox™. Examples and exercises demonstrate the use of appropriate MATLAB and Statistics and Machine Learning Toolbox functionality throughout the analysis process; from exploratory analysis, to confirmatory analysis and simulation. Topics include: - - - Calculating summary statistics - - Visualizing data - - Fitting distributions - - Performing tests of significance - - Fitting regression models - - Interpolating data - - Generating random numbers - -Participants must have a Mathworks account in order to access their online training materials. - diff --git a/content/tutorials/multigpu-inference/index.md b/content/tutorials/multigpu-inference/index.md index 1cc49f84..f208d700 100644 --- a/content/tutorials/multigpu-inference/index.md +++ b/content/tutorials/multigpu-inference/index.md @@ -2,10 +2,12 @@ title: "Multi-GPU LLM Inference" date: "2025-02-23T00:00:00" authors: [kal] -categories: ["HPC", "Machine learning"] -tags: ["HPC"] +categories: ["GPU", "HPC", "Machine learning"] +tags: ["GPU", "HPC"] summary: "This tutorial is an introduction to multi-GPU strategies for large language model (LLM) inference using tools like Accelerate, DeepSpeed, and vLLM." +weight: 370 + notes: multigpu-inference --- diff --git a/content/tutorials/omero-hands-on/index.md b/content/tutorials/omero-hands-on/index.md deleted file mode 100755 index e1ba3daa..00000000 --- a/content/tutorials/omero-hands-on/index.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -title: "OMERO Hands-On Tutorial" -summary: "Getting started with the OMERO image database." -authors: [cag] -date: "2019-10-10T21:13:14-05:00" -categories: ["Image Processing","Omero"] -tags: ["Image_Processing","Omero"] -draft: true - -notes: omero-hands-on -url_dataset: data/omero-workshop-images.zip -url_code: https://github.com/ksiller/omero - - ---- - -{{< figure src="/img/ome-main-nav.svg" >}} - -OMERO is image management software that allows you to organize, view, annotate, analyze, and share your data from a single centralized repository. - -"This hands-on tutorial will show you how to import images into the OMERO database, organize and tag images, and how to use OMERO with third-party software such as Fiji/ImageJ, MATLAB™, and Python." - -Download the sample dataset and scripts so you can follow along with the -examples. diff --git a/content/tutorials/python-high-performance/index.md b/content/tutorials/python-high-performance/index.md index 7ccee73c..404d3050 100644 --- a/content/tutorials/python-high-performance/index.md +++ b/content/tutorials/python-high-performance/index.md @@ -1,17 +1,13 @@ --- title: "High Performance Programming in Python" -summary: Tips and tricks for improving the performance of your Python code. -# authors: [kah,khs] +summary: "Like most interpreted languages, Python can be slow. This tutorial will describe tips and tricks to help speed it up." date: 2020-02-02T00:06:47-04:00 -categories: ["Python", "Programming", "HPC"] -tags: ["Python","Programming","HPC"] -authors: ["kah"] -draft: true +categories: ["Python", "Programming", "Parallel Programming", "HPC"] +tags: ["Python","Programming", "Parallel_Programming","HPC"] +authors: ["khs","kah"] -url_notes: courses/python-high-performance +weight: 840 ---- - -My program takes forever! Can I do something about it? Like most interpreted languages, Python can be slow. Loops are a particular problem, but there are other pitfalls that may affect the performance of your code. This workshop will show you tips and tricks to speed up your code. +notes: python-high-performance -Participants should have experience programming in Python. +--- diff --git a/content/tutorials/python-machine-learning/index.md b/content/tutorials/python-machine-learning/index.md index e40d0bf0..91050223 100644 --- a/content/tutorials/python-machine-learning/index.md +++ b/content/tutorials/python-machine-learning/index.md @@ -6,11 +6,13 @@ date : "2022-06-09T00:00:00" categories : ["Machine Learning", "Python"] tags : ["Machine_Learning","Python"] +weight: 320 + notes: python-machine-learning url_code: notes/python-machine-learning/code/ML_with_Python.zip --- -This tutorial wil go over Machine Learning concepts using Python discussing the following topics: +This tutorial wil go over Machine Learning concepts using Python. It discusses the following topics: * Overview of Machine Learning * Decision Trees * Coding Decision Trees diff --git a/content/tutorials/pytorch-hpc/index.md b/content/tutorials/pytorch-hpc/index.md new file mode 100755 index 00000000..e99d4e1e --- /dev/null +++ b/content/tutorials/pytorch-hpc/index.md @@ -0,0 +1,13 @@ +--- +title: "Introduction to PyTorch on HPC" +summary: "This tutorial provides a practical introduction to building artificial neural networks using PyTorch, a powerful and flexible deep learning framework. The course covers the fundamentals of PyTorch, including tensors, automatic differentiation, and model building." + +date: 2024-06-27T21:13:14-05:00 +authors: [abd, bmr] +categories: ["GPU","Machine Learning","Deep Learning"] +tags: ["GPU","Machine_Learning","Deep_Learning"] + +weight: 330 + +notes: pytorch-hpc +--- diff --git a/content/tutorials/qiime2/index.md b/content/tutorials/qiime2/index.md index 6c8f945c..751fb173 100644 --- a/content/tutorials/qiime2/index.md +++ b/content/tutorials/qiime2/index.md @@ -1,11 +1,13 @@ --- title: "Introduction to QIIME 2" date: "2025-06-10T00:00:00" -authors: [] +authors: [gka] categories: ["Bioinformatics", "Data Analytics"] tags: ["Bioinformatics", "Data_Analytics"] summary: "This tutorial is an introduction to using QIIME 2 for micribiome analysis." +weight: 580 + notes: qiime2 --- diff --git a/content/tutorials/rapids/index.md b/content/tutorials/rapids/index.md index f248ce85..56b436e8 100755 --- a/content/tutorials/rapids/index.md +++ b/content/tutorials/rapids/index.md @@ -5,7 +5,8 @@ authors: [rs] date: 2020-10-13T00:00:00-05:00 categories: ["Python","GPU","HPC"] tags: ["Python","GPU","HPC"] -weight: 180 + +weight: 770 notes: rapids diff --git a/content/tutorials/rio-intro/index.md b/content/tutorials/rio-intro/index.md index 1e399841..45129e11 100644 --- a/content/tutorials/rio-intro/index.md +++ b/content/tutorials/rio-intro/index.md @@ -5,9 +5,9 @@ summary: "This tutorial is an introduction to using the High Security HPC platfo # Schedule page publish date (NOT talk date). publishDate: "2025-01-30T00:00:00" -authors: [as, pbo, Camden Duy] -tags: [HPC] -categories: ["HPC"] +authors: [as, pbo, cmd ] +tags: [HPC_Intro] +categories: ["HPC Intro"] # Markdown Slides (optional). # Associate this talk with Markdown slides. diff --git a/content/tutorials/rivanna-alphafold/index.md b/content/tutorials/rivanna-alphafold/index.md deleted file mode 100644 index 2de11e89..00000000 --- a/content/tutorials/rivanna-alphafold/index.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -title: "AlphaFold on Rivanna" -date: "2025-05-20" -authors: [mab] -categories: "" -tags: ["Bioinformatics"] -summary: "This tutorial introduces the basics of GPU computing and demonstrates how to run AlphaFold on the Rivanna cluster to predict protein structures." - -notes: rivanna-alphafold -url_code: - ---- - diff --git a/content/tutorials/seurat-bioinformatics/index.md b/content/tutorials/seurat-bioinformatics/index.md index d248bc79..b91bfc87 100644 --- a/content/tutorials/seurat-bioinformatics/index.md +++ b/content/tutorials/seurat-bioinformatics/index.md @@ -2,10 +2,12 @@ title: "Bioinformatics Data Visualization with Seurat" date: "2025-08-18T00:00:00" authors: [gka, mab] -categories: ["HPC", "Bioinformatics"] +categories: ["Bioinformatics"] tags: ["Bioinformatics"] summary: "This tutorial is a step-by-step walkthrough of the Seurat workflow for single-cell RNA-seq analysis." +weight: 560 + notes: seurat-bioinformatics --- diff --git a/content/tutorials/slurm-from-cli/index.md b/content/tutorials/slurm-from-cli/index.md index d9974fe7..642c5edb 100644 --- a/content/tutorials/slurm-from-cli/index.md +++ b/content/tutorials/slurm-from-cli/index.md @@ -6,11 +6,11 @@ summary: "This tutorial is an introduction to submitting Slurm jobs through a te publishDate: "2023-01-04T00:00:00Z" authors: [as,kah] -tags: ["HPC","Slurm"] -categories: ["HPC","Slurm"] +tags: ["HPC_Intro","Slurm"] +categories: ["HPC Intro","Slurm"] notes: slurm-from-cli -weight: 30 +weight: 230 --- diff --git a/content/tutorials/unix-tutorial/index.md b/content/tutorials/unix-tutorial/index.md index 5d26efc8..c0d6befa 100755 --- a/content/tutorials/unix-tutorial/index.md +++ b/content/tutorials/unix-tutorial/index.md @@ -1,11 +1,12 @@ --- title: Unix Tutorials for Beginners summary: "This set of tutorials is an introduction to using the Unix operating system from the command line." +authors: [ kah ] # Schedule page publish date (NOT talk date). publishDate: "2017-01-01T00:00:00" -categories: ["Unix","HPC"] -tags: ["Unix","HPC"] +categories: ["Unix"] +tags: ["Unix"] # Markdown Slides (optional). # Associate this talk with Markdown slides. @@ -14,7 +15,7 @@ tags: ["Unix","HPC"] # Otherwise, set `slides = ""`. notes: unix-tutorial -weight: 40 +weight: 190 --- diff --git a/content/tutorials/using-the-shell/index.md b/content/tutorials/using-the-shell/index.md index 70a3767a..f28ff641 100644 --- a/content/tutorials/using-the-shell/index.md +++ b/content/tutorials/using-the-shell/index.md @@ -1,15 +1,15 @@ --- -title: "Using the Shell on Open OnDemand" +title: "Using the Shell Through Open OnDemand" date: 2020-02-28T00:00:00-05:00 authors: [kah,rs] -categories: ["Programming","HPC", "Unix","Open OnDemand"] -tags: [Programming,HPC,Unix,Open_OnDemand] -summary: This tutorial will explain how to access the shell on UVA HPC, as well as several useful shell commands for working with HPC. +categories: ["HPC Intro", "Unix","Open OnDemand"] +tags: [HPC_Intro,Unix,Open_OnDemand] +summary: This tutorial will demonstrate how to access the shell on UVA HPC through the Open OnDemand interface, and will teach several useful shell commands for working with HPC. notes: hpc-intro/features_of_ood/features_shell_access -weight: 110 +weight: 180 --- diff --git a/content/tutorials/vscode-intro/index.md b/content/tutorials/vscode-intro/index.md index 02d768a2..11269616 100644 --- a/content/tutorials/vscode-intro/index.md +++ b/content/tutorials/vscode-intro/index.md @@ -1,13 +1,15 @@ --- title: Introduction to VS Code for HPC -summary: "This tutorial is an introduction to using VS Code for HPC" +summary: "Using VS Code on a remote HPC system may be different from local use. This tutorial introduces best practices for VS Code on HPC." # Schedule page publish date (NOT talk date). publishDate: "2025-01-30T00:00:00" -authors: [Hana Parece] -tags: [HPC] -categories: ["HPC"] +authors: [hp] +tags: [HPC, Programming] +categories: ["HPC", "Programming"] + +weight: 240 # Markdown Slides (optional). # Associate this talk with Markdown slides. @@ -18,6 +20,4 @@ categories: ["HPC"] pdf: vscode-intro -weight: 12 - --- diff --git a/content/tutorials/working-with-files/index.md b/content/tutorials/working-with-files/index.md index 65ccbf71..d3e5e6b2 100644 --- a/content/tutorials/working-with-files/index.md +++ b/content/tutorials/working-with-files/index.md @@ -2,14 +2,14 @@ title: "Working with Files" date: 2020-02-28T00:00:00-05:00 authors: [kah,rs] -categories: ["HPC", "Data Transfer","Open OnDemand"] -tags: [HPC,Data_transfer,Open_OnDemand] +categories: ["HPC Intro", "Data Transfer","Open OnDemand"] +tags: [HPC_Intro ,Data_transfer,Open_OnDemand] summary: This tutorial will teach you how to work with files on UVA HPC through your browser, including creating, deleting, editing, and more. notes: hpc-intro/features_of_ood/features_files_tab -weight: 110 +weight: 130 --- diff --git a/content/tutorials/working-with-jobs-ood/index.md b/content/tutorials/working-with-jobs-ood/index.md index d8981e39..c8692fbe 100644 --- a/content/tutorials/working-with-jobs-ood/index.md +++ b/content/tutorials/working-with-jobs-ood/index.md @@ -2,14 +2,14 @@ title: "Working with Jobs in Open OnDemand" date: 2020-02-28T00:00:00-05:00 authors: [kah,rs] -categories: ["HPC","Open OnDemand"] -tags: [HPC,Open_OnDemand] -summary: This tutorial will explain how to view and create jobs on UVA HPC through Open OnDemand. +categories: ["HPC Intro","Open OnDemand","Slurm"] +tags: [HPC_Intro,Open_OnDemand,Slurm] +summary: This tutorial will explain how to view and create Slurm jobs on UVA HPC through Open OnDemand. notes: hpc-intro/features_of_ood/features_jobs_tab -weight: 110 +weight: 140 --- diff --git a/layouts/partials/docs_sidebar.html b/layouts/partials/docs_sidebar.html index 57eb7ca3..bfa2b128 100644 --- a/layouts/partials/docs_sidebar.html +++ b/layouts/partials/docs_sidebar.html @@ -21,7 +21,7 @@ {{ if eq site.Params.search.engine 1 }} - + {{ end }} diff --git a/layouts/partials/search.html b/layouts/partials/search.html index 5f850a84..0ad30758 100644 --- a/layouts/partials/search.html +++ b/layouts/partials/search.html @@ -1,18 +1,24 @@ -
- -When we try going to [localhost:8888](localhost:8888), the app isn't running yet. We had a command at the end of our Dockerfile to start up Shiny server and run our app, but we just replaced that with `/bin/bash`. So within our new container, let's run `/usr/bin/shiny-server.sh`. - -Now we see that error message again when we reload the page in the browser. Hit **Ctrl + C** to stop running the app. - -Navigate to `/var/log/shiny-server`. Here you will see log files--use the `cat` command to take a look at them. In this example, we are missing the `fresh` package from our Docker image. We need to add that to our Dockerfile and rebuild the image. diff --git a/content/notes/containers/shinyapp-dockerbuild.md b/content/notes/containers/shinyapp-dockerbuild.md deleted file mode 100755 index cc2127fc..00000000 --- a/content/notes/containers/shinyapp-dockerbuild.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -title: Building the App Image -date: "2023-05-01T00:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 220 -menu: - containers: - parent: Serving a ShinyApp ---- - -Now that we've written the Dockerfile, it's time to build our image! To do that, we will use the `docker build` command. - -In the same directory that your Dockerfile is in, you can run the following: - -``` -docker build -t
- There are two main families of databases: Relational and NoSQL. -
--
-
- Relational databases store information in an orderly, column, row, and table schema. They “relate” the tables together to present different views of the data. -
- NoSQL databases are much less structured. This means they can store different data alongside each other – which makes things both easier to store but harder to query across. -
- There are additional types of databases, such as ledger, time-series and others. Those are beyond the scope of this introduction. -
- -# Relational Databases (RDBMS) - -Most users have at least heard of relational databases such as: - -* MySQL / MariaDB -* PostgreSQL -* Microsoft SQL Server -* Oracle - -Relational databases operate on the concepts of tables, relations, schemas, data types, indices, SQL, joins and basic "CRUD" operations. - - C = Create (Insert) - R = Read (Select) - U = Update (Update) - D = Delete (Delete) - -Take the example of an online store, where data revolves around the ideas of items, orders and customers. When a customer makes a purchase in our store, the data from the transaction is actually broken apart into tables of related data. Here’s one way of seeing that process: - - - -The database for such an online store might have a handful of related tables: - -1. **Orders** -2. **Customers** -3. **Credit Cards** -4. **Items** - -Relational database tables use unique keys as a way to relate one table with another, and so the "orders" table -might simply aggregate keys drawn from other tables for each order. This allows each table to have a clear definition -of what data fields, and their data types, are expected with every transaction. Data coming in must be broken apart -to conform to this data structure, and data going out must be drawn back together from across the tables. - -But this “breaking apart” process is actually an intensive, time-consuming process. Data being sent off to -any particular table has to be validated by data type (strings, integers, dates, decimals, binary, etc.), length, -and NULL before it can be inserted into a particular data table. This happens across multiple tables at -the same time, and ensures that the entire transaction completes successfully or is rolled back. - -Impedance Mismatch - a set of conceptual and technical difficulties that are often encountered when interacting with a relational database management system. - -SQL, "structured query language" is the language spoken by most relational databases. While there are slight variations -in SQL syntax between RDBMS platforms (a semicolon here, a percent sign there), they all generally read the same to -anyone familiar with general SQL queries. - -Create a table: -```sql - create table researchers - (researcherID int NOT NULL AUTO_INCREMENT, - first varchar(15), - last varchar(20), - email varchar(30), - age int, - PRIMARY KEY (ID) - ); -``` - -Insert an item into a table: -```sql - insert into researchers - (first, last, email, age) - values ('Jane', 'Doe', 'jdoe@georgia.edu', 34); -``` - -Select (read) all items from a table: -```sql - select * from researchers; -``` - -Select (read) a single item from a table: -```sql - select * from researchers where researcherID = 147; - select * from researchers where first = 'Jane'; - select first, last from researchers where age = 34; -``` - ---- - -# NoSQL Databases - -NoSQL databases come in at least two main groupings: **Aggregate oriented** or **Node-Arc/Graph**. - -## 1. Aggregate-Oriented Databases - -* Key-Value - Redis, Memcached -* Document - DynamoDB, MongoDB -* Column-Family - Cassandra, BigTable - -NoSQL databases share very few common characteristics. Perhaps the only one is that they are **schema-less**. Typical aggregate-oriented NoSQL databases will store an aggregation in the form of strings or entire documents. That is usually in plain text, often in a specific format or notation, such as JSON or XML. - -Here are some sample entries from a simple Key-Value datastore: - -
-
-
-
-
-
-| Key | -Value | -
|---|---|
| access_key | -ABCDEfghijklmnop123456789xyzabc | -
| secret_key | -23481283852384128328a | -
| current_count | -472 | -
| jobs_remaining | -13 | -
| last-winner | -Darla Johnson | -
| last-winner-date | -08/17/2014 08:42:13.015 UTC | -
-In the case of document NoSQL databases, the “value” portion of the entry can get much larger. - -Here is an example of an entry in JSON. Note that the entire entry (or “document”) breaks down into a hierarchy of data: fields and their values, and dictionaries of multiple values, - -``` -{ - "success": { - "total": 1 - }, - "contents": { - "quotes": [ - { - "quote": "Remove the temptation to settle for anything short of what you deserve.", - "length": "71", - "author": "Lorii Myers", - "tags": [ - "expectation", - "inspire", - "perfection" - ], - "category": "inspire", - "date": "2017-09-08", - "permalink": "https://theysaidso.com/quote/ZWrV624xU_q6_KYYlrQpYgeF/lorii-myers-remove-the-temptation-to-settle-for-anything-short-of-what-you-deser", - "title": "Inspiring Quote of the day", - "background": "https://theysaidso.com/img/bgs/man_on_the_mountain.jpg", - "id": "ZWrV624xU_q6_KYYlrQpYgeF" - } - ], - "copyright": "2017-19 theysaidso.com" - } -} -``` - -Also consider that subsequent entries into this table may or may not contain a background image, or the same number of tags, or the precise data structure of this -entry. NoSQL evolved out of the need to quickly collect varied data at very high rates and so it does not suffer from impedance mismatch. Rather, it suffers from -its difficulty to aggregate or join. - -## 2. Node-Arc / Graph Databases - -Graph, or Node-arc databases are entirely different, in that they try to store and represent connectivity between nodes in a constellation, and their relationships. So a “query” of a graph database might inform you about the networks of other nodes related to the node you are interested in, and the types and strengths of those relationships, among other uses. Some examples of Graph DBs are: - -* Neo4j -* TinkerPop -* Infinite - - - - ---- - -# Using Databases in Your Research - -We are frequently asked by researchers how to incorporate databases into their work. Here are four suggestions: - -
-
-
-
-
-
- » Track Results
-- Track the status of your completed work by adding a record to a table upon completion. This lets you - know what work remains open and information about its processing. -
-
-
-
-
- » Queue Up Your Work
-- Collect and store data about future work you need to complete, the steps required, and the expected lifecycle - of each step. While this might be easy to do in Excel, you could grow this into a database that orchestrates - some of these steps for you. -
-
-
-
-
-
-
-
- » Index Everything
-- Maintain a searchable history of source data, result sets, and code used to process them. - This could include links to related data, articles published, GitHub code repositories, and more. -
-
-
-
-
- » Automate
-- If you are awash in source data or have a backlog of files to process, consider automating it by using a database. - Your code, instead of handling a single file at a time, could read each row in the database and process files - indexed in a table. A single HPC job could process thousands of files! -
-- -**Note:** Research Computing may be able to provide support for your database needs. Please schedule a consultation request on our website by filling out [this form](https://www.rc.virginia.edu/form/support-request/?category=Consultation). - - -# Other Resources - -Here is a great overview of databases and their histories: - -{{< youtube qI_g07C_Q5I >}} - -Martin Fowler - NoSQL - YouTube diff --git a/content/notes/databases-intro/reldb.png b/content/notes/databases-intro/reldb.png deleted file mode 100644 index 56d5e2da..00000000 Binary files a/content/notes/databases-intro/reldb.png and /dev/null differ diff --git a/content/notes/rivanna-alphafold/23-alphafold-db.md b/content/notes/hpc-alphafold/23-alphafold-db.md similarity index 57% rename from content/notes/rivanna-alphafold/23-alphafold-db.md rename to content/notes/hpc-alphafold/23-alphafold-db.md index e05f36d5..1aaf13fa 100644 --- a/content/notes/rivanna-alphafold/23-alphafold-db.md +++ b/content/notes/hpc-alphafold/23-alphafold-db.md @@ -4,13 +4,13 @@ date: 2025-05-20T00:23:54Z type: docs weight: 2300 menu: - rivanna-alphafold: + hpc-alphafold: name: Protein Structure Database --- -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_57.png width=75% height=75% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_57.png width=75% height=75% >}} -The [AlphaFold Protein Structure Database](www.alphafold.ebi.ac.uk) has 200 million structure predictions. Even so, running the monomer version is still necessary in special use-cases such as engineered proteins, chimeric proteins, novel proteins translated from newly recognized Open Reading Frames (ORFs), and more. For protein complexes, it is best to use the multimer version. +The [AlphaFold Protein Structure Database](https://www.alphafold.ebi.ac.uk) has 200 million structure predictions. Even so, running the monomer version is still necessary in special use-cases such as engineered proteins, chimeric proteins, novel proteins translated from newly recognized Open Reading Frames (ORFs), and more. For protein complexes, it is best to use the multimer version. DeepMind and EMBL’s European Bioinformatics Institute (EMBL-EBI) have partnered to create AlphaFold DB to make these predictions freely available to the scientific community. The latest database release contains over 200 million entries, providing broad coverage of UniProt (the standard repository of protein sequences and annotations). AlphaFold DB provides individual downloads for the human proteome, the proteomes of 47 key research and global health organisms, and the manually curated Swiss-Prot subset of UniProt. diff --git a/content/notes/rivanna-alphafold/24-limitations-alphafold-db.md b/content/notes/hpc-alphafold/24-limitations-alphafold-db.md similarity index 99% rename from content/notes/rivanna-alphafold/24-limitations-alphafold-db.md rename to content/notes/hpc-alphafold/24-limitations-alphafold-db.md index 98d3b8aa..ba9ad1fe 100644 --- a/content/notes/rivanna-alphafold/24-limitations-alphafold-db.md +++ b/content/notes/hpc-alphafold/24-limitations-alphafold-db.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 2350 menu: - rivanna-alphafold: + hpc-alphafold: parent: Protein Structure Database --- The AlphaFold DB uses a monomeric model similar to the CASP14 version. As a result, many of the same limitations are expected: diff --git a/content/notes/rivanna-alphafold/25-alphafold-alternatives.md b/content/notes/hpc-alphafold/25-alphafold-alternatives.md similarity index 96% rename from content/notes/rivanna-alphafold/25-alphafold-alternatives.md rename to content/notes/hpc-alphafold/25-alphafold-alternatives.md index a7231d96..2fe8defc 100644 --- a/content/notes/rivanna-alphafold/25-alphafold-alternatives.md +++ b/content/notes/hpc-alphafold/25-alphafold-alternatives.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 2400 menu: - rivanna-alphafold: + hpc-alphafold: --- diff --git a/content/notes/rivanna-alphafold/_index.md b/content/notes/hpc-alphafold/_index.md similarity index 80% rename from content/notes/rivanna-alphafold/_index.md rename to content/notes/hpc-alphafold/_index.md index f1bbafd7..39b0531a 100644 --- a/content/notes/rivanna-alphafold/_index.md +++ b/content/notes/hpc-alphafold/_index.md @@ -1,12 +1,12 @@ --- -title: AlphaFold on Rivanna +title: AlphaFold on HPC date: 2025-05-20T00:23:54Z authors: [mab] type: docs weight: 1 menu: - rivanna-alphafold: + hpc-alphafold: --- This workshop will cover: diff --git a/content/notes/rivanna-alphafold/alphafold/4-alphafold-intro.md b/content/notes/hpc-alphafold/alphafold/4-alphafold-intro.md similarity index 98% rename from content/notes/rivanna-alphafold/alphafold/4-alphafold-intro.md rename to content/notes/hpc-alphafold/alphafold/4-alphafold-intro.md index 44e45027..f78f4340 100644 --- a/content/notes/rivanna-alphafold/alphafold/4-alphafold-intro.md +++ b/content/notes/hpc-alphafold/alphafold/4-alphafold-intro.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 400 menu: - rivanna-alphafold: + hpc-alphafold: name: AlphaFold --- diff --git a/content/notes/rivanna-alphafold/alphafold/5-prediction-pipeline.md b/content/notes/hpc-alphafold/alphafold/5-prediction-pipeline.md similarity index 84% rename from content/notes/rivanna-alphafold/alphafold/5-prediction-pipeline.md rename to content/notes/hpc-alphafold/alphafold/5-prediction-pipeline.md index e7835787..dfeb2122 100644 --- a/content/notes/rivanna-alphafold/alphafold/5-prediction-pipeline.md +++ b/content/notes/hpc-alphafold/alphafold/5-prediction-pipeline.md @@ -4,11 +4,11 @@ date: 2025-05-20T00:23:54Z type: docs weight: 450 menu: - rivanna-alphafold: + hpc-alphafold: parent: AlphaFold --- -{{< figure src=/notes/rivanna-alphafold/img/pipeline-steps.png caption="Prediction pipeline" width=70% height=70% >}} +{{< figure src=/notes/hpc-alphafold/img/pipeline-steps.png caption="Prediction pipeline" width=70% height=70% >}} 1. **User Input (FASTA)** The user provides a FASTA file: one sequence for monomer predictions or multiple sequences for multimer predictions. @@ -45,6 +45,6 @@ menu: 5. **Ranking** The five models are ranked based on a confidence score, and the highest-ranking structures are output as **PDB files**. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_10.png width=90% height=90% caption="AlphaFold model architecture." >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_10.png width=90% height=90% caption="AlphaFold model architecture." >}} -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_12.png width=70% height=70% caption="Each of AlphaFold’s five model predictions is run in parallel, relaxed independently, and ranked by confidence." >}} \ No newline at end of file +{{< figure src=/notes/hpc-alphafold/img/Alphafold_12.png width=70% height=70% caption="Each of AlphaFold’s five model predictions is run in parallel, relaxed independently, and ranked by confidence." >}} \ No newline at end of file diff --git a/content/notes/rivanna-alphafold/alphafold/6-outputs-alphafold.md b/content/notes/hpc-alphafold/alphafold/6-outputs-alphafold.md similarity index 97% rename from content/notes/rivanna-alphafold/alphafold/6-outputs-alphafold.md rename to content/notes/hpc-alphafold/alphafold/6-outputs-alphafold.md index f5fdf4ca..7bb42be6 100644 --- a/content/notes/rivanna-alphafold/alphafold/6-outputs-alphafold.md +++ b/content/notes/hpc-alphafold/alphafold/6-outputs-alphafold.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 550 menu: - rivanna-alphafold: + hpc-alphafold: parent: AlphaFold --- diff --git a/content/notes/rivanna-alphafold/alphafold/7-confidence-metrics.md b/content/notes/hpc-alphafold/alphafold/7-confidence-metrics.md similarity index 97% rename from content/notes/rivanna-alphafold/alphafold/7-confidence-metrics.md rename to content/notes/hpc-alphafold/alphafold/7-confidence-metrics.md index db397d8e..770e5d90 100644 --- a/content/notes/rivanna-alphafold/alphafold/7-confidence-metrics.md +++ b/content/notes/hpc-alphafold/alphafold/7-confidence-metrics.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 580 menu: - rivanna-alphafold: + hpc-alphafold: parent: AlphaFold --- diff --git a/content/notes/rivanna-alphafold/img/Alphafold_10.png b/content/notes/hpc-alphafold/img/Alphafold_10.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_10.png rename to content/notes/hpc-alphafold/img/Alphafold_10.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_12.png b/content/notes/hpc-alphafold/img/Alphafold_12.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_12.png rename to content/notes/hpc-alphafold/img/Alphafold_12.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_17.png b/content/notes/hpc-alphafold/img/Alphafold_17.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_17.png rename to content/notes/hpc-alphafold/img/Alphafold_17.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_18.png b/content/notes/hpc-alphafold/img/Alphafold_18.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_18.png rename to content/notes/hpc-alphafold/img/Alphafold_18.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_2.jpg b/content/notes/hpc-alphafold/img/Alphafold_2.jpg similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_2.jpg rename to content/notes/hpc-alphafold/img/Alphafold_2.jpg diff --git a/content/notes/rivanna-alphafold/img/Alphafold_20.jpg b/content/notes/hpc-alphafold/img/Alphafold_20.jpg similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_20.jpg rename to content/notes/hpc-alphafold/img/Alphafold_20.jpg diff --git a/content/notes/rivanna-alphafold/img/Alphafold_21.png b/content/notes/hpc-alphafold/img/Alphafold_21.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_21.png rename to content/notes/hpc-alphafold/img/Alphafold_21.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_22.png b/content/notes/hpc-alphafold/img/Alphafold_22.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_22.png rename to content/notes/hpc-alphafold/img/Alphafold_22.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_23.png b/content/notes/hpc-alphafold/img/Alphafold_23.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_23.png rename to content/notes/hpc-alphafold/img/Alphafold_23.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_24.png b/content/notes/hpc-alphafold/img/Alphafold_24.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_24.png rename to content/notes/hpc-alphafold/img/Alphafold_24.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_27.png b/content/notes/hpc-alphafold/img/Alphafold_27.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_27.png rename to content/notes/hpc-alphafold/img/Alphafold_27.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_28.png b/content/notes/hpc-alphafold/img/Alphafold_28.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_28.png rename to content/notes/hpc-alphafold/img/Alphafold_28.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_29.png b/content/notes/hpc-alphafold/img/Alphafold_29.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_29.png rename to content/notes/hpc-alphafold/img/Alphafold_29.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_3.png b/content/notes/hpc-alphafold/img/Alphafold_3.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_3.png rename to content/notes/hpc-alphafold/img/Alphafold_3.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_30.png b/content/notes/hpc-alphafold/img/Alphafold_30.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_30.png rename to content/notes/hpc-alphafold/img/Alphafold_30.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_31.png b/content/notes/hpc-alphafold/img/Alphafold_31.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_31.png rename to content/notes/hpc-alphafold/img/Alphafold_31.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_32.png b/content/notes/hpc-alphafold/img/Alphafold_32.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_32.png rename to content/notes/hpc-alphafold/img/Alphafold_32.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_33.png b/content/notes/hpc-alphafold/img/Alphafold_33.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_33.png rename to content/notes/hpc-alphafold/img/Alphafold_33.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_34.png b/content/notes/hpc-alphafold/img/Alphafold_34.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_34.png rename to content/notes/hpc-alphafold/img/Alphafold_34.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_35.png b/content/notes/hpc-alphafold/img/Alphafold_35.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_35.png rename to content/notes/hpc-alphafold/img/Alphafold_35.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_36.jpg b/content/notes/hpc-alphafold/img/Alphafold_36.jpg similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_36.jpg rename to content/notes/hpc-alphafold/img/Alphafold_36.jpg diff --git a/content/notes/rivanna-alphafold/img/Alphafold_4.png b/content/notes/hpc-alphafold/img/Alphafold_4.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_4.png rename to content/notes/hpc-alphafold/img/Alphafold_4.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_40.png b/content/notes/hpc-alphafold/img/Alphafold_40.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_40.png rename to content/notes/hpc-alphafold/img/Alphafold_40.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_41.jpg b/content/notes/hpc-alphafold/img/Alphafold_41.jpg similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_41.jpg rename to content/notes/hpc-alphafold/img/Alphafold_41.jpg diff --git a/content/notes/rivanna-alphafold/img/Alphafold_42.png b/content/notes/hpc-alphafold/img/Alphafold_42.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_42.png rename to content/notes/hpc-alphafold/img/Alphafold_42.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_43.png b/content/notes/hpc-alphafold/img/Alphafold_43.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_43.png rename to content/notes/hpc-alphafold/img/Alphafold_43.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_44.png b/content/notes/hpc-alphafold/img/Alphafold_44.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_44.png rename to content/notes/hpc-alphafold/img/Alphafold_44.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_45.png b/content/notes/hpc-alphafold/img/Alphafold_45.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_45.png rename to content/notes/hpc-alphafold/img/Alphafold_45.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_47.png b/content/notes/hpc-alphafold/img/Alphafold_47.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_47.png rename to content/notes/hpc-alphafold/img/Alphafold_47.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_48.png b/content/notes/hpc-alphafold/img/Alphafold_48.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_48.png rename to content/notes/hpc-alphafold/img/Alphafold_48.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_5.png b/content/notes/hpc-alphafold/img/Alphafold_5.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_5.png rename to content/notes/hpc-alphafold/img/Alphafold_5.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_50.jpg b/content/notes/hpc-alphafold/img/Alphafold_50.jpg similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_50.jpg rename to content/notes/hpc-alphafold/img/Alphafold_50.jpg diff --git a/content/notes/rivanna-alphafold/img/Alphafold_51.png b/content/notes/hpc-alphafold/img/Alphafold_51.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_51.png rename to content/notes/hpc-alphafold/img/Alphafold_51.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_53.png b/content/notes/hpc-alphafold/img/Alphafold_53.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_53.png rename to content/notes/hpc-alphafold/img/Alphafold_53.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_54.png b/content/notes/hpc-alphafold/img/Alphafold_54.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_54.png rename to content/notes/hpc-alphafold/img/Alphafold_54.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_57.png b/content/notes/hpc-alphafold/img/Alphafold_57.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_57.png rename to content/notes/hpc-alphafold/img/Alphafold_57.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_58.png b/content/notes/hpc-alphafold/img/Alphafold_58.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_58.png rename to content/notes/hpc-alphafold/img/Alphafold_58.png diff --git a/content/notes/rivanna-alphafold/img/Alphafold_7.png b/content/notes/hpc-alphafold/img/Alphafold_7.png similarity index 100% rename from content/notes/rivanna-alphafold/img/Alphafold_7.png rename to content/notes/hpc-alphafold/img/Alphafold_7.png diff --git a/content/notes/rivanna-alphafold/img/background-colors.png b/content/notes/hpc-alphafold/img/background-colors.png similarity index 100% rename from content/notes/rivanna-alphafold/img/background-colors.png rename to content/notes/hpc-alphafold/img/background-colors.png diff --git a/content/notes/rivanna-alphafold/img/molecule-file-browser.png b/content/notes/hpc-alphafold/img/molecule-file-browser.png similarity index 100% rename from content/notes/rivanna-alphafold/img/molecule-file-browser.png rename to content/notes/hpc-alphafold/img/molecule-file-browser.png diff --git a/content/notes/rivanna-alphafold/img/pipeline-steps.png b/content/notes/hpc-alphafold/img/pipeline-steps.png similarity index 100% rename from content/notes/rivanna-alphafold/img/pipeline-steps.png rename to content/notes/hpc-alphafold/img/pipeline-steps.png diff --git a/content/notes/rivanna-alphafold/img/rivanna-steps.png b/content/notes/hpc-alphafold/img/rivanna-steps.png similarity index 100% rename from content/notes/rivanna-alphafold/img/rivanna-steps.png rename to content/notes/hpc-alphafold/img/rivanna-steps.png diff --git a/content/notes/rivanna-alphafold/img/vmd-export.png b/content/notes/hpc-alphafold/img/vmd-export.png similarity index 100% rename from content/notes/rivanna-alphafold/img/vmd-export.png rename to content/notes/hpc-alphafold/img/vmd-export.png diff --git a/content/notes/rivanna-alphafold/img/vmd-part-3.png b/content/notes/hpc-alphafold/img/vmd-part-3.png similarity index 100% rename from content/notes/rivanna-alphafold/img/vmd-part-3.png rename to content/notes/hpc-alphafold/img/vmd-part-3.png diff --git a/content/notes/rivanna-alphafold/img/vmd-part-4.png b/content/notes/hpc-alphafold/img/vmd-part-4.png similarity index 100% rename from content/notes/rivanna-alphafold/img/vmd-part-4.png rename to content/notes/hpc-alphafold/img/vmd-part-4.png diff --git a/content/notes/rivanna-alphafold/intro/0-intro.md b/content/notes/hpc-alphafold/intro/0-intro.md similarity index 84% rename from content/notes/rivanna-alphafold/intro/0-intro.md rename to content/notes/hpc-alphafold/intro/0-intro.md index 6fd6a513..8a4fc9bc 100644 --- a/content/notes/rivanna-alphafold/intro/0-intro.md +++ b/content/notes/hpc-alphafold/intro/0-intro.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 200 menu: - rivanna-alphafold: + hpc-alphafold: --- This section will cover: @@ -19,4 +19,4 @@ This section will cover: * Pipeline and Architecture -* Confidence Metrics \ No newline at end of file +* Confidence Metrics diff --git a/content/notes/rivanna-alphafold/intro/1-gpu-vs-cpu.md b/content/notes/hpc-alphafold/intro/1-gpu-vs-cpu.md similarity index 89% rename from content/notes/rivanna-alphafold/intro/1-gpu-vs-cpu.md rename to content/notes/hpc-alphafold/intro/1-gpu-vs-cpu.md index 72c34c9d..83ed031a 100644 --- a/content/notes/rivanna-alphafold/intro/1-gpu-vs-cpu.md +++ b/content/notes/hpc-alphafold/intro/1-gpu-vs-cpu.md @@ -4,11 +4,11 @@ date: 2025-05-20T00:23:54Z type: docs weight: 210 menu: - rivanna-alphafold: + hpc-alphafold: parent: Introduction --- -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_2.jpg width=70% height=70% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_2.jpg width=70% height=70% >}} - The **CPU (Central Processing Unit)** is the general-purpose processor responsible for handling most computing tasks in a system. - The **GPU (Graphics Processing Unit)** is now widely used for accelerating scientific and high-performance computing tasks. @@ -28,9 +28,9 @@ menu: **Metaphor: GPU vs CPU** -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_3.png width=50% height=50% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_3.png width=50% height=50% >}} -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_4.png width=50% height=50% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_4.png width=50% height=50% >}} **CPUs are like scooters**: simple, efficient, and great for quickly handling one task or a few tasks at a time. diff --git a/content/notes/rivanna-alphafold/intro/2-protein-folding-problem.md b/content/notes/hpc-alphafold/intro/2-protein-folding-problem.md similarity index 95% rename from content/notes/rivanna-alphafold/intro/2-protein-folding-problem.md rename to content/notes/hpc-alphafold/intro/2-protein-folding-problem.md index b8c0c013..5b657131 100644 --- a/content/notes/rivanna-alphafold/intro/2-protein-folding-problem.md +++ b/content/notes/hpc-alphafold/intro/2-protein-folding-problem.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 250 menu: - rivanna-alphafold: + hpc-alphafold: parent: Introduction --- @@ -15,4 +15,4 @@ As of 2025, only about 174,000 protein structures have been solved and deposited This massive gap between known sequences and solved structures is known as the **protein folding problem**. -**AlphaFold** addresses this problem using AI to predict protein structures from amino acid sequences — at scale and with high accuracy. \ No newline at end of file +**AlphaFold** addresses this problem using AI to predict protein structures from amino acid sequences — at scale and with high accuracy. diff --git a/content/notes/rivanna-alphafold/intro/3-casp-competitions.md b/content/notes/hpc-alphafold/intro/3-casp-competitions.md similarity index 73% rename from content/notes/rivanna-alphafold/intro/3-casp-competitions.md rename to content/notes/hpc-alphafold/intro/3-casp-competitions.md index fa7a6a08..9045dbfc 100644 --- a/content/notes/rivanna-alphafold/intro/3-casp-competitions.md +++ b/content/notes/hpc-alphafold/intro/3-casp-competitions.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 300 menu: - rivanna-alphafold: + hpc-alphafold: parent: Introduction --- @@ -14,12 +14,12 @@ Participants are challenged to predict structures of proteins whose experimental CASP uses the **Global Distance Test - Total Score** (GDT-TS) as a performance metric, which measures what percentage of α-carbons are within a threshold distance (in angstroms) of the experimental structure. The higher the GDT-TS score, the more accurate the model's prediction is. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_5.png width=25% height=25% caption="Example of a predicted protein structure (orange) compared to the experimentally solved structure (black). CASP evaluates prediction accuracy using these kinds of comparisons." >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_5.png width=25% height=25% caption="Example of a predicted protein structure (orange) compared to the experimentally solved structure (black). CASP evaluates prediction accuracy using these kinds of comparisons." >}} AlphaFold’s strong performance in **CASP13** and **CASP14** proved its ability to predict protein structures with near-experimental accuracy. This established AlphaFold as a game changer in the world of structural biology and computational science. [Learn More](https://www.nature.com/articles/d41586-020-03348-4) -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_7.png width=60% height=60% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_7.png width=60% height=60% >}} diff --git a/content/notes/rivanna-alphafold/out/14-outputs.md b/content/notes/hpc-alphafold/out/14-outputs.md similarity index 80% rename from content/notes/rivanna-alphafold/out/14-outputs.md rename to content/notes/hpc-alphafold/out/14-outputs.md index 2718fde4..0b415011 100644 --- a/content/notes/rivanna-alphafold/out/14-outputs.md +++ b/content/notes/hpc-alphafold/out/14-outputs.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1650 menu: - rivanna-alphafold: + hpc-alphafold: name: Outputs --- @@ -16,6 +16,6 @@ After running AlphaFold, your output directory will include a mix of files from > Files labeled `ranked_0` represent the **best-ranked model** based on AlphaFold's internal confidence scoring. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_24.png width=40% height=40% caption="Example output directory" >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_24.png width=40% height=40% caption="Example output directory" >}} diff --git a/content/notes/rivanna-alphafold/out/15-sequence-coverage-plot.md b/content/notes/hpc-alphafold/out/15-sequence-coverage-plot.md similarity index 69% rename from content/notes/rivanna-alphafold/out/15-sequence-coverage-plot.md rename to content/notes/hpc-alphafold/out/15-sequence-coverage-plot.md index 889d4477..fa5e8dc2 100644 --- a/content/notes/rivanna-alphafold/out/15-sequence-coverage-plot.md +++ b/content/notes/hpc-alphafold/out/15-sequence-coverage-plot.md @@ -4,13 +4,13 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1700 menu: - rivanna-alphafold: + hpc-alphafold: parent: Outputs --- Sequence coverage plots (`*_sequence_coverage.pdf`) show the alignment depth along the length of the protein. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_27.png width=85% height=85% caption="Sequence coverage plot for CB1. Darker colors indicate higher identity to the query sequence.">}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_27.png width=85% height=85% caption="Sequence coverage plot for CB1. Darker colors indicate higher identity to the query sequence.">}} This example has a depth of ~13,000 sequences, with varying levels of homology to the query. The CB1 protein is a 7-transmembrane (7TM) helical protein with less-structured tails on both termini — which is also reflected in its 3D structure predictions. diff --git a/content/notes/rivanna-alphafold/out/15a-postprocessing.md b/content/notes/hpc-alphafold/out/15a-postprocessing.md similarity index 98% rename from content/notes/rivanna-alphafold/out/15a-postprocessing.md rename to content/notes/hpc-alphafold/out/15a-postprocessing.md index 77a7cd13..f4c63e66 100644 --- a/content/notes/rivanna-alphafold/out/15a-postprocessing.md +++ b/content/notes/hpc-alphafold/out/15a-postprocessing.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1740 menu: - rivanna-alphafold: + hpc-alphafold: parent: Outputs --- diff --git a/content/notes/rivanna-alphafold/out/16-plddt-plot.md b/content/notes/hpc-alphafold/out/16-plddt-plot.md similarity index 82% rename from content/notes/rivanna-alphafold/out/16-plddt-plot.md rename to content/notes/hpc-alphafold/out/16-plddt-plot.md index 7caabbdb..aa3e4392 100644 --- a/content/notes/rivanna-alphafold/out/16-plddt-plot.md +++ b/content/notes/hpc-alphafold/out/16-plddt-plot.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1750 menu: - rivanna-alphafold: + hpc-alphafold: parent: Outputs --- @@ -14,9 +14,9 @@ This output plot shows the **per-residue confidence scores** (pLDDT) for the top For example, CB1 is a 7-transmembrane (7TM) domain protein with less-structured tails on both termini which is reflected in the plot. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_29.png width=80% height=80% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_29.png width=80% height=80% >}} -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_30.png width=30% height=30% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_30.png width=30% height=30% >}} ### Key Details diff --git a/content/notes/rivanna-alphafold/out/17-pae-plot.md b/content/notes/hpc-alphafold/out/17-pae-plot.md similarity index 89% rename from content/notes/rivanna-alphafold/out/17-pae-plot.md rename to content/notes/hpc-alphafold/out/17-pae-plot.md index 576ba088..26cac732 100644 --- a/content/notes/rivanna-alphafold/out/17-pae-plot.md +++ b/content/notes/hpc-alphafold/out/17-pae-plot.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1800 menu: - rivanna-alphafold: + hpc-alphafold: parent: Outputs --- @@ -18,7 +18,7 @@ The color scale indicates _expected distance error_ in angstroms. Purple is 'go For example, the CB1 protein is a 7-transmembrane domain GPCR with less-structured tails on both termini, which appear as regions of higher predicted error. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_31.png width=60% height=60% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_31.png width=60% height=60% >}} Source: Arnold, M. J. (2021) AlphaPickle doi.org/10.5281/zenodo.5708709 diff --git a/content/notes/rivanna-alphafold/out/18-csv-outputs.md b/content/notes/hpc-alphafold/out/18-csv-outputs.md similarity index 65% rename from content/notes/rivanna-alphafold/out/18-csv-outputs.md rename to content/notes/hpc-alphafold/out/18-csv-outputs.md index 7a2ef5c0..ea1fe27b 100644 --- a/content/notes/rivanna-alphafold/out/18-csv-outputs.md +++ b/content/notes/hpc-alphafold/out/18-csv-outputs.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1850 menu: - rivanna-alphafold: + hpc-alphafold: parent: Outputs --- @@ -12,13 +12,13 @@ AlphaPickle generates CSV files that can be used for downstream analysis and plo **`PAE.csv`** -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_32.png width=70% height=70% caption="Example PAE.csv file contents." >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_32.png width=70% height=70% caption="Example PAE.csv file contents." >}} `PAE.csv` is an `n_residue` x `n_residue` matrix that is used for generating PAE heatmaps. **`pLDDT.csv`** -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_33.png width=20% height=20% caption="Example pLDDT.csv file contents." >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_33.png width=20% height=20% caption="Example pLDDT.csv file contents." >}} `pLDDT.csv` is an `n_residue`-long list which is helpful for generating pLDDT plots. diff --git a/content/notes/rivanna-alphafold/out/19-pdb-structure.md b/content/notes/hpc-alphafold/out/19-pdb-structure.md similarity index 80% rename from content/notes/rivanna-alphafold/out/19-pdb-structure.md rename to content/notes/hpc-alphafold/out/19-pdb-structure.md index 2d153a2d..df2660f2 100644 --- a/content/notes/rivanna-alphafold/out/19-pdb-structure.md +++ b/content/notes/hpc-alphafold/out/19-pdb-structure.md @@ -4,18 +4,18 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1900 menu: - rivanna-alphafold: + hpc-alphafold: parent: Outputs --- -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_34.png width=40% height=40% caption="Predicted 3D structure of CB1." >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_34.png width=40% height=40% caption="Predicted 3D structure of CB1." >}} The above image shows an example of a 3D structure from a `ranked_0.pdb` file. The structure is colored by `pLDDT`, where dark blue represents high confidence and red represents low confidence. CB1 is a 7-transmembrane domain (7TM) septa-helical protein with less-structured tails on both termini. This is reflected in the 3D structure, where disordered regions ("ribbons") are low-confidence and helices are high-confidence. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_35.png width=40% height=40% caption="Schematic diagram of CB1.">}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_35.png width=40% height=40% caption="Schematic diagram of CB1.">}} This schematic complements the 3D model by showing the membrane topology of CB1. The arrangement of helices and loops explains the structured and unstructured regions seen in the AlphaFold prediction. diff --git a/content/notes/rivanna-alphafold/running/10-working-directory-alphafold.md b/content/notes/hpc-alphafold/running/10-working-directory-alphafold.md similarity index 90% rename from content/notes/rivanna-alphafold/running/10-working-directory-alphafold.md rename to content/notes/hpc-alphafold/running/10-working-directory-alphafold.md index dd5ac5f8..ef1bbf47 100644 --- a/content/notes/rivanna-alphafold/running/10-working-directory-alphafold.md +++ b/content/notes/hpc-alphafold/running/10-working-directory-alphafold.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 800 menu: - rivanna-alphafold: + hpc-alphafold: parent: Running AlphaFold On Rivanna --- @@ -48,4 +48,4 @@ cd alphafold ls -lh ``` -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_21.png width=90% height=90% caption="Example of AlphaFold working directory contents">}} \ No newline at end of file +{{< figure src=/notes/hpc-alphafold/img/Alphafold_21.png width=90% height=90% caption="Example of AlphaFold working directory contents">}} \ No newline at end of file diff --git a/content/notes/rivanna-alphafold/running/11-input-fasta.md b/content/notes/hpc-alphafold/running/11-input-fasta.md similarity index 71% rename from content/notes/rivanna-alphafold/running/11-input-fasta.md rename to content/notes/hpc-alphafold/running/11-input-fasta.md index 9d62731a..0758cc46 100644 --- a/content/notes/rivanna-alphafold/running/11-input-fasta.md +++ b/content/notes/hpc-alphafold/running/11-input-fasta.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1100 menu: - rivanna-alphafold: + hpc-alphafold: parent: Running AlphaFold On Rivanna --- @@ -16,6 +16,6 @@ As an example, you can download a sequence from UniProt, such as [P21554 (CNR1)] Next, get your FASTA file onto Rivanna in your working directory. You can do this using OOD, Globus, or `scp` (command line tool), or you can make the file directly on Rivanna with a text editor like `nano` or `vim`. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_22.png width=40% height=40% caption="Schematic diagram of the expected structure for CNR1">}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_22.png width=40% height=40% caption="Schematic diagram of the expected structure for CNR1">}} -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_23.png width=90% height=90% caption="Text representation of a FASTA file for CNR1">}} \ No newline at end of file +{{< figure src=/notes/hpc-alphafold/img/Alphafold_23.png width=90% height=90% caption="Text representation of a FASTA file for CNR1">}} \ No newline at end of file diff --git a/content/notes/rivanna-alphafold/running/12-submit-slurm.md b/content/notes/hpc-alphafold/running/12-submit-slurm.md similarity index 94% rename from content/notes/rivanna-alphafold/running/12-submit-slurm.md rename to content/notes/hpc-alphafold/running/12-submit-slurm.md index 31b713f6..c6625a31 100644 --- a/content/notes/rivanna-alphafold/running/12-submit-slurm.md +++ b/content/notes/hpc-alphafold/running/12-submit-slurm.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 1150 menu: - rivanna-alphafold: + hpc-alphafold: parent: Running AlphaFold On Rivanna --- diff --git a/content/notes/rivanna-alphafold/running/7-loggingon.md b/content/notes/hpc-alphafold/running/7-loggingon.md similarity index 68% rename from content/notes/rivanna-alphafold/running/7-loggingon.md rename to content/notes/hpc-alphafold/running/7-loggingon.md index ec864a60..4c92ae91 100644 --- a/content/notes/rivanna-alphafold/running/7-loggingon.md +++ b/content/notes/hpc-alphafold/running/7-loggingon.md @@ -4,22 +4,22 @@ date: 2025-05-20T00:23:54Z type: docs weight: 605 menu: - rivanna-alphafold: + hpc-alphafold: parent: Running AlphaFold On Rivanna --- First, log in to the [Open OnDemand portal](ood.hpc.virginia.edu) and navigate to 'Desktop'. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_17.png width=80% height=80% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_17.png width=80% height=80% >}} From there, you can launch a Desktop session on Rivanna. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_18.png width=60% height=60% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_18.png width=60% height=60% >}} Choose your desired resources (cores, memory, time, etc.). Note: You must have a valid allocation to launch a job. -The `rivanna-training` allocation is no longer supported for AlphaFold jobs. +The `hpc-training` allocation is no longer supported for AlphaFold jobs. If you are part of a lab or course, use the allocation provided to you (e.g., `yourgroup-allocation`). If you're unsure what to use, contact your PI or the RC support team for help. @@ -29,7 +29,7 @@ Once your session is ready, click the blue "Launch Desktop" button. You’ll be placed in a Linux environment like this: -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_20.jpg width=70% height=70% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_20.jpg width=70% height=70% >}} Use the MATE Terminal to run commands or load software modules. diff --git a/content/notes/rivanna-alphafold/running/7a-prep-wd.md b/content/notes/hpc-alphafold/running/7a-prep-wd.md similarity index 98% rename from content/notes/rivanna-alphafold/running/7a-prep-wd.md rename to content/notes/hpc-alphafold/running/7a-prep-wd.md index 16fc11ea..860794da 100644 --- a/content/notes/rivanna-alphafold/running/7a-prep-wd.md +++ b/content/notes/hpc-alphafold/running/7a-prep-wd.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 610 menu: - rivanna-alphafold: + hpc-alphafold: parent: Running AlphaFold On Rivanna --- diff --git a/content/notes/rivanna-alphafold/running/7b-prep-conda-env.md b/content/notes/hpc-alphafold/running/7b-prep-conda-env.md similarity index 96% rename from content/notes/rivanna-alphafold/running/7b-prep-conda-env.md rename to content/notes/hpc-alphafold/running/7b-prep-conda-env.md index 96f3095e..8ccd7865 100644 --- a/content/notes/rivanna-alphafold/running/7b-prep-conda-env.md +++ b/content/notes/hpc-alphafold/running/7b-prep-conda-env.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 620 menu: - rivanna-alphafold: + hpc-alphafold: parent: Running AlphaFold On Rivanna --- diff --git a/content/notes/rivanna-alphafold/running/8-running-on-rivanna.md b/content/notes/hpc-alphafold/running/8-running-on-rivanna.md similarity index 97% rename from content/notes/rivanna-alphafold/running/8-running-on-rivanna.md rename to content/notes/hpc-alphafold/running/8-running-on-rivanna.md index bd7a00e1..f5032edb 100644 --- a/content/notes/rivanna-alphafold/running/8-running-on-rivanna.md +++ b/content/notes/hpc-alphafold/running/8-running-on-rivanna.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 600 menu: - rivanna-alphafold: + hpc-alphafold: name: Running AlphaFold On Rivanna --- diff --git a/content/notes/hpc-alphafold/vmd/20-visualizing-structures.md b/content/notes/hpc-alphafold/vmd/20-visualizing-structures.md new file mode 100644 index 00000000..e1db76a1 --- /dev/null +++ b/content/notes/hpc-alphafold/vmd/20-visualizing-structures.md @@ -0,0 +1,31 @@ +--- +title: Visualizing 3D Structures +date: 2025-05-20T00:23:54Z +type: docs +weight: 1950 +menu: + hpc-alphafold: + name: Visualizing 3D Structures +--- + +To view your predicted structure generated by AlphaFold in 3D, you can use **Visual Molecular Dynamics (VMD)**. +{{< figure src=/notes/hpc-alphafold/img/Alphafold_40.png width=60% height=60% >}} + +Steps: +* Log onto Open On Demand ([https://hpc-portal.hpc.virginia.edu/](https://hpc-portal.hpc.virginia.edu/)) +* Choose the ‘Desktop’ interactive app and fill out the webform with your required resources. +* Click launch. +* Load VMD in the terminal: +```bash +module load vmd +vmd +``` +The VMD windows will launch. From there, you can open your `.pdb` file from the AlphaFold output directory to view the 3D model (colored by `pLDDT`). + +{{< figure src=/notes/hpc-alphafold/img/Alphafold_41.jpg width=65% height=65% caption="Loading your .pdb file (a)." >}} +{{< figure src=/notes/hpc-alphafold/img/molecule-file-browser.png width=65% height=65% caption="Loading your .pdb file (b)." >}} +{{< figure src=/notes/hpc-alphafold/img/vmd-part-3.png width=60% height=60% caption="Loading your .pdb file (c)." >}} +{{< figure src=/notes/hpc-alphafold/img/vmd-part-4.png width=50% height=50% caption="Loading your .pdb file (d)." >}} + + + diff --git a/content/notes/rivanna-alphafold/vmd/21-modify-vmd.md b/content/notes/hpc-alphafold/vmd/21-modify-vmd.md similarity index 57% rename from content/notes/rivanna-alphafold/vmd/21-modify-vmd.md rename to content/notes/hpc-alphafold/vmd/21-modify-vmd.md index 8b4c16fa..49a06253 100644 --- a/content/notes/rivanna-alphafold/vmd/21-modify-vmd.md +++ b/content/notes/hpc-alphafold/vmd/21-modify-vmd.md @@ -4,13 +4,13 @@ date: 2025-05-20T00:23:54Z type: docs weight: 2150 menu: - rivanna-alphafold: + hpc-alphafold: parent: Visualizing 3D Structures --- Once your PDB file is loaded in VMD, you can change the visual style using the Graphical Representations menu. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_45.png width=50% height=50% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_45.png width=50% height=50% >}} ### Representation settings @@ -19,10 +19,10 @@ Once your PDB file is loaded in VMD, you can change the visual style using the G **Coloring Method:** Set to `Beta` to visualize `pLDDT` confidence values (stored in the beta column). -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_47.png width=30% height=30% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_47.png width=30% height=30% >}} ### Result -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_48.png width=70% height=70% caption="Resulting visualization colored by pLDDT via 'beta' slot." >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_48.png width=70% height=70% caption="Resulting visualization colored by pLDDT via 'beta' slot." >}} diff --git a/content/notes/rivanna-alphafold/vmd/22-background-color-vmd.md b/content/notes/hpc-alphafold/vmd/22-background-color-vmd.md similarity index 78% rename from content/notes/rivanna-alphafold/vmd/22-background-color-vmd.md rename to content/notes/hpc-alphafold/vmd/22-background-color-vmd.md index c4e44791..32a63351 100644 --- a/content/notes/rivanna-alphafold/vmd/22-background-color-vmd.md +++ b/content/notes/hpc-alphafold/vmd/22-background-color-vmd.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 2200 menu: - rivanna-alphafold: + hpc-alphafold: parent: Visualizing 3D Structures --- @@ -18,8 +18,8 @@ To change the background color: 4. In the *Names* list, select **Background**. 5. Choose a new color from the *Colors* list or modify the RGB sliders in the *Color Definitions* tab. -{{< figure src=/notes/rivanna-alphafold/img/background-colors.png width=75% height=75% >}} -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_54.png width=65% height=65% >}} +{{< figure src=/notes/hpc-alphafold/img/background-colors.png width=75% height=75% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_54.png width=65% height=65% >}} >This isn’t a full VMD tutorial, but it is beneficial to play around with it as needed to help you in your research. diff --git a/content/notes/rivanna-alphafold/vmd/22-exporting-vmd.md b/content/notes/hpc-alphafold/vmd/22-exporting-vmd.md similarity index 79% rename from content/notes/rivanna-alphafold/vmd/22-exporting-vmd.md rename to content/notes/hpc-alphafold/vmd/22-exporting-vmd.md index 1b9a3105..7fb469cf 100644 --- a/content/notes/rivanna-alphafold/vmd/22-exporting-vmd.md +++ b/content/notes/hpc-alphafold/vmd/22-exporting-vmd.md @@ -4,7 +4,7 @@ date: 2025-05-20T00:23:54Z type: docs weight: 2250 menu: - rivanna-alphafold: + hpc-alphafold: parent: Visualizing 3D Structures --- @@ -13,17 +13,17 @@ The simplest way to produce raster image files in VMD is to use the `Snapshot` f To use the snapshot feature, simply open the Render form and choose the snapshot option. VMD will capture the contents of the graphics window, and attempt to save the resulting image to the filename given in the Render form. You may find that it is important not to have other windows or cursors in front of the VMD graphics display when using snapshot, since the resulting images may include obscuring windows or cursors. This is a platform-dependent behavior, so you will need to determine if your system does this or not. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_50.jpg width=60% height=60% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_50.jpg width=60% height=60% >}} -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_51.png width=60% height=60% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_51.png width=60% height=60% >}} In the pop-up `imagemagick` window, right-click and hit `save` (see below). -{{< figure src=/notes/rivanna-alphafold/img/vmd-export.png width=50% height=50% >}} +{{< figure src=/notes/hpc-alphafold/img/vmd-export.png width=50% height=50% >}} Next, hit `format`, and a new popup appears. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_53.png width=75% height=75% >}} +{{< figure src=/notes/hpc-alphafold/img/Alphafold_53.png width=75% height=75% >}} Select your image format (`TIFF` or `PNG` is recommended) and hit `select`. Type the file name, including the extension, and hit `save`. diff --git a/content/notes/hpc-intro/_index.md b/content/notes/hpc-intro/_index.md index cd7c1827..0fa8a8bd 100644 --- a/content/notes/hpc-intro/_index.md +++ b/content/notes/hpc-intro/_index.md @@ -18,4 +18,4 @@ menu: UVA's primary resource for high\-performance computing provides a platform for computationally-intensive research across a variety of disciplines. -{{< figure src="/notes/hpc-intro/img/rivanna_racks.png" alt="A data center with racks of servers." caption="Some servers in UVA's HPC cluster" >}} +{{< figure src="/notes/hpc-intro/img/rivanna_racks.jpg" alt="A data center with racks of servers." caption="Some servers in UVA's HPC cluster" >}} diff --git a/content/notes/hpc-intro/img/rivanna_racks.jpg b/content/notes/hpc-intro/img/rivanna_racks.jpg new file mode 100644 index 00000000..9ed770b7 Binary files /dev/null and b/content/notes/hpc-intro/img/rivanna_racks.jpg differ diff --git a/content/notes/matlab-parallel-programming/matlab-parallel-programming_1.md b/content/notes/matlab-parallel-programming/matlab-parallel-programming_1.md index cf780df5..94cb87b0 100644 --- a/content/notes/matlab-parallel-programming/matlab-parallel-programming_1.md +++ b/content/notes/matlab-parallel-programming/matlab-parallel-programming_1.md @@ -27,7 +27,7 @@ However, the challenge lies in effectively utilizing this powerful hardware, whi - **Accelerate workflows** with minimal changes to existing code - **Scale applications** seamlessly from desktop to clusters or cloud for more computational power and memory -{{< figure src=/notes/matlab-parallel-programming/img/matlab-gpu-multicore-cpu.png >}} +{{< figure src="/notes/matlab-parallel-programming/img/matlab-gpu-multicore-cpu.png" >}} Let's look at some examples of customer across multiple industries who have been using MathWorks parallel computing tools and why they chose parallel computing @@ -36,13 +36,13 @@ Let's look at some examples of customer across multiple industries who have been --- -| {{< figure src=/notes/matlab-parallel-programming/img/automotive-test-analysis.png width="300px" >}} | {{< figure src=/notes/matlab-parallel-programming/img/discrete-model-fleet-performance.png width="300px" >}} | +| {{< figure src="/notes/matlab-parallel-programming/img/automotive-test-analysis.png" width="300px" >}} | {{< figure src="/notes/matlab-parallel-programming/img/discrete-model-fleet-performance.png" width="300px" >}} | |:-----------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------:| | **Automotive Test Analysis and Visualization** | **Discrete-Event Model of Fleet Performance** | | - 3–4 months of development time saved | - Simulation time reduced from months to hours | | - Validation time sped up 2X | - Simulation time sped up 20X | -| {{< figure src=/notes/matlab-parallel-programming/img/heart-transplant-studies.png width="300px" >}} | {{< figure src=/notes/matlab-parallel-programming/img/derived-market-data.jpg width="300px" >}} | +| {{< figure src="/notes/matlab-parallel-programming/img/heart-transplant-studies.png" width="300px" >}} | {{< figure src="/notes/matlab-parallel-programming/img/derived-market-data.jpg" width="300px" >}} | |:-----------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------:| | **Heart Transplant Studies** | **Calculating Derived Market Data** | | - 4 weeks reduced to 5 days | - Implementation time reduced by months | diff --git a/content/notes/matlab-statistics/additionalRes1.png b/content/notes/matlab-statistics/additionalRes1.png deleted file mode 100644 index 1a05713a..00000000 Binary files a/content/notes/matlab-statistics/additionalRes1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/centrality1.png b/content/notes/matlab-statistics/centrality1.png deleted file mode 100644 index 36adf03e..00000000 Binary files a/content/notes/matlab-statistics/centrality1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/centrality2.png b/content/notes/matlab-statistics/centrality2.png deleted file mode 100644 index 77747144..00000000 Binary files a/content/notes/matlab-statistics/centrality2.png and /dev/null differ diff --git a/content/notes/matlab-statistics/centrality3.png b/content/notes/matlab-statistics/centrality3.png deleted file mode 100644 index a90ac400..00000000 Binary files a/content/notes/matlab-statistics/centrality3.png and /dev/null differ diff --git a/content/notes/matlab-statistics/distribution1.png b/content/notes/matlab-statistics/distribution1.png deleted file mode 100644 index 0f5497e1..00000000 Binary files a/content/notes/matlab-statistics/distribution1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/distribution2.png b/content/notes/matlab-statistics/distribution2.png deleted file mode 100644 index 5fa596e7..00000000 Binary files a/content/notes/matlab-statistics/distribution2.png and /dev/null differ diff --git a/content/notes/matlab-statistics/distribution3.png b/content/notes/matlab-statistics/distribution3.png deleted file mode 100644 index 664d8d5c..00000000 Binary files a/content/notes/matlab-statistics/distribution3.png and /dev/null differ diff --git a/content/notes/matlab-statistics/exploreData1.png b/content/notes/matlab-statistics/exploreData1.png deleted file mode 100644 index 89c5cac5..00000000 Binary files a/content/notes/matlab-statistics/exploreData1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/exploreData2.png b/content/notes/matlab-statistics/exploreData2.png deleted file mode 100644 index fc7ac5a5..00000000 Binary files a/content/notes/matlab-statistics/exploreData2.png and /dev/null differ diff --git a/content/notes/matlab-statistics/index.md b/content/notes/matlab-statistics/index.md deleted file mode 100644 index 64a64301..00000000 --- a/content/notes/matlab-statistics/index.md +++ /dev/null @@ -1,200 +0,0 @@ ---- -date : "2019-06-23T08:37:46-05:00" -title : "Statistical Methods with MATLAB" -type: article -toc : true ---- - -# Overview - -MATLAB is an integrated technical computing environment from the MathWorks that combines array-based numeric computation, advanced graphics and visualization, and a high-level programming language. Separately licensed toolboxes provide additional domain-specific functionality. - -**Matlab Academy: Statistical Methods with Matlab** - -**Documentation: Statistics and Machine Learning Toolbox (help page)** - -**Documentation: Statistics and Machine Learning Toolbox (product page)** -
-
- - -## Course Overview - -**Video: Statistical Methods with Matlab** - -## Exploring Data - -### Visualizing Data Sets -{{< figure src="exploreData1.png" >}} - -{{< figure src="exploreData2.png" >}} - -**Documentation: histogram ** ** boxplot ** ** scatter ** - -
-### Measures of Centrality and Spread -{{< figure src="centrality1.png" >}} -{{< figure src="centrality2.png" >}} -{{< figure src="centrality3.png" >}} - -** Documentation: mean ** ** median ** -** mode ** ** trimmean ** - -
- -{{< figure src="spread1.png" >}} -{{< figure src="spread2.png" >}} -{{< figure src="spread3.png" >}} - -** Documentation: std ** ** iqr ** -** range ** ** var ** - - -
- - - -### Distributions -{{< figure src="distribution3.png" >}} -{{< figure src="spread4.png" >}} - -
-{{< figure src="distribution1.png" >}} -{{< figure src="distribution2.png" >}} - -**Documentation: normpdf ** ** unifpdf ** ** randn ** ** rand ** - -### Summary -{{< figure src="summary1.png" >}} - - -
-{{< figure src="summary2.png" >}} - - -
- -{{< figure src="summary3.png" >}} - - - -
- -## Fitting a Curve to Data - -### Linear Regression -{{< figure src="regression1.png" >}} -{{< figure src="regression2.png" >}} -{{< figure src="regression4.png" >}} -{{< figure src="regression3.png" >}} - -** Documentation: fit ** - -
- -### Evaluating Goodness of Fit -{{< figure src="regression5.png" >}} -{{< figure src="regression6.png" >}} -{{< figure src="regression7.png" >}} - -
- -### Nonlinear Regression -{{< figure src="regression8.png" >}} -{{< figure src="regression9.png" >}} -{{< figure src="regression10.png" >}} - -### Summary -{{< figure src="summary4.png" >}} -{{< figure src="summary5.png" >}} -{{< figure src="summary6.png" >}} - -## Interpolating Data - -### Linear Interpolation -{{< figure src="interp1.png" >}} -{{< figure src="interp2.png" >}} -{{< figure src="interp3.png" >}} -{{< figure src="interp4.png" >}} -{{< figure src="interp5.png" >}} -{{< figure src="interp6.png" >}} - -### Nonlinear Interpolation - -{{< figure src="interp7.png" >}} -{{< figure src="interp8.png" >}} -{{< figure src="interp9.png" >}} - -** Documentation: interp1 ** - -
- -### Summary -{{< figure src="summary7.png" >}} -{{< figure src="summary8.png" >}} - -## Additional Resources - -{{< figure src="additionalRes1.png" >}} - -
-**MATLAB Central** ** MathWorks Support ** - -
- -## Exercises - -### Visualizing Data sets - -**Exercise: Visualize Height and Weight Data** - -### Measure of Centrality and Spread - -**Exercise: Find the Mean and Median** - -**Exercise: Find the Standard Deviation and IQR** - -### Distributions - -**Exercise: Fit and Plot a Normal Distribution** - -**Exercise: Generating Random Numbers** - -### Review: Exploring Data - -**Exercise: Earthquakes** - -### Linear Regression - -**Exercise: Fit a Line to Data** - -**Exercise: Fit a Polynomial to Data** - -### Evaluating the Goodness of Fit - -**Exercise: Evaluate and Improve the Fit** - - -
-
-### Nonlinear Regression - -**Exercise: Fit a Nonlinear Model** - -### Review: Fitting a Curve to Data - -**Exercise: Temperature Fluctuations** - -### Linear Interpolation - -**Exercise: Fill in Missing Data** - -**Exercise: Resample Data** - -### Nonlinear Interpolation - -**Exercise: Resample Data with Different Interpolation Methods** - -### Review: Interpolation - -**Exercise: Stock Prices** diff --git a/content/notes/matlab-statistics/interp1.png b/content/notes/matlab-statistics/interp1.png deleted file mode 100644 index 1b5705de..00000000 Binary files a/content/notes/matlab-statistics/interp1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interp2.png b/content/notes/matlab-statistics/interp2.png deleted file mode 100644 index a5914fd6..00000000 Binary files a/content/notes/matlab-statistics/interp2.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interp3.png b/content/notes/matlab-statistics/interp3.png deleted file mode 100644 index 59a2c0e8..00000000 Binary files a/content/notes/matlab-statistics/interp3.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interp4.png b/content/notes/matlab-statistics/interp4.png deleted file mode 100644 index 201f6a1c..00000000 Binary files a/content/notes/matlab-statistics/interp4.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interp5.png b/content/notes/matlab-statistics/interp5.png deleted file mode 100644 index 98d2c184..00000000 Binary files a/content/notes/matlab-statistics/interp5.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interp6.png b/content/notes/matlab-statistics/interp6.png deleted file mode 100644 index 61ace1d0..00000000 Binary files a/content/notes/matlab-statistics/interp6.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interp7.png b/content/notes/matlab-statistics/interp7.png deleted file mode 100644 index 502cf0ff..00000000 Binary files a/content/notes/matlab-statistics/interp7.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interp8.png b/content/notes/matlab-statistics/interp8.png deleted file mode 100644 index 842c8a97..00000000 Binary files a/content/notes/matlab-statistics/interp8.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interp9.png b/content/notes/matlab-statistics/interp9.png deleted file mode 100644 index d1f0f1dc..00000000 Binary files a/content/notes/matlab-statistics/interp9.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret1.png b/content/notes/matlab-statistics/interpret1.png deleted file mode 100644 index f39e9865..00000000 Binary files a/content/notes/matlab-statistics/interpret1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret10.png b/content/notes/matlab-statistics/interpret10.png deleted file mode 100644 index 45455bc7..00000000 Binary files a/content/notes/matlab-statistics/interpret10.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret2.png b/content/notes/matlab-statistics/interpret2.png deleted file mode 100644 index 58819e7f..00000000 Binary files a/content/notes/matlab-statistics/interpret2.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret3.png b/content/notes/matlab-statistics/interpret3.png deleted file mode 100644 index 6ea1ec8c..00000000 Binary files a/content/notes/matlab-statistics/interpret3.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret4.png b/content/notes/matlab-statistics/interpret4.png deleted file mode 100644 index f38c6543..00000000 Binary files a/content/notes/matlab-statistics/interpret4.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret5.png b/content/notes/matlab-statistics/interpret5.png deleted file mode 100644 index 26faaa66..00000000 Binary files a/content/notes/matlab-statistics/interpret5.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret6.png b/content/notes/matlab-statistics/interpret6.png deleted file mode 100644 index 3b643931..00000000 Binary files a/content/notes/matlab-statistics/interpret6.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret7.png b/content/notes/matlab-statistics/interpret7.png deleted file mode 100644 index 1bcf6ffe..00000000 Binary files a/content/notes/matlab-statistics/interpret7.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret8.png b/content/notes/matlab-statistics/interpret8.png deleted file mode 100644 index f17b453b..00000000 Binary files a/content/notes/matlab-statistics/interpret8.png and /dev/null differ diff --git a/content/notes/matlab-statistics/interpret9.png b/content/notes/matlab-statistics/interpret9.png deleted file mode 100644 index 33fb8f31..00000000 Binary files a/content/notes/matlab-statistics/interpret9.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression1.png b/content/notes/matlab-statistics/regression1.png deleted file mode 100644 index ddba8446..00000000 Binary files a/content/notes/matlab-statistics/regression1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression10.png b/content/notes/matlab-statistics/regression10.png deleted file mode 100644 index 5cbb9007..00000000 Binary files a/content/notes/matlab-statistics/regression10.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression2.png b/content/notes/matlab-statistics/regression2.png deleted file mode 100644 index 2feaae00..00000000 Binary files a/content/notes/matlab-statistics/regression2.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression3.png b/content/notes/matlab-statistics/regression3.png deleted file mode 100644 index c884b3e9..00000000 Binary files a/content/notes/matlab-statistics/regression3.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression4.png b/content/notes/matlab-statistics/regression4.png deleted file mode 100644 index 6a2af1ad..00000000 Binary files a/content/notes/matlab-statistics/regression4.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression5.png b/content/notes/matlab-statistics/regression5.png deleted file mode 100644 index 82ff00c1..00000000 Binary files a/content/notes/matlab-statistics/regression5.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression6.png b/content/notes/matlab-statistics/regression6.png deleted file mode 100644 index 152706e3..00000000 Binary files a/content/notes/matlab-statistics/regression6.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression7.png b/content/notes/matlab-statistics/regression7.png deleted file mode 100644 index 985743b1..00000000 Binary files a/content/notes/matlab-statistics/regression7.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression8.png b/content/notes/matlab-statistics/regression8.png deleted file mode 100644 index 1e87519b..00000000 Binary files a/content/notes/matlab-statistics/regression8.png and /dev/null differ diff --git a/content/notes/matlab-statistics/regression9.png b/content/notes/matlab-statistics/regression9.png deleted file mode 100644 index 44257c9b..00000000 Binary files a/content/notes/matlab-statistics/regression9.png and /dev/null differ diff --git a/content/notes/matlab-statistics/spread1.png b/content/notes/matlab-statistics/spread1.png deleted file mode 100644 index 1b3960ef..00000000 Binary files a/content/notes/matlab-statistics/spread1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/spread2.png b/content/notes/matlab-statistics/spread2.png deleted file mode 100644 index f1fa8750..00000000 Binary files a/content/notes/matlab-statistics/spread2.png and /dev/null differ diff --git a/content/notes/matlab-statistics/spread3.png b/content/notes/matlab-statistics/spread3.png deleted file mode 100644 index 10729062..00000000 Binary files a/content/notes/matlab-statistics/spread3.png and /dev/null differ diff --git a/content/notes/matlab-statistics/spread4.png b/content/notes/matlab-statistics/spread4.png deleted file mode 100644 index b49fda81..00000000 Binary files a/content/notes/matlab-statistics/spread4.png and /dev/null differ diff --git a/content/notes/matlab-statistics/summary1.png b/content/notes/matlab-statistics/summary1.png deleted file mode 100644 index cea0add6..00000000 Binary files a/content/notes/matlab-statistics/summary1.png and /dev/null differ diff --git a/content/notes/matlab-statistics/summary2.png b/content/notes/matlab-statistics/summary2.png deleted file mode 100644 index 70a08ba4..00000000 Binary files a/content/notes/matlab-statistics/summary2.png and /dev/null differ diff --git a/content/notes/matlab-statistics/summary3.png b/content/notes/matlab-statistics/summary3.png deleted file mode 100644 index 67653041..00000000 Binary files a/content/notes/matlab-statistics/summary3.png and /dev/null differ diff --git a/content/notes/matlab-statistics/summary4.png b/content/notes/matlab-statistics/summary4.png deleted file mode 100644 index 3784c5e4..00000000 Binary files a/content/notes/matlab-statistics/summary4.png and /dev/null differ diff --git a/content/notes/matlab-statistics/summary5.png b/content/notes/matlab-statistics/summary5.png deleted file mode 100644 index a8fb2afb..00000000 Binary files a/content/notes/matlab-statistics/summary5.png and /dev/null differ diff --git a/content/notes/matlab-statistics/summary6.png b/content/notes/matlab-statistics/summary6.png deleted file mode 100644 index 5f6f120b..00000000 Binary files a/content/notes/matlab-statistics/summary6.png and /dev/null differ diff --git a/content/notes/matlab-statistics/summary7.png b/content/notes/matlab-statistics/summary7.png deleted file mode 100644 index ec444cbc..00000000 Binary files a/content/notes/matlab-statistics/summary7.png and /dev/null differ diff --git a/content/notes/matlab-statistics/summary8.png b/content/notes/matlab-statistics/summary8.png deleted file mode 100644 index 2e527de7..00000000 Binary files a/content/notes/matlab-statistics/summary8.png and /dev/null differ diff --git a/content/courses/python-high-performance/.f2py_f2cmap b/content/notes/python-high-performance/.f2py_f2cmap similarity index 100% rename from content/courses/python-high-performance/.f2py_f2cmap rename to content/notes/python-high-performance/.f2py_f2cmap diff --git a/content/courses/python-high-performance/AnacondaNavigatorFastX.png b/content/notes/python-high-performance/AnacondaNavigatorFastX.png similarity index 100% rename from content/courses/python-high-performance/AnacondaNavigatorFastX.png rename to content/notes/python-high-performance/AnacondaNavigatorFastX.png diff --git a/content/courses/python-high-performance/DMP.png b/content/notes/python-high-performance/DMP.png similarity index 100% rename from content/courses/python-high-performance/DMP.png rename to content/notes/python-high-performance/DMP.png diff --git a/content/courses/python-high-performance/SMP.png b/content/notes/python-high-performance/SMP.png similarity index 100% rename from content/courses/python-high-performance/SMP.png rename to content/notes/python-high-performance/SMP.png diff --git a/content/courses/python-high-performance/_index.md b/content/notes/python-high-performance/_index.md similarity index 99% rename from content/courses/python-high-performance/_index.md rename to content/notes/python-high-performance/_index.md index 96827934..60806e31 100644 --- a/content/courses/python-high-performance/_index.md +++ b/content/notes/python-high-performance/_index.md @@ -5,10 +5,10 @@ toc: true date: "2020-11-17T00:00:00" tags: ["Programming","Python","HPC"] categories: ["Programming","Python","HPC"] +weight: 1 menu: hp-python: name: High-Performance Python - weight: 1 --- Python, like most interpreted languages, can be very slow. But there are best practices and some programming tricks that can speed it up considerably. This can make the difference between finishing the work in an acceptable time, or being unable to finish a project. diff --git a/content/courses/python-high-performance/codes/.ipynb_checkpoints/Untitled-checkpoint.ipynb b/content/notes/python-high-performance/codes/.ipynb_checkpoints/Untitled-checkpoint.ipynb similarity index 100% rename from content/courses/python-high-performance/codes/.ipynb_checkpoints/Untitled-checkpoint.ipynb rename to content/notes/python-high-performance/codes/.ipynb_checkpoints/Untitled-checkpoint.ipynb diff --git a/content/courses/python-high-performance/codes/.ipynb_checkpoints/dask_delayed-checkpoint.py b/content/notes/python-high-performance/codes/.ipynb_checkpoints/dask_delayed-checkpoint.py similarity index 100% rename from content/courses/python-high-performance/codes/.ipynb_checkpoints/dask_delayed-checkpoint.py rename to content/notes/python-high-performance/codes/.ipynb_checkpoints/dask_delayed-checkpoint.py diff --git a/content/courses/python-high-performance/codes/.virtual_documents/Untitled.ipynb b/content/notes/python-high-performance/codes/.virtual_documents/Untitled.ipynb similarity index 100% rename from content/courses/python-high-performance/codes/.virtual_documents/Untitled.ipynb rename to content/notes/python-high-performance/codes/.virtual_documents/Untitled.ipynb diff --git a/content/courses/python-high-performance/codes/MonteCarlo.py b/content/notes/python-high-performance/codes/MonteCarlo.py similarity index 100% rename from content/courses/python-high-performance/codes/MonteCarlo.py rename to content/notes/python-high-performance/codes/MonteCarlo.py diff --git a/content/courses/python-high-performance/codes/MonteCarloPi.py b/content/notes/python-high-performance/codes/MonteCarloPi.py similarity index 100% rename from content/courses/python-high-performance/codes/MonteCarloPi.py rename to content/notes/python-high-performance/codes/MonteCarloPi.py diff --git a/content/courses/python-high-performance/codes/MonteCarloPiMC.py b/content/notes/python-high-performance/codes/MonteCarloPiMC.py similarity index 100% rename from content/courses/python-high-performance/codes/MonteCarloPiMC.py rename to content/notes/python-high-performance/codes/MonteCarloPiMC.py diff --git a/content/courses/python-high-performance/codes/MonteCarloPiMPI.py b/content/notes/python-high-performance/codes/MonteCarloPiMPI.py similarity index 100% rename from content/courses/python-high-performance/codes/MonteCarloPiMPI.py rename to content/notes/python-high-performance/codes/MonteCarloPiMPI.py diff --git a/content/courses/python-high-performance/codes/Untitled.ipynb b/content/notes/python-high-performance/codes/Untitled.ipynb similarity index 100% rename from content/courses/python-high-performance/codes/Untitled.ipynb rename to content/notes/python-high-performance/codes/Untitled.ipynb diff --git a/content/notes/python-high-performance/codes/__pycache__/dask_mpi.cpython-39.pyc b/content/notes/python-high-performance/codes/__pycache__/dask_mpi.cpython-39.pyc new file mode 100644 index 00000000..341425c0 Binary files /dev/null and b/content/notes/python-high-performance/codes/__pycache__/dask_mpi.cpython-39.pyc differ diff --git a/content/notes/python-high-performance/codes/__pycache__/fibonacci.cpython-39.pyc b/content/notes/python-high-performance/codes/__pycache__/fibonacci.cpython-39.pyc new file mode 100644 index 00000000..baa22610 Binary files /dev/null and b/content/notes/python-high-performance/codes/__pycache__/fibonacci.cpython-39.pyc differ diff --git a/content/courses/python-high-performance/codes/aops.py b/content/notes/python-high-performance/codes/aops.py similarity index 100% rename from content/courses/python-high-performance/codes/aops.py rename to content/notes/python-high-performance/codes/aops.py diff --git a/content/courses/python-high-performance/codes/arith.c b/content/notes/python-high-performance/codes/arith.c similarity index 100% rename from content/courses/python-high-performance/codes/arith.c rename to content/notes/python-high-performance/codes/arith.c diff --git a/content/courses/python-high-performance/codes/arith.h b/content/notes/python-high-performance/codes/arith.h similarity index 100% rename from content/courses/python-high-performance/codes/arith.h rename to content/notes/python-high-performance/codes/arith.h diff --git a/content/courses/python-high-performance/codes/build/temp.linux-x86_64-3.9/integrate_cyf.o b/content/notes/python-high-performance/codes/build/temp.linux-x86_64-3.9/integrate_cyf.o similarity index 100% rename from content/courses/python-high-performance/codes/build/temp.linux-x86_64-3.9/integrate_cyf.o rename to content/notes/python-high-performance/codes/build/temp.linux-x86_64-3.9/integrate_cyf.o diff --git a/content/courses/python-high-performance/codes/build_arith.py b/content/notes/python-high-performance/codes/build_arith.py similarity index 100% rename from content/courses/python-high-performance/codes/build_arith.py rename to content/notes/python-high-performance/codes/build_arith.py diff --git a/content/courses/python-high-performance/codes/cupy_example.py b/content/notes/python-high-performance/codes/cupy_example.py similarity index 100% rename from content/courses/python-high-performance/codes/cupy_example.py rename to content/notes/python-high-performance/codes/cupy_example.py diff --git a/content/courses/python-high-performance/codes/da_mean.png b/content/notes/python-high-performance/codes/da_mean.png similarity index 100% rename from content/courses/python-high-performance/codes/da_mean.png rename to content/notes/python-high-performance/codes/da_mean.png diff --git a/content/courses/python-high-performance/codes/dask_delayed.py b/content/notes/python-high-performance/codes/dask_delayed.py similarity index 100% rename from content/courses/python-high-performance/codes/dask_delayed.py rename to content/notes/python-high-performance/codes/dask_delayed.py diff --git a/content/courses/python-high-performance/codes/dask_df_example.py b/content/notes/python-high-performance/codes/dask_df_example.py similarity index 100% rename from content/courses/python-high-performance/codes/dask_df_example.py rename to content/notes/python-high-performance/codes/dask_df_example.py diff --git a/content/courses/python-high-performance/codes/dask_df_mpi.py b/content/notes/python-high-performance/codes/dask_df_mpi.py similarity index 100% rename from content/courses/python-high-performance/codes/dask_df_mpi.py rename to content/notes/python-high-performance/codes/dask_df_mpi.py diff --git a/content/courses/python-high-performance/codes/dask_dist_example.py b/content/notes/python-high-performance/codes/dask_dist_example.py similarity index 100% rename from content/courses/python-high-performance/codes/dask_dist_example.py rename to content/notes/python-high-performance/codes/dask_dist_example.py diff --git a/content/courses/python-high-performance/codes/dask_mpi_example.py b/content/notes/python-high-performance/codes/dask_mpi_example.py similarity index 100% rename from content/courses/python-high-performance/codes/dask_mpi_example.py rename to content/notes/python-high-performance/codes/dask_mpi_example.py diff --git a/content/courses/python-high-performance/codes/dask_mpi_futures.py b/content/notes/python-high-performance/codes/dask_mpi_futures.py similarity index 100% rename from content/courses/python-high-performance/codes/dask_mpi_futures.py rename to content/notes/python-high-performance/codes/dask_mpi_futures.py diff --git a/content/courses/python-high-performance/codes/dask_mpi_init.py b/content/notes/python-high-performance/codes/dask_mpi_init.py similarity index 100% rename from content/courses/python-high-performance/codes/dask_mpi_init.py rename to content/notes/python-high-performance/codes/dask_mpi_init.py diff --git a/content/courses/python-high-performance/codes/dask_viz.py b/content/notes/python-high-performance/codes/dask_viz.py similarity index 100% rename from content/courses/python-high-performance/codes/dask_viz.py rename to content/notes/python-high-performance/codes/dask_viz.py diff --git a/content/courses/python-high-performance/codes/fibonacci-cProfile.py b/content/notes/python-high-performance/codes/fibonacci-cProfile.py similarity index 100% rename from content/courses/python-high-performance/codes/fibonacci-cProfile.py rename to content/notes/python-high-performance/codes/fibonacci-cProfile.py diff --git a/content/courses/python-high-performance/codes/fibonacci.py b/content/notes/python-high-performance/codes/fibonacci.py similarity index 100% rename from content/courses/python-high-performance/codes/fibonacci.py rename to content/notes/python-high-performance/codes/fibonacci.py diff --git a/content/courses/python-high-performance/codes/fibonacci.py.lprof b/content/notes/python-high-performance/codes/fibonacci.py.lprof similarity index 100% rename from content/courses/python-high-performance/codes/fibonacci.py.lprof rename to content/notes/python-high-performance/codes/fibonacci.py.lprof diff --git a/content/courses/python-high-performance/codes/fibonacci_direct.py b/content/notes/python-high-performance/codes/fibonacci_direct.py similarity index 100% rename from content/courses/python-high-performance/codes/fibonacci_direct.py rename to content/notes/python-high-performance/codes/fibonacci_direct.py diff --git a/content/courses/python-high-performance/codes/fractions.cxx b/content/notes/python-high-performance/codes/fractions.cxx similarity index 100% rename from content/courses/python-high-performance/codes/fractions.cxx rename to content/notes/python-high-performance/codes/fractions.cxx diff --git a/content/courses/python-high-performance/codes/fractions.f90 b/content/notes/python-high-performance/codes/fractions.f90 similarity index 100% rename from content/courses/python-high-performance/codes/fractions.f90 rename to content/notes/python-high-performance/codes/fractions.f90 diff --git a/content/courses/python-high-performance/codes/fractions.h b/content/notes/python-high-performance/codes/fractions.h similarity index 100% rename from content/courses/python-high-performance/codes/fractions.h rename to content/notes/python-high-performance/codes/fractions.h diff --git a/content/courses/python-high-performance/codes/groupby.png b/content/notes/python-high-performance/codes/groupby.png similarity index 100% rename from content/courses/python-high-performance/codes/groupby.png rename to content/notes/python-high-performance/codes/groupby.png diff --git a/content/courses/python-high-performance/codes/integrate.py b/content/notes/python-high-performance/codes/integrate.py similarity index 100% rename from content/courses/python-high-performance/codes/integrate.py rename to content/notes/python-high-performance/codes/integrate.py diff --git a/content/courses/python-high-performance/codes/integrate_cyf.c b/content/notes/python-high-performance/codes/integrate_cyf.c similarity index 100% rename from content/courses/python-high-performance/codes/integrate_cyf.c rename to content/notes/python-high-performance/codes/integrate_cyf.c diff --git a/content/courses/python-high-performance/codes/integrate_cyf.cpython-39-x86_64-linux-gnu.so b/content/notes/python-high-performance/codes/integrate_cyf.cpython-39-x86_64-linux-gnu.so similarity index 100% rename from content/courses/python-high-performance/codes/integrate_cyf.cpython-39-x86_64-linux-gnu.so rename to content/notes/python-high-performance/codes/integrate_cyf.cpython-39-x86_64-linux-gnu.so diff --git a/content/courses/python-high-performance/codes/integrate_cyf.pyx b/content/notes/python-high-performance/codes/integrate_cyf.pyx similarity index 100% rename from content/courses/python-high-performance/codes/integrate_cyf.pyx rename to content/notes/python-high-performance/codes/integrate_cyf.pyx diff --git a/content/courses/python-high-performance/codes/integrate_cyf_test.py b/content/notes/python-high-performance/codes/integrate_cyf_test.py similarity index 100% rename from content/courses/python-high-performance/codes/integrate_cyf_test.py rename to content/notes/python-high-performance/codes/integrate_cyf_test.py diff --git a/content/courses/python-high-performance/codes/loops.py b/content/notes/python-high-performance/codes/loops.py similarity index 100% rename from content/courses/python-high-performance/codes/loops.py rename to content/notes/python-high-performance/codes/loops.py diff --git a/content/courses/python-high-performance/codes/mppool.py b/content/notes/python-high-performance/codes/mppool.py similarity index 100% rename from content/courses/python-high-performance/codes/mppool.py rename to content/notes/python-high-performance/codes/mppool.py diff --git a/content/courses/python-high-performance/codes/mpprocess.py b/content/notes/python-high-performance/codes/mpprocess.py similarity index 100% rename from content/courses/python-high-performance/codes/mpprocess.py rename to content/notes/python-high-performance/codes/mpprocess.py diff --git a/content/courses/python-high-performance/codes/mpstarmap.py b/content/notes/python-high-performance/codes/mpstarmap.py similarity index 100% rename from content/courses/python-high-performance/codes/mpstarmap.py rename to content/notes/python-high-performance/codes/mpstarmap.py diff --git a/content/courses/python-high-performance/codes/numba_cuda_example.py b/content/notes/python-high-performance/codes/numba_cuda_example.py similarity index 100% rename from content/courses/python-high-performance/codes/numba_cuda_example.py rename to content/notes/python-high-performance/codes/numba_cuda_example.py diff --git a/content/courses/python-high-performance/codes/numba_vectorize.py b/content/notes/python-high-performance/codes/numba_vectorize.py similarity index 100% rename from content/courses/python-high-performance/codes/numba_vectorize.py rename to content/notes/python-high-performance/codes/numba_vectorize.py diff --git a/content/courses/python-high-performance/codes/numba_vectorize_todevice.py b/content/notes/python-high-performance/codes/numba_vectorize_todevice.py similarity index 100% rename from content/courses/python-high-performance/codes/numba_vectorize_todevice.py rename to content/notes/python-high-performance/codes/numba_vectorize_todevice.py diff --git a/content/courses/python-high-performance/codes/pimc.sh b/content/notes/python-high-performance/codes/pimc.sh similarity index 100% rename from content/courses/python-high-performance/codes/pimc.sh rename to content/notes/python-high-performance/codes/pimc.sh diff --git a/content/courses/python-high-performance/codes/plot_mp_scaling.py b/content/notes/python-high-performance/codes/plot_mp_scaling.py similarity index 100% rename from content/courses/python-high-performance/codes/plot_mp_scaling.py rename to content/notes/python-high-performance/codes/plot_mp_scaling.py diff --git a/content/courses/python-high-performance/codes/pycuda_example.py b/content/notes/python-high-performance/codes/pycuda_example.py similarity index 100% rename from content/courses/python-high-performance/codes/pycuda_example.py rename to content/notes/python-high-performance/codes/pycuda_example.py diff --git a/content/courses/python-high-performance/codes/pycuda_numba.py b/content/notes/python-high-performance/codes/pycuda_numba.py similarity index 100% rename from content/courses/python-high-performance/codes/pycuda_numba.py rename to content/notes/python-high-performance/codes/pycuda_numba.py diff --git a/content/courses/python-high-performance/codes/pympi.slurm b/content/notes/python-high-performance/codes/pympi.slurm similarity index 100% rename from content/courses/python-high-performance/codes/pympi.slurm rename to content/notes/python-high-performance/codes/pympi.slurm diff --git a/content/courses/python-high-performance/codes/replace_forloop.py b/content/notes/python-high-performance/codes/replace_forloop.py similarity index 100% rename from content/courses/python-high-performance/codes/replace_forloop.py rename to content/notes/python-high-performance/codes/replace_forloop.py diff --git a/content/courses/python-high-performance/codes/replace_forloop_comp.py b/content/notes/python-high-performance/codes/replace_forloop_comp.py similarity index 100% rename from content/courses/python-high-performance/codes/replace_forloop_comp.py rename to content/notes/python-high-performance/codes/replace_forloop_comp.py diff --git a/content/courses/python-high-performance/codes/run_dask_mpi.slurm b/content/notes/python-high-performance/codes/run_dask_mpi.slurm similarity index 100% rename from content/courses/python-high-performance/codes/run_dask_mpi.slurm rename to content/notes/python-high-performance/codes/run_dask_mpi.slurm diff --git a/content/courses/python-high-performance/codes/setup.py b/content/notes/python-high-performance/codes/setup.py similarity index 100% rename from content/courses/python-high-performance/codes/setup.py rename to content/notes/python-high-performance/codes/setup.py diff --git a/content/courses/python-high-performance/codes/tasks.py b/content/notes/python-high-performance/codes/tasks.py similarity index 100% rename from content/courses/python-high-performance/codes/tasks.py rename to content/notes/python-high-performance/codes/tasks.py diff --git a/content/courses/python-high-performance/codes/testdecorator.py b/content/notes/python-high-performance/codes/testdecorator.py similarity index 100% rename from content/courses/python-high-performance/codes/testdecorator.py rename to content/notes/python-high-performance/codes/testdecorator.py diff --git a/content/courses/python-high-performance/codes/wrap_fractions.cxx b/content/notes/python-high-performance/codes/wrap_fractions.cxx similarity index 100% rename from content/courses/python-high-performance/codes/wrap_fractions.cxx rename to content/notes/python-high-performance/codes/wrap_fractions.cxx diff --git a/content/courses/python-high-performance/compiled_code.md b/content/notes/python-high-performance/compiled_code.md similarity index 100% rename from content/courses/python-high-performance/compiled_code.md rename to content/notes/python-high-performance/compiled_code.md diff --git a/content/courses/python-high-performance/data.zip b/content/notes/python-high-performance/data.zip similarity index 100% rename from content/courses/python-high-performance/data.zip rename to content/notes/python-high-performance/data.zip diff --git a/content/courses/python-high-performance/data/0.json b/content/notes/python-high-performance/data/0.json similarity index 100% rename from content/courses/python-high-performance/data/0.json rename to content/notes/python-high-performance/data/0.json diff --git a/content/courses/python-high-performance/data/1.json b/content/notes/python-high-performance/data/1.json similarity index 100% rename from content/courses/python-high-performance/data/1.json rename to content/notes/python-high-performance/data/1.json diff --git a/content/courses/python-high-performance/data/2.json b/content/notes/python-high-performance/data/2.json similarity index 100% rename from content/courses/python-high-performance/data/2.json rename to content/notes/python-high-performance/data/2.json diff --git a/content/courses/python-high-performance/data/3.json b/content/notes/python-high-performance/data/3.json similarity index 100% rename from content/courses/python-high-performance/data/3.json rename to content/notes/python-high-performance/data/3.json diff --git a/content/courses/python-high-performance/data/4.json b/content/notes/python-high-performance/data/4.json similarity index 100% rename from content/courses/python-high-performance/data/4.json rename to content/notes/python-high-performance/data/4.json diff --git a/content/courses/python-high-performance/data/5.json b/content/notes/python-high-performance/data/5.json similarity index 100% rename from content/courses/python-high-performance/data/5.json rename to content/notes/python-high-performance/data/5.json diff --git a/content/courses/python-high-performance/data/6.json b/content/notes/python-high-performance/data/6.json similarity index 100% rename from content/courses/python-high-performance/data/6.json rename to content/notes/python-high-performance/data/6.json diff --git a/content/courses/python-high-performance/data/7.json b/content/notes/python-high-performance/data/7.json similarity index 100% rename from content/courses/python-high-performance/data/7.json rename to content/notes/python-high-performance/data/7.json diff --git a/content/courses/python-high-performance/data/8.json b/content/notes/python-high-performance/data/8.json similarity index 100% rename from content/courses/python-high-performance/data/8.json rename to content/notes/python-high-performance/data/8.json diff --git a/content/courses/python-high-performance/data/9.json b/content/notes/python-high-performance/data/9.json similarity index 100% rename from content/courses/python-high-performance/data/9.json rename to content/notes/python-high-performance/data/9.json diff --git a/content/courses/python-high-performance/distributed-parallelization.md b/content/notes/python-high-performance/distributed-parallelization.md similarity index 100% rename from content/courses/python-high-performance/distributed-parallelization.md rename to content/notes/python-high-performance/distributed-parallelization.md diff --git a/content/courses/python-high-performance/fibonacci-profiler.png b/content/notes/python-high-performance/fibonacci-profiler.png similarity index 100% rename from content/courses/python-high-performance/fibonacci-profiler.png rename to content/notes/python-high-performance/fibonacci-profiler.png diff --git a/content/courses/python-high-performance/gpu_acceleration.md b/content/notes/python-high-performance/gpu_acceleration.md similarity index 100% rename from content/courses/python-high-performance/gpu_acceleration.md rename to content/notes/python-high-performance/gpu_acceleration.md diff --git a/content/courses/python-high-performance/mp-scaling.png b/content/notes/python-high-performance/mp-scaling.png similarity index 100% rename from content/courses/python-high-performance/mp-scaling.png rename to content/notes/python-high-performance/mp-scaling.png diff --git a/content/courses/python-high-performance/mpi-parallelization.md b/content/notes/python-high-performance/mpi-parallelization.md similarity index 100% rename from content/courses/python-high-performance/mpi-parallelization.md rename to content/notes/python-high-performance/mpi-parallelization.md diff --git a/content/courses/python-high-performance/multiprocessing.md b/content/notes/python-high-performance/multiprocessing.md similarity index 100% rename from content/courses/python-high-performance/multiprocessing.md rename to content/notes/python-high-performance/multiprocessing.md diff --git a/content/courses/python-high-performance/profiling-timing.md b/content/notes/python-high-performance/profiling-timing.md similarity index 100% rename from content/courses/python-high-performance/profiling-timing.md rename to content/notes/python-high-performance/profiling-timing.md diff --git a/content/courses/python-high-performance/serial-optimization.md b/content/notes/python-high-performance/serial-optimization.md similarity index 94% rename from content/courses/python-high-performance/serial-optimization.md rename to content/notes/python-high-performance/serial-optimization.md index fa513084..427afd5f 100644 --- a/content/courses/python-high-performance/serial-optimization.md +++ b/content/notes/python-high-performance/serial-optimization.md @@ -11,7 +11,7 @@ menu: We can represent the optimization process with a flowchart: -{{< diagram >}} +{{< diagram alt="Flowchart of the optimization process: profile code, tune the slowest section, check for performance improvement, then repeat or try a different solution." >}} graph TD; A(Profile or time) --> B(Tune the slowest section); B --> C{Performance increase?}; @@ -20,7 +20,7 @@ C -- No --> E(Try a different solution); D --> A {{< /diagram >}} -There are many approaches to speeding up sections, some specific to Python and some more generic. In this section we will consider several possibilities. +There are many many approaches to speeding up sections, some specific to Python and some more generic. In this section we will consider several possibilities. ### Avoid for Loops diff --git a/content/courses/python-high-performance/testdecorator.py.prof b/content/notes/python-high-performance/testdecorator.py.prof similarity index 100% rename from content/courses/python-high-performance/testdecorator.py.prof rename to content/notes/python-high-performance/testdecorator.py.prof diff --git a/content/notes/python-machine-learning/_index.md b/content/notes/python-machine-learning/_index.md index 23dfae59..5091e17b 100644 --- a/content/notes/python-machine-learning/_index.md +++ b/content/notes/python-machine-learning/_index.md @@ -26,4 +26,4 @@ In this tutorial we will be covering the following topics: As mentioned above, example codes will be provided for respective topics. Prior experience with the Python programming language and some familiarity with machine learning concepts are helpful for this tutorial. Please download and unzip the following file to follow along on code activities. -{{< file-download file="notes/python-machine-learning/code/ML_with_Python.zip" text="ML_with_Python.zip" >}} +{{< file-download file="/notes/python-machine-learning/code/ML_with_Python.zip" text="ML_with_Python.zip" >}} diff --git a/content/courses/pytorch-hpc/_index.md b/content/notes/pytorch-hpc/_index.md similarity index 94% rename from content/courses/pytorch-hpc/_index.md rename to content/notes/pytorch-hpc/_index.md index b9501ae8..8b5e00c0 100644 --- a/content/courses/pytorch-hpc/_index.md +++ b/content/notes/pytorch-hpc/_index.md @@ -1,14 +1,15 @@ --- date : "2025-02-26T00:00:00-05:00" title : "Introduction to PyTorch for HPC" -summary: "An introduction to devlopment with PyTorch and HPC." -authors: [abd] +#summary: "An introduction to using PyTorch on an HPC system." categories: ["Deep Learning","Python","HPC","Machine Learning"] tags: [Deep_learning,Machine_learning,Python,HPC] toc: true -type: book +type: docs weight: 1 - +menu: + pytorch-hpc: + name: Pytorch for HPC --- ### Overview diff --git a/content/notes/pytorch-hpc/basics.md b/content/notes/pytorch-hpc/basics.md deleted file mode 100644 index e69de29b..00000000 diff --git a/content/courses/pytorch-hpc/gpu.md b/content/notes/pytorch-hpc/gpu.md similarity index 97% rename from content/courses/pytorch-hpc/gpu.md rename to content/notes/pytorch-hpc/gpu.md index e7472e64..dcc98248 100644 --- a/content/courses/pytorch-hpc/gpu.md +++ b/content/notes/pytorch-hpc/gpu.md @@ -2,8 +2,12 @@ date : "2025-02-26T00:00:00" title: "GPU Acceleration for Deep Learning" toc: true -type: book +type: docs + weight: 15 +menu: + pytorch-hpc: + parent: Pytorch for HPC --- @@ -93,4 +97,4 @@ Alternatively you can use the [SLURM Script Generator](https://www.rc.virginia.e For more information on SLURM visit: https://www.rc.virginia.edu/userinfo/hpc/software/pytorch/, https://www.rc.virginia.edu/userinfo/hpc/slurm/ -For information on multi-GPU use: https://pytorch.org/tutorials/beginner/ddp_series_multigpu.html \ No newline at end of file +For information on multi-GPU use: https://pytorch.org/tutorials/beginner/ddp_series_multigpu.html diff --git a/content/courses/pytorch-hpc/img/leakyrelu.png b/content/notes/pytorch-hpc/img/leakyrelu.png similarity index 100% rename from content/courses/pytorch-hpc/img/leakyrelu.png rename to content/notes/pytorch-hpc/img/leakyrelu.png diff --git a/content/courses/pytorch-hpc/img/nn.png b/content/notes/pytorch-hpc/img/nn.png similarity index 100% rename from content/courses/pytorch-hpc/img/nn.png rename to content/notes/pytorch-hpc/img/nn.png diff --git a/content/courses/pytorch-hpc/img/relu.png b/content/notes/pytorch-hpc/img/relu.png similarity index 100% rename from content/courses/pytorch-hpc/img/relu.png rename to content/notes/pytorch-hpc/img/relu.png diff --git a/content/courses/pytorch-hpc/img/sigmoid.png b/content/notes/pytorch-hpc/img/sigmoid.png similarity index 100% rename from content/courses/pytorch-hpc/img/sigmoid.png rename to content/notes/pytorch-hpc/img/sigmoid.png diff --git a/content/courses/pytorch-hpc/img/tanh.png b/content/notes/pytorch-hpc/img/tanh.png similarity index 100% rename from content/courses/pytorch-hpc/img/tanh.png rename to content/notes/pytorch-hpc/img/tanh.png diff --git a/content/notes/pytorch-hpc/index.md b/content/notes/pytorch-hpc/index.md deleted file mode 100644 index 8eb700bb..00000000 --- a/content/notes/pytorch-hpc/index.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -# Course title, summary, and position. -title: "Introduction to Pytorch for HPC" -authors: [abd] - -# Page metadata. -date: "2025-02-26T00:00:00" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. - -# Add menu entry to sidebar. -menu: - pytorch-hpc: - name: Introduction - weight: 5 ---- - - - - diff --git a/content/courses/pytorch-hpc/optimize.md b/content/notes/pytorch-hpc/optimize.md similarity index 98% rename from content/courses/pytorch-hpc/optimize.md rename to content/notes/pytorch-hpc/optimize.md index 9d2f5843..cc552acf 100644 --- a/content/courses/pytorch-hpc/optimize.md +++ b/content/notes/pytorch-hpc/optimize.md @@ -2,9 +2,11 @@ date : "2025-02-26T00:00:00" title: "Optimizing Neural Networks" toc: true -type: book +type: docs weight: 25 - +menu: + pytorch-hpc: + parent: Pytorch for HPC --- diff --git a/content/courses/pytorch-hpc/overview.md b/content/notes/pytorch-hpc/overview.md similarity index 99% rename from content/courses/pytorch-hpc/overview.md rename to content/notes/pytorch-hpc/overview.md index 60e66692..17fc4af2 100644 --- a/content/courses/pytorch-hpc/overview.md +++ b/content/notes/pytorch-hpc/overview.md @@ -2,9 +2,12 @@ date : "2025-02-26T00:00:00" title: "Introduction to PyTorch" toc: true -type: book +type: docs weight: 5 +menu: + pytorch-hpc: + parent: Pytorch for HPC --- diff --git a/content/courses/pytorch-hpc/project.md b/content/notes/pytorch-hpc/project.md similarity index 97% rename from content/courses/pytorch-hpc/project.md rename to content/notes/pytorch-hpc/project.md index c96fb1a7..54f5c3b1 100644 --- a/content/courses/pytorch-hpc/project.md +++ b/content/notes/pytorch-hpc/project.md @@ -2,11 +2,14 @@ date : "2025-02-26T00:00:00" title: "PyTorch Experimentation and Project Setup" toc: true -type: book +type: docs weight: 30 ---- +menu: + pytorch-hpc: + parent: Pytorch for HPC +--- ### **Structuring a PyTorch Project** A well-organized project structure makes debugging, collaboration, and experimentation easier. @@ -28,7 +31,6 @@ pytorch_project/ ├── slurm_train.sh # SLURM script for HPC training └── README.md # Documentation ``` ---- ### Managing Experiments with Logging Frameworks Tracking experiments is essential for understanding model performance. @@ -85,4 +87,3 @@ train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=32, shuffle valid_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=32, shuffle=False) test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=32, shuffle=False) ``` - diff --git a/content/courses/pytorch-hpc/simple_nn.md b/content/notes/pytorch-hpc/simple_nn.md similarity index 99% rename from content/courses/pytorch-hpc/simple_nn.md rename to content/notes/pytorch-hpc/simple_nn.md index 8f087e6c..418f1598 100644 --- a/content/courses/pytorch-hpc/simple_nn.md +++ b/content/notes/pytorch-hpc/simple_nn.md @@ -2,8 +2,11 @@ date : "2025-02-26T00:00:00" title: "Building a Simple Neural Network" toc: true -type: book +type: docs weight: 10 +menu: + pytorch-hpc: + parent: Pytorch for HPC --- diff --git a/content/courses/pytorch-hpc/transfer.md b/content/notes/pytorch-hpc/transfer.md similarity index 98% rename from content/courses/pytorch-hpc/transfer.md rename to content/notes/pytorch-hpc/transfer.md index 8c6341dc..66e5c052 100644 --- a/content/courses/pytorch-hpc/transfer.md +++ b/content/notes/pytorch-hpc/transfer.md @@ -2,9 +2,13 @@ date : "2025-02-26T00:00:00" title: "Transfer Learning" toc: true -type: book +type: docs weight: 20 +menu: + pytorch-hpc: + parent: Pytorch for HPC + --- ### **What is Transfer Learning?** diff --git a/content/notes/rio-intro/_index.md b/content/notes/rio-intro/_index.md index ad3d2b06..8b0f6f34 100644 --- a/content/notes/rio-intro/_index.md +++ b/content/notes/rio-intro/_index.md @@ -1,9 +1,10 @@ --- title: Intro to High Security HPC (Rio) +authors: [as, pbo, cmd] date: 2025-11-12T03:53:56Z -authors: [as, pbo, Camden Duy] -categories: [HPC] -tags: [HPC] +authors: [as, pbo, cmd] +categories: [HPC Intro] +tags: [HPC_Intro] type: docs weight: 1 diff --git a/content/notes/rivanna-alphafold/vmd/20-visualizing-structures.md b/content/notes/rivanna-alphafold/vmd/20-visualizing-structures.md deleted file mode 100644 index d7f2e746..00000000 --- a/content/notes/rivanna-alphafold/vmd/20-visualizing-structures.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -title: Visualizing 3D Structures -date: 2025-05-20T00:23:54Z -type: docs -weight: 1950 -menu: - rivanna-alphafold: - name: Visualizing 3D Structures ---- - -To view your predicted structure generated by AlphaFold in 3D, you can use **Visual Molecular Dynamics (VMD)**. -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_40.png width=60% height=60% >}} - -Steps: -* Log onto Open On Demand ([https://rivanna-portal.hpc.virginia.edu/](https://rivanna-portal.hpc.virginia.edu/)) -* Choose the ‘Desktop’ interactive app and fill out the webform with your required resources. -* Click launch. -* Load VMD in the terminal: -```bash -module load vmd -vmd -``` -The VMD windows will launch. From there, you can open your `.pdb` file from the AlphaFold output directory to view the 3D model (colored by `pLDDT`). - -{{< figure src=/notes/rivanna-alphafold/img/Alphafold_41.jpg width=65% height=65% caption="Loading your .pdb file (a)." >}} -{{< figure src=/notes/rivanna-alphafold/img/molecule-file-browser.png width=65% height=65% caption="Loading your .pdb file (b)." >}} -{{< figure src=/notes/rivanna-alphafold/img/vmd-part-3.png width=60% height=60% caption="Loading your .pdb file (c)." >}} -{{< figure src=/notes/rivanna-alphafold/img/vmd-part-4.png width=50% height=50% caption="Loading your .pdb file (d)." >}} - - - diff --git a/content/notes/unix-tutorial/_index.md b/content/notes/unix-tutorial/_index.md index 5718fd37..1c24b3b8 100644 --- a/content/notes/unix-tutorial/_index.md +++ b/content/notes/unix-tutorial/_index.md @@ -3,11 +3,11 @@ title: "UNIX Tutorials for Beginners" type: docs toc: true weight: 1 -date: 2023-12-11T00:00:00Z +date: "2023-12-11-14T00:00:00" +author: [kah] menu: unix-tutorials: - --- These tutorials are derived from the excellent tutorials from the University of Surrey, UK, with some minor modifications for our site. The originals can be found [here](http://www.ee.surrey.ac.uk/Teaching/Unix/). diff --git a/content/notes/unix-tutorial/unix_tutorial_1.md b/content/notes/unix-tutorial/unix_tutorial_1.md index 07b54115..3c2200ec 100644 --- a/content/notes/unix-tutorial/unix_tutorial_1.md +++ b/content/notes/unix-tutorial/unix_tutorial_1.md @@ -161,7 +161,7 @@ In newer Linux systems, most commands have a `--help` option ```bash % wc --help ``` -However, `man` sends output through a [pager](/notes/unix-tutorials/unix_tutorial_2), whereas --help prints directly to the console. +However, `man` sends output through a [pager](/notes/unix-tutorial/unix_tutorial_2/), whereas --help prints directly to the console. Another useful command, ``` diff --git a/content/tutorials/benchmark-parallel-programs/index.md b/content/tutorials/benchmark-parallel-programs/index.md index b1428f7c..0693abb0 100755 --- a/content/tutorials/benchmark-parallel-programs/index.md +++ b/content/tutorials/benchmark-parallel-programs/index.md @@ -8,5 +8,5 @@ summary: Learn how to benchmark your program for parallel SLURM jobs. notes: benchmark-parallel-programs -weight: 130 +weight: 830 --- diff --git a/content/tutorials/bioinfo-intro/index.md b/content/tutorials/bioinfo-intro/index.md index 7c3bd509..03c55a6a 100644 --- a/content/tutorials/bioinfo-intro/index.md +++ b/content/tutorials/bioinfo-intro/index.md @@ -6,6 +6,8 @@ categories: ["Bioinformatics"] tags: ["Bioinformatics"] summary: "An introduction to bioinformatics concepts, sequencing technologies, common analysis tools, and their use on high-performance computing systems." +weight: 505 + notes: bioinfo-intro ---- \ No newline at end of file +--- diff --git a/content/tutorials/bioinfo-tools-riv/index.md b/content/tutorials/bioinfo-tools-riv/index.md index 8fb185c5..b5a4cb6c 100755 --- a/content/tutorials/bioinfo-tools-riv/index.md +++ b/content/tutorials/bioinfo-tools-riv/index.md @@ -6,8 +6,10 @@ summary: "This tutorial is an introduction to using several popular bioinformati publishDate: "2021-03-02T00:00:00" authors: [gka] -tags: [HPC,Bioinformatics] -categories: ["HPC","Bioinformatics"] +tags: [Bioinformatics] +categories: ["Bioinformatics"] + +weight: 520 # Markdown Slides (optional). # Associate this talk with Markdown slides. @@ -17,7 +19,7 @@ categories: ["HPC","Bioinformatics"] url_slides: notes/bioinfo-tools-riv -weight: 500 +weight: 400 --- diff --git a/content/tutorials/biopython/index.md b/content/tutorials/biopython/index.md deleted file mode 100644 index 794f98d2..00000000 --- a/content/tutorials/biopython/index.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -title: "Biopython" - -summary: "An introduction to using the Biopython package to develop Python scripts for bioinformatics analysis." - -date: 2020-10-01T00:00:00-05:00 -authors: ["khs"] -tags: [Python,Bioinformatics] -categories: ["Python", "Bioinformatics"] -weight: 520 - -notes: biopython ---- -In this workshop participants will learn how to use the biopython package to develop Python scripts for bioinformatics analysis. Participants will learn how to download public datasets from online databases, parse bioinformatics file formats, and implement some sequence analysis workflows. diff --git a/content/tutorials/building-running-c-cpp-fortran/index.md b/content/tutorials/building-running-c-cpp-fortran/index.md index c2857f4e..7e097e2e 100755 --- a/content/tutorials/building-running-c-cpp-fortran/index.md +++ b/content/tutorials/building-running-c-cpp-fortran/index.md @@ -8,6 +8,6 @@ summary: This tutorial will teach you to compile, link, and run programs written notes: "building-running-c-cpp-fortran" -weight: 110 +weight: 800 --- diff --git a/content/tutorials/connecting-to-hpc/index.md b/content/tutorials/connecting-to-hpc/index.md index 57ef1e3e..8b92194b 100644 --- a/content/tutorials/connecting-to-hpc/index.md +++ b/content/tutorials/connecting-to-hpc/index.md @@ -2,14 +2,14 @@ title: "Connecting to HPC" date: 2020-02-28T00:00:00-05:00 authors: [kah,rs] -categories: ["Programming","HPC","Open OnDemand"] -tags: [Programming,HPC,Open_OnDemand] +categories: ["Programming","HPC Intro","Open OnDemand"] +tags: [Programming,HPC_Intro,Open_OnDemand] summary: This tutorial will teach you multiple methods for connecting UVA's HPC system in the web browser. notes: hpc-intro/connecting_to_the_system/connecting_logging_on -weight: 110 +weight: 100 --- diff --git a/content/tutorials/containers-for-hpc/index.md b/content/tutorials/containers-for-hpc/index.md new file mode 100755 index 00000000..74660f0c --- /dev/null +++ b/content/tutorials/containers-for-hpc/index.md @@ -0,0 +1,12 @@ +--- +title: "Building and Using Containers for HPC" +summary: "An introduction to using and building software containers for use on HPC systems." +date: 2020-10-29T00:00:00-05:00 +authors: [rs] +categories: ["HPC","Containers"] +tags: [HPC, Containers] + +notes: containers-for-hpc + +weight: 890 +--- diff --git a/content/tutorials/containers/index.md b/content/tutorials/containers/index.md deleted file mode 100755 index a7dbda15..00000000 --- a/content/tutorials/containers/index.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -title: "Introduction to Building and Deploying Containers" -summary: "" -authors: [cag] -date: 2023-05-12T00:00:00-05:00 -tags: ["Containers","R","Shiny","App_Development"] -categories: ["Containers","Docker", "R", "Shiny", "App Development"] -draft: false -toc: True - -notes: containers - -weight: 250 - ---- -In this workshop we will go over how to containerize our apps with Docker and maintain them with GitHub. diff --git a/content/tutorials/databases-intro/index.md b/content/tutorials/databases-intro/index.md deleted file mode 100644 index dc8f1c6a..00000000 --- a/content/tutorials/databases-intro/index.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -title: Introduction to Databases -summary: "This tutorial is an introduction to relational and NoSQL databases." - -# Schedule page publish date (NOT talk date). -publishDate: "2024-04-01T00:00:00" - -categories: ["Databases"] -tags: ["Databases"] - -notes: databases-intro - -weight: 200 - ---- diff --git a/content/tutorials/deep-learning-distributed/index.md b/content/tutorials/deep-learning-distributed/index.md index c3c03fdc..5502c77a 100644 --- a/content/tutorials/deep-learning-distributed/index.md +++ b/content/tutorials/deep-learning-distributed/index.md @@ -5,7 +5,8 @@ date: "2024-06-06T00:00:00" authors: [as,jmh] categories: ["GPU","Machine Learning","Python","Deep Learning"] tags: ["GPU","Machine_Learning","Python","Deep_Learning"] -weight: 300 + +weight: 320 notes: deep-learning-distributed url_code: notes/deep-learning-distributed/code/distributed_dl.zip diff --git a/content/tutorials/deep-learning-hpc/index.md b/content/tutorials/deep-learning-hpc/index.md index da28d7b3..7ea791b0 100755 --- a/content/tutorials/deep-learning-hpc/index.md +++ b/content/tutorials/deep-learning-hpc/index.md @@ -1,12 +1,14 @@ --- title: "Introduction to Deep Learning Using HPC" -summary: "This tutorial provides an introduction to deep learning using high power computing. This workshop will cover accessing deep learning containers, resource allocation and helpful tools, how to choose a GPU, and deep learning slurm scripts." +summary: "This tutorial provides an introduction to deep learning using high performance computing. This workshop will cover accessing deep learning containers, resource allocation and helpful tools, how to choose a GPU, and deep learning slurm scripts." date: 2024-06-27T21:13:14-05:00 -authors: [uvarc] +authors: [abd] categories: ["GPU","Machine Learning","Deep Learning"] tags: ["GPU","Machine_Learning","Deep_Learning"] +weight: 310 + notes: deep-learning-hpc --- diff --git a/content/tutorials/dl-drug-discovery/index.md b/content/tutorials/dl-drug-discovery/index.md index dfa3c97b..f9561be0 100644 --- a/content/tutorials/dl-drug-discovery/index.md +++ b/content/tutorials/dl-drug-discovery/index.md @@ -2,10 +2,12 @@ title: "Deep Learning in Drug Discovery" date: "2025-05-14T00:00:00" authors: [ppr] -categories: ["HPC","Deep learning"] -tags: ["HPC","Deep learning"] +categories: [Deep learning"] +tags: ["Deep_learning"] summary: "This tutorial introduces the deep learning approach to computational drug discovery." +weight: 590 + notes: dl-drug-discovery ---- \ No newline at end of file +--- diff --git a/content/tutorials/genomics/index.md b/content/tutorials/genomics/index.md index d0ef56dd..c5d86db1 100644 --- a/content/tutorials/genomics/index.md +++ b/content/tutorials/genomics/index.md @@ -5,12 +5,12 @@ summary: "This tutorial introduces tools and resources for performing genome ass date: "2025-07-10T00:00:00" authors: [dat, mab] -tags: ["Bioinformatics", "Data_Analytics", "HPC"] -categories: ["Bioinformatics","Data Analytics", "HPC"] +tags: ["Bioinformatics", "Data_Analytics"] +categories: ["Bioinformatics","Data Analytics"] -notes: genomics +weight: 530 -weight: 100 +notes: genomics --- diff --git a/content/tutorials/git-intro/index.md b/content/tutorials/git-intro/index.md index b5ab887d..32e7cec7 100755 --- a/content/tutorials/git-intro/index.md +++ b/content/tutorials/git-intro/index.md @@ -4,10 +4,12 @@ summary: "Learn to use the git version-control system to manage your software pr Date: "2015-07-23T21:13:14-05:00" -authors: [uvarc] +authors: [kah] tags: [Programming,Git] categories: ["Programming","Git"] +weight: 810 + # Markdown Slides (optional). # Associate this talk with Markdown slides. # Simply enter your slide deck's filename without extension. @@ -16,7 +18,5 @@ categories: ["Programming","Git"] #url_slides: slides/git-intro/git-slides.html notes: git-intro -weight: 100 - --- diff --git a/content/tutorials/globus-data-transfer/index.md b/content/tutorials/globus-data-transfer/index.md index 53560fdd..b7a8505f 100644 --- a/content/tutorials/globus-data-transfer/index.md +++ b/content/tutorials/globus-data-transfer/index.md @@ -8,5 +8,6 @@ date : 2024-02-10T00:00:00-05:00 notes: globus-data-transfer url_video: https://youtu.be/bMcSqbDiFuo -weight: 50 + +weight: 150 --- diff --git a/content/tutorials/gpu-applications-rivanna/index.md b/content/tutorials/gpu-applications-rivanna/index.md index de0cc865..9b2a2189 100755 --- a/content/tutorials/gpu-applications-rivanna/index.md +++ b/content/tutorials/gpu-applications-rivanna/index.md @@ -8,7 +8,7 @@ tags: ["GPU", "HPC"] notes: gpu-applications-rivanna -weight: 160 +weight: 810 --- diff --git a/content/tutorials/hpc-alphafold/index.md b/content/tutorials/hpc-alphafold/index.md new file mode 100644 index 00000000..94cd33ac --- /dev/null +++ b/content/tutorials/hpc-alphafold/index.md @@ -0,0 +1,16 @@ +--- +title: "AlphaFold on HPC" +date: "2025-05-20" +authors: [mab] +categories: ["GPU","HPC","Bioinformatics"] +tags: ["GPU","HPC","Bioinformatics"] +summary: "This tutorial introduces the basics of GPU computing and demonstrates how to run AlphaFold on the HPC cluster to predict protein structures." + +weight: 550 + +notes: hpc-alphafold + +#url_code: + +--- + diff --git a/content/tutorials/hpc-best-practices/index.md b/content/tutorials/hpc-best-practices/index.md index 7ffa640c..44bb71fd 100644 --- a/content/tutorials/hpc-best-practices/index.md +++ b/content/tutorials/hpc-best-practices/index.md @@ -6,6 +6,8 @@ authors: [teh] tags: ["HPC","Unix"] summary: "This tutorial introduces various techniques and strategies to help users more efficiently use UVA's HPC System." +weight: 250 + notes: hpc-best-practices --- diff --git a/content/tutorials/hpc-from-terminal/index.md b/content/tutorials/hpc-from-terminal/index.md index 447d01d4..3e722f14 100644 --- a/content/tutorials/hpc-from-terminal/index.md +++ b/content/tutorials/hpc-from-terminal/index.md @@ -1,16 +1,16 @@ --- title: Using UVA's HPC System from the Terminal -summary: "This tutorial is an introduction to using the Unix command line on UVA's HPC System." +summary: "This tutorial is an introduction to using the Unix command line on UVA's HPC System, including several commands specific to this environment." # Schedule page publish date (NOT talk date). publishDate: "2023-01-04T00:00:00" authors: [as,gka,kah] -tags: ["HPC","Unix"] -categories: ["HPC","Unix"] +tags: ["HPC_Intro","Unix"] +categories: ["HPC Intro","Unix"] notes: hpc-from-terminal url_slides: slides/hpc-from-terminal/HPC_From_Terminal.pptx -weight: 20 +weight: 210 --- diff --git a/content/tutorials/hpc-intro/index.md b/content/tutorials/hpc-intro/index.md index 567d5d97..d30ab5ec 100755 --- a/content/tutorials/hpc-intro/index.md +++ b/content/tutorials/hpc-intro/index.md @@ -6,8 +6,8 @@ summary: "This tutorial is an introduction to using UVA's high-performance compu publishDate: "2021-04-13T00:00:00" authors: [jmh,kah,pbo] -tags: [HPC] -categories: ["HPC"] +tags: [HPC_Intro] +categories: ["HPC Intro"] # Markdown Slides (optional). # Associate this talk with Markdown slides. @@ -19,4 +19,4 @@ notes: hpc-intro url_video: https://www.youtube.com/playlist?list=PLT4bryHgBcRP7N-hB9u6EWs6tq_2nMoRO weight: 10 ---- \ No newline at end of file +--- diff --git a/content/tutorials/interactive-apps-ood/index.md b/content/tutorials/interactive-apps-ood/index.md index b073a0bc..a2b268d2 100644 --- a/content/tutorials/interactive-apps-ood/index.md +++ b/content/tutorials/interactive-apps-ood/index.md @@ -1,15 +1,14 @@ --- title: "Interactive Apps with Open OnDemand" date: 2020-02-28T00:00:00-05:00 -authors: [kah,rs] -categories: ["HPC","Open OnDemand"] -tags: [HPC,Open_OnDemand] -summary: This tutorial will introduce several interactive apps built-into Open OnDemand. - +authors: [kah, rs] +categories: ["HPC Intro","Open OnDemand"] +tags: [HPC_Intro, Open_OnDemand] +summary: This tutorial will introduce several interactive apps built in to Open OnDemand. notes: hpc-intro/interactive_apps/interactive -weight: 110 +weight: 120 --- diff --git a/content/tutorials/llms-hpc/index.md b/content/tutorials/llms-hpc/index.md index 9e3765db..b95e3a68 100644 --- a/content/tutorials/llms-hpc/index.md +++ b/content/tutorials/llms-hpc/index.md @@ -6,6 +6,8 @@ categories: ["HPC", "Machine learning"] tags: ["HPC"] summary: "This tutorial is an introduction to running large language models on UVA's HPC system." +weight: 350 + notes: llms-hpc --- diff --git a/content/tutorials/matlab-data-science/index.md b/content/tutorials/matlab-data-science/index.md index 1bd3b993..a029d823 100755 --- a/content/tutorials/matlab-data-science/index.md +++ b/content/tutorials/matlab-data-science/index.md @@ -6,7 +6,8 @@ date: 2020-09-08T21:13:14-05:00 authors: [teh] categories: ["MATLAB","Machine Learning","Data Analytics","Deep Learning"] tags: ["MATLAB","Machine_Learning","Data_Analytics","Deep_Learning"] -weight: 240 + +weight: 650 notes: matlab-data-science diff --git a/content/tutorials/matlab-data-viz/index.md b/content/tutorials/matlab-data-viz/index.md index e14030f1..4b17df77 100755 --- a/content/tutorials/matlab-data-viz/index.md +++ b/content/tutorials/matlab-data-viz/index.md @@ -5,7 +5,8 @@ authors: [cag] date: 2019-10-16T21:13:14-05:00 categories: ["MATLAB","Data Analytics"] tags: ["MATLAB","Data_Analytics"] -weight: 250 + +weight: 620 notes: matlab-data-visualization diff --git a/content/tutorials/matlab-deep-learning/index.md b/content/tutorials/matlab-deep-learning/index.md index fbf4acdc..c953da3d 100755 --- a/content/tutorials/matlab-deep-learning/index.md +++ b/content/tutorials/matlab-deep-learning/index.md @@ -5,7 +5,8 @@ date: 2019-10-16T21:13:14-05:00 authors: [cag] categories: ["MATLAB","Machine Learning","Deep Learning"] tags: ["MATLAB","Machine_Learning","Deep_Learning"] -weight: 350 + +weight: 660 notes: matlab-deep-learning diff --git a/content/tutorials/matlab-image-processing/index.md b/content/tutorials/matlab-image-processing/index.md index 4ad28b5c..54ed6fd0 100644 --- a/content/tutorials/matlab-image-processing/index.md +++ b/content/tutorials/matlab-image-processing/index.md @@ -6,7 +6,8 @@ date: 2020-03-03T21:13:14-05:00 categories: ["Image Processing","MATLAB"] tags: ["Image_Processing","MATLAB"] draft: false -weight: 220 + +weight: 610 notes: matlab-image-processing diff --git a/content/tutorials/matlab-optimization/index.md b/content/tutorials/matlab-optimization/index.md index 270920bc..e25e64cb 100644 --- a/content/tutorials/matlab-optimization/index.md +++ b/content/tutorials/matlab-optimization/index.md @@ -5,7 +5,9 @@ tags : [ "MATLAB" ] title : "Optimization Techniques in MATLAB" summary : "This workshop introduces applied optimization in the MATLAB environment, focusing on using the Optimization Toolbox™ and the Global Optimization Toolbox™." authors : [teh] -weight: 260 + +weight: 630 + notes: matlab-optimization --- diff --git a/content/tutorials/matlab-parallel/index.md b/content/tutorials/matlab-parallel/index.md index dc0b3698..c6aeb3b0 100755 --- a/content/tutorials/matlab-parallel/index.md +++ b/content/tutorials/matlab-parallel/index.md @@ -3,9 +3,10 @@ title: "Parallel Computing with MATLAB" summary: "In this hands-on workshop, you will be introduced to parallel and distributed computing in MATLAB™ for speeding up your application and offloading work." authors: [teh] date: "2019-06-23T08:37:46-05:00" -categories: [ "HPC", "Parallel Computing", "MATLAB" ] +categories: [ "HPC", "Parallel Programming", "MATLAB" ] tags: [ "HPC", "Parallel_Programming", "MATLAB" ] -weight: 280 + +weight: 680 url_slides: slides/matlab-parallel/Parallel_Computing_with_MATLAB.pptx url_code: scripts/matlab-parallel/Parallel-Computing-Workshop.zip diff --git a/content/tutorials/matlab-statistics/index.md b/content/tutorials/matlab-statistics/index.md deleted file mode 100644 index 9ddf1b06..00000000 --- a/content/tutorials/matlab-statistics/index.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -title : "Statistical Methods with MATLAB" -summary : "Using statistical methods with MATLAB™" -date : "2019-06-23T08:37:46-05:00" -authors : [uvarc] -categories: ["Matlab","Data Analytics"] -tags: ["Matlab","Data_Analytics"] -weight: 230 - -notes: matlab-statistics ---- - -This workshop provides hands-on experience for performing statistical data analysis with MATLAB® and the Statistics and Machine Learning Toolbox™. Examples and exercises demonstrate the use of appropriate MATLAB and Statistics and Machine Learning Toolbox functionality throughout the analysis process; from exploratory analysis, to confirmatory analysis and simulation. Topics include: - - - Calculating summary statistics - - Visualizing data - - Fitting distributions - - Performing tests of significance - - Fitting regression models - - Interpolating data - - Generating random numbers - -Participants must have a Mathworks account in order to access their online training materials. - diff --git a/content/tutorials/multigpu-inference/index.md b/content/tutorials/multigpu-inference/index.md index 1cc49f84..f208d700 100644 --- a/content/tutorials/multigpu-inference/index.md +++ b/content/tutorials/multigpu-inference/index.md @@ -2,10 +2,12 @@ title: "Multi-GPU LLM Inference" date: "2025-02-23T00:00:00" authors: [kal] -categories: ["HPC", "Machine learning"] -tags: ["HPC"] +categories: ["GPU", "HPC", "Machine learning"] +tags: ["GPU", "HPC"] summary: "This tutorial is an introduction to multi-GPU strategies for large language model (LLM) inference using tools like Accelerate, DeepSpeed, and vLLM." +weight: 370 + notes: multigpu-inference --- diff --git a/content/tutorials/omero-hands-on/index.md b/content/tutorials/omero-hands-on/index.md deleted file mode 100755 index e1ba3daa..00000000 --- a/content/tutorials/omero-hands-on/index.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -title: "OMERO Hands-On Tutorial" -summary: "Getting started with the OMERO image database." -authors: [cag] -date: "2019-10-10T21:13:14-05:00" -categories: ["Image Processing","Omero"] -tags: ["Image_Processing","Omero"] -draft: true - -notes: omero-hands-on -url_dataset: data/omero-workshop-images.zip -url_code: https://github.com/ksiller/omero - - ---- - -{{< figure src="/img/ome-main-nav.svg" >}} - -OMERO is image management software that allows you to organize, view, annotate, analyze, and share your data from a single centralized repository. - -"This hands-on tutorial will show you how to import images into the OMERO database, organize and tag images, and how to use OMERO with third-party software such as Fiji/ImageJ, MATLAB™, and Python." - -Download the sample dataset and scripts so you can follow along with the -examples. diff --git a/content/tutorials/python-high-performance/index.md b/content/tutorials/python-high-performance/index.md index 7ccee73c..404d3050 100644 --- a/content/tutorials/python-high-performance/index.md +++ b/content/tutorials/python-high-performance/index.md @@ -1,17 +1,13 @@ --- title: "High Performance Programming in Python" -summary: Tips and tricks for improving the performance of your Python code. -# authors: [kah,khs] +summary: "Like most interpreted languages, Python can be slow. This tutorial will describe tips and tricks to help speed it up." date: 2020-02-02T00:06:47-04:00 -categories: ["Python", "Programming", "HPC"] -tags: ["Python","Programming","HPC"] -authors: ["kah"] -draft: true +categories: ["Python", "Programming", "Parallel Programming", "HPC"] +tags: ["Python","Programming", "Parallel_Programming","HPC"] +authors: ["khs","kah"] -url_notes: courses/python-high-performance +weight: 840 ---- - -My program takes forever! Can I do something about it? Like most interpreted languages, Python can be slow. Loops are a particular problem, but there are other pitfalls that may affect the performance of your code. This workshop will show you tips and tricks to speed up your code. +notes: python-high-performance -Participants should have experience programming in Python. +--- diff --git a/content/tutorials/python-machine-learning/index.md b/content/tutorials/python-machine-learning/index.md index e40d0bf0..91050223 100644 --- a/content/tutorials/python-machine-learning/index.md +++ b/content/tutorials/python-machine-learning/index.md @@ -6,11 +6,13 @@ date : "2022-06-09T00:00:00" categories : ["Machine Learning", "Python"] tags : ["Machine_Learning","Python"] +weight: 320 + notes: python-machine-learning url_code: notes/python-machine-learning/code/ML_with_Python.zip --- -This tutorial wil go over Machine Learning concepts using Python discussing the following topics: +This tutorial wil go over Machine Learning concepts using Python. It discusses the following topics: * Overview of Machine Learning * Decision Trees * Coding Decision Trees diff --git a/content/tutorials/pytorch-hpc/index.md b/content/tutorials/pytorch-hpc/index.md new file mode 100755 index 00000000..e99d4e1e --- /dev/null +++ b/content/tutorials/pytorch-hpc/index.md @@ -0,0 +1,13 @@ +--- +title: "Introduction to PyTorch on HPC" +summary: "This tutorial provides a practical introduction to building artificial neural networks using PyTorch, a powerful and flexible deep learning framework. The course covers the fundamentals of PyTorch, including tensors, automatic differentiation, and model building." + +date: 2024-06-27T21:13:14-05:00 +authors: [abd, bmr] +categories: ["GPU","Machine Learning","Deep Learning"] +tags: ["GPU","Machine_Learning","Deep_Learning"] + +weight: 330 + +notes: pytorch-hpc +--- diff --git a/content/tutorials/qiime2/index.md b/content/tutorials/qiime2/index.md index 6c8f945c..751fb173 100644 --- a/content/tutorials/qiime2/index.md +++ b/content/tutorials/qiime2/index.md @@ -1,11 +1,13 @@ --- title: "Introduction to QIIME 2" date: "2025-06-10T00:00:00" -authors: [] +authors: [gka] categories: ["Bioinformatics", "Data Analytics"] tags: ["Bioinformatics", "Data_Analytics"] summary: "This tutorial is an introduction to using QIIME 2 for micribiome analysis." +weight: 580 + notes: qiime2 --- diff --git a/content/tutorials/rapids/index.md b/content/tutorials/rapids/index.md index f248ce85..56b436e8 100755 --- a/content/tutorials/rapids/index.md +++ b/content/tutorials/rapids/index.md @@ -5,7 +5,8 @@ authors: [rs] date: 2020-10-13T00:00:00-05:00 categories: ["Python","GPU","HPC"] tags: ["Python","GPU","HPC"] -weight: 180 + +weight: 770 notes: rapids diff --git a/content/tutorials/rio-intro/index.md b/content/tutorials/rio-intro/index.md index 1e399841..45129e11 100644 --- a/content/tutorials/rio-intro/index.md +++ b/content/tutorials/rio-intro/index.md @@ -5,9 +5,9 @@ summary: "This tutorial is an introduction to using the High Security HPC platfo # Schedule page publish date (NOT talk date). publishDate: "2025-01-30T00:00:00" -authors: [as, pbo, Camden Duy] -tags: [HPC] -categories: ["HPC"] +authors: [as, pbo, cmd ] +tags: [HPC_Intro] +categories: ["HPC Intro"] # Markdown Slides (optional). # Associate this talk with Markdown slides. diff --git a/content/tutorials/rivanna-alphafold/index.md b/content/tutorials/rivanna-alphafold/index.md deleted file mode 100644 index 2de11e89..00000000 --- a/content/tutorials/rivanna-alphafold/index.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -title: "AlphaFold on Rivanna" -date: "2025-05-20" -authors: [mab] -categories: "" -tags: ["Bioinformatics"] -summary: "This tutorial introduces the basics of GPU computing and demonstrates how to run AlphaFold on the Rivanna cluster to predict protein structures." - -notes: rivanna-alphafold -url_code: - ---- - diff --git a/content/tutorials/seurat-bioinformatics/index.md b/content/tutorials/seurat-bioinformatics/index.md index d248bc79..b91bfc87 100644 --- a/content/tutorials/seurat-bioinformatics/index.md +++ b/content/tutorials/seurat-bioinformatics/index.md @@ -2,10 +2,12 @@ title: "Bioinformatics Data Visualization with Seurat" date: "2025-08-18T00:00:00" authors: [gka, mab] -categories: ["HPC", "Bioinformatics"] +categories: ["Bioinformatics"] tags: ["Bioinformatics"] summary: "This tutorial is a step-by-step walkthrough of the Seurat workflow for single-cell RNA-seq analysis." +weight: 560 + notes: seurat-bioinformatics --- diff --git a/content/tutorials/slurm-from-cli/index.md b/content/tutorials/slurm-from-cli/index.md index d9974fe7..642c5edb 100644 --- a/content/tutorials/slurm-from-cli/index.md +++ b/content/tutorials/slurm-from-cli/index.md @@ -6,11 +6,11 @@ summary: "This tutorial is an introduction to submitting Slurm jobs through a te publishDate: "2023-01-04T00:00:00Z" authors: [as,kah] -tags: ["HPC","Slurm"] -categories: ["HPC","Slurm"] +tags: ["HPC_Intro","Slurm"] +categories: ["HPC Intro","Slurm"] notes: slurm-from-cli -weight: 30 +weight: 230 --- diff --git a/content/tutorials/unix-tutorial/index.md b/content/tutorials/unix-tutorial/index.md index 5d26efc8..c0d6befa 100755 --- a/content/tutorials/unix-tutorial/index.md +++ b/content/tutorials/unix-tutorial/index.md @@ -1,11 +1,12 @@ --- title: Unix Tutorials for Beginners summary: "This set of tutorials is an introduction to using the Unix operating system from the command line." +authors: [ kah ] # Schedule page publish date (NOT talk date). publishDate: "2017-01-01T00:00:00" -categories: ["Unix","HPC"] -tags: ["Unix","HPC"] +categories: ["Unix"] +tags: ["Unix"] # Markdown Slides (optional). # Associate this talk with Markdown slides. @@ -14,7 +15,7 @@ tags: ["Unix","HPC"] # Otherwise, set `slides = ""`. notes: unix-tutorial -weight: 40 +weight: 190 --- diff --git a/content/tutorials/using-the-shell/index.md b/content/tutorials/using-the-shell/index.md index 70a3767a..f28ff641 100644 --- a/content/tutorials/using-the-shell/index.md +++ b/content/tutorials/using-the-shell/index.md @@ -1,15 +1,15 @@ --- -title: "Using the Shell on Open OnDemand" +title: "Using the Shell Through Open OnDemand" date: 2020-02-28T00:00:00-05:00 authors: [kah,rs] -categories: ["Programming","HPC", "Unix","Open OnDemand"] -tags: [Programming,HPC,Unix,Open_OnDemand] -summary: This tutorial will explain how to access the shell on UVA HPC, as well as several useful shell commands for working with HPC. +categories: ["HPC Intro", "Unix","Open OnDemand"] +tags: [HPC_Intro,Unix,Open_OnDemand] +summary: This tutorial will demonstrate how to access the shell on UVA HPC through the Open OnDemand interface, and will teach several useful shell commands for working with HPC. notes: hpc-intro/features_of_ood/features_shell_access -weight: 110 +weight: 180 --- diff --git a/content/tutorials/vscode-intro/index.md b/content/tutorials/vscode-intro/index.md index 02d768a2..11269616 100644 --- a/content/tutorials/vscode-intro/index.md +++ b/content/tutorials/vscode-intro/index.md @@ -1,13 +1,15 @@ --- title: Introduction to VS Code for HPC -summary: "This tutorial is an introduction to using VS Code for HPC" +summary: "Using VS Code on a remote HPC system may be different from local use. This tutorial introduces best practices for VS Code on HPC." # Schedule page publish date (NOT talk date). publishDate: "2025-01-30T00:00:00" -authors: [Hana Parece] -tags: [HPC] -categories: ["HPC"] +authors: [hp] +tags: [HPC, Programming] +categories: ["HPC", "Programming"] + +weight: 240 # Markdown Slides (optional). # Associate this talk with Markdown slides. @@ -18,6 +20,4 @@ categories: ["HPC"] pdf: vscode-intro -weight: 12 - --- diff --git a/content/tutorials/working-with-files/index.md b/content/tutorials/working-with-files/index.md index 65ccbf71..d3e5e6b2 100644 --- a/content/tutorials/working-with-files/index.md +++ b/content/tutorials/working-with-files/index.md @@ -2,14 +2,14 @@ title: "Working with Files" date: 2020-02-28T00:00:00-05:00 authors: [kah,rs] -categories: ["HPC", "Data Transfer","Open OnDemand"] -tags: [HPC,Data_transfer,Open_OnDemand] +categories: ["HPC Intro", "Data Transfer","Open OnDemand"] +tags: [HPC_Intro ,Data_transfer,Open_OnDemand] summary: This tutorial will teach you how to work with files on UVA HPC through your browser, including creating, deleting, editing, and more. notes: hpc-intro/features_of_ood/features_files_tab -weight: 110 +weight: 130 --- diff --git a/content/tutorials/working-with-jobs-ood/index.md b/content/tutorials/working-with-jobs-ood/index.md index d8981e39..c8692fbe 100644 --- a/content/tutorials/working-with-jobs-ood/index.md +++ b/content/tutorials/working-with-jobs-ood/index.md @@ -2,14 +2,14 @@ title: "Working with Jobs in Open OnDemand" date: 2020-02-28T00:00:00-05:00 authors: [kah,rs] -categories: ["HPC","Open OnDemand"] -tags: [HPC,Open_OnDemand] -summary: This tutorial will explain how to view and create jobs on UVA HPC through Open OnDemand. +categories: ["HPC Intro","Open OnDemand","Slurm"] +tags: [HPC_Intro,Open_OnDemand,Slurm] +summary: This tutorial will explain how to view and create Slurm jobs on UVA HPC through Open OnDemand. notes: hpc-intro/features_of_ood/features_jobs_tab -weight: 110 +weight: 140 --- diff --git a/layouts/partials/docs_sidebar.html b/layouts/partials/docs_sidebar.html index 57eb7ca3..bfa2b128 100644 --- a/layouts/partials/docs_sidebar.html +++ b/layouts/partials/docs_sidebar.html @@ -21,7 +21,7 @@ {{ if eq site.Params.search.engine 1 }} - + {{ end }} diff --git a/layouts/partials/search.html b/layouts/partials/search.html index 5f850a84..0ad30758 100644 --- a/layouts/partials/search.html +++ b/layouts/partials/search.html @@ -1,18 +1,24 @@ -