Skip to content

Iron Out The Kinks

Jump to bottom
Mike Caprio edited this page Dec 9, 2018 · 3 revisions

Create Tools to Automate Rendering 3D Models of Marine Microbes from 2D Cross-sections

Hackathon Findings

Hackathon Projects

Background

The oceans are home to vast multitudes of micro-organisms. Up to 90% of the biomass in the ocean (the weight of all its life) is made up of single-celled microbes. At the American Museum of Natural History, efforts are underway to discover and understand this host of living creatures, which could be a variety of many millions of (or even a billion!) different kinds of microbes. In the normal course of studying living things, specimens are added to collections and their genetic and physical structure is analyzed - but how do we study the structure of a single celled organism? When it comes to studying marine microbes, they must be collected from marine expeditions and cultured (kept alive in solutions) in laboratory conditions; AMNH keeps many hundreds of living cultures for study, some of which are the only living cultures of those microbes in the world.

Some images of marine protists as seen under an optical microscope

Protists are microbes with the general structure and organization of a cell - a nucleus, tiny internal organs (organelles), and membranes - but they are not classifiable as animals, plants, or fungi. One particularly fascinating line of study involves microscopic observation of the organelles and other interior structures of marine protists to compare them to other types of single celled organisms. But it isn't possible to study these incredibly tiny structures with even the best optical microscopes, so instead an electron microscope (in particular, a transmission electron microscope, or TEM) is used to image these structures, may of which are only a few nanometers big. For this particular area of study the museum is "dissecting" single celled protists by cutting them into cross-sections, identifying and locating subcellular structures (in particular, microtubules) in each section, and reconstructing those shapes as 3D models.

The process of creating and imaging cross-sections of single celled organisms is extensive. First, the cultures of the organisms are prepared with various chemical preservative, drying, and freezing techniques, prior to being fixed in place in a resin. The hardened resin sample is then repeatedly cut into 50-70-nanometer-thin "deli slices" by a machine called an ultramicrotome, which uses a diamond knife honed to the thinness of atoms to cut the sample (depending upon its size, a single cell cut this thin can produce between eighty to several hundred slices). The cross-sectioned slices are arranged in ribbons across tiny discs, then stained with lead and depleted uranium; these discs are then ready to be inserted into a transmission electron microscope to be imaged with a digital camera.

Left, a resin sample in the ultramicrotome; Right, depleted uranium solution

Unfortunately this reconstruction is a lengthy, manual process. Because the slices of specimens are cut so thinly, they are often distorted, skewed, rotated, or otherwise misaligned on their imaging disc. Other times there are bits of dust or splatters of oil that accumulate in the preparation process that cause image artifacts (dark spots). After good cross-sectional images of a specimen have been identified, they must be aligned properly (like registering frames for an animation) and sometimes edited to fix those distortions. Next, the images become layers in a vector based editing program (Adobe Illustrator version 3) where the structures in cross-section are "traced out" as vector shapes. This vector file then gets loaded into a 3D editing tool (Maya, which can import Adobe Illustrator 3 files) where the pieces from all layers are combined together, spaced to the fixed 50-70 nanometer thickness of the sections. Additional corrections for distortions of the layers are also made at this stage as needed; in particular, the microtubules need to be made properly cylindrical. Although it takes a lot of time and effort, this digital reconstruction technique ultimately provides some of the most accurate observational models ever made for this kind of study.

Solutions

A 3D model showing nucleus (blue), mitochondrion (purple), and multicolored microtubules

Above is an example of the final product we hope to achieve (in this case, entirely adjusted manually). We would like to see solutions that automate some of the manual efforts and make it easier for us to produce a model like this. Streamlining any stages of the process, or in an ideal world, combining a number of these automations together into a single tool, would solve the challenge.

Some needed solutions include:

  • Selecting candidate cells. As of now, the TIFF images captured by the TEM are first checked by human eye, then candidate specimens are selected for further adjustment. Is it possible to choose candidate specimens using computer vision?

  • Aligning slices. Register the orientation of the slices so that changing shapes line up. What do we focus on? The contour of the membrane of the organism; other cells around the organism can be helpful as well. Can you utilize an existing tool or create a new tool that accomplishes this?

  • Correcting distortions. Specimens from the Museum's transmission electron microscope (TEM) are cut very thinly, and distort in the process. Once slices are aligned with one another, they may need to be adjusted to compensate for nonlinear compression, skew, and other distortions. This could also include image adjustment of contrast and brightness; slices can also have oil spots, staining artifacts, dust, oil, or other image artifacts to clean up.

  • An Illustrator 3 replacement. Build a tool that will allow the user to create a layer that contains each image and trace the contours and positions of microtubules and membranes as vector shapes. Using known dimensions of depth of layer and length and width of specimen, we can create 3D "building blocks" of tubule structures. We would then want an export of the reconstruction of tubules and membranes as a Maya ASCII file.

  • Create a "save file". If it is possible to create a unified tool that contains automations for the process, we also want a save file for the entire process. This would be a new file format that contains the layers, vectors, and other metadata (like a PSD file) so that the process can be returned to later to add additional missing tubes or make other fixes.

  • Maya automation. Once the vector drawings from the 2D images are combined into a model, the outlines of smooth structures are invariably highly erratic, and need to be "cleaned up" - imagine it as a bunch of kinks in a long cord. We need an automated way to smooth out the cord and iron out the kinks without making it completely straight.

Animation made from six cross-section TIFF images. The images have been rotated to match one another, but the different distortions in each image are apparent.

Resources

Be sure to check the Online Resources and Data Sets page to see if there might be any general purpose code or utilities you might use, especially for computer vision and image processing.

  • Challenge repository files: A few publicly available files (unprocessed images, processed images, Adobe Illustrator files, and Maya ASCII files) for one of the protists is available in this repository.
  • Additional images and files are available for two other protists are available on a hard drive. Teams that wish to work on this challenge can ask hackathon organizers for access to the hard drive to copy files locally.

**Please note: because some of the files for this challenge are not yet published we ask that you not keep the files listed as "under NDA" on the hard drive after the hackathon is over. As with all challenges, there will be opportunities for those interested in continuing to work on their projects after the hackathon! **

2D image processing:

  • ImageJ2: The lab is currently experimenting with ImageJ to try and do some of the pre-processing of the 2D TEM images, including automated alignment/registration of each cell.
  • FIJI - Fiji Is Just ImageJ: This version of ImageJ may be easier to get started with, but may also not have the same capabilities as the full toolkit.
  • CellProfiler: This toolkit may have some capabilities of identifying structures within cells or otherwise contributing to 2D image processing.

3D image manipulation:

Maya:

Challenge owner: Eunsoo Kim and Aaron Heiss

Challenges --|-- Online Resources And Data Sets --|-- Code of Conduct --|-- Home

Challenges

Online Resources And Data Sets

Museum Stakeholders

Home

Clone this wiki locally