1 21-BioImage

Previous: 20-ImageBasics.html

1.1 Biological imaging

Biological imaging may refer to any imaging technique used in biology.

https://en.wikipedia.org/wiki/Biological_imaging
https://en.wikipedia.org/wiki/Bioimage_informatics
https://en.wikipedia.org/wiki/Automated_tissue_image_analysis
https://en.wikipedia.org/wiki/Digital_pathology
https://en.wikipedia.org/wiki/Image_segmentation
https://en.wikipedia.org/wiki/Medical_imaging
https://en.wikipedia.org/wiki/Computer-aided_diagnosis

1.1.1 Medical imaging

https://en.wikipedia.org/wiki/Medical_imaging

• Medical imaging is the technique and process of creating visual representations of the interior of a body for clinical analysis and medical intervention, as well as visual representation of the function of some organs or tissues (physiology).
• Medical imaging seeks to reveal internal structures hidden by the skin and bones, as well as to diagnose and treat disease.
• Medical imaging also establishes a database of normal anatomy and physiology to make it possible to identify abnormalities.
• Although imaging of removed organs and tissues can be performed for medical reasons, such procedures are usually considered part of pathology instead of medical imaging.
• As a discipline and in its widest sense, it is part of biological imaging and incorporates radiology which uses the imaging technologies of X-ray radiography, magnetic resonance imaging, medical ultrasonography or ultrasound, endoscopy, elastography, tactile imaging, thermography, medical photography and nuclear medicine functional imaging techniques as positron emission tomography (PET) and Single-photon emission computed tomography (SPECT).

1.1.2 Bioimage informatics

https://en.wikipedia.org/wiki/Bioimage_informatics

• Bioimage informatics is a sub-field of bioinformatics and computational biology.
• It focuses on the use of computational techniques to analyze bioimages, especially cellular and molecular images, at large scale and high throughput.
• The goal is to obtain useful knowledge out of complicated and heterogeneous image and related meta-data.
• Automated microscopes are able to collect large numbers of images with minimal intervention.
• This has led to a data explosion, which absolutely requires automatic processing.
• Additionally, and surprisingly, for several of these tasks, there is evidence that automated systems can perform better than humans.
• In addition, automated systems are unbiased, unlike human based analysis whose evaluation may (even unconsciously) be influenced by the desired outcome.
• There has been an increasing focus on developing novel image processing, computer vision, data mining, database and visualization techniques to extract, compare, search and manage the biological knowledge in these data-intensive problems.
  
  A stained histologic specimen, sandwiched between a glass microscope slide and cover-slip, mounted on the stage of a light microscope.

Fluorescent image of a cell in telophase.
Multiple dyes were imaged and are shown in different colours.

Microscopic view of a histologic specimen of human lung tissue,
stained with hematoxylin and eosin.

Overview of a typical process:

1.2 Types of problem

1.2.1 Subcellular Location Analysis

• Subcellular location analysis was one of the initial problems in this field.
• In its supervised mode, the problem is to learn a classifier that can recognize the major cell organelles from cell images.
• Methods used are based on machine learning, building a discriminative classifier based on numeric features computed from the image.
  
• Sub-cellular Location Example.
• Examples of different patterns are mapped into a two-dimensional space by computing different image features.
• Image of unknown proteins are similarly mapped into this space and a nearest neighbor search or other classifier can be used for assigning a location to this unclassified protein.

1.2.2 Automated tissue image analysis

• Automated tissue image analysis is a process by which computer-controlled automatic test equipment is used to evaluate tissue samples, using computations to derive quantitative measurements from an image to avoid subjective errors.
• In a typical application, automated tissue image analysis could be used to measure the aggregate activity of cancer cells in a biopsy of a cancerous tumor taken from a patient.
• In breast cancer patients, for example, automated tissue image analysis may be used to test for high levels of proteins known to be present in more aggressive forms of breast cancers.
  

1.2.3 High-Content Screening

• High throughput screens using automated imaging technology (sometimes called high-content screening) have become a standard method for both drug discovery and basic biological research.
• Using multi-well plates, robotics, and automated microscopy, the same assay can be applied to a large library of possible reagents (typically either small molecules or RNAi) very rapidly, obtaining thousands of images in a short amount of time.
• Due to the high volume of data generated, automatic image analysis is a necessity.
  
  An automated confocal image reader.

1.2.4 Segmentation (or labeling)

• Image segmentation is the process of partitioning a digital image into multiple segments
  • (sets of pixels, also known as super-pixels).
• The goal of segmentation is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze.
• Image segmentation is typically used to locate objects and boundaries (lines, curves, etc.) in images.
• More precisely, image segmentation is the process of assigning a label to every pixel in an image such that pixels with the same label share certain characteristics.
  
  Example image for segmentation problem.
  Shown are nuclei of mouse NIH 3T3,
  stained with Hoechst and a segmentation in red.

Volume segmentation of a 3D-rendered CT scan of the thorax:
The anterior thoracic wall, the airways and the pulmonary vessels anterior to the root of the lung have been digitally removed in order to visualize thoracic contents:
* blue: pulmonary arteries
* red: pulmonary veins (and also the abdominal wall)
* yellow: the mediastinum
* violet: the diaphragm

1.2.5 Tracking

• Tracking is another traditional image processing problem which appears in bioimage informatics.
• The problem is to relate objects that appear in subsequent frames of a film.
• As with segmentation, the problem can be posed in both two- and three-dimensional forms.
  

1.2.6 Registration

• Image registration is the process of transforming different sets of data into one coordinate system.
• Data may be multiple photographs, data from different sensors, times, depths, or viewpoints.
• It is used in computer vision, medical imaging, military automatic target recognition, and compiling and analyzing images and data from satellites.
• Registration is needed to compare or integrate the data obtained from these different measurements.

Segment, then register:

1.2.7 Classification/diagnosis

Imaging methods:

Breast cancer image (e.g., a target classification)

1.2.8 Reconstruction

Taking sequential microscope image slices and reconstructing a 3D view on tissue.

+++++++++++ Cahoot-20-2
https://umsystem.instructure.com/courses/97521/quizzes/298142

1.3 Common bioimage informatics methods and their applications

Many of these methods are general,
but are particularly common in bioimage analysis.

1.3.1 Overview

A conceptual pipeline.
The specimen is imaged using any of today’s microscopes,
modeled by the input image f(v) passing through the blocks of
PSF (properties of the microscope, described by convolution with h(v)) and
A/D conversion (analog-to-digital conversion, effects of sampling and digitization together with uncertainty introduced by various sources of noise),
producing a digital image g_n.
That digital image is then restored either via a
de-noising followed by de-convolution, or via
joint de-noising/de-convolution,
producing a digital image f_n .
Various options are possible, the image could go through a
registration/mosaicing processing, producing r_n,
segmentation/tracing/tracking, producing s_n,
data analysis/modeling/simulations block, with the output y_n.
At the input/output of each block,
one can join the pathway to either skip a block,
or send feedback to previous block(s) in the system.

Stages in image analysis illustrated using algal image:
* (a) detail from image;
* (b) same detail after application of 5 × 5 moving median filter;
* (c) histogram, on a square root scale, of pixel values after filtering, with an arrow to indicate the threshold;
* (d) result of thresholding image at pixel value 120, to produce a binary image;
* (e) result of applying morphological opening to (d);
* (f) separated objects in (e) counted.

1.3.2 Bioimage formats

* Images viewed as matrices.
* The overview is not meant to be exhaustive, but reflects some of the more frequently used modes of image acquisition in biological and medical imaging, where the number of dimensions is typically one to five, with each dimension corresponding to an independent physical parameter:
* three to space,
* one (usually denoted t) to time, and
* one to wavelength, or color, or more generally to any spectral parameter (we denote this dimension s here).
* In other words, images are discrete functions, I (x, y, z, t, s),
* with each set of coordinates yielding the value of a unique sample
* (indicated by the small squares, the number of which is obviously arbitrary here).
* Note that the dimensionality of an image (indicated in the top row) is given by the number of coordinates that are varied during acquisition.
* To avoid confusion in characterizing an image, it is advisable to add adjectives indicating which dimensions were scanned, rather than mentioning just dimensionality.
* For example, a 4D image may either be a spatially 2D multi-spectral time-lapse image, or a spatially 3D time-lapse image.

1.3.3 Major terms

Image processing
* takes an image as input and produces a modified version of it
* (in the case shown, the object contours are enhanced using an operation known as edge detection, described in more detail elsewhere in this booklet).

Image analysis
* concerns the extraction of object features from an image.

Computer graphics
* is the inverse of image analysis:
* it produces an image from given primitives, which could be numbers (the case shown), or parameterized shapes, or mathematical functions.

Computer vision
* aims at producing a high-level interpretation of what is contained in an image; this is also known as image understanding.

Visualization
* transforms higher-dimensional image data into a more primitive representation to facilitate exploring the data.

1.3.4 Intensity transformation

• Among the simplest image processing operations are those that pass along each image pixel, and produce an output value that depends only on the corresponding input value and some mapping function.
• These are also called point operations.
• If the mapping function is the same for each pixel, we speak of a global intensity transformation.

* Examples of intensity transformations based on a global mapping function:
* contrast stretching, intensity inversion, intensity thresholding, and histogram equalization.
* The top row shows the images used as input for each of the transformations.
* The second row shows for each image the mapping function used (denoted M), with the histogram of the input image shown in the background.
* The bottom row shows for each input image the corresponding output image resulting from applying the mapping function:
* O( x, y ) = M( I (x, y )).
* It is clear from the mapping functions that contrast stretching and histogram equalization both distribute the most frequently occurring intensities over a wider range of values, thereby increasing image contrast.
* The former transformation is suitable in the case of uni-modal histograms, whereas the latter is particularly suited for images having multi-modal histograms.

Check out in class:
https://scikit-image.org/docs/stable/auto_examples/color_exposure/plot_equalize.html#sphx-glr-auto-examples-color-exposure-plot-equalize-py

1.3.5 Local image filtering

• Instead of considering just the corresponding input pixel when computing a value for each output pixel (as in intensity transformation), one could also take into account the values of adjacent input pixels.
• Image processing operations based on this principle are called neighborhood operations, or image filtering operations
• They are usually designed to filter out (enhance or reduce) specific image information.
  They can be classified into linear versus nonlinear filters:

1. Linear filtering operations:
   • Compute the output pixel value as a linear combination (weighing and summation) of the values of the corresponding input pixel and its neighbors.
   • Examples of kernels include:
     • averaging,
     • sharpening,
     • smoothing, and
     • derivative filters of varying sizes
2. Nonlinear filtering operations
   • Combine neighboring input pixel values in a nonlinear fashion in producing an output pixel value.
   • They can not be described as a convolution process.
   • Examples include:
     • median filtering (which for each output pixel computes the value as the median of the corresponding input values in a neighborhood of given size) and
     • min-filtering or max-filtering (where the output value is computed as, respectively, the minimum or the maximum value in a neighborhood around the corresponding input pixel).

1.3.5.1 Linear

https://en.wikipedia.org/wiki/Kernel_(image_processing)

* Principles and examples of convolution filtering.
* The value of an output pixel is computed as a linear combination (weighing and summation) of the value of the corresponding input pixel and of its neighbors.
* The weight factor assigned to each input pixel is given by the convolution kernel (denoted K).
* In principle, kernels can be of any size.
* Examples of commonly used kernels of size 3 × 3 pixels include
* the averaging filter,
* the sharpening filter, and
* the Sobel x- or y-derivative (approximation) filters.
* The Gaussian filter is often used as a smoothing filter.
* It has a free parameter (standard deviation σ) which determines the size of the kernel (usually cut off at m = 3σ) and therefore the degree of smoothing.
* The derivatives of this kernel are often used to compute image derivatives at different scales, as for example in Canny edge detection.
* The scale parameter, σ, should be chosen such that the resulting kernel matches the structures to be filtered.

Local filters
http://scipy-lectures.org/advanced/image_processing/index.html#image-filtering
http://scipy-lectures.org/packages/scikit-image/#local-filters

1.3.5.2 Non-linear

Code to show:
https://gitlab.com/bio-data/computer-vision/bio_images/morphology.py

The basic morphological operators are erosion, dilation, opening and closing.
https://en.wikipedia.org/wiki/Mathematical_morphology

https://en.wikipedia.org/wiki/Erosion_(morphology)
If when centered on a pixel,
any of kernel does not match,
set all in kernel to value of middle pixel.
The erosion of a point is the minimum of the points in its neighborhood,
with that neighborhood defined by the structuring element.
Erosion = minimum filter.
Replace the value of a pixel by the minimal value covered by the structuring element.

Suppose:
A is a 13 x 13 matrix, and
B is a 3 x 3 matrix:
1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 0 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1

Assuming that the origin B is at its center,
for each pixel in A superimpose the origin of B,
if B is completely contained by A,
then the pixel is retained,
else deleted.

Therefore the Erosion of A by B is given by this 13 x 13 matrix:
0 0 0 0 0 0 0 0 0 0 0 0 0
0 1 1 1 1 0 0 0 1 1 1 1 0
0 1 1 1 1 0 0 0 1 1 1 1 0
0 1 1 1 1 1 1 1 1 1 1 1 0
0 1 1 1 1 1 1 1 1 1 1 1 0
0 1 1 1 1 1 1 1 1 1 1 1 0
0 1 1 1 1 1 1 1 1 1 1 1 0
0 1 1 1 1 1 1 1 1 1 1 1 0
0 1 1 1 1 1 1 1 1 1 1 1 0
0 1 1 1 1 1 1 1 1 1 1 1 0
0 1 1 1 1 1 1 1 1 1 1 1 0
0 1 1 1 1 1 1 1 1 1 1 1 0
0 0 0 0 0 0 0 0 0 0 0 0 0

This means that only when B is completely contained inside A,
that the pixels values are retained,
otherwise it gets deleted or eroded.

https://en.wikipedia.org/wiki/Dilation_(morphology)
If when centered on a pixel,
any pixel does not match the kernel,
set all pixels in element to opposite of that pixel.
Dilation = maximum filter.
Replace the value of a pixel by the maximal value covered by the structuring element.

Suppose:
A is the following 11 x 11 matrix, and
B is the following 3 x 3 matrix:
0 0 0 0 0 0 0 0 0 0 0
0 1 1 1 1 0 0 1 1 1 0
0 1 1 1 1 0 0 1 1 1 0
0 1 1 1 1 1 1 1 1 1 0
0 1 1 1 1 1 1 1 1 1 0 1 1 1
0 1 1 0 0 0 1 1 1 1 0 1 1 1
0 1 1 0 0 0 1 1 1 1 0 1 1 1
0 1 1 0 0 0 1 1 1 1 0
0 1 1 1 1 1 1 1 0 0 0
0 1 1 1 1 1 1 1 0 0 0
0 0 0 0 0 0 0 0 0 0 0

For each pixel in A that has a value of 1, superimpose B,
with the center of B aligned with the corresponding pixel in A.
Each pixel of every superimposed B is included in the dilation of A by B.
The dilation of A by B is given by this 11 x 11 matrix:
1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 0 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 0 0
1 1 1 1 1 1 1 1 1 0 0

https://en.wikipedia.org/wiki/Opening_(morphology)
Opening - the dilation of the erosion

https://en.wikipedia.org/wiki/Closing_(morphology)
Closing - the erosion of the dilation

See ndimage api for these methods:
http://scipy-lectures.org/intro/scipy.html#mathematical-morphology
http://scipy-lectures.org/advanced/image_processing/index.html#mathematical-morphology
http://scipy-lectures.org/packages/scikit-image/#mathematical-morphology

* Principles and examples of binary morphological filtering.
* An object in the image is described as the set (denoted X) of all coordinates of pixels belonging to that object.
* Morphological filters process this set using a second set, known as the structuring element (denoted S).
* Here the discussion is limited to structuring elements that are symmetrical with respect to their center element, s = (0, 0), indicated by the dot.
* dilation of X is defined as
* the set of all coordinates x for which the cross section of S placed at x (denoted Sx) with X is not empty
* The erosion of X is defined as
* the set of all x for which Sx is a subset of X.
* A dilation followed by an erosion (or vice versa) is called a closing (versus opening).
* These operations are named after the effects they produce, as illustrated.
* Many interesting morphological filters can be constructed by taking differences of two or more operations, such as in morphological edge detection.
* Other applications include
* skeletonization, which consists in a sequence of thinning operations producing the basic shape of objects, and
* granulometry, which uses a family of opening operations with increasingly larger structuring elements to compute the size distribution of objects in an image.

Turbinate image:
(a) as printed,
(b) after enhancement.

+++++++++++ Cahoot-20-3
https://umsystem.instructure.com/courses/97521/quizzes/298143

1.3.6 Geometric transformations

• Images (or parts thereof) need to undergo spatial (or geometrical) transformation prior to further processing or analysis.
• There are two aspects to this type of operation:

1. coordinate transformation and

• This concerns the mapping of input pixel positions to output pixel positions (and vice versa).
• Depending on the complexity of the problem, one commonly uses a rigid, or an affine, or a curved transformation.

2. image re-sampling.

• Image re-sampling concerns the issue of computing output pixel values based on:
• the input pixel values and the coordinate transformation.
• This is also known as image interpolation, for which many methods exist.

Geometrical image transformation by:

coordinate transformation
* Is concerned with how input pixel positions are mapped to output pixel positions.
* Many types of transformations (denoted T) exist.
* The most frequently used types are (in increasing order of complexity)
* rigid transformations (translations and rotations),
* affine transformations (rigid transformations plus scalings and skewings), and
* curved transformations (affine transformations plus certain nonlinear or elastic deformations). These are defined (or can be approximated) by polynomial functions (with degree n depending on the complexity of the transformation).

image re-sampling
* concerns the computation of the pixel values of the output image (denoted O) from the pixel values of the input image (denoted I).
* This is done by using the inverse transformation (denoted T−1) to map output grid positions (x0, y0) to input positions (x, y).
* The value at this point is then computed by interpolation from the values at neighboring grid positions, using a weighing function, also known as the interpolation kernel (denoted K).

1.3.7 Image restoration

* Examples of the effects of image restoration operations:
* background subtraction,
* noise reduction, and
* deconvolution.
* Intensity gradients may be removed by subtracting a background image.
* In some cases, this image may be obtained from the raw image itself by mathematically fitting a polynomial surface function through the intensities at selected points (indicated by the squares) corresponding to the background.
* Several filtering methods exist to reduce noise:
* Gaussian filtering blurs not only noise but all image structures.
* Median filtering is somewhat better at retaining object edges but has the tendency to eliminate very small objects (compare the circles in each image).
* The magnitude of these effects depends on the filter size.
* Non-linear diffusion filtering was designed specifically to preserve object edges while reducing noise.
* De-convolution methods aim to undo the blurring effects of the microscope optics and to restore small details.
* More sophisticated methods are also capable of the appearance of reducing noise.
* Whether noise itself is truly reduced, is another issue.

1.3.8 Co-localization analysis

An interesting question in many biological studies is:
To what degree two or more molecular objects (typically proteins) are active in the same specimen?

* Commonly used measures for quantitative co-localization analysis
* The aim of all these measures is to express in numbers, the degree of overlap between two fluorophores (captured in well separated channels), indicating the presence of the corresponding labeled molecules in the same or proximate physical locations (up to the optical resolution of the microscope).
* A visual impression of the co-occurrence of fluorophore intensities (I₁ and I₂) is given by the joint histogram
* (also referred to as the scatter plot or fluorogram).
* Some co-localization measures are:
* computed over the entire images,
* restricted to certain intensity ranges (indicated by the squares in the joint histograms).
* Among the first are
* Pearson’s correlation coefficient (denoted r_P) and
* the so-called overlap coefficient (denoted r and computed from the sub-coefficients k₁ and k₂).
* Both coefficients are insensitive to intensity scalings (due to photo-bleaching or a difference in signal amplification), while the former is also insensitive to intensity offsets (different background levels).
* The value of r_P may range from −1 to 1, and is therefore at odds with intuition.
* Its squared value is perhaps more valuable as it expresses the quality of a least-squares fitting of a line through the points in the scatter plot.
* The other measures range from 0 to 1.
* The value of r is meaningful only when the amount of fluorescence is approximately equal in both channels, that is when k₁ and k₂ have similar values.
* Manders’ co-localization coefficients (denoted m1 and m2) are intuitively most clear, but require careful separation of signal and background in both channels:
* the denominators are computed over the entire images, but the numerators sum only those intensities in one channel for which the corresponding intensity in the other channel is within a predefined range
* (the left and right and the top and bottom lines of the square region indicated in the joint histogram, for I₁ and I₂ respectively).

1.3.9 Neuron tracing and quantification

• Another biological image analysis problem, which occurs for example when studying the molecular mechanisms involved in neurite outgrowth and differentiation, is the length measurement of elongated image structures.
• For practical reasons, many neuronal morphology studies were, and still are, performed using 2D imaging.
• This often results in ambiguous images.
• At many places it is unclear whether neurites are branching or crossing.
• Tracing such structures and building neuritic trees for morphological analysis requires the input of human experts to resolve ambiguities.

* Tracing of neurite outgrowth using interactive segmentation methods.
* To reduce background intensity gradients (shading effects) or discontinuities (due to the stitching of scans with different background levels), the image features exploited here are the second-order derivatives, obtained by convolution with the second-order Gaussian derivative kernels at a proper scale (to suppress noise).
* These constitute a so-called Hessian matrix at every pixel in the image.
* Its eigenvalues and eigenvectors are used to construct an ellipse (as indicated), whose size is representative of the local neurite contrast and whose orientation corresponds to the local neurite orientation.
* In turn, these properties are used to compute a cost image (with dark values indicating a lower cost and bright values a higher cost) and vector field (not shown), which together guide a search algorithm that finds the paths of minimum cumulative cost between a start point and all other points in the image.
* By using graphics routines, the path to the current cursor position (indicated by the cross) is shown at interactive speed while the user selects the optimal path based on visual judgment.
* Once tracing is finished, neurite lengths and statistics can be computed automatically.
* This is the underlying principle of the NeuronJ tracing tool, freely available as a plugin to the ImageJ program.

1.3.10 Particle detection and tracking, and Cell tracking

• One of the major challenges of biomedical research in the post-genomic era is the unraveling of not just the spatial, but the spatiotemporal relationships of complex bio-molecular systems.
• Naturally this involves the acquisition of time-lapse image series, and the tracking of objects over time.
• From an image analysis point of view, a distinction can be made between tracking of single molecules (or complexes) and tracking of entire cells

Particle tracking methods consist of two stages
1. the detection of individual particles per time frame, and
2. the linking of particles detected in successive frames

• Motion estimation of cells is another frequently occurring problem in biological research.

• In particle tracking studies, for example, cell movement may muddle the motion analysis of intra-cellular components and needs to be corrected for.

• Contrary to single molecules or molecular complexes, which are sub-resolution objects appearing as PSF-shaped spots in the images, cells are relatively large objects (with respect to pixel size), with a distinct shape.

Challenges in particle and cell tracking.
* Regarding particle tracking, currently one of the best approaches to detection of fluorescent tags is by least-squares fitting of a model of the intensity distribution to the image data.
* Because the tags are sub-resolution particles, they appear as diffraction-limited spots in the images and therefore can be modeled well by a mixture of Gaussian functions, each with its own amplitude scaling factor, standard deviation, and center position.
* Usually the detection is done separately for each time step, resulting in a list of potential particle positions and corresponding features, to be linked between time steps.
* The linking is hampered by the fact that the number of detected particles may be different for each time step.
* In cell tracking, a contour model (surface model in the case of 3D time-lapse experiments) is often used for segmentation.
* Commonly used models consist of control points, which are interpolated using smooth basis functions (typically B-splines) to form continuous, closed curves.
* The model must be flexible enough to handle geometrical as well as topological shape changes (cell division).
* The fitting is done by (constrained) movement of the control points to minimize some predefined energy functional computed from image-dependent information (intensity distributions inside and outside the curve) as well as image-independent information (a-priori knowledge about cell shape and dynamics).
* Finally, trajectories can be visualized by representing them as tubes (segments) and spheres (time points) and using surface rendering.

Reminder:

1.3.11 Visualization

Several visualization methods exist, which vary in efficiency:

1.3.11.1 Ray

• Visualization methods that produce a viewable image of higher-dimensional image data without requiring an explicit geometrical representation of that data are called volume rendering methods.
• A commonly used, flexible and easy to understand volume rendering method is ray casting, or ray tracing.
• With this method, the value of each pixel in the view image is determined by “casting a ray” into the image data and evaluating the data encountered along that ray using a predefined ray function.

1.3.11.2 Surface rendering

• In contrast with volume rendering methods, which in principle take into account all data along rays and therefore enable the visualization of object interiors, surface rendering methods visualize only object surfaces.
• Generally, this requires a mathematical description of the surfaces in terms of primitive geometrical entities: points, lines, triangles, polygons, or polynomial curves and surface patches, in particular splines.
• Such descriptions are derived from a segmentation of the image data into meaningful parts (objects versus background).
• This constitutes the most critical aspect of surface rendering: the value of the visualization depends almost entirely on the correctness of the segmentation.

* Visualization of volumetric image data using volume rendering and surface rendering methods.
* Volume rendering methods do not require an explicit geometrical representation of the objects of interest present in the data.
* A commonly used volume rendering method is ray casting: for each pixel in the view image, a ray is cast into the data, and the intensity profile along the ray is fed to a ray function, which determines the output value, such as the maximum, average, or minimum intensity, or accumulated “opacity” (derived from intensity or gradient magnitude information).
* By contrast, surface rendering methods require a segmentation of the objects (usually obtained by thresholding), from which a surface representation (triangulation) is derived, allowing very fast rendering by graphics hardware.
* To reduce the effects of noise, Gaussian smoothing is often applied as a pre-processing step prior to segmentation.
* As shown, both operations have a substantial influence on the final result: by slightly changing the degree of smoothing or the threshold level, objects may appear (dis)connected while in fact they are not.
* Therefore it is recommended to establish optimal parameter values for both steps while inspecting the effects on the original image data rather than looking directly at the renderings.

1.4 General vision methods: edges and gradients

1.4.1 Low-level feature detecting algorithms

https://en.wikipedia.org/wiki/Feature_detection_(computer_vision)

1.4.1.1 Edges

https://en.wikipedia.org/wiki/Edge_detection

Canny:
https://en.wikipedia.org/wiki/Canny_edge_detection

Example problem:
https://en.wikipedia.org/wiki/Sobel_operator

How to detect edges?
Edge detection: zoom in 1 row

Of one row:
intensity,
derivative, and
smoothed derivative

Do this with rows, then cols, and you get edges

Sobel does x-derivative approx and x-derivative approx
(Convolution kernels above)

+++++++++++ Cahoot-20-4
https://umsystem.instructure.com/courses/97521/quizzes/298144

http://scipy-lectures.org/advanced/image_processing/index.html#edge-detection
(sobel)

1.4.1.2 Gradients

1.4.1.3 Histogram of oriented gradients (HOG)

Feature histogram

HOGles

* Middle row displays roughly what the computer with HOG might see.
* Top row is HOG.
* Doing HOG first helps to scan/detect objects (a form of feature extraction)

1.5 Labeling and segmentation

What is segmentation on biological images?

How do you label all of the cell or nucleus data in these images with the same algorithm?
Below are images from your next assignment:
../../../../index/Classes/Bioinformatics/Content.html
(watch video in class)

1.5.1 Segmentation theory

1.5.1.1 Segmentation by histogram thresholding

https://en.wikipedia.org/wiki/Otsu’s_method
In Otsu’s method we exhaustively search for the threshold that minimizes the intra-class variance (the variance within the class), defined as a weighted sum of variances of the two classes
The algorithm assumes that the image contains two classes of pixels following bi-modal histogram (foreground pixels and background pixels), it then calculates the optimum threshold separating the two classes so that their combined spread (intra-class variance) is minimal, or equivalently (because the sum of pairwise squared distances is constant), so that their inter-class variance is maximal.

https://en.wikipedia.org/wiki/Balanced_histogram_thresholding
This method weighs the histogram, checks which of the two sides is heavier, and removes weight from the heavier side until it becomes the lighter.
It repeats the same operation until the edges of the weighing scale meet.

1.5.1.2 Segmentation with clustering algorithms

Recall k-means
1. Pick K cluster centers, either randomly or based on some heuristic method, for example K-means++
2. Assign each pixel in the image to the cluster that minimizes the distance between the pixel and the cluster center
3. Re-compute the cluster centers by averaging all of the pixels in the cluster
4. Repeat steps 2 and 3 until convergence is attained (i.e. no pixels change clusters)

Original image

k-means on that image’s X-features (pixel values)

* Recall/consider?
* What is distance?
* How were colors chosen?

• Clustering here sets boundaries for a groupwise thresholding.
• In this case, distance is the squared or absolute difference between a pixel and a cluster center.
• The difference is typically based on pixel color, intensity, texture, and location, or a weighted combination of these factors.

1.5.1.3 Segmentation via edge detection

https://en.wikipedia.org/wiki/Edge_detection
* Edge detection is a well-developed field on its own within image processing.
* Region boundaries and edges are closely related, since there is often a sharp adjustment in intensity at the region boundaries.
* Edge detection techniques have therefore been used as the base of another segmentation technique.
* Segmentation methods can also be applied to edges obtained from edge detectors.

1.5.1.4 Many many more segmentation methods

https://en.wikipedia.org/wiki/Image_segmentation

1.5.2 Python demos to go over in class

1.5.2.1 scipy from scipy-lectures

May be more out-of-date on some functions
* http://scipy-lectures.org/intro/scipy.html#connected-components-and-measurements-on-images
* http://scipy-lectures.org/advanced/image_processing/index.html#segmentation
* (based on threshold and more)
* http://scipy-lectures.org/advanced/image_processing/index.html#measuring-objects-properties-ndimage-measurements

1.5.2.2 scipy docs

• https://docs.scipy.org/doc/scipy/reference/tutorial/ndimage.html#segmentation-and-labeling
  • (no easy images for in-class)
• https://docs.scipy.org/doc/scipy/reference/tutorial/ndimage.html#object-measurements
  • (no easy images for in-class)

1.5.2.3 scikit-image from scipy-lectures

May be more out-of-date on some functions
* http://scipy-lectures.org/packages/scikit-image/#image-segmentation (based on threshold and more)
* For example: http://scipy-lectures.org/packages/scikit-image/auto_examples/plot_labels.html#sphx-glr-packages-scikit-image-auto-examples-plot-labels-py (in class review)
* http://scipy-lectures.org/packages/scikit-image/#measuring-regions-properties (for labeling)
* http://scipy-lectures.org/packages/scikit-image/#data-visualization-and-interaction

1.5.2.4 scikit-image docs

• https://scikit-image.org/docs/stable/user_guide/tutorial_segmentation.html
• https://scikit-image.org/docs/stable/auto_examples/color_exposure/plot_regional_maxima.html#sphx-glr-auto-examples-color-exposure-plot-regional-maxima-py
• https://scikit-image.org/docs/stable/auto_examples/features_detection/plot_template.html#sphx-glr-auto-examples-features-detection-plot-template-py (template matching can be useful for consistent objects, but not all)
• https://scikit-image.org/docs/stable/auto_examples/features_detection/plot_blob.html#sphx-glr-auto-examples-features-detection-plot-blob-py
• https://scikit-image.org/docs/stable/auto_examples/segmentation/plot_peak_local_max.html#sphx-glr-auto-examples-segmentation-plot-peak-local-max-py
• https://scikit-image.org/docs/stable/auto_examples/segmentation/plot_label.html#sphx-glr-auto-examples-segmentation-plot-label-py (simple, good in class)
• https://scikit-image.org/docs/stable/auto_examples/segmentation/plot_random_walker_segmentation.html#sphx-glr-auto-examples-segmentation-plot-random-walker-segmentation-py
• https://scikit-image.org/docs/stable/auto_examples/segmentation/plot_watershed.html#sphx-glr-auto-examples-segmentation-plot-watershed-py
• https://scikit-image.org/docs/stable/auto_examples/applications/plot_coins_segmentation.html#sphx-glr-auto-examples-applications-plot-coins-segmentation-py
  • (good comparison of methods, overview; do in Spyder)

1.5.3 How to judge accuracy of a labeling

https://en.wikipedia.org/wiki/Jaccard_index
Recall the rand index we covered previously!

Jaccard index, also known as Intersection over Union (IOU),
and the Jaccard similarity coefficient
(originally given the French name coefficient de communauté by Paul Jaccard),
is a statistic used for gauging the similarity and diversity of sample sets.
The Jaccard coefficient measures similarity between finite sample sets,
and is defined as the size of the intersection,
divided by the size of the union of the sample sets:

$J(A,B)={{|AB|} }={{|AB|} } $

If A and B are both empty,
we define J(A,B) = 1

\(0\leq J(A,B)\leq 1\)

Union —————————— Intersection

1.6 Intro on nuclei from data science bowl

https://kaggle.com/competitions/data-science-bowl-2018/stage1/teaching-notebook-for-total-imaging-newbies.py
https://kaggle.com/competitions/data-science-bowl-2018/stage1/basic-skimage-nuclei.py