Previous: 20-ImageBasics.html

**Biological imaging** may refer to any imaging
technique used in biology.

https://en.wikipedia.org/wiki/Biological_imaging

https://en.wikipedia.org/wiki/Bioimage_informatics

https://en.wikipedia.org/wiki/Automated_tissue_image_analysis

https://en.wikipedia.org/wiki/Digital_pathology

https://en.wikipedia.org/wiki/Image_segmentation

https://en.wikipedia.org/wiki/Medical_imaging

https://en.wikipedia.org/wiki/Computer-aided_diagnosis

https://en.wikipedia.org/wiki/Medical_imaging

**Medical imaging**is the technique and process of creating visual representations of the interior of a body for clinical analysis and medical intervention, as well as visual representation of the function of some organs or tissues (physiology).- Medical imaging seeks to reveal internal structures hidden by the skin and bones, as well as to diagnose and treat disease.
- Medical imaging also establishes a database of normal anatomy and physiology to make it possible to identify abnormalities.
- Although imaging of removed organs and tissues can be performed for medical reasons, such procedures are usually considered part of pathology instead of medical imaging.
- As a discipline and in its widest sense, it is part of biological imaging and incorporates radiology which uses the imaging technologies of X-ray radiography, magnetic resonance imaging, medical ultrasonography or ultrasound, endoscopy, elastography, tactile imaging, thermography, medical photography and nuclear medicine functional imaging techniques as positron emission tomography (PET) and Single-photon emission computed tomography (SPECT).

https://en.wikipedia.org/wiki/Bioimage_informatics

**Bioimage informatics**is a sub-field of bioinformatics and computational biology.- It focuses on the use of computational techniques to analyze bioimages, especially cellular and molecular images, at large scale and high throughput.
- The goal is to obtain useful knowledge out of complicated and heterogeneous image and related meta-data.
- Automated microscopes are able to collect large numbers of images with minimal intervention.
- This has led to a data explosion, which absolutely requires automatic processing.
**Additionally, and surprisingly, for several of these tasks, there is evidence that automated systems can perform better than humans.****In addition, automated systems are unbiased, unlike human based analysis whose evaluation may (even unconsciously) be influenced by the desired outcome.**- There has been an increasing focus on developing novel image
processing, computer vision, data mining, database and visualization
techniques to extract, compare, search and manage the biological
knowledge in these data-intensive problems.

A stained histologic specimen, sandwiched between a glass microscope slide and cover-slip, mounted on the stage of a light microscope.

Fluorescent image of a cell in telophase.

Multiple dyes were imaged and are shown in different colours.

Microscopic view of a histologic specimen of human lung tissue,

stained with hematoxylin and eosin.

Overview of a typical process:

- Subcellular location analysis was one of the initial problems in this field.
- In its supervised mode, the problem is to learn a classifier that can recognize the major cell organelles from cell images.
- Methods used are based on machine learning, building a
discriminative classifier based on numeric features computed from the
image.

- Sub-cellular Location Example.
- Examples of different patterns are mapped into a two-dimensional space by computing different image features.
- Image of unknown proteins are similarly mapped into this space and a nearest neighbor search or other classifier can be used for assigning a location to this unclassified protein.

- Automated tissue image analysis is a process by which computer-controlled automatic test equipment is used to evaluate tissue samples, using computations to derive quantitative measurements from an image to avoid subjective errors.
- In a typical application, automated tissue image analysis could be used to measure the aggregate activity of cancer cells in a biopsy of a cancerous tumor taken from a patient.
- In breast cancer patients, for example, automated tissue image
analysis may be used to test for high levels of proteins known to be
present in more aggressive forms of breast cancers.

- High throughput screens using automated imaging technology (sometimes called high-content screening) have become a standard method for both drug discovery and basic biological research.
- Using multi-well plates, robotics, and automated microscopy, the same assay can be applied to a large library of possible reagents (typically either small molecules or RNAi) very rapidly, obtaining thousands of images in a short amount of time.
- Due to the high volume of data generated, automatic image analysis
is a necessity.

An automated confocal image reader.

- Image segmentation is the process of partitioning a digital image
into multiple
**segments**- (sets of pixels, also known as super-pixels).

- The goal of segmentation is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze.
- Image segmentation is typically used to locate objects and boundaries (lines, curves, etc.) in images.
- More precisely, image segmentation is the process of assigning a
**label**to every pixel in an image such that pixels with the same label share certain characteristics.

Example image for segmentation problem.

Shown are nuclei of mouse NIH 3T3,

stained with Hoechst and a segmentation in red.

Volume segmentation of a 3D-rendered CT scan of the thorax:

The anterior thoracic wall, the airways and the pulmonary vessels
anterior to the root of the lung have been digitally removed in order to
visualize thoracic contents:

* blue: pulmonary arteries

* red: pulmonary veins (and also the abdominal wall)

* yellow: the mediastinum

* violet: the diaphragm

- Tracking is another traditional image processing problem which appears in bioimage informatics.
- The problem is to relate objects that appear in subsequent frames of a film.
- As with segmentation, the problem can be posed in both two- and
three-dimensional forms.

- Image registration is the process of transforming different sets of data into one coordinate system.
- Data may be multiple photographs, data from different sensors, times, depths, or viewpoints.
- It is used in computer vision, medical imaging, military automatic target recognition, and compiling and analyzing images and data from satellites.
- Registration is needed to compare or integrate the data obtained from these different measurements.

Segment, then register:

Imaging methods:

Breast cancer image (e.g., a target classification)

Taking sequential microscope image slices and reconstructing a 3D
view on tissue.

**+++++++++++ Cahoot-20-2**

https://umsystem.instructure.com/courses/97521/quizzes/298142

Many of these methods are general,

but are particularly common in bioimage analysis.

A conceptual pipeline.

The specimen is imaged using any of today’s microscopes,

modeled by the input image f(v) passing through the blocks of

PSF (properties of the microscope, described by convolution with h(v))
and

A/D conversion (analog-to-digital conversion, effects of sampling and
digitization together with uncertainty introduced by various sources of
noise),

producing a digital image g_{n}.

That digital image is then restored either via a

de-noising followed by de-convolution, or via

joint de-noising/de-convolution,

producing a digital image f_{n} .

Various options are possible, the image could go through a

registration/mosaicing processing, producing r_{n},

segmentation/tracing/tracking, producing s_{n,}

data analysis/modeling/simulations block, with the output
y_{n}.

At the input/output of each block,

one can join the pathway to either skip a block,

or send feedback to previous block(s) in the system.

Stages in image analysis illustrated using algal image:

* (a) detail from image;

* (b) same detail after application of 5 × 5 moving median filter;

* (c) histogram, on a square root scale, of pixel values after
filtering, with an arrow to indicate the threshold;

* (d) result of thresholding image at pixel value 120, to produce a
binary image;

* (e) result of applying morphological opening to (d);

* (f) separated objects in (e) counted.

* Images viewed as matrices.

* The overview is not meant to be exhaustive, but reflects some of the
more frequently used modes of image acquisition in biological and
medical imaging, where the number of dimensions is typically one to
five, with each dimension corresponding to an independent physical
parameter:

* three to space,

* one (usually denoted t) to time, and

* one to wavelength, or color, or more generally to any spectral
parameter (we denote this dimension s here).

* In other words, images are discrete functions, I (x, y, z, t,
s),

* with each set of coordinates yielding the value of a unique
sample

* (indicated by the small squares, the number of which is obviously
arbitrary here).

* Note that the dimensionality of an image (indicated in the top row) is
given by the number of coordinates that are varied during
acquisition.

* To avoid confusion in characterizing an image, it is advisable to add
adjectives indicating which dimensions were scanned, rather than
mentioning just dimensionality.

* For example, a 4D image may either be a spatially 2D multi-spectral
time-lapse image, or a spatially 3D time-lapse image.

**Image processing**

* takes an image as input and produces a modified version of it

* (in the case shown, the object contours are enhanced using an
operation known as edge detection, described in more detail elsewhere in
this booklet).

**Image analysis**

* concerns the extraction of object features from an image.

**Computer graphics**

* is the inverse of image analysis:

* it produces an image from given primitives, which could be numbers
(the case shown), or parameterized shapes, or mathematical
functions.

**Computer vision**

* aims at producing a high-level interpretation of what is contained in
an image; this is also known as image understanding.

**Visualization**

* transforms higher-dimensional image data into a more primitive
representation to facilitate exploring the data.

- Among the simplest image processing operations are those that pass along each image pixel, and produce an output value that depends only on the corresponding input value and some mapping function.
- These are also called
**point operations**. - If the mapping function is the same for each pixel, we speak of a global intensity transformation.

* Examples of intensity transformations based on a global mapping
function:

* contrast stretching, intensity inversion, intensity thresholding, and
histogram equalization.

* The top row shows the images used as input for each of the
transformations.

* The second row shows for each image the mapping function used (denoted
M), with the histogram of the input image shown in the background.

* The bottom row shows for each input image the corresponding output
image resulting from applying the mapping function:

* O( x, y ) = M( I (x, y )).

* It is clear from the mapping functions that contrast stretching and
histogram equalization both distribute the most frequently occurring
intensities over a wider range of values, thereby increasing image
contrast.

* The former transformation is suitable in the case of uni-modal
histograms, whereas the latter is particularly suited for images having
multi-modal histograms.

Check out in class:

https://scikit-image.org/docs/stable/auto_examples/color_exposure/plot_equalize.html#sphx-glr-auto-examples-color-exposure-plot-equalize-py

- Instead of considering just the corresponding input pixel when
computing a value for each output pixel (as in intensity
transformation), one could also take into account the
**values of adjacent input pixels**. - Image processing operations based on this principle are called
**neighborhood operations**, or**image filtering operations** - They are usually designed to filter out (enhance or reduce) specific
image information.

They can be classified into linear versus nonlinear filters:

**Linear**filtering operations:- Compute the output pixel value as a linear combination (weighing and summation) of the values of the corresponding input pixel and its neighbors.
- Examples of kernels include:
- averaging,
- sharpening,
- smoothing, and
- derivative filters of varying sizes

**Nonlinear**filtering operations- Combine neighboring input pixel values in a nonlinear fashion in producing an output pixel value.
- They can not be described as a convolution process.
- Examples include:
- median filtering (which for each output pixel computes the value as the median of the corresponding input values in a neighborhood of given size) and
- min-filtering or max-filtering (where the output value is computed as, respectively, the minimum or the maximum value in a neighborhood around the corresponding input pixel).

https://en.wikipedia.org/wiki/Kernel_(image_processing)

* Principles and examples of convolution filtering.

* The value of an output pixel is computed as a linear combination
(weighing and summation) of the value of the corresponding input pixel
and of its neighbors.

* The weight factor assigned to each input pixel is given by the
convolution kernel (denoted K).

* In principle, kernels can be of any size.

* Examples of commonly used kernels of size 3 × 3 pixels include

* the averaging filter,

* the sharpening filter, and

* the **Sobel** x- or y-derivative (approximation)
filters.

* The Gaussian filter is often used as a smoothing filter.

* It has a free parameter (standard deviation σ) which determines the
size of the kernel (usually cut off at m = 3σ) and therefore the degree
of smoothing.

* The derivatives of this kernel are often used to compute image
derivatives at different scales, as for example in Canny edge
detection.

* The scale parameter, σ, should be chosen such that the resulting
kernel matches the structures to be filtered.

Local filters

http://scipy-lectures.org/advanced/image_processing/index.html#image-filtering

http://scipy-lectures.org/packages/scikit-image/#local-filters

Code to show:

`https://gitlab.com/bio-data/computer-vision/bio_images/morphology.py`

The basic morphological operators are erosion, dilation, opening and
closing.

https://en.wikipedia.org/wiki/Mathematical_morphology

https://en.wikipedia.org/wiki/Erosion_(morphology)

If when centered on a pixel,

any of kernel does not match,

set all in kernel to value of middle pixel.

The erosion of a point is the minimum of the points in its
neighborhood,

with that neighborhood defined by the structuring element.

Erosion = minimum filter.

Replace the value of a pixel by the minimal value covered by the
structuring element.

Suppose:

A is a 13 x 13 matrix, and

B is a 3 x 3 matrix:

`1 1 1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 0 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1 1 1`

Assuming that the origin B is at its center,

for each pixel in A superimpose the origin of B,

if B is completely contained by A,

then the pixel is retained,

else deleted.

Therefore the Erosion of A by B is given by this 13 x 13
matrix:

`0 0 0 0 0 0 0 0 0 0 0 0 0`

`0 1 1 1 1 0 0 0 1 1 1 1 0`

`0 1 1 1 1 0 0 0 1 1 1 1 0`

`0 1 1 1 1 1 1 1 1 1 1 1 0`

`0 1 1 1 1 1 1 1 1 1 1 1 0`

`0 1 1 1 1 1 1 1 1 1 1 1 0`

`0 1 1 1 1 1 1 1 1 1 1 1 0`

`0 1 1 1 1 1 1 1 1 1 1 1 0`

`0 1 1 1 1 1 1 1 1 1 1 1 0`

`0 1 1 1 1 1 1 1 1 1 1 1 0`

`0 1 1 1 1 1 1 1 1 1 1 1 0`

`0 1 1 1 1 1 1 1 1 1 1 1 0`

`0 0 0 0 0 0 0 0 0 0 0 0 0`

This means that only when B is completely contained inside A,

that the pixels values are retained,

otherwise it gets deleted or eroded.

https://en.wikipedia.org/wiki/Dilation_(morphology)

If when centered on a pixel,

any pixel does not match the kernel,

set all pixels in element to opposite of that pixel.

Dilation = maximum filter.

Replace the value of a pixel by the maximal value covered by the
structuring element.

Suppose:

A is the following 11 x 11 matrix, and

B is the following 3 x 3 matrix:

`0 0 0 0 0 0 0 0 0 0 0`

`0 1 1 1 1 0 0 1 1 1 0`

`0 1 1 1 1 0 0 1 1 1 0`

`0 1 1 1 1 1 1 1 1 1 0`

`0 1 1 1 1 1 1 1 1 1 0 1 1 1`

`0 1 1 0 0 0 1 1 1 1 0 1 1 1`

`0 1 1 0 0 0 1 1 1 1 0 1 1 1`

`0 1 1 0 0 0 1 1 1 1 0`

`0 1 1 1 1 1 1 1 0 0 0`

`0 1 1 1 1 1 1 1 0 0 0`

`0 0 0 0 0 0 0 0 0 0 0`

For each pixel in A that has a value of 1, superimpose B,

with the center of B aligned with the corresponding pixel in A.

Each pixel of every superimposed B is included in the dilation of A by
B.

The dilation of A by B is given by this 11 x 11 matrix:

`1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 0 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 1 1`

`1 1 1 1 1 1 1 1 1 0 0`

`1 1 1 1 1 1 1 1 1 0 0`

https://en.wikipedia.org/wiki/Opening_(morphology)

Opening - the dilation of the erosion

https://en.wikipedia.org/wiki/Closing_(morphology)

Closing - the erosion of the dilation

See ndimage api for these methods:

http://scipy-lectures.org/intro/scipy.html#mathematical-morphology

http://scipy-lectures.org/advanced/image_processing/index.html#mathematical-morphology

http://scipy-lectures.org/packages/scikit-image/#mathematical-morphology

* Principles and examples of binary morphological filtering.

* An object in the image is described as the set (denoted X) of all
coordinates of pixels belonging to that object.

* Morphological filters process this set using a second set, known as
the structuring element (denoted S).

* Here the discussion is limited to structuring elements that are
symmetrical with respect to their center element, s = (0, 0), indicated
by the dot.

* **dilation** of X is defined as

* the set of all coordinates x for which the cross section of S placed
at x (denoted Sx) with X is not empty

* The **erosion** of X is defined as

* the set of all x for which Sx is a subset of X.

* A dilation followed by an erosion (or vice versa) is called a
**closing** (versus opening).

* These operations are named after the effects they produce, as
illustrated.

* Many interesting morphological filters can be constructed by taking
differences of two or more operations, such as in morphological edge
detection.

* Other applications include

* skeletonization, which consists in a sequence of thinning operations
producing the basic shape of objects, and

* granulometry, which uses a family of opening operations with
increasingly larger structuring elements to compute the size
distribution of objects in an image.

Turbinate image:

(a) as printed,

(b) after enhancement.

**+++++++++++ Cahoot-20-3**

https://umsystem.instructure.com/courses/97521/quizzes/298143

- Images (or parts thereof) need to undergo spatial (or geometrical) transformation prior to further processing or analysis.
- There are two aspects to this type of operation:

- coordinate transformation and

- This concerns the mapping of input pixel positions to output pixel positions (and vice versa).
- Depending on the complexity of the problem, one commonly uses a rigid, or an affine, or a curved transformation.

- image re-sampling.

- Image re-sampling concerns the issue of computing output pixel values based on:
- the input pixel values and the coordinate transformation.
- This is also known as image interpolation, for which many methods exist.

Geometrical image transformation by:

**coordinate transformation**

* Is concerned with how input pixel positions are mapped to output pixel
positions.

* Many types of transformations (denoted T) exist.

* The most frequently used types are (in increasing order of
complexity)

* rigid transformations (translations and rotations),

* affine transformations (rigid transformations plus scalings and
skewings), and

* curved transformations (affine transformations plus certain nonlinear
or elastic deformations). These are defined (or can be approximated) by
polynomial functions (with degree n depending on the complexity of the
transformation).

**image re-sampling**

* concerns the computation of the pixel values of the output image
(denoted O) from the pixel values of the input image (denoted I).

* This is done by using the inverse transformation (denoted T−1) to map
output grid positions (x0, y0) to input positions (x, y).

* The value at this point is then computed by interpolation from the
values at neighboring grid positions, using a weighing function, also
known as the interpolation kernel (denoted K).

* Examples of the effects of image restoration operations:

* background subtraction,

* noise reduction, and

* deconvolution.

* Intensity gradients may be removed by subtracting a background
image.

* In some cases, this image may be obtained from the raw image itself by
mathematically fitting a polynomial surface function through the
intensities at selected points (indicated by the squares) corresponding
to the background.

* Several filtering methods exist to reduce noise:

* Gaussian filtering blurs not only noise but all image
structures.

* Median filtering is somewhat better at retaining object edges but has
the tendency to eliminate very small objects (compare the circles in
each image).

* The magnitude of these effects depends on the filter size.

* Non-linear diffusion filtering was designed specifically to preserve
object edges while reducing noise.

* De-convolution methods aim to undo the blurring effects of the
microscope optics and to restore small details.

* More sophisticated methods are also capable of the appearance of
reducing noise.

* Whether noise itself is truly reduced, is another issue.

An interesting question in many biological studies is:

To what degree two or more molecular objects (typically proteins) are
active in the same specimen?

* Commonly used measures for quantitative co-localization analysis

* The aim of all these measures is to express in numbers, the degree of
overlap between two fluorophores (captured in well separated channels),
indicating the presence of the corresponding labeled molecules in the
same or proximate physical locations (up to the optical resolution of
the microscope).

* A visual impression of the co-occurrence of fluorophore intensities
(I_{1} and I_{2}) is given by the joint histogram

* (also referred to as the scatter plot or fluorogram).

* Some co-localization measures are:

* computed over the entire images,

* restricted to certain intensity ranges (indicated by the squares in
the joint histograms).

* Among the first are

* Pearson’s correlation coefficient (denoted r_{P}) and

* the so-called overlap coefficient (denoted r and computed from the
sub-coefficients k_{1} and k_{2}).

* Both coefficients are insensitive to intensity scalings (due to
photo-bleaching or a difference in signal amplification), while the
former is also insensitive to intensity offsets (different background
levels).

* The value of r_{P} may range from −1 to 1, and is therefore at
odds with intuition.

* Its squared value is perhaps more valuable as it expresses the quality
of a least-squares fitting of a line through the points in the scatter
plot.

* The other measures range from 0 to 1.

* The value of r is meaningful only when the amount of fluorescence is
approximately equal in both channels, that is when k_{1} and
k_{2} have similar values.

* Manders’ co-localization coefficients (denoted m1 and m2) are
intuitively most clear, but require careful separation of signal and
background in both channels:

* the denominators are computed over the entire images, but the
numerators sum only those intensities in one channel for which the
corresponding intensity in the other channel is within a predefined
range

* (the left and right and the top and bottom lines of the square region
indicated in the joint histogram, for I_{1} and I_{2}
respectively).

- Another biological image analysis problem, which occurs for example when studying the molecular mechanisms involved in neurite outgrowth and differentiation, is the length measurement of elongated image structures.
- For practical reasons, many neuronal morphology studies were, and still are, performed using 2D imaging.
- This often results in ambiguous images.
- At many places it is unclear whether neurites are branching or crossing.
- Tracing such structures and building neuritic trees for morphological analysis requires the input of human experts to resolve ambiguities.

* Tracing of neurite outgrowth using interactive segmentation
methods.

* To reduce background intensity gradients (shading effects) or
discontinuities (due to the stitching of scans with different background
levels), the image features exploited here are the second-order
derivatives, obtained by convolution with the second-order Gaussian
derivative kernels at a proper scale (to suppress noise).

* These constitute a so-called Hessian matrix at every pixel in the
image.

* Its eigenvalues and eigenvectors are used to construct an ellipse (as
indicated), whose size is representative of the local neurite contrast
and whose orientation corresponds to the local neurite
orientation.

* In turn, these properties are used to compute a cost image (with dark
values indicating a lower cost and bright values a higher cost) and
vector field (not shown), which together guide a search algorithm that
finds the paths of minimum cumulative cost between a start point and all
other points in the image.

* By using graphics routines, the path to the current cursor position
(indicated by the cross) is shown at interactive speed while the user
selects the optimal path based on visual judgment.

* Once tracing is finished, neurite lengths and statistics can be
computed automatically.

* This is the underlying principle of the NeuronJ tracing tool, freely
available as a plugin to the ImageJ program.

- One of the major challenges of biomedical research in the post-genomic era is the unraveling of not just the spatial, but the spatiotemporal relationships of complex bio-molecular systems.
- Naturally this involves the acquisition of time-lapse image series, and the tracking of objects over time.
- From an image analysis point of view, a distinction can be made between tracking of single molecules (or complexes) and tracking of entire cells

Particle tracking methods consist of two stages

1. the detection of individual particles per time frame, and

2. the linking of particles detected in successive frames

Motion estimation of cells is another frequently occurring problem in biological research.

In particle tracking studies, for example, cell movement may muddle the motion analysis of intra-cellular components and needs to be corrected for.

Contrary to single molecules or molecular complexes, which are sub-resolution objects appearing as PSF-shaped spots in the images, cells are relatively large objects (with respect to pixel size), with a distinct shape.

Challenges in particle and cell tracking.

* Regarding particle tracking, currently one of the best approaches to
detection of fluorescent tags is by least-squares fitting of a model of
the intensity distribution to the image data.

* Because the tags are sub-resolution particles, they appear as
diffraction-limited spots in the images and therefore can be modeled
well by a mixture of Gaussian functions, each with its own amplitude
scaling factor, standard deviation, and center position.

* Usually the detection is done separately for each time step, resulting
in a list of potential particle positions and corresponding features, to
be linked between time steps.

* The linking is hampered by the fact that the number of detected
particles may be different for each time step.

* In cell tracking, a contour model (surface model in the case of 3D
time-lapse experiments) is often used for segmentation.

* Commonly used models consist of control points, which are interpolated
using smooth basis functions (typically B-splines) to form continuous,
closed curves.

* The model must be flexible enough to handle geometrical as well as
topological shape changes (cell division).

* The fitting is done by (constrained) movement of the control points to
minimize some predefined energy functional computed from image-dependent
information (intensity distributions inside and outside the curve) as
well as image-independent information (a-priori knowledge about cell
shape and dynamics).

* Finally, trajectories can be visualized by representing them as tubes
(segments) and spheres (time points) and using surface rendering.

Reminder:

Several visualization methods exist, which vary in efficiency:

- Visualization methods that produce a viewable image of higher-dimensional image data without requiring an explicit geometrical representation of that data are called volume rendering methods.
- A commonly used, flexible and easy to understand volume rendering method is ray casting, or ray tracing.
- With this method, the value of each pixel in the view image is determined by “casting a ray” into the image data and evaluating the data encountered along that ray using a predefined ray function.

- In contrast with volume rendering methods, which in principle take into account all data along rays and therefore enable the visualization of object interiors, surface rendering methods visualize only object surfaces.
- Generally, this requires a mathematical description of the surfaces in terms of primitive geometrical entities: points, lines, triangles, polygons, or polynomial curves and surface patches, in particular splines.
- Such descriptions are derived from a segmentation of the image data into meaningful parts (objects versus background).
- This constitutes the most critical aspect of surface rendering: the value of the visualization depends almost entirely on the correctness of the segmentation.

* Visualization of volumetric image data using volume rendering and
surface rendering methods.

* Volume rendering methods do not require an explicit geometrical
representation of the objects of interest present in the data.

* A commonly used volume rendering method is ray casting: for each pixel
in the view image, a ray is cast into the data, and the intensity
profile along the ray is fed to a ray function, which determines the
output value, such as the maximum, average, or minimum intensity, or
accumulated “opacity” (derived from intensity or gradient magnitude
information).

* By contrast, surface rendering methods require a segmentation of the
objects (usually obtained by thresholding), from which a surface
representation (triangulation) is derived, allowing very fast rendering
by graphics hardware.

* To reduce the effects of noise, Gaussian smoothing is often applied as
a pre-processing step prior to segmentation.

* As shown, both operations have a substantial influence on the final
result: by slightly changing the degree of smoothing or the threshold
level, objects may appear (dis)connected while in fact they are
not.

* Therefore it is recommended to establish optimal parameter values for
both steps while inspecting the effects on the original image data
rather than looking directly at the renderings.

https://en.wikipedia.org/wiki/Feature_detection_(computer_vision)

https://en.wikipedia.org/wiki/Edge_detection

Canny:

https://en.wikipedia.org/wiki/Canny_edge_detection

Example problem:

https://en.wikipedia.org/wiki/Sobel_operator

How to detect edges?

Edge detection: zoom in 1 row

Of one row:

intensity,

derivative, and

smoothed derivative

Do this with rows, then cols, and you get edges

**Sobel does x-derivative approx and x-derivative
approx**

(Convolution kernels above)

**+++++++++++ Cahoot-20-4**

https://umsystem.instructure.com/courses/97521/quizzes/298144

http://scipy-lectures.org/advanced/image_processing/index.html#edge-detection

(sobel)

Feature histogram

HOGles

* Middle row displays roughly what the computer with HOG might
see.

* Top row is HOG.

* Doing HOG first helps to scan/detect objects (a form of feature
extraction)

What is segmentation on biological images?

How do you label all of the cell or nucleus data in these images with
the same algorithm?

Below are images from your next assignment:

../../../../index/Classes/Bioinformatics/Content.html

(watch video in class)

https://en.wikipedia.org/wiki/Otsu’s_method

In Otsu’s method we exhaustively search for the threshold that minimizes
the intra-class variance (the variance within the class), defined as a
weighted sum of variances of the two classes

The algorithm assumes that the image contains two classes of pixels
following bi-modal histogram (foreground pixels and background pixels),
it then calculates the optimum threshold separating the two classes so
that their combined spread (intra-class variance) is minimal, or
equivalently (because the sum of pairwise squared distances is
constant), so that their inter-class variance is maximal.

https://en.wikipedia.org/wiki/Balanced_histogram_thresholding

This method weighs the histogram, checks which of the two sides is
heavier, and removes weight from the heavier side until it becomes the
lighter.

It repeats the same operation until the edges of the weighing scale
meet.

Recall k-means

1. Pick K cluster centers, either randomly or based on some heuristic
method, for example K-means++

2. Assign each pixel in the image to the cluster that minimizes the
distance between the pixel and the cluster center

3. Re-compute the cluster centers by averaging all of the pixels in the
cluster

4. Repeat steps 2 and 3 until convergence is attained (i.e. no pixels
change clusters)

Original image

k-means on that image’s X-features (pixel values)

* Recall/consider?

* What is distance?

* How were colors chosen?

- Clustering here sets boundaries for a groupwise thresholding.
- In this case, distance is the squared or absolute difference between a pixel and a cluster center.
- The difference is typically based on pixel color, intensity, texture, and location, or a weighted combination of these factors.

https://en.wikipedia.org/wiki/Edge_detection

* Edge detection is a well-developed field on its own within image
processing.

* Region boundaries and edges are closely related, since there is often
a sharp adjustment in intensity at the region boundaries.

* Edge detection techniques have therefore been used as the base of
another segmentation technique.

* Segmentation methods can also be applied to edges obtained from edge
detectors.

https://en.wikipedia.org/wiki/Image_segmentation

May be more out-of-date on some functions

*
http://scipy-lectures.org/intro/scipy.html#connected-components-and-measurements-on-images

*
http://scipy-lectures.org/advanced/image_processing/index.html#segmentation

* (based on threshold and more)

*
http://scipy-lectures.org/advanced/image_processing/index.html#measuring-objects-properties-ndimage-measurements

- https://docs.scipy.org/doc/scipy/reference/tutorial/ndimage.html#segmentation-and-labeling
- (no easy images for in-class)

- https://docs.scipy.org/doc/scipy/reference/tutorial/ndimage.html#object-measurements
- (no easy images for in-class)

May be more out-of-date on some functions

* http://scipy-lectures.org/packages/scikit-image/#image-segmentation
(based on threshold and more)

* For example:
http://scipy-lectures.org/packages/scikit-image/auto_examples/plot_labels.html#sphx-glr-packages-scikit-image-auto-examples-plot-labels-py
(in class review)

*
http://scipy-lectures.org/packages/scikit-image/#measuring-regions-properties
(for labeling)

*
http://scipy-lectures.org/packages/scikit-image/#data-visualization-and-interaction

- https://scikit-image.org/docs/stable/user_guide/tutorial_segmentation.html
- https://scikit-image.org/docs/stable/auto_examples/color_exposure/plot_regional_maxima.html#sphx-glr-auto-examples-color-exposure-plot-regional-maxima-py
- https://scikit-image.org/docs/stable/auto_examples/features_detection/plot_template.html#sphx-glr-auto-examples-features-detection-plot-template-py (template matching can be useful for consistent objects, but not all)
- https://scikit-image.org/docs/stable/auto_examples/features_detection/plot_blob.html#sphx-glr-auto-examples-features-detection-plot-blob-py
- https://scikit-image.org/docs/stable/auto_examples/segmentation/plot_peak_local_max.html#sphx-glr-auto-examples-segmentation-plot-peak-local-max-py
- https://scikit-image.org/docs/stable/auto_examples/segmentation/plot_label.html#sphx-glr-auto-examples-segmentation-plot-label-py (simple, good in class)
- https://scikit-image.org/docs/stable/auto_examples/segmentation/plot_random_walker_segmentation.html#sphx-glr-auto-examples-segmentation-plot-random-walker-segmentation-py
- https://scikit-image.org/docs/stable/auto_examples/segmentation/plot_watershed.html#sphx-glr-auto-examples-segmentation-plot-watershed-py
- https://scikit-image.org/docs/stable/auto_examples/applications/plot_coins_segmentation.html#sphx-glr-auto-examples-applications-plot-coins-segmentation-py
- (good comparison of methods, overview; do in Spyder)

https://en.wikipedia.org/wiki/Jaccard_index

Recall the rand index we covered previously!

**Jaccard index**, also known as **Intersection
over Union (IOU),**

and the **Jaccard similarity coefficient**

(originally given the French name coefficient de communauté by Paul
Jaccard),

is a statistic used for gauging the similarity and diversity of sample
sets.

The **Jaccard coefficient** measures similarity between
finite sample sets,

and is defined as the size of the intersection,

divided by the size of the union of the sample sets:

$J(A,B)={{|AB|} }={{|AB|} } $

If A and B are both empty,

we define J(A,B) = 1

\(0\leq J(A,B)\leq 1\)

Union —————————— Intersection

https://kaggle.com/competitions/data-science-bowl-2018/stage1/teaching-notebook-for-total-imaging-newbies.py

https://kaggle.com/competitions/data-science-bowl-2018/stage1/basic-skimage-nuclei.py