Statistical Shape and Deformation Analysis: Methods, Implementation and Applications

By Guoyan Zheng, Shuo Li and Gabor Szekely

Statistical Shape and Deformation Analysis: Methods, Implementation and Applications contributes enormously to solving different problems in patient care and physical anthropology, ranging from improved automatic registration and segmentation in medical image computing to the study of genetics, evolution and comparative form in physical anthropology and biology.

This book gives a clear description of the concepts, methods, algorithms and techniques developed over the last three decades that is followed by examples of their implementation using open source software.

Applications of statistical shape and deformation analysis are given for a wide variety of fields, including biometry, anthropology, medical image analysis and clinical practice.

• Presents an accessible introduction to the basic concepts, methods, algorithms and techniques in statistical shape and deformation analysis
• Includes implementation examples using open source software
• Covers real-life applications of statistical shape and deformation analysis methods

• Language: English
• Publisher: Academic Press
• Release date: Mar 23, 2017
• ISBN: 9780128104941

Guoyan Zheng

Prof. Zheng is the Head of the Information Processing in Medical Imaging Group, the Institute for Surgical Technology and Biomechanics, University of Bern. In 2010, he did his habilitation was awarded the title ‘Privatdozent’ from the same university. His research interests include medical image computing, machine learning, computer assisted interventions, medical robotics, and multi-modality image analysis. He has published over 160 peer-reviewed journal and conference papers and was granted 6 US and European patents. He has won over ten national and international awards/prizes including the best basic science paper published in the Journal of Laryngology and Otology in year 2011, the 2009 Ypsomed Innovation Prize, and the best technical paper award in the 2006 annual conference of the International Society of Computer Assisted Orthopaedic Surgery. He is on the program committee of the 16th, the 18th and the 19th International Conference on Medical Image Computing and Computer Assisted Interventions (MICCAI 2013, 2015 and 2016). He is the general chair of the 7th International Workshop on Medical Imaging and Augmented Reality (MIAR 2016).

Book preview

outstanding.

Part I

Basic Concepts, Methods and Algorithms

Outline

Chapter 1. Automated Image Interpretation Using Statistical Shape Models

Chapter 2. Statistical Deformation Model: Theory and Methods

Chapter 3. Correspondence Establishment in Statistical Shape Modeling: Optimization and Evaluation

Chapter 4. Landmark-Based Statistical Shape Representations

Chapter 5. Probabilistic Morphable Models

Chapter 6. Object Statistics on Curved Manifolds

Chapter 7. Shape Modeling Using Gaussian Process Morphable Models

Chapter 8. Bayesian Statistics in Computational Anatomy

Chapter 1

Automated Image Interpretation Using Statistical Shape Models

Claudia Lindner    The University of Manchester, Centre for Imaging Sciences, Manchester, United Kingdom

Abstract

Image interpretation covers a wide range of techniques to extract meaningful information from raw image data. Statistical Shape Models (SSMs) provide a means to describe the variation in shape of an object class across a set of images, allowing the qualitative and quantitative analysis of image data. For the detailed analysis of the shape of an object using SSMs, the object of interest needs to be annotated using many points in every image. Depending on the object type (e.g. faces or skeletal structures), this annotation may not only be tedious and time-consuming but may also require significant expertise to correctly place the points. In this chapter, we are going to introduce the basic methodology behind SSMs, and describe a number of automated annotation methods to enable fast and objective shape analyses with a wide variety of applications.

Keywords

Image analysis; Landmark detection; Segmentation; Shape model matching; Shape analysis; Regression forests; Statistical Shape Models; Active Shape Models; Active Appearance Models; Constrained Local Models

Chapter Outline

Acknowledgements

1.1  Introduction

1.2  Statistical Shape Analysis

1.2.1  Defining Shape

1.2.2  Statistical Shape Models

1.2.3  Automated Shape Annotation

1.3  Feature Point Detection Using Shape Model Matching

1.3.1  Active Shape Models

1.3.2  Active Appearance Models

1.3.3  Constrained Local Models

1.3.4  Random Forest Regression-Voting Constrained Local Models

1.4  Fully Automated Image Analysis via Shape Model Matching

1.4.1  Object Detection Methods

1.4.2  Combining Object Detection with Shape Model Matching

1.5  Automated Image Interpretation and Its Applications

1.6  Limitations of Statistical Shape Models for Image Interpretation

1.7  Conclusion

References

Acknowledgements

The author is grateful to Prof. Tim Cootes for fruitful discussions, to Dr. Paul Bromiley for supplying the data for Fig. 1.11, and to Jessie Thomson for helpful comments on the chapter. The author acknowledges funding from the Engineering and Physical Sciences Research Council, UK (EP/M012611/1).

1.1 Introduction

Image interpretation covers a wide range of techniques to extract meaningful information from raw image data. Statistical Shape Models (SSMs) provide a means to describe the variation in shape of an object class across a set of images, allowing the qualitative and quantitative analysis of image data. For the detailed analysis of the shape of an object using SSMs, the object of interest needs to be annotated using many points in every image. Depending on the object type (e.g. faces or skeletal structures), this annotation may not only be tedious and time-consuming but may also require significant expertise to correctly place the points. In this chapter, we are going to introduce the basic methodology behind SSMs, and describe a number of automated annotation methods to enable fast and objective shape analyses with a wide variety of applications.

1.2 Statistical Shape Analysis

An object in an image can be characterized by both shape and texture. While shape provides geometric information about the object, texture gives its intensity appearance. Hence, shape-based image interpretation utilizes the contour of an object while texture-based image interpretation utilizes its gray-level (or color-level) variation. Within an object class (e.g. hands, faces) the shape and texture may vary widely across images. For example with reference to medical images, these differences may be related to one or more clinical variables such as age and gender, anatomical variation between individuals, or disease progression. Furthermore, differences in the shape and texture of an object may also result from the usage of different imaging techniques or image acquisition protocols.

A detailed introduction to statistical shape analysis can be found in [30]. Below, we provide a summary of the main aspects of statistical shape analysis as deemed relevant to the remainder of this chapter.

1.2.1 Defining Shape

Traditionally, morphometric analyses were based on a set of predefined measurements such as lengths and angles. Though predefined measurements are often easily obtainable, they do not capture the overall shape of the object and predominantly measure size rather than shape.

If we are not interested in the location, orientation or scale …, then we find ourselves working with … change of shape. [44, p. 428]

According to Kendall [44], shape describes all geometric information of an object disregarding location, orientation and scale of the data. Modern morphometrics, thus, suggests to analyze the contour of an object so as to capture its overall shape. This is commonly done by placing a number of landmark points along the object's contour. A landmark¹ is considered to be a point of correspondence that marks a specific part of the contour, or structure, of an object across images of the same object class. The shape of the object is then expressed by the combination of all landmark points. When identifying landmarks, the following criteria should be taken into account [74]. Landmarks should:

• provide adequate coverage of the morphology of the object;

• be chosen so as to quantify any significant shape change;

• be placed such that they can be found repeatedly and reliably;

• mark consistent positions relative to other landmarks.

In studies involving images of organisms, landmarks are often manually chosen to mark points of anatomical significance (e.g. finger tips in images of the hands) in combination with evenly spaced points along the object's contour. When landmarks are evenly spaced it is not always obvious how to select the (number of) points required such that the variation in shape is well represented and that all points are in correspondence across the object class. The latter applies to both object classes with and without anatomical landmarks or key points of interest. Furthermore, this holds true for shape surfaces in 3D where manual identification of suitable landmarks is often difficult and impractical. Given a representative set of objects for an object class of interest, a number of automated methods are available to establish correspondence in these cases; see [26] for an overview. A technique referred to as Minimum Description Length (MDL) has emerged as the state-of-the-art in this field [25]. MDL methods establish point correspondences automatically by minimizing the description length of a shape model created from the point placements, aiming for models with high generalization ability, specificity and compactness. However, automated methods to establish point correspondences often require the shape to be readily defined as a curve i.e. the shape of the object of interest to be outlined in every image – which is not always available.

A set of landmark points placed at key features of an object provides a description of the geometry of the object and allows the detailed statistical analysis of the global shape of the object. Though various statistical analyses could be used for this purpose, Principal Component Analysis (PCA) is commonly applied in this context.

1.2.2 Statistical Shape Models

Statistical Shape Models (SSMs)² are commonly used to study the morphometrics of a deformable object. They describe the shape of the object by applying PCA to a set of landmark points, and are based on the assumption that each shape is a deformed version of a reference shape. SSMs aim to establish the shape variation of an object class in order to create a statistical model that gives a parameterized representation of this variation in shape.

Building an SSM is based on a set of landmark points capturing the shape of the object in every image. The types of shape deformation present across the data are identified and redundancies in the shape distribution of the object class are removed. SSMs yield a compact representation of the shape of the object class, and can be used in a variety of application areas. They are not only useful for the analysis of shape differences in the data, on which the model was built, but also to analyze the shape of new data and to synthesize new shapes that are similar to the original data.

A prerequisite for building an SSM is a set of annotated images. The dataset that the model is built on is commonly referred to as the training dataset. Given a set of n training images, for each image the shape of the object of interest is assumed to be described by ℓ landmark points placed at key features and/or along the contour of the object. Note that the landmark points may be in any dimension but in this chapter we focus on the 2D case.³ The landmark points of any image I . For boundary landmarks (i.e. landmark points along the contour of the object), all landmarks are given in the order that they are connected with each other (if applicable).

A detailed description of SSMs can be found in Cootes et al. [16] and Davies et al. [26]. The following provides an overview of the ideas and algorithms behind building SSMs.

Shape Alignment via Generalized Procrustes Analysis

with parameters θ that with x . Best . The alignment process is conducted in an iterative manner, where the shape of each training image is repeatedly aligned with the respective mean over all aligned shapes until the mean shape converges. To initialize the alignment, the mean shape is set to the landmark points of the first shape in the training set.

After all shapes have been aligned, PCA is applied to build the SSM.

Modeling Shape Variation via Principal Component Analysis

PCA is a dimensionality reduction method that can be used to represent the shape variation of n is calculated over all training shapes. This is then used to analyze how the elements of the different shape vectors vary together by calculating the covariance matrix over all training data

(1.1)

Covariance matrix C provides the basis to compute the principal components, its 2ℓ containing the 2ℓ orthogonal shape eigenvectors of the covariance matrix C in its columns as well as a 2ℓ. The landmarks of any aligned shape can then be defined by the mean shape vector plus a linear combination of eigenvectors of C

(1.2)

Matrix P, shape mode vector b define an SSM where b , and the sum of all eigenvalues gives the overall variance of the training data. Eigenvectors with larger eigenvalues describe more significant shape variations whereas those with lower eigenvalues express smaller, often more local, variations. In general, most of the overall variation can be explained by a subset of the 2ℓ eigenvectors and eigenvalues. To reduce dimensionality while still preserving a sufficient large proportion of the overall shape variance of the training set, it is usually satisfactory to only consider the t . An alternative approach to defining t, the number of shape modes to be included, is to choose enough modes so that the model can approximate any training shape to within a given accuracy (e.g. within one pixel). Based on the above, every aligned shape vector x can be approximated by

(1.3)

containing the t eigenvectors corresponding to the t being a t-dimensional vector representing the t shape mode parameters. To accurately describe shape vector x after dimensionality reduction, a residual term needs to be introduced such that

(1.4)

where residuals r can be used to estimate how well the SSM fits to a dataset. The larger the residuals the more the model deviates from the data.

The shape represented by Eq. to be Gaussian in general, it was later recommended to use an upper bound on the Mahalanobis distance instead [16]. For a new with C belongs to the distribution of shape variation given by the training set.

, point in these directions and provide the axes of the new coordinate system where every vector x . Similarly, a few parameters (or shape modes) may be sufficient to represent high dimensional data as, for example, given by a dataset that uses many landmarks to capture the contour of an object.

Figure 1.1 , where p 1 and p 2 are the two principle axes.

Fig. 1.2 shows the first three shape modes of a proximal femur shape model that was trained on 2516 female AP pelvic radiographs from the Osteoarthritis Initiative dataset (https://oai.epi-ucsf.org/). These three modes together explain 70% of the variance in proximal femur morphology across all radiographs analyzed. This figure demonstrates that an SSM mode does not simply reflect one morphometric measurement such as the femoral neck width but rather represents a combination of several related differences between proximal femurs of the study population.

Figure 1.2 First three shape modes of variation in proximal femur morphology and the effect of varying the PCA parameters between ±2.5 SD for an SSM trained on 2516 female AP pelvic radiographs from the Osteoarthritis Initiative dataset ( https://oai.epi-ucsf.org/ ). Each figure shows the average (solid line) and ±2.5 SD.

needs to be applied

(1.5)

that best match the model to the image (i.e. such that residuals r are small) need to be identified.

1.2.3 Automated Shape Annotation

SSMs provide a method to quantify the shape of an object using only a limited number of parameters. Applying SSMs in the setting of morphometric analyses (e.g. of skeletal structures in medical images) has the advantage of capturing the global shape of the object of interest rather than reducing it to a set of fixed geometric measurements such as lengths and angles.

Annotating images manually is prone to inconsistencies, time-consuming and subjective. This poses a problem in particular for application areas with large datasets and high accuracy requirements. Over the years, several methods have been developed to automate the annotation process. These methods broadly fall into two categories – feature point detection and groupwise image registration. Feature point detection aims to directly predict the position of every landmark in every image, and is often combined with some kind of shape model to regularize the output of the detection. Groupwise image registration, in contrast, aims to bring a set of related images into spatial alignment which can then be used to find sets of corresponding points across images using a single annotated reference image. Furthermore, several hybrid methods combining feature point detection and image registration have been presented (see e.g. Zhang and Brady [76] or Guo et al. [41]). An alternative approach to generating an SSM, without the need for annotating new images, would be to directly predict the parameters of the shape model. For more details, see for example the literature on Marginal Space Learning [77] or Shape Particle Filtering [27].

Groupwise image registration requires all images to be similar in terms of the position, orientation and scale of the object of interest in the image (e.g. as in MRI or CT datasets). If this is not the case, a semi-automatic registration approach can be followed by guiding the registration with a sparse set of manually annotated points in every image. For example, the three images in Fig. 1.3 are not well aligned if the object of interest was the house. However, if one was to manually annotate two corners of the house (crosses) then these could be used to initialize and guide the registration. Depending on the application area and the object of interest, the number of points required to guide the registration may significantly increase. More details on groupwise image registration can be found elsewhere (see e.g. Cootes et al. [17] and Sotiras et al. [66]). In this chapter, the focus is on feature point detection methods to generate the annotations as these provide the opportunity to fully automate the annotation process – even if the images are not aligned and vary significantly in the position, orientation and scale of the object of interest.

Figure 1.3 Example dataset where the object of interest, the house, is not sufficiently aligned across images for groupwise image registration to be applied without some form of initialization.

1.3 Feature Point Detection Using Shape Model Matching

For an automated annotation system to be able to reliably deal with a range of intra-class variation and to provide robust results, it needs to be both general and specific. To accomplish this, it is inevitable for the system to incorporate prior knowledge about the expected shape of the object and/or its texture. Prior knowledge is particularly useful in cases of outliers (e.g. due to occlusions) and noise as well as when there is significant variation in shape and/or texture across images. Most recent feature point detection methods include prior knowledge via machine learning approaches. While this allows the annotation system to learn about the expected shape and/or texture variation, it also means that these methods will only perform well if the shape/texture variation in the training data is representative of the variation in the given object class.⁴

Texture-based feature point detection techniques focus on identifying the best position for every landmark by analyzing the texture in a region of interest of a given image (see e.g. Reinders et al. [61], Feris et al. [32] or Patil et al. [60]). Shape-based approaches, in contrast, aim to identify the shape that best matches the search image. Hence, in the latter a single point is not necessarily placed at its best position based on the texture in the region around the landmark point but at its best position given the overall shape in the image. Shape-based methods will only identify point positions that provide an overall valid shape example of the object class, which can be a big advantage in some application areas. A variety of shape models can be used in this context, ranging from simple geometric relationships between sets of points [5,58] over graphical representations of shape [28,78] to SSMs. We refer to methods that regularize the output of a feature point candidate detector using an SSM as shape model matching methods. Of note is that shape model matching methods often have a local texture-based feature point detection step, and hence tend to combine shape with texture information.

In the following, we describe a range of shape model matching methods.

1.3.1 Active Shape Models

Active Shape Models (ASMs) initialized to zero) at the estimated position, orientation and scale, a model instance x is created and the following model matching steps are applied:

1. Each landmark point in x is optimized by moving it to the locally best ;

ensure that the optimized model instance is consistent with the shape variation learned from the training images. Model instance x is re-calculated;

3. Steps 1 and 2 are repeated until the shape model instance converges to the shape in the search image.

The very first ASM algorithms used edge detection methods based on the profile normal to the shape model contour to identify the locally best position for each landmark (see Fig. 1.4). This required all landmarks to lie on the locally strongest edge. However, depending on the dataset, the locally strongest edge may not be the edge representing the contour of the object of interest. Therefore, later ASM versions improved upon this by using statistical models of the gray-value structure along the profile normal to the SSM model contour.

Figure 1.4 Applying edge detection to identify the best landmark point position.

Local Gray-Value Models ASMs that use local gray-value models learn the nearby structure of every landmark point and hence do not require landmarks to lie on the locally strongest edge but only on locally distinctive structures [15].

Model Building For every landmark point kand every training image inormal to the shape model contour is sampled as illustrated in Fig. 1.5. Either side of the landmark, s . For every landmark point k. The probability of the corresponding landmark point in the search image to fit to this distribution is then related to the squared Mahalanobis distance between the normalized profile vector at that point and the mean of the landmark profile model. Minimizing the latter optimizes the quality-of-fit of the landmark point position to the learned profile model for that landmark.

Figure 1.5 Modeling local gray-value profiles to identify the best landmark point position.

Model Matching To find the best fit to a search image based on an initial estimate, for every landmark point k, centered at the estimated landmark point position is sampled (, is calculated. Landmark point k are updated to best match these positions. As with the strongest edge approach, this procedure is repeated iteratively until the shape model instance converges to the shape in the search image.

To further increase the robustness of ASMs, a multi-resolution framework following a Gaussian pyramid approach can be used. This requires several ASMs to be trained, one for each stage of the framework. During shape model matching this approach involves first a search for the point locations in a coarse, low resolution version of the image and then a series of refinement searches at stages of increasing resolutions. Keeping the number of pixels constant across stages allows the coarse model to take a larger area of the image into account, guiding the model towards the correct position. For more details on how to efficiently match an SSM to a search image using ASMs the reader is referred to [12,14,16].

Despite these improvements to increase the robustness of ASMs, they are very sensitive to the landmark positions used for initialization. This limits their application areas. Particularly in medical imaging, anatomical landmarks may not always be placed on an edge while there may be edges of non-relevant structures close by. Furthermore, the patterns of intensities around anatomical landmarks are not always sufficiently distinctive across subjects. Therefore, ASMs are likely to drift off or get stuck if not initialized appropriately (see Fig. 1.6). Over the years, several extensions to the traditional ASM algorithm have been suggested aiming to overcome these shortcomings [23,50,52,56,68,75].

Figure 1.6 Two examples of where an insufficient initialization in position, orientation or scale causes a gray-level model based ASM to get stuck in a local optimum; showing the initialization of the model as well as the result after five search iterations (additional iterations did not improve the performance).

1.3.2 Active Appearance Models

Statistical Appearance Models (SAMs) combine SSMs as described above with a model of texture variation retrieved from a shape-free image patch. Active Appearance Models (AAMs) have been proposed as a shape model matching method that uses a joint shape and texture SAM for feature point detection [13].

Texture Models

.

In addition to texture variations caused by shape change, there may also be texture variations that are due to other reason such as global lighting variations. To account for this, the texture vector in the image coordinate frame needs to be normalized. Projections between image coordinate frame and reference coordinate frame, i.e. normalization and de-normalization, are transformation-based.

that is used to project the object from the model coordinate frame to the image coordinate frame can be defined by

(1.6)

can be normalized according to

(1.7)

where α represents a scaling factor, β gives the mean pixel value as an offset factor, 1 is a vector of ones, and k is the number of elements of gand 1. The latter is recursively calculated and re-estimated using α and β as in Eq. (1.6), where one of the texture training vectors is chosen as the first estimate of the mean.

, PCA can then be applied to generate a shape-free texture model

(1.8)

defines the variation of the gray-value patterns. The texture model described in Eq. can be applied

(1.9)

As for SSMs, dimensionality reduction can be achieved by only including the first t texture modes that capture a certain proportion of the overall texture variation.

Statistical Appearance Models

can be defined by

(1.10)

defines an SAM

(1.11)

is a vector of appearance coefficients describing both the shape and texture of the object. Compared to the shape and texture models defined above, SAMs do not include a mean appearance vector as both the shape and the texture coefficients have zero mean by construction. The shape and texture of an SAM can also be defined directly via

(1.12)

Active Appearance Models

Active Appearance Models (AAMs) describe an optimization problem to minimize the difference between an appearance model instance and the object of interest in an image. Given an SAM as described above, the aim is to adjust the model parameters such that the appearance model instance matches the target object as closely as possible.

In contrast to ASMs, instead of optimizing the position of individual landmarks, AAMs aim to match the entire object simultaneously. An AAM is matched to an image by minimizing the residual between the object's texture in the image and that synthesized by the SAM. As defined in Eq. in the search image (i.e. an estimate of transformation parameters θ to , where ψ is the vector of transformation parameters α and βand ψ are learned from the training set. Initial values for θ can be learned from the training set or obtained from a manual initialization of the pose of the object in the image.

. Both the image and the synthesized texture vectors are given in the reference coordinate frame and the difference can be easily determined by

(1.13)

with p as well as transformation parameters θ (translation, rotation and scale) and ψ (scale and offset). To minimize the difference given by vector r, parameters p need to be incrementally adjusted. Without loss of generality, it is assumed that there is an approximately linear relationship between changes in the parameters and the difference between the image patch and the synthesized model patch.

The difference vector r is iteratively optimized by minimizing its sum of