Automatic Segmentation
Susumu Mori1

1Johns Hopkins School of Medicine, United States


In this presentation, the basic concept of anatomical segmentation is explained. This presentation first discusses about the importance of tissue segmentation for modern medical data analysis, in which difficult neurological conditions are often the target of the research. Then different types of segmentation approaches are explained. In the last section, an interesting paradox of the tissue segmentation, namely the lack of ground truth, is discussed in detail. This presentation should be informative for both method developers and users.


Anatomical segmentation of brain MR images is an old research topic; a simple PubMed search with “brain”, “MRI”, “segmentation”, “algorithm”, returns more than 1,800 publications. There are many tools publicly available to automatically define structures of interest. Yet, segmentation is still an extremely difficult technology and far from complete. This is partly due to technical difficulties in developing capable segmentation algorithms. However, what has not been well discussed are more fundamental issues like, “what is segmentation” to begin with and “why we need it.” In this educational session, we will go over some of these fundamental issues.

Why do we need segmentation?

With a 1 mm voxel size, a brain of more than 1L of volume would contain more than 1 million voxels. Suppose a patient brain has abnormality with the size of 10,000 voxels, it occupies less than 1% of the total voxels. This means, 99% of the observation units would bring in noise to the statistical analysis, which substantially weakens the statistical power. Data reduction is, therefore, often necessary, which is typically performed by infusing semantic labels (i.e. segmentation) and hypothesis about the disease of interest. For example, a brain image with millions of voxels could be converted to a two-element vector, [the volume of the hippocampus, the volume of the ventricle], if we hypothesize that these are the two most important structures. Of course, this hypothesis could be wrong and such reduction may throw away essential information encoded in the images. Nonetheless, segmentation could be an effective way for data reduction.

What types of segmentation approaches are available?

The single-atlas-based segmentation is conceptually the easiest to understand (Fig. 1). In this approach, we need to prepare one atlas in which structures are defined. Then this atlas can be transformed to the shape of a patient image. The transformation deforms the shape of the atlas into the shape of the new image, thereby carrying the anatomical boundaries defined in the atlas to the new image. The result should be accurate as long as the transformation is perfect. In reality, everybody’s brain is different and it is a difficult task to achieve a perfect transformation from one brain to the other. To accommodate the range of anatomical variability and transformation errors, some segmentation algorithms adopted probabilistic atlases, which were created from tens of pre-segmented atlases. After the information from multiple atlases are averaged in a standardized space, each voxel of the atlas contains labeling probability. In more modern approaches called multi-atlas methods, a probabilistic atlas is not created and the tens of atlases are directly registered to the subject image. Although this would require much more computation power and the management of the multiple atlases becomes more complex (the entire atlas libraries need to be distributed to each user and update becomes challenging), it provides much more flexible environment. For example, if the atlas library covers a wide range of ages, age-matched atlases can be used for optimum anatomical estimates.

How do we know that segmentation results are accurate?

One interesting fact is, there is no objective and quantitative way to accurately measure the accuracy. If such a metric is available to us, we would have already used it as a part of segmentation algorithms so that we can maximize the accuracy. In the field of computational anatomy, the final judgment of the accuracy is provided by manual delineation. There is a profound issue because the very notion of “anatomical structure” could be artificial. Fig. 2 shows two examples to make this point. The left panels are obtained from conventional in vivo MRI, in which these structures seem clearly discernible. The middle panels are from high-resolution MRI and the right panels are histology images. As the resolution becomes higher, these anatomical notions start to disappear and boundary definitions become obscure. This is because many anatomical structures are merely concepts of anatomy defined by human. As a matter of fact, past studies reported that as simple as hippocampus volumes could vary over 100% depending on the anatomical definitions they adopted [1]. These examples tell that the anatomical definitions are not based the ground truth, but by human-created rule books. The three-dimensional anatomical boundaries defined in atlases are exactly the rules adopted by the creators. So the accuracy should be discussed within the context of “how faithfully followed the rule”, but not by “how true they are.”


Segmentation-based image analysis is expected to be increasingly important in this era of BigData and precision medicine, in which each patient brings rich medical data and data reduction in each measurement domain is essential. As the segmentation technology is still evolving rapidly, it is advised to keep learning new tools and evaluate their performance carefully.


No acknowledgement found.


1: Geuze E, Vermetten E, Bremner JD (2005) MR-based in vivo hippocampal volumetrics: 1. Review of methodologies currently employed. Mol Psychiatry 10, 147-159.


Fig. 1: Three approaches for atlas-based brain segmentation; Single-atlas, probabilistic atlas, and multi-atlas approaches.

Fig. 2: Examples of structures which can be readily defined using in vivo MRI, but their entities do not have clear anatomical boundaries at cellular levels. (A): the caudate (red arrows) and the adjacent structure (putamen) are continuous structures (yellow arrows), which are penetrated by bundles of white matter tracts. (B): the corticospinal tract (red arrow) seems one homogeneous structure in low-resolution in vivo imaging but images in higher resolution reveals complex and heterogeneous architectures, in which the size and volume are difficult to quantify. Histology panels are copied from The Human Nervous System, Paxinos and Mai, Elsevier and The Human Brain, Nolte, Mosby with permissions.

Proc. Intl. Soc. Mag. Reson. Med. 26 (2018)