Hostname: page-component-586b7cd67f-2plfb Total loading time: 0 Render date: 2024-11-27T20:56:16.314Z Has data issue: false hasContentIssue false
  • English
  • Français

Quantifying Variability of Manual Annotation in Cryo-Electron Tomograms

Published online by Cambridge University Press:  26 May 2016

Corey W. Hecksel ,
Michele C. Darrow ,
Wei Dai ,
Jesús G. Galaz-Montoya ,
Jessica A. Chin ,
Patrick G. Mitchell ,
Shurui Chen ,
Jemba Jakana ,
Michael F. Schmid  and
Wah Chiu
Corey W. Hecksel
Affiliation:
Molecular Virology and Microbiology Department, Baylor College of Medicine, Houston, TX 77030, USA National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Michele C. Darrow
Affiliation:
Verna and Marrs McLean Department of Biochemistry and Molecular Biology, Baylor College of Medicine, Houston, TX 77030, USA National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Wei Dai
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Jesús G. Galaz-Montoya
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Jessica A. Chin
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Patrick G. Mitchell
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Shurui Chen
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Jemba Jakana
Affiliation:
National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Michael F. Schmid
Affiliation:
Verna and Marrs McLean Department of Biochemistry and Molecular Biology, Baylor College of Medicine, Houston, TX 77030, USA National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
Wah Chiu*
Affiliation:
Molecular Virology and Microbiology Department, Baylor College of Medicine, Houston, TX 77030, USA Verna and Marrs McLean Department of Biochemistry and Molecular Biology, Baylor College of Medicine, Houston, TX 77030, USA National Center for Macromolecular Imaging, Baylor College of Medicine, Houston, TX 77030, USA
*
*Corresponding author. [email protected]
Get access

Abstract

Although acknowledged to be variable and subjective, manual annotation of cryo-electron tomography data is commonly used to answer structural questions and to create a “ground truth” for evaluation of automated segmentation algorithms. Validation of such annotation is lacking, but is critical for understanding the reproducibility of manual annotations. Here, we used voxel-based similarity scores for a variety of specimens, ranging in complexity and segmented by several annotators, to quantify the variation among their annotations. In addition, we have identified procedures for merging annotations to reduce variability, thereby increasing the reliability of manual annotation. Based on our analyses, we find that it is necessary to combine multiple manual annotations to increase the confidence level for answering structural questions. We also make recommendations to guide algorithm development for automated annotation of features of interest.

Keywords

cryo-electron tomographysegmentationannotationvalidationDice coefficient
Type
Technique and Instrumentation Development
Information
Microscopy and Microanalysis , Volume 22 , Issue 3 , June 2016 , pp. 487 - 496
Copyright
Copyright © Microscopy Society of America 2016

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

†

Current address: Diamond Light Source Ltd, Science Division, Fermi Ave, Didcot, Oxfordshire OX11 0DX, UK.

‡

Current address: Department of Cell Biology and Neuroscience, Center for Integrative Proteomics Research, Rutgers University, 174 Frelinghuysen Road, Piscataway, NJ 08854-8076, USA.

a

Corey W. Hecksel and Michele C. Darrow contributed equally to this work.

References

Asano, S., Fukuda, Y., Beck, F., Aufderheide, A., Förster, F., Danev, R. & Baumeister, W. (2015). Proteasomes. A molecular census of 26S proteasomes in intact neurons. Science 347, 439–442.Google Scholar
Dai, W., Fu, C., Raytcheva, D., Flanagan, J., Khant, H.A., Liu, X., Rochat, R.H., Haase-Pettingell, C., Piret, J., Ludtke, S.J., Nagayama, K., Schmid, M.F., King, J.A. & Chiu, W. (2013). Visualizing virus assembly intermediates inside marine cyanobacteria. Nature 502, 707–710.Google Scholar
Darrow, M.C., Sergeeva, O.A., Isas, J.M., Galaz-Montoya, J., King, J.A., Langen, R., Schmid, M.F. & Chiu, W. (2015). Structural mechanisms of mutant huntingtin aggregation suppression by synthetic chaperonin-like CCT5 complex explained by cryo-electron tomography. J Biol Chem 290, 17451–17461.Google Scholar
Frangakis, A.S. & Förster, F. (2004). Computational exploration of structural information from cryo-electron tomograms. Curr Opin Struct Biol 14, 325–331.Google Scholar
Frangakis, A.S. & Hegerl, R. (2001). Noise reduction in electron tomographic reconstructions using nonlinear anisotropic diffusion. J Struct Biol 135, 239–250.Google Scholar
Garduño, E., Wong-Barnum, M., Volkmann, N. & Ellisman, M.H. (2008). Segmentation of electron tomographic data sets using fuzzy set theory principles. J Struct Biol 162, 368–379.Google Scholar
Gilliam, J.C., Chang, J.T., Sandoval, I.M., Zhang, Y., Li, T., Pittler, S.J., Chiu, W. & Wensel, T.G. (2012). Three-dimensional architecture of the rod sensory cilium and its disruption in retinal neurodegeneration. Cell 151, 1029–1041.Google Scholar
GraphPad QuickCalcs (2015). GraphPad QuickCalcs: T test calculator. Available at http://www.graphpad.com/quickcalcs/ttest1.cfm (retrieved July 6, 2015).Google Scholar
Hoppe, W. (1974 a). Three-demensional electron microscopic reconstruction of an object. Naturwissenschaften 61, 534–536.Google Scholar
Hoppe, W. (1974 b). Towards three-dimensional “electron microscopy” at atomic resolution. Naturwissenschaften 61, 239–249.Google Scholar
Ibiricu, I., Huiskonen, J.T., Döhner, K., Bradke, F., Sodeik, B. & Grünewald, K. (2011). Cryo electron tomography of herpes simplex virus during axonal transport and secondary envelopment in primary neurons. PLoS Pathog 7, e1002406.Google Scholar
Koning, R.I., Zovko, S., Bárcena, M., Oostergetel, G.T., Koerten, H.K., Galjart, N., Koster, A.J. & Mieke Mommaas, A. (2008). Cryo electron tomography of vitrified fibroblasts: Microtubule plus ends in situ. J Struct Biol 161, 459–468.Google Scholar
Kremer, J.R., Mastronarde, D.N. & McIntosh, J.R. (1996). Computer visualization of three-dimensional image data using IMOD. J Struct Biol 116, 71–76.Google Scholar
Lepault, J., Booy, F.P. & Dubochet, J. (1983). Electron microscopy of frozen biological suspensions. J Microsc 129, 89–102.Google Scholar
Lučić, V., Förster, F. & Baumeister, W. (2005). Structural studies by electron tomography: From cells to molecules. Annu Rev Biochem 74, 833–865.Google Scholar
Lučić, V., Rigort, A. & Baumeister, W. (2013). Cryo-electron tomography: The challenge of doing structural biology in situ. J Cell Biol 202, 407–419.Google Scholar
Maimon, T., Elad, N., Dahan, I. & Medalia, O. (2012). The human nuclear pore complex as revealed by cryo-electron tomography. Structure 20, 998–1006.Google Scholar
Martinez-Sanchez, A., Garcia, I. & Fernandez, J.-J. (2011). A differential structure approach to membrane segmentation in electron tomography. J Struct Biol 175, 372–383.Google Scholar
Martinez-Sanchez, A., Garcia, I. & Fernandez, J.-J. (2013). A ridge-based framework for segmentation of 3D electron microscopy datasets. J Struct Biol 181, 61–70.Google Scholar
Maurer, U.E., Sodeik, B. & Grünewald, K. (2008). Native 3D intermediates of membrane fusion in herpes simplex virus 1 entry. Proc Natl Acad Sci U S A 105, 10559–10564.Google Scholar
Moussavi, F., Heitz, G., Amat, F., Comolli, L.R., Koller, D. & Horowitz, M. (2010). 3D segmentation of cell boundaries from whole cell cryogenic electron tomography volumes. J Struct Biol 170, 134–145.Google Scholar
Nguyen, H. & Ji, Q. (2008). Shape-driven three-dimensional watersnake segmentation of biological membranes in electron tomography. IEEE Trans Med Imaging 27, 616–628.Google Scholar
Page, C., Hanein, D. & Volkmann, N. (2015). Accurate membrane tracing in three-dimensional reconstructions from electron cryotomography data. Ultramicroscopy 155, 20–26.Google Scholar
Patla, I., Volberg, T., Elad, N., Hirschfeld-Warneken, V., Grashoff, C., Fässler, R., Spatz, J.P., Geiger, B. & Medalia, O. (2010). Dissecting the molecular architecture of integrin adhesion sites by cryo-electron tomography. Nat Cell Biol 12, 909–915.Google Scholar
Rigort, A., Günther, D., Hegerl, R., Baum, D., Weber, B., Prohaska, S., Medalia, O., Baumeister, W. & Hege, H.-C. (2012). Automated segmentation of electron tomograms for a quantitative description of actin filament networks. J Struct Biol 177, 135–144.Google Scholar
Rusu, M., Starosolski, Z., Wahle, M., Rigort, A. & Wriggers, W. (2012). Automated tracing of filaments in 3D electron tomography reconstructions using Sculptor and Situs. J Struct Biol 178, 121–128.Google Scholar
Sandberg, K. (2007). Methods for image segmentation in cellular tomography. Methods Cell Biol 79, 769–798.Google Scholar
Shahmoradian, S.H., Galaz-Montoya, J.G., Schmid, M.F., Cong, Y., Ma, B., Spiess, C., Frydman, J., Ludtke, S.J. & Chiu, W. (2013). TRiC’s tricks inhibit huntingtin aggregation. eLife 2, e00710.Google Scholar
Shahmoradian, S.H., Galiano, M.R., Wu, C., Chen, S., Rasband, M.N., Mobley, W.C. & Chiu, W. (2014). Preparation of primary neurons for visualizing neurites in a frozen-hydrated state using cryo-electron tomography. J Vis Exp 84, e50783.Google Scholar
Tang, G., Peng, L., Baldwin, P.R., Mann, D.S., Jiang, W., Rees, I. & Ludtke, S.J. (2007). EMAN2: An extensible image processing suite for electron microscopy. J Struct Biol 157, 38–46.Google Scholar
Taylor, K.A. & Glaeser, R.M. (1974). Electron diffraction of frozen, hydrated protein crystals. Science 186, 1036–1037.Google Scholar
Tsai, W.-T., Hassan, A., Sarkar, P., Correa, J., Metlagel, Z., Jorgens, D.M. & Auer, M. (2014). From voxels to knowledge: A practical guide to the segmentation of complex electron microscopy 3D-data. J Vis Exp 90, e51673.Google Scholar
van der Heide, P., Xu, X.-P., Marsh, B.J., Hanein, D. & Volkmann, N. (2007). Efficient automatic noise reduction of electron tomographic reconstructions based on iterative median filtering. J Struct Biol 158, 196–204.Google Scholar
Volkmann, N. (2002). A novel three-dimensional variant of the watershed transform for segmentation of electron density maps. J Struct Biol 138, 123–129.Google Scholar
Volkmann, N. (2010). Methods for segmentation and interpretation of electron tomographic reconstructions. In Methods in Enzymology. Volume 483, Cryo-EM, Part C: Analyses, Interpretation, and Case studies (chapter 2), G.J. Jensen (Ed.), pp. 31–46. Cambridge: Academic Press. Available at http://www.sciencedirect.com/science/article/pii/S0076687910830022 (retrieved November 7, 2014).Google Scholar
Wang, R., Stone, R.L., Kaelber, J.T., Rochat, R.H., Nick, A.M., Vijayan, K.V., Afshar-Kharghan, V., Schmid, M.F., Dong, J.-F., Sood, A.K. & Chiu, W. (2015). Electron cryotomography reveals ultrastructure alterations in platelets from patients with ovarian cancer. Proc Natl Acad Sci U S A 112, 14266–14271.Google Scholar
Woodward, C.L., Cheng, S.N. & Jensen, G.J. (2014). Electron cryotomography studies of maturing HIV-1 particles reveal the assembly pathway of the viral core. J Virol 89, 1267–1277.Google Scholar
Zhao, X., Zhang, K., Boquoi, T., Hu, B., Motaleb, M.A., Miller, K.A., James, M.E., Charon, N.W., Manson, M.D., Norris, S.J., Li, C. & Liu, J. (2013). Cryoelectron tomography reveals the sequential assembly of bacterial flagella in Borrelia burgdorferi. Proc Natl Acad Sci U S America 110, 14390–14395.Google Scholar
Supplementary material: File

Hecksel supplementary material S1

Supplementary Table

Download Hecksel supplementary material S1(File)
File 103.6 KB
Supplementary material: File

Hecksel supplementary material S2

Supplementary Table

Download Hecksel supplementary material S2(File)
File 146.7 KB

Hecksel supplementary material S3

Supplementary Movie

Download Hecksel supplementary material S3(Video)
Video 26.2 MB

Hecksel supplementary material S4

Supplementary Movie

Download Hecksel supplementary material S4(Video)
Video 11.4 MB