Hostname: page-component-78c5997874-j824f Total loading time: 0 Render date: 2024-11-15T05:20:16.469Z Has data issue: false hasContentIssue false
  • English
  • Français

Fast Improvement of TEM Images with Low-Dose Electrons by Deep Learning

Published online by Cambridge University Press:  10 December 2021

Hiroyasu Katsuno Open the ORCID record for Hiroyasu Katsuno [Opens in a new window] ,
Yuki Kimura Open the ORCID record for Yuki Kimura [Opens in a new window] ,
Tomoya Yamazaki Open the ORCID record for Tomoya Yamazaki [Opens in a new window]  and
Ichigaku Takigawa
Hiroyasu Katsuno*
Affiliation:
Institute of Low Temperature Science, Hokkaido University, Kita-19, Nishi-8, Kita-ku, Sapporo, Hokkaido 060-0819, Japan
Yuki Kimura
Affiliation:
Institute of Low Temperature Science, Hokkaido University, Kita-19, Nishi-8, Kita-ku, Sapporo, Hokkaido 060-0819, Japan
Tomoya Yamazaki
Affiliation:
Institute of Low Temperature Science, Hokkaido University, Kita-19, Nishi-8, Kita-ku, Sapporo, Hokkaido 060-0819, Japan
Ichigaku Takigawa
Affiliation:
RIKEN, Center for Advanced Intelligence Project, 1-4-1 Nihonbashi, Chuo-Ku, Tokyo 103-0027, Japan Institute for Chemical Reaction Design and Discovery (WPI-ICReDD), Hokkaido University, N21 W10, Kita-ku, Sapporo, Hokkaido 001-0021, Japan
*
*Corresponding author: Hiroyasu Katsuno, E-mail: [email protected]
Get access

Abstract

Low electron dose observation is indispensable for observing various samples using a transmission electron microscope; consequently, image processing has been used to improve transmission electron microscopy (TEM) images. To apply such image processing to in situ observations, we here apply a convolutional neural network to TEM imaging. Using a dataset that includes short-exposure images and long-exposure images, we develop a pipeline for processed short-exposure images, based on end-to-end training. The quality of images acquired with a total dose of approximately $5$ $e^{-}$ per pixel becomes comparable to that of images acquired with a total dose of approximately $1{,}000$ $e^{-}$ per pixel. Because the conversion time is approximately 8 ms, in situ observation at 125 fps is possible. This imaging technique enables in situ observation of electron-beam-sensitive specimens.

Keywords

deep learningfast improvementnanoparticlestransmission electron microscopy
Type
Software and Instrumentation
Information
Microscopy and Microanalysis , Volume 28 , Issue 1 , February 2022 , pp. 138 - 144
Check for updates
Copyright
Copyright © The Author(s), 2021. Published by Cambridge University Press on behalf of the Microscopy Society of America

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Chen, C, Chen, Q, Xu, J & Koltun, V (2018). Learning to see in the dark. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3291–3300.CrossRefGoogle Scholar
Chlanda, P & Sachse, M (2014). Cryo-electron microscopy of vitreous sections. Methods Mol Biol 1117, 193–214.CrossRefGoogle ScholarPubMed
Dabov, K, Foi, A, Katkovnik, V & Egiazarian, K (2007). Image denoising by sparse 3-D transform-domain. IEEE Trans Image Process 16, 2080–2095.CrossRefGoogle ScholarPubMed
De Jonge, N, Houben, L, D-Borkowski, RE & Ross, FM (2019). Resolution and aberration correction in liquid cell transmission electron microscopy. Nat Rev Mater 4, 61–78.CrossRefGoogle Scholar
De Jonge, N & Ross, FM (2011). Electron microscopy of pecimens in liquid. Nat Nanotechnol 6, 695–704.CrossRefGoogle Scholar
Elad, M & Aharon, M (2006). Image denoising via sparse and redundant. IEEE Trans Image Process 15, 3736–3745.CrossRefGoogle ScholarPubMed
Falk, T, Mai, D, Bensch, R, Özgün, Ç, Abdulkadir, A, Marrakchi, Y, Böhm, A, Deubner, J, Jäckel, Z, Seiwald, K, Dovzhenko, A, Tietz, O, Bosco, CD, Walsh, S, Saltukoglu, DL, Tay, TL, Prinz, M, Palme, K, Simons, M, Diester, I, Brox, T & Ronneberger, O (2019). U-Net: Deep learning for cell counting, detection, and morphometry. Nat Methods 16, 67–70.CrossRefGoogle ScholarPubMed
Fernández-Morán, H & Dahl, AO (1952). Electron microscopy of ultrathin frozen sections of pollen grains. Science 116, 465–467.CrossRefGoogle ScholarPubMed
Gu, S, Zhang, L, Zou, W & Feng, X (2014). Weighted nuclear norm minimization with application to image denoising. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2862–2869.CrossRefGoogle Scholar
He, K, Zhang, X, Ren, S & Sun, J (2016). Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778.CrossRefGoogle Scholar
Kingma, DP & Ba, J (2015). Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning. Available at arXiv:1412.6980v5Google Scholar
Li, PH, Lindsey, LF, Januszewski, M, Zheng, Z, Bates, AS, Taisz, I, Tyka, M, Nichols, M, Li, F, Perlman, E, Maitin-Shepard, J, Blakely, T, Leavitt, L, Jefferis, GSXE, Bock, D & Jain, V (2020). Automated reconstruction of a serial-section EM drosophila brain with flood-filling networks and local realignment. bioRxiv:605634.Google Scholar
Lim, J, Kim, J-H, Sim, J-Y & Kim, C-S (2015). Robust contrast enhancement of noisy low-light images: Denoising-enhancement completion. In 2015 IEEE International Conference on Image Processing (ICIP), pp. 4131–4135.CrossRefGoogle Scholar
Loh, YP & Chan, CS (2019). Getting to know low-light images with the exclusively dark dataset. Comput Vision Image Understanding 178, 30–42.CrossRefGoogle Scholar
Madsen, J, Liu, P, Kling, J, Wagner, JB, Hansen, TW, Winther, O & Schiøtz, J (2018). A deep learning approach to identify local structures in atomic-resolution transmission electron microscopy images. Adv Theory Simul 1, 1800037.CrossRefGoogle Scholar
Ronneberger, O, Fischer, P & Thomas, B (2015). U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015, Lecture Notes in Computer Science, Vol. 9351, Navab N, Hornegger J, Wells W & Frangi A (Eds.), pp. 234–241. Cham: Springer.CrossRefGoogle Scholar
Sadre, R, Ophus, C, Butko, A & Weber, GH (2020). Deep learning segmentation of complex features in atomic-resolution phase contrast transmission electron microscopy images. arXiv:2012.05322v1.Google Scholar
Schneider, NM, Norton, MM, Mendel, BJ, Gorgan, JM, Ross, FM, Bau, HH (2014). Electron-water interactions and implications for liquid cell electron microscopy. J Phys Chem C 118, 22373–22382.CrossRefGoogle Scholar
Shi, W, Caballero, J, Hauzar, F, Totz, J, Aitken, AP, Bishop, R, Rueckert, D & Wang, Z (2016). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883.CrossRefGoogle Scholar
Yakubovskiy, P (2020). Segmentation models: Python library with neural networks for image segmentation based on PyTorch. GitHub repository. Available at https://github.com/qubvel/segmentation_models.pytorch.Google Scholar