Hostname: page-component-586b7cd67f-l7hp2 Total loading time: 0 Render date: 2024-12-01T00:07:28.389Z Has data issue: false hasContentIssue false
  • English
  • Français

Optimizing low-Reynolds-number predation via optimal control and reinforcement learning

Published online by Cambridge University Press:  22 June 2022

Guangpu Zhu ,
Wen-Zhen Fang  and
Lailai Zhu Open the ORCID record for Lailai Zhu [Opens in a new window]
Guangpu Zhu
Affiliation:
Department of Mechanical Engineering, National University of Singapore, 117575, Republic of Singapore
Wen-Zhen Fang
Affiliation:
Department of Mechanical Engineering, National University of Singapore, 117575, Republic of Singapore Key Laboratory of Thermo-Fluid Science and Engineering, MOE, Xi'an Jiaotong University, Xi'an 710049, PR China
Lailai Zhu*
Affiliation:
Department of Mechanical Engineering, National University of Singapore, 117575, Republic of Singapore
*
†Email address for correspondence: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

We seek the best stroke sequences of a finite-size swimming predator chasing a non-motile point or finite-size prey at low Reynolds number. We use optimal control to seek the globally optimal solutions for the former and reinforcement learning (RL) for general situations. The predator is represented by a squirmer model that can translate forward and laterally, rotate and generate a stresslet flow. We identify the predator's best squirming sequences to achieve the time-optimal (TO) and efficiency-optimal (EO) predation. For a point prey, the TO squirmer executing translational motions favours a two-fold $L$-shaped trajectory that enables it to exploit the disturbance flow for accelerated predation; using a stresslet mode expedites significantly the EO predation, allowing the predator to catch the prey faster yet with lower energy consumption and higher predatory efficiency; the predator can harness its stresslet disturbance flow to suck the prey towards itself; compared to a translating predator, its compeer combining translation and rotation is less time-efficient, and the latter occasionally achieves the TO predation via retreating in order to advance. We also adopt RL to reproduce the globally optimal predatory strategy of chasing a point prey, qualitatively capturing the crucial two-fold attribute of a TO path. Using a numerically emulated RL environment, we explore the dependence of the optimal predatory path on the size of prey. Our results might provide useful information that help in the design of synthetic microswimmers such as in vivo medical microrobots capable of capturing and approaching objects in viscous flows.

JFM classification

Biological Fluid Dynamics: Micro-organism dynamics Biological Fluid Dynamics: Swimming/flying Micro-/Nano-fluid dynamics: Microscale transport
Type
JFM Papers
Information
Journal of Fluid Mechanics , Volume 944 , 10 August 2022 , A3
Check for updates
Copyright
© The Author(s), 2022. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

‡

G. Zhu and W.-Z. Fang contributed equally to this work.

References

REFERENCES

Achiam, J. 2018 Spinning up in deep reinforcement learning. https://spinningup.openai.com/en/latest/.Google Scholar
Alageshan, J.K., Verma, A.K., Bec, J. & Pandit, R. 2020 Machine learning strategies for path-planning microswimmers in turbulent flows. Phys. Rev. E 101 (4), 043110.CrossRefGoogle ScholarPubMed
Andersen, A. & Kiørboe, T. 2020 The effect of tethering on the clearance rate of suspension-feeding plankton. Proc. Natl Acad. Sci. USA 117 (48), 30101–30103.CrossRefGoogle ScholarPubMed
Blake, J.R. 1971 A spherical envelope approach to ciliary propulsion. J. Fluid Mech. 46 (1), 199–208.CrossRefGoogle Scholar
Borra, F., Biferale, L., Cencini, M. & Celani, A. 2022 Reinforcement learning for pursuit and evasion of microswimmers at low Reynolds number. Phys. Rev. Fluids 7 (2), 023103.CrossRefGoogle Scholar
Ceylan, H., Yasa, I.C., Kilic, U., Hu, W. & Sitti, M. 2019 Translational prospects of untethered medical microrobots. Prog. Biomed. Engng 1 (1), 012002.CrossRefGoogle Scholar
Clark, A.J. & Adelberg, E.A. 1962 Bacterial conjugation. Annu. Rev. Microbiol. 16 (1), 289–319.Google ScholarPubMed
Colabrese, S., Gustavsson, K., Celani, A. & Biferale, L. 2017 Flow navigation by smart microswimmers via reinforcement learning. Phys. Rev. Lett. 118 (15), 158004.CrossRefGoogle ScholarPubMed
Daddi-Moussa-Ider, A., Löwen, H. & Liebchen, B. 2021 Hydrodynamics can determine the optimal route for microswimmer navigation. Commun. Phys. 4 (1), 15.CrossRefGoogle Scholar
Dashiff, A., Junka, R.A., Libera, M. & Kadouri, D.E. 2011 Predation of human pathogens by the predatory bacteria Micavibrio aeruginosavorus and Bdellovibrio bacteriovorus. J. Appl. Microbiol. 110 (2), 431–444.CrossRefGoogle ScholarPubMed
Datt, C. & Elfring, G.J. 2019 Active particles in viscosity gradients. Phys. Rev. Lett. 123 (15), 158006.CrossRefGoogle ScholarPubMed
Datt, C., Zhu, L., Elfring, G.J. & Pak, O.S. 2015 Squirming through shear-thinning fluids. J. Fluid Mech. 784, R1.CrossRefGoogle Scholar
Dölger, J., Nielsen, L.T., Kiørboe, T. & Andersen, A. 2017 Swimming and feeding of mixotrophic biflagellates. Sci. Rep. 7 (1), 39892.CrossRefGoogle ScholarPubMed
Fenchel, T. 1980 Suspension feeding in ciliated protozoa: structure and function of feeding organelles. Arch. Protistenkd. 123 (3), 239–260.CrossRefGoogle Scholar
Fenchel, T. 1986 The ecology of heterotrophic microflagellates. In Advances in Microbial Ecology (ed. K.E. Nelson), pp. 57–97. Springer.CrossRefGoogle Scholar
Gazzola, M., Tchieu, A.A., Alexeev, D., de Brauer, A. & Koumoutsakos, P. 2016 Learning to school in the presence of hydrodynamic interactions. J. Fluid Mech. 789, 726–749.CrossRefGoogle Scholar
Hansen, B., Bjornsen, P.K. & Hansen, P.J. 1994 The size ratio between planktonic predators and their prey. Limnol. Oceanogr. 39 (2), 395–403.CrossRefGoogle Scholar
Ishikawa, T., Brumley, D.R. & Pedley, T.J. 2021 Rheology of a concentrated suspension of spherical squirmers: monolayer in simple shear flow. J. Fluid Mech. 914, A26.CrossRefGoogle Scholar
Ishimoto, K. & Gaffney, E.A. 2013 Squirmer dynamics near a boundary. Phys. Rev. E 88 (6), 062702.CrossRefGoogle Scholar
Jabbarzadeh, M. & Fu, H.C. 2018 Viscous constraints on microorganism approach and interaction. J. Fluid Mech. 851, 715–738.CrossRefGoogle Scholar
Jakobsen, H.H. 2001 Escape response of planktonic protists to fluid mechanical signals. Mar. Ecol. Prog. Ser. 214, 67–78.CrossRefGoogle Scholar
Jiao, Y., Ling, F., Heydari, S., Kanso, E., Heess, N. & Merel, J. 2021 Learning to swim in potential flow. Phys. Rev. Fluids 6 (5), 050505.CrossRefGoogle Scholar
Kiørboe, T. 2003 Marine snow microbial communities: scaling of abundances with aggregate size. Aquat. Microb. Ecol. 33 (1), 67–75.CrossRefGoogle Scholar
Kiørboe, T. 2016 Fluid dynamic constraints on resource acquisition in small pelagic organisms. Eur. Phys. J. 225 (4), 669–683.Google Scholar
Kiørboe, T., Andersen, A., Langlois, V.J., Jakobsen, H.H. & Bohr, T. 2009 Mechanisms and feasibility of prey capture in ambush-feeding zooplankton. Proc. Natl Acad. Sci. USA 106 (30), 12394–12399.CrossRefGoogle ScholarPubMed
Kiørboe, T., Jiang, H., Gonçalves, R.J., Nielsen, L.T. & Wadhwa, N. 2014 Flow disturbances generated by feeding and swimming zooplankton. Proc. Natl Acad. Sci. USA 111 (32), 11738–11743.CrossRefGoogle ScholarPubMed
Kuhnle, A., Schaarschmidt, M. & Fricke, K. 2017 Tensorforce: a tensorflow library for applied reinforcement learning. Web page. https://tensorforce.readthedocs.io/en/latest/.Google Scholar
Kurzthaler, C., Zhu, L., Pahlavan, A.A. & Stone, H.A. 2020 Particle motion nearby rough surfaces. Phys. Rev. Fluids 5 (8), 082101.CrossRefGoogle Scholar
Lambert, R.A., Picano, F., Breugem, W. -P. & Brandt, L. 2013 Active suspensions in thin films: nutrient uptake and swimmer motion. J. Fluid Mech. 733, 528–557.CrossRefGoogle Scholar
Langlois, V.J., Andersen, A., Bohr, T., Visser, A.W. & Kiørboe, T. 2009 Significance of swimming and feeding currents for nutrient uptake in osmotrophic and interception-feeding flagellates. Aquat. Microb. Ecol. 54 (1), 35–44.Google Scholar
Lehman, J. & Stanley, K.O. 2011 Novelty search and the problem with objectives. In Genetic Programming Theory and Practice IX (ed. R. Riolo, E. Vladislavleva & J.H. Moore), pp. 37–56. Springer.CrossRefGoogle Scholar
Li, G., Lauga, E. & Ardekani, A.M. 2021 Microswimming in viscoelastic fluids. J. Non-Newtonian Fluid Mech. 197, 104655.CrossRefGoogle Scholar
Liepins, G.E. & Vose, M.D. 1991 Deceptiveness and genetic algorithm dynamics. In Foundations of Genetic Algorithms (ed. G.J.E. Rawlins), vol. 1, pp. 36–50. Elsevier.CrossRefGoogle Scholar
Lighthill, M.J. 1952 On the squirming motion of nearly spherical deformable bodies through liquids at very small Reynolds numbers. Commun. Pure Appl. Maths 5, 109–118.CrossRefGoogle Scholar
Lintuvuori, J.S., Würger, A. & Stratford, K. 2017 Hydrodynamics defines the stable swimming direction of spherical squirmers in a nematic liquid crystal. Phys. Rev. Lett. 119 (6), 068001.CrossRefGoogle Scholar
Magar, V., Goto, T. & Pedley, T.J. 2003 Nutrient uptake by a self-propelled steady squirmer. Q. J. Mech. Appl. Maths 56 (1), 65–91.CrossRefGoogle Scholar
Mathijssen, A.J.T.M., Jeanneret, R. & Polin, M. 2018 Universal entrainment mechanism controls contact times with motile cells. Phys. Rev. Fluids 3 (3), 033103.CrossRefGoogle Scholar
Michelin, S. & Lauga, E. 2011 Optimal feeding is optimal swimming for all Péclet numbers. Phys. Fluids 23 (10), 101901.CrossRefGoogle Scholar
Mirzakhanloo, M., Esmaeilzadeh, S. & Alam, M.-R. 2020 Active cloaking in Stokes flows via reinforcement learning. J. Fluid Mech. 903, A34.CrossRefGoogle Scholar
More, R.V. & Ardekani, A.M. 2020 Motion of an inertial squirmer in a density stratified fluid. J. Fluid Mech. 905, A9.CrossRefGoogle Scholar
Muiños-Landin, S., Fischer, A., Holubec, V. & Cichos, F. 2021 Reinforcement learning with artificial microswimmers. Sci. Rob. 6 (52), eabd9285.Google ScholarPubMed
Nasiri, M. & Liebchen, B. 2022 Reinforcement learning of optimal active particle navigation. arXiv:2202.00812.Google Scholar
Nelson, B.J., Kaliakatsos, I.K. & Abbott, J.J. 2010 Microrobots for minimally invasive medicine. Annu. Rev. Biomed. Engng 12, 55–85.CrossRefGoogle ScholarPubMed
Pak, O.S. & Lauga, E. 2014 Generalized squirming motion of a sphere. J. Engng Maths 88 (1), 1–28.CrossRefGoogle Scholar
Pérez, J., Moraleda-Muñoz, A., Marcos-Torres, F.J. & Muñoz-Dorado, J. 2016 Bacterial predation: 75 years and counting! Environ. Microbiol. 18 (3), 766–779.CrossRefGoogle ScholarPubMed
Pushkin, D.O., Shum, H. & Yeomans, J.M. 2013 Fluid transport by individual microswimmers. J. Fluid Mech. 726, 5–25.CrossRefGoogle Scholar
Qiu, J., Huang, W., Xu, C. & Zhao, L. 2020 Swimming strategy of settling elongated micro-swimmers by reinforcement learning. Sci. China 63 (8), 284711.Google Scholar
Qiu, J., Mousavi, N., Gustavsson, K., Xu, C., Mehlig, B. & Zhao, L. 2022 Navigation of micro-swimmers in steady flow: the importance of symmetries. J. Fluid Mech. 932, A10.CrossRefGoogle Scholar
Reigh, S.Y., Zhu, L., Gallaire, F. & Lauga, E. 2017 Swimming with a cage: low-Reynolds-number locomotion inside a droplet. Soft Matt. 13 (17), 3161–3173.CrossRefGoogle Scholar
Rieck, M., Bittner, M., Grüter, B., Diepolder, J. & Piprek, P. 1999 Falcon.m user guide.Google Scholar
Schneider, E. & Stark, H. 2019 Optimal steering of a smart active particle. Europhys. Lett. 127 (6), 64003.CrossRefGoogle Scholar
Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. 2017 Proximal policy optimization algorithms. arXiv:1707.06347.Google Scholar
Sengupta, A., Kruppa, T. & Löwen, H. 2011 Chemotactic predator–prey dynamics. Phys. Rev. E 83 (3), 031914.CrossRefGoogle ScholarPubMed
Słomka, J., Alcolombri, U., Secchi, E., Stocker, R. & Fernandez, V.I. 2020 Encounter rates between bacteria and small sinking particles. New J. Phys. 22 (4), 043016.CrossRefGoogle Scholar
Spagnolie, S.E. & Lauga, E. 2012 Hydrodynamics of self-propulsion near a boundary: predictions and accuracy of far-field approximations. J. Fluid Mech. 700, 105–147.CrossRefGoogle Scholar
Strickler, J.R. 1998 Observing free-swimming copepods mating. Phil. Trans. R. Soc. Lond. B 353 (1369), 671–680.CrossRefGoogle Scholar
Sutton, R.S. & Barto, A.G. 2018 Reinforcement Learning: An Introduction. MIT Press.Google Scholar
Svensen, C. & Kiørboe, T. 2000 Remote prey detection in Oithona similis: hydromechanical versus chemical cues. J. Plankton Res. 22 (6), 1155–1166.CrossRefGoogle Scholar
Tam, D. & Hosoi, A.E. 2011 Optimal feeding and swimming gaits of biflagellated organisms. Proc. Natl Acad. Sci. USA 108 (3), 1001–1006.CrossRefGoogle ScholarPubMed
Tsang, A.C.H., Tong, P.W., Nallan, S. & Pak, O.S. 2020 Self-learning how to swim at low Reynolds number. Phys. Rev. Fluids 5 (7), 074101.CrossRefGoogle Scholar
Verma, S., Novati, G. & Koumoutsakos, P. 2018 Efficient collective swimming by harnessing vortices through deep reinforcement learning. Proc. Natl Acad. Sci. USA 115 (23), 5849–5854.CrossRefGoogle ScholarPubMed
Visser, A. 2001 Hydromechanical signals in the plankton. Mar. Ecol. Prog. Ser. 222, 1–24.CrossRefGoogle Scholar
Visser, A.W. & Kiørboe, T. 2006 Plankton motility patterns and encounter rates. Oecologia 148 (3), 538–546.CrossRefGoogle ScholarPubMed
Wächter, A. & Biegler, L.T. 2006 On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Math. Program. 106 (1), 25–57.CrossRefGoogle Scholar
Zhu, L. & Gallaire, F. 2017 Bifurcation dynamics of a particle-encapsulating droplet in shear flow. Phys. Rev. Lett. 119 (6), 064502.CrossRefGoogle ScholarPubMed
Zhu, L., Lauga, E. & Brandt, L. 2013 Low-Reynolds-number swimming in a capillary tube. J. Fluid Mech. 726, 285–311.CrossRefGoogle Scholar