GPU-based Large-eddy Simulation of Mixed-phased Clouds

Evgeny V. Mortikov; Elizaveta M. Gashchuk; Andrey V. Debolskiy

doi:10.14529/jsfi250405

Authors

Evgeny V. Mortikov Research Computing Center, Lomonosov Moscow State University, Moscow, Russia; Moscow Center of Fundamental and Applied Mathematics, Moscow, Russia https://orcid.org/0000-0002-9683-5701
Elizaveta M. Gashchuk Research Computing Center, Lomonosov Moscow State University, Moscow, Russia; Moscow Center of Fundamental and Applied Mathematics, Moscow, Russia https://orcid.org/0000-0002-5748-1787
Andrey V. Debolskiy Research Computing Center, Lomonosov Moscow State University, Moscow, Russia; Moscow Center of Fundamental and Applied Mathematics, Moscow, Russia; A. M. Obukhov Institute of Atmospheric Physics, Moscow, Russia https://orcid.org/0000-0002-0182-5675

DOI:

https://doi.org/10.14529/jsfi250405

Keywords:

large-eddy simulation, boundary layer turbulence, cloud-resolving modeling, computational performance, graphics processing unit

Abstract

We discuss the development of the large-eddy simulation (LES) model of the atmospheric boundary layer with embedded two-moment bulk cloud microphysics scheme well-suited for massively-parallel heterogeneous supercomputers based on GPU (Graphics Processing Units) architecture. To evaluate the LES model and its computational efficiency, we consider the numerical setup corresponding to the development of an intense Arctic cold-air outbreak case. It is shown that the dynamic closure approach for calculation of subgrid scale fluxes, applied to both heat and moisture transport, allows to correctly reproduce moist convective boundary layers with mixed-phased clouds even with coarse grid resolution. Implementation of state-of-the-art microphysics scheme for GPU systems not only led to significant speedup of the computations, but in general improved the multi-GPU scaling of the model.

References

Baldauf, M., Seifert, A., Förstner, J., et al.: Operational convective-scale numerical weather prediction with the cosmo model: Description and sensitivities. Monthly Weather Review 139, 3887–3905 (12 2011). https://doi.org/10.1175/MWR-D-10-05013.1

Bhattacharjee, S., Mortikov, E., Debolskiy, A., et al.: Direct numerical simulation of a turbulent channel flow with Forchheimer drag. Boundary-Layer Meteorology 185(2), 259–276 (2022). https://doi.org/10.1007/s10546-022-00731-8

Bieringer, P.E., Piña, A.J., Lorenzetti, D.M., et al.: A Graphics Processing Unit (GPU) Approach to Large Eddy Simulation (LES) for Transport and Contaminant Dispersion. Atmosphere 12(7), 890 (2021). https://doi.org/10.3390/atmos12070890

Bigg, E.K.: The formation of atmospheric ice crystals by the freezing of droplets. Quarterly Journal of the Royal Meteorological Society 79(342), 510–519 (1953). https://doi.org/10.1002/qj.49707934207

Bou-Zeid, E., Meneveau, C., Parlange, M.: A scale-dependent Lagrangian dynamic model for large eddy simulation of complex turbulent flows. Physics of Fluids 17(2), 025105 (2005). https://doi.org/10.1063/1.1839152

Brown, D.L., Cortez, R., Minion, M.L.: Accurate projection methods for the incompressible Navier–Stokes equations. Journal of Computational Physics 168(2), 464–499 (2001). https://doi.org/10.1006/jcph.2001.6715

Cotton, R.J., Field, P.R.: Ice nucleation characteristics of an isolated wave cloud. Quarterly Journal of the Royal Meteorological Society 128(585), 2417–2437 (2002). https://doi.org/10.1256/qj.01.150

Debolskiy, A.V., Mortikov, E.M., Poliukhov, A.A.: Computational efficiency and adaptation of radiation transfer module within Large-eddy simulation model. Numerical Methods and Programming 26(4), 515–533 (2025). https://doi.org/10.26089/NumMet.v26r434

Debolskiy, A.V., Mortikov, E.V., Glazunov, A.V., Lüpkes, C.: Evaluation of surface layer stability functions and their extension to first order turbulent closures for weakly and strongly stratified stable boundary layer. Boundary-Layer Meteorology 187(1), 73–93 (2023). https://doi.org/10.1007/s10546-023-00784-3

Ding, C., He, Y.: A ghost cell expansion method for reducing communications in solving PDE problems. In: Proceedings of the 2001 ACM/IEEE Conference on Supercomputing. p. 50. SC ’01, Association for Computing Machinery, New York, NY, USA (2001). https://doi.org/10.1145/582034.582084

Donahue, A., Caldwell, P.: Impact of physics parameterization ordering in a global atmosphere model. Journal of Advances in Modeling Earth Systems 10(2), 481–499 (2018). https://doi.org/10.1002/2017MS001067

Gaschuk, E.M., Ezhkova, A.A., Onoprienko, V.A., et al.: Passive tracer transport in ocean modeling: implementation on GPUs, efficiency and optimizations. Lobachevskii Journal of Mathematics 44(8), 3040–3058 (2023). https://doi.org/10.1134/S1995080223080152

Geerts, B., Giangrande, S., Mcfarquhar, G., et al.: "COMBLE" Beginnings: The Cold-Air Outbreaks in the Marine Boundary Layer Experiment. Bulletin of the American Meteorological Society 104, 327–330 (03 2025). https://doi.org/10.1175/BAMS-D-21-0044.A

Geerts, B., Giangrande, S.E., McFarquhar, G.M., et al.: The COMBLE Campaign: A study of marine boundary layer clouds in Arctic cold-air outbreaks. Bulletin of the American Meteorological Society 103(5), E1371–E1389 (2022). https://doi.org/10.1175/BAMS-D-21-0044.1

Germano, M., Piomelli, U., Moin, P., Cabot, W.H.: A dynamic subgrid-scale eddy viscosity model. Physics of Fluids A: Fluid Dynamics 3(7), 1760–1765 (1991). https://doi.org/10.1063/1.857955

Giorgetta, M.A., Brokopf, R., Crueger, T., et al.: ICON-A, the Atmosphere Component of the ICON Earth System Model: I. Model Description. Journal of Advances in Modeling Earth Systems 10(7), 1613–1637 (2018). https://doi.org/10.1029/2017MS001242

Gladskikh, D., Ostrovsky, L., Troitskaya, Y., et al.: Turbulent transport in a stratified shear flow. Journal of Marine Science and Engineering 11(1), 136 (2023). https://doi.org/10.3390/jmse11010136

Glazunov, A., Mortikov, E., Debolskiy, A., Pashkin, A.: Large eddy simulation in the urban environment with simplified and realistic surface morphology. Russian Meteorology and Hydrology 50, 491–506 (2025). https://doi.org/10.3103/S1068373925060056

Glazunov, A., Rannik, U., Stepanenko, V., et al.: Large-eddy simulation and stochastic modeling of Lagrangian particles for footprint determination in the stable boundary layer. Geoscientific Model Development 9(9), 2925–2949 (2016). https://doi.org/10.5194/gmd-9-2925-2016

Glazunov, A.V., Mortikov, E.V., Barskov, K.V., et al.: Layered structure of stably stratified turbulent shear flows. Izvestiya, Atmospheric and Oceanic Physics 55(4), 312–323 (2019). https://doi.org/10.1134/S0001433819040042

Heus, T., van Heerwaarden, C.C., Jonker, H.J.J., et al.: Formulation of the Dutch Atmospheric Large-Eddy Simulation (DALES) and overview of its applications. Geoscientific Model Development 3(2), 415–444 (2010). https://doi.org/10.5194/gmd-3-415-2010

Juliano, T., Tornow, F., Fridlind, A.: COMBLE Model-Observation Intercomparison Project Cookbook. https://arm-development.github.io/comble-mip/README.html (2023), accessed: 2025-11-04

Juliano, T.W., Tornow, Florian Fridlind, A.M., Ackerman, A.S., et al.: The Cold-Air Outbreaks in the Marine Boundary Layer Experiment model-observation intercomparison project (COMBLE-MIP), Part I: Model specification, observational constraints, and preliminary findings. Geoscientific Model Development (2025), (in review)

Kadantsev, E., Mortikov, E., Glazunov, A., et al.: On dissipation timescales of the basic second-order moments: the effect on the energy and flux budget (EFB) turbulence closure for stably stratified turbulence. Nonlinear Processes in Geophysics 31(3), 395–408 (2024). https://doi.org/10.5194/npg-31-395-2024

Kadantsev, E., Mortikov, E., Zilitinkevich, S.: The resistance law for stably stratified atmospheric planetary boundary layers. Quarterly Journal of the Royal Meteorological Society 147(737), 2233–2243 (2021). https://doi.org/10.1002/qj.4019

Khain, A.P., Beheng, K.D., Heymsfield, A., et al.: Representation of microphysical processes in cloud-resolving models: Spectral (bin) microphysics versus bulk parameterization. Reviews of Geophysics 53(2), 247–322 (2015). https://doi.org/10.1002/2014RG000468

Lilly, D.K.: A proposed modification of the Germano subgrid-scale closure method. Physics of Fluids A 4(3), 633–635 (1992). https://doi.org/10.1063/1.858280

Lund, T.S.: On the use of discrete filters for large eddy simulation. In: Annual Research Briefs, pp. 83–95. Center for Turbulence Research, Stanford University: Stanford (1997)

Maronga, B., Banzhaf, S., Burmeister, C., et al.: Overview of the PALM model system 6.0. Geoscientific Model Development 13(3), 1335–1372 (2020). https://doi.org/10.5194/gmd-13-1335-2020

Meneveau, C., Lund, T.S., Cabot, W.H.: A Lagrangian dynamic subgrid-scale model of turbulence. Journal of Fluid Mechanics 319, 353–385 (1996). https://doi.org/10.1017/S0022112096007379

Morinishi, Y., Lund, T., Vasilyev, O., Moin, P.: Fully conservative higher order finite difference schemes for incompressible flow. Journal of Computational Physics 143(1), 90–124 (1998). https://doi.org/10.1006/jcph.1998.5962

Mortikov, E.V.: Numerical simulation of the motion of an ice keel in a stratified flow. Izvestiya, Atmospheric and Oceanic Physics 52(1), 108–115 (2016), https://journal-vniispk.ru/0001-4338/article/view/148416

Mortikov, E.V., Debolskiy, A.V.: Direct numerical simulation of stratified turbulent flows and passive tracer transport on HPC systems: Comparison of CPU architectures. Supercomputing Frontiers and Innovations 8(4), 50–68 (2021). https://doi.org/10.14529/jsfi210405

Mortikov, E.V., Debolskiy, A.V., Glazunov, A.V., et al.: Planetary boundary layer scheme in the INMCM Earth system model. Russian Journal of Numerical Analysis and Mathematical Modelling 39(6), 343–352 (2024). https://doi.org/10.1515/rnam-2024-0029

Mortikov, E.V., Glazunov, A.V., Lykosov, V.N.: Numerical study of plane Couette flow: turbulence statistics and the structure of pressure-strain correlations. Russian Journal of Numerical Analysis and Mathematical Modelling 34(2), 119–132 (2019). https://doi.org/10.1515/rnam-2019-0010

Ovchinnikov, M., Ackerman, A.S., Avramov, A., et al.: Intercomparison of large-eddy simulations of Arctic mixed-phase clouds: Importance of ice size distribution assumptions. Journal of Advances in Modeling Earth Systems 6(1), 223–248 (2014). https://doi.org/10.1002/2013MS000282

Reisner, J., Rasmussen, R.M., Bruintjes, R.T.: Explicit forecasting of supercooled liquid water in winter storms using the MM5 mesoscale model. Quarterly Journal of the Royal Meteorological Society 124(548), 1071–1107 (1998). https://doi.org/10.1002/qj.49712454804

Sagaut, P.: Large eddy simulation for incompressible flows. Springer (2006). https://doi.org/10.1007/b137536

Satoh, M., Stevens, B., Judt, F., et al.: Global cloud-resolving models. Current Climate Change Reports 5, 172–184 (2019). https://doi.org/10.1007/s40641-019-00131-0

Schalkwijk, J., Jonker, H., Siebesma, A.P., Meijgaard, E.: Weather Forecasting Using GPU-Based Large-Eddy Simulations. Bulletin of the American Meteorological Society 96, 715–723 (2015). https://doi.org/10.1175/BAMS-D-14-00114.1

Seifert, A., Beheng, K.D.: A two-moment cloud microphysics parameterization for mixedphase clouds. Part 2: Maritime vs. continental deep convective storms. Meteorology and Atmospheric Physics 92(1-2), 67–82 (2006). https://doi.org/10.1007/s00703-005-0113-3

Seifert, A., Heus, T.: Large-eddy simulation of organized precipitating trade wind cumulus clouds. Atmospheric Chemistry and Physics 13(11), 5631–5645 (2013). https://doi.org/10.5194/acp-13-5631-2013

Seifert, A., Beheng, K.D.: A double-moment parameterization for simulating autoconversion, accretion and selfcollection. Atmospheric Research 59-60, 265–281 (2001). https://doi.org/10.1016/S0169-8095(01)00126-0

Seifert, A., Stevens, B.: Microphysical scaling relations in a kinematic model of isolated shallow cumulus clouds. Journal of the Atmospheric Sciences 67, 1575–1590 (2010). https://doi.org/10.1175/2009JAS3319.1

Siebesma, A.P., Bretherton, C., Brown, A., et al.: A large eddy simulation intercomparison study of shallow cumulus convection. Journal of the Atmospheric Sciences 60, 1201–1219 (2003). https://doi.org/10.1175/1520-0469(2003)60<1201:ALESIS>2.0.CO;2

Smagorinsky, J.: Global atmospheric modeling and the numerical simulation of climate. In: Hess, W.N. (ed.) Weather and climate modification, pp. 633–686. John Wiley & Sons, Ltd: New York (1974)

Suiazova, V., Debolskiy, A., Mortikov, E.: Study of surface layer characteristics in the presence of suspended snow particles using observational data and Large-Eddy Simulation. Izv. Atmos. Ocean. Phys. 60(2), 183–195 (2024). https://doi.org/10.1134/S000143382470021X

Suiazova, V., Debolskiy, A., Mortikov, E.: Modeling turbulent flows over a heterogeneous surface using mesoscale and large eddy simulations. Russian Meteorology and Hydrology 50, 417–426 (2025). https://doi.org/10.3103/S106837392505005X

Tarasova, M., Debolskiy, A., Mortikov, E., et al.: On the parameterization of the mean wind profile for urban canopy models. Lobachevskii Journal of Mathematics 45, 3198–3210 (2024). https://doi.org/10.1134/S1995080224603801

Tkachenko, E., Debolskiy, A., Mortikov, E.: Intercomparison of subgrid scale models in large-eddy simulation of sunset atmospheric boundary layer turbulence: Computational aspects. Lobachevskii Journal of Mathematics 42, 1580–1595 (2021). https://doi.org/10.1134/S1995080221070234

Tomita, H.: New microphysical schemes with five and six categories by diagnostic generation of cloud ice. Journal of the Meteorological Society of Japan. Ser. II 86A, 121–142 (2008). https://doi.org/10.2151/jmsj.86A.121

vanZanten, M.C., Stevens, B., Nuijens, L., et al.: Controls on precipitation and cloudiness in simulations of trade-wind cumulus as observed during RICO. Journal of Advances in Modeling Earth Systems 3(2), M06001 (2011). https://doi.org/10.1029/2011MS000056

Varentsov, A.I., Mortikov, E.V., Glazunov, A.V., et al.: Large-eddy simulation of aerosol transport over different urban local climate zones. Geography, Environment, Sustainability 18(3), 68–79 (2025). https://doi.org/10.24057/2071-9388-2025-3925

Voevodin, V., Debolskiy, A., Mortikov, E.: Facilitating the Process of Performance Analysis of HPC Applications. Lobachevskii Journal of Mathematics 44, 3178–3190 (2023). https://doi.org/10.1134/S1995080223080589

Wisner, C., Orville, H.D., Myers, C.: A Numerical Model of a Hail-Bearing Cloud. Journal of the Atmospheric Sciences 29(6), 1160–1181 (1972). https://doi.org/10.1175/1520-0469(1972)029<1160:ANMOAH>2.0.CO;2

Wu, P., Ovchinnikov, M., Xiao, H., et al.: Effect of ice number concentration on the evolution of boundary layer clouds during Arctic marine cold-air outbreaks. Journal of Geophysical Research: Atmospheres 130(3), e2024JD041282 (2025). https://doi.org/10.1029/2024JD041282

Zasko, G.V., Glazunov, A.V., Mortikov, E.V., et al.: Optimal Energy Growth in Stably Stratified Turbulent Couette Flow. Boundary-Layer Meteorology 187(1), 395–421 (2023). https://doi.org/10.1007/s10546-022-00744-3

Zilitinkevich, S., Druzhinin, O., Glazunov, A., et al.: Dissipation rate of turbulent kinetic energy in stably stratified sheared flows. Atmospheric Chemistry and Physics 19(4), 2489–2496 (2019). https://doi.org/10.5194/acp-19-2489-2019