Recurrent Monitoring of Supercomputer Noise - Voevodin, Vadim V.; Nikitenko, Dmitry A. (Supercomputing Frontiers and Innovations, 2024)
The presence of noise in supercomputers has long been known, but its scale and impact on the behavior of user applications are nevertheless considerably unclear. Therefore, we decided to develop an approach to determining the noise level on an ongoing basis, which makes it possible to assess the global impact of noise over a long time period. This paper describes a method for recurrent monitoring the noise level and analyzing collected statistics on a real modern supercomputer, and also presents the implementation and evaluation of this method on the Lomonosov-2 supercomputer. The usage of the proposed approach in practice made it possible to identify previously unknown issues and peculiarities, like detection of a faulty compute node, presence of nodes that tend to be more susceptible to noise or the global nature of the noise, which leads to the appearance of noise at multiple nodes simultaneously. This method can as well be ported to other similar computing systems without significant changes.