Ice cubes typically have a temperature of around -18°C, while tap water is around 10°C. When the cubes are dropped in, the relative warmth makes their outer layers start to expand, while their cold inner layers remain unchanged. This creates tension between the layers which can tear the cube apart.
Whether it does or not depends on the presence of ‘defects’ in the ice cubes, which are caused by uneven cooling and/or dissolved oxygen. These lead to the layers of ice being poorly connected to one another, and if the defects are big enough, they’ll allow the internal tension to crack the whole ice cube apart