I was thinking more that if you have a highly specular (reflective) object casting a shadow on a highly specular (reflective) ground plane, there will be more light bounces between them that require evaluation. If your samples aren't high enough, you aren't evaluating them, thus noise.
Rather than raising your light samples, it seems worth it to me to check to see if your material is 50% or more specularity and then reducing it if the material doesn't require that much specularity.
Then just try another render. You may get a smoother shadow at lower samples with the benefit of less rendering overhead.
I think I have to stand a bit corrected, but I don´t think that has to do with highly specular surface or pure roughness, it´s just that the ray samples need to be evaluated per surface when ray hit´s.
The surface noise in the parts of any surface that is in the shadows, it will not matter much if you raise the light samples, the light samples seem to only bother about the softness in the shadow if it has any softness angle set to it to produce softer shadow at the ends of the shadow, or if using domelight or a very high angle similar to producing completely soft shadows, and that was what was the issue. (increase light samples)
Another issue is the actual (shadow) noise on any surface being in the shadow, that can´t be cleared up by increasing light samples, but have to be adressed with cam samples, going from 4-15 perhaps.
But you also talk about roughness and specularity which I believe introduces the other type of noise such as fireflies, which is a bit different.