This might be a result of too much or little of Tolerance and Smoothness settings in the LiveMatte Settings.