Why stylized faces are the hard case
Face detection and face swapping are both built around the proportions of real human faces — eye spacing, nose length, jaw width. Anime and cartoon styles deliberately break those proportions: eyes are enormous, noses are minimal, chins are pointed. The further a style departs from real anatomy, the less reliably a face is detected at all, and the less naturally a real face maps onto it.
In practice this produces a clean gradient. Semi-realistic and detailed anime styles, or 3D-animated characters with broadly human proportions, work well. Highly abstract or chibi styles frequently will not detect a face, and when they do, the result tends to look like a photo pasted over a drawing rather than a character with a new face.