“I always say I’ve learned that girls are God’s revenge,” he jokes to PEOPLE. “It’s payback for my past.”

Born on Aug. 9, Emily joins big siblings Sarah Brianna, 2½, and Colin Michael, 4½ – Stanley’s two children with wife Erin, an attorney – and 17-year-old Evan Shane, the rocker’s son from a previous relationship.

“Honestly, it’s just the greatest gift,” he says of adding another little girl to the family.

“There’s a relationship and a bond between dads and daughters that only a dad can understand. It’s very unique. To have my last two children be girls is...