Definition of "Downside" :

noun: a negative aspect of something that is generally positive

"There is a downside even to motherhood."