"Overwhelmed with joy" is the correct idiomatic expression used to show that someone is overcome by a particular emotion.
Prepositions with Idioms: Prepositions like "with" often appear in set phrases, and "overwhelmed with" is one of those fixed expressions used to describe an emotional reaction.