The sentence describes an unreal situation in the past.
This is a third conditional structure:
“If + past perfect, would have + past participle”.
So, “If you had studied, you would have been confident.”
Other options don’t fit the standard conditional format.