The total electrostatic potential energy \( U \) of the system with three charges at the vertices of an equilateral triangle can be given by the sum of the pairwise interactions:
\[
U = \frac{kQq}{a} + \frac{kQq}{a} + \frac{-kq^2}{a}
\]
Since the total potential energy depends on the interaction between all pairs of charges, and given that charges are at the vertices of an equilateral triangle, the total potential energy is simplified to:
\[
U = -\frac{kQ^2}{3a}
\]
Thus, the correct answer is \( -\frac{kQ^2}{3a} \).