The gravitational potential energy between two masses \( m_1 \) and \( m_2 \) separated by a distance \( r \) is given by:
\[
U = -\frac{G m_1 m_2}{r}
\]
For the three masses \( m \), \( 2m \), and \( 3m \) placed at the vertices of an equilateral triangle of side \( a \), the potential energy is the sum of the potential energies between each pair of masses.
Step 1: Calculate potential energy between each pair of masses.
Between \( m \) and \( 2m \):
\[
U_{m, 2m} = -\frac{G m \cdot 2m}{a} = -\frac{2 G m^2}{a}
\]
Between \( 2m \) and \( 3m \):
\[
U_{2m, 3m} = -\frac{G \cdot 2m \cdot 3m}{a} = -\frac{6 G m^2}{a}
\]
Between \( m \) and \( 3m \):
\[
U_{m, 3m} = -\frac{G \cdot m \cdot 3m}{a} = -\frac{3 G m^2}{a}
\]
Step 2: Total gravitational potential energy.
The total potential energy is the sum of the individual potential energies:
\[
U_{\text{total}} = U_{m, 2m} + U_{2m, 3m} + U_{m, 3m}
\]
\[
U_{\text{total}} = -\frac{2 G m^2}{a} - \frac{6 G m^2}{a} - \frac{3 G m^2}{a} = -\frac{11 G m^2}{a}
\]
Final Answer:
\[
\boxed{\frac{-11 G m^2}{a}}
\]