Let the magnitudes of the two forces be \( F_1 \) and \( F_2 \), where \( F_1 = 5x \) and \( F_2 = 3x \) (since their magnitudes are in the ratio 5:3). The resultant \( R \) of the two forces is given by the formula:
\[
R = \sqrt{F_1^2 + F_2^2 + 2 F_1 F_2 \cos \theta}
\]
where \( \theta = 60^\circ \) is the angle between the two forces. Substituting the known values:
\[
35 = \sqrt{(5x)^2 + (3x)^2 + 2 \times 5x \times 3x \times \cos 60^\circ}
\]
Simplify:
\[
35 = \sqrt{25x^2 + 9x^2 + 30x^2 \times \frac{1}{2}}
\]
\[
35 = \sqrt{25x^2 + 9x^2 + 15x^2} = \sqrt{49x^2}
\]
\[
35 = 7x
\]
\[
x = 5
\]
Now, substitute \( x = 5 \) into the expressions for \( F_1 \) and \( F_2 \):
\[
F_1 = 5x = 25 \, \text{N}, \quad F_2 = 3x = 15 \, \text{N}
\]
Thus, the magnitudes of the two forces are \( 25 \, \text{N} \) and \( 15 \, \text{N} \).
The correct answer is option (3), \( 25 \, \text{N} \) and \( 15 \, \text{N} \).