Step 1: Recognizing the relationship between integrals.
The function \( x^2 e^{-x^2} \) is the derivative of \( -\frac{1}{2} e^{-x^2} \). Thus, we use integration by parts to evaluate the integral:
\[
\int_0^t x^2 e^{-x^2} dx = \frac{1}{2} \left( - e^{-t^2} \right).
\]
Step 2: Taking the limit.
Taking the limit as \( t \to \infty \), we get:
\[
\lim_{t \to \infty} \int_0^t x^2 e^{-x^2} dx = \frac{\sqrt{\pi}}{4}.
\]
Step 3: Conclusion.
Thus, the correct answer is (A).