Let the given expression be \( E = (1 + \alpha x + \beta x^2)(1+x)^{11} \).
We know that the general term in the binomial expansion of \( (1+x)^n \) is \( T_{k+1} = \binom{n}{k}x^k \).
So, for \( (1+x)^{11} \), the general term is \( \binom{11}{k}x^k \).
The expansion of \( E \) can be written as:
\[
E = 1 \cdot (1+x)^{11} + \alpha x \cdot (1+x)^{11} + \beta x^2 \cdot (1+x)^{11}
\]
Let's find the coefficient of \( x^{10} \):
The terms that contribute to \( x^{10} \) are:
% Option
(A) From \( 1 \cdot (1+x)^{11} \): The term with \( x^{10} \) is \( 1 \cdot \binom{11}{10}x^{10} \).
% Option
(B) From \( \alpha x \cdot (1+x)^{11} \): The term with \( x^{10} \) is \( \alpha x \cdot \binom{11}{9}x^9 = \alpha \binom{11}{9}x^{10} \).
% Option
(C) From \( \beta x^2 \cdot (1+x)^{11} \): The term with \( x^{10} \) is \( \beta x^2 \cdot \binom{11}{8}x^8 = \beta \binom{11}{8}x^{10} \).
The coefficient of \( x^{10} \) is:
\[
C_{10} = \binom{11}{10} + \alpha \binom{11}{9} + \beta \binom{11}{8}
\]
We know the binomial coefficients:
\[
\binom{11}{10} = \binom{11}{1} = 11, \binom{11}{9} = \binom{11}{2} = \frac{11 \times 10}{2} = 55, \binom{11}{8} = \binom{11}{3} = \frac{11 \times 10 \times 9}{6} = 165
\]
Given \( C_{10} = 396 \):
\[
11 + 55\alpha + 165\beta = 396
\]
Divide by 11:
\[
1 + 5\alpha + 15\beta = 36 \Rightarrow 5\alpha + 15\beta = 35 \Rightarrow \alpha + 3\beta = 7 \text{(Equation 1)}
\]
Now, let's find the coefficient of \( x^{11} \):
The terms that contribute to \( x^{11} \) are:
% Option
(A) From \( 1 \cdot (1+x)^{11} \): The term with \( x^{11} \) is \( \binom{11}{11}x^{11} \).
% Option
(B) From \( \alpha x \cdot (1+x)^{11} \): The term with \( x^{11} \) is \( \alpha \binom{11}{10}x^{11} \).
% Option
(C) From \( \beta x^2 \cdot (1+x)^{11} \): The term with \( x^{11} \) is \( \beta \binom{11}{9}x^{11} \).
The coefficient of \( x^{11} \) is:
\[
C_{11} = \binom{11}{11} + \alpha \binom{11}{10} + \beta \binom{11}{9}
\]
\[
\binom{11}{11} = 1, \binom{11}{10} = 11, \binom{11}{9} = 55
\]
Given \( C_{11} = 144 \):
\[
1 + 11\alpha + 55\beta = 144 \Rightarrow 11\alpha + 55\beta = 143
\]
Divide by 11:
\[
\alpha + 5\beta = 13 \text{(Equation 2)}
\]
Now we have the system of equations:
\begin{align}
\alpha + 3\beta &= 7 \text{(1)}
\alpha + 5\beta &= 13 \text{(2)}
\end{align}
Subtract Equation (1) from Equation (2):
\[
(\alpha + 5\beta) - (\alpha + 3\beta) = 13 - 7 \Rightarrow 2\beta = 6 \Rightarrow \beta = 3
\]
Substitute \( \beta = 3 \) into Equation (1):
\[
\alpha + 3(3) = 7 \Rightarrow \alpha + 9 = 7 \Rightarrow \alpha = -2
\]
Finally, we need to find \( \alpha^2 + \beta^2 \):
\[
\alpha^2 + \beta^2 = (-2)^2 + (3)^2 = 4 + 9 = 13
\]