To find the value of \(\lambda^2+\lambda\), given that the planes \(\overrightarrow{r}.(2\hat{i}-\lambda\hat{j}+3\hat{k})=0\) and \(\overrightarrow{r}.(\lambda\hat{i}+5\hat{j}-\hat{k})=5\) are perpendicular, we use the condition that the normal vectors of two perpendicular planes have a dot product equal to zero.
The normal vector of the first plane is \(\mathbf{n_1}=(2, -\lambda, 3)\) and the normal vector of the second plane is \(\mathbf{n_2}=(\lambda, 5, -1)\).
Their dot product is:
\( \mathbf{n_1} \cdot \mathbf{n_2} = 2\lambda + (-\lambda) \cdot 5 + 3 \cdot (-1) = 0 \)
This simplifies to:
\( 2\lambda - 5\lambda - 3 = 0 \)
\( -3\lambda - 3 = 0 \)
\( -3(\lambda + 1) = 0 \)
Which gives:
\( \lambda + 1 = 0 \Rightarrow \lambda = -1 \)
Now, calculating \(\lambda^2 + \lambda\):
\( \lambda^2 + \lambda = (-1)^2 + (-1) = 1 - 1 = 0 \)
Thus, the value of \(\lambda^2 + \lambda\) is 0.