The correct answer is
\(\frac{4r}{3}.\)A sphere of fixed radius
\((r)\) is given.
Let
\(R\) and
\(h\) be the radius and the height of the cone respectively.

The volume
\((V)\) of the cone is given by,
\(V=\frac{1}{3}πR^2h\)Now, from the right triangle BCD, we have:
\(BC=\sqrt{r^2-R^2}\)\(∴h=r+\sqrt{r^2-R^2}\)\(∴V=\frac{1}{3}πR^2(r+\sqrt{r^2-R^2})\)\(=\frac{1}{3}πR^2r+\frac{1}{3}πR^2\sqrt{r^2-R^2}\)\(∴\frac{dV}{dR}=\frac{2}{3}πRr+\frac{2π}{3}πR\sqrt{r^2-R^2}+\frac{R^2}{3}.\frac{(-2R)}{2\sqrt{r^2-R^2}}\)\(=\frac{2}{3}πRr+\frac{2π}{3πR}\sqrt{r^2-R^2}-\frac{R^3}{3\sqrt{r^2-R^2}}\)\(=\frac{2}{3}πRr+\frac{2πR(r^2-R^2)-\pi R^3}{3\sqrt{r^2-R^2}}\)\(=\frac{2}{3}πRr+\frac{2πRr^2-3\pi R^3}{3\sqrt{r^2-R^2}}\)Now,
\(\frac{dV}{dR^2}=0\)\(⇒\frac{2πrR}{3}=\frac{3πR^3-2πRr^3}{3\sqrt{r^2-R^2}}\)\(⇒2r\sqrt{r^2-R^2}=3R^2-2r^2\)\(⇒4r^2(r^2-R^2)=3R^2-2r^2\)\(=4r^4-4r^2R^2=9R^4+4r^4-12R^2r^2\)\(⇒9R^4-8r^2R^2=0\)\(⇒9R^2=8r^2\)\(⇒R^2=\frac{8r^2}{9}\)Now,
\(\frac{d^2V}{dR^2}=\frac{2πr}{3}+\frac{3\sqrt{r^2-R^2}(2πr^2-9πR^2)-(2πRr^2-3πR^3)(-6R)\frac{1}{2\sqrt{r^2-R^2}}}{9(r^2-R^2)}\)\(=\frac{2πr}{3}+\frac{3\sqrt{r^2-R^2}(2πr^2-9πR^2)-(2πRr^2-3πR^3)(3R)\frac{1}{2\sqrt{r^2-R^2}}}{9(r^2-R^2)}\)Now when
\(R^2=\frac{8r^2}{9}\),it can be shown that
\(\frac{d^2V}{dR^2}<0.\)∴ The volume is the maximum when
\(R^2=\frac{8r^2}{9}.\)When
\(R^2=\frac{8r^2}{9}.\),height of the cone
\(=r+\sqrt{r^2-\frac{8r^2}{9}}\)\(=r+\sqrt{\frac{r^2}{9}}=r+\frac{r}{3}=\frac{4r}{3}.\)Hence, it can be seen that the altitude of the right circular cone of maximum volume that can be inscribed in a sphere of radius
\(r\) is
\(\frac{4r}{3}.\)