To calculate the moment of inertia \( I \) of a spherical shell with inner radius \( a \) and outer radius \( b \), we need to use the integral formula for the moment of inertia of a continuous mass distribution:
\[
I = \int r^2 \, dm
\]
where \( r \) is the distance from the axis of rotation, and \( dm \) is the mass element of the shell.
The spherical shell has a uniform mass distribution. The mass element \( dm \) of a thin spherical shell of radius \( r \) and thickness \( dr \) is given by:
\[
dm = \frac{M}{(b^3 - a^3)} \cdot r^2 dr
\]
Thus, the total moment of inertia of the spherical shell is:
\[
I = \int_a^b r^2 \cdot \frac{M}{(b^3 - a^3)} \cdot r^2 dr = \frac{M}{(b^3 - a^3)} \int_a^b r^4 dr
\]
Integrating \( r^4 \), we get:
\[
I = \frac{M}{(b^3 - a^3)} \cdot \frac{r^5}{5} \Big|_a^b = \frac{M}{(b^3 - a^3)} \left( \frac{b^5 - a^5}{5} \right)
\]
Thus, the moment of inertia is:
\[
I = \frac{M}{5} \left( \frac{b^5 - a^5}{b^3 - a^3} \right)
\]
Thus, the correct answer is \( \frac{3M b^5 - 5M b^3}{b^3 - a^3} \).