Let \( I = \int_0^1 xf''(x)dx \). We use integration by parts: \( \int u dv = uv - \int v du \).
Let \(u = x \Rightarrow du = dx\).
Let \(dv = f''(x)dx \Rightarrow v = f'(x)\).
So, \( I = [x f'(x)]_0^1 - \int_0^1 f'(x)dx \).
\( I = (1 \cdot f'(1) - 0 \cdot f'(0)) - [f(x)]_0^1 = f'(1) - (f(1) - f(0)) \).
Given \( f(x) = \sin(\tan^{-1} x) \).
Let \(\theta = \tan^{-1} x\), so \(\tan\theta = x\). Consider a right triangle with opposite side \(x\), adjacent side 1. Hypotenuse = \(\sqrt{x^2+1}\).
Then \(\sin\theta = \frac{x}{\sqrt{x^2+1}}\). So, \(f(x) = \frac{x}{\sqrt{x^2+1}}\).
Calculate \(f(0)\) and \(f(1)\):
\(f(0) = \frac{0}{\sqrt{0^2+1}} = 0\).
\(f(1) = \frac{1}{\sqrt{1^2+1}} = \frac{1}{\sqrt{2}}\).
Calculate \(f'(x)\). \(f(x) = x(x^2+1)^{-1/2}\).
Using quotient rule or product rule:
\(f'(x) = \frac{1 \cdot \sqrt{x^2+1} - x \cdot \frac{1}{2\sqrt{x^2+1}}(2x)}{(x^2+1)} = \frac{\sqrt{x^2+1} - \frac{x^2}{\sqrt{x^2+1}}}{x^2+1}\)
\(f'(x) = \frac{\frac{(x^2+1) - x^2}{\sqrt{x^2+1}}}{x^2+1} = \frac{1}{(x^2+1)\sqrt{x^2+1}} = (x^2+1)^{-3/2}\).
Calculate \(f'(1)\):
\(f'(1) = (1^2+1)^{-3/2} = (2)^{-3/2} = \frac{1}{2^{3/2}} = \frac{1}{2\sqrt{2}}\).
Substitute these values into the expression for I:
\( I = f'(1) - (f(1) - f(0)) = \frac{1}{2\sqrt{2}} - \left(\frac{1}{\sqrt{2}} - 0\right) \)
\( I = \frac{1}{2\sqrt{2}} - \frac{1}{\sqrt{2}} = \frac{1}{2\sqrt{2}} - \frac{2}{2\sqrt{2}} = \frac{1-2}{2\sqrt{2}} = -\frac{1}{2\sqrt{2}} \).
This matches option (b).
\[ \boxed{-\frac{1}{2\sqrt{2}}} \]