Let the three consecutive odd numbers be \( x-2, x, x+2 \).
Then their average is:
\[
\frac{(x-2) + x + (x+2)}{3} = \frac{3x}{3} = x
\]
Let the three even numbers be \( y-2, y, y+2 \). Their average is also \( y \)
According to the question:
\[
y - 2 = x + 2 + 9 \Rightarrow y = x + 13
\]
Also given:
\[
y^2 - x^2 = 507
\]
Substitute \( y = x + 13 \):
\[
(x + 13)^2 - x^2 = 507 \Rightarrow x^2 + 26x + 169 - x^2 = 507
\Rightarrow 26x = 338 \Rightarrow x = 13
\]
Then smallest odd number:
\[
x - 2 = 13 - 2 = \boxed{11}
\]