Minimize $f(x,y,z)=x^2+y^2+z^2$ subject to
\[
g_1(x,y,z)=z-xy-5=0, \qquad g_2(x,y,z)=x+y+z-1=0.
\]
Use Lagrange multipliers: $\nabla f=\lambda \nabla g_1+\mu \nabla g_2$.
\[
(2x,2y,2z)=\lambda(-y,-x,1)+\mu(1,1,1).
\]
From the first two coordinates,
\[
2x+\lambda y=\mu,\qquad 2y+\lambda x=\mu
\Rightarrow (2-\lambda)(x-y)=0.
\]
If $x=y$, then from constraints $z=1-2x$ and $z=x^2+5$ give
$x^2+2x+4=0$, impossible; hence $\lambda=2$.
Then from the third coordinate $2z=\lambda+\mu=2+\mu\Rightarrow \mu=2z-2$.
Using $2x+2y=\mu$ gives $x+y=z-1$. With $x+y+z=1$ we get $z=1$ and $x+y=0$.
Using $z=xy+5$ yields $xy=-4$; together with $y=-x$ gives $x=\pm2$, $y=\mp2$.
Thus $P$ is $(2,-2,1)$ or $(-2,2,1)$ and
\[
\|OP\|=\sqrt{2^2+(-2)^2+1^2}=\sqrt{9}= \boxed{3}.
\]