The solution of the inequality system can be found by solving both inequalities and analyzing the solution intervals. The solution for \( 5x + 2 < 3x + 8 \) gives \( x < 3 \), and solving \( \frac{x^2}{x-1} < 4 \) gives the final solution set as \( (-\infty, 1) \cup (2, 3) \cup (5, \infty) \).