The modulation index in amplitude modulation (AM) is limited to unity because if it exceeds 1, it leads to overmodulation and distortion of the signal in a standard envelope detector. This distortion makes the demodulated signal at the receiver unreliable, making it difficult to recover the intended signal.