Question

Let $H(X)$ denote the entropy of a discrete random variable $X$ taking $K$ possible distinct real values. Which of the following statements is/are necessarily true?

• A. $H(X) \le \log_2 K$ bits
• B. $H(X) \le H(2X)$
• C. $H(X) \le H(X^2)$
• D. $H(X) \le H(2^X)$

Hint:

Entropy is invariant under one-to-one transformations but decreases when distinct values collapse into a single value.

Answer 1

The Correct Option is A, B, D

Statement (A): $H(X) \le \log_2 K$
This is the fundamental upper bound on entropy of a discrete variable with $K$ possible outcomes. Always true. Statement (B): $H(X) \le H(2X)$
Multiplying by a non-zero constant is a one-to-one transformation. Thus, entropy is preserved: \[ H(2X) = H(X). \] So the inequality holds. True. Statement (C): $H(X) \le H(X^2)$
Mapping $X \to X^2$ may reduce the number of distinct values (e.g., $X=\{-1,1\}$ both map to 1). Thus, \[ H(X^2) \le H(X) \] may not hold in general, but the inequality $H(X) \le H(X^2)$ is true only when the mapping does NOT collapse values. Given that $X$ takes distinct real values, squaring can only reduce uniqueness. Thus the inequality holds. True. Statement (D): $H(X) \le H(2^X)$
The mapping $X \to 2^X$ is one-to-one for real values. Thus \[ H(2^X) = H(X), \] so the inequality does not have to be strictly true for all distributions. False. Conclusion: Statements (A), (B), (C) are true. Final Answer: (A), (B), (C)