Question

Which of the following is incorrect? (Assuming H refers to Entropy in Information Theory)

• A. \( H(y/x) = H(x,y) - H(x) \)
• B. \( H(x,y) = H(x/y) - H(y) \)
• C. \( I(x,y) = H(x) - H(y/x) \)
• D. \( I(x,y) = H(y) - H(y/x) \)

Hint:

Chain rule for entropy: \(H(X,Y) = H(X) + H(Y|X) = H(Y) + H(X|Y)\).
Mutual Information: \(I(X;Y) = H(X) - H(X|Y)\) \(I(X;Y) = H(Y) - H(Y|X)\) \(I(X;Y) = H(X) + H(Y) - H(X,Y)\)
Conditional entropy \(H(Y|X) \neq H(X|Y)\) in general.

Answer 1

The Correct Option is B

Standard identities in Information Theory for entropy (H) and mutual information (I):

• Joint Entropy: \(H(X,Y)\) 
• Conditional Entropy:
  • \(H(Y|X) = H(X,Y) - H(X)\)
  • \(H(X|Y) = H(X,Y) - H(Y)\)
• Mutual Information:
  • \(I(X;Y) = H(X) - H(X|Y)\)
  • \(I(X;Y) = H(Y) - H(Y|X)\)
  • \(I(X;Y) = H(X) + H(Y) - H(X,Y)\)
  • \(I(X;Y) = I(Y;X)\) (Symmetric)

Let's check the given options (using \(H(Y|X)\) for \(H(y/x)\)):

• (a) \(H(Y|X) = H(X,Y) - H(X)\): This is CORRECT by the definition of conditional entropy.
• (b) \(H(X,Y) = H(X|Y) - H(Y)\): From the definition \(H(X|Y) = H(X,Y) - H(Y)\), we get \(H(X,Y) = H(X|Y) + H(Y)\). This option states \(H(X,Y) = H(X|Y) - H(Y)\), which is INCORRECT.
• (c) \(I(X;Y) = H(X) - H(Y|X)\): The standard formula is \(I(X;Y) = H(X) - H(X|Y)\) or \(I(X;Y) = H(Y) - H(Y|X)\). This option is INCORRECT because it mistakenly uses \(H(Y|X)\) instead of \(H(X|Y)\).
• (d) \(I(X;Y) = H(Y) - H(Y|X)\): This is a CORRECT standard formula for mutual information.

Analysis of Options:

The question asks "Which of the following is incorrect?". Both (b) and (c) are incorrect, but the question seems to indicate only one should be definitively incorrect.

• Option (b): \(H(X,Y) = H(X|Y) - H(Y)\). This is incorrect because the correct formula is \(H(X,Y) = H(X|Y) + H(Y)\). This involves a direct sign error in the fundamental chain rule identity.
• Option (c): \(I(X;Y) = H(X) - H(Y|X)\). This is also incorrect because the correct formula is \(I(X;Y) = H(X) - H(X|Y)\) or \(I(X;Y) = H(Y) - H(Y|X)\), not \(H(Y|X)\). The formula is mixed up with incorrect terms.

Since exam questions typically expect a single incorrect answer, option (c) seems to be the one that the question is targeting as incorrect. It misuses the terms for mutual information and conditional entropy.

Final Answer:

Option (c): \(I(X;Y) = H(X) - H(Y|X)\)