According to Beer-Lambert's law, the absorbance $A$ is proportional to the concentration $C$:
$A = \log_{10}\left(\dfrac{1}{T}\right) = \varepsilon C l$,
where $T$ is the transmittance, $\varepsilon$ is the molar absorptivity, and $l$ is the path length.
If the concentration is doubled, i.e., $C' = 2C$, then:
$A' = \varepsilon \cdot 2C \cdot l = 2A$
Now from Beer’s law again:
$A' = \log_{10}\left(\dfrac{1}{T'}\right) = 2 \log_{10}\left(\dfrac{1}{T}\right)$
So,
$\log_{10}\left(\dfrac{1}{T'}\right) = \log_{10}\left(\dfrac{1}{T^2}\right)$
$⇒ T' = T^2$
Hence, the new transmittance is $T^2$.