Entropy inequalities - 趣味の研究

Definition.

Let $h_p$ be Renyi entropy,

$h_p=\frac{1}{1-p}\log(\int_{\mathrm{R^d}}dx f(x)^p)$ for probability density function $f(x)$

$h=h_1$ is Shannon entropy.

$h_{\infty}=-\log\|f\|_{\infty}$ and $\|f\|_{\infty}=\mathrm{ess} \sup f(x)$ .

and

$N_p=\exp(\frac{2h_p}{d})$ .

We derive some properties by assuming

$\det(K_X)\|f\|_{\infty}^2\leq C$ .

where $K_x$ is covariant matrix and $C$ is constant.

Theorem 1 ( joint entropy inequatliy)

Let $X$ be probability variables in $\mathrm{R^d}$ and $p\geq 1$ .

Let $\|f_k\|_{\infty}^2\det(K_{X})\leq C$ and $\lambda_l$ be eigen values of $K_X$ .

If $\lambda_l\geq 1$ for all $l$ ,

$h_p(X_1,X_2, \cdot\cdot\cdot, X_d)\geq\frac{1}{d}\sum_l h_p(X_l) -\frac{1}{2}\log(2\pi eC)$

This inequaity is the extension of the inequality in discrete case

$h_p(X_1,X_2, \cdot\cdot\cdot, X_d)\geq\frac{1}{d}\sum_l h_p(X_l)$ .

Theorem 2 (d-dimentinal reverse EPI)

Let $X_k$ be uncorrelated probability variables in $\mathrm{R^d}$ .

Let $\|f_k\|_{\infty}^2\det(K_{X_k})\leq C$ for all k, and let

$\lambda_{k;l}$ be eigen values of $K_{X_k}$ .

If d = 1,

$\sum_{k=1}^n {N_{p}(X_k)}\geq \frac{1}{2\pi eC}N_p(\sum_{k=1}^n X_k)$

for $p\geq 1$ .

If $\lambda_{l,k}\geq 1$ for all $l,k$ and $d\geq 2$ ,

$\sum_{k=1}^n {N_{p}(X_k)}^d\geq \frac{1}{2\pi eC}N_p(\sum_{k=1}^n X_k)$

for $p\geq 1$ .

These are the reverse EPI.

Theorem 3. (Renyi EPI for order p < 1)

Let $X_k$ be independent probability variables in $\mathrm{R^d}$ .

Let $\|f_k\|_{\infty}^2\det(K_{X_k})\leq C$ for all k.

where, $K_X$ is covariant matrix.

Then,

for $\frac{d}{d+2}\leq p$ < $1$ ,

$N_p(\sum_{k=1}^n X_k)\geq \frac{A_{p,d}}{eC^{\frac{1}{d}}}\sum_{k=1}^n N_{p}(X_k)$

This inequality is the extension of Renyi EPI for p<1.

where

$A_{p,d}=[{(\Gamma(\frac{1}{1-p}-\frac{d}{2})^{\frac{p}{1-p}}\Gamma(\frac{p}{1-p}) )}^{\frac{2}{d}}\pi]/[{(\Gamma(\frac{1}{1-p})^{\frac{p}{1-p}}\Gamma(\frac{p}{1-p}-\frac{d}{2}) )}^{\frac{2}{d}}{(\beta(1-p) )}]$

$\beta=\frac{1}{2p-d(1-p)}$

Proposition 1.

$h_p\geq -\log(\|f\|_{\infty})$

$N_p(X)\geq N_{\infty}(X)$

This inequality holds either discrete case or continuous case.

We easily show these results by using $f\leq\|f\|_{\infty}$ .

Proposition 2.(entropy upper bound)

For discrete or continuous d-dimentional probability variable $X$ ,

$\exp(2h_1(X) )\leq {(2\pi e)}^d\det(K_X)$ .

Lemma 1.

For continuous probability variable,

$h_p\leq h_q$ if $q\leq p$ .

Proof of Theorem 1.

Using Proposition2 and Lemma1,

$\sum_l \exp(2h_p(X_l) )\leq \sum_l\exp(2h_1(X_l) )\leq 2\pi e \sum_l K_{X,ll}=2\pi e \mathrm{Tr}(K_X)=2\pi e\sum_l \lambda_l$ eq(1)

By the asuumption $\lambda_l\geq 1$ and $\|f_k\|_{\infty}^2\det(K_{X})\leq C$ ,

$\sum_l \lambda_l\leq d\Pi_l\lambda_l = d\det(K_X)\leq dC\|f\|_{\infty}^{-2}$ eq(2)

By combining Proposition 1., eq(1) and eq(2),

$\sum_l \exp(2h_p(X_l) )\leq 2\pi eCd\exp(2h_p(X_1,X_2, \cdot\cdot\cdot, X_d) )$

Using Jensen's Inequality,

$\frac{1}{d}\sum_l \exp(2h_p(X_l) )\geq \exp(\sum_l \frac{2h_p(X_l)}{d})$ .

We derive

$\frac{1}{d}\sum_l h_p(X_l)\leq\frac{1}{2}\log(2\pi eC)+h_p(X_1,X_2, \cdot\cdot\cdot, X_d)$

Proof of Theorem 2.

For $Y=\sum_k X_k$ , using Proposition2.,

$N_p(Y)\leq N_1(Y)\leq 2\pi e\det(K_Y)^{\frac{1}{d}}$

Using $\frac{1}{d}\mathrm{Tr}A\geq {(\det A)}^\frac{1}{d}$ , and uncorrelated condition $\mathrm{Tr}K_Y=\sum_k \mathrm{Tr}K_{X_k}$ ,

we derive

$2\pi e\det(K_Y)^{\frac{1}{d}}\leq \frac{2\pi e}{d}\mathrm{Tr}K_Y=\frac{2\pi e}{d}\sum_k\mathrm{Tr}K_{X_k}=\frac{2\pi e}{d}\sum_k\sum_l\lambda_{k;l}$

By assumption $\lambda_{k:l}\geq 1$ ,

$\sum_l\lambda_{k;l}\leq d\Pi_l\lambda_{k;l}=d\det(K_{X_k})$ .

By combining Proposition 1. and assumption $\|f_k\|_{\infty}^2\det(K_{X_k})\leq C$ ,

$N_p(Y)\leq 2\pi e\sum_k\det(K_{X_k})\leq 2\pi eC\sum_kN_{\infty}(X_k)^d\leq 2\pi eC\sum_k N_p(X_k)^d$ .

Proof of Theorem 3.

Lemma 2.

For $\frac{d}{d+2}\leq p$ < $1$ ,

$N_p(X)\leq A_{p,d}^{-1}\det(K_X)$

We can derive this inequality from Proposition 1.3. in [4].

By Proposition 1.3. in [4],

$g(x)=B_p\int_{\mathrm{R^d}}d^dx\frac{1}{{(1+\beta(1-p)(\bf{x-\mu})^TK_x^{-1}(\bf{x-\mu }) )}^{\frac{1}{(1-p)}}}$ maximise Renyi entropy.

Using $\int_{\mathrm{R^d}}d^dx=\frac{4\pi^{\frac{d}{2}}}{\Gamma(\frac{d}{2})}\int_0^{\infty}dx x^{d-1}$ and the definition of beta function,

we derive

$\exp(h_{p,g})=\|g\|_p^{\frac{p}{1-p}}=\det(K_X)[{(\Gamma(\frac{1}{1-p})\Gamma(\frac{p}{1-p}-\frac{d}{2}) )}{(\beta(1-p) )}^{\frac{d}{2}}]/[{(\Gamma(\frac{1}{1-p}-\frac{d}{2})\Gamma(\frac{p}{1-p}) )}\pi{^\frac{d}{2}}]=A_{p,d}^{-\frac{d}{2}}\det(K_X)$

where

$\beta=\frac{1}{2p-d(1-p)}$ .

For any probability variable $X$ with covariant matrix $K_X$ ,

$\exp(2h_{g,p})\geq \exp(2h_p(X) )=A_{p,d}^{-d}\det(K_X)$ .

Lemma 3.

Let Y be $Y=\sum_k X_k$ for independent probability variable $X_k$ .

$N_{\infty}(Y) \geq\frac{1}{e}\sum_{k=1}^n N_{\infty}(X_k)$

This Lemma is shown as Theorem 2.14. in [1].

By combining the assumption and Lemma 2.,

$N_p(X_k)\leq A_{p,d}^{-1}\det(K_{X_k})^{\frac{1}{d}} \leq A_{p,d}^{-1}C^{\frac{1}{d}}N_{\infty}(X_k)$ .

By combining Lemma 3. and Proposition 1,

$\sum_k N_p(X_k) \leq A_{p,d}^{-1}C^{\frac{1}{d}}\sum_k N_{\infty}(X_k) \leq A_{p,d}^{-1}C^{\frac{1}{d}}eN_{\infty}(Y)\leq A_{p,d}^{-1}C^{\frac{1}{d}}e N_p(Y)$ .

References.

[1] A.Marsiglietti, V.Kostina2, P.Xu. "A lower bound on the differential entropy of log-concave random vectors with applications"

[2] E.Ram, I.Sason. "On Renyi Entropy Power Inequalities".

[3] A. Marsiglietti, V.Kostina."A lower bound on the differential entropy of log-concave random vectors with applications"

[4]O.Johnson , C.Vignat ."Some results concerning maximum Rényi entropy distributions"