Beirlant, J., Dudewicz, E., Gyorfi, L. and van der Meulen, E.: Nonparametric entropy estimation: An overview. Int. J. of Math. and Stat. Sci., 6. pp.17-39. 1997.  @ NUS  Home/Search   Document Not in Database   Summary   Related Articles   Check

....in statistical pattern recognition, adaptive vector quantization, image registration and indexing, and other areas. Non parametric estimation of Shannon entropy has been of interest to many in non parametric statistics, pattern recognition, model identification, image registration and other areas [18, 27, 1, 44, 4, 45, 11]. Estimation of entropy arises as a step towards Shannon entropy estimation, e.g. Mokkadem [32] constructed a non parametric estimate of the Shannon entropy from a convergent sequence of entropy estimates. However, estimation of the entropy is of interest in its own right. The problem ....

....multiple modalities via the Jensen difference [31, 30, 19] The most natural estimation method is to substitute a non parametric density estimator f into the expression for entropy. This method has been widely applied to estimation of the Shannon entropy and is called plug in estimation in [4]. Other methods of Shannon entropy estimation discussed in [4] include sample spacing estimators, restricted to d ##, and estimates based on nearest neighbor distances. Three general classes of methods can be identified: parametric estimators, non parametric estimators based on density or ....

[Article contains additional citation context not shown here]

J. Beirlant, E. J. Dudewicz, L. Gyorfi, and E. van der Meulen, "Nonparametric entropy estimation: an overview," Intern. J. Math. Stat. Sci., vol. 6, no. 1, pp. 17--39, june 1997. 24

....RATIO (ER) METHOD To measure the amount of disorder , based upon the probability density function (pdf) p(x) of data, the differential entropy is used: H(x) Gamma R 1 Gamma1 p(x) ln p(x)dx. Particularly convenient is the Kozachenko Leonenko (K L) estimate of the differential entropy [4] H(x) j=1 ln (N ae j ) ln 2 CE (1) owing to its flexibility with respect to the dimensionality of the data set. In Eq. 1, N is the number of samples in the data set, ae j is the Euclidean distance of the j th delay vector to its nearest neighbour, and CE ( 0:5772) is the Euler constant. ....

J. Beirlant, E.J. Dudewicz, L. Gyorfi, and E.C. van der Meulen, "Nonparametric entropy estimation: An overview," International J. Mathematical and Statistical Sciences, vol. 6, pp. 17--39, 1997.

....Mutual Information In estimating mutual information, we wish to avoid strong prior modelling assumptions, i.e. jointly Gaussian measurements. There has been considerable research into useful nonparametric methods for estimating information theoretic quantities; for an overview, see e.g. [4]. Kernel density estimation methods are often used as an appealing alternative when no prior knowledge of the distribution is available. Similarly, these kernel based methods can be used to estimate mutual information e#ectively. Using estimates with smooth, di#erentiable kernel shapes will also ....

.... max # # (13) Because our variables of interest are continuous, it is convenient to write the mutual information in terms of joint and marginal entropy, as: i ) H(f i ) H(f H(f i ) 14) There are a number of possible kernel based estimates of entropy available [4]. In practice we use either a leave one out resubstitution estimate: HRS (x) 15) or an integrated squared error estimate from [6] H ISE =H(1) 1 p(x) dx (16) where 1 is the uniform density on a fixed range, and p(x) K # (x x j ) These methods ....

J. Beirlant, E. J. Dudewicz, L. Gyorfi, and E. C. van der Meulen. Nonparametric entropy estimation: An overview. International Journal of Math. Stat. Sci., 6(1):17-- 39, June 1997.

....distributions, mutual information cannot be computed analytically. Moreover, the pdf s themselves are rarely available. In this section, we develop numerical methods to estimate mutual information based on available wavelet coe#cient data. This problem is at least as di#cult as pdf estimation; see [42, 43] for estimation of entropy in a general, theoretical context. 5.1 Nonparametric estimators Given two random vectors X and Y with known joint pdf p(x, y) I(X; Y ) is defined by the integral (1) We let X be the wavelet coe#cient, and Y be the neighborhood statistic T = f(NX) the parent PX, or ....

J. Beirlant, E. J. Dudewica, L. Gyofi, and E. van der Meulen, "Non--parametric entropy estimation: An overview," Int. J. Math. Stat. Sci., vol. 6, no. 1, pp. 17--39, 1997.

.... hn (X)j jh(Y ) 0 hn (Y )j jh(X;Y ) 0 hn (X;Y )j so that the asymptotic unbiasedness, consistency, or the order of consistency of entropy estimates imply analogous properties of In (X;Y ) There is extensive literature dealing with entropy estimates, overviewed recently by Beirlant et al. [2]. Any of the entropy estimates applicable to multidimensional observations can be used for information estimation in the sense explained above. Of all these estimators, the most systematically studied seem to be the histograms associated with products Pn 2Q n of partitions of the marginal ....

....cells of a constant dominating probability rather than of a constant Lebesgue volume. An even more inspiring method (applicable, however, only to one dimensional spaces) is the finite partitioning by sample quantiles, see [3] the sample 1=n quantiles are equivalent to the spacings discussed in [2], see also [16] The random grouping of data into mn cells specified by the j=mn quantiles of empirical distribution, 1 j mn 0 1,is an example of adaptive partitioning, leading moreover to the uniform distribution of observations into cells (the number of observations in a cell differs by at ....

J. Beirlant, E. J. Dudewicz, L. Gy orfi, and E. C. van der Meulen, "Nonparametric entropy estimation: An overview," Int. J. Math. Stat. Sci., vol. 6, no. 1, pp. 17--39, 1997.

.... that the density has compact support, and is bounded away from zero on its support, see Hall [13] van Es [21] For a thorough review of the issues in entropy estimation, in particular also (the lack of) the reasonability of various assumptions, see Beirlant, Dudewicz, Gyorfi, and van der Meulen [3]. The principal tool for the proof of (1.3) is provided by submartingales and submartingale inequalities, see Breiman [4] This comes about as follows. We let G denote the cumulative distribution function corresponding to the probability density function (pdf) g, and let Gn denote the empirical ....

....this is not o(n 01=2 ) So we do get consistent estimators, but not asymptotic normality. The same applies to the estimator of Ahmad and Lin [1] ACKNOWLEDGMENT The authors wish to thank L. Gy orfi for providing us with a preprint of the review paper Beirlant, Dudewicz, Gy orfi, van der Meulen [3]. ....

J. Beirlant, E. Dudewicz, L. Gy orfi, and E. G. van der Meulen, "Nonparametric entropy estimation: An overview," Int. J Math Stat Sci., vol. 6, pp. 17--39, 1997.

....Estimation of Mutual Information For most distributions, mutual information cannot be computed analytically. In this section, we develop numerical methods to estimate mutual information based on available wavelet coe#cient data. This problem is at least as di#cult as pdf estimation; see [35, 36] for estimation of entropy in a general, theoretical context. 5.1 Consistent Estimators Given two random vectors X and Y with known joint pdf p(x, y) I(X; Y ) is defined by the integral (1) We let X be the wavelet coe#cient, and Y be the neighborhood statistic T = f(NX) the parent PX, or ....

J. Beirlant, E. J. Dudewica, L. Gyofi, and E. van der Meulen, "Non--parametric entropy estimation: An overview," Int. J. Math. Stat. Sci., vol. 6, no. 1, pp. 17--39, 1997.

....of entropy which completely bypasses the di cult intermediate step of multivariate density estimation required by previous estimators. The problem of entropy estimation has long been of interest to the engineering, physics, and statistics communities, e.g. see the recent paper by Beirlant etal [11] for a thorough overview of the topic of Shannon entropy estimation. The general entropy estimation problem is relevant to pattern analysis, process complexity assessment, model identi cation, tests of distributions, and other applications where invariance to scale, translation and other ....

J. Beirlant, E. J. Dudewica, L. Gyor, and E. van der Meulen, \Nonparametric entropy estimation: an overview," Intern. J. Math. Stat. Sci., vol. 6, no. 1, pp. 17-39, 1997.

....in order to decide whether f is equal to f o . Idivergence estimation also arises in image registration where the I divergence can be directly related to mutual information between two images f and f o [8] For an overview of entropy and I divergence estimation applications the reader can refer to [2] and [1] In this paper we present a methodology robust estimation of I (f; f o ) for unknown f and arbitrary dominating density f o . This methodology performs a nonlinear transformation on the data sample Xn , producing a transformed data sample Yn , and constructs a graph, called the ....

J. Beirlant, E. J. Dudewicz, L. Gyorfi, and E. van der Meulen, "Nonparametric entropy estimation: an overview," Intern. J. Math. Stat. Sci., vol. 6, no. 1, pp. 17--39, 1997.

....is not consistent. For example, in estimating the di erential entropy one can have a fast estimate based on one spacing, which is a density estimate such that the pointwise variance tends to in nity, causing a universal additive bias for the corresponding entropy estimate (cf. Beirlant et al. [4]) Similarly, here the estimate would be L n = 1 (Y i;n Y i ) Y i;n being the response variable that corresponds to X i;n . One may expect some fast rate of convergence since 1 i 1 i;n and Y i;n Y i Em(X) 7 The problem, however, is that one data ....

Beirlant, J., Dudewicz, E. J., Gyor , L. and van der Meulen, E. C. (1997). Nonparametric entropy estimation: an overview. International J. Math. Stat. Sci., 6, pp. 17-39.

No context found.

Beirlant, J., Dudewicz, E., Gyorfi, L. and van der Meulen, E.: Nonparametric entropy estimation: An overview. Int. J. of Math. and Stat. Sci., 6. pp.17-39. 1997.

No context found.

J. Beirlant, E. J. Dudewicz, L. Gyorfi, and E. C. van der Meulen. Nonparametric entropy estimation: An overview. International Journal of Math. Stat. Sci., 6(1):17--39, June 1997.

No context found.

J. Beirlant, E. J. Dudewicz, L. Gy orfi, and E. C. van der Meulen, "Nonparametric entropy estimation: An overview," International Journal of Math. Stat. Sci., vol. 6, no. 1, pp. 17--39, June 1997.

No context found.

J.Beirlant,E.J.Dudewicz,L.Gyrfi,E.C.vander Meulen, "Nonparametric Entropy Estimation: An Overview," Intern. J. Math. Stat. Sci., vol. 6, no. 1, pp. 17-39, June 1997.

No context found.

J. Beirlant, E. Dudewicz, L. Gyor, and E. van der Meulen, "Nonparametric entropy estimation: An overview," International Journal of Mathematical and Statistical Sciences, vol. 6, no. 1, pp. 17--39, 1997.

Online articles have much greater impact   More about CiteSeer.IST at NUS   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST at NUS - Copyright Penn State and NEC. Hosted by the School of Computing, National University of Singapore.