Normalized entropy: a comparison with traditional techniques in variable selection

Abstract

A variable selection procedure in regression analysis using a normalized entropy measure was firstly proposed in 1996, by Amos Golan, George Judge and Douglas Miller, in the book Maximum Entropy Econometrics - Robust Estimation with Limited Data. To the best of the authors' knowledge, the idea has not been explored in the literature since then, despite many noteworthy advantages that have been pointed out by Amos Golan and coauthors, such as: it is simple to perform, even for a large number of variables (useful in some big data problems); it allows the use of non-sample information (easily incorporated in the optimization structure); and it can be implemented for ill-posed models (frequently observed in real-world problems). Following a recent work that illustrates how normalized entropy can represent a promising approach to identify pure noise models, this paper revises the procedure of normalized entropy, proposes some improvements, and illustrates its performance when compared with some well-known traditional techniques in variable selection problems.