Neural Network as a Tool for Feature Selection

It has been widely know that neural networks can serve as a powerful tool for pattern classification, especially when the distribution of the objective classes is unknown or can't be expressed as mathematical models.  There are also studies that have shown that neural networks can be used as a tool for feature extraction, i.e., to produce new features based on the original features or the inputs to a neural network.  The set of new features usually contains fewer and more informative features so that future classification can be conducted at a lower computational cost using only the condensed new features. 

In this web page, the phrase Feature Selection is used in the sense that no new features are produced.  Instead, the only purpose is to find, among the original inputs, which ones are the most useful for a particular case of pattern classification.  In other words, which inputs can be left out of the classification process without causing dramatic decrease in classification accuracies. 

For example, a synthetic data set can be created like this:  Numerical characters from 0 to 9 are represented by 35 units in a 5 by 7 (row by column) array as in the following image:

If random Gaussian-distributed noise is added, the characters may look like these:

There are 400 files containing such noise-added 25 by 14 (row by column) arrays that were generated and are available for download (binary byte files in zip format, 173K).  If the numbers from the 35 units in the array are used as an input to one of 35 input nodes, a three-layer (one hidden) feed-forward network without direction connection between the input and output nodes can be trained to classified such noise-added array to achieve accuracies close to 100%. 

However, from the images above it is obvious that some of the units in the 3 by 5 array are useless for the classification.  For example, the three units in the middle of the the second, third, fifth and sixth row do not provide any more information than the variation of the Gaussian noise.  Neither do the units in the lower right corner of the array and the right-most one in the middle row.  If the input from these units are not used, the classification accuracy should not be affected adversely.  To go one step further, because there are correlations among the remaining 21 units, if some of them are left out, the classification accuracy should not suffer drastically.

For feature selection, a new method was developed to interpret the weights of a trained multi-layer feed-forward network.  Inputs are left out one after another.  With the synthetic data in this example, it was shown that the useless inputs were always deleted before any useful ones were.  Because the weights of a network are initialized randomly, the process was run 16 times.  The following will show the order in which the 22 inputs were reduced to 3.  The 22 inputs included all the useful units, and the one in the lower right corner of the 5 by 7 array, just to show the useless one always got deleted first. 

In the figure below, from left to right lies the 16 processes.  The 5 by 7 array in the images above is rotated 90 degrees clockwise for easier arrangement on this page.  There are 16 such arrays in the first row.  A "-" denotes a unit that has been deleted from the array.  At the beginning, the rotated array is the same for all the processes.  The first row in the figure shows that it was always the useless unit, now at the lower left corner of the array after rotation, that got deleted.  As a result, there are 21 "+" symbols representing 21 units remaining in the array.  Each array in the second row has 20 unit left, 19 in the third, and so on.

+++++++ +++++++ +++++++ +++++++ +++++++ +++++++ +++++++ +++++++ +++++++ +++++++ +++++++ +++++++ +++++++ +++++++ +++++++ +++++++
+--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+
+--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+
+--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+
-++-+++ -++-+++ -++-+++ -++-+++ -++-+++ -++-+++ -++-+++ -++-+++ -++-+++ -++-+++ -++-+++ -++-+++ -++-+++ -++-+++ -++-+++ -++-+++

+++++++ +++++++ +++++++ +++++++ -++++++ +++++++ +++++++ -++++++ +++++++ +++++++ +++++++ +++++++ +++++++ +++-+++ +++++++ +++++++
+--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+
+--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+
---+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+
-++-+++ -++-+++ -++-++- -++-++- -++-+++ -++-++- -++-++- -++-+++ -++-++- -++-+++ -++-++- -++-++- -++-++- -++-+++ -++-+++ -++-+++

+++++++ +++++++ -++++++ +++++++ -++++++ +++++++ -++++++ -++-+++ +++++++ +++++++ +++++++ -++++++ +++-+++ +++-+++ +++-+++ +++-+++
+--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+ ---+--+
+--+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+
---+--+ ---+--+ +--+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+
-++-++- -+--+++ -++-++- -++-++- -++-++- -++-++- -++-++- -++-+++ -++-++- -++-++- -++-++- -++-++- -++-++- -++-+++ -++-+++ -++-+++

+++++++ +++++++ -++++++ +++-+++ -++-+++ +++-+++ -++-+++ -++-+++ +++++++ +++-+++ ++++++- -++++++ -++-+++ +++-+++ +++-+++ +++-+++
+--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+ ---+--+
+--+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+
---+--+ ---+--+ ---+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+
-+--++- -+--++- -++-++- -++-++- -++-++- -++-++- -++-++- -+--+++ -+--++- -++-++- -++-++- -+--++- -++-++- -++-++- -++-++- -++-++-

+++-+++ +++++++ -++++++ +++-++- -++-+++ +++-+++ -++-++- -++-+++ ++++++- +++-+++ +++-++- -++-+++ -++-+++ +++-++- +++-+++ +++-+++
+--+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+ ---+--+
+--+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+
---+--+ ---+--+ ---+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+
-+--++- -+--++- -+--++- -++-++- -+--++- -+--++- -++-++- -+--++- -+--++- -+--++- -++-++- -+--++- -+--++- -++-++- -++-++- -+--++-

+++-++- ++++++- -++-+++ +++-++- -++-+++ +++-+++ -++-++- -++-++- +++-++- +++-++- +++-++- -++-+++ -++-++- +++-++- +++-++- +++-+++
+--+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+ ---+--+
+--+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+
---+--+ ---+--+ ---+--+ ---+--+ ---+--+ ---+--+ +--+--+ +--+--+ ---+--+ ---+--+ +--+--+ ---+--+ +--+--+ +--+--+ ---+--+ ---+--+
-+--++- -+--++- -+--++- -+--++- -+--++- -+--++- -+--++- -+--++- -+--++- -+--++- -+--++- -+--++- -+--++- -+--++- -++-++- -+--++-

-++-++- +++-++- -++-+++ -++-++- -++-+++ +++-++- -++-++- -++-++- +++-++- +++-++- +++-++- -++-++- -++-++- +++-++- +++-++- +++-++-
+--+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+ ---+--+
+--+--+ +--+--+ +-----+ +--+--+ +--+--+ ---+--+ +--+--+ +--+--+ ---+--+ ---+--+ ---+--+ +--+--+ +--+--+ +--+--+ ---+--+ +--+--+
---+--+ ---+--+ ---+--+ ---+--+ ------+ ---+--+ +--+--+ +--+--+ ---+--+ ---+--+ +--+--+ ---+--+ ---+--+ ---+--+ ---+--+ ---+--+
-+--++- -+--++- -+--++- -+--++- -+--++- -+--++- -+--++- -+--++- -+--++- -+--++- -+--+-- -+--++- -+--++- -+--++- -+--++- -+--++-

-++-++- +++-++- -++-++- -++-++- -++-++- +++-++- -++-++- -++-++- +++-++- +++-++- +++-++- -++-++- -++-++- +++-++- +++-++- +++-++-
+--+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ ---+--+ +--+--+ +--+--+ ---+--+ +--+--+ +--+--+ ---+--+ +--+--+ ---+--+
+--+--+ +--+--+ +-----+ +--+--+ +--+--+ ---+--+ +--+--+ +-----+ ------+ ---+--+ ---+--+ +--+--+ +--+--+ +--+--+ ---+--+ +-----+
---+--+ ---+--+ ---+--+ ---+--+ ------+ ---+--+ +--+--+ +--+--+ ---+--+ ------+ +--+--+ ---+--+ ------+ ------+ ---+--+ ---+--+
----++- ----++- -+--++- -+--+-- -+--++- -+--+-- -+---+- -+--++- -+--++- -+--++- -+--+-- ----++- -+--++- -+--++- ----++- -+--++-

-++-++- +++-++- -++-++- -++-++- -++-++- +++-++- -++-++- -++-++- +++-++- +++-++- +++-++- -++-++- -++-++- +++-++- +++-++- +++-++-
+--+--+ ---+--+ +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ ---+--+ +--+--+ +--+--+ ---+--+ +--+--+ +--+--+ ---+--+ ---+--+ ---+--+
+--+--+ +--+--+ +-----+ +--+--+ +--+--+ ---+--- +--+--+ +-----+ ------- ---+--- ------+ +--+--+ +--+--+ +--+--+ ---+--+ ------+
---+--+ ---+--+ ---+--- ---+--+ ------+ ---+--+ +--+--+ +--+--+ ---+--+ ------+ +--+--+ ---+--+ ------+ ------+ ---+--+ ---+--+
-----+- -----+- -+--++- ----+-- -+---+- -+--+-- -----+- ----++- -+--++- -+--++- -+--+-- -----+- ----++- -+---+- ----++- -+--++-

-++-++- +++-++- -++-++- -++-++- -++-++- +++-++- -++-++- -++-++- +++-++- +++-++- +++-++- -++-++- -++-++- +++-++- +++-++- +++-++-
+-----+ ---+--- +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ ---+--+ +--+--+ +--+--+ ---+--+ +--+--+ +--+--+ ---+--+ ---+--+ ---+--+
+--+--+ +--+--+ ------+ ---+--+ +--+--- ------- +--+--+ +------ ------- ---+--- ------+ +--+--+ +--+--+ ---+--+ ---+--+ -------
---+--+ ---+--+ ---+--- ---+--+ ------+ ---+--+ +--+--- +--+--+ ---+--+ ------+ +--+--+ ------+ ------+ ------+ ---+--+ ---+--+
-----+- -----+- -+--++- ----+-- -+---+- -+--+-- -----+- ----++- -+---+- ----++- ----+-- -----+- ----+-- -+---+- ----+-- -+--++-

-++-++- +++-++- -++-++- -++-++- -++-++- +++-++- -++-++- -++-++- +++-++- -++-++- +++-++- -++-++- -++-++- +++-++- +++-++- +++-++-
+-----+ ---+--- +--+--+ +--+--+ +--+--+ +--+--+ ---+--+ ---+--+ ---+--+ +--+--+ ---+--+ ---+--+ ---+--+ ---+--+ ------+ ---+--+
+--+--- ---+--+ ------+ ---+--+ +--+--- ------- ---+--+ +------ ------- ---+--- ------+ +--+--+ +--+--+ ---+--- ---+--+ -------
---+--+ ---+--+ ---+--- ------+ ------+ ---+--+ +--+--- ---+--+ ---+--+ ------+ +--+--- ------+ ------+ ------+ ---+--+ ---+--+
-----+- -----+- ----++- ----+-- -----+- ----+-- -----+- ----++- -+---+- ----++- ----+-- -----+- ----+-- -+---+- ----+-- ----++-

-++-++- +++-++- -++-+-- -++-++- -++-++- +++-++- -++-++- -++-++- +++-++- -++-++- +++-++- -++-++- -++-++- +++-++- +++-++- +++-++-
------+ ---+--- +--+--+ +--+--+ ---+--+ ---+--+ ---+--+ ---+--+ ---+--+ +--+--+ ---+--+ ---+--+ ---+--+ ---+--+ ------+ ---+--+
+--+--- ---+--+ ------+ ---+--- +--+--- ------- ---+--+ +------ ------- ---+--- ------+ +--+--+ +--+--- ---+--- ---+--+ -------
---+--+ ------+ ---+--- ------+ ------+ ---+--+ +------ ---+--+ ---+--+ ------+ ---+--- ------- ------+ ------+ ---+--- ---+--+
-----+- -----+- ----++- ----+-- -----+- ----+-- -----+- -----+- -----+- ----+-- ----+-- -----+- ----+-- -----+- ----+-- ----+--

--+-++- ++--++- -++-+-- --+-++- -++-++- +++--+- -++-+-- -++-++- +++-++- -++-+-- +++-++- --+-++- -++-+-- +++-++- +++-++- +++-+--
------+ ---+--- +--+--+ +--+--+ ---+--+ ---+--+ ---+--+ ------+ ---+--+ +--+--+ ------+ ---+--+ ---+--+ ---+--+ ------+ ---+--+
+--+--- ---+--+ ------+ ---+--- +------ ------- ---+--+ +------ ------- ---+--- ------+ +--+--+ +--+--- ---+--- ------+ -------
---+--+ ------+ ---+--- ------+ ------+ ---+--+ +------ ---+--+ ---+--- ------+ ---+--- ------- ------+ ------- ---+--- ---+--+
-----+- -----+- ----+-- ----+-- -----+- ----+-- -----+- -----+- -----+- ----+-- ----+-- -----+- ----+-- -----+- ----+-- ----+--

--+-++- ++---+- --+-+-- --+-+-- -++-++- ++---+- --+-+-- --+-++- +-+-++- --+-+-- ++--++- --+--+- -++-+-- +++-++- +-+-++- +++-+--
------+ ---+--- +--+--+ +--+--+ ---+--+ ---+--+ ---+--+ ------+ ---+--+ +--+--+ ------+ ---+--+ ---+--+ ---+--+ ------+ ------+
+--+--- ---+--+ ------+ ---+--- +------ ------- ---+--+ +------ ------- ---+--- ------+ +--+--+ +------ ------- ------+ -------
---+--- ------+ ---+--- ------+ ------- ---+--+ +------ ---+--+ ---+--- ------+ ---+--- ------- ------+ ------- ---+--- ---+--+
-----+- -----+- ----+-- ----+-- -----+- ----+-- -----+- -----+- -----+- ----+-- ----+-- -----+- ----+-- -----+- ----+-- ----+--

--+-++- ++---+- --+-+-- --+-+-- -++--+- ++---+- --+-+-- --+-++- +-+-+-- --+-+-- ++---+- --+--+- --+-+-- +-+-++- +-+-++- +++-+--
------+ ---+--- +--+--+ +-----+ ---+--+ ------+ ---+--+ ------+ ---+--+ +--+--+ ------+ ---+--+ ---+--+ ---+--+ ------+ ------+
+------ ---+--+ ------- ---+--- +------ ------- ------+ +------ ------- ------- ------+ +-----+ +------ ------- ------- -------
---+--- ------- ---+--- ------+ ------- ---+--+ +------ ---+--- ---+--- ------+ ---+--- ------- ------+ ------- ---+--- ---+---
-----+- -----+- ----+-- ----+-- -----+- ----+-- -----+- -----+- -----+- ----+-- ----+-- -----+- ----+-- -----+- ----+-- ----+--

--+--+- ++---+- --+-+-- --+-+-- --+--+- ++---+- --+-+-- --+--+- +-+-+-- --+-+-- ++---+- --+--+- --+-+-- +-+-+-- +-+-+-- ++--+--
------+ ---+--- +-----+ +------ ---+--+ ------+ ---+--+ ------+ ------+ +--+--+ ------+ ---+--+ ---+--+ ---+--+ ------+ ------+
+------ ------+ ------- ---+--- +------ ------- ------- +------ ------- ------- ------- +------ +------ ------- ------- -------
---+--- ------- ---+--- ------+ ------- ---+--- +------ ---+--- ---+--- ------- ---+--- ------- ------- ------- ---+--- ---+---
-----+- -----+- ----+-- ----+-- -----+- ----+-- -----+- -----+- -----+- ----+-- ----+-- -----+- ----+-- -----+- ----+-- ----+--

--+--+- ++---+- --+-+-- --+-+-- --+--+- ++---+- --+-+-- --+--+- +-+-+-- --+-+-- ++---+- --+--+- --+-+-- +-+-+-- --+-+-- ++--+--
------+ ---+--- ------+ ------- ---+--+ ------+ ---+--+ ------+ ------+ +--+--+ ------+ ---+--+ ---+--+ ---+--+ ------+ ------+
------- ------+ ------- ---+--- +------ ------- ------- ------- ------- ------- ------- ------- +------ ------- ------- -------
---+--- ------- ---+--- ------+ ------- ---+--- +------ ---+--- ---+--- ------- ---+--- ------- ------- ------- ---+--- ---+---
-----+- ------- ----+-- ----+-- ------- ------- ------- -----+- ------- ------- ------- -----+- ------- ------- ----+-- -------

--+--+- +----+- --+-+-- --+-+-- --+--+- +----+- --+-+-- --+--+- --+-+-- --+-+-- +----+- --+--+- --+-+-- --+-+-- --+-+-- +---+--
------+ ---+--- ------+ ------- ------+ ------+ ---+--+ ------+ ------+ +-----+ ------+ ---+--+ ------+ ---+--+ ------+ ------+
------- ------+ ------- ---+--- +------ ------- ------- ------- ------- ------- ------- ------- +------ ------- ------- -------
---+--- ------- ---+--- ------- ------- ---+--- ------- ---+--- ---+--- ------- ---+--- ------- ------- ------- ---+--- ---+---
------- ------- ------- ----+-- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -------

--+---- +----+- --+---- --+-+-- -----+- +----+- --+-+-- --+--+- --+-+-- --+-+-- +------ --+--+- ----+-- --+-+-- --+-+-- +------
------+ ------- ------+ ------- ------+ ------+ ---+--- ------+ ------+ ------+ ------+ ------+ ------+ ---+--- ------+ ------+
------- ------+ ------- ------- +------ ------- ------- ------- ------- ------- ------- ------- +------ ------- ------- -------
---+--- ------- ---+--- ------- ------- ------- ------- ------- ------- ------- ---+--- ------- ------- ------- ------- ---+---
------- ------- ------- ----+-- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -------

The following tables show, for each of the 16 processes above, how the classification accuracies changed as units were deleted one after another.  The columns in the tables correspond to the processes above in the same from left to right order.  In the first table, the numbers are the percentages of correct classification for the worst classified class among the 10 numerical characters.

95.9  95.2  94.4  95.0  95.4  95.4  95.5  95.3  95.8  95.8  95.8  96.2  95.7  95.7  95.2  95.6  95.5
95.5  95.8  95.5  95.2  95.5  95.7  95.5  94.9  95.8  95.5  95.6  95.3  95.3  95.2  95.6  95.4  95.5
95.1  94.9  94.5  95.4  94.9  95.2  95.5  94.6  95.3  94.8  94.4  95.6  95.1  95.4  95.1  95.6  95.1
95.5  93.9  94.4  95.0  94.6  94.7  94.9  95.4  95.1  95.2  94.4  95.1  95.0  94.9  95.2  95.6  94.9
93.7  93.7  94.3  95.2  94.9  94.4  94.3  93.1  93.7  94.9  95.6  93.0  95.1  94.7  94.3  94.5  94.3
92.9  93.4  93.0  94.6  92.8  93.7  93.6  93.8  93.4  93.1  94.7  93.1  93.3  94.6  94.2  93.5  93.6
93.4  93.3  93.1  92.9  93.8  92.5  92.7  93.1  93.8  93.2  92.8  93.1  92.7  93.9  94.3  92.7  93.2
92.7  93.3  90.7  93.2  91.1  92.0  92.7  92.8  92.9  92.2  92.3  93.2  93.1  93.0  93.1  92.7  92.6
90.1  89.8  92.0  91.2  91.9  92.6  91.7  91.6  90.8  92.4  91.5  90.1  91.1  92.2  89.9  91.0  91.2
89.2  89.3  88.8  89.9  91.9  91.2  89.0  88.7  90.2  91.3  90.7  90.2  89.2  90.6  88.9  90.2  90.0
89.2  89.4  88.6  88.0  90.3  88.7  87.1  88.4  89.3  88.9  89.6  88.4  86.5  89.2  88.9  88.7  88.7
88.1  87.8  87.5  87.4  87.8  87.7  88.0  87.1  87.3  89.1  87.1  87.8  88.1  87.5  87.4  87.6  87.7
85.1  87.3  81.0  88.1  85.2  85.3  85.9  85.5  86.1  86.5  85.9  85.5  86.1  85.6  85.0  85.9  85.6
78.4  82.5  80.8  80.1  82.9  80.5  79.5  80.8  83.6  79.2  81.1  77.5  78.9  83.4  80.0  78.7  80.5
79.0  77.7  70.2  71.6  79.5  76.5  69.9  76.4  78.9  69.9  78.0  75.7  75.8  81.4  77.0  77.2  75.9
76.0  75.3  71.9  69.4  74.4  71.2  69.1  75.9  70.4  68.1  73.8  72.3  66.4  77.9  73.9  74.7  72.5
68.8  73.3  67.8  69.4  70.8  70.4  67.0  71.9  71.0  63.9  70.0  71.3  67.6  67.9  69.4  71.2  69.5
58.8  66.4  60.6  61.6  65.2  67.2  64.7  57.6  64.9  60.9  65.4  61.0  63.6  68.0  60.0  69.7  63.5
30.3  31.6  26.6  26.3  33.9  21.0  17.9  26.7  23.4  37.5  24.1  27.4  40.4  20.7  27.2  27.4  27.7

The second table contains the percentage of correction classification averaged over the ten classes.

97.7  97.8  97.7  97.6  97.8  97.8  97.9  97.4  97.9  97.7  97.7  98.0  97.9  97.8  97.6  97.6  97.7
97.7  97.8  97.9  97.5  98.1  97.9  97.7  97.5  97.9  97.6  97.8  97.8  97.8  97.7  97.8  97.6  97.8
97.7  97.3  97.1  97.6  97.3  97.5  97.7  97.1  97.6  97.5  97.3  97.8  97.5  97.5  97.4  97.6  97.5
97.6  96.9  97.4  97.5  97.6  97.7  97.3  97.3  97.4  97.5  97.3  97.5  97.5  97.3  97.4  97.4  97.4
96.8  96.9  97.0  97.1  97.0  97.1  96.7  96.6  96.6  97.0  97.4  96.9  97.4  96.8  96.9  96.8  96.9
96.6  96.3  96.2  96.8  96.4  96.7  96.5  96.2  96.4  96.7  97.1  96.4  97.0  97.0  96.8  96.4  96.6
96.4  96.6  96.3  96.0  96.6  96.2  96.1  96.5  96.7  96.6  96.6  96.0  96.2  96.3  97.0  96.0  96.4
96.3  95.9  94.9  96.3  95.6  96.0  95.9  96.0  96.1  95.8  95.8  96.3  96.1  96.0  96.5  96.3  96.0
95.4  94.8  94.9  95.1  95.6  95.5  95.0  94.6  94.9  95.5  95.2  94.9  95.1  95.5  95.6  94.4  95.1
94.2  93.9  93.1  94.3  95.1  94.0  94.0  93.8  93.5  93.7  94.2  94.3  94.3  93.8  93.9  94.2  94.0
93.8  93.3  92.7  93.4  93.3  92.4  92.5  93.0  92.4  93.1  93.9  93.9  93.4  93.3  93.7  92.1  93.1
91.7  91.9  92.2  93.0  92.2  92.0  91.8  91.5  91.7  92.2  92.0  92.7  92.5  91.7  92.6  91.7  92.1
90.0  91.6  89.2  91.8  90.6  90.5  91.2  89.8  90.1  91.3  90.9  89.9  91.1  91.2  91.2  90.5  90.7
87.3  89.6  87.5  88.2  88.8  88.4  87.9  88.1  87.5  88.8  89.5  88.5  87.9  88.6  89.0  87.1  88.3
84.7  85.4  83.5  85.4  84.8  86.0  84.2  84.3  84.5  85.4  86.7  85.3  84.3  85.9  86.7  85.8  85.2
82.0  82.3  80.8  81.6  81.6  82.4  81.3  82.0  80.8  82.8  83.5  81.2  80.6  83.6  81.5  82.0  81.9
77.9  79.4  79.5  80.2  78.0  79.8  76.9  78.5  79.1  77.2  80.1  79.0  78.6  78.3  79.2  79.3  78.8
71.5  75.4  73.7  74.3  74.3  76.3  75.3  72.2  75.0  74.6  75.1  73.5  74.8  75.5  73.8  77.0  74.5
64.6  61.5  65.3  57.9  65.9  62.0  64.8  64.6  65.2  64.3  62.3  66.3  66.2  64.7  61.9  60.6  63.6

In both tables above, the percentages in the first row were obtained when 22 units were used.  The ones in the second row were from 21 units, and so on.  It can be seen that the removal of the first unit, which was the useless one in the lower right corner of the original 5 by 7 array, did not decrease the accuracies at all.  From the first table, it can be seen that at least 5 units are needed for all the classes to be correctly classified with an accuracy greater than 50%.  It's easy to see that if all the units comprising an important component of the array had been deleted at an early stage of the process, the accuracy for at least one class would have gone down prematurely.  For example, if the three units in the middle of the fourth row had been the first three to be deleted, the accuracy would have been below 50% starting from the fourth row in the first table, because there would have been no way to distinguish the characters 8 and 0.  The 4 units from the left in the first row are also indispensable for telling characters 1 and 7 apart.  During none of the 16 processes, an important component like these has been deleted prematurely.  While there were 6 units left, it was always the least important one that was removed.  The last 5 units were always strategically distributed to optimize the last possible chance of correctly classifying all characters.

Projects Main Page || Neural Network Main Page
Character Recognition || SPIE Challenge || Diabetes Forecast || Gene Recognition