Multi-layer Perceptron Pattern Classification

Artificial neural network can classify groups. Multi-layer perceptron (back propagation) is suitable and widely used to solve complex classification problems. The aim of the project was to "illustrate the learning behavior of back-propagation neural network for pattern classification". In particular, the task was to write a MATLAB program to implement the back propagation algorithm to classify two classes defined by the following conditional probability density functions.

Procedure

First, a training data was generated using MATLAB function randn and variances. Second, a feed forward network of one input layer (input themselves), two hidden layers, and one output layer was constructed. Then, the neural network was optimized by appropriately selecting the number of neurons on each layer, the learning rate, and the momentum rate by trial and error.
 

Implimentation Code

%Lab2

format long

% intialize variables
data_size = 11000;

% variance and mean vectors for classes
variance_class_1 = 1 ;
mean_vector_class_1 = [0;0] ;
variance_class_2 = 4 ;
mean_vector_class_2 = [2;0] ;

% random data x for clases 1 & 2
C1 = zeros(2,data_size);
C2 = zeros(2,data_size);

for k=1:data_size
C1(1,k) = mean_vector_class_1(1,1) ;
C1(2,k) = mean_vector_class_1(2,1) ;
C2(1,k) = mean_vector_class_2(1,1) ;
C2(2,k) = mean_vector_class_2(2,1) ;
end 
C1_x = C1 + sqrt(1)*randn(2, data_size) ;
C2_x = C2 + sqrt(2)*randn(2, data_size) ;
% Back Propagation Neural Network

net = newff([-10 10; -10 10], [8, 4, 1], {'tansig','tansig','purelin'});
net.trainParam.show = 50;        % Show results every 50 iterations
net.trainParam.epochs = 200;     % Max number of iterations
net.trainParam.goal = 1e-2;      % Error tolerance stopping criterion
net.trainParam.mu = 0.9;         % Momentum 
net.trainParam.lr = 0.01;        % Learning rate

% Train network '0.5' output for class C1 and '0.95' for class C2
for i=1:1000
targets_C1(i) = 0.05;
targets_C2(i) = 0.95;
end

targets = [targets_C1 targets_C2];
train_data = [C1_x(1,1:1000) C2_x(1,1:1000); C1_x(2,1:1000) C2_x(2,1:1000)];
net = train(net, train_data, targets); 
classification_boundary = 0.5;

% Accuracy calculation for test data
network_output_C1 = sim(net, [C1_x(1,1000:11000); C1_x(2,1000:11000)]);
network_output_C2 = sim(net, [C2_x(1,1000:11000); C2_x(2,1000:11000)]);

correct_C1 = 0;
correct_C2 = 0;
for i=1:10000
    if (network_output_C1(i) < classification_boundary ) 
        correct_C1 = correct_C1 + 1;
    end
end
for i=1:10000
   if (network_output_C2(i) > classification_boundary ) 
        correct_C2 = correct_C2 + 1;
    end
end
accuracy_for_C1 = (correct_C1 / 10000)* 100
accuracy_for_C2 = (correct_C2 / 10000)* 100

% The two nets are equally probable
accuracy_net = 0.5*accuracy_for_C1 + 0.5*accuracy_for_C2

Conclusion

The final network can be characterized as shown in the graph. The optimum learning rate was 0.01 and momentum rate was 0.5. The best average accuracy of classification for both classes was 78%.