Actions

dANN

Backprop Feedforward Neural Network

From Syncleus Wiki

dANN Information
General
Description An Artificial Intelligence Library written in Java.
Last Activity Today
License OSCL Type C
Homepage dANN
Download
Distributions 2.1
Documentation Documentation Snapshot
Development http://gerrit.syncleus.com/dANN
JIRA Bug Tracking
Sonar Technical Debt
Support
IRC Room #dANN on irc.freenode.org
Mailing Lists Directory of Mailing Lists

Feedforward Backprop networks are covered in the com.syncleus.dann.neural.backprop.brain package. There are three classes in particular we are interested in, each is a more specific extension of the previous one: AbstractFeedforwardBrain, AbstractFullyConnectedFeedforwardBrain, and FullyConnectedFeedforwardBrain.

AbstractFeedforwardBrain will handle initializing a network in groups of layers. The actual creation of the neurons are left to child classes through the "createNeuron" abstract method. Any child class is also expected to handle how the layers connect to each other.

AbstractFullyConnectedFeedforwardBrain extends AbstractFeedforwardBrain further by also connecting the layers of the network such that each layer connects all of its neurons to every neuron in the next level. Child classes are still expected to initialize the neurons by implementing the "createNeuron" method.

FullyConnectedFeedforwardBrain is a regular (non-abstract) class that extends AbstractFullyConnectedFeedforwardBrain and implements the "createNeuron" method such that all neurons are created with the same set of parameters. Typically unless you need a network with a unique architecture you'll simply use this class.

Example: XOR

We want to start off with a simple example where a FullyConnectedFeedforwardBrain is used to create a network and then teach it to learn to mimic an XOR logic gate. An XOR gate is pretty straight forward itself. It takes 2 or more inputs; if one and only one input is on (in this case +1.0) then the output will also be on, otherwise the output is off (-1.0 in this case). While a 2 input XOR is the simplest form we will try to do a 3 input XOR to make it a bit more interesting.

The first step is to create the brain and tell it some properties it will use to construct its network:

final double learningRate = 0.0175;
final ActivationFunction activationFunction = new SineActivationFunction();
final int[] topology = {3, 3, 1};
 
FullyConnectedFeedforwardBrain brain =
     new FullyConnectedFeedforwardBrain(topology,
                                        learningRate,
                                        activationFunction);

The first parameter, topology, tells the brain how many neurons will be in each layer. In this example there are 3 neurons in the input layer, 3 in a hidden layer, and 1 in the output layer. The next parameter, learningRate, is applied to all the neurons in the network. This parameter should be between 1.0 (exclusive) and 0.0 (exclusive). We wont go into the details here about choosing an appropriate learning rate. Finally the last parameter is the activation function. Like the learning rate the choice of activation function depends on the problem your solving and often is a matter of trial and error.

Next we want to get a copy of the input and output neurons used by the brain:

final ArrayList<InputNeuron> inputs =
     new ArrayList<InputNeuron>(brain.getInputNeurons());
InputNeuron inputA = inputs.get(0);
InputNeuron inputB = inputs.get(1);
InputNeuron inputC = inputs.get(2);
 
final ArrayList<OutputNeuron> outputs =
     new ArrayList<OutputNeuron>(brain.getOutputNeurons());
OutputBackpropNeuron output = (OutputBackpropNeuron) outputs.get(0);

Now we need to train the network several times so it can learn what we expect of it. We will want to train it as an XOR:

for(int lcv = 0;lcv < 250; lcv++)
{
	inputA.setInput(-1.0);
	inputB.setInput(-1.0);
	inputC.setInput(-1.0);
	brain.propagate();
	output.setDesired(-1.0);
	brain.backPropagate();
 
	inputA.setInput(1.0);
	inputB.setInput(-1.0);
	inputC.setInput(-1.0);
	brain.propagate();
	output.setDesired(1.0);
	brain.backPropagate();
 
	inputA.setInput(-1.0);
	inputB.setInput(1.0);
	inputC.setInput(-1.0);
	brain.propagate();
	output.setDesired(1.0);
	brain.backPropagate();
 
	inputA.setInput(-1.0);
	inputB.setInput(-1.0);
	inputC.setInput(1.0);
	brain.propagate();
	output.setDesired(1.0);
	brain.backPropagate();
 
	inputA.setInput(1.0);
	inputB.setInput(1.0);
	inputC.setInput(-1.0);
	brain.propagate();
	output.setDesired(-1.0);
	brain.backPropagate();
 
	inputA.setInput(1.0);
	inputB.setInput(-1.0);
	inputC.setInput(1.0);
	brain.propagate();
	output.setDesired(-1.0);
	brain.backPropagate();
 
	inputA.setInput(-1.0);
	inputB.setInput(1.0);
	inputC.setInput(1.0);
	brain.propagate();
	output.setDesired(-1.0);
	brain.backPropagate();
 
	inputA.setInput(1.0);
	inputB.setInput(1.0);
	inputC.setInput(1.0);
	brain.propagate();
	output.setDesired(-1.0);
	brain.backPropagate();
}

With the network fully trained now the only thing left to do is check to make sure it accurately predicts the output according to the pattern we trained it for:

inputA.setInput(-1.0);
inputB.setInput(-1.0);
inputC.setInput(-1.0);
brain.propagate();
assert (output.getOutput() < 0.0)
 
inputA.setInput(1.0);
inputB.setInput(1.0);
inputC.setInput(1.0);
brain.propagate();
assert (output.getOutput() < 0.0)
 
inputA.setInput(-1.0);
inputB.setInput(1.0);
inputC.setInput(-1.0);
brain.propagate();
assert (output.getOutput() > 0.0)

You'll notice we are only checking if the output is less than or greater than 0.0. This is because it is natural for the network to output numbers like 0.99872 instead of 1. This is because the output of the network represents a sort of certainty of its answer. The closer the number is to 1 or -1 the more certain the network is of its answer. Any output around 0.0 indicated the network isn't sure what the answer is. Keep in mind this only applies to certain Activation Functions like Sine. Others might limit the output to a range of 0.0 and 1.0.

See Also