Depth | Artificial neural network analysis

X1,x2,..., xN: input of neurons. These can be actually observed from the input layer or are a hidden layer of intermediate values ​​(hidden layer is a layer consisting of all the nodes between the input and output. Help the neural network to learn the complex relationship between data. We do not understand it does not matter, behind Speaking of multi-layer neural networks will be explained again.

X0: Offset unit. This is the input to which the constant value is added to the activation function (in a mathematical analogy y=ax+b is the constant b that makes the straight line not the origin). Intercept items, which usually have a +1 value.

W0,w1,w2,...,wN: The weight for each input. Even the bias unit is weighty.

a: output of neurons. Calculated as follows:

What needs to be explained is that f in the formula is a known activation function, f makes the neural network (single or even multi-layer) very flexible and has the ability to estimate complex nonlinear relationships. In the simple case it can be a Gaussian function, a logic function, a hyperbolic function or even a linear function. Using a neural network allows it to implement three basic functions: AND, OR, NOT.

Here's a basic f function, which is actually a step function we know well:

Example 1: The and function is implemented as follows:

Neuron output: a = f ( -1.5 + x1 + x2 )

The truth table is as follows. It can be seen that the column "a" is consistent with the column "X1 AND X2":

Example 2: The or function is implemented as follows:

The neuron output is:

a = f(-0.5 + x1 + x2 )

The truth table is as follows, where column "a" is consistent with "X1 OR X2":

We can see from Example 1 and Example 2 that we can implement the or function by simply changing the weight of the bias unit. If either one of x1 or x2 is positive, the total weight is positive.

Example 3: The not function is implemented as follows:

The neuron output is: a = f( 1 - 2 x1 )

The truth table is as follows:

The author hopes that through these examples we can intuitively understand how a neuron works in a neural network. Of course, there are multiple choices of functions in the neural network. In addition to the functions used in the above examples, here we introduce a commonly used function - sigmoid function, to help everyone understand. The function image looks like this:

It can be seen from the figure that the advantage of the sigmoid function is that the output range is limited, so the data is not easy to divergence during the transfer process, and its output range is (0, 1), and the probability can be represented at the output layer. Of course, it also has the corresponding disadvantage, that is, the gradient is too small when saturated. We can understand the following and examples.

In the example we saw earlier, the AND, OR, and non-functions are linearly separable. The perceptron only has the output layer neurons to perform the activation function processing, ie it has only one layer of functional neurons and the learning ability is very limited (this is before this article One of the key issues in the perceptron model mentioned). Therefore, it is necessary to use multi-layer neurons for nonlinear separability. The following section introduces the multi-layer neural network.

Neural networks are divided into three types of layers:

Input layer: The leftmost layer of the neural network through which the samples that need to be trained to observe are input.

Hidden layer: A layer consisting of all nodes between input and output. Helps neural networks learn complex relationships among data.

Output layer: The first layer of the neural network is obtained from the first two layers. In the case of 5 classifications, the output layer will have 5 neurons.

Why is a multi-layered network useful? Here we try to understand how neural networks use multi-layer to simulate complex relationships. To understand this further, we need to give an example of the same or a function. The truth table is as follows:

Here we can see that when the input is the same, the output is 1, and otherwise it is 0. This relationship cannot be completed by a single neuron and therefore requires a multi-layered network. The idea behind using multiple layers is that the complex relationship function can be broken down into simple functions and their combinations. Let us decompose the same or function:

X1 XNOR X2 = NOT (X1 XOR X2 )

= NOT [(A+B).(A'+B') ]

= (A+B)'+ (A'+B')'

=(A'.B') + (AB)

(Note: The symbol "+" here stands for or, the symbol "." stands for and the symbol "'" "-" stands for non.)

So we can use the simple example explained earlier.

Method 1: X1 XNOR X2 = (A'.B') + (AB)

Designing neurons to simulate A'.B' is a challenge that can be achieved by:

Neuron output: a = f( 0.5 – x1 – x2 )

The truth table is:

Now, everyone can treat A'.B' and AB separately as two separate parts and combine them into a multi-layered network. The semantic diagram of the network is as follows:

Here we can see that on the first level, we determine A'.B' and AB respectively. At the second level, we will output and implement the OR function at the top. The final overall network implementation is shown in the following figure:

If you look closely, you will find that this is just a combination of the neurons we previously painted:

A1: achieved A'.B';

A2: Achieved AB;

A3: Establish OR on a1 and a2 to achieve an efficient implementation (A'.B' + AB).

Verify its functionality through a truth table:

Now everyone should probably understand intuitively how multi-layer networks work. In the above example, we had to calculate A'.B' and AB separately. If we want to simply implement the same or a function based on the basic AND, OR, and NOT functions, we look at method 2 below.

Method 2: X1 XNOR X2 = NOT[ (A+B).(A'+B') ]

Here we use the formula obtained in the second step of the previous decomposition to achieve, the network semantics are as follows:

Everyone can see that neural networks have to use three hidden layers here. The overall network implementation is similar to what we did before:

A1: Equivalent to A;

A2: A';

A3: Equivalent to B;

A4: Achieved B';

A5: Achieved OR, actually A+B;

A6: The OR is implemented, which is actually A'+B';

A7: AND is implemented, which is actually (A+B). (A'+B');

A8: NOT is implemented, actually NOT [(A+B).(A'+B')], which is the final XNOR output

The truth table is as follows:

Note: A typical neuron (except the bias unit) provides data for each neuron in the next layer. In this case, we have avoided several connections from the first layer to the second layer because their weight is 0, and adding them easily affects everyone's clear understanding of the calculations between neurons. So here is not drawn as below:

It can be seen that method 2 we have successfully achieved the same or a function. The idea of ​​method 2 is how to break a complex function into multiple layers. However, this method is more complicated than Method 1, so everyone should prefer Method 1.

Before I learned neural network, I also read a lot of information, and some of the information was in the cloud. The author hopes that after reading this article, we can avoid detours and understand neural networks. In the end, I quoted Wu's book and concluded by saying: “There are a lot of technical terms that make you very awkward. The 'artificial neural network' belongs to this category, and at least the first time I heard the word was stunned. You think, in people’s minds, people’s brain structure is still not clear at all, and this leads to an “artificial” neural network that seems to be using a computer to simulate the human brain. The structure is so complicated that everyone's first reaction must be that the artificial neural network must be very advanced.If we are fortunate enough to meet a kindly and articulate scientist or professor, he is willing to spend an hour or two and put the artificial neural network in the simplest way. The bottom line tells you that you will find, 'Oh, it turned out to be this.' If we unfortunately come across a lover, he will tell you solemnly, 'I am using artificial neural networks' or 'I study The topic of the problem is artificial neural networks, and then there is no such thing as following. In this way, in addition to respecting him, you cannot help but feel inferiority. Of course there is good intention but it is not good at expressing. People tried to make this concept clear to you, but he used some more ugly terms to talk about Yunshan Mist. Finally, you found that listening to him for several hours, the result was even more confusing. You lost nothing but wasted time. You come to a conclusion: In any case, I don't need to understand it in my life.

Don't think that I was joking. These are my personal experiences. First of all, I have not met the good people who have spoken to me in an hour or two. Next, I met a group of people who showed off in front of me. As young people, I always wanted to understand what I didn’t understand, so I decided Go to attend a class. However, after listening to it for about two or three times, I stopped going because because it was a waste of time, I didn't seem to get anything. Fortunately, I did not use it for the time being, and I no longer care. Later, during my doctoral studies in the United States, I like to read books before going to bed. I just hold a few textbooks about artificial neural networks on the bed and I actually read them. Then use it to do two or three projects, be regarded as learned. At this point, looking back at the 'Artificial Neural Network' is actually not complicated. It is not difficult to get started. I just went the wrong way.

Lei Feng Net Note: This article is authorized by the bright wind Taiwan Lei Feng network release, if you need to reprint please contact the original author.