Perceptron & Backpropagation: A Deep Dive into Supervised Learning

To start, let’s rewind a little back into machine learning. Machine learning is the branch of Artificial Intelligence that deals with teaching a machine how to give the appropriate response to a given query without the intervention of humans. It is making the computers learn to adapt to certain responses learn from their mistakes and give a better and more accurate response. So how exactly does a machine learn and what does it use?

Machines and computers use algorithms to learn and adapt. There are four of these algorithm types that are employed that allow them to extract appropriate patterns and respond automatically. 

1. Supervised learning
2. Unsupervised learning
3. Reinforcement learning
4. Evolutionary learning

Here, we will take a deeper look at only supervised learning. 

A massive amount of data is produced daily, that is, in terabytes in some cases. This huge amount of data needs processing to be of any use and humans can only analyze so much. This is why these four algorithms are there. Humans provide some amount of supervision, and these models do all the processing work themselves.

However, there is a distinction between algorithms and models. Regarding machine learning, they are often used interchangeably. An algorithm is a set of steps taken to solve a problem whereas a model is the representation learned from the specific data with defined parameters. Therefore, we can say that the algorithm is the process of building a model. The difference can be presented in the form of a function in the following way: If \(y=f(x)\) then \(y\) is the model and \(f\) is the algorithm. \( x\) Is the data that is processed. 

The Perceptron

The perceptron is an artificial neural network inspired by how a human’s neuron works. It was introduced in 1958 by Frank Rosenblatt and can perform binary classification tasks which means that it can classify the data into two possible categories typically represented by binary labels 1 or 0. Rosenblatt was a psychologist and computer scientist who was interested in developing a machine that could learn and recognize patterns in data. Perceptron is a single-layered neural link that requires four parameters making it the simplest algorithm. Initially, it was used for simple tasks as it could only take a limited number of inputs to provide an output. It learned through the response provided at each output which the perceptron remembered. The limitations caused the perceptron’s popularity to decrease, and research stopped around the 1960s and 1970s. (Banoula, 2023)

The Back-Propagation Algorithm

In the 1980s, a multi-layered neural network was introduced called the backpropagation algorithm. This rekindled the spark of interest in neural networks. The backpropagation algorithm is considered a type of supervised learning that requires a known output for each input value to give an accurate result. It involves working backwards, from outputs to inputs to decipher how to reduce the number of errors and improve the quality of the neural network. It does not require any parameters. This model is way more flexible and can be applied to a broader range of scenarios. 

The Supervised Learning Algorithm

As the name suggests, supervised learning models learn under the supervision of humans. This means that it is provided with already defined labels on the datasets to match specific inputs. This is called the training data. The machine or computer is fed with this data that allows it to learn and predict a new set of data called the testing data. The testing data is used to measure the accuracy of the result. Finally, it performs classification and regression tasks. 

Now the target data is usually written as a set of data (\(x_{i},t_{i}\)). The inputs are \(x_{i}\) and the outputs are \(t_{i}\).  The subscript i indicates the index of the data and suggests that there is a lot. It reaches up to n which is the upper limit.

The supervised learning algorithm tells the relationship between the input and output where the input is \( x\) and the output is \( y\) or \( t\) (\( t\) for target). The goal of supervised learning is to learn a function whose inputs and outputs are always changing. The basic steps are listed below:

• Collecting labelled data
• Data cleaning and preprocessing
• Model selection (Regression or Classification)
• Training the model
• Model evaluation

In mathematical terms, given \( x_{1},y_{1},…,(x_{n},y_{n})\), choosing a function \( fx=y\) is what supervised learning does. 

The two types of supervised learning are classification and regression. Classification is when the data is categorized based on qualitative analysis and regression involves the relationship between the \( x\) and \( y\) based on quantitative analysis. Simply put, classification models give yes or no answers, and regression models present a numerical answer.

Classification is when $$y_{i} \in \{\>a\>finite\>set\>of\>data\>\}$$

Regression is when $$y_{i} \in R \>or\> y_{i} \in R^{d}$$

Where \( R\) is the set of real values and \( R^{d} \) is \( d\)-dimensional real space (sometimes).

All these formulas are the simplest representations of supervised learning. The overall formula is

$$M = \left\{ \sum (x_i, y_i) \right\}^{N}_{i}$$

Where \( N\) is the number of training examples and \( x_{i}\) is a \( d\)-dimensional vector (notice how it’s in bold) also known as features. The \( y_{i}\) is the output value which is a categorical variable for a classification problem and a real value for a regression problem. 

Conclusion

To conclude, supervised learning is a useful concept in machine learning where models learn to make predictions under the guidance of labeled data. This approach involves collecting labelled datasets, training the model on the labelled data, and evaluating its performance so that it can learn a function that maps inputs to outputs accurately.

The perceptron and the backpropagation algorithm are notable examples of supervised learning models. While the perceptron is a simple single-layered neural network capable of binary classification tasks, the backpropagation algorithm introduced in the 1980s enabled the development of more complex multi-layered neural networks, leading to renewed interest in neural networks.

Supervised learning can be categorized into classification and regression tasks. In classification, the model categorizes data into classes, while in regression, the model predicts numerical values. These tasks involve finding the relationship between input features and output labels.

Overall, supervised learning plays a crucial role in various applications, including pattern recognition, image and speech recognition, medical diagnosis, and more. Its ability to learn from labelled data under human supervision makes it a powerful tool for solving real-world problems and advancing artificial intelligence.

Citations

• Banoula, M. (2021, May 26). What is Perceptron? A Beginner’s Guide for 2023 | Simplilearn. https://www.simplilearn.com/tutorials/deep-learning-tutorial/perceptron#what_is_a_binary_classifier_in_machine_learning
• Marsland, S. (2014). Machine learning: An algorithmic perspective (2nd ed.). CRC Press.