Reinforcement Learning: What is it?

One of the humans’ most efficient learning procedures involves learning from mistakes. This way, we can focus more on the topics that we made mistakes in to deepen our understanding and expand our knowledge. It has been an ever-fascinating and intriguing field of research where these questions arise. “Can machines be made to learn from their mistakes?” or “Can machines learn to strategize their problem solving by trial and error?” (Marriott & Harrison, 1997)

Reinforcement learning is the type of machine learning algorithm that is the answer to these questions. With its roots in psychology and neuroscience, it has emerged as a powerful learning algorithm that is inspired by the way humans learn through feedback.  A clear goal is defined, and reinforcement learning tries to reach that goal through trial and error. It is not provided with any labels on data, so it navigates that on its own.

Reinforcement learning tries to maximize cumulative rewards by interacting actively with the environment. It consists of four key components – Agents, Environment, Actions, and Rewards. Agents are the decision-makers that learn by interacting with the environment. The environment is the context or framework that the agent uses to learn and take feedback. After learning from the environment, actions are the decisions made by the agents to reach a goal. Finally, rewards are the feedback or response provided by the elements in the environment that tell the agent whether the decision is sound or not.

Just like how humans are encouraged through the festival of praise, the model knows that it can move forward with the decision through positive feedback. Certain values are assigned to the positive and negative responses. The cumulative reward is the sum of the positive feedback values or the final value.

Working of the Algorithm

The agent navigates through an environment and makes a decision. This decision results in the altered shape of the environment which tells the agent whether to move forward with the next decision or gain more rewards from the current state. In this way, the algorithm employs the if-then policy. If a decision leads to more reward, proceed forward, or else gain more points in the current state.

Types of Reinforcement Learning

There are two primary types of reinforcement learning. 

1. Model-based – it involves a defined environment that does not change often. This makes the learning process for the algorithm easier as it is provided with clear reward expectations and transition state dynamics.
2. Model-free – employed when the environment dynamics are variable and not explicitly modelled. Feedback is provided on every state transition. The agent updates its policy according to these rewards.

Real-World Application

Consider a real-world example, reinforcement learning is used for maximizing ROI (Return On Investment) by allocating resources to the most promising opportunities. It does this with the help of the enormous amount of data being generated everyday which includes our search history, past interactions with certain ads, demographic information, and other contextual data. By processing this data, the algorithms gain insights into which ads are more likely to resonate with specific user segments and drive desired actions, such as clicks or conversions. 

Since reinforcement learning is a continuous learning model, it decided to put up an ad campaign. The user’s response is then observed, how a user interacts with the ad, whether he ignores it or shows interest. The model monitors this engagement and optimizes its strategies based on it. This process is iterative as the model adapts to different feedback and ultimately makes the best decisions that gain more rewards. This is how an ad campaign is optimized in real-time.

Ultimately, the model focuses on optimizing those performance metrics that directly impact ROI. These metrics can be click-through rates, conversion rates, cost per acquisition etc. The reinforcement algorithm tries to dynamically allocate a larger portion of the resources to the opportunities with more promising results. For example, if an ad placement to a certain group of people yields higher conversion rates, then more budget is allocated to the campaign. Fitness enthusiasts are more likely to interact with products aimed at fitness and gym than others. 

The model also learns to optimize itself with the evolving market trends, customer preferences and competitive landscapes but since reinforcement learning models are designed to adapt to change, advertisement strategies remain effective and competitive. 

Citation

Marriott S, Harrison RF. Can Machines Ever Learn from Their Own Mistakes? Measurement and Control. 1997;30(10):300-307. doi:10.1177/002029409703001003