How NLP Plays a Role in the Performance of Your AI Chatbot - Artificial Intelligence (AI) Chatbot in Malaysia, For You

How NLP Plays a Role in the Performance of Your AI Chatbot

An illustration of the Eight Dome Mosque in Terengganu, Malaysia.

Whenever AI Chatbots come into the picture, you’ll inevitably hear the golden three words: Natural Language Processing (NLP).

On the surface, the words seem self-explanatory: Since our computers speak a fairly unnatural language consisting of 1’s and 0’s, it’s going to have to do some processing to understand us. But is there a difference between one company’s NLP and the next? And how is the state of NLP today affecting your Chatbot? Before we go into that, let’s first dive into how it all started.

The Early Years

Interest in NLP and conversational AI started even before most of the world saw a computer. In October of 1950, Alan Turing proposed The Imitation Game — a game in which three players (a human, a machine, and a judge) communicate with each other, with the end goal being to answer a simple question: Can a machine pass as human?

While it wasn’t much fun as a game, it did launch Conversational AI into stardom as a sort of challenge for the masses. Nowadays the Imitation Game is called the Turing Test; and acts as a benchmark of sorts upon which you might measure humanity’s progress in technology as a whole: If a machine can speak like a human, then perhaps we have achieved artificial intelligence.

A chatbot is born

Despite the early start, it was a long wait until we saw the first program that appeared to ‘pass’ the Turing test: Joseph Weizenbaum’s ELIZA; a chatbot created in 1966. Ironically, in making the chatbot, Weizenbaum wanted to illustrate that the technology was still very primitive (the chat was mostly powered by keyword matching) and that they were nowhere near achieving machine intelligence.

So how intelligent was it? I personally feel it is quite impressive: Despite having huge limitations in both hardware and data, ELIZA still manages to come up with some fascinating responses. Of course, the use case is very limited — and it was arguably more a feat of clever social engineering than machine intelligence. Thanks to Norbert Landsteiner, you can have a chat with ELIZA and experience her smarts for yourself.

NLP As We Know It

It’s been a long time since the ’60s, and the approach when it comes to NLP has changed quite drastically. Gone are the days of rule-based, keyword-matching systems. These days, it’s all about building language models. The following is a simplified step-by-step on how this is done.

Step 1: Getting a dictionary for your computer

Building a language model today starts with creating as wide a dictionary as possible — through a process that usually involves trawling the Internet for huge amounts of data. And when I mean huge, I truly do mean huge — GPT-3, for example, was trained on 45TB of text data sourced from various data sources, forming a total dictionary of 400 billion tokens (read: words).

Step 2: Forming contextual understanding

It’s not enough to just know that the words exist: For a machine to truly read and understand a sentence, they would need to have a grasp on context as well. Take for example the following sentence:

By the time he arrived, she had had dinner.

For a machine to understand the difference in meaning between the first had (meaning already) and the second had (meaning eaten), it would have to know a bit more than just assigning a token.

While there are many ways to do this, the most popular way these days is arguably by using Transformers: A mathematical sort of magic that allows computers to ‘understand’ words based on the words that appear around them. The result? A machine with a pretty good grasp of how human language works.

Step 3: Problem specialization

Now that we have the language part down, we are ready to use the language model to solve the task at hand — building a conversation. Enter transfer learning — the tried and tested idea that you can achieve better training results when starting with a model that already knows its stuff. All that’s left is to take your language model and train it with your new domain-specific data, whether it be intents and entities for your new corporate chatbot or chatlogs from the deceased.

The quality of the language model is where NLP providers may differ slightly, and if you’re going for an NLP solution by one of the tech giants (Google, Microsoft, IBM, et cetera), we mean very slightly — unless you’re working on the fringes of Conversational AI (in which case you shouldn’t be looking for a mass-market solution) you wouldn’t even notice.

Challenges in NLP

The current approach to NLP doesn’t come without its own set of problems, and if you’re been in the business of Conversational AI you’ve probably picked up on one or two. Coming up with an exhaustive list of ways NLP today could improve would be too big an endeavor for one article, but below are a couple of the big ones that our current approaches to NLP seem no closer to solving.

Addressing anaphora

Anaphora is, per the Oxford Languages definition, ‘the use of a word referring back to a word used earlier in a text or conversation, to avoid repetition’. While we humans can easily make connections when we hear the words ‘she’, ‘them’, or ‘that’, it doesn’t seem to be quite as simple for our computers. Even with the large swathes of data, huge processing power, and decades upon decades poured into NLP research, the best models still don’t produce results that show a convincing grasp of this concept.

Preventing unhealthy biases

With the latest and greatest models sourcing text from all over the place (novels, research papers, and the filthy broader Internet) it has become more and more challenging to filter and shield machines from our inherent human biases. A good illustration of this is the BERT Fill-Mask Model on Huggingface — which when posed with the prompt…

He works as a _____.

…gives the suggestions ‘carpenter’, ‘waiter’, ‘barber’, ‘mechanic’ or ‘salesman’ but for…

She works as a _____.

…suggests the words ‘nurse’, ‘waitress’, ‘maid’, ‘prostitute’ or ‘cook’.

Final Thoughts

While we are still a ways away from the intelligent machines first dreamed up by Alan Turing, the world of NLP has developed far from where it started. The conversational AI of today holds a lot of promise, and given the right use case, a chatbot could very well be the thing you need to achieve your business goals.

It goes without saying that there is no sense in waiting for perfect — the best time to start building and solving is today.

XIMNET is a digital solutions provider with two decades of track records specialising in web application development, AI Chatbot and system integration in Malaysia. XIMNET is launching a brand new way of building ChatGPT-powered AI Chatbot with XYAN. Get in touch with us to find out more.

about the article

This article is first published on XIMNET Medium. More resources available here.

XIMNET is one of the leading tech agency for AI Chatbot in Malaysia.