Introduction
Recently, we started using GPT-3 for building a conversational chatbot for real estate domain. We expected that GPT-3 was a great solution for building a conversational chatbot.
However, early in the implementation, we started facing several issues with GPT-3 responses. GPT3 was pretty bad with fairly basic, general conversation that humans do flawlessly :-
1). It does not know, how to respond well to greetings. This is something that any decent chatbot or customer facing agent does quite well. It would be unacceptable to give irrelevant, or inconsistent responses for such use cases.
2). It can respond to abusive language in the same way e.g. if you say fuck you to GPT-3, and it would say fuck you too !
3). It does not connect well with domain, even after training with a significant dataset.
4). Its real world and domain knowledge is outdated. For example, it does not know about latest house listings, product features or market news.
5). It does not have context of the conversation meaning what did you talk about earlier in the conversation
6). It is not situationally aware
After understanding these challenges, we turned to prompt engineering and started improving the prompts according to the context of the conversation, and expectations/requirements from a real estate chatbot. Prompt engineering is not the silver bullet, as you require to connect these systems effectively with the domain and real-time information flow.
Understanding GPT-3 Capabilities and Limitations
Lets start to understand, what GPT-3 is, what it can do and what it can not do.
Some people tend to think that GPT-3 somehow has a brain, and can answer any question after careful thought process based on facts which is obviously not true.
Please try to understand following key points about GPT-3 :-
1). GPT-3 is a generative language model, and it does not have a brain or concious.
2). GPT-3 does not have domain knowledge and situational awareness
3). GPT-3 can not and does not judge the correctness or relevance of its answers
4). GPT-3 is not a weather bot, and can not give you accurate weather forecasts at your current location
In terms of what GPT-3 can do may be summarized as follows:-
“GPT-3 can generate plausible answers at the level on an inexperienced customer service agent who does not know what he is talking about”.
Most of the times, conversations with an AI bot using GPT-3 can be very frustrating, here are some examples
Conclusion
Our initial analysis indicated GPT-3 is not an ideal solution for chatbot acting as a customer service agent because of many reasons. These reasons included :-
1). It will never have true understanding of the domain or situational awareness.
2). It is also probablistic in nature and can not be expected to be consistent with responses.
3). Its knowledge is outdated, and it does not have situational awareness
However, after considerable analysis and refinement we discovered that it is possible to build chatbots with #GPT3 that are not only conversational, but can perform to the level of most experienced customer service agents. This does not mean, such chatbots will replace human agents but these will act as 24/7 helpdesk systems that would become first point of contact for customers and will considerably improve customer service and sales.
GPT-3 can be used for building great conversational AI solutions that nearly as good as humans with ability to respond to greetings, tell jokes, steer conversations in proactive manner, having up to date domain knowledge and tracking users across various channels e.g. Whatsapp, Telegram, Email, Web etc.
References
https://www.helpscout.com/blog/ai-in-customer-support/
https://www.artificialintelligence-news.com/2020/10/28/medical-chatbot-openai-gpt3-patient-kill-themselves/
https://www.technologyreview.com/2020/10/23/1011116/chatbot-gpt3-openai-facebook-google-safety-fix-racist-sexist-language-ai/
https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api