During summer 2017, I joined PayPal’s Next-Gen Commerce (NGC) Team. It’s an amazing team working on innovative products. Internally it’s known as one of the special projects team. I was grateful I could work on that team and had to learn from great mentors like Greg, Shilpa, and Dereck.
I was working there as a software engineering intern. I had to work with another talented intern, a designer, William Ferguson. Surprisingly they also housed us together, so we got to know each other and collaborate a lot.
One of the product they were building back then was the PayPal messenger bot. They were prototyping it and testing how helpful it was. But one major request from another department inside PayPal is actually from the Customer Service team. Apparently they got swamped by customer request. How swamped? well, about 1,500 messages per day!
So our team came up with an idea to alleviate a lot of those burden by leveraging our messenger bot. Thus, William and I did our research for this problem and we analyzed the customer service messages. We realized that when we categorize the problems that most people come with when they reach out to the customer service team, a lot of them could be done by simply referring them to the right pages. We realized that the top 6 most common issues that user faces, actually corresponds to about 49% of total user issues.
Once we’re done analyzing the problem, William came up with the design, and I came up with the engineering architecture. The bot uses a combination of rule-based and NLP to guess what the user needs, and responds accordingly giving links or informations needed to help them.
This is done by downloading the entire dataset of the customer service chat. And for each chat log, I label the message as one of the 6 issues that the bot handles. This is not done manually of course, there are some tags in the chat log that can be leveraged. Once the data are labeled, I cleaned the dataset by taking out private data and unnecessary words. Then I trained a model to recognize the chat words with the issues as the label. This way when a user reach out to the bot’s messenger with a phrase like “A merchant charged me the wrong amount” the model could guess that it’s likely to be a dispute issue.
The hardest part building this is to design ambiguous cases, for example cases like when the user came with “I don’t recognize this transaction” could lead to multiple potential outcomes. The user account could have been hacked, or it could be an error from the merchant. But then we solve this by introducing tree-based logic in the bot’s algorithm. This means that when the bot arrived at this kinds of ambiguity, it should be smart enough to ask for more clarification such as “were you initiate this transaction?” or “do you recognize the merchant?” which could lead the user to one solution.
Finally, to test the effectiveness of our product, we ran a user experience test to measure the performance of the bot. The bot successfully resolved user’s issues 75% of the time which translates to more than 37,000 of potential customer service time saved!
And finally we pushed the bot to production, and we got a shoutout from TechCrunch! Way to go NGC team!