In June, I participated in the McGill Innovation Lab’s AI for Social Good Hackathon in Montreal. With my team, we came together to create a Chatbot that can interact with the user, understand an inquiry (about a common language word) and send back a link to the Handspeak website. On that web page, the user will be able to watch a video that illustrates the word in ASL (American Sign Language). In this post, I will go over our thinking process and how we were able to make the application.
Note: The code for this project can be found here.
1) Finding the Idea
We were actually struggling for a good idea, looking for a clean dataset and see how we can apply ML/AI to it. It took a huge amount of time, but we were able to set on an idea before the supper: SensAI, a chatbot that returns ASL videos. As none of us created a chatbot before, we thought it would be a great project to work on.
2) Creating Our Own Dataset
As there weren’t any clean existing dataset of ASL translated videos of common English word, we decided to use Handspeak. The handspeak website has a fair amount of videos for common words (7553 videos, one for each word). We thought it would be a great start to find links to these videos and being able to map them to the word associated. We used Python to scrape the website and create a dataset. Although the links for the video location couldn’t be easily shared because of permission issues, the dataset still allowed us to map the numbers to the correct word and its webpage. The Python program we wrote for this is called DBExtract.py. The program scrapes the website’s structure and saves all important information in a csv file.
- Link: Handspeak website
3) Using Wit.AI
We then created an account on Wit.AI, an online application owned by Facebook. This application allows the user to create a chatbot, train it on sample expressions to recognize what the user is saying and allowing to return an output by the bot. Furthermore, it also allows to do back-end work using Node.js, Python and other languages. We used it to first train our model to recognize expressions like “How do you say WORD?” and “What is WORD in ASL?”. The back-end to keep track of these expressions and return the output based on the confidence perentage was done using the node-wit repositary on github, available here. Make sure to have Node.js installed before you use the code.
- Link: Wit.AI quickstart tutorial
4) Creating a Facebook Page to Host the Messenger Bot using Ngork
The next step was to go on Facebook to create a new facebook page. We found this nice tutorial that explains step by step how to create a chatbot on facebook and how to connect it to our code. However, instead of using Heroku to host the chatbot, we simply wanted it to run locally on our own laptop. Hence, we decided to use Ngork instead. Here is another extremely handy tutorial that explains how to run the code using it. Make sure you have a hidden file .env where all your tokens are initialized with the proper value.
5) Test and Results
To test the chatbot, run on your command-line both the messenger.js file using Node and open a local port using ngrok.
./ngrok http 8445
You are now connected and can see immediate result. Here, I tested the chatbot and was pleasantly surprised to see that everything is working as expected. If you look at the command-line, you will also be able to track the confidence percentage for each expression being entered.
A video demo in .mp4 format is saved on my github account, but for now, here are some screen shots:
Sadly, even though we wanted to do Natural Language Processing (NLP) through the chatbot, our team all agreed we ended up not really having a ML based project. Either way, it was a wonderful learning experience overall and we were happy to have a complete project to present at the end. We learned to scrape a website, create a messenger bot and use Node.js. Hopefully, next time, we will get to work on a more AI intensive project.