A large percentage of free available language-learning apps available to the public are not as intuitive, appealing, or interactive to many users across the world. In order to solve this issue, we needed to come up with a revolutionary approach to promote language education in both an efficient and accessible system for all – tackling the diversity theme.
Lexigo is an AI-Powered Application that provides a unique language-learning experience which is globally accessible on IOS/AndroidOS. By opening the phone application you are able to walk around your environment in real-time and seek objects that interest you, and provide a translation for the selected language. This occurs while adding the objects to a list of learned words, to test in the future. The user will have the option to select the quiz mode, where the scanned objects get their own multiple choice dropdowns of similar words to the object and the user must use their knowledge to choose the correct word.
To create our application we used both React Native’s and Expo’s frameworks to build the fundamentals of our project and continue to expand our ideations through the allotted time period. By using GitHub, we created separate branches and we divided & conquered different portions of the app. For example, one was responsible for translation, one for object detection, one for front end, and one for back end. We were able to refine our product, bringing everything together and finalizing the first prototype. One of our final steps was to test and optimize our app in a repetitive manner to ensure all team members were satisfied. We also made it essential to focus on UI/UX to provide users with an enjoyable experience and intuitive design.
We aimed to include an API for both object detection and language translation. Unfortunately, all our searches yielded the same models and systems which weren’t efficient. As a result the translations take too long to render due to the increased amount of API calls made and our object detector buffers and is not always 100% accurate. And to top it all off, the API we used had rate limits which we passed and therefore couldn’t use anymore as it was paid. To work around this challenge we had to adapt quickly and adjust our object detection models to the open-source framework TensorFlow.
Overall, the Creation and implementation of a base object-detection system was a huge accomplishment as most of us had little to no prior ML or AI experience. Additionally, this hackathon was one of our 1st in-person hackathon experiences for the majority of our teams which we found very enjoyable as a result. Finally, many of us had not worked with React Native for app development before, which initially provided a challenge to become accustomed to the new syntax and environment. However, it was overall a knowledgeable learning experience and we all learned something new!
We learned how to use HuggingFace API to translate languages, to deploy and train an ML model that provides object detection and language translation. We learnt how to use Expo on React Native and ExpoGo for the mobile app. Additionally, we all learnt how to collaborate, organized and delegated tasks efficiently being the first hackathon experience we collectively collaborated together on.
To take Lexigo to the next level we can introduce a more accurate object detection model to provide a more interactive, reliable system for users. The introduction of a leaderboard system to keep track of recently learned vocabulary and the inclusion of more in-depth statistics including the use of graphs to showcase how many new words you learnt in a selected time-span. The full implementation of the Quiz feature was unfortunately not fully implemented within the time-span of this hackathon which would be able to provide a more efficient learning technique. Additionally, the use of One vs. One competition to let users get matched, and play against others in under a minute by solving as many multiple choice options for each object as possible. Finally, we really wanted to include more languages in our settings page to both allow more variety and also provide the option to change the applications language to your own.