Machine Control & Monitoring with Unity and AI Chatbot

Hackathon EPFL Lausanne, Switzerland
November 30 - December 1

Authors

• Arnau Claramunt
• Genís López
• Jaume López
• Pay Mayench

        

---

Project Overview

This project is the implementation of the Bobst Company challenge presented at the LauzHack24 Hackathon. We assembled a 3D printed conveyor belt machine, represent the virtual model in Unity and use a chatbot with a small LLM to also interact with the machine.

Key Features:

• 3D-Printed Conveyor Belt controlled by a Raspberry Pi.
• Real-time data collection (speed, box counter, energy usage).
• AI-powered chatbot for troubleshooting and monitoring via Ollama's Llama 3.2.
• Interactive HMI built in Unity for real-time control and visualization.
• Dockerized architecture for easy deployment.
• Bridge between Unity and Ollama with FastAPI.

---

🛠️ Technologies & Tools

Component	Technology	Purpose
Hardware	Raspberry Pi	Controls conveyor belt and collects metrics.
Modeling	Unity	Creates a 3D visualization of the machine.
API Layer	FastAPI	Communication between AI and Unity and Unity to Raspberry Pi.
AI	Ollama (Llama 3.2)	Context-aware troubleshooting LLM chatbot.
Deployment	Docker	Simplified deployment of all components.

---

Bob the AI Chatbot

Meet Bob, our AI-powered assistant built on Ollama's Llama 3.2. Bob is integrated into the system, providing real-time assistance, troubleshooting, and even direct control over the conveyor belt.

---

🧠 How Bob Works

1. Understand Context:
   Bob uses the machine's current state (e.g., motor status, speed, output metrics) as context for user interactions.

2. Interpret User Queries:
   Natural language queries are passed to Bob along with the machine state for precise and actionable responses.

3. Execute Actions:
   When appropriate, Bob translates its recommendations into machine actions via API calls to the Unity that then calls the Raspberry Pi.

🔮 Future Improvements

• Add more sensors for enhanced data insights.
• Introduce predictive maintenance using AI.
• Expand HMI with detailed production analytics.

---

Screenshots

Conveyor belt machine:

The machine operates with a safety feature that performs a secure stop to prevent accidents when a hand is detected near the belt:

Safety_stop.mp4

Unity interface:

API for talking to the Raspberry Pi 5:

---

Setup instructions

1. Clone the Repository:

git clone [email protected]:EncryptEx/machine-control-monitoring.git
cd machine-control-monitoring

2. Build and run Dockers containers: (form the chatbot_api folder)

docker-compose up --build

3. Launch Unity Application:
   Open the Unity project in your IDE and run the scene.

4. Interact with Bob:
   Use the chatbot interface in Unity to troubleshoot and control the conveyor.