Bin Picking Cell Control System

This project implements a control system for a robotic bin picking cell, including WMS API, Robot Server, ROS2 nodes, and HMI interface.

🆕 Apr 2026 — The AI Command Interface was added as an extension to explore AI-driven application workflows within a working robotics system. It covers the full pipeline — from data handling and AI Client integration to model usage and inference at the application level — built on top of the existing ROS2-based bin picking system.

• [Watch the 1. video] https://drive.google.com/file/d/11evNwT5sQRn601x7im3lHK87BcNyySlo/view?usp=sharing
• [Watch the 2. video] https://drive.google.com/file/d/1BL_pPy4zq_ODYh0MyYfKxksL2QmgBRdx/view?usp=sharing

System Architecture

• WMS API (Port 8080): Handles pick requests from Warehouse Management System (WMS)
• Robot Server (Port 8081): Manages robot operations and cell status
• ROS2 Nodes: Control various cell components (scanner, door, emergency button, stack light)
• HMI (Port 8082): Web-based interface for monitoring cell status
• AI Agent (Port 8083): Natural language command interface powered by local LLM (Ollama)

Project Structure

bin_picking_cell/
├── docker/
│   ├── docker-compose.yml     # docker files
│   ├── ai_agent.Dockerfile    # AI Agent container
│   └── ...
├── ros2_ws/src
│   ├── bin_picking_control/   # ROS2 nodes
│   └── bin_picking_interface/ # costomized msgs and srvs
├── src/
│   ├── ai_agent/              # AI Agent server
│   ├── hmi/                   # HMI interface
│   ├── robot_system/          # Robot Cell server
│   └── wms_api/               # WMS API

Prerequisites

• Docker and Docker Compose

• ROS2 Humble (if running without Docker)

• Python 3.10+

• Web browser

• Ollama running locally with qwen2.5:7b model pulled

  ollama pull qwen2.5:7b

  Ollama must listen on all interfaces (not just localhost) so the Docker container can reach it:

  # Edit Ollama systemd service
  sudo systemctl edit ollama

  Add the following and save:

  [Service]
  Environment="OLLAMA_HOST=0.0.0.0"

  sudo systemctl daemon-reload
  sudo systemctl restart ollama

Quick Start with Docker

1. Clone the repository:

   git clone https://github.com/XunG28/bin_picking_cell.git
   
   cd bin_picking_cell
   

2. Build and start the containers:

   # Terminal 0: Creating Docker Containers
   docker compose -f docker/docker-compose.yml up --build
   
   

3. Access the HMI:

• Open your web browser
• Navigate to http://localhost:8082

System Components

1. WMS API (Port 8080)

• Handles pick requests
• Monitors robot service status
• Endpoints:
  • POST /pick: Submit pick requests
  • GET /system_status: Get cell status
  • GET /latest_operation: Get latest operation details

2. Robot Server (Port 8081)

• Processes pick requests
• Manages ROS2 communications
• Monitors cell components status

3. ROS2 Nodes

• Scanner Node: Generates random barcodes, topic /barcode

• Door Handle Node: Monitors door state, topic /door_state

• Emergency Button Node: Manages emergency state, topic /emergency_state

• Stack Light Node: Indicates system status, topic /stack_light_state

• Control Services: services that can change the states of Door handle and Emergency button

4. HMI Interface

• Real-time status display
• System state visualization
• Operation history
• AI Command panel with live safety warnings and command history

5. AI Agent (Port 8083)

This module is a practical demo application showing how an AI Client can be integrated into a real-world robotics system. It handles the full inference pipeline from natural language input to a validated, structured pick request — bridging conversational input with the existing WMS API.

• Natural language pick request interface
• Powered by local Ollama LLM (qwen2.5:7b, temperature=0 for deterministic output)
• Endpoint: POST /ai_command — accepts {"command": "natural language string"}
• Parses order ID and quantity from free-text input (typo-tolerant)
• Validates extracted values (positive integers only) and provides intelligent suggestions on errors
• Detects multiple orders in a single message and asks the user to send one at a time
• Forwards valid requests to WMS API and returns the full pick result
• Response status types:
  • ok: order successfully picked (green in HMI)
  • error: order understood but robot execution failed (red in HMI)
  • missing: incomplete or invalid input with a suggestion (yellow in HMI)

Testing the AI Command Interface

Open the HMI at http://localhost:8082 and use the AI Command panel. The interface supports natural language input and handles the following scenarios:

Scenario	Example Input
Normal pick	Pick 3 items for order 42
Not enough items	Pick 100 items for order 999
Safety door open	trigger door, then send a valid pick
Emergency active	trigger emergency, then send a valid pick
Missing order ID	pick 3 items
Missing quantity	order 123
Invalid values	Pick -3 items for order 33
Multiple orders	order 321 qty 7; order 555 qty 6

Testing the System

1. Check ROS2 Nodes

   # Terminal 1: Check all ROS2 nodes in ros2_nodes container
   
   docker exec -it ros2_nodes bash
   
   # Check barcode
   ros2 topic echo /barcode
   
   # Check door state
   ros2 topic echo /door_state
   
   # Check emergency state
   ros2 topic echo /emergency_state
   
   # Check stack light state
   ros2 topic echo /stack_light_state
   
   

2. Send a Pick Request

   # Normal scenario (when the door is closed, without emergency stop):
   # Terminal 2: docker_wms_api container
   
   docker exec -it docker-wms_api-1 bash 
   
   # Send a pick request
   curl -X POST http://localhost:8080/pick \
   -H "Content-Type: application/json" \
   -d '{"pickId": 222, "quantity": 2}'
   
   

3. Test Safety Features

   #Terminal 1: ros2_nodes container
   
   docker exec -it ros2_nodes bash 
   
   # operation1: Open door
   ros2 service call /set_door_state std_srvs/srv/SetBool "data: false"
   
   # operation2: Press emergency button
   ros2 service call /press_emergency std_srvs/srv/Trigger
   
   # operation3: Close door
   ros2 service call /set_door_state std_srvs/srv/SetBool "data: true"
   
   # operation4: Release emergency button
   ros2 service call /release_emergency std_srvs/srv/Trigger
   
   

after every operation, we can see the status of the system in the HMI interface at http://localhost:8082 or the log infos in the Terminal 0.

```  
# Terminal 0: Another way to see the status of the system
cd bin_picking_cell/docker
docker compose logs -f
```

We can test the system after every operation by sending a pick request to the WMS API.

```
# Terminal 2: docker_wms_api container

docker exec -it docker_wms_api bash 

# Send a pick request, change the pickId and quantity to test different scenarios
curl -X POST http://localhost:8080/pick \
-H "Content-Type: application/json" \
-d '{"pickId": 123, "quantity": 3}'

```

Only when the status is operational will it return success, and other statuses will return failure.

System Flow

Request Flow

External Request -> WMS API Container (Port 8080)
                   |
                   v
            Robot Server Container (Port 8081)
                   |
                   v
            Pick Manager (ROS2 Service: execute_pick)
                   |
                   v
            Check ROS2 Node States:
            - Stack Light State (0: operational, 1: paused, -1: emergency)
            - Door Handle State (open/closed)
            - Emergency Button State (pressed/released)
            - Barcode Scanner (get codes)
                   |
                   v
            Execute Pick Based on States:
            - Emergency button pressed -> Fail
            - Door open -> Fail
            - Stack light not operational -> Fail
            - Insufficient barcodes -> Fail
            - All normal -> Success with barcodes
                   |
                   v
            Response to Robot Server
                   |
                   v
            Response to WMS API
                   |
                   v
            Response to External Request

Container Communication

WMS Container (8080) <--HTTP--> Robot Server Container (8081) <--ROS2--> ROS2 Nodes Container
(wms_api-1)                     (robot_server)                         (ros2_nodes)
     ^
     | HTTP
AI Agent Container (8083) <--HTTP--> Ollama (host, port 11434)

Environment Separation

This project has the feature of auto-configuration in local/multi-container, therefore can adapt to more complex environments.

AI Agent (8083)    -> WMS API (8080)     -> Robot Server (8081)    -> Pick Manager    -> ROS2 Nodes
Pure Python           Pure Python           ROS2 + Python              Pure ROS2         Pure ROS2
     |
Ollama (host)