mbed: Gesture recognition

We have the new gesture sensor development kit, MtSense04-Pi, which carries a gesture recognition sensor with low power consumption while being small in size. Use HID library on mbed and make the gesture sensor a remote presenter.

MtSense04-Pi carries the hand gesture sensor PAJ7620 that is developed by our partner Pixart Imaging Inc. Use MtSense04-Pi to measure the hand gestures and convert the hand gestures into the mapped key with MtConnect04S.

This gesture recognition example uses the mbed environment. If you are new, you can check out the getting started tutorial first.

Prepare

gesture-sensor-kit

This example uses MtConnect04S, the Bluetooth LE board, MtSense04-Pi, the gesture sensor, the extension board MtAid01-mbed. Stack them together.
gesture-kit-stacked

 

Build

Go to the repository on mbed, and click on Import into Compiler.螢幕快照 2016-12-30 下午12.16.45.pngmbed will open the Compiler page in another browser tab and pop-up a dialog will pop up. Select Import. You will then see the source code.%e8%9e%a2%e5%b9%95%e5%bf%ab%e7%85%a7-2016-12-30-%e4%b8%8b%e5%8d%884-47-38

You can check the file explorer on the left side. It imports libraries of mbed

  • BLE_API and nRF51822 which are fundamental functions of Bluetooth.
  • PAJ7620U2 for sensor’s I²C actions.
  • BLE_HID is a Bluetooth HID implementation.

Everything is good to go, so click on Compile. It will start to build the application. When it is done, the browser will download a .hex file automatically. You can save it to anywhere you want. It is the firmware binary which we will run on MtConnect04S.

If you are familiar with Nordic SDK, the .hex file downloaded includes both application and SoftDevice.

Run

Drag the .hex file to the USB Drive MTCONNEC04. The MtAid01-mbed board will program MtConnect04S. When it is done, the drive will be removed automatically, and you may see an unexpected removing drive message. Then, push the button on MtAid01-mbed, or replug USB to re-power up the kit.

You can then connect it to the PC. We tested on Windows 10, macOS Sierra and the author of BLE_HID tested it on Windows 8.

For Windows 10, open Settings, and select Devices. On the Bluetooth page, you will see a device named MtM Gesture. Click on Pair to connect to the PC.

win10-settings-bluetooth

Your gesture keyboard is now ready. Open PowerPoint to test. The motion sensor kit recognizes nine different gestures:

Hand gesture Mapping key
Moving up Up arrow
Moving down Down arrow
Moving left Left arrow
Moving right Right arrow
Moving forward Page down
Moving backward Page up
Circle clockwise F5
Circle counter clockwise ESC
Waving “Bye Bye” (Continuous inputs)

Modify

If you’d like to change the behavior, it is easy. On the mbed Compiler page, open main.cpp in the left file explorer and find main(). There is a switch case in while loop. It will read the measured value from the sensor and send mapping keys. Have fun!mbed-gesture-main-function

Known issue

Due to the library BLE_HID implementation, you have to unpair and pair again with the device and PC/MAC.

This tutorial is applicable with
 ✅MtConnect03
 ✅MtConnect04
 ✅MtConnect04S

Please feel free to contact us, if you need further information