This project demonstrates real-time human pose estimation in the browser using a webcam feed. It utilizes TensorFlow.js with the MoveNet model to detect body keypoints. When a specific pose (raising the right hand above the shoulder) is detected, it triggers a sine wave sound generated using the Web Audio API.
The entire application is contained within a single HTML file (index.html 또는 motion_sound.html).
- Real-time pose estimation from webcam video using TensorFlow.js and MoveNet.
- Visualization of detected keypoints and skeleton on an HTML Canvas.
- Detection of a specific pose: Right hand raised above the shoulder.
- Generation and playback of a sine wave sound using the Web Audio API upon pose detection (no external audio files needed).
- Client-side implementation (runs entirely in the user's browser).
- Simple and contained within a single HTML file.
- HTML5 (Webcam via
getUserMedia, Canvas) - CSS3 (Basic Styling)
- JavaScript (ES6+)
- TensorFlow.js - Core library for machine learning in JavaScript.
- TensorFlow.js Pose Detection API (MoveNet) - Pre-trained model for efficient pose estimation.
- Web Audio API - For generating and playing sounds directly in the browser.
-
Clone or Download:
- Clone this repository:
git clone https://github.com/hwkims/js-pose-estimation-web-audio.git cd js-pose-estimation-web-audio - Or, download the
index.html(ormotion_sound.html) file directly.
- Clone this repository:
-
Run a Local Web Server:
- Due to browser security restrictions (CORS,
getUserMedia), you need to run this HTML file from a local web server. You cannot simply open the file directly using thefile://protocol. - Using Python 3:
python -m http.server
- Using Node.js (requires
http-serverinstalled globally):npm install -g http-server # If not already installed http-server .
- Using VS Code: Use the "Live Server" extension.
- Due to browser security restrictions (CORS,
-
Open in Browser:
- Open your web browser and navigate to the local server address (e.g.,
http://localhost:8000,http://127.0.0.1:8000, or the address provided by Live Server).
- Open your web browser and navigate to the local server address (e.g.,
-
Grant Camera Permission:
- Allow the browser to access your webcam when prompted.
-
Interact:
- Stand in front of the camera. You should see your skeleton overlayed on the video feed.
- Raise your right hand clearly above your right shoulder. A sine wave sound should play briefly.
- Performance may vary depending on your device's hardware.
- The MoveNet 'Thunder' model is used for better accuracy, but 'Lightning' can be used for potentially faster performance on lower-end devices (you can change this in the
loadDetectorfunction in the script). - Audio playback might require an initial user interaction (like clicking anywhere on the page) in some browsers due to autoplay policies. The code includes a listener to attempt resuming the
AudioContexton the first click. - Pose detection sensitivity can be adjusted by changing the
minConfidencevariable or the pose checking logic incheckRightHandUp.
##📄 License
This project is open source and available under the MIT License. (추후 LICENSE 파일을 추가하거나 원하시는 라이선스를 명시하세요)
Feel free to contribute or suggest improvements!