2021-06-12_FreeMoCap_Clips_16MB.mp4
(As of May 2022) We're currently working on building a proper API with documentation, but for now enjoy this pile of not-quite-spaghetti-but-definitely-lasagna-flavored code and sloppy ReadMe π
Required
-
Windows only for now (sorry! Mac and Linux support coming very soon!π )
-
A Python 3.7 environment -
- We recommend installing Anaconda from here (https://www.anaconda.com/products/individual#Downloads) to create your Python environment.
-
Two or more USB webcams attached to viable USB ports
USB hubs typically don't workI think they do now?- Note that two cameras is the minimum required for 3d reconstruction. However, with just two views, many points will be occluded/not visible to both cameras. For better performance, use three (or four or more?) cameras
-
Each camera must get a clean unobstructed view of a Charuco board at some (See below).
Open an Anaconda-enabled command prompt or powershell window and perform the following steps:
- Create a Python3.7 Anaconda environment
$ conda create -n freemocap-env python=3.7
- Activate that newly created environment
$ conda activate freemocap-env
- Install freemocap from PyPi using
pip
$ pip install freemocap
That should be it!
tl;dr- Activate the the freemocap Python environment and run the following lines of code (either in a script or in a console)
import freemocap
freemocap.RunMe()But COOL KIDS will install Blender (blender.org and generate an awesome .blend file animation by setting useBlender=True:
import freemocap
freemocap.RunMe(useBlender=True)This two-line script is a copy of the freemocap_runme_script.py file, which can be run by entering the following command into a command prompt or powershell:
(freemoocap-env)$ python freemocap_runme_script.py
- You will know if it's
Anaconda Enabledbecause you will see a little(base)to the left of each line, which denotes that your(base)environment is currently active. - We recommend Windows Terminal so you can enjoy all the Rich⨠formatted text output, but you'll need to do a bit of work to connect it to Anaconda (e.g. these instructions )
- If that seems intimidating (or just too much work), just press the
Windowskey, typeAnaconda Promptand run everything from there.
- If that seems intimidating (or just too much work), just press the
- e.g. if your freemocap environment is named
freemocap-env, type:
(base)$ conda activate freemocap-env
- If successful, the
(base)to the left of each line will change to(freemocap-env), indicating that your freemocap environment is now active (typeconda info --envsorconda info -efor a list of all available environments)
- Activate an instance of an
ipythonconsole by typingipythoninto the command window and pressing 'Enter'
(freemocap-env)$ ipython
[1]: import freemocap5) Execute the freemocap.RunMe() command (with default parameters, see #runme-input-parameters for more info)
[2]: freemocap.RunMe() #<-this is where the magic happens!---β¨πβ¨---.
You can re-start the processing pipeline from any of the following processing stages (defined below)by specifying the SessionID desired stage in the call to freemocap.RunMe()
So to process the session named sesh_2021-11-21_19_42_07 starting from stage 3 (aka, skipping the 1- recording and 2- synchronization stages), run:
import freemocap
freemocap.RunMe(sessionID="sesh_2021-11-21_19_42_07", stage=3)Note - if you leave sessionID unspecified but set stage to a number higher than 1, it will attempt to use the last recorded session (but this can be buggy atm)
-
Stage 1 - Record Videos
- Record raw videos from attached USB webcams and timestamps for each frame
- Raw Videos saved to
FreeMoCap_Data/[Session Folder]/RawVideos
-
Stage 2 - Synchronize Videos
- Use recorded timestamps to re-save raw videos as synchronized videos (same start and end and same number of frames). Videos saved to
- Synchronized Videos saved to
FreeMoCap_Data/[Session Folder]/SynchedVideos
-
Stage 3 - Calibrate Capture Volume
- Use Anipose's Charuco-based calibration method to determine the location of each camera during a recording session and calibrate the capture volume
- Calibration info saved to
[sessionID]_calibration.tomland[sessionID]_calibration.pickle
-
Stage 4 - Track 2D points in videos and Reconstruct 3D <-This is where the magic happens β¨
- Apply user specified tracking algorithms to Synchronized videos (currently supporting MediaPipe, OpenPose, and DeepLabCut) to generate 2D data
- Save to
FreeMoCap_Data/[Session Folder]/DataArrays/folder (e.g.mediaPipeData_2d.npy)
- Save to
- Combine 2d data from each camera with calibration data from Stage 3 to reconstruct the 3d trajectory of each tracked point
- Save to
/DataArraysfolder (e.g.openPoseSkel_3d.npy)
- Save to
- NOTE - you might think it would make sense to separate the 2d tracking and 3d reconstruction into different stages, but the way the code is currently set up it's cleaner to combine them into the same processing stage Β―\_(γ)_/Β―
- Apply user specified tracking algorithms to Synchronized videos (currently supporting MediaPipe, OpenPose, and DeepLabCut) to generate 2D data
-
Stage 5 - Use Blender to generate output data files (optional, requires Blender installed. set
freemocap.RunMe(useBlender=True)to use)- Hijack a user-installed version of Blender to format raw mocap data into a
.blendfile including the raw data as keyframed emtpies with a (sloppy, inexpertly) rigged and meshed armatured based on the Rigify Human Metarig - Save
.blendfile to[Session_Folder]/[Session_ID]/[Session_ID].blend - You can double click that
.blendfile to open it in Blender. - For instructions on how to navigate a Blender Scene, try this YouTube Tutorial
- Hijack a user-installed version of Blender to format raw mocap data into a
-
Stage 6 - Save Skeleton Animation!
- Create a Matplotlib based output animation video.
- Saves Animation video to:
[Session Folder]/[SessionID]_animVid.mp4 - Note - This part takes for-EVER π
The freemocap.RunMe() function takes a number of parameters that can be used to alter it's default behavior in important ways. Here are the default parameters along with a followed by a brief description of each one.
#in `freemocap/fmc_runme.py`
def RunMe(sessionID=None,
stage=1,
useOpenPose=False,
runOpenPose = True,
useMediaPipe=True,
runMediaPipe=True,
useDLC=False,
dlcConfigPath=None,
debug=False,
setDataPath = False,
userDataPath = None,
recordVid = True,
showAnimation = True,
reconstructionConfidenceThreshold = .7,
charucoSquareSize = 36, #mm
calVideoFrameLength = .5,
startFrame = 0,
useBlender = False,
resetBlenderExe = False,
get_synced_unix_timestamps = True,
good_clean_frame_number = 0,
bundle_adjust_3d_points=False
):-
sessionID- Type - (str)
- [Default] - None.
- Indentifying string to use for this session.
- If creating a new session, default behavior is to autogerate SessionID is based on date and time that the session was recorded
- If re-processing a previously recorded session, this value specifies which session to reprocess (must be the name of a folder within the
FreeMoCap_Datafolder)
-
stage- [Type] - Int
- [Default] - 1
- Which processing stage to start from. Processing stages are deined in more detail in #processing-stages
stage 1 - Record Raw Videos stage 2 - Synchronize Videos stage 3 - Camera Calibration stage 4 - 2d Tracking and 3d Calibration stage 5 - Create output files (using Blender) stage 6 - Create output animation (Matplotlib) -
useMediaPipe- [Type] - BOOL
- [Default] - False,
- Whether or not to use the MediaPipe tracking method in
stage=4
-
runMediaPipe- [Type] - BOOL
- [Default] - False,
- Whether or not to RUN the MediaPipe tracking method in
stage=4(will use previously processed data. This can save a lot of time when re-processing long videos)
-
useOpenPose- [Type] - BOOL
- [Default] - False,
- Whether or not to use the OpenPose tracking method in
stage=4
-
runOpenPose- [Type] - BOOL
- [Default] - False,
- Whether or not to RUN the OpenPose tracking method in
stage=4(will use previously processed data. This can save a lot of time when re-processing long videos)
-
useDeepLabCut- [Type] - BOOL
- [Default] - False,
- Whether or not to use the DeepLabCut model/project specified at
dlcConfigPathto track objects instage=4
-
setDataPath- [Type] - BOOL
- [Default] - False,
- Trigger the GUI that prompts user to specify location of
FreeMoCap_Data
-
userDataPath- [Type] - BOOL
- [Default] - False,
- path to the location of
FreeMoCap_Data
-
recordVid- [Type] - BOOL
- [Default] - False,
- wehether to save the matplotlib animation to an
.mp4file
-
showAnimation- [Type] - BOOL
- [Default] - False,
- wehether to save the matplotlib animation to an
.mp4file
-
reconstructionConfidenceThreshold- [Type] - float in range(0,1),
- [Default] - .7
- Threshold 'confidence' value to include a point in the 3d reconstruction step
-
charucoSquareSize- [Type] = int
- [Default] = 36,
- The size of a side of a black square in the Charuco board used in this calibration. The default value of 36 is approximately appropriate for a print out on an 8 in bu 10 in paper (US Letter, approx A4)
-
calVideoLength- [Type] = int, float in range (0,1), or [int, int]
- [Default] = .5,
- What portion of the videos to use in the Anipose calibration step in
stage=3.-1uses the whole recording, a number between 0 and 1 defines a proprotion of the video to use, and a tuple of two numbers defines the start and end frame
-
startFrame- [Type] = int
- [Default] = 0,
- what frame of the video to start the animation in
stage=6
-
useBlender- [Type] = BOOL
- [Default] = True,
- Whether to use Blender to create output
.blend,.fbx,.usd,and.gltffiles
-
resetBlenderExe- [Type] = BOOL
- [Default] = False,
- Whether to launch GUI to set Blender .exe path (usually something like
C:/Program Files/Blender Foundation/2.95/)
-
get_synced_unix_timestamps- [Type] = BOOL
- [Default] = True,
- Whether to save camera timestamps in
Unix Epoch Timein addition to the default 'counting up from zero' timestamps. Very helpful for synchronizing FreeMoCap with other softwares
-
good_clean_frame_number- [Type] = int
- [Default] = 0,
- A frame where the subject is standing in something like a T-pose or an A-pose, which will be used to scale the armature created via the
useBlender=Trueoption. If set to default (0) the software will attempt to locate this frame automatically by looking for a frame where all markers are visible with highconfidencevalues (but this is buggy)
-
bundle_adjust_3d_points[EXPERIMENTAL as of May 2022]- [Type] = BOOL
- [Default] = False,
- When set to
True, the system will run a bundle adjust optimization of all recorded 3d points produced instage=4usinganiposelib'soptim_pointsmethod. This takes a rather long time, but can signicantly clean up the resulting recordings. However,it may also "over smooth" the data. We're in the process of testing this method out now
-
Our calibration method relies on Anipose's Charuco-based calibration method to determine the location of each camera during a recording session. This information is later used to create the 3d reconstruction of the tracked points
-
IMPORTANT The Charuco board shown to the camera MUST be generated with the
cv2.aruco.DICT_4X4_250dictionary! -
Ah high resoultion
pngof this Charuco board is in this repository at/charuco_board_image_highRes.png -
- To generate your own board, use the following python commands (or equivalent). DO NOT CHANGE THE PARAMETERS OR THE CALIBRATION WILL NOT WORK:
import cv2 aruco_dict = cv2.aruco.Dictionary_get(cv2.aruco.DICT_4X4_250) #note `cv2.aruco` can be installed via `pip install opencv-contrib-python` board = cv2.aruco.CharucoBoard_create(7, 5, 1, .8, aruco_dict) charuco_board_image = board.draw((2000,2000)) #`2000` is the resolution of the resulting image. Increase this number if printing a large board (bigger is better! Esp for large spaces! cv2.imwrite('charuco_board_image.png',charuco_board_image)
- To generate your own board, use the following python commands (or equivalent). DO NOT CHANGE THE PARAMETERS OR THE CALIBRATION WILL NOT WORK:
Both Deeplabcut and OpenPose are technically supported, but both are rather under-tested at the moment.
-
To use DeepLabCut, install with set
freemocap.RunMe(useDLC=True)- Installation instructions for DeepLabCut may be found on their github - https://github.com/DeepLabCut/DeepLabCut
-
If you would like to use OpenPose for body tracking, install Cuda and the Windows Portable Demo of OpenPose and set
freemocap.RunMe(useOpenPose=True).-
Install CUDA: https://developer.nvidia.com/cuda-downloads
-
Install OpenPose (Windows Portable Demo): https://github.com/CMU-Perceptual-Computing-Lab/openpose/releases/tag/v1.6.0
-
Follow the GitHub Repository and/or Join the Discord (https://discord.gg/HX7MTprYsK) for updates!
β¨πβ¨