Measurements from Body Meshes

[kristijanbartol]

In the past few months, I have been using SMPL mesh estimation to extract body measurements. It seems that body measurement is still not a hot topic compared to mesh regression* (my guess is that image-based shape estimation methods are still not accurate enough to even go one step further). However, I believe it would be interesting to be able to conveniently obtain body measurements once you have your body model parameters.

I already have some specific private implementations. Would we be interested in this feature (maybe towards some other major release >v1.0)?

References*

The following is the list of papers that estimate body models from images or other data in order to extract body measurements.

Body measurement based on SMPL mesh estimation

[1] B. M. Smith, V. Chari, A. Agrawal, J. M. Rehg and R. Sever, "Towards Accurate 3D Human Body Reconstruction from Silhouettes," 2019 International Conference on 3D Vision (3DV), 2019, pp. 279-288, doi: 10.1109/3DV.2019.00039.
[2] E. Dibra, H. Jain, C. Öztireli, R. Ziegler and M. Gross, "Human Shape from Silhouettes Using Generative HKS Descriptors and Cross-Modal Neural Networks," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5504-5514, doi: 10.1109/CVPR.2017.584.
[3] Dibra, E.; Jain, H.; Öztireli, C.; Ziegler, R.; Gross, M. HS-Nets: Estimating Human Body Shape from Silhouettes with Convolutional Neural Networks. In Proceedings of the 2016 Fourth International Conference on 3D Vision (3DV), Stanford, CA, USA, 25–28 October 2016; pp. 108–117.
[4] Pujades, S.; Mohler, B.J.; Thaler, A.; Tesch, J.; Mahmood, N.; Hesse, N.; Bülthoff, H.H.; Black, M.J. The Virtual Caliper: Rapid Creation of Metrically Accurate Avatars from 3D Measurements. IEEE Trans. Vis. Comput. Graph. 2019, 25, 1887–1897.
[5] Yan, S.; Wirta, J.; Kämäräinen, J.K. Silhouette Body Measurement Benchmarks. In Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy, 10–15 January 2021; pp. 7804–7809.
[6] Bartol, K.; Bojanić, D.; Petković, T.; Peharec, S.; Pribanić, T. Linear Regression vs. Deep Learning: A Simple Yet Effective Baseline for Human Body Measurement. Sensors 2022, 22, 1885. https://doi.org/10.3390/s22051885

In particular, note that this paper [7] Sengupta, A. et al. “Probabilistic Estimation of 3D Human Shape and Pose with a Semantic Local Parametric Model.” BMVC (2021) even specifies the indices corresponding to SMPL vertices.

Mesh regression models

SMPLify-X, ExPose, VIBE, and all other very popular papers (compared to the above ones).

[caizhongang]

@kristijanbartol Definitely YES, the community would love your contribution! I think body measurement may be formulated as accurate shape estimation. Hence, I find this CVPR 2022 Best Paper Candidate [8] to be the latest relevant work.

[8] Choutas, V., Müller, L., Huang, C. P., Tang, S., Tzionas, D., Black, M. J., Accurate 3D Body Shape Regression using Metric and Semantic Attributes, CVPR 2022

[kristijanbartol]

That's awesome! I would simply start with being able to extract the measurements from body mesh vertices, regardless of the method that uses these measurements. There will be a lot of fixed indices for body measurement extraction, especially due to different body models. This might even be problematic if we would want to add even more models such as STAR. That's why I opened a discussion to see how to best tackle this.

[DavidBoja]

Hi,

I would like to continue what @kristijanbartol started.

I already implemented an SMPL measurer for my purposes here. The measurements are implemented as either lengths or circumferences. The lengths are computed as landmark distances defined on the SMPL body, whereas the circumferences are implemented as plane cuts of the SMPL body with planes at pre-defined locations (you can take a look at the Figure on the repo i sent). This is similar to the mentioned paper [8], with the addition of having more measurements defined, and a more general definition of circumference measurements — whereas [8] only cut the SMPL body with planes parallel to the floor, my implementation can cut the SMPL body with arbitrary planes.

Additionally the measuring can be done for an SMPL body model defined by either its shape parameters or by its vertices. This is convenient if additional optimization has been done to the SMPL vertices, after which they do not correspond fully to a shape parameter set.

I would like to do something very similar here with the addition of being model agnostic (smpl/smplx/star).

I went through the codebase and some general steps would be:

1. Add anthropometry folder to mmhuman3d/core/conventions with all the measurement definitions
2. Add landmark_mapping to mmhuman3d/core/conventions with landmarks definitions for each body model
3. Add face segmentations to mmhuman3d/core/conventions/segmentation for each body model
4. Add measure.py code that actually measures the bodies somwhere?? — not sure where?
5. create demo.py in mmhuman3d/demo
6. create tests in mmhuman3d/tests
7. add visualization scripts in mmhuman3d/core/visualization

Does that sound about right?

Before starting, however, I have a couple of questions:

1. Can you explain why is the smpl segmetnation in this repo mmhuman3d/core/conventions/segmentation/smpl.py
 different than the original segmentation here?
2. Where should I put the code that measures the body models? Maybe create an anthropometry folder within the models mmhuman3d/models/anthropometry ?
3. As far I can tell, there are no 3D visualizations here. My code currently makes use of the plotly package to create a 3D plot that opens in the browser (see Fig in the repo i sent). Do you have any advice for this -- do I just add the plotly dependency, or do i just render everything into 2D? Visualizations are convenient, but not necessary, so this step might be omitted.

Let me know what you think :).

[8] Choutas, V., Müller, L., Huang, C. P., Tang, S., Tzionas, D., Black, M. J., Accurate 3D Body Shape Regression using Metric and Semantic Attributes, CVPR 2022

[DavidBoja]

@caizhongang Do you have any thoughts on this?

[DavidBoja]

@caizhongang Do you have any thoughts on this?

[caizhongang]

@DavidBoja , thank you very much for your proposal and sorry for the late reply (we were quite occupied working on something that we hope to be very useful to the community).

Your project looks amazing!

---

1. Add anthropometry folder to <ins>mmhuman3d/core/conventions</ins> with all the measurement definitions
2. Add landmark_mapping to <ins>mmhuman3d/core/conventions</ins> with landmarks definitions for each body model
3. Add face segmentations to <ins>mmhuman3d/core/conventions/segmentation</ins> for each body model
5. Add measure.py code that actually measures the bodies somwhere?? — not sure where?
6. create demo.py in <ins>mmhuman3d/demo</ins>
7. create tests in <ins>mmhuman3d/tests</ins>
8. add visualization scripts in <ins>mmhuman3d/core/visualization</ins>

I think these steps looks great, and for 5, I think 'mmhuman3d/core/evaluation/‘ will be the right place. Later we will also extend it to SMPL-X for evaluation results on datasets such as HBW. As for 8, the actual entry point (main function) of visualization may be put together with the demo script (probably as an argparse item), that calls your function in mmhuman3d/core/visualization/

---

1. Can you explain why is the smpl segmetnation in this repo <ins>mmhuman3d/core/conventions/segmentation/smpl.py</ins>
 different than the original segmentation [here](https://meshcapade.wiki/assets/SMPL_body_segmentation/smpl/smpl_vert_segmentation.json)?

@WenjiaWang0312 do you have any ideas?

2. Where should I put the code that measures the body models? Maybe create an anthropometry folder within the models <ins>mmhuman3d/models/anthropometry</ins> ?

I think this sounds good.

3. As far I can tell, there are no 3D visualizations here. My code currently makes use of the plotly package to create a 3D plot that opens in the browser (see Fig in the repo i sent). Do you have any advice for this -- do I just add the plotly dependency, or do i just render everything into 2D? Visualizations are convenient, but not necessary, so this step might be omitted.

We are working on a visualization tool based on node.js (for interactive visualization). Would you mind leaving the visualization part to us?

---

Again, I'm sure the community will look forward to your implementation!

[DavidBoja]

Great! Thanks for the inputs.

1. Waiting on @WenjiaWang0312 for the smpl segmentation :)
2. I will put the code that actually measures the body models in mmhuman3d/models/anthropometry then
3. I'll also add a function that evaluates the mean absolute error for 2 sets of body measurements (ground truth and estimated measurements for example) in mmhuman3d/core/evaluation/
4. Sure thing - I'll let the 3D visualizations to you. I will think about wether 2D visualizations and rendering makes sense after I have a working measuring code.

Thanks again!

[WenjiaWang0312]

Hi, there. The segmentation is the same. I store continuous indexes to a list of [start, end] to save space and make the .py file simpler. With a function to decode it, we commonly only need to compute once.