About SCAIL pose Location Drift

[rex-29]

Hi! Thanks for your great work!!! I am testing some cases with half body image and full body driving video, then i got some result as below

The pose generate locate at the bottom of video, is there any solution for this? THX!

[teal024]

Because we manually set the camera movement bounds, when the aspect ratios of two images differ too much—such as a half-body shot versus a full-body shot—the movement can hit the upper limit of the bounds, causing abnormal behavior. The original bounds assume that the driving video and the ref frame has similar propotions, but users can manually adjust the bounds to cater for such scenarios.
For example, change line 125 of https://github.com/zai-org/SCAIL-Pose/blob/master/NLFPoseExtract/align3d.py to result = minimize(objective, initial_params, bounds=[(0.2, 5), (0.2, 5), (-imshape[1], imshape[1]), (-imshape[0], imshape[0])]) may just fix that issue.

[rex-29]

Because we manually set the camera movement bounds, when the aspect ratios of two images differ too much—such as a half-body shot versus a full-body shot—the movement can hit the upper limit of the bounds, causing abnormal behavior. The original bounds assume that the driving video and the ref frame has similar propotions, but users can manually adjust the bounds to cater for such scenarios. For example, change line 125 of https://github.com/zai-org/SCAIL-Pose/blob/master/NLFPoseExtract/align3d.py to result = minimize(objective, initial_params, bounds=[(0.2, 5), (0.2, 5), (-imshape[1], imshape[1]), (-imshape[0], imshape[0])]) may just fix that issue.

Thanks a lot! It's indeed solved some cases!
Except the equation you mentioned above, I found we can change FOV value in def intrinsic_matrix_from_field_of_view, it can change the pose's depth of the video, can we set specific value for different case?

[rex-29]

intrinsic_matrix_from_field_of_view

And I found the face and hands will drift after i change the fov degrees

[teal024]

intrinsic_matrix_from_field_of_view

And I found the face and hands will drift after i change the fov degrees

I guess the drift is because of 2 factors: (a). changing of fov only moves the 3D skeleton but keep the face/hands unchanged (b). original face/hands offset in shift_dwpose_according_to_nlf (https://github.com/zai-org/SCAIL-Pose/blob/master/NLFPoseExtract/nlf_render.py#L27) calculated from on the projected 2d offset after 3D-align(solving new 3D Cam parameters) is based on the original fov. If you want the hand to be attached to the forearm i think there the offset should be calculated based on both the offset due to changing of fov and 3D-align offset caculated on new fovs.

[rex-29]

intrinsic_matrix_from_field_of_view

And I found the face and hands will drift after i change the fov degrees

I guess the drift is because of 2 factors: (a). changing of fov only moves the 3D skeleton but keep the face/hands unchanged (b). original face/hands offset in shift_dwpose_according_to_nlf (https://github.com/zai-org/SCAIL-Pose/blob/master/NLFPoseExtract/nlf_render.py#L27) calculated from on the projected 2d offset after 3D-align(solving new 3D Cam parameters) is based on the original fov. If you want the hand to be attached to the forearm i think there the offset should be calculated based on both the offset due to changing of fov and 3D-align offset caculated on new fovs.

Which means I need to run both default fovs and new fovs to get the offsets, then use the offset map to hands and face?

[teal024]

Which means I need to run both default fovs and new fovs to get the offsets, then use the offset map to hands and face?

I think so

[rex-29]

Which means I need to run both default fovs and new fovs to get the offsets, then use the offset map to hands and face?

I think so

Use the offset solve the problem well for me. Thanks!

[rex-29]

Sometimes I got result like this, is it caused by taichi_cylinder?