ALL DETAILS OF deeptracking project.
1.Graphics driver for navida
2.The default is python3 (the author uses python3 programming)
1# update-alternatives --list python 2update-alternatives: error: no alternatives for python
If the above error message appears, it means that an alternate version of Python has not been recognized by the update-alternatives command. Want to solve this problem, we need to update the alternative list, put python2.7 and python3.4 into it.
1# update-alternatives --install / usr / bin / python python /usr/bin/python2.7 1 2update-alternatives: using /usr/bin/python2.7 to provide / usr / bin / python (python) in auto mode 3# update-alternatives --install / usr / bin / python python /usr/bin/python3.4 2 4update-alternatives: using /usr/bin/python3.4 to provide / usr / bin / python (python) in auto mode
The --install option uses several parameters for creating symbolic links. The last parameter specifies the priority of this option. If we do not set alternative options manually, the one with the highest priority will be selected. In this example, we set the priority for /usr/bin/python3.4 to 2, so the update-alternatives command will automatically set it to the default Python version.
1# python --version 2Python 3.4.2 Next, we again list the available Python alternatives.
1# update-alternatives --list python 2/usr/bin/python2.7 3/usr/bin/python3.4 Now, we can use the command below at any time in the listed Python alternative version of the arbitrary switch.
1# sudo update-alternatives --config python set the python3 as default.
3.Install pip In the official website to download. Py file sudo python can install it to python3
4.install tqdm
5.numpy sudo apt-get install python3-numpy(the version can not be 1.14.0 or this can make the error showed bellow) 6.opengl and glfw installed in accordance with libfreenect2 Then : 6.1 Install libfreenect2 + py3freenect2 First of all, to install libfreenect2 Because of the subsequent installation process will use one of these very important library opengl and glfw (follow-up by pip install their respective python parser so that can be import in Python)
libfreenect2: https://github.com/OpenKinect/libfreenect2.git
py3freenect:https://github.com/MathGaron/py3freenect2.git
afer all do: pip install PyOpenGL PyOpenGL_accelerate pip install glfw 7.plyfile
pip install plyfile(the author recommend to use version 0.4 and you can install it by source !!!!!!) 8.scipy
sudo apt-get install python3-scipy 9.numpngw，pandas，skimage,slackclient
sudo pip install numpngw sudo pip install pandas sudo pip install scikit-image sudo pip install slackclient 10.PIL
sudo apt-get install python3-pil
11.opencv3.1.0(can not be opencv3.4 cause I failed many times !!!!!) Official website and this can be installed crrectly due to follow-up to install python-aruco (very tedious requirements are highly easy to install the wrong, I remember must be 1.3.0 version !!!!!!) (red is a required item) All in accordance with the official website document install in linux operation, in addition to cmake part
sudo apt-get install build-essential sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev cd ~ / opencv mkdir build cd build cmake -D CMAKE_BUILD_TYPE = RELEASE -D CMAKE_INSTALL_PREFIX = / usr / local -D BUILD_NEW_PYTHON_SUPPORT = ON -D PYTHON_EXECUTABLE = $ (which python3) .. make -j7 sudo make install
12.Python-aruco (error prone)
After installing opencv3.4 you need to install aruco1.3.0: The official website to download version 1.3.0 unzip then go into the package to install it according to readme, then follow the link to operate: https://github.com/fehlfarbe/python-aruco/tree/aruco1.3.0 Note that you can skip over numpy Installation If there is a problem in the implementation of swigbuild.sh In the issue to find the answer Eg EIGEN, arrayobject some of the first document can not find the issue in the issue of the answer, if not in accordance with the software installed on it will be incorrect, Be patient and try it several times, later may also need: export LD_LIBRARY_PATH = / usr / local / lib /: $ LD_LIBRARY_PATH can be imported in Python.
13.pytorch (because of my python3.4 so github source is installed) https://github.com/pytorch/pytorch.git
- Hugh Perkins's pytorch The general process is Figure 1, of course, will jump to install torch: https://github.com/torch/distro.git Click the link in Figure 2 to get it: http://torch.ch/docs/getting-started.html#_ Torch installed, and then return to the first figure in the process to go.
All dependencies are generally set up. Of course, the author provided the docker file, but the fatal error was that it could not be configured with python-aruco (tried n different builds).
Data augmentation: In order to rebuild a more realistic scene, the author made data augmentation to the acquired training data, which is based on randomly providing the background for the training data and the corresponding occlusion of the 3D virtual object. Since the author did not provide the corresponding data,so readers need to download it by yourself.
1)For 3-dimensional coverings:
You can download a random occlusion from http://graphics.stanford.edu/data/3Dscanrep/.
2)For background image: As the background image required to be consistent with the format of the pictures provided by the author(raw_training dataset), so we just selected sun3d database as mentioned in the author's paper, Readers can go to the official website: http://sun3d.cs.princeton.edu/ download
BTW,you also need special tools to download data sets from this website
At this point, the data required for data augmentation on the completion of the introduction.
Deeptracking code installation git clone (not need mention) After all the environment dependences are installed sudo python3 setup.py install
Code operation mode(required by the author) Since the author imported the data mainly by the json file when running the project, the code is usually run like this mode : python xxxx.py -c xxxx.json
1)json file settings First find the location of the relevant dataset combined with the author suggested in the json file to find the corresponding file link location.
2)Replace plyparser Since the author did not provide code for ply parsing , so readers need to replace the file bellow:
Transforms contain utility functions to manipulate pointclouds
date : 2017-20-03
__author__ = "Mathieu Garon"
__version__ = "0.0.1"
from plyfile import PlyData, PlyElement
import os
import numpy as np
from PIL import Image
class PlyParser:
def __init__(self, path):
self.path = path
self.data = PlyData.read(path)
def get_texture(self):
for comment in self.data.comments:
if "TextureFile" in comment:
tex_path = os.path.join(os.path.dirname(self.path), comment.split(" ")[-1])
return np.array(Image.open(tex_path)).astype(np.uint8)
return None
def get_vertex(self):
for element in self.data.elements:
if element.name == "vertex":
return PlyParser.recarray_to_array(element.data[["x", "y", "z"]], np.float32)
raise KeyError("No field vertex with x, y, z in ply file.")
def get_vertex_color(self):
for element in self.data.elements:
if element.name == "vertex":
try:
return PlyParser.recarray_to_array(element.data[["red", "green", "blue"]], np.uint8)
except ValueError:
break
raise KeyError("No field vertex with red, green, blue in ply file.")
def get_vertex_normals(self):
for element in self.data.elements:
if element.name == "vertex":
try:
return PlyParser.recarray_to_array(element.data[["nx", "ny", "nz"]], np.float32)
except ValueError:
break
raise KeyError("No field vertex with normals nx, ny, nz in ply file.")
def get_texture_coord(self):
for element in self.data.elements:
if element.name == "vertex":
try:
return PlyParser.recarray_to_array(element.data[["texture_u", "texture_v"]], np.float32)
except ValueError:
break
raise KeyError("No field vertex with texture coord 'texture_u' and 'texture_v' in ply file.")
def get_faces(self):
for element in self.data.elements:
if element.name == "face":
try:
faces_object = element.data["vertex_indices"]
except ValueError:
break
# take for granted that all faces are of same lenght
faces = np.ndarray((len(faces_object), len(faces_object[0])), dtype=np.uint32)
for i, face in enumerate(faces_object):
faces[i, :] = face
return faces
raise KeyError("No field face with vertex_indices.")
@staticmethod
def recarray_to_array(array, type):
return array.view(type).reshape(array.shape + (-1,))
@staticmethod
def save_points(points, path):
vertex = np.zeros(points.shape[0], dtype=([('x', 'f4'), ('y', 'f4'), ('z', 'f4')]))
vertex.fill(255)
vertex['x'] = points[:, 0]
vertex['y'] = points[:, 1]
vertex['z'] = points[:, 2]
el = PlyElement.describe(vertex, 'vertex')
PlyData([el], text=ascii).write(path)
So far all the basic operation of the project is to complete the introduction.
Implementation process According to the author's paper and ppt , we need to generate_synthetic_data.py, generate_real_data.py to generate the previous frame and current frame in the figure below, and then use the merge_datasets.py tools in deeptracking to mix them as a training set. Similarly, the same procedure can be used to generate the verification set.
generate_synthetic_data.py Running Results:
As shown in the figure, there will be overflow in the normal calculation in the dragon model. The resulting picture is very far from the paper. According to the paper, the picture should be a picture of the dragon with no background in Figure 1. For other models such as skull
generate_real_data.py Demonstration result:
There is nothing wrong, but you can see that this should be a complete 3d model but seems it produce less than ideal results.
train.py results:
Just to be a small demonstration, my padding data is very small but the machine actually collapsed. Just to prove that the environment is indeed built correctly it is only the hardware constraints led to the results are not satisfactory.
Since the validation function requires well-trained model result data, I have not been able to demonstrate any training so can not show the validate part:
2018-01-29 20:55:35 __main __ [INFO] train.py:222 Setup Datasets
2018-01-29 20:55:35 __main __ [INFO] train.py:99 Setup Train: / home / gaofei / dragon
2018-01-29 20:55:36 __main __ [INFO] train.py:106 Computed mean: [1.18879998 2.31146669 0.65359998 -304.39163208 3.55537772
4.312222 3.11995554 -305.23696899]
Computed Std: [0. 0. 0. 0. 2.48835552 2.70862198
2.3448 1.06384277]
2018-01-29 20:55:36 __main __ [INFO] train.py:107 Setup Valid: / home / gaofei / skull
2018-01-29 20:55:36 __main __ [INFO] train.py:225 Setup Model
2018-01-29 20:55:41 __main __ [DEBUG] train.py:227 Backend: cuda
nn.Sequential {
[input -> (1) -> (2) -> (3) -> (4) -> (5) ->
(1): nn.Sequential {
[- (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> 10) -> (12) -> (13) -> (14) -> (15) -> output]
(1): nn.ParallelTable {
input
| `-> (1): nn.Sequential {
| [input -> (1) -> (2) -> (3) -> (4) -> output]
| (1): nn.SpatialConvolution (4 -> 24, 5x5)
| (2): nn.SpatialBatchNormalization (4D) (24)
| (3): nn.ELU (alpha: 1.000000)
| (4): nn.SpatialMaxPooling (2x2, 2,2)
|}
`-> (2): nn.Sequential {
[input -> (1) -> (2) -> (3) -> (4) -> output]
(1): nn.SpatialConvolution (4 -> 24, 5x5)
(2): nn.SpatialBatchNormalization (4D) (24)
(3): nn.ELU (alpha: 1.000000)
(4): nn.SpatialMaxPooling (2x2, 2,2)
}
... -> output
}
(2): nn.JoinTable
(3): nn.SpatialConvolution (48 -> 48, 3x3)
(4): nn.SpatialBatchNormalization (4D) (48)
(5): nn.ELU (alpha: 1.000000)
(6): nn.SpatialMaxPooling (2x2, 2,2)
(7): nn.SpatialConvolution (48 -> 48, 3x3)
(8): nn.SpatialBatchNormalization (4D) (48)
(9): nn.ELU (alpha: 1.000000)
(10): nn.SpatialMaxPooling (2x2, 2,2)
(11): nn.SpatialConvolution (48 -> 48, 3x3)
(12): nn.SpatialBatchNormalization (4D) (48)
(13): nn.ELU (alpha: 1.000000)
(14): nn.SpatialMaxPooling (2x2, 2,2)
(15): nn.View (-1, 2352)
}
(2): nn.Dropout (0.5, busy)
(3): nn.Linear (2352 -> 50)
(4): nn.ELU (alpha: 1.000000)
(5): nn.Linear (50 -> 6)
(6): nn.Tanh
}
2018-01-29 20:55:41 __main __ [SLACK] train.py:236 Train start at 2018/01/29 20:55:41
2018-01-29 20:55:43 __main __ [INFO] train.py:180 [100%]: Train loss: nan
2018-01-29 20:55:43 __main __ [INFO] train.py:182 Time / batch: 0.0005429645379384358h
2018-01-29 20:55:43 __main __ [SLACK] train.py:243 [Epoch 0] Train loss: 0 Val loss: nan
2018-01-29 20:55:43 __main __ [INFO] train.py:180 [100%]: Train loss: nan
2018-01-29 20:55:43 __main __ [INFO] train.py:182 Time / batch: 3.4365852673848473e-05h
2018-01-29 20:55:43 __main __ [SLACK] train.py:243 [Epoch 1] Train loss: 0 Val loss: nan
2018-01-29 20:55:43 __main __ [INFO] train.py:180 [100%]: Train loss: nan
2018-01-29 20:55:43 __main __ [INFO] train.py:182 Time / batch: 2.897653314802382e-05h
2018-01-29 20:55:43 __main __ [SLACK] train.py:243 [Epoch 2] Train loss: 0 Val loss: nan
2018-01-29 20:55:44 __main __ [INFO] train.py:180 [100%]: Train loss: nan
2018-01-29 20:55:44 __main __ [INFO] train.py:182 Time / batch: 3.191981050703261e-05h
2018-01-29 20:55:44 __main __ [SLACK] train.py:243 [Epoch 3] Train loss: 0 Val loss: nan
2018-01-29 20:55:44 __main __ [INFO] train.py:180 [100%]: Train loss: nan
2018-01-29 20:55:44 __main __ [INFO] train.py:182 Time / batch: 3.512461980183919e-05h
2018-01-29 20:55:44 __main __ [SLACK] train.py:243 [Epoch 4] Train loss: nan Val loss: nan
2018-01-29 20:55:44 __main __ [INFO] train.py:180 [100%]: Train loss: nan
2018-01-29 20:55:44 __main __ [INFO] train.py:182 Time / batch: 2.8053919474283855e-05h
2018-01-29 20:55:44 __main __ [SLACK] train.py:243 [Epoch 5] Train loss: nan Val loss: nan
2018-01-29 20:55:44 __main __ [SLACK] train.py:256 Train Terminated at 2018/01/29 20:55:44
2018-01-29 20:55:44 __main __ [SLACK] train.py:257 Total Epoch: 0
Best Validation Loss: 1000
Training results can be seen up (text)and it is also the needs the in the up picture. And tools can be generated tools csv documents will be displayed:
As you can see, occlution can also be used as a verification part because it is also a video file, so it matches what the author says to accurately track the occluded objects in real time with 6 free-throws, like a pair of fluoroscopy eyes. But my graphics card generated all the jagged things pictured here, which should produce the same thing as the one below. As to why there is such a problem, my opinion is caused by the computing power of the graphics card. The author's opinion is a code issue. Currently, he is carrying out the second phase of upgrade and is also handling my problem. https://github.com/lvsn/deeptracking/issues/9
Please refer to my questions on the code in the project on the issue of the code, you can see the specific process of each step, mainly the graphics calculated a lot of non and beneficial phenomenon.
At this point all the code is introduced, the main environment is too difficult to build. I am currently looking for ways to build a docker image that frees me from the hassles of building environments. Thanks to this project, I have conceptualized the process of implementing a specific project, and because the results are suboptimal
My original, declined reproduced, if necessary, contact the author: Myzhencai@hotmail.com