Skip to content

Commit 2bbd6f0

Browse files
goldbattlegoldbattle
authored
Merge pull request #186 from rpng/develop_v2.4
Development v2.4 - Camera classes, Masks, Formatting and Smaller Fixes
2 parents c6f7e94 + 1cbde30 commit 2bbd6f0

File tree

144 files changed

+92637
-1589
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

144 files changed

+92637
-1589
lines changed

‎.gitignore

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
cmake-build-debug
1+
cmake-build-*
22
build
33
Build
44
*.*~

‎Dockerfile

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
FROM osrf/ros:noetic-desktop-full
2+
3+
# =========================================================
4+
# =========================================================
5+
6+
# Are you are looking for how to use this docker file?
7+
# - https://docs.openvins.com/dev-docker.html
8+
# - https://docs.docker.com/get-started/
9+
# - http://wiki.ros.org/docker/Tutorials/Docker
10+
11+
# =========================================================
12+
# =========================================================
13+
14+
# Dependencies we use, catkin tools is very good build system
15+
# Also some helper utitiles for fast in terminal edits (nano etc)
16+
RUN apt-get update && apt-get install -y libeigen3-dev nano git cmake
17+
RUN sudo apt-get install -y python3-catkin-tools python3-osrf-pycommon
18+
19+
# Install deps needed for clion remote debugging
20+
# https://blog.jetbrains.com/clion/2020/01/using-docker-with-clion/
21+
RUN apt-get update && apt-get install -y ssh build-essential gcc g++ \
22+
gdb clang cmake rsync tar python && apt-get clean
23+
RUN ( \
24+
echo 'LogLevel DEBUG2'; \
25+
echo 'PermitRootLogin yes'; \
26+
echo 'PasswordAuthentication yes'; \
27+
echo 'Subsystem sftp /usr/lib/openssh/sftp-server'; \
28+
) > /etc/ssh/sshd_config_test_clion \
29+
&& mkdir /run/sshd
30+
RUN useradd -m user && yes password | passwd user
31+
RUN usermod -s /bin/bash user
32+
CMD ["/usr/sbin/sshd", "-D", "-e", "-f", "/etc/ssh/sshd_config_test_clion"]
33+

‎ReadMe.md

+3-1
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ Please take a look at the feature list below for full details on what the system
1818

1919
## News / Events
2020

21+
* **July 19, 2021** - Camera classes, masking support, alignment utility, and other small fixes. See v2.4 [PR#117](https://github.com/rpng/open_vins/pull/186) for details.
2122
* **December 1, 2020** - Released improved memory management, active feature pointcloud publishing, limiting number of features in update to bound compute, and other small fixes. See v2.3 [PR#117](https://github.com/rpng/open_vins/pull/117) for details.
2223
* **November 18, 2020** - Released groundtruth generation utility package, [vicon2gt](https://github.com/rpng/vicon2gt) to enable creation of groundtruth trajectories in a motion capture room for evaulating VIO methods.
2324
* **July 7, 2020** - Released zero velocity update for vehicle applications and direct initialization when standing still. See [PR#79](https://github.com/rpng/open_vins/pull/79) for details.
@@ -62,9 +63,10 @@ Please take a look at the feature list below for full details on what the system
6263
* Stereo camera
6364
* Binocular camera
6465
* KLT or descriptor based
66+
* Masked tracking
6567
* Static IMU initialization (sfm will be open sourced later)
6668
* Zero velocity detection and updates
67-
* Out of the box evaluation on EurocMav and TUM-VI datasets
69+
* Out of the box evaluation on EurocMav, TUM-VI, UZH-FPV, KAIST Urban and VIO datasets
6870
* Extensive evaluation suite (ATE, RPE, NEES, RMSE, etc..)
6971

7072

‎docs/bib/extra.bib

+13
Original file line numberDiff line numberDiff line change
@@ -216,3 +216,16 @@ @inproceedings{Davidson2009ENC
216216
year={2009},
217217
url = {http://www.tkt.cs.tut.fi/research/nappo_files/1_C2.pdf},
218218
}
219+
220+
@article{Jeon2021RAL,
221+
title={Run your visual-inertial odometry on NVIDIA Jetson: Benchmark tests on a micro aerial vehicle},
222+
author={Jeon, Jinwoo and Jung, Sungwook and Lee, Eungchang and Choi, Duckyu and Myung, Hyun},
223+
journal={IEEE Robotics and Automation Letters},
224+
volume={6},
225+
number={3},
226+
pages={5332--5339},
227+
year={2021},
228+
publisher={IEEE}
229+
}
230+
231+

‎docs/dev-docker.dox

+194
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
/**
2+
3+
4+
@page dev-docker Developing with Docker
5+
@tableofcontents
6+
7+
8+
@section dev-docker-install Installing Docker
9+
10+
This will differ on which operating system you have installed, this guide is for linux-based systems.
11+
Please take a look at the official Docker [Get Docker](https://docs.docker.com/get-docker/) guide.
12+
There is also a guide from ROS called [getting started with ROS and Docker](http://wiki.ros.org/docker/Tutorials/Docker).
13+
On Ubuntu one should be able to do the following to get docker:
14+
15+
@code{.shell-session}
16+
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
17+
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
18+
sudo apt-get update
19+
sudo apt-get install docker-ce docker-ce-cli containerd.io
20+
@endcode
21+
22+
From there we can install [NVIDIA Container Toolkit](https://github.com/NVIDIA/nvidia-docker) to allow for the docker to use our GPU and for easy GUI pass through.
23+
You might also want to check out [this](https://roboticseabass.wordpress.com/2021/04/21/docker-and-ros/) blogpost for some more details.
24+
25+
@code{.shell-session}
26+
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
27+
&& curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \
28+
&& curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
29+
sudo apt-get update
30+
sudo apt-get install -y nvidia-docker2
31+
sudo systemctl restart docker
32+
sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi #to verify install
33+
@endcode
34+
35+
From this point we should be able to "test" that everything is working ok.
36+
First on the host machine we need to allow for x11 windows to connect.
37+
38+
@code{.shell-session}
39+
xhost +
40+
@endcode
41+
42+
We can now run the following command which should open gazebo GUI on your main desktop window.
43+
44+
@code{.shell-session}
45+
docker run -it --net=host --gpus all \
46+
--env="NVIDIA_DRIVER_CAPABILITIES=all" \
47+
--env="DISPLAY" \
48+
--env="QT_X11_NO_MITSHM=1" \
49+
--volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \
50+
osrf/ros:noetic-desktop-full \
51+
bash -it -c "roslaunch gazebo_ros empty_world.launch"
52+
@endcode
53+
54+
Alternatively we can launch directly into a bash shell and run commands from in there.
55+
This basically gives you a terminal in the docker container.
56+
57+
@code{.shell-session}
58+
docker run -it --net=host --gpus all \
59+
--env="NVIDIA_DRIVER_CAPABILITIES=all" \
60+
--env="DISPLAY" \
61+
--env="QT_X11_NO_MITSHM=1" \
62+
--volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \
63+
osrf/ros:noetic-desktop-full \
64+
bash
65+
# you should be able to launch rviz once in bash
66+
rviz
67+
@endcode
68+
69+
70+
@section dev-docker-openvins Running OpenVINS with Docker
71+
72+
Clone the OpenVINS repository, build the container and then launch it.
73+
The [Dockerfile](https://github.com/rpng/open_vins/blob/master/Dockerfile) will not build the repo by default, thus you will need to build the project.
74+
75+
76+
@m_class{m-block m-warning}
77+
78+
@par Use a Workspace Directory Mount
79+
Here it is important to note that we are going to create a dedicated ROS *workspace* which will then be loaded into the workspace.
80+
Thus if you are going to develop packages alongside OpenVINS you would make sure you have cloned your source code into the same workspace.
81+
The workspace local folder will be mounted to `/catkin_ws/` in the docker, thus all changes from the host are mirrored.
82+
83+
@code{.shell-session}
84+
mkdir -p ~/workspace/catkin_ws_ov/src
85+
cd ~/workspace/catkin_ws_ov/src
86+
git clone https://github.com/rpng/open_vins.git
87+
cd open_vins
88+
docker build -t openvins .
89+
cd ~/workspace/catkin_ws_ov
90+
@endcode
91+
92+
If the dockerfile breaks, you can remove the image and reinstall using the above
93+
94+
@code{.shell-session}
95+
docker image list
96+
docker image rm openvins --force
97+
@endcode
98+
99+
From here it is a good idea to create a nice helper command which will launch the docker and also pass the GUI to your host machine.
100+
Here you can append it to the bottom of the ~/.bashrc so that we always have it on startup or just run the two commands on each restart
101+
102+
@m_class{m-block m-warning}
103+
104+
@par Directory Binding
105+
You will need to specify *absolute directory paths* to the workspace and dataset folders on the host you want to bind.
106+
Bind mounts are used to ensure that the host directory is directly used and all edits made on the host are sync'ed with
107+
the docker container. See the docker [bind mounts](https://docs.docker.com/storage/bind-mounts/) documentation. You
108+
can add and remove mounts from this command as you see the need.
109+
110+
@code{.shell-session}
111+
nano ~/.bashrc
112+
# add to the bashrc file
113+
xhost + &> /dev/null
114+
export DOCKER_CATKINWS=/home/username/workspace/catkin_ws_ov
115+
export DOCKER_DATASETS=/home/username/datasets
116+
alias ov_docker="docker run -it --net=host --gpus all \
117+
--env=\"NVIDIA_DRIVER_CAPABILITIES=all\" --env=\"DISPLAY\" \
118+
--env=\"QT_X11_NO_MITSHM=1\" --volume=\"/tmp/.X11-unix:/tmp/.X11-unix:rw\" \
119+
--mount type=bind,source=$DOCKER_CATKINWS,target=/catkin_ws \
120+
--mount type=bind,source=$DOCKER_DATASETS,target=/datasets \
121+
openvins $1"
122+
# save and exit
123+
source ~/.bashrc
124+
@endcode
125+
126+
Now we can launch RVIZ and also compile the OpenVINS codebase.
127+
From two different terminals on the host machine one can run the following:
128+
129+
@code{.shell-session}
130+
ov_docker roscore
131+
ov_docker rosrun rviz rviz -d /catkin_ws/src/open_vins/ov_msckf/launch/display.rviz
132+
ov_docker bash
133+
@endcode
134+
135+
Now once inside the docker with the bash shell we can build and launch an example simulation:
136+
137+
@code{.shell-session}
138+
cd catkin_ws
139+
catkin build
140+
source devel/setup.bash
141+
roslaunch ov_msckf pgeneva_sim.launch
142+
@endcode
143+
144+
@m_class{m-block m-danger}
145+
146+
@par Real-time Performance
147+
On my machine running inside of the docker container is not real-time in nature. I am not sure why this is the case
148+
if someone knows if something is setup incorrectly please open a github issue. Thus it is recommended to only use
149+
the "serial" nodes which allows for the same parameters to be used as when installing directly on an OS.
150+
151+
152+
@section dev-docker-clion Using Jetbrains Clion and Docker
153+
154+
Jetbrains provides some instructions on their side and a youtube video.
155+
Basically, Clion needs to be configured to use an external compile service and this service
156+
needs to be exposed from the docker container.
157+
I still recommend users compile with `catkin build` directly in the docker, but this will allow for debugging and syntax insights.
158+
159+
- https://blog.jetbrains.com/clion/2020/01/using-docker-with-clion/
160+
- https://www.youtube.com/watch?v=h69XLiMtCT8
161+
162+
After building the OpenVINS image (as above) we can do the following which will start a detached process in the docker.
163+
This process will allow us to connect Clion to it.
164+
165+
@code{.shell-session}
166+
docker run -d --cap-add sys_ptrace -p127.0.0.1:2222:22 --name clion_remote_env openvins
167+
@endcode
168+
169+
We can now change Clion to use the docker remote:
170+
171+
1. In short, you should add a new Toolchain entry in settings under Build, Execution, Deployment as a Remote Host type.
172+
2. Click in the Credentials section and fill out the SSH credentials we set-up in the Dockerfile
173+
- Host: localhost
174+
- Port: 2222
175+
- Username: user
176+
- Password: password
177+
3. Add a CMake profile that uses this toolchain and you’re done.
178+
4. Change build target to be this new CMake profile (optionally just edit / delete the default)
179+
180+
To add support for ROS you will need to manually set environmental variables in the CMake profile.
181+
These were generated by going into the ROS workspace, building a package, and then looking at `printenv` output.
182+
It should be under `Settings > Build,Execution,Deployment > CMake > (your profile) > Environment`.
183+
This might be a brittle method, but not sure what else to do... (also see [this](https://www.allaban.me/posts/2020/08/ros2-setup-ide-docker/) blog post).
184+
185+
@code{.shell-session}
186+
LD_PATH_LIB=/catkin_ws/devel/lib:/opt/ros/noetic/lib;PYTHON_EXECUTABLE=/usr/bin/python3;PYTHON_INCLUDE_DIR=/usr/include/python3.8;ROS_VERSION=1;CMAKE_PREFIX_PATH=/catkin_ws/devel:/opt/ros/noetic;LD_LIBRARY_PATH=/catkin_ws/devel/lib:/opt/ros/noetic/lib;PATH=/opt/ros/noetic/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;PKG_CONFIG_PATH=/catkin_ws/devel/lib/pkgconfig:/opt/ros/noetic/lib/pkgconfig;PYTHONPATH=/opt/ros/noetic/lib/python3/dist-packages;ROSLISP_PACKAGE_DIRECTORIES=/catkin_ws/devel/share/common-lisp;ROS_PACKAGE_PATH=/catkin_ws/src/open_vins/ov_core:/catkin_ws/src/open_vins/ov_data:/catkin_ws/src/open_vins/ov_eval:/catkin_ws/src/open_vins/ov_msckf:/opt/ros/noetic/share
187+
@endcode
188+
189+
When you build in Clion you should see in `docker stats` that the `clion_remote_env` is building the files and maxing out the CPU during this process.
190+
Clion should send the source files to the remote server and then on build should build and run it remotely within the docker container.
191+
A user might also want to edit `Build,Execution,Deployment > Deployment` settings to exclude certain folders from copying over.
192+
See this [jetbrains documentation page](https://www.jetbrains.com/help/clion/remote-projects-support.html) for more details.
193+
194+
*/

‎docs/dev-welcome.dox

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,11 @@
44
@page dev-welcome Internals and Developers
55

66
- @subpage dev-coding-style --- General coding styles and conventions
7+
- @subpage dev-docker --- How to use docker images to develop with
78
- @subpage dev-docs --- Developer guide on how documentation can be built
89
- @subpage dev-index --- Description of the covariance index system
910
- @subpage dev-roadmap --- Where we plan to go in the future
1011
- @subpage dev-profiling --- Some notes on performing profiling
1112

1213

13-
1414
*/

‎docs/gs-datasets.dox

+36-11
Original file line numberDiff line numberDiff line change
@@ -157,6 +157,15 @@ The dataset was collected in Korea with a vehicle equipped with stereo camera pa
157157
The camera is 10 Hz, while the Xsens IMU is 100 Hz sensing rate.
158158
A groundtruth "baseline" trajectory is also provided which is the resulting output from fusion of the FoG, RKT GPS, and wheel encoders.
159159

160+
@m_class{m-block m-warning}
161+
162+
@par Dynamic Environments
163+
A challenging open research question is being able to handle dynamic objects seen from the cameras.
164+
By default we rely on our tracking 8 point RANSAC to handle these dynamics objects.
165+
In the most of the KAIST datasets the majority of the scene can be taken up by other moving vehicles, thus the performance can suffer.
166+
Please be aware of this fact.
167+
168+
160169
@code{.shell-session}
161170
git clone https://github.com/rpng/file_player.git
162171
git clone https://github.com/irapkaist/irp_sen_msgs.git
@@ -166,18 +175,8 @@ rosrun file_player file_player
166175

167176
To use the dataset, the dataset's file player can be used to publish the sensor information on to ROS.
168177
See the above commands on what packages you need to clone into your ROS workspace.
169-
One can record a rosbag after manually and use the serial OpenVINS processing node, or use the live node and manually playback the datasets.
170178
It is important to *disable* the "skip stop section" to ensure that we have continuous sensor feeds.
171-
Typically we process the datasets at 2x rate so we get a 20 Hz image feed and the datasets can be processed in a more efficient manor.
172-
173-
@m_class{m-block m-warning}
174-
175-
@par Dynamic Environments
176-
A challenging open research question is being able to handle dynamic objects seen from the cameras.
177-
By default we rely on our tracking 8 point RANSAC to handle these dynamics objects.
178-
In the most of the KAIST datasets the majority of the scene can be taken up by other moving vehicles, thus the performance can suffer.
179-
Please be aware of this fact.
180-
179+
Typically we process the datasets at 1.5x rate so we get a ~20 Hz image feed and the datasets can be processed in a more efficient manor.
181180

182181
@m_div{m-text-center}
183182
| Dataset Name | Length (km) | Dataset Link | Groundtruth Traj. | Example Launch |
@@ -188,6 +187,32 @@ Typically we process the datasets at 2x rate so we get a 20 Hz image feed and th
188187
@m_enddiv
189188

190189

190+
@section gs-data-kaist-vio KAIST VIO Dataset
191+
192+
The [KAIST VIO dataset](https://github.com/zinuok/kaistviodataset) @cite Jeon2021RAL is a dataset of a MAV in an indoor 3.15 x 3.60 x 2.50 meter environment which undergoes various trajectory motions.
193+
The camera is intel realsense D435i 25 Hz, while the IMU is 100 Hz sensing rate from the pixelhawk 4 unit.
194+
A groundtruth "baseline" trajectory is also provided from a OptiTrack Mocap system at 50 Hz, the bag files have the marker body frame to IMU frame already applied.
195+
This topic has been provided in ov_data for convinces sake.
196+
197+
@m_div{m-text-center}
198+
| Dataset Name | Length (km) | Dataset Link | Groundtruth Traj. | Example Launch |
199+
|-------------:|--------|--------------|------------------|------------------|
200+
| circle | 29.99 | [download](https://urserver.kaist.ac.kr/publicdata/KAIST_VIO_Dataset/circle/circle.bag) | [link](https://github.com/rpng/open_vins/tree/master/ov_data/kaist_vio) | [launch](https://github.com/rpng/open_vins/blob/master/ov_msckf/launch/pgeneva_ros_kaistvio.launch) |
201+
| circle_fast | 64.15 | [download](https://urserver.kaist.ac.kr/publicdata/KAIST_VIO_Dataset/circle/circle_fast.bag) | [link](https://github.com/rpng/open_vins/tree/master/ov_data/kaist_vio) | [launch](https://github.com/rpng/open_vins/blob/master/ov_msckf/launch/pgeneva_ros_kaistvio.launch) |
202+
| circle_head | 35.05 | [download](https://urserver.kaist.ac.kr/publicdata/KAIST_VIO_Dataset/circle/circle_head.bag) | [link](https://github.com/rpng/open_vins/tree/master/ov_data/kaist_vio) | [launch](https://github.com/rpng/open_vins/blob/master/ov_msckf/launch/pgeneva_ros_kaistvio.launch) |
203+
| infinite | 29.35 | [download](https://urserver.kaist.ac.kr/publicdata/KAIST_VIO_Dataset/infinite/infinite.bag) | [link](https://github.com/rpng/open_vins/tree/master/ov_data/kaist_vio) | [launch](https://github.com/rpng/open_vins/blob/master/ov_msckf/launch/pgeneva_ros_kaistvio.launch) |
204+
| infinite_fast | 54.24 | [download](https://urserver.kaist.ac.kr/publicdata/KAIST_VIO_Dataset/infinite/infinite_fast.bag) | [link](https://github.com/rpng/open_vins/tree/master/ov_data/kaist_vio) | [launch](https://github.com/rpng/open_vins/blob/master/ov_msckf/launch/pgeneva_ros_kaistvio.launch) |
205+
| infinite_head | 37.45 | [download](https://urserver.kaist.ac.kr/publicdata/KAIST_VIO_Dataset/infinite/infinite_head.bag) | [link](https://github.com/rpng/open_vins/tree/master/ov_data/kaist_vio) | [launch](https://github.com/rpng/open_vins/blob/master/ov_msckf/launch/pgeneva_ros_kaistvio.launch) |
206+
| rotation | 7.82 | [download](https://urserver.kaist.ac.kr/publicdata/KAIST_VIO_Dataset/rotation/rotation.bag) | [link](https://github.com/rpng/open_vins/tree/master/ov_data/kaist_vio) | [launch](https://github.com/rpng/open_vins/blob/master/ov_msckf/launch/pgeneva_ros_kaistvio.launch) |
207+
| rotation_fast | 14.55 | [download](https://urserver.kaist.ac.kr/publicdata/KAIST_VIO_Dataset/rotation/rotation_fast.bag) | [link](https://github.com/rpng/open_vins/tree/master/ov_data/kaist_vio) | [launch](https://github.com/rpng/open_vins/blob/master/ov_msckf/launch/pgeneva_ros_kaistvio.launch) |
208+
| square | 41.94 | [download](https://urserver.kaist.ac.kr/publicdata/KAIST_VIO_Dataset/square/square.bag) | [link](https://github.com/rpng/open_vins/tree/master/ov_data/kaist_vio) | [launch](https://github.com/rpng/open_vins/blob/master/ov_msckf/launch/pgeneva_ros_kaistvio.launch) |
209+
| square_fast | 44.07 | [download](https://urserver.kaist.ac.kr/publicdata/KAIST_VIO_Dataset/square/square_fast.bag) | [link](https://github.com/rpng/open_vins/tree/master/ov_data/kaist_vio) | [launch](https://github.com/rpng/open_vins/blob/master/ov_msckf/launch/pgeneva_ros_kaistvio.launch) |
210+
| square_head | 50.00 | [download](https://urserver.kaist.ac.kr/publicdata/KAIST_VIO_Dataset/square/square_head.bag) | [link](https://github.com/rpng/open_vins/tree/master/ov_data/kaist_vio) | [launch](https://github.com/rpng/open_vins/blob/master/ov_msckf/launch/pgeneva_ros_kaistvio.launch) |
211+
@m_enddiv
212+
213+
214+
215+
191216

192217

193218

0 commit comments

Comments
 (0)