This repository and the associated datasets constitute a framework for developing learning-based solutions to prediction, planning and simulation problems in self-driving. State-of-the-art solutions to these problems still require significant amounts of hand-engineering and unlike, for example, perception systems, have not benefited much from deep learning and the vast amount of driving data available.
The purpose of this framework is to enable engineers and researchers to experiment with data-driven approaches to planning and simulation problems using real world driving data and contribute to state-of-the-art solutions.
This software is developed by Lyft Level 5 self-driving division and is open to external contributors.
You can use this framework to build systems which:
- Turn prediction, planning and simulation problems into data problems and train them on real data.
- Use neural networks to model key components of the Autonomous Vehicle (AV) stack.
- Use historical observations to predict future movement of cars around an AV.
- Plan behavior of an AV in order to imitate human driving.
- Study the improvement in performance of these systems as the amount of data increases.
We provide several notebooks with examples and applications.
Our visualisation notebook is the perfect place to start if you want to know more about L5Kit.
Related to our 2020 competition, we provide a notebook to train and test our baseline model for predicting future agents trajectories.
We provide 3 notebooks for a deep dive into planning for a Self Driving Vehicle (SDV). Please refer to our README for a full description of what you can achieve using them:
- you can train your first ML policy for planning using our training notebook
- you can evaluate your model in the open-loop setting using our open-loop evaluation notebook
- you can evaluate your model in the closed-loop setting using our closed-loop evaluation notebook
We also provide pre-trained models for this task. Please refer to the training notebook.
We provide a simulation notebook to test interaction between agents and the SDV when they are both controlled by a ML policy.
- test your ML policy for simulation using our simulation evaluation notebook
- 04-16-2021: We've just released a new notebook for the ML simulation task!
- 12-03-2020: We've just released a series of notebooks to train and evaluate an ML planning model. We've also included pre-trained models! Learn more about this in the dedicated README
- 11-26-2020: 2020 Kaggle Lyft Motion Prediction for Autonomous Vehicles Competition ended. We had more than 900 teams taking part in it!
- 11-16-2020: Dataset paper presented at CoRL 2020
- 09-29-2020: L5Kit v1.0.1 released
- 08-25-2020: 2020 Kaggle Lyft Motion Prediction for Autonomous Vehicles Competition started
- 08-24-2020: L5Kit v1.0.6 and Dataset v1.1 (includes traffic light support) released!
- 06-25-2020: Docs and API available at https://lyft.github.io/l5kit/ (thanks Kevin Zhao!)
The framework consists of three modules:
- Datasets - data available for training ML models.
- L5Kit - the core library supporting functionality for reading the data and framing planning and simulation problems as ML problems.
- Examples - an ever-expanding collection of jupyter notebooks which demonstrate the use of L5Kit to solve various AV problems.
To use the framework you will need to download the Lyft Level 5 Prediction dataset from https://self-driving.lyft.com/level5/data/. It consists of the following components:
- 1000 hours of perception output logged by Lyft AVs operating in Palo Alto. This data is stored in 30 second chunks using the zarr format.
- A hand-annotated, HD semantic map. This data is stored using protobuf format.
- A high-definition aerial map of the Palo Alto area. This image has 8cm per pixel resolution and is provided by NearMap.
To read more about the dataset and how it was generated, read the dataset whitepaper.
Note (08-24-20): The new version of the dataset includes dynamic traffic light support. Please update your L5Kit version to v1.0.6 to start using this functionality.
Register at https://self-driving.lyft.com/level5/data/ and download the 2020 Lyft prediction dataset. Store all files in a single folder to match this structure:
prediction-dataset/
+- scenes/
+- sample.zarr
+- train.zarr
+- train_full.zarr
+- aerial_map/
+- aerial_map.png
+- semantic_map/
+- semantic_map.pb
+- meta.json
You may find other downloaded files and folders (mainly from aerial_map), but they are not currently required by L5Kit
L5Kit is a library which lets you:
- Load driving scenes from zarr files
- Read semantic maps
- Read aerial maps
- Create birds-eye-view (BEV) images which represent a scene around an AV or another vehicle
- Sample data
- Train neural networks
- Visualize results
The examples folder contains examples in jupyter notebook format which you can use as a foundation for building your ML planning and simulation solutions. Currently we provide two examples, with more to come soon:
A tutorial on how to load and visualize samples from a dataset using L5Kit.
An example of training a neural network to predict the future positions of cars nearby an AV. This example is a baseline solution for the Lyft 2020 Kaggle Motion Prediction Challenge.
Follow this workflow if:
- you're not interested in developing and/or contributing to L5Kit;
- you don't need any features from a specific branch or latest master and you're fine with the latest release;
pip install l5kitYou should now be able to import from L5Kit (e.g. from l5kit.data import ChunkedDataset should work)
Examples are not shipped with the package, but you can download the zip release from: L5Kit Releases
Please download the zip matching your installed version (you can run pip freeze | grep l5kit to get the right version)
Unzip the files and grab the example folder in the root of the project.
jupyter notebook examples/visualisation/visualise_data.ipynbFollow this workflow if:
- you want to test latest master or another branch;
- you want to contribute to L5Kit;
- you want to test the examples using a non-release version of the code;
git clone https://github.com/lyft/l5kit.git
cd l5kit/l5kitPlease note the double l5kit in the path, as we need to cd where setup.py file is.
We support deterministic build through pipenv.
Once you've installed pipenv (or made it available in your env) run:
pipenv sync --devThis will install all dependencies (--dev includes dev-packages too) from the lock file.
If you don't care about determinist builds or you're having troubles with packages resolution (Windows, Python<3.7, etc..),
you can install directly from the setup.py by running:
pip install -e ."[dev]"If you run into trouble installing L5Kit on Windows, you may need to
- install Pytorch and torchvision manually first (select the correct version required by your system, i.e. GPU or CPU-only), then run L5Kit install (remove the packages torch and torchvision from
setup.py) - install Microsoft C++ Build Tools.
sphinx-apidoc --module-first --separate -o API/ l5kit/l5kit l5kit/l5kit/tests*
sphinx-build . docsjupyter notebook examples/visualisation/visualise_data.ipynbWe use Apache 2 license for the code in this repo.
The framework was developed at Lyft Level 5 and is maintained by the following authors and contributors:
- Guido Zuidhof
- Luca Bergamini
- John Houston
- Yawei Ye
- Suraj MS
- Oliver Scheel
- Emil Praun
- Liam Kelly
- Vladimir Iglovikov
- Chih Hu
- Peter Ondruska
If you are using L5Kit or dataset in your work please cite the following whitepaper:
@misc{lyft2020,
title={One Thousand and One Hours: Self-driving Motion Prediction Dataset},
author={John Houston and Guido Zuidhof and Luca Bergamini and Yawei Ye and Ashesh Jain and Sammy Omari and Vladimir Iglovikov and Peter Ondruska},
year={2020},
eprint={2006.14480},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
If you find problem or have questions about L5Kit please feel free to create github issue or reach out to [email protected]!