Release 0.6.8

This is the next point release for Velociraptor - Digging deeper!

Detailed release notes are posted at https://docs.velociraptor.app/blog/2023/2023-02-13-release-notes-0.6.8/

This release introduces an upgraded communication protocol between client and server. Previously the server would keep track of various collection statistics but now these are tracked on the client, making the server a lot more efficient. The new communication protocol is a lot faster and allows the server to support a larger number of clients. When using a 0.6.8 server with older clients, the older protocol is still used to maintain backwards compatibility.

Notable features

• Send ClientInfo messages all the time. Previously client information like hostname was collected during interrogation by collecting Generic.Client.Info. In this version, Velociraptor will send periodic updates every day to keep client information in sync.
• Use pageable tables for the VFS: The previous VFS view would store the entire directory listing in a single table making it impossible to view very large directories like C:\Windows or System32. This version updates the way the VFS is handled to transfer directory listing more efficiently, and uses a pageable table to view arbitrarily large directories. Performance is much improved for example recursive directory sync on C:\Windows tables about 90 sec to sync 250k files.
• Refactored hunt and collection export code - exporting a hunt or collection creates a zip file containing the files collected and query results. The code for exporting this zip file was optimized to produce this export much faster and with less memory requirements. The GUI is also much improved.
• Batch client log messages into JSONL groups - Previously clients would send each query log in a separate message, in order to ensure they get to the server ASAP. This release batches these into larger messages reducing the server's load and improving efficiency.
• Implemented a preview Column renderer - Sometimes it is handy to send from the client some raw data to provide context around e.g. yara hits. Previously it was only possible to view data that was encoded within the result set JSON causing very large JSON payloads. In this version it is possible to use the upload() function to upload an arbitrary sized file to the server, and mark the column as type "upload_preview" . The GUI will then render a preview of the file (if it is an image, it will be shown as a thumbnail) otherwise a hex viewer.
• Uploads are now deduplicated on store_as_name: Previously each call to the upload() function would result in the file being uploaded which made using upload() a bit tricky in order to avoid uploading the same file multiple times. Now it is not necessary as uploads will be automatically deduplicated so can be called multiple times on the same file.
• Added a tempfile based materializer to have safe queries : Some VQL queries would materialize a query into an array, e.g. with an expression like

LET X <= SELECT * FROM glob(...)

Previously this expanded the query into memory and would cause huge memory requirements and possible out of memory crashes. In this release VQL implements a temp file implementation of the LET operator which automatically switches to a temp file when the variable receives more than 1000 rows by default. This keeps memory use controlled and allows these kinds of queries to be run safely.

• Allow client side collections to be traced: We often get questions about what happened to a collection that seems to be hung? Previously the only way to gather client side information was to collect a Generic.Client.Profile collection. This required running it at just the right time and did not guarantee that we would get helpful insight of what the query and the client binary were doing. In this release it is possible to specify a trace collection for any collection to automatically collect client side state as the collection is progressing.

• Added progress reporting to offline collector: When copying large files it is useful to get some kind of feedback as the copy operation is progressing.

• Track tool definitions by defining artifact: Previously once a tool was defined, the admin could only override the tool but not reset it to a new version. If a tool was defined by 2 artifacts it might be that each artifact declares a different version. The release allows the admin to reset the tool definition to any one of the artifacts that define it.

Including many bug fixes and stability improvements.

Known Issues

Release 0.6.8-2 fixes a number of minor issues:

1. Flow Deletion did not remove uploaded bulk files.
2. Protect CryptCATAdmin and WinVerifyTrust functions behind dangerous api flag due to occassional crashes
3. Trace file generator regression
4. Dedudplicate labels in GUI
5. Suppress logging to files for admin commands
6. Add client id back to client monitoring events