24.04 Support

[ekristen]

This is being worked on, we had to wait for some upstream dependencies to support 24.04 officially.

[ekristen]

Biggest impediment here is going to be python based things, we pretty much have to setup virtualenvs for everything.

[ekristen]

@digitalsleuth FYSA

[digitalsleuth]

@ekristen Makes sense, I'll get on the first of them tonight, and review the Python 2 tools again to see if there are either viable alternatives or the possibility of a quick-and-easy port.

[ekristen]

Let's coordinate efforts. Pick a tool or two keep it small. I've already started a bunch of changes and I want to try and prevent too many conflicts.

[digitalsleuth]

Sure thing, just let me know what you've already started on and I'll tackle one of the others.

[ekristen]

I've got a working branch (not a ton pushed, more local changes at the moment) -- just keep your PRs scoped to one thing at a time so we can more easily deconflict any merge conflicts.

support-24.04, open prs against this branch.

[ekristen]

@digitalsleuth we might be able to use just a couple virtualenvs for everything vs 1 virtual env per tool, keep that in mind. I'm pushing some changes soon.

[digitalsleuth]

@ekristen I've been working on some states and slowing moving Python 2 packages to Python 3, but I keep bumping up against the new "pipx" issue in 24.04. Essentially, Ubuntu no longer likes installing pip packages globally, and the following error comes up:

              × This environment is externally managed
              ╰─> To install Python packages system-wide, try apt install
                  python3-xyz, where xyz is the package you are trying to
                  install.

                  If you wish to install a non-Debian-packaged Python package,
                  create a virtual environment using python3 -m venv path/to/venv.
                  Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
                  sure you have python3-full installed.

                  If you wish to install a non-Debian packaged Python application,
                  it may be easiest to use pipx install xyz, which will manage a
                  virtual environment for you. Make sure you have pipx installed.

                  See /usr/share/doc/python3.12/README.venv for more information.

              note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
              hint: See PEP 668 for the detailed specification.

I didn't want to assume what you'd like for the approach to this, so I figured I'd just reach out and ask first.
Thoughts?

[ekristen]

I think we have to go to doing virtualenv for very specific thing we want to install and install to folders in /opt and symlink to usr/local/bin

[digitalsleuth]

Sounds good, I'll start working on it.

[digitalsleuth]

So far, the list of python3 packages which looks like they'll have to have a virtualenv (or "may" need to be dropped) are:

• defang
• machinae
• imagemounter
• ioc_writer
• pillow (tougher because it's a library)
• pyhindsight
• python-registry (there are alternatives)
• stix-validator
• stix

The remainder actually have apt packages in Noble for the library and / or tool itself. Prepending python3- to install the apt packages, those are:

• pefile
• bitstring
• colorama
• geoip2
• ijson
• keyrings.alt
• lxml
• dateutil
• evtx
• libevtx (python3 bindings only. Not currently used by SIFT packages, but could be useful. The new standard apt package is now called libevtx-utils vice just libevtx)
• magic
• setuptools-rust
• setuptools
• virustotal-api
• wheel
• yara

I'm currently testing the python3 virtualenv options for the first part of the list, then can move to the second part of the list (the new apt packages).

[ekristen]

I don't have a problem adding the apt packages, but most of the python packages were in support of a specific tool being installed, which any tooling we are going to have to use virtualenv going forward so, I we should focus on which tools need which python packages and focus on virtualenvs and symlinks for those scripts to work.

[digitalsleuth]

Just to clarify, are we retaining backwards compatibility with 22 and 20?

[ekristen]

22 ideally, drop 20 Sent from my iPhoneOn Oct 19, 2024, at 08:44, Digital Sleuth ***@***.***> wrote: Just to clarify, are we retaining backwards compatibility with 22 and 20? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[digitalsleuth]

I've been working on moving away from Python 2 and have re-created a couple of the "scripts" in Python 3:
pe-scanner
and
pe-carver

If you're satisfied with these re-writes, I'll replace the old ones with these new ones.