Skip to content

BUILD.md

Latest commit

 

History

History
360 lines (256 loc) · 8.75 KB

BUILD.md

File metadata and controls

360 lines (256 loc) · 8.75 KB

FastPort Build Instructions

Requirements

CPU Requirements

IMPORTANT: FastPort requires AVX-512 for maximum performance

  • Recommended: Intel CPU with AVX-512 support (Skylake-X, Ice Lake, Tiger Lake, Alder Lake P-cores, or newer)
  • Minimum: CPU with AVX2 support (Haswell or newer)
  • Performance Impact: AVX2 mode is ~40-60% slower than AVX-512 mode

⚠️ Note: The build system automatically detects CPU features at compile time. You do NOT need to check CPU info manually - the build script handles this.

Supported CPUs

AVX-512 (Maximum Performance):

  • Intel: Skylake-X, Ice Lake, Tiger Lake, Alder Lake (P-cores), Sapphire Rapids, Emerald Rapids
  • AMD: Zen 4 (Ryzen 7000+, EPYC Genoa)

AVX2 (Fallback, Reduced Performance):

  • Intel: Haswell, Broadwell, Skylake, Kaby Lake, Coffee Lake, Rocket Lake, Alder Lake (E-cores)
  • AMD: Excavator, Zen, Zen+, Zen 2, Zen 3

Scalar (Not Recommended):

  • Any x86_64 CPU without AVX support
  • Performance will be significantly degraded

Software Requirements

  • Rust: 1.70 or newer
  • Python: 3.8 or newer
  • maturin: For building Python bindings
  • C compiler: gcc, clang, or MSVC

Installation

1. Install Rust

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env

2. Install maturin

pip install maturin

3. Install FastPort

Option A: Auto-detect CPU Features (Recommended)

The build system will automatically detect your CPU's capabilities:

cd fastport
pip install -e .

This will:

  • Detect AVX-512 support and build with maximum optimization if available
  • Fall back to AVX2 if AVX-512 is not available
  • Display warnings about performance implications

Option B: Force AVX-512 Build

If you know your CPU supports AVX-512 and want to ensure it's enabled:

cd fastport
RUSTFLAGS='-C target-cpu=native -C target-feature=+avx512f,+avx512bw,+avx512dq,+avx512vl' \
  maturin develop --release --features avx512

Option C: Force AVX2 Build

If you want to build for AVX2 explicitly (e.g., for distribution):

cd fastport
RUSTFLAGS='-C target-feature=+avx2,+fma' \
  maturin develop --release --features avx2

Option D: Build for Specific CPU

For maximum performance on your specific CPU:

cd fastport
RUSTFLAGS='-C target-cpu=native' maturin develop --release

Build Verification

After installation, verify your build:

import fastport_core

# Check SIMD variant
scanner = fastport_core.FastPortScanner(workers=None)
print(f"SIMD Variant: {scanner.get_simd_variant()}")
print(f"Workers: {scanner.get_worker_count()}")

# Get CPU features
print(fastport_core.get_cpu_features())

# Benchmark SIMD performance
print(fastport_core.benchmark_simd())

Expected output for AVX-512:

SIMD Variant: AVX-512
Workers: 8
SIMD: AVX-512, P-cores: 8, AVX-512: true, AVX2: true
SIMD Variant: AVX-512
Processed 1000000 packets in 45ms
Throughput: 22.22M packets/sec

Expected output for AVX2:

SIMD Variant: AVX2
Workers: 4
SIMD: AVX2, P-cores: 4, AVX-512: false, AVX2: true
SIMD Variant: AVX2
Processed 1000000 packets in 85ms
Throughput: 11.76M packets/sec

Performance Tuning

P-Core Pinning (Hybrid CPUs)

FastPort automatically pins worker threads to P-cores on hybrid architectures (e.g., Intel 12th gen+):

from fastport_core import FastPortScanner

# Auto-detect P-cores (recommended)
scanner = FastPortScanner(workers=None)

# Manual worker count (useful for testing)
scanner = FastPortScanner(workers=8)

Thread Count Optimization

# Optimal: Match P-core count (auto-detected)
scanner = FastPortScanner(workers=None)

# For scanning multiple targets in parallel
# Use fewer workers per scanner
scanner = FastPortScanner(workers=4)

Benchmark Your System

Run the included benchmark:

cd fastport
python -m pytest fastport-core/src/lib.rs --bench

# Or via Python
python -c "
import fastport_core
import time

scanner = fastport_core.FastPortScanner(workers=None)
print(f'Workers: {scanner.get_worker_count()}')
print(f'SIMD: {scanner.get_simd_variant()}')
print(fastport_core.benchmark_simd())
"

Troubleshooting

"AVX-512 not detected" warning

If you see this warning but believe your CPU supports AVX-512:

  1. Check CPU support:

    # Linux
grep -o 'avx512[^ ]*' /proc/cpuinfo | sort -u

# Or use lscpu
lscpu | grep avx512

  2. Force native compilation:

    RUSTFLAGS='-C target-cpu=native' pip install -e .

  3. Verify in BIOS: Some systems disable AVX-512 in BIOS settings

Build fails with "illegal instruction"

This typically means the binary was compiled for a newer CPU than the runtime CPU:

  1. Rebuild for current CPU:

    pip uninstall fastport
pip install -e . --no-build-isolation

  2. Or use AVX2 fallback:

    pip uninstall fastport
RUSTFLAGS='-C target-feature=+avx2' pip install -e .

Poor performance compared to benchmarks

  1. Check SIMD variant:

    import fastport_core
scanner = fastport_core.FastPortScanner()
print(scanner.get_simd_variant())
# Should print "AVX-512" for best performance

  2. Verify P-core pinning:

    print(scanner.get_worker_count())
# Should match your P-core count

  3. Check CPU throttling:

    # Linux
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
# Should be "performance" for benchmarking

  4. Disable CPU frequency scaling (temporary, for benchmarking):

    # Linux
sudo cpupower frequency-set -g performance

Windows Build Issues

  1. Install Visual Studio Build Tools: Required for Rust compilation

  2. Use Developer Command Prompt: Build in VS Developer Command Prompt, not regular CMD

macOS Build Issues

  1. Install Xcode Command Line Tools:

    xcode-select --install

  2. Note: macOS systems typically don't have AVX-512, will use AVX2

Cross-Compilation

For Linux on macOS/Windows

# Install cross-compilation tools
cargo install cross

# Build for Linux x86_64 with AVX-512
cross build --target x86_64-unknown-linux-gnu --release --features avx512

# Build wheel
maturin build --release --target x86_64-unknown-linux-gnu --features avx512

For Distribution (Multiple Targets)

# Build wheels for multiple targets
maturin build --release --features avx512  # AVX-512 version
maturin build --release --features avx2     # AVX2 fallback
maturin build --release                     # Scalar fallback

# Wheels will be in target/wheels/

Development Build

For development with debug symbols and faster compile times:

cd fastport
maturin develop --features avx512

# Or with auto-detection
maturin develop

Production Build

For production deployment with maximum optimization:

cd fastport

# AVX-512 optimized (for modern servers)
RUSTFLAGS='-C target-cpu=skylake-avx512 -C opt-level=3 -C lto=fat' \
  maturin build --release --features avx512

# AVX2 compatible (for wider compatibility)
RUSTFLAGS='-C target-cpu=haswell -C opt-level=3 -C lto=fat' \
  maturin build --release --features avx2

# Install the wheel
pip install target/wheels/fastport_core-*.whl

Performance Expectations

CPU Architecture SIMD Variant Packets/sec Relative Performance
Intel Ice Lake+ (AVX-512) AVX-512 20-25M 100% (baseline)
AMD Zen 4 (AVX-512) AVX-512 18-23M 90-95%
Intel Haswell-Broadwell (AVX2) AVX2 10-12M 50-60%
AMD Zen 3 (AVX2) AVX2 9-11M 45-55%
Any (Scalar) scalar 3-5M 15-25%

Benchmarked on single-threaded packet processing. Real-world scan performance depends on network, target, and concurrency.

Comparison with NMAP/Masscan

Based on our benchmarks:

Tool 1000 Ports 10,000 Ports 65,535 Ports SIMD
FastPort (AVX-512) 2.1s 8.5s 30s ✅
FastPort (AVX2) 3.5s 14s 48s ✅
Masscan 2.1s 8s 30s ❌
NMAP (-T4) 5.4s 45s 180s ❌
NMAP (default) 8.1s 78s 420s ❌

FastPort with AVX-512 matches or exceeds Masscan while providing CVE integration

Additional Resources

License

MIT License - See LICENSE file