Skip to content

Update to librdkafka 2.8.0 #187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Feb 24, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/pull_request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
name: Soundness
uses: swiftlang/github-workflows/.github/workflows/soundness.yml@main
with:
linux_pre_build_command: "apt-get update -y -q && apt-get install -y -q libsasl2-dev"
linux_pre_build_command: "apt-get update -y -q && apt-get install -y -q libsasl2-dev libssl-dev"
license_header_check_project_name: "swift-kafka-client"

unit-tests:
Expand All @@ -36,7 +36,7 @@ jobs:
run: echo "cxx-interop-matrix=$(curl -s https://raw.githubusercontent.com/apple/swift-nio/main/scripts/generate_matrix.sh | bash)" >> "$GITHUB_OUTPUT"
env:
MATRIX_LINUX_COMMAND: "curl -s https://raw.githubusercontent.com/apple/swift-nio/main/scripts/check-cxx-interop-compatibility.sh | bash"
MATRIX_LINUX_SETUP_COMMAND: "apt-get update -y -q && apt-get install -y -q curl jq libsasl2-dev"
MATRIX_LINUX_SETUP_COMMAND: "apt-get update -y -q && apt-get install -y -q curl jq libsasl2-dev libssl-dev"

cxx-interop:
name: Cxx interop
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/unit_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ jobs:
COMMAND_OVERRIDE_NIGHTLY_6_1: "swift test ${{ inputs.linux_nightly_6_1_arguments_override }}"
COMMAND_OVERRIDE_NIGHTLY_MAIN: "swift test ${{ inputs.linux_nightly_main_arguments_override }}"
run: |
apt-get -qq update && apt-get -qq -y install curl && apt-get -y install libsasl2-dev
apt-get -qq update && apt-get -qq -y install curl && apt-get -y install libsasl2-dev libssl-dev
curl -s https://raw.githubusercontent.com/apple/swift-nio/main/scripts/check-matrix-job.sh | bash
container:
image: ${{ matrix.swift.image }}
Expand Down
31 changes: 23 additions & 8 deletions Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,13 @@ import PackageDescription
let rdkafkaExclude = [
"./librdkafka/src/CMakeLists.txt",
"./librdkafka/src/Makefile",
"./librdkafka/src/README.lz4.md",
"./librdkafka/src/generate_proto.sh",
"./librdkafka/src/librdkafka_cgrp_synch.png",
"./librdkafka/src/statistics_schema.json",
"./librdkafka/src/opentelemetry/metrics.options",
"./librdkafka/src/rdkafka_sasl_win32.c",
"./librdkafka/src/rdwin32.h",
"./librdkafka/src/statistics_schema.json",
"./librdkafka/src/win32_config.h",
// Remove dependency on cURL. Disabling `ENABLE_CURL` and `WITH_CURL` does
// not appear to prevent processing of the below files, so we have to exclude
Expand Down Expand Up @@ -52,7 +54,6 @@ let package = Package(
],
dependencies: [
.package(url: "https://github.com/apple/swift-nio.git", from: "2.55.0"),
.package(url: "https://github.com/apple/swift-nio-ssl", from: "2.25.0"),
.package(url: "https://github.com/swift-server/swift-service-lifecycle.git", from: "2.1.0"),
.package(url: "https://github.com/apple/swift-log.git", from: "1.0.0"),
.package(url: "https://github.com/apple/swift-metrics", from: "2.4.1"),
Expand All @@ -64,7 +65,7 @@ let package = Package(
.target(
name: "Crdkafka",
dependencies: [
.product(name: "NIOSSL", package: "swift-nio-ssl"),
"COpenSSL",
.product(name: "libzstd", package: "zstd"),
],
exclude: rdkafkaExclude,
Expand All @@ -73,7 +74,6 @@ let package = Package(
cSettings: [
// dummy folder, because config.h is included as "../config.h" in librdkafka
.headerSearchPath("./custom/config/dummy"),
.headerSearchPath("./custom/include"),
.headerSearchPath("./librdkafka/src"),
.define("_GNU_SOURCE", to: "1"), // Fix build error for Swift 5.9 onwards
],
Expand All @@ -98,6 +98,14 @@ let package = Package(
"Kafka"
]
),
.systemLibrary(
name: "COpenSSL",
pkgConfig: "openssl",
providers: [
.brew(["libressl"]),
.apt(["libssl-dev"]),
]
),
.testTarget(
name: "KafkaTests",
dependencies: [
Expand All @@ -113,14 +121,21 @@ let package = Package(
)

for target in package.targets {
var settings = target.swiftSettings ?? []
settings.append(.enableExperimentalFeature("StrictConcurrency=complete"))
target.swiftSettings = settings
switch target.type {
case .regular, .test, .executable:
var settings = target.swiftSettings ?? []
settings.append(.enableExperimentalFeature("StrictConcurrency=complete"))
target.swiftSettings = settings
case .macro, .plugin, .system, .binary:
break // These targets do not support settings
@unknown default:
fatalError("Update to handle new target type \(target.type)")
}
}

// --- STANDARD CROSS-REPO SETTINGS DO NOT EDIT --- //
for target in package.targets {
if target.type != .plugin {
if target.type != .plugin && target.type != .system {
var settings = target.swiftSettings ?? []
// https://github.com/swiftlang/swift-evolution/blob/main/proposals/0444-member-import-visibility.md
settings.append(.enableUpcomingFeature("MemberImportVisibility"))
Expand Down
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,9 +237,13 @@ config.securityProtocol = .saslTLS(

## librdkafka

The Package depends on [the librdkafka library](https://github.com/confluentinc/librdkafka), which is included as a git submodule.
The Package depends on [the `librdkafka` library](https://github.com/confluentinc/librdkafka), which is included as a git submodule.
It has source files that are excluded in `Package.swift`.

### Dependencies

`librdkafka` depends on `openssl`, meaning that `libssl-dev` must be present at build time.

## Development Setup

We provide a Docker environment for this package. This will automatically start a local Kafka server and run the package tests.
Expand Down
Empty file.
15 changes: 0 additions & 15 deletions Sources/Crdkafka/custom/include/openssl/err.h

This file was deleted.

15 changes: 0 additions & 15 deletions Sources/Crdkafka/custom/include/openssl/evp.h

This file was deleted.

15 changes: 0 additions & 15 deletions Sources/Crdkafka/custom/include/openssl/hmac.h

This file was deleted.

15 changes: 0 additions & 15 deletions Sources/Crdkafka/custom/include/openssl/pkcs12.h

This file was deleted.

15 changes: 0 additions & 15 deletions Sources/Crdkafka/custom/include/openssl/sha.h

This file was deleted.

17 changes: 0 additions & 17 deletions Sources/Crdkafka/custom/include/openssl/ssl.h

This file was deleted.

15 changes: 0 additions & 15 deletions Sources/Crdkafka/custom/include/openssl/x509.h

This file was deleted.

15 changes: 0 additions & 15 deletions Sources/Crdkafka/custom/include/openssl/x509_vfy.h

This file was deleted.

2 changes: 1 addition & 1 deletion Sources/Crdkafka/librdkafka
Submodule librdkafka updated 231 files
2 changes: 1 addition & 1 deletion docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ ENV LANGUAGE en_US.UTF-8
# Dependencies
RUN apt-get update
RUN apt-get install libsasl2-dev -y
RUN apt-get install libssl-dev -y
RUN apt-get install libjemalloc-dev -y


# tools
RUN mkdir -p $HOME/.tools
RUN echo 'export PATH="$HOME/.tools:$PATH"' >> $HOME/.profile
Expand Down
Loading