Updated Fixes for the Linux version of GRDB

[thinkpractice]

Fixes for the Linux version of GRDB

UPDATE this PR merges some upstream changes into my fork.

This PR contains some fixes for the Linux version of GRDB, mainly these fixes include:

• Fixing linker errors on Linux
• Some fixes for differences between Foundation on Linux and Apple platforms
• Fixing the unittests and making them succeed on Linux
• Ignoring Apple platform specific functionality like Combine or NSFileCoordinator, etc.

Some of these fixes are inspired by #1708 by @marcprux. This PR attemps to make these fixes in
a platform-agnostic way so that GRDB will build with SPM on more platforms than Linux alone,
for instance, on Android and Windows. These patches fix the build of GRDB on Swift 6.1 and later.
There are still some issues building on Swift 6.0, which I will attempt to fix later.

Fixing the GRDB Build

Most of GRDB actually builds on Linux just fine. The only change is that I needed to do to make it build was adding a

@preconcurrency import Dispatch

to handle non-Sendable protocol conformance in GRDB/Core/DispatchQueueActor.swift. Not sure if that's the right approach but it made everything build!

Fixing GRDB linking on Linux

The main issue with GRDB on Linux is actually a linker error. The standard libsqlite3-dev installed with apt or dnf doesn't have WAL enabled and when trying to use GRDB in another swift package or app in Linux, you get a linker error. Functions like sqlite3_snapshot_get, sqlite3_snapshot_open are not found by the linker,
and therefore the package isn't usable.

There would be two ways around it I think, either:

• include a custom sqlite build for GRDB on Linux, or
• fix linking by adding extra compiler directives/conditions for Linux

I chose the latter for now as it is a more easy fix. I updated the following conditional compilation directive:

#if SQLITE_ENABLE_SNAPSHOT || (!GRDBCUSTOMSQLITE && !GRDBCIPHER)

to:

#if (SQLITE_ENABLE_SNAPSHOT || (!GRDBCUSTOMSQLITE && !GRDBCIPHER)) && !os(Linux)

This is a quick fix, but it would be better to change the SQLITE_ENABLE_SNAPSHOT to return false on Linux.

Note this should be better handled as a fix to SQLITE_ENABLE_SNAPSHOT as later on Linux may also provide
a custom sqlite build to enable WAL. In those cases, the code inside these blocks should be compiled.

This compiler directive is present (and updated) in the following files:

• GRDB/Core/DatabasePool.swift
• GRDB/Core/DatabaseSnapshotPool.swift
• GRDB/Core/WALSnapshot.swift
• GRDB/Core/WALSnapshotTransaction.swift
• GRDB/ValueObservation/Observers/ValueConcurrentObserver.swift

Updating the compiler directive as above will fix the build with the standard libsqlite3-dev on Linux. If that's fine we
can keep it at that. I'm not completely sure yet what WAL does but if I understand correctly, using it is mostly a
performance consideration. If it would be a nice to have on Linux too, I can research adding a custom sqlite build for
Linux with WAL enabled. I wonder why it is not included in the standard libsqlite3-dev though? Adding the changes above
will fix linking of the GRDB code.

Fixes for open source Foundation versions

Open source versions of Foundation are slightly different than the Foundation Apple bundles with XCode. Also, some
packages, like for instance CoreGraphics, are not available on non-Apple platforms. Some primitive types like CGFloat
are available on non-Apple platforms and are defined in Foundation. In some cases, there are also differences where a
initializer is defined as convenience init in open source Foundation and as a normal init in Apple Foundation. This
causes some problems with Apple specific code in some cases; it will fail to build on open source Foundation. In GRDB in
many cases this code is surrounded by a #if !os(Linux) compiler directive. The fixes below try to solve the build issues on
Linux and try to get rid of these compiler directives. Also fixing the unit tests at the same time.

CGFloat.swift

CoreGraphics is not available on Linux. But CGFloat is available via Foundation. Changed the compiler directive in CGFloat.swift and its test from:

#if canImport(CoreGraphics)

to:

#if canImport(CoreGraphics)
    import CoreGraphics
#elseif !os(Darwin)
    import Foundation
#endif

The same fix should be added to GRDBTests/CGFloatTests.swift.

Decimal.swift

Removed the #if !os(Linux) compiler directive in the Decimal extension, as it compiles fine under swift 6. As discussed
with @groue the minimum version for GRDB is swift 6.0. I tested these changes with swift 6.1 and swift 6.2 and they work fine.

Note: swift 6.0 doesn't build, we should look into fixing the build for this version.

NSString, NSDate, Date

Some foundation NS* types (NSString, NSData, Date, etc.) were excluding the DatabaseValueConvertible extensions on Linux but build fine on swift 6.1 and swift 6.2.

Removed #if !os(Linux) in:

• NSString.swift
• NSData.swift
• Date.swift

NSNumber

There's a problem with the following code in NSNumber.swift on Linux:

 public static func fromDatabaseValue(_ dbValue: DatabaseValue) -> Self? {
   switch dbValue.storage {
   case .int64(let int64):
         return self.init(value: int64)
   case .double(let double):
         return self.init(value: double)
   case let .string(string):
         // Must match Decimal.fromDatabaseValue(_:)
         guard let decimal = Decimal(string: string, locale: posixLocale) else { return nil }
         return NSDecimalNumber(decimal: decimal) as? Self
   default:
         return nil
   }
}

both the .int64 and .double case above do not compile. The error is the following:

error: constructing an object of class type 'Self' with a metatype value must use a 'required' initializer

The NSNumber(value: Int64) and NSNumber(value: Double) are marked as convenience init on Linux. This may cause the build to fail. The issue with this approach is that it doesn't work for NSDecimalNumber, which is a subclass of NSNumber.
It looks like the main issue to begin with is that NSNumber defines NSNumber(value: Int64) and NSNumber(value: Double as convenience init, whereas their NSDecimalNumber equivalents are non-convenience. I could solve it by changing the code
above like this:

public static func fromDatabaseValue(_ dbValue: DatabaseValue) -> Self? {
        switch dbValue.storage {
        case .int64(let int64) where self is NSDecimalNumber.Type:
            let number = NSDecimalNumber(value: int64)
            return number as? Self
        case .int64(let int64):
            let number = NSNumber(value: int64)
            return number as? Self
        case .double(let double) where self is NSDecimalNumber.Type:
            let number = NSDecimalNumber(value: double)
            return number as? Self
        case .double(let double):
            let number = NSNumber(value: double)
            return number as? Self
        case .string(let string):
            // Must match Decimal.fromDatabaseValue(_:)
            guard let decimal = Decimal(string: string, locale: posixLocale) else { return nil }
            return NSDecimalNumber(decimal: decimal) as? Self
        default:
            return nil
        }
    }
}

This is a lot more verbose than the existing code. Also, this code may fail if there are other subclasses of NSNumber
that need to be supported. For each of these classes we should add a case for .int64 and double. I think an alternative
approach would be better, but I don't have one at the moment.

All tests in:

• Tests/GRDBTests/FoundationNSNumberTests.swift
• Tests/GRDBTests/FoundationNSDecimalNumberTests.swift

succeeded with these changes.

URL.swift

The NSURL is a specific case. Apparently, the NSURL absoluteString property is non-optional on Linux and optional on Apple platforms. The fix is the following:

 /// Returns a TEXT database value containing the absolute URL.
        public var databaseValue: DatabaseValue {
            #if !os(Darwin)
                absoluteString.databaseValue
            #else
                absoluteString?.databaseValue ?? .null
            #endif
        }

UUID.swift

UUID.swift contains a compiler directive excluding the DatabaseValueConvertible from building
on Linux and Windows. On Linux the problem is the following:

 public static func fromDatabaseValue(_ dbValue: DatabaseValue) -> Self? {
        switch dbValue.storage {
        case .blob(let data) where data.count == 16:
            return data.withUnsafeBytes {
                self.init(uuidBytes: $0.bindMemory(to: UInt8.self).baseAddress)
            }
        case .string(let string):
            return self.init(uuidString: string)
        default:
            return nil
        }
    }

The NSUUID(uuidBytes:) constructor takes an optional UnsafePointer<UInt8>, see here, whereas this constructor on Linux takes a non-optional.

This could be fixed with a guard statement. However, similarly to the case for NSNumber in addition, the
self.init(uuidString: string) doesn't compile on Linux. The error is:

Constructing an object of class type 'Self' with a metatype value must use a 'required' initializer

Added the following fix:

public static func fromDatabaseValue(_ dbValue: DatabaseValue) -> Self? {
        switch dbValue.storage {
        case .blob(let data) where data.count == 16:
            return data.withUnsafeBytes {
                #if canImport(Darwin)
                    self.init(uuidBytes: $0.bindMemory(to: UInt8.self).baseAddress)
                #else
                    guard let uuidBytes = $0.bindMemory(to: UInt8.self).baseAddress else {
                        return nil as Self?
                    }
                    return NSUUID(uuidBytes: uuidBytes) as? Self
                #endif
            }
        case .string(let string):
            return NSUUID(uuidString: string) as? Self
        default:
            return nil
        }
    }

This also makes all unittests succeed. It should be checked whethere we would want to support any subclass of NSUUID.
If so, a similar solution as for NSNumber above can be used.

Use of DispatchQueue private APIs

In some of the tests for the DispatchQueue APIs, some private DispatchQueue apis are used to get the label of the
current DispatchQueue. These APIs are unavailable on non-Darwin platforms. It's mainly about some tests in for example DatabasePoolConcurrencyTests.swift. For instance, the following test:

func testDefaultLabel() throws {
        let dbPool = try makeDatabasePool()
        dbPool.writeWithoutTransaction { db in
            XCTAssertEqual(db.configuration.label, nil)
            XCTAssertEqual(db.description, "GRDB.DatabasePool.writer")

            // This test CAN break in future releases: the dispatch queue labels
            // are documented to be a debug-only tool.
            let label = String(utf8String: __dispatch_queue_get_label(nil))
            XCTAssertEqual(label, "GRDB.DatabasePool.writer")
         

I understood from @groue that the goal of those tests is to make sure that the Xcode, LLDB, and crash reports mention the label of the current database in the name of the current DispatchQueue. This is a nicety that aims at helping debugging, and worth a test. The label of the current dispatch queue can only be tested with the private __dispatch_queue_get_label api, which is only available on Darwin systems.

Those tests can be avoided on non-Darwin platforms, as @marcprux did in #1708:

#if canImport(Darwin) // __dispatch_queue_get_label unavailable on non-Darwin platforms
// This test CAN break in future releases: the dispatch queue labels
// are documented to be a debug-only tool.
let label = String(utf8String: __dispatch_queue_get_label(nil))
XCTAssertEqual(label, "GRDB.DatabasePool.writer")
#endif

I implemented this approach for:

• Utils.swift
• DatabasePoolConcurrencyTests.swift
• DatabaseQueueTests.swift
• DatabaseSnapshotTests.swift

NSError

Bridging of DatabaseError to NSError does not seem to work on Linux. The testNSErrorBridging therefore
fails, we should probably ignore the test on non-Darwin platforms as NSError is mostly a left-over from ObjectiveC times
and APIs.

   func testNSErrorBridging() throws {
        #if !canImport(Darwin)
            throw XCTSkip("NSError bridging not available on non-Darwin platforms")
        #else
            let dbQueue = try makeDatabaseQueue()
            try dbQueue.inDatabase { db in
                try db.create(table: "parents") { $0.column("id", .integer).primaryKey() }
                try db.create(table: "children") { $0.belongsTo("parent") }
                do {
                    try db.execute(sql: "INSERT INTO children (parentId) VALUES (1)")
                } catch let error as NSError {
                    XCTAssertEqual(DatabaseError.errorDomain, "GRDB.DatabaseError")
                    XCTAssertEqual(error.domain, DatabaseError.errorDomain)
                    XCTAssertEqual(error.code, 787)
                    XCTAssertNotNil(error.localizedFailureReason)
                }
            }
        #endif
    }

Fixing Unit tests

Tests using NSFileCoordinator

There are some tests using NSFileCoordinator which is not available on non-Darwin platforms. Most notably
testConcurrentOpening in DatabasePoolConcurrencyTests.swift contributes 400+ test failures when run on
Linux. testConcurrentOpening, that runs about 500 tests (a for loop of 50 iterations runs 10 times some things in parallel
with DispatchQueue.concurrentPerform. The test uses the NSFileCoordinator which isn't available on Linux.

These tests are ignored because they test whether NSFileCoordinator works with GRDB. This is Darwin specific functionality:

We're talking about testConcurrentOpening, right? The goal of DispatchQueue.concurrentPerform is just to spawn parallel
jobs. In this case each job opens a DatabasePool, protected by a NSFileCoordinator, and we're asserting that not error is
thrown. This is a test for the correct integration of GRDB and NSFileCoordinator, so that we can confidently recommend
using NSFileCoordinator as a way to protect a database that is shared between multiple processes.

We could say that those tests should be removed, arguing that they are just asserting the behavior of NSFileCoordinator and
that "testing the framework" is a bad practice - but we're also asserting that DatabasePool evolution must be compatible
with user code that uses NSFileCoordinator and that any deadlock must be prevented.)

Arguably DispatchQueue.concurrentPerform could be replaced with any other modern technique that guarantees a minimum level of parallelism, such as withTaskGroup.

We should check whether we need to test similar behaviour on Linux. For now we just ignore the test.
@marcprux solved it like this:

func testConcurrentOpening() throws { 
     #if !canImport(ObjectiveC) 
     throw XCTSkip("NSFileCoordinator unavailable") 
     #else 

We solved it similarly.

Other Unit tests

• XCTIssue is not available on non-Darwin platforms, it's used in:
  • GRDBTests/FailureTestCase.swift ignored this test completely with #if os(Darwin) on non-Darwin platforms
  • GRDBTests/ValueObservationRecorderTests.swift ignored this test also.
• Support.swift actually needs Combine, so all tests that use it need Combine as well:
  • DatabaseMigratorTests.swift has several tests using the Test class defined in Support.swift; used an #if !canImport(Combine) clause to skip several tests in this test suite.
  • ValueObservationTests.swift also uses this Test class
• DatabaseRegionObservationTests.swift tests some Combine functionality in testAnyDatabaseWriter; also wrapped them in a #if canImport(Combine)
• When integrating the upstream changes I had some conflicts in DatabaseMigratorTests.swift I tried to merge them but not sure everything went ok. I was struggling a bit integrating them. This file should be checked if all upstream changes have been merged in.

TODOs

• Fix the build under swift 6.0
• Check if we can provide an alternative for XCTIssue
• See if we can find an alternative for __dispatch_queue_get_label on Linux
• Setup a CI.yml pipeline to build the Linux version of the package automatically and run the unit tests

With this we:

Executed 2768 tests, with 46 tests skipped and 0 failures (0 unexpected) in 234.863 (234.863) seconds

[thinkpractice]

@groue I found out there's some weird behaviour with my vscode. One time tabs are converted to spaces, and then another indentation which was 4 spaces gets converted back to a tab 😠 Will need to check how to solve this. Already changed my editor settings, so either I missed some setting or it's a bug in vscode....

Found it apparently there is something like "editor.trimAutoWhitespace" in vscode which removes "trailing auto inserted whitespace"

[thinkpractice]

@groue I have been updating some more files to reflect the original formatting and make it easier for me to see my changes. Before these changes would be synced to this PR, but now I'm not seeing them anymore. Is this something that sound familiar to you? I have already tried the solution mentioned here, i.e. updating the base to the same branch and saving it. So far, this hasn't helped, and if I try it again I'm getting an error saying the base is already being updated. I will check later if I can see the changes, but if they don't show up, do you have any tips?

UPDATE some of my changes are starting to show up... so I guess I it's working again. Let's see if the other changes I made later will show up as well!

[groue]

@thinkpractice I have pushed on the groue:grdb_linux_changes branch all changes of your own branch, fully rebased on the latest development branch, with all code formatting removed. I took care that all commits are attributed to you.

I had to heavily amend your initial changes, so I hope I haven't broken or forgotten anything.

If this branch builds as well as yours on Linux, and if you agree, maybe you could force-push this branch to yours? That will make the review, and future changes, much easier.

[thinkpractice]

@groue did you see I spend some time cleaning up the code as well? Hope you could use that. Thanks for taking care of the rest! Will take care to format my code in the right way on any future changes. I am fine with force pushing the changes to my branch if that makes things easier. I am still traveling so I cannot really test your changes now (only brought my MacBook). I will be home next weekend so then I can test everything. Hope that’s ok!

[groue]

did you see I spend some time cleaning up the code as well? Hope you could use that.

Yes, thanks for your efforts! Now starting over from the development branch was the best way to reach the ideal minimal diff.

Will take care to format my code in the right way on any future changes.

I hope .vscode/settings.json solves this for good, yes.

I am fine with force pushing the changes to my branch if that makes things easier. I am still traveling so I cannot really test your changes now (only brought my MacBook). I will be home next weekend so then I can test everything. Hope that’s ok!

If you look at the new commits, you can see that they perform minimal changes. They do indeed make things much easier. I just don't know if I faithfully reported all your changes. So please take your time, check this branch on your Linux system, and make sure I did not forget anything! 🙏