swift: Runtime error (`EXC_BAD_ACCESS`) when performing certain Postgres operations via PostgresNIO.

Description This occurs when the program is built in Release mode.

This program makes a connection to a remote Postgres database instance and attempts to run: DELETE FROM "SomeTable" WHERE "channel" = $1 Although the connection successfully closes, a runtime error on the Swift end is generated.

Steps to reproduce Install Docker and Postgres locally if not already installed.

First, set up the Postgres server and database (here, we listen on port 5432):

docker run --detach --name test --restart=unless-stopped --publish 5432:5432 --env POSTGRES_HOST_AUTH_METHOD=trust postgres:latest 

Insert the table that we will attempt to delete from the Swift program:

psql -h localhost -U postgres

postgres=# CREATE TABLE "SomeTable" ( channel varchar(40) );
CREATE TABLE
postgres=# exit

The Swift program requires the following packages:

NIO and NIOConcurrencyHelpers (https://github.com/apple/swift-nio.git)
Vapor (https://github.com/vapor/vapor.git)
RxSwift (https://github.com/ReactiveX/RxSwift.git)
PostgresKit (https://github.com/vapor/postgres-kit.git)
SQLKit (https://github.com/vapor/sql-kit.git)

Copy and paste this program into a new project with the above packages, build in Release mode, and run, while the Docker image and Postgres server are running:

import NIO; import Vapor; import RxSwift; import PostgresKit; import SQLKit; import NIOConcurrencyHelpers
let eventLoopGroup = MultiThreadedEventLoopGroup(numberOfThreads: 1)
print(TestInterface.postgres())
class Driver {
    let pool: EventLoopGroupConnectionPool<PostgresConnectionSource>
    private var connection: PostgresConnection?
    init() {
        let configuration = PostgresConfiguration(hostname: "localhost", port: 5432, username: "postgres", password: "", database: "postgres")
        pool = EventLoopGroupConnectionPool(source: PostgresConnectionSource(configuration: configuration), maxConnectionsPerEventLoop: 1, requestTimeout: .seconds(10), on: eventLoopGroup)
    }
}
internal final class PGTriggerManifest: TriggerDriver {
    private let pool: EventLoopGroupConnectionPool<PostgresConnectionSource>
    private let channel: String
    internal init(pool: EventLoopGroupConnectionPool<PostgresConnectionSource>, channel: String) {self.pool = pool; self.channel = channel}
    internal func deactivate() -> EventLoopFuture<Void> {self.unregisterListener().map {}.flatMap { self.deleteChannelTriggers() }}
    private func deleteChannelTriggers() -> EventLoopFuture<Void> {retryOnDeadlock {let db = self.pool.database(logger: Logger(label: "\(PGTriggerManifest.self):DBLogger")).sql(); return TriggerManifest.Latest.deleteTriggers(channel: self.channel, from: db)}}
    private func retryOnDeadlock<Result>(maxRetries: Int = 3, _ query: @escaping () -> EventLoopFuture<Result>) -> EventLoopFuture<Result> {
        let promise = pool.eventLoopGroup.next().makePromise(of: Result.self)
        func attempt() {let future = query(); future.cascadeSuccess(to: promise)}
        attempt(); return promise.futureResult
    }
    private func unregisterListener() -> EventLoopFuture<Void> {return pool.eventLoopGroup.future()}
}
public protocol TriggerDriver {func deactivate() -> EventLoopFuture<Void>}
public class TestInterface {
     let triggerManifest: TriggerDriver
     public init(triggerManifest: TriggerDriver, eventLoopGroup: EventLoopGroup) {self.triggerManifest = triggerManifest}
     public func deactivateTriggerManifest() -> EventLoopFuture<Void> {triggerManifest.deactivate()}
     deinit {do {try self.deactivateTriggerManifest().wait()} catch {}}
     public func withTransaction<T>(_ closure: @escaping ((TestInterface) -> EventLoopFuture<T>)) -> EventLoopFuture<T> {
         eventLoopGroup.makeFutureWithTask {
             func internalClosure(transaction: TestInterface) async throws -> T {try await closure(transaction).get()}
             return try await self.withTransaction(internalClosure)
         }
     }
     public func withTransaction<T>(_ closure: @escaping ((TestInterface) async throws -> T)) async throws -> T {fatalError()}
     static func postgres() -> TestInterface {return TestInterface(triggerManifest: PGTriggerManifest(pool: Driver().pool, channel: UUID().uuidString), eventLoopGroup: eventLoopGroup)}
 }
public enum TriggerManifest {}
public extension TriggerManifest {enum Latest {public static let schema = "SomeTable"; public enum FieldKeys {public static let channel: String = "channel"}}}
public extension TriggerManifest.Latest {static func deleteTriggers(channel: String, from sqlDatabase: SQLDatabase) -> EventLoopFuture<Void> {sqlDatabase.delete(from: schema).where(SQLIdentifier(FieldKeys.channel), .equal, SQLBind(channel)).run()}}

Expected behavior The program should continue to run, without crashing, after the transaction completes.

Environment

  • Swift compiler version info: Toolchain 2023-03-26a
  • Xcode version info: 14.2
  • Deployment target: macOS 13.2.1.

Additional Information The location of the crash ends up being somewhere in final class PSQLRowStream, which is a part of the postgres-nio package. It often occurs in line 54, which contains self.logger = queryContext.logger, but sometimes, the crash occurs a few lines above (in the switch rowSource statement).

If port 5432 is busy, and the server needs to run on a different port, don’t forget to change the corresponding port in line 8 of the reproducer.

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 18 (13 by maintainers)

Commits related to this issue

Most upvoted comments

@weissi - Sorry for the slow reply, this does indeed appear to be fixed as of the above commit. We no longer observe this segmentation fault in optimized builds of impacted code in nightly or 5.9 release toolchains since the patch on our end.

This revision ought to print “Ready to exit.” in Debug mode, and then wait. Since it was originally designed to be a server, it is meant to be running continuously. It’ll be sitting at try self.close().wait(); when the process can be stopped. I’ve verified with a packet sniffer that the DB transaction goes through before this point. In Release mode, the code will crash as before.

import NIO; import Vapor; import RxSwift; import PostgresKit; import SQLKit; import NIOConcurrencyHelpers
let eventLoopGroup = MultiThreadedEventLoopGroup(numberOfThreads: 1)
print(TestInterface.postgres())
class Driver {
    let pool: EventLoopGroupConnectionPool<PostgresConnectionSource>
    private var connection: PostgresConnection?
    init() {
        let configuration = PostgresConfiguration(hostname: "localhost", port: 5432, username: "postgres", password: "", database: "postgres")
        pool = EventLoopGroupConnectionPool(source: PostgresConnectionSource(configuration: configuration), maxConnectionsPerEventLoop: 1, requestTimeout: .seconds(10), on: eventLoopGroup)
    }
}
internal final class PGTriggerManifest: TriggerDriver {
    private let pool: EventLoopGroupConnectionPool<PostgresConnectionSource>
    private let channel: String
    internal init(pool: EventLoopGroupConnectionPool<PostgresConnectionSource>, channel: String) {self.pool = pool; self.channel = channel}
    internal func deactivate() -> EventLoopFuture<Void> {self.unregisterListener().map {}.flatMap { self.deleteChannelTriggers() }}
    private func deleteChannelTriggers() -> EventLoopFuture<Void> {retryOnDeadlock {let db = self.pool.database(logger: Logger(label: "\(PGTriggerManifest.self):DBLogger")).sql(); return TriggerManifest.Latest.deleteTriggers(channel: self.channel, from: db)}}
    private func retryOnDeadlock<Result>(maxRetries: Int = 3, _ query: @escaping () -> EventLoopFuture<Result>) -> EventLoopFuture<Result> {
        let promise = pool.eventLoopGroup.next().makePromise(of: Result.self)
        func attempt() {let future = query(); future.cascadeSuccess(to: promise)}
        attempt(); return promise.futureResult
    }
    private func unregisterListener() -> EventLoopFuture<Void> {return pool.eventLoopGroup.future()}
}
public protocol TriggerDriver {func deactivate() -> EventLoopFuture<Void>}
public class TestInterface {    
    let triggerManifest: TriggerDriver
    public init(triggerManifest: TriggerDriver, eventLoopGroup: EventLoopGroup) {self.triggerManifest = triggerManifest; self.eventLoopGroup = eventLoopGroup}
    public func deactivateTriggerManifest() -> EventLoopFuture<Void> {triggerManifest.deactivate()}
    deinit {do {print("Ready to exit."); try self.close().wait(); try self.deactivateTriggerManifest().wait()} catch {}}
    public func withTransaction<T>(_ closure: @escaping ((TestInterface) -> EventLoopFuture<T>)) -> EventLoopFuture<T> {
        eventLoopGroup.makeFutureWithTask {
            func internalClosure(transaction: TestInterface) async throws -> T {try await closure(transaction).get()}
            return try await self.withTransaction(internalClosure)
        }
    }
    public func withTransaction<T>(_ closure: @escaping ((TestInterface) async throws -> T)) async throws -> T {fatalError()}
    static func postgres() -> TestInterface {return TestInterface(triggerManifest: PGTriggerManifest(pool: Driver().pool, channel: UUID().uuidString), eventLoopGroup: MultiThreadedEventLoopGroup(numberOfThreads: 1))}
    public func close() -> EventLoopFuture<Void> {(true ? deactivateTriggerManifest() : eventLoopGroup.future()).flatMapAlways { _ in return self.eventLoopGroup.next().makePromise().futureResult }}
    public let eventLoopGroup: EventLoopGroup
}
public enum TriggerManifest {}
public extension TriggerManifest {enum Latest {public static let schema = "SomeTable"; public enum FieldKeys {public static let channel: String = "channel"}}}
public extension TriggerManifest.Latest {static func deleteTriggers(channel: String, from sqlDatabase: SQLDatabase) -> EventLoopFuture<Void> {sqlDatabase.delete(from: schema).where(SQLIdentifier(FieldKeys.channel), .equal, SQLBind(channel)).run()}}

@weissi - This does still reproduce with top-of-tree Swift. Steps to reproduce are still a bit clunky, because it requires interacting with a Postgres database. If you follow the instructions above to set up the server and database, then insert the table that’s needed, I’ve attached a SwiftPM project that segfaults on a swift run -c release.

The issue has been isolated to the avoidingStateMachineCoW function above, and a version of PostgresNIO that does away with that function does not lead to this segmentation fault. We’re still looking into what specifically is going wrong with that function.

PostgresNIOCrasher.zip