Golang | Pulsar Functions Runtime Death Exception

[divya-inaza]

I am getting the below error while running a pulsar function created using GO SDK.

Function Container is dead with following exception. Restarting.
java.lang.RuntimeException:
at org.apache.pulsar.functions.runtime.process.ProcessRuntime.tryExtractingDeathException(ProcessRuntime.java:400) ~[org.apache.pulsar-pulsar-functions-runtime-2.10.1.jar:2.10.1]
at org.apache.pulsar.functions.runtime.process.ProcessRuntime.isAlive(ProcessRuntime.java:387) ~[org.apache.pulsar-pulsar-functions-runtime-2.10.1.jar:2.10.1]
at org.apache.pulsar.functions.runtime.RuntimeSpawner.lambda$start$0(RuntimeSpawner.java:88) ~[org.apache.pulsar-pulsar-functions-runtime-2.10.1.jar:2.10.1]
at org.apache.pulsar.common.util.Runnables$CatchingAndLoggingRunnable.run(Runnables.java:54) [org.apache.pulsar-pulsar-common-2.10.1.jar:2.10.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]

[flowchartsman]

This is unfortunately a pretty generic response. Debugging pulsar function crashes in go can be kind of a pain, especially if they happen before pf.Start is called, since there's no logging at that point. Usually you can find the more specific cause in the function worker or broker logs, depending on how you're running the function. If you can, pulsar-admin localrun can potentially provide some insight.

[divya-inaza]

Thanks for your response.

The Create API returns a successful response. However, the function stops running due to this Death Exception error.

When I try to restart the function, I receive a 204 response. In spite of this, when I try to get the status of the function, it gives me the below error.

standalone_1 | 2022-12-03T11:50:35,902+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L765 E3 from bookie: 127.0.0.1:3181
standalone_1 | 2022-12-03T11:50:35,902+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.client.PendingReadOp - Read of ledger entry failed: L765 E3-E3, Sent to [127.0.0.1:3181], Heard from [] : bitset = {}, Error = 'Bookie operation timeout'. First unread entry is (-1, rc = null)
standalone_1 | 2022-12-03T11:50:36,057+0000 [main] ERROR org.apache.pulsar.functions.worker.FunctionActioner - t09/n09/FOURTHFUNC Error starting function
standalone_1 | org.apache.distributedlog.exceptions.BKTransmitException: Encountered issue on reading entry 0 @ log segment [LogSegmentId:765, firstTxId:1024, lastTxId:9223372036854775807, version:VERSION_V5_SEQUENCE_ID, completionTime:1670066952044, recordCount:42692, regionId:0, status:0, logSegmentSequenceNumber:1, lastEntryId:171, lastSlotId:0, inprogress:false, minActiveDLSN:DLSN{logSegmentSequenceNo=1, entryId=0, slotId=0}, startSequenceId:0] : -23
standalone_1 | at org.apache.distributedlog.impl.logsegment.BKLogSegmentEntryReader.readEntriesFromReadAheadCache(BKLogSegmentEntryReader.java:814) ~[org.apache.distributedlog-distributedlog-core-4.14.5.jar:4.14.5]
standalone_1 | at org.apache.distributedlog.impl.logsegment.BKLogSegmentEntryReader.safeRun(BKLogSegmentEntryReader.java:733) ~[org.apache.distributedlog-distributedlog-core-4.14.5.jar:4.14.5]
standalone_1 | at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.14.5.jar:4.14.5]
standalone_1 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
standalone_1 | at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
standalone_1 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
standalone_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
standalone_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
standalone_1 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
standalone_1 | at java.lang.Thread.run(Thread.java:829) ~[?:?]

It would be great if you can take a look at this.

[divya-inaza]

It was just the build issue.

I ran the below command before the go build command & it fixed my issue.

$Env:GOOS = "linux"; $Env:GOARCH = "amd64"