Shapefile reader throws java EOFException on valid cadastral shapefile during full scan

[dseerutt]

Using Java 21, Spark 4 locally (unit test) and Sedona 1.8.1, I get this exception trying to read a valid Shapefile file :
26/09/28 16:20:17 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 9) java.io.EOFException at java.base/java.io.DataInputStream.readFully(DataInputStream.java:210) at org.apache.sedona.core.formatMapper.shapefileParser.parseUtils.shp.ShpFileParser$SafeReader.read(ShpFileParser.java:142) at org.apache.sedona.core.formatMapper.shapefileParser.parseUtils.shp.ShpFileParser.parseRecordPrimitiveContent(ShpFileParser.java:97) at org.apache.sedona.core.formatMapper.shapefileParser.shapes.ShapeFileReader.nextKeyValue(ShapeFileReader.java:97) at org.apache.sedona.sql.datasources.shapefile.ShapefilePartitionReader.next(ShapefilePartitionReader.scala:197) at org.apache.spark.sql.execution.datasources.v2.PartitionReaderWithPartitionValues.next(PartitionReaderWithPartitionValues.scala:49) at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.hasNext(DataSourceRDD.scala:122) at org.apache.spark.sql.execution.datasources.v2.MetricsIterator.hasNext(DataSourceRDD.scala:160) at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.$anonfun$hasNext$1(DataSourceRDD.scala:64) at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.$anonfun$hasNext$1$adapted(DataSourceRDD.scala:64) at scala.Option.exists(Option.scala:406) at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.hasNext(DataSourceRDD.scala:64) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:583) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.hashAgg_doAggregateWithoutKey_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:50) at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:583) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:143) at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:57) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:111) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54) at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:171) at org.apache.spark.scheduler.Task.run(Task.scala:147) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$5(Executor.scala:647) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:80) at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:77) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:650) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583)

The file cadastre-06-parcelles-shp.zip used is provided by the French government, it can be downloaded here : https://cadastre.data.gouv.fr/data/[etalab-cadastre/2026-06-01/shp/departements/06/](https://cadastre.data.gouv.fr/data/etalab-cadastre/2026-06-01/shp/departements/06/).

QGIS opens the shapefile successfully.
QGIS validity check reports 51 invalid geometries.
Removing those geometries makes the issue disappear.
However, it is not yet clear whether those geometries are the root cause or whether they expose a bug in the Sedona shapefile reader.

I was able to execute these queries :

• load() ✅
• printSchema() ✅
• limit(775694).count() ✅
• limit(775695).count() ❌

I didn't find a way to bypass this issue as the error happens when Sedona loads the records, I can't exclude nor fix the broken geometries.
Can you please investigate ?

[jiayuasu]

Hi @dseerutt

The invalid geometries are not the cause. The .shx index in that zip does not follow the shapefile spec. Every index entry gives the record's content length as 4 words (8 bytes) larger than the length in the matching .shp record header, and the spec requires the two to be equal. The extra 4 words are exactly the size of the record header. The .shp itself is fine.

Sedona read each record using the .shx length and only ever moved forward in the file. So it read 8 bytes into the next record every time, and the offset grew by 8 bytes per record until it ran past the end of the file. With the 2026-06-01 file for département 06, that happens after 776,497 of the 792,590 records. QGIS finds each record by its offset, so the extra length does no harm there. When you removed the invalid geometries, QGIS saved a new, correct .shx, which is why the error went away.

Note that count() hid a second problem: geometries from the second row on were parsed from the wrong bytes, and most came back as null. Any results you got from this file with the .shx present are wrong, even where no exception was t

Until the fix is released, delete or rename parcelles.shx. Without an index, Sedona reads the .shp record by record using its own headers. I checked this on your file: all 792,590 polygons come back correctly.

The fix is in #3422 . With it, Sedona seeks to each .shx offset and takes the record length from the .shp header. It should be in the next release.

[dseerutt]

Hi,
Thank you very much for this very detailed answer.
I will be able to use your fix soon, in the end of October if I understand your milestones well.
I also would like to thank you for your temporary removal fix, it's very handy.

Best regards