Skip to content

Conversation

LinMingQiang
Copy link
Contributor

Purpose

When I execute a spark 3.4- UT in idea, such as DDLTest, get NoClassDefFoundError

Caused by: java.lang.NoClassDefFoundError: org/apache/parquet/hadoop/ParquetEmptyBlockException
	at org.apache.paimon.format.parquet.ParquetUtil.getParquetReader(ParquetUtil.java:81)
	at org.apache.paimon.format.parquet.ParquetUtil.extractColumnStats(ParquetUtil.java:51)
	at org.apache.paimon.format.parquet.ParquetSimpleStatsExtractor.extractWithFileInfo(ParquetSimpleStatsExtractor.java:78)
	at org.apache.paimon.format.parquet.ParquetSimpleStatsExtractor.extract(ParquetSimpleStatsExtractor.java:71)
	at org.apache.paimon.io.StatsCollectingSingleFileWriter.fieldStats(StatsCollectingSingleFileWriter.java:105)
	at org.apache.paimon.io.RowDataFileWriter.result(RowDataFileWriter.java:112)

There is a parquet dependency conflict in spark 3.4-, spark 3.5 does not have this problem because its parquet version does not conflict.

截屏2025-03-02 11 57 47

Linked issue: close #xxx

Tests

API and Format

Documentation

@JingsongLi
Copy link
Contributor

+1

@JingsongLi JingsongLi merged commit a968660 into apache:master Mar 3, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants