[SystemDS-#3524] Multi-threading of transformdecode/[SystemDS-#3521] Improved Feature Transformations #2275

Isso-W · 2025-06-19T19:46:55Z

This pull request introduces a new framework for column decoding in Apache SystemDS, with the addition of a base class ColumnDecoder and several specialized implementations (ColumnDecoderBin, ColumnDecoderComposite, ColumnDecoderRecode,ColumnDecoderPassthrough and ColumnDecoderDummycode). These changes provide a flexible and extensible structure for decoding encoded data in matrix-to-frame transformations. Below are the most important changes grouped by theme:

Core Framework for Column Decoding

Added ColumnDecoder as an abstract base class to define the structure for decoding operations, including methods for decoding (columnDecode), handling sub-range decoding (subRangeDecoder), and metadata initialization (initMetaData). It also implements Externalizable for efficient serialization.

Current Issues

ColumnDecoderDummycode ist not supported yet, as well as the test case ColumnDecoderMixedMethodsTest
There are still some issues with DecoderDummyCode, it do not work together with DecoderRecode

…e done, test passes for Bin

codecov · 2025-06-20T11:37:54Z

Codecov Report

Attention: Patch coverage is 52.98913% with 173 lines in your changes missing coverage. Please review.

Project coverage is 72.55%. Comparing base (11c5a74) to head (64651ec).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
.../runtime/transform/decode/ColumnDecoderRecode.java	45.33%	39 Missing and 2 partials ⚠️
.../sysds/runtime/transform/decode/ColumnDecoder.java	35.71%	27 Missing ⚠️
...runtime/transform/decode/ColumnDecoderFactory.java	65.71%	23 Missing and 1 partial ⚠️
...sds/runtime/transform/decode/ColumnDecoderBin.java	57.40%	21 Missing and 2 partials ⚠️
...ntime/transform/decode/ColumnDecoderDummycode.java	55.76%	20 Missing and 3 partials ⚠️
...ntime/transform/decode/ColumnDecoderComposite.java	52.27%	21 Missing ⚠️
...ime/transform/decode/ColumnDecoderPassThrough.java	47.61%	11 Missing ⚠️
...sds/runtime/transform/decode/DecoderComposite.java	0.00%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2275      +/-   ##
============================================
- Coverage     72.58%   72.55%   -0.04%     
- Complexity    46221    46275      +54     
============================================
  Files          1489     1496       +7     
  Lines        174193   174561     +368     
  Branches      34182    34232      +50     
============================================
+ Hits         126434   126646     +212     
- Misses        38196    38347     +151     
- Partials       9563     9568       +5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

phaniarnab · 2025-06-24T21:24:07Z

Thank you for the patch. I will take a look into this next week @Isso-W.
Meanwhile, please add the missing license headers to the new files.

small error in test array

# Conflicts: # src/test/java/org/apache/sysds/test/functions/transform/ColumnDecoderMixedMethodsTest.java

phaniarnab · 2025-07-03T09:51:50Z

src/main/java/org/apache/sysds/runtime/transform/decode/ColumnDecoderRecode.java

+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
+import java.util.*;


Avoid all import. Only import the required classes.

phaniarnab · 2025-07-03T09:54:22Z

src/main/java/org/apache/sysds/runtime/transform/decode/ColumnDecoderRecode.java

+                for( int j=0; j<_colList.length; j++ ) {
+                    int colID = _colList[j];
+                    double val = UtilFunctions.objectToDouble(
+                            out.getSchema()[colID-1], out.get(i, colID-1));
+                    long key = UtilFunctions.toLong(val);
+                    out.set(i, colID-1, getRcMapValue(j, key));
+                }


Why are you iterating all the columns? A column decoder should be called for each column.

phaniarnab · 2025-07-03T09:55:47Z

src/main/java/org/apache/sysds/runtime/transform/decode/DecoderRecode.java

+ 		if( _onOut ) { //recode on output (after dummy)
 			for( int i=rl; i<ru; i++ ) {
 				for( int j=0; j<_colList.length; j++ ) {
 					int colID = _colList[j];
 					double val = UtilFunctions.objectToDouble(
 							out.getSchema()[colID-1], out.get(i, colID-1));
+


Remove these empty lines that you added.

phaniarnab · 2025-07-03T09:59:19Z

src/main/java/org/apache/sysds/runtime/transform/decode/ColumnDecoder.java

+    protected int[] _colList;
+    protected String[] _colnames = null;
+    protected ColumnDecoder(ValueType[] schema, int[] colList) {
+        _schema = schema;
+        _colList = colList;
+    }


Why a column list? A column encoder should work on a single column.

phaniarnab · 2025-07-03T10:00:33Z

src/main/java/org/apache/sysds/runtime/transform/decode/ColumnDecoderBin.java

+        long b1 = System.nanoTime();
+        out.ensureAllocatedColumns(in.getNumRows());
+
+        final int outColIndex = _colList[0] - 1;


Why the outColIndex is always _colList[0] - 1?

phaniarnab · 2025-07-03T10:01:33Z

src/main/java/org/apache/sysds/runtime/transform/decode/ColumnDecoderBin.java

+            for (int j = 0; j < _colList.length; j++) {
+                double val = in.get(i, j);
+                if (!Double.isNaN(val)) {
+                    int key = (int) Math.round(val);
+                    double bmin = _binMins[j][key - 1];
+                    double bmax = _binMaxs[j][key - 1];
+                    double oval = bmin + (bmax - bmin) / 2 + (val - key) * (bmax - bmin);
+                    out.getColumn(_colList[j] - 1).set(i, oval);
+                } else {
+                    out.getColumn(_colList[j] - 1).set(i, val);
+                }
+            }


I don't understand why you are iterating all columns in a column decoder.

phaniarnab · 2025-07-03T10:03:12Z

src/main/java/org/apache/sysds/runtime/transform/decode/ColumnDecoderDummycode.java

+                for( int j=0; j<_colList.length; j++ )
+                    for( int k=_clPos[j]; k<_cuPos[j]; k++ )
+                        if( in.get(i, k-1) != 0 ) {
+                            int col = _colList[j] - 1;
+                            Object val = UtilFunctions.doubleToObject(out.getSchema()[col], k-_clPos[j]+1);
+                            synchronized(out) { out.set(i, col, val); }
+                        }
+        }


A column decoder should work on a single column that is provided.

phaniarnab · 2025-07-03T10:06:32Z

@Isso-W, please address the comments. And fix the tests as well. Your tests are failing in the transform package, which you should be able to reproduce.

…erly on Bin and Pass-through.

# Conflicts: # src/test/java/org/apache/sysds/test/functions/transform/ColumnDecoderMixedMethodsTest.java

phaniarnab · 2025-07-21T07:39:29Z

The latest changes look good @Isso-W.
Please make sure all the unit tests pass and we get a green tick.
As discussed, for the remaining time, focus on extending the FTBench benchmark with decoding scripts and report performance improvement by multithreading for those use cases.

phaniarnab · 2025-08-10T10:03:51Z

@Isso-W, can you please post your plots of FTBench (decoding) here for others to see?

Isso-W · 2025-08-10T18:52:26Z

Sure @phaniarnab, here is the plot.
Full decode means the time cost for whole decode function in dml
Pure decoder means the time cost for decoder part in decoder function in Java code
And there is an additional comparison for these both.

I also created a new test using part of FTbench T9 by running DML, I can also put that test with responding json in PR to make the plot reproduciable. B.t.w. will this PR be merged into main?

….csv, flight.csv is too large to push

Isso-W and others added 17 commits May 17, 2025 06:07

Basic Structure for ColumnDecoder

365d2b8

Basic Structure for ColumnDecoder

d5dc7c1

Main logic in ColumnDecoderComposite done, ColumnDecoderBin as exampl…

3712b9c

…e done, test passes for Bin

update two encode methods for PassThrough and Recode

ca836a4

Main logic in ColumnDecoderComposite done, ColumnDecoderBin as exampl…

f43a853

…e done, test passes for Bin

Mixed test and optimization on Pass through

84077b0

Middle version

8fcbbcc

Middle version

c02fa76

Middle version

3c66fcc

Middle version

8da2e0d

Middle version

19a4437

Bin, Recode, Passthrough done

34a4341

update dummy funtion for Processing columns in parallel

0c5acdb

Middle version

98865e7

Middle version

2886537

Middle version

b75b7d1

Update pom.xml

61f6d39

github-project-automation bot added this to SystemDS PR Queue Jun 19, 2025

github-project-automation bot moved this to In Progress in SystemDS PR Queue Jun 19, 2025

Isso-W and others added 6 commits June 30, 2025 12:15

Add Licenses to new classes

839b050

Merge remote-tracking branch 'origin/main'

595f4b7

Update ColumnDecoderMixedMethodsTest.java

bfcb133

small error in test array

little fix to multi-thread, update time measure methode

0e67c0a

Merge remote-tracking branch 'origin/main'

91fbc54

# Conflicts: # src/test/java/org/apache/sysds/test/functions/transform/ColumnDecoderMixedMethodsTest.java

multi-thread fix, bin optimization

ec7fe78

phaniarnab reviewed Jul 3, 2025

View reviewed changes

Huge refactor: Changed input structure for decoders. Now working prop…

6dfa925

…erly on Bin and Pass-through.

EmilyHongYT and others added 13 commits July 10, 2025 01:22

update record funtion for Processing columns and test it

b8c4f88

small fixes on bin

d62f358

Merge remote-tracking branch 'origin/main'

56e7cd9

# Conflicts: # src/test/java/org/apache/sysds/test/functions/transform/ColumnDecoderMixedMethodsTest.java

update recode & dummy funtion for Processing columns and test it

d7a3228

update Factory & PassThrough part for Processing columns and test it

0a35c28

print runtime for dummy and recode when processing single column

81c405f

fixed dummy decoder

91cc343

update recode & dummy funtion for Processing columns and test it

8448990

update Factory & PassThrough part for Processing columns and test it

dd14314

print runtime for dummy and recode when processing single column

41cb69b

fixed dummy decoder

ba3dc04

little improvement

bb92e76

Bug fixes and replace decoder

abafe03

Isso-W and others added 4 commits July 21, 2025 11:41

Update pom.xml

9b67321

Merge branch 'apache:main' into main

64651ec

cleaned code and add documentations

808cba2

Merge remote-tracking branch 'origin/main'

0156c58

remove internal time counter, add test class using FT Benchmark adult…

337334e

….csv, flight.csv is too large to push

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SystemDS-#3524] Multi-threading of transformdecode/[SystemDS-#3521] Improved Feature Transformations #2275

[SystemDS-#3524] Multi-threading of transformdecode/[SystemDS-#3521] Improved Feature Transformations #2275

Uh oh!

Isso-W commented Jun 19, 2025

Uh oh!

codecov bot commented Jun 20, 2025 •

edited

Loading

Uh oh!

phaniarnab commented Jun 24, 2025

Uh oh!

phaniarnab Jul 3, 2025

Uh oh!

phaniarnab Jul 3, 2025

Uh oh!

phaniarnab Jul 3, 2025

Uh oh!

phaniarnab Jul 3, 2025

Uh oh!

phaniarnab Jul 3, 2025

Uh oh!

phaniarnab Jul 3, 2025

Uh oh!

phaniarnab Jul 3, 2025

Uh oh!

phaniarnab commented Jul 3, 2025

Uh oh!

phaniarnab commented Jul 21, 2025

Uh oh!

phaniarnab commented Aug 10, 2025

Uh oh!

Isso-W commented Aug 10, 2025

Uh oh!

Uh oh!

[SystemDS-#3524] Multi-threading of transformdecode/[SystemDS-#3521] Improved Feature Transformations #2275

Are you sure you want to change the base?

[SystemDS-#3524] Multi-threading of transformdecode/[SystemDS-#3521] Improved Feature Transformations #2275

Uh oh!

Conversation

Isso-W commented Jun 19, 2025

Core Framework for Column Decoding

Current Issues

Uh oh!

codecov bot commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

phaniarnab commented Jun 24, 2025

Uh oh!

phaniarnab Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

phaniarnab Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

phaniarnab Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

phaniarnab Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

phaniarnab Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

phaniarnab Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

phaniarnab Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

phaniarnab commented Jul 3, 2025

Uh oh!

phaniarnab commented Jul 21, 2025

Uh oh!

phaniarnab commented Aug 10, 2025

Uh oh!

Isso-W commented Aug 10, 2025

Uh oh!

Uh oh!

codecov bot commented Jun 20, 2025 •

edited

Loading