Skip to content

Commit 59994ec

Browse files
authored
Add results from large-scale multiclass classification benchmarks (#436)
No need for a changelog entry as this change impacts no users.
1 parent 4a9d91b commit 59994ec

File tree

1 file changed

+15
-1
lines changed

1 file changed

+15
-1
lines changed

benchmark/README.md

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -194,4 +194,18 @@ Experimental setting: 20 Optuna search trials for XGBoost, CatBoost and LightGBM
194194

195195
#### `scale: large`
196196

197-
TODO
197+
Experimental setting: 20 Optuna search trials for XGBoost, CatBoost and LightGBM. 3 Optuna search trials and 10 epochs training for deep learning models.
198+
199+
| | dataset_0 | dataset_1 | dataset_2 |
200+
| :------------------ | :--------------------- | :--------------------- | :----------------------- |
201+
| XGBoost | Too slow\* | Too slow\* | Too slow\* |
202+
| CatBoost | Too slow\* | Too slow\* | Too slow\* |
203+
| LightGBM | Too slow\* | Too slow\* | Too slow\* |
204+
| Trompt | OOM | 0.889±0.063 (55428s) | 0.804±0.013 (23304s) |
205+
| ResNet | 0.892±0.002 (417s) | **0.999±0.001 (396s)** | 0.915±0.001 (405s) |
206+
| MLP | 0.770±0.001 (170s) | 0.549±0.000 (223s) | 0.895±0.001 (192s) |
207+
| FTTransformerBucket | 0.897±0.004 (4436s) | 0.502±0.000 (1892s) | 0.888±0.009 (4414s) |
208+
| ExcelFormer | OOM | TODO (code error) | **0.951±0.002 (10236s)** |
209+
| FTTransformer | 0.872±0.005 (7004s) | 0.540±0.068 (3355s) | 0.908±0.004 (7514s) |
210+
| TabNet | **0.912±0.004 (219s)** | 0.995±0.001 (301s) | 0.919±0.003 (187s) |
211+
| TabTransformer | 0.843±0.003 (2810s) | 0.657±0.187 (2843s) | 0.854±0.001 (284s) |

0 commit comments

Comments
 (0)