Plugin forking threads for each chunk

[djmetzle]

We're seeing a threading and concurrency problem trying to upgrade our CC webpack plugin.

We've moved from the old v3 version:
https://www.npmjs.com/package/webpack-closure-compiler

To the latest v4 plugin:
https://www.npmjs.com/package/closure-webpack-plugin

We're seeing a pretty wild performance degradation here. The previous v3 version used a tuneable concurrency flag to limit the number of parallel compilations performed by the plugin:
https://github.com/roman01la/webpack-closure-compiler/blob/3677e5e6672af68eb4608491d39d9e39ba8bf55e/index.js#L99

This would use async.queue to only execute CC at a specified concurrency.

Testing the new plugin with the default configuration from the docs we see a separate thread spun for every chunk in our webpack compilation. We saw thousands of java threads, and knocked down our development environments.

For additional detail, we tested some different combinations of options. Using any of the platform options does not affect the result. We can use platform: javascript and then still see hundreds of node threads, as opposed to JVM threads.

We also tested switching the mode to AGRESSIVE_BUNDLE. When we run in AGRESSIVE mode though, we do not see more than one compilation at a time. Our compilation is not compatible with that option, but it is worth noting that the problem does not happen when using the AGRESSIVE option.

---

On investigation, it seems like the cause of this explosion in threads might be a bug. The code bifurcates on the bundling type:

closure-webpack-plugin/src/closure-compiler-plugin.js

Lines 448 to 452 in 1246f6e

	if (this.options.mode === 'AGGRESSIVE_BUNDLE') {
	this.aggressiveBundle(compilation, originalChunks, cb);
	} else {
	this.standardBundle(compilation, originalChunks, cb);
	}

In the standard codepath, there is a forEach over each chunk in the compilation:

closure-webpack-plugin/src/closure-compiler-plugin.js

Line 467 in 1246f6e

originalChunks.forEach((chunk) => {

which pushes the chunk onto a compilations list:

closure-webpack-plugin/src/closure-compiler-plugin.js

Lines 497 to 498 in 1246f6e

	compilations.push(
	this.runCompiler(compilation, compilationOptions, sources, chunkDefs)

And then Promise.alls them:

closure-webpack-plugin/src/closure-compiler-plugin.js

Line 553 in 1246f6e

Promise.all(compilations)

But the AGRESSIVE code path only does a single compilation promise:

closure-webpack-plugin/src/closure-compiler-plugin.js

Lines 776 to 777 in 1246f6e

	this.runCompiler(compilation, compilationOptions, allSources)
	.then((outputFiles) => {

We suspect this explains the explosion in background work, and the difference in behavior between STANDARD and AGRESSIVE.

---

We're running webpack v4 on node 12 with an otherwise uninteresting configuration. 270 entrypoints, 2500 modules, and about 300 emitted chunks.

[ChadKillingsworth]

In AGRESSIVE mode, all of the chunks are passed to the compiler in one single compilation. The dependency tree is normalized and relationships are clearly defined.

In STANDARD mode, each entry point equates to a compilation. In this mode, webpack can duplicate a module into multiple output chunks. Closure Compiler doesn't support this so each chunk tree be compiled individually.

Your use case sounds like it's pushing limits here. What would be needed is a thread manager to limit the number of parallel compilations.

[djmetzle]

@ChadKillingsworth
Thanks for the reply!

Your use case sounds like it's pushing limits here. What would be needed is a thread manager to limit the number of parallel compilations.

We weren't having problems with the v3 plugin though. The previous v3 version used a tuneable concurrency flag to limit the number of parallel compilations performed by the plugin.

I think that we need something like async.queue, or some other approach, like a "thread manager" to limit parallel work.