v0.2.0 5x decreased performance

[erikpro008]

I have tried the new version training on device "mps" Macbook and it is 5x slower than the previous version. The GPU does not seem to utilize 100% because normally I would hear fan noises, so there might be a bottleneck with the 'hyper_connections'.

Happy new year ^^.

[erikpro008]

bottleneck identified :
setting from "num_residual_streams = 4" to "num_residual_streams = 1" increases the speed again.

[lucidrains]

hello Erik and happy new years! yes indeed, had no idea someone would play around with this so soon

num_residual_streams of 1 turns the feature off, so that makes sense

let me see if i can speed it up, and in the worst case, just remove it

does it at least converge for you with > 1 stream?

[moiseshorta]

@lucidrains chiming in on your last question

does it at least converge for you with > 1 stream?

I tried training with the default value of 2 and it does converge, albeit way noisier and slower than without this update.

[lucidrains]

@moiseshorta ah, thanks for reporting that, perhaps it does not play well with the down and upsampling

do you want to try 0.2.2 and see if it is a bit better? i'll add the ability to customize depth to the middle of the unet maybe sometime next week

[moiseshorta]

@lucidrains sure, I'll give it a try. would you have an explanation of what would be the effects of these changes?

I'll write back in some days to see.

just as context: in my experiments I'm trying to see how to optimize for best convergence results and quality in the least amount of time possible, as the code I'm designing will be aimed at users with limited accessibility to GPU compute (i.e. google colab free tier)