Skip to content

QiNCO FSQ #208

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 8 commits into from
Closed

QiNCO FSQ #208

wants to merge 8 commits into from

Conversation

kavorite
Copy link

@kavorite kavorite commented Apr 5, 2025 •
edited
Loading

enable implicit codebooks in residual fsq

kavorite and others added 7 commits April 4, 2025 23:53
@kavorite
qinco+fsq
16aa74c
@kavorite
refactor for better consistency with rvq interface
296efee 
- allow passing in codebook transformations at same call boundaries
- use above
@kavorite
Merge branch 'lucidrains:master' into feat/qinco-fsq
129833e
@kavorite
print worst utilization
12c8d57
@kavorite
consume rvq mlps
cb9890a
@kavorite
apply codebook xforms consistently
9ba1ef1
@kavorite
utilization: apples to apples
9f25623 
report first quantizer utilization so that it's more comparable to
non-residual quantization methods
@kavorite kavorite marked this pull request as draft April 5, 2025 11:14
@kavorite
Copy link
Author

kavorite commented Apr 5, 2025

Results (fashion-mnist):

  • Reconstruction performance under various hyperparams appears to be consistently ~10% better than without qinco (0.116 vs 0.104)
  • Utilization suffers (50% vs 70% under new grid parameters)
  • Runtime, interestingly, takes about a 10% penalty, the same as the performance gains.

@kavorite kavorite marked this pull request as ready for review April 5, 2025 11:25
@lucidrains
Copy link
Owner

@kavorite did you mean to use the codebook transform function here? on second thought, how would this work

@lucidrains lucidrains closed this Apr 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kavorite @lucidrains