Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate table compression #54

Open
lemire opened this issue Jan 18, 2025 · 2 comments
Open

Investigate table compression #54

lemire opened this issue Jan 18, 2025 · 2 comments

Comments

@lemire
Copy link
Member

lemire commented Jan 18, 2025

We can save about 37KB of tables, while possibly improving the performance.

See

@the-moisrex
Copy link
Contributor

the-moisrex commented Jan 18, 2025

Also Unicode 16.0 has come and has some updates on normalization algorithms and some new code points that change the tables (most likely).

And, also, I remember that you had NFKC and NFKD composition and decomposition code points in your tables which is not being used, but I might be remembering another repo!

These are in your slow path algorithms mostly, so I'm aware they're not a priority.

@the-moisrex
Copy link
Contributor

It's about 110 KiB save:

  • IDNA Mapping tables: yours 84.47 KiB -> mine 45.03 KiB
  • Decomposition tables: yours 73.43 -> mine 41 KiB
  • Composition tables: yours 45.23 KiB -> mine 14 KiB
  • CCC tables: yours 21 KiB -> mine 13 KiB

So, I have saved 224.13 - 113.03 ~= 111 KiB.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants