๐ฅ Fast State-of-the-Art Tokenizers optimized for Research and Production + merge frequencies
Rust ยท active 2025-03-27 โ 2026-08-10 (UTC)
Activity over time
Daily event counts in the loaded window
Line chart, 502 days from 2025-03-27 to 2026-08-10. Pushes: 404 total, peak 17 in a day. Pull requests: 101 total, peak 9 in a day. Issues: 63 total, peak 7 in a day. Comments: 344 total, peak 44 in a day. Stars: 534 total, peak 10 in a day.
- Pushes
- Pull requests
- Issues
- Comments
- Stars
Stars, PRs, issues and forks are under-captured in the later part of this window. GH Archive progressively stopped capturing non-push events during 2026 โ โ95% or worse by the end of the window. Every series here except Pushes fades for that reason, so a decline above reflects the archive, not this repository. Pushes stay reliable throughout, so read them, and the contributor counts derived from them, as the real signal. Data health has the measurements.
Top contributors
Pushes, PRs, issues, reviews and comments โ stars and forks excluded, so this is contribution rather than popularity
| Contributor | Contributions | Pushes | PRs | Comments |
|---|---|---|---|---|
| ArthurZucker | 497 | 269 | 29 | 125 |
| SBrandeis | 77 | 74 | 2 | 0 |
| McPatate | 49 | 28 | 4 | 12 |
| Narsil | 45 | 16 | 11 | 12 |
| HuggingFaceDocBuilderDev | 36 | 0 | 0 | 36 |
| ngoldbaum | 17 | 0 | 1 | 15 |
| Copilot | 17 | 1 | 0 | 10 |
| dependabot[bot] | 10 | 0 | 10 | 0 |
| michaelfeil | 8 | 0 | 2 | 3 |
| sftse | 7 | 0 | 0 | 6 |
| sanderland | 6 | 0 | 0 | 3 |
| paulinebm | 6 | 6 | 0 | 0 |
| b00f | 5 | 0 | 2 | 2 |
| cclauss | 5 | 0 | 2 | 3 |
| stephantul | 5 | 0 | 0 | 5 |
| mullerhai | 4 | 0 | 0 | 3 |
| github-actions[bot] | 4 | 0 | 0 | 4 |
| zhimin-z | 4 | 0 | 0 | 4 |
| assafvayner | 4 | 4 | 0 | 0 |
| hf-security-analysis[bot] | 3 | 2 | 1 | 0 |
Recent activity
Latest issues, pull requests and releases
- Pull request#2298SBrandeis2026-08-05 21:54
- Pull request#2255SBrandeis2026-07-28 21:53
- Pull request#2136McPatate2026-07-01 20:14
- Pull request#2087pjdurden2026-06-06 23:59
- Pull request#2078SMC172026-05-29 00:20
- Pull request#2077sebpop2026-05-28 23:09
- Pull request#2072sebpop2026-05-26 17:33
- Issue comment#1684ArthurZucker2026-05-25 07:31Prebuilding tokenizers for Windows arm
- Pull request#2064adityasingh24002026-05-21 23:31
- Pull request#2056ArthurZucker2026-05-13 02:42
- Issue comment#2049HuggingFaceDocBuilderDev2026-05-02 13:50Add encode_byte_offsets methods for byte-level alignment
- Pull request#2049kashif2026-05-02 13:47
- Issue comment#2035Brianzhengca2026-05-02 03:48Massive performance degradation and memory spike when using text_pair with long inputs
- Issue comment#2044vyalamar2026-04-28 22:07Batch encode: simple lock-free scheduler
- Issue comment#2044vyalamar2026-04-28 18:39Batch encode: simple lock-free scheduler
- Issue comment#2041github-actions[bot]2026-04-27 05:59bindings/python: free-threaded Python (3.14t) support
- Pull request#2041ArthurZucker2026-04-27 05:45
- Pull request#2040ArthurZucker2026-04-27 05:26
- Pull request#2037KimBioInfoStudio2026-04-26 16:42
- Issue comment#1985McPatate2026-04-19 13:21feat: add new faster whitespace split pretok
- Issue comment#2022ArthurZucker2026-04-10 15:38perf: skip alignment tracking in encode_fast normalization
- Issue comment#1942github-actions[bot]2026-04-10 12:47๐จ feat: add role_to_token field for special token metadata
- Issue comment#2010github-actions[bot]2026-04-10 10:49feat: remove string fmt allocation in hot loop in `BpeBuilder::build`
- Issue comment#2012github-actions[bot]2026-04-10 10:48feat: open-addressing merge table with cache-line-local linear probing for BPE
- Pull request#2018ArthurZucker2026-04-10 10:42
Totals cover only the window loaded into ClickHouse and count events, not GitHub's lifetime totals โ 534 stars here means stars gained during the window, not the repo's star count.