Unsupervised text tokenizer for Neural Network-based text generation.
active 2025-03-15 → 2026-07-30 (UTC)
Activity over time
Daily event counts in the loaded window
Line chart, 503 days from 2025-03-15 to 2026-07-30. Pushes: 140 total, peak 7 in a day. Pull requests: 86 total, peak 6 in a day. Issues: 69 total, peak 8 in a day. Comments: 187 total, peak 18 in a day. Stars: 529 total, peak 6 in a day.
- Pushes
- Pull requests
- Issues
- Comments
- Stars
Stars, PRs, issues and forks are under-captured in the later part of this window. GH Archive progressively stopped capturing non-push events during 2026 — −95% or worse by the end of the window. Every series here except Pushes fades for that reason, so a decline above reflects the archive, not this repository. Pushes stay reliable throughout, so read them, and the contributor counts derived from them, as the real signal. Data health has the measurements.
Top contributors
Pushes, PRs, issues, reviews and comments — stars and forks excluded, so this is contribution rather than popularity
| Contributor | Contributions | Pushes | PRs | Comments |
|---|---|---|---|---|
| taku910 | 258 | 135 | 13 | 60 |
| dependabot[bot] | 60 | 5 | 51 | 4 |
| crusaderky | 42 | 0 | 5 | 30 |
| jeffhataws | 9 | 0 | 0 | 6 |
| google-cla[bot] | 6 | 0 | 0 | 6 |
| reneleonhardt | 5 | 0 | 0 | 5 |
| tkng | 4 | 0 | 1 | 3 |
| dbowring | 4 | 0 | 1 | 3 |
| Ahajha | 4 | 0 | 0 | 4 |
| jmsunseri | 4 | 0 | 2 | 2 |
| kkew3 | 4 | 0 | 3 | 1 |
| lucianchauvin | 4 | 0 | 0 | 4 |
| rgommers | 4 | 0 | 0 | 3 |
| BLKSerene | 4 | 0 | 1 | 3 |
| teamclouday | 3 | 0 | 2 | 1 |
| eix128 | 3 | 0 | 0 | 2 |
| wagenrace | 3 | 0 | 1 | 2 |
| anishesg | 3 | 0 | 1 | 2 |
| 010gvr | 2 | 0 | 0 | 1 |
| oss-fuzz-robot | 2 | 0 | 0 | 0 |
Recent activity
Latest issues, pull requests and releases
- Pull request#1294dependabot[bot]2026-07-25 03:12
- Pull request#1276kkew32026-06-20 10:48
- Issue#1030taku9102026-06-11 06:24No typings in Python package
- Issue#1264taku9102026-06-06 06:34Use Abseil in the public API.
- Issue comment#1245taku9102026-05-31 14:07Deprecate vocab management methods.
- Issue comment#1255Deepak-png9812026-05-29 06:49Fix user_defined_symbols encoding for WORD model
- Issue comment#1252taku9102026-05-25 05:15Question about speed
- Issue#1252jovan20092026-05-23 17:06Question about speed
- Issue#1246vinovo2026-05-19 23:38CMake: cl.exe-only `/D` and `/wd` flags injected for Clang on Windows in the bundled-protobuf path
- Pull request#1242taku9102026-05-17 01:08
- Issue#571taku9102026-05-11 03:08Sentencepiece with pre-defined vocabulary
- Issue comment#571taku9102026-05-11 03:08Sentencepiece with pre-defined vocabulary
- Issue#881taku9102026-05-10 10:28Duplicate tokens in BPE vocabulary
- Issue comment#1024taku9102026-05-10 10:04Wrong calculation of max_score in unigram_model.cc
- Pull request#1231Alearner122026-05-08 17:46
- Issue#1047taku9102026-05-08 08:17With unigram algorithm, constant piece at end of each sentences does not become a token
- Issue comment#1047taku9102026-05-08 08:17With unigram algorithm, constant piece at end of each sentences does not become a token
- Issue comment#1149taku9102026-05-07 01:18Use official abseil-cpp
- Issue#1227taku9102026-05-04 08:48OSS-Fuzz issue 509386852 [Crashes in SetSentencepieces]
- Issue#984taku9102026-05-03 00:38Allow whitespace-only pieces
- Issue#984taku9102026-05-03 00:38Allow whitespace-only pieces
- Issue#1156taku9102026-05-02 08:54Python package does not have license
- Issue#1149taku9102026-05-02 06:47Use official abseil-cpp
- Issue#1220taku9102026-05-02 06:46OSS-Fuzz issue 507172876 [ IsUnusedInlined is called with invalid id.]
- Issue comment#1220taku9102026-05-02 06:46OSS-Fuzz issue 507172876 [ IsUnusedInlined is called with invalid id.]
Totals cover only the window loaded into ClickHouse and count events, not GitHub's lifetime totals — 529 stars here means stars gained during the window, not the repo's star count.