ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts. We released a collection of ModuleFormer-based Language Models (MoLM) ranging in scale from 4 billion to 8 billion parameters.
active 2023-11-22 → 2024-03-27 (UTC)
Activity over time
Daily event counts in the loaded window
Line chart, 127 days from 2023-11-22 to 2024-03-27. Pushes: 5 total, peak 2 in a day. Pull requests: 0 total, peak 0 in a day. Issues: 0 total, peak 0 in a day. Comments: 0 total, peak 0 in a day. Stars: 0 total, peak 0 in a day.
- Pushes
- Pull requests
- Issues
- Comments
- Stars
Top contributors
Pushes, PRs, issues, reviews and comments — stars and forks excluded, so this is contribution rather than popularity
| Contributor | Contributions | Pushes | PRs | Comments |
|---|---|---|---|---|
| LuciferianInk | 5 | 5 | 0 | 0 |
Recent activity
Latest issues, pull requests and releases
No issue or PR events — this repo's activity is pushes only.
Totals cover only the window loaded into ClickHouse and count events, not GitHub's lifetime totals — 0 stars here means stars gained during the window, not the repo's star count.