gigatoken

Everything FervorAI has published that touches gigatoken — 1 piece, newest first.

• Gigatoken Tokenizes Text at Gigabytes per Second. The Real Question Is Whether You Tokenize Gigabytes.

  Gigatoken delivers a real ~1000x-over-HuggingFace (68-681x over tiktoken