Russell Ballestrini
9e6bbe4b19
- Rename tokenizer.js to token_estimator.js to reflect its purpose - Update import in translation.js - Add development journal documenting NVIDIA 4090 performance metrics
20 lines
No EOL
653 B
ReStructuredText
20 lines
No EOL
653 B
ReStructuredText
====================
|
|
Development Journal
|
|
====================
|
|
|
|
July 2, 2025
|
|
============
|
|
|
|
Token Estimator Implementation
|
|
-------------------------------
|
|
|
|
Built `token_estimator.js` - a simplified BPE-based token counter for translation ETA estimation.
|
|
|
|
**Not compatible with tiktoken** - this is estimation-only, ~85% accuracy. Good enough for progress bars, not for exact tokenization.
|
|
|
|
**NVIDIA 4090 Performance Metrics:**
|
|
- Prompt processing: 3,144 tokens/sec
|
|
- Generation: 121 tokens/sec
|
|
- Translation estimates: 1K tokens ≈ 11 seconds
|
|
|
|
The estimator uses real production metrics from our Hermes machine to provide realistic translation timing. |