121 lines
No EOL
4.4 KiB
ReStructuredText
121 lines
No EOL
4.4 KiB
ReStructuredText
====================
|
|
Development Journal
|
|
====================
|
|
|
|
July 2, 2025
|
|
============
|
|
|
|
Token Estimator Implementation
|
|
-------------------------------
|
|
|
|
Built `token_estimator.js` - a simplified BPE-based token counter for translation ETA estimation.
|
|
|
|
**Not compatible with tiktoken** - this is estimation-only, ~85% accuracy. Good enough for progress bars, not for exact tokenization.
|
|
|
|
**NVIDIA 4090 Performance Metrics:**
|
|
- Prompt processing: 3,144 tokens/sec
|
|
- Generation: 121 tokens/sec
|
|
- Translation estimates: 1K tokens ≈ 11 seconds
|
|
|
|
The estimator uses real production metrics from our Hermes machine to provide realistic translation timing.
|
|
|
|
July 3, 2025
|
|
============
|
|
|
|
Complete Translation Infrastructure Overhaul
|
|
--------------------------------------------
|
|
|
|
**Modular Language System Implementation**
|
|
|
|
Restructured the entire translation system to use individual language files instead of one monolithic object. Each of the 19 supported languages now has its own dedicated JS file in `/src/languages/`.
|
|
|
|
**Language Coverage:**
|
|
- 🇺🇸 English (en.js) - 46 translation keys
|
|
- 🇨🇳 Chinese Simplified (zh.js)
|
|
- 🇹🇼 Chinese Traditional (zh-tw.js)
|
|
- 🇮🇳 Hindi (hi.js)
|
|
- 🇪🇸 Spanish (es.js)
|
|
- 🇫🇷 French (fr.js)
|
|
- 🇸🇦 Arabic (ar.js)
|
|
- 🇧🇩 Bengali (bn.js)
|
|
- 🇷🇺 Russian (ru.js)
|
|
- 🇧🇷 Portuguese (pt.js)
|
|
- 🇵🇰 Urdu (ur.js)
|
|
- 🇮🇩 Indonesian (id.js)
|
|
- 🇩🇪 German (de.js)
|
|
- 🇯🇵 Japanese (ja.js)
|
|
- 🇰🇪 Swahili (sw.js)
|
|
- 🇮🇳 Marathi (mr.js)
|
|
- 🇮🇳 Telugu (te.js)
|
|
- 🇹🇷 Turkish (tr.js)
|
|
- 🇰🇷 Korean (ko.js)
|
|
|
|
**Dynamic UI Refresh System**
|
|
|
|
Implemented `refreshUILanguage()` function that updates all UI text without requiring browser refresh:
|
|
|
|
- **Declarative translations**: Elements with `data-i18n` attributes automatically update
|
|
- **Parameter support**: `data-i18n-params` for dynamic text with variables like `{lang}` and `{error}`
|
|
- **Smart element detection**: Handles inputs, buttons, placeholders, and modal titles
|
|
- **Event system**: Dispatches `languageChanged` custom events for component integration
|
|
- **Backward compatibility**: Maintains `window.refreshUILanguage` for existing code
|
|
|
|
**Translation Verification System**
|
|
|
|
Created `verify-translations.js` - a Node.js script that ensures translation consistency:
|
|
|
|
- **Key extraction**: Parses JS files to extract translation object keys
|
|
- **Completeness verification**: Compares all languages against English reference
|
|
- **Missing/extra key detection**: Reports inconsistencies with colored console output
|
|
- **Exit codes**: Returns 0 for success, 1 for issues (CI/CD friendly)
|
|
|
|
**Adding New Languages - Workflow:**
|
|
|
|
1. **Before adding**: Run verification to ensure current state is clean
|
|
```bash
|
|
cd /home/fox/git/ai.unturf.com/src/languages
|
|
node verify-translations.js
|
|
```
|
|
Should show: "🎉 All languages have complete translations!"
|
|
|
|
2. **Create new language file**: Copy structure from `en.js`
|
|
```bash
|
|
cp en.js new-lang.js # Replace 'new-lang' with actual language code
|
|
```
|
|
|
|
3. **Translate all values**: Keep the same keys, translate only the string values
|
|
|
|
4. **Update index.js**: Add import and export for the new language
|
|
```javascript
|
|
import { newLang } from "./new-lang.js";
|
|
// Add to UI_TRANSLATIONS object
|
|
```
|
|
|
|
5. **Verify completeness**: Run script again to check for issues
|
|
```bash
|
|
node verify-translations.js
|
|
```
|
|
|
|
6. **Fix any issues**: Script will show missing/extra keys in red
|
|
- Missing keys: Add them with proper translations
|
|
- Extra keys: Remove them or add to English reference if needed
|
|
|
|
7. **Final verification**: Run until you see the success message again
|
|
|
|
**File Structure:**
|
|
```
|
|
src/languages/
|
|
├── index.js # Aggregates all translations
|
|
├── en.js # English reference (46 keys)
|
|
├── [18 other languages] # Complete translations
|
|
└── verify-translations.js # Sanity check script
|
|
```
|
|
|
|
**Benefits:**
|
|
- **Maintainability**: Each language isolated in its own file
|
|
- **Consistency**: 100% key coverage verified across all 19 languages
|
|
- **Real-time UX**: Language switching without page refresh
|
|
- **Developer experience**: Automated verification prevents translation drift
|
|
- **Scalability**: Easy to add new languages following the established pattern
|
|
|
|
This system supports the full user journey from language detection to dynamic interface updates, ensuring a seamless multilingual experience for all Hermes AI users. |