fix intro generation getting cut off by being more generous with output tokens

- reduce aggressive buffering that was leaving too little room for intro output
- reserve up to 1000 tokens or 30% of model capacity for intro generation
- increase minimum output tokens from 100 to 200
- should fix truncated intros especially on Groq models with 8K completion limits
This commit is contained in:
Russell Ballestrini 2025-07-12 13:27:08 -04:00
parent 6645061372
commit c014adde73

View file

@ -144,8 +144,11 @@ async function* sendMessageWithCustomHistory(messageHistory) {
// Estimate input tokens for intro generation
const inputText = messageHistory.map(msg => msg.content).join('');
const inputTokens = Math.ceil(inputText.length / 4);
const buffer = Math.max(2048, Math.floor(inputTokens * 1.5)); // Use 1.5x input tokens as buffer, minimum 2048
const calculatedAvailable = Math.max(100, modelMaxTokens - inputTokens - buffer);
// For intro generation, be more generous - we want complete intros
// Reserve a reasonable amount for output instead of aggressive buffering
const outputReserve = Math.min(1000, Math.floor(modelMaxTokens * 0.3)); // Reserve up to 1000 tokens or 30% for output
const calculatedAvailable = Math.max(200, modelMaxTokens - inputTokens - outputReserve);
// Cap to model's max completion tokens limit
const maxCompletionTokens = getSelectedModelMaxCompletionTokens();