russell@unturf.com
a89cc53fed
Rescan both targets against all 5 MOADs (2026-04-03). New defect: - transformers-0004: wav2vec2, wav2vec2_phoneme, esm tokenizers call self.all_special_ids/@property inside per-token decode loops, rebuilding list every iteration. O(T) -> O(1) fix: cache set before loop. wav2vec2_phoneme also has type mismatch (str vs list[int]), making the check always False, leaking special tokens. 9/9 unit tests PASS. Existing defects confirmed still present (not re-filed): - transformers-0001/0002/0003: unchanged from 2026-03-31 scan. - vllm-0001/0002: unchanged from 2026-03-31 scan. MOAD-0002/0003/0004/0005: CLEAN on both targets (see SCAN-2026-04-03.md). SCAN-TODO.md: marked transformers and vllm as complete with full summary. Also includes UNDF stamps on jicofo-0001, jicofo-0002, langchain-0002 patches from prior generate_undf.py run.
29 lines
1.4 KiB
Diff
29 lines
1.4 KiB
Diff
# UNDF: UNDF-2026-000001220
|
|
--- a/libs/langchain/langchain_classic/retrievers/multi_query.py
|
|
+++ b/libs/langchain/langchain_classic/retrievers/multi_query.py
|
|
@@ -44,7 +44,22 @@ DEFAULT_QUERY_PROMPT = PromptTemplate(
|
|
|
|
|
|
def _unique_documents(documents: Sequence[Document]) -> list[Document]:
|
|
- return [doc for i, doc in enumerate(documents) if doc not in documents[:i]]
|
|
+ """Return documents with duplicates removed, preserving first-seen order.
|
|
+
|
|
+ Original implementation was O(D^2): for each doc it created a list slice
|
|
+ documents[:i] (O(D) allocation) and scanned it with `not in` (O(D) scan),
|
|
+ giving O(D^2) time and memory for D = Q * k total retrieved documents.
|
|
+
|
|
+ Fixed implementation is O(D) using a hashable proxy key built from
|
|
+ (id, page_content, metadata items). Documents with unhashable metadata
|
|
+ values are handled by stringifying them as a fallback.
|
|
+ """
|
|
+ seen: set[tuple] = set()
|
|
+ result: list[Document] = []
|
|
+ for doc in documents:
|
|
+ # Build a hashable proxy key from (id, page_content, metadata items).
|
|
+ # metadata values may be unhashable (e.g. list), so stringify on fallback.
|
|
+ meta_key: tuple = tuple(sorted((k, str(v)) for k, v in doc.metadata.items()))
|
|
+ key = (doc.id, doc.page_content, meta_key)
|
|
+ if key not in seen:
|
|
+ seen.add(key)
|
|
+ result.append(doc)
|
|
+ return result
|