russell@unturf.com
6b975a3b9e
DuckDB (C++ query engine): - duckdb-0001: Binder::AddCorrelatedColumn vector dedup O(C²) MEDIUM 200x - duckdb-0002: HasCorrelatedExpressions vector scan O(N×M) MEDIUM 100x - duckdb-0003: ComputeOverlappingBindings vector scan O(N×H) MEDIUM 219x - duckdb-0004: Deliminator group-join binding check O(G×J) MEDIUM 125x Apache Arrow (C++ analytics): - arrow-0001: AsofJoin IsTimeOrKeyColumn vector scan O(F×K) MEDIUM 114x - arrow-0002: Scanner AddFieldsNeededForFilter vector dedup O(F×C) MEDIUM 250x All 6/6 unit tests PASS.
29 lines
1.6 KiB
Diff
29 lines
1.6 KiB
Diff
# UNDF: (leave blank)
|
||
# Apache Arrow CWE-407: ScanV2Options::AddFieldsNeededForFilter O(F×C)
|
||
# File: cpp/src/arrow/dataset/scanner.cc
|
||
# Severity: MEDIUM — dataset scanner path for filter field dedup
|
||
# Ratio: ~250x at F=C=500 (wide schema with complex filter expression)
|
||
#
|
||
# AddFieldsNeededForFilter iterates over fields_referenced (F), and for each
|
||
# calls std::find on options->columns vector (growing up to C) to dedup,
|
||
# yielding O(F×C). With wide schemas and complex filters, both F and C grow.
|
||
# Fix: build an unordered_set of existing columns for O(1) lookup.
|
||
--- a/cpp/src/arrow/dataset/scanner.cc
|
||
+++ b/cpp/src/arrow/dataset/scanner.cc
|
||
@@ -82,10 +82,13 @@
|
||
Status ScanV2Options::AddFieldsNeededForFilter(ScanV2Options* options) {
|
||
std::vector<FieldRef> fields_referenced = FieldsInExpression(options->filter);
|
||
+ // Build O(1) lookup set for existing columns; std::find was O(C) per field → O(F×C) total
|
||
+ std::unordered_set<FieldPath, FieldPath::Hash> existing_columns(
|
||
+ options->columns.begin(), options->columns.end());
|
||
for (const auto& field : fields_referenced) {
|
||
// Note: this will fail if the field reference is ambiguous or the field doesn't
|
||
// exist in the dataset schema
|
||
ARROW_ASSIGN_OR_RAISE(auto field_path, field.FindOne(*options->dataset->schema()));
|
||
- if (std::find(options->columns.begin(), options->columns.end(), field_path) ==
|
||
- options->columns.end()) {
|
||
+ if (existing_columns.find(field_path) == existing_columns.end()) {
|
||
options->columns.push_back(std::move(field_path));
|
||
+ existing_columns.insert(options->columns.back());
|
||
}
|
||
}
|