ETL-style parallel markdown backfill with real-time progress

- ProcessPoolExecutor: 1 process per CPU core (bypasses GIL)
- ThreadPoolExecutor: 6 threads per process for I/O throughput
- Thread-local SQLite connections with WAL mode
- Manager().Value() for cross-process progress counter
- Real-time tqdm updates polling shared counter every 50ms
- ~19 pages/sec on 4-core system (4887 pages in 2 min)
This commit is contained in:
Russell Ballestrini 2025-12-30 14:22:29 -05:00
parent 293ed64a3b
commit 1346590d39
5 changed files with 840 additions and 864 deletions

View file

@ -426,8 +426,12 @@ class SmartMarkdownConverter:
src = self._resolve_url(child.get('src', ''))
alt = child.get('alt', '')
if src:
# Images should be block-level, not inline
parts.append(f"\n\n![{alt}]({src})\n\n")
# Emoji images (alt like :smile:) stay inline
if alt.startswith(':') and alt.endswith(':'):
parts.append(f"![{alt}]({src})")
else:
# Regular images are block-level
parts.append(f"\n\n![{alt}]({src})\n\n")
elif tag == 'br':
parts.append('\n')