ETL-style parallel markdown backfill with real-time progress
- ProcessPoolExecutor: 1 process per CPU core (bypasses GIL) - ThreadPoolExecutor: 6 threads per process for I/O throughput - Thread-local SQLite connections with WAL mode - Manager().Value() for cross-process progress counter - Real-time tqdm updates polling shared counter every 50ms - ~19 pages/sec on 4-core system (4887 pages in 2 min)
This commit is contained in:
parent
293ed64a3b
commit
1346590d39
5 changed files with 840 additions and 864 deletions
|
|
@ -426,8 +426,12 @@ class SmartMarkdownConverter:
|
|||
src = self._resolve_url(child.get('src', ''))
|
||||
alt = child.get('alt', '')
|
||||
if src:
|
||||
# Images should be block-level, not inline
|
||||
parts.append(f"\n\n\n\n")
|
||||
# Emoji images (alt like :smile:) stay inline
|
||||
if alt.startswith(':') and alt.endswith(':'):
|
||||
parts.append(f"")
|
||||
else:
|
||||
# Regular images are block-level
|
||||
parts.append(f"\n\n\n\n")
|
||||
|
||||
elif tag == 'br':
|
||||
parts.append('\n')
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue