Russell Ballestrini
cbed6d9d9a
- Use lxml parser (2-5x faster than html.parser), fallback to html.parser - Combine 20+ noise selectors into single query (1 DOM traversal vs 20) - Add hint parameter to skip content detection for known types - Added lxml to requirements.txt
27 lines
354 B
Text
27 lines
354 B
Text
# Core crawler
|
|
aiohttp>=3.9.0
|
|
aiofiles>=23.0.0
|
|
beautifulsoup4>=4.12.0
|
|
html5lib>=1.1
|
|
miniuri>=0.1.0
|
|
|
|
# Database
|
|
sqlalchemy[asyncio]>=2.0.0
|
|
aiosqlite>=0.19.0
|
|
|
|
# Storage
|
|
filevault>=0.1.0
|
|
|
|
# SERP server
|
|
fastapi>=0.104.0
|
|
uvicorn>=0.24.0
|
|
python-multipart>=0.0.6
|
|
|
|
# Progress bar
|
|
tqdm>=4.66.0
|
|
|
|
# Optional
|
|
pillow>=10.0.0
|
|
html2text>=2024.2.26
|
|
markdown>=3.5.0
|
|
lxml
|