- This file contains the specific dependencies required for running the application in Windows Subsystem for Linux (WSL).
- Adding this file ensures that developers working on WSL can easily set up their environment and install all necessary packages.
- This improves consistency across development environments and reduces setup time for new contributors or team members.
- Modified verify_target method to skip updating the target if the NamespaceRequest is already verified and the last verification was within the past day.
- This change reduces unnecessary modifications to the target, improving efficiency and maintaining data integrity.
- Added logging to indicate when target verification is skipped due to verification status.
hopefully this:
* lowers cpu usage, although it can't hurt.
* reduces database churn and locking.
modified: remarkbox/models/namespace_request.py
- Added logic to skip scraping if a NamespaceRequest is already verified and the last scrape was within the past day.
- This change reduces unnecessary scraping, improving efficiency and reducing load on target servers.
- Added logging to indicate when a scrape is skipped due to verification status.
modified: remarkbox/models/namespace_request.py
- Implemented logic to follow HTTP redirects up to a specified maximum depth (max_redirects) in the scrape_target method.
- Added logging to track each redirect step, providing better visibility into the redirect chain.
- Set a limit on the number of redirects to prevent infinite loops in case of circular redirects.
- This change ensures that the UUID is checked on the final destination page, improving the accuracy of the scraping process.
modified: remarkbox/models/namespace_request.py
- Replaced custom URI parsing logic with the miniuri library to simplify and standardize URI handling.
- Updated the _get_domain_from_target method to use miniuri for extracting the hostname from target URLs.
- This change enhances code readability and leverages miniuri's robust URI parsing capabilities.
modified: remarkbox/models/namespace_request.py
- Added a threading lock to the NamespaceRequest class to ensure only one scrape operation is performed at a time for a given request.
- Used non-blocking lock acquisition to allow early exit if a scrape is already in progress, reducing redundant requests.
- Ensured lock release in a finally block to maintain proper lock management.
- This change aims to mitigate potential DoS issues by preventing multiple concurrent scrapes and reducing the risk of rate-limiting by target servers.
modified: remarkbox/models/namespace_request.py
