-
https://scrapy.org/ and https://spidermon.readthedocs.io/en/latest/
-
drip crawler https://rapidapi.com/markhorverse-markhorverse-default/api/dripcrawler
-
js browser automation comparison
-
no-code/low-code tools
- https://www.youtube.com/watch?v=QxHE4af5BQE
- https://github.com/confident-ai/deepeval https://docs.confident-ai.com/
- https://github.com/deepchecks/deepchecks https://docs.deepchecks.com/stable/getting-started/welcome.html
- https://github.com/Skyvern-AI/skyvern
- https://github.com/browser-use/browser-use
- https://convergence.ai/proxy_lite/
- https://github.com/unclecode/crawl4ai
- https://github.com/D4Vinci/Scrapling
- https://github.com/kreuzberg-dev/kreuzcrawl
- https://scrapeops.io/python-scrapy-playbook/scrapy-playwright/
- https://www.zenrows.com/blog/scrapy-playwright#install-scrapy-playwright
- https://playwright.dev/python/docs/debug
- rust
- https://www.zenrows.com/blog/rust-web-scraping#extract-html-data
- https://www.scrapingbee.com/blog/web-scraping-rust/k
- https://itehax.com/blog/web-scraping-using-rust
- https://github.com/kxzk/scraping-with-rust
- https://www.scrapingdog.com/blog/web-scraping-with-rust/
- https://www.reddit.com/r/rust/comments/18e5wlf/web_scraping_with_playwright/ https://github.com/mattsse/chromiumoxide
- https://substack.thewebscraping.club/
- AWS lambda (serverless)
alterantive browsers
others
!antibotin discord or self hosted to check- https://www.wappalyzer.com/
- https://scrapingfish.com/webscraping-benchmark
- https://www.zenrows.com/solutions/web-unblocker
- https://substack.thewebscraping.club/p/why-scraper-is-blocked
- ip rotation