87 Views

Python スクレイピングのやりかた

scraping のルールについては、
スクレイピング対象のサイトの
robots.txtを読むこと。

サンプル
https://4knn.tv/robots.txt

クローリング(Crawling)ロボットにおけるクロールに対してのルール表記。

Allow: は、許可
Disallow:は、 非許可

https://www.bloomberg.com/robots.txt
# Bot rules:
# 1. A bot may not injure a human being or, through inaction, allow a human being to come to harm.
# 2. A bot must obey orders given it by human beings except where such orders would conflict with the First Law.
# 3. A bot must protect its own existence as long as such protection does not conflict with the First or Second Law.
# If you can read this then you should apply here https://www.bloomberg.com/careers/

参考図書


https://amzn.to/2JaTClm