scraping のルールについては、
スクレイピング対象のサイトの
robots.txtを読むこと。
サンプル
http://4knn.tv/robots.txt
クローリング（Crawling）ロボットにおけるクロールに対してのルール表記。
Allow: は、許可
Disallow:は、 非許可
https://www.bloomberg.com/robots.txt
# Bot rules:
# 1. A bot may not injure a human being or, through inaction, allow a human being to come to harm.
# 2. A bot must obey orders given it by human beings except where such orders would conflict with the First Law.
# 3. A bot must protect its own existence as long as such protection does not conflict with the First or Second Law.
# If you can read this then you should apply here https://www.bloomberg.com/careers/
参考図書
https://amzn.to/2JaTClm
