Python Forum
Requests_HTML not getting all data on Amazon
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Requests_HTML not getting all data on Amazon
#2
I was able to fix this with exception handling, not sure why I was getting the Nonetypes in the first place, but I did get past it.
 for review in reviews:
            try:
                title = review.find('a[data-hook=review-title]', first=True).text
            except AttributeError:
                title = None
            try:
                rating = review.find('i[data-hook=review-star-rating] span', first=True).text
            except AttributeError:
                rating = None
            try:
                body = review.find('span[data-hook=review-body] span', first=True).text.replace('\n','').strip()  # exchange newlines with a space
            except AttributeError:
                body = None

            data = {                                             # dictionary formatting the data with title hooks for the analyzer to link to
                "title": title,
                'rating': rating,
                'body': body
            }

            total.append(data)
               
        return total
making this change got me past it.

I hope this helps someone else with the same issue.
Reply


Messages In This Thread
RE: Requests_HTML not getting all data on Amazon - by aaander - Nov-19-2022, 02:09 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Getting a URL from Amazon using requests-html, or beautifulsoup aaander 1 1,773 Nov-06-2022, 10:59 PM
Last Post: snippsat
  Can't open Amazon page Pavel_47 3 3,334 Oct-21-2020, 09:13 AM
Last Post: Aspire2Inspire
  New in Python Amazon Scraping brian1425 1 2,106 Jul-10-2020, 01:00 PM
Last Post: snippsat
  error installing requests_html davidm 4 4,022 Mar-06-2020, 03:23 PM
Last Post: snippsat
  Amazon AWS - how to install the library chatterbot wpaiva 9 4,021 Feb-01-2020, 08:18 AM
Last Post: brighteningeyes
  Execute search query on Amazon website Pavel_47 7 3,603 Nov-07-2019, 10:43 AM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020