Nested looping issue - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html) +--- Thread: Nested looping issue (/thread-13031.html) |
Nested looping issue - smk90 - Sep-24-2018 Hi, I am a Python newbie, I am trying to use Python recursively loop through a list and provide a cumulative value of a metric for each 'match' in a list. My code is nearly there, but I have an issue where I am getting the same (incorrect) value every single time for each of the items in the list. Any tips where this is going wrong? I am pretty sure it is the for data in soup.findAll('div',{'class':'ml1-SoccerStatsBar '}): line but not sure why. #use Selenium & Beautiful Soup from selenium import webdriver import time from bs4 import BeautifulSoup #define URL/driver my_url = "https://www.bet365.com/#/IP/" driver = webdriver.Edge() driver.get(my_url) #allow a sleep of 10 seconds time.sleep(10) #parse the page pSource= driver.page_source soup = BeautifulSoup(pSource, "html.parser") #containers tag - per match containers = soup.findAll("div", {"class": "ipn-TeamStack "}) for container in containers: match = container.find_all('div') texts = [div.text for div in match] #Total Match Shots cumul_match_shots = 0 for data in soup.findAll('div',{'class':'ml1-SoccerStatsBar '}): for result in data.find_all('span'): a = result.text if len(a) > 0: cumul_match_shots += int(a) #print out values print(texts) print(cumul_match_shots) #close the webpage driver.close() RE: Nested looping issue - Larz60+ - Sep-24-2018 that page doesn't have a div with class name of 'ipn-TeamStack' show your error traceback and/or results RE: Nested looping issue - stullis - Sep-24-2018 Shouldn't line 26 iterate over match instead of soup.findAll? Using soup.findAll would draw in data from the whole webpage instead of the excerpt you've identified in the outer loop. RE: Nested looping issue - Larz60+ - Sep-24-2018 what do you expect will happen if findAll doesn't find anything? for data in soup.findAll('div',{'class':'ml1-SoccerStatsBar '}):The loop will never execute! the statement: ('div',{'class':'ml1-SoccerStatsBar '}) means to find all div tags who have a class='ml1-SoccerStatsBar ' (including the space since it's part of the string. As stated before, that web page doesn't have a single entry that satisfies that case. |