Aug-14-2020, 03:36 PM
from selenium import webdriver from selenium.webdriver.firefox.options import Options from selenium.webdriver.support.wait import WebDriverWait from bs4 import BeautifulSoup option = webdriver.FirefoxOptions() option.add_argument('-headless') driver=webdriver.Firefox(options=option) driver.get("https://bors.e24.no/#!/instrument/PCIB.OSE/orderdepth") element = WebDriverWait(driver, 20).until(lambda x: x.find_elements_by_class_name("number")) soup=BeautifulSoup(driver.page_source,'html.parser') data=[price.text for price in soup.find_all('td', {'class':'number'})] print(data) driver.quit()The problem was the time needed to load this dynamic page completely.
That's were WebDriverWait came in handy.
In my case, chromedriver for 85 was useless. Both Firefox and Unix did the job.
Maybe this solution helps someone.