Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Extract data from a table
#4
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support.wait import WebDriverWait
from bs4 import BeautifulSoup

option = webdriver.FirefoxOptions()
option.add_argument('-headless')
driver=webdriver.Firefox(options=option)

driver.get("https://bors.e24.no/#!/instrument/PCIB.OSE/orderdepth")
element = WebDriverWait(driver, 20).until(lambda x: x.find_elements_by_class_name("number"))
soup=BeautifulSoup(driver.page_source,'html.parser')
data=[price.text for price in soup.find_all('td', {'class':'number'})]
print(data)
driver.quit()
The problem was the time needed to load this dynamic page completely.
That's were WebDriverWait came in handy.
In my case, chromedriver for 85 was useless. Both Firefox and Unix did the job.
Maybe this solution helps someone.
Reply


Messages In This Thread
Extract data from a table - by Bob_M - Aug-05-2020, 12:49 PM
RE: Extract data from a table - by kashcode - Aug-14-2020, 10:07 AM
RE: Extract data from a table - by metulburr - Aug-14-2020, 11:39 AM
RE: Extract data from a table - by Bob_M - Aug-14-2020, 03:36 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Scraping data from table into existing dataframe vincer58 1 2,108 Jan-09-2022, 05:15 PM
Last Post: vincer58
  Extract data from sports betting sites nestor 3 5,784 Mar-30-2021, 04:37 PM
Last Post: Larz60+
  Inserting data from a table to another (in same db) firebird 5 2,584 Oct-05-2020, 06:04 AM
Last Post: buran
  Scraping a dynamic data-table in python through AJAX request filozofo 1 3,977 Aug-14-2020, 10:13 AM
Last Post: kashcode
  Extract data with Selenium and BeautifulSoup nestor 3 4,059 Jun-06-2020, 01:34 AM
Last Post: Larz60+
  Extract json-ld schema markup data and store in MongoDB Nuwan16 0 2,535 Apr-05-2020, 04:06 PM
Last Post: Nuwan16
  Extract data from a webpage cycloneseb 5 2,996 Apr-04-2020, 10:17 AM
Last Post: alekson
  Cannot Extract data through charts online AgileAVS 0 1,899 Feb-01-2020, 01:47 PM
Last Post: AgileAVS
  Cannot extract data from the next pages nazmulfinance 4 2,891 Nov-11-2019, 08:15 PM
Last Post: nazmulfinance
  Table data with BeatifulSoup gerry84 11 7,500 Oct-23-2019, 10:09 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020