The Dormouse's story
Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well.
...
""" ``` Running the "three sisters" document through Beautiful Soup gives us a `BeautifulSoup` object, which represents the document as a nested data structure: ```python from bs4 import BeautifulSoup soup = BeautifulSoup(html_doc, 'html.parser') print(soup.prettify()) # # ## # The Dormouse's story # #
## Once upon a time there were three little sisters; and their names were # # Elsie # # , # # Lacie # # and # # Tillie # # ; and they lived at the bottom of a well. #
## ... #
# # ``` Here are some simple ways to navigate that data structure: ```python soup.title #The Dormouse's story
soup.p['class'] # u'title' soup.a # Elsie soup.find_all('a') # [Elsie, # Lacie, # Tillie] soup.find(id="link3") # Tillie ``` One common task is extracting all the URLs found within a page's `` tags: ```python for link in soup.find_all('a'): print(link.get('href')) # http://example.com/elsie # http://example.com/lacie # http://example.com/tillie ``` Another common task is extracting all the text from a page: ```python print(soup.get_text()) # The Dormouse's story # # The Dormouse's story # # Once upon a time there were three little sisters; and their names were # Elsie, # Lacie and # Tillie; # and they lived at the bottom of a well. # # ... ``` ## Example The following program scrapes the 247ctf.com scoreboard: ```python import requests from bs4 import BeautifulSoup page = requests.get('https://247ctf.com/scoreboard') soup = BeautifulSoup(page.content, 'html.parser') table = soup.find('table') table_body = table.find('tbody') rows = table_body.find_all('tr') for row in rows: print('------------------------------------------------------') cols = [x.text.strip() for x in row.find_all('td')] print(f"{cols[2]} is in {cols[0]} place with {cols[4]}.") ```