Beautiful Soup (HTML parser)
Python HTML/XML parser
- 2 min read
- 2 chapters
Related on Lably
Through Python (programming language), HTML and Control flow
History
Beautiful Soup was started in 2004 by Leonard Richardson. It takes its name from the poem Beautiful Soup from Alice's Adventures in Wonderland and is a reference to the term "tag soup" meaning poorly-structured HTML code. Richardson continues to contribute to the project, which is additionally supported by paid open-source maintainers from the company Tidelift.
Versions
Beautiful Soup 3 was the official release line of Beautiful Soup from May 2006 to March 2012. The current release is Beautiful Soup 4.x.
In 2021, Python 2.7 support was retired and the release 4.9.3 was the last to support Python 2.7.
Sign in to keep reading
One more chapter is free with a Lably account.
Usage
Beautiful Soup represents parsed data as a tree which can be searched and iterated over with ordinary Python loops.
Code example
The example below uses the Python standard library's urllib to load Wikipedia's main page, then uses Beautiful Soup to parse the document and search for all links within.
<span id="mwUA"></span><span class="ch" id="mwUQ">#!/usr/bin/env python3</span> <span class="c1" id="mwUg"># Anchor extraction from HTML document</span> <span class="kn" id="mwUw">from</span><span class="w" id="mwVA"> </span><span class="nn" id="mwVQ">bs4</span><span class="w" id="mwVg"> </span><span class="kn" id="mwVw">import</span> <span class="n" id="mwWA">BeautifulSoup</span> <span class="kn" id="mwWQ">from</span><span class="w" id="mwWg"> </span><span class="nn" id="mwWw">urllib.request</span><span class="w" id="mwXA"> </span><span class="kn" id="mwXQ">import</span> <span class="n" id="mwXg">urlopen</span> <span class="k" id="mwXw">with</span> <span class="n" id="mwYA">urlopen</span><span class="p" id="mwYQ">(</span><span class="s2" id="mwYg">"https://en.wikipedia.org/wiki/Main_Page"</span><span class="p" id="mwYw">)</span> <span class="k" id="mwZA">as</span> <span class="n" id="mwZQ">response</span><span class="p" id="mwZg">:</span> <span class="n" id="mwZw">soup</span> <span class="o" id="mwaA">=</span> <span class="n" id="mwaQ">BeautifulSoup</span><span class="p" id="mwag">(</span><span class="n" id="mwaw">response</span><span class="p" id="mwbA">,</span> <span class="s2" id="mwbQ">"html.parser"</span><span class="p" id="mwbg">)</span> <span class="k" id="mwbw">for</span> <span class="n" id="mwcA">anchor</span> <span class="ow" id="mwcQ">in</span> <span class="n" id="mwcg">soup</span><span class="o" id="mwcw">.</span><span class="n" id="mwdA">find_all</span><span class="p" id="mwdQ">(</span><span class="s2" id="mwdg">"a"</span><span class="p" id="mwdw">):</span> <span class="nb" id="mweA">print</span><span class="p" id="mweQ">(</span><span class="n" id="mweg">anchor</span><span class="o" id="mwew">.</span><span class="n" id="mwfA">get</span><span class="p" id="mwfQ">(</span><span class="s2" id="mwfg">"href"</span><span class="p" id="mwfw">,</span> <span class="s2" id="mwgA">"/"</span><span class="p" id="mwgQ">))</span>
Another example is using the Python requests library to get divs on a URL.
<span id="mwiQ"></span><span class="kn" id="mwig">import</span><span class="w" id="mwiw"> </span><span class="nn" id="mwjA">requests</span> <span class="kn" id="mwjQ">from</span><span class="w" id="mwjg"> </span><span class="nn" id="mwjw">bs4</span><span class="w" id="mwkA"> </span><span class="kn" id="mwkQ">import</span> <span class="n" id="mwkg">BeautifulSoup</span> <span class="n" id="mwkw">url</span> <span class="o" id="mwlA">=</span> <span class="s2" id="mwlQ">"https://wikipedia.org"</span> <span class="n" id="mwlg">response</span> <span class="o" id="mwlw">=</span> <span class="n" id="mwmA">requests</span><span class="o" id="mwmQ">.</span><span class="n" id="mwmg">get</span><span class="p" id="mwmw">(</span><span class="n" id="mwnA">url</span><span class="p" id="mwnQ">)</span> <span class="n" id="mwng">soup</span> <span class="o" id="mwnw">=</span> <span class="n" id="mwoA">BeautifulSoup</span><span class="p" id="mwoQ">(</span><span class="n" id="mwog">response</span><span class="o" id="mwow">.</span><span class="n" id="mwpA">text</span><span class="p" id="mwpQ">,</span> <span class="s2" id="mwpg">"html.parser"</span><span class="p" id="mwpw">)</span> <span class="n" id="mwqA">headings</span> <span class="o" id="mwqQ">=</span> <span class="n" id="mwqg">soup</span><span class="o" id="mwqw">.</span><span class="n" id="mwrA">find_all</span><span class="p" id="mwrQ">(</span><span class="s2" id="mwrg">"div"</span><span class="p" id="mwrw">)</span> <span class="k" id="mwsA">for</span> <span class="n" id="mwsQ">heading</span> <span class="ow" id="mwsg">in</span> <span class="n" id="mwsw">headings</span><span class="p" id="mwtA">:</span> <span class="nb" id="mwtQ">print</span><span class="p" id="mwtg">(</span><span class="n" id="mwtw">heading</span><span class="o" id="mwuA">.</span><span class="n" id="mwuQ">text</span><span class="o" id="mwug">.</span><span class="n" id="mwuw">strip</span><span class="p" id="mwvA">())</span>