The World of Python Programming

Complete Guide to Python urllib: GET/POST & Web Scraping

APIs & External Libraries
目次

1. Introduction

The Python standard library urllib is a powerful tool that helps when interacting with external data over HTTP. For example, it can be used to fetch web page data or send requests to APIs to retrieve information. Using this library expands the possibilities for web application development and enables efficient data handling.

The Need for urllib and Comparison with Other Libraries

urllib comes bundled with Python, so no additional installation is required, making it convenient for beginners to start using immediately. The requests library, which offers similar functionality, is also popular, but requests is an external library and must be installed. As a standard library, urllib is well suited for learning the basics of HTTP communication in Python and is recommended as a first step to understand how web requests work.

Overview of this article

This article covers practical uses—from sending basic GET/POST requests with urllib and parsing URLs to web scraping and API integration. Each step is explained sequentially so even beginners can follow along; if you want to work with web data in Python, this guide will be a useful reference.

2. Basic Usage of urllib

urllib provides a wide range of features including basic sending of web requests, response handling, and URL parsing. In this section, we explain the basic methods for sending GET and POST requests using the urllib.request module.

How to Send GET Requests

GET requests are mainly used to retrieve information from web pages. The code below shows an example of using the urllib.request.urlopen function to get a page’s HTML data from a URL.
import urllib.request

url = 'https://example.com'
response = urllib.request.urlopen(url)
html = response.read().decode('utf-8')
print(html)
The code above retrieves the HTML content of the specified URL and outputs it as text. GET requests do not require sending data and are suitable when you simply want to obtain information.

How to Send POST Requests

POST requests are used to send data to a server. For example, they are used when you need to make changes on the server side, such as sending data to an API or submitting form information.
import urllib.request
import urllib.parse

url = 'https://example.com/api'
data = {'key1': 'value1', 'key2': 'value2'}
data = urllib.parse.urlencode(data).encode('utf-8')
request = urllib.request.Request(url, data=data)
response = urllib.request.urlopen(request)
result = response.read().decode('utf-8')
print(result)
In this example, a dictionary of data is URL-encoded and sent in a POST request as bytes. The response returned by the server is read and the result is displayed.

Error Handling

With web requests, the server may not respond or may return errors. Therefore, urllib provides the HTTPError and URLError exceptions for error handling.
import urllib.request
import urllib.error

url = 'https://example.com/api'

try:
    response = urllib.request.urlopen(url)
    html = response.read().decode('utf-8')
    print(html)
except urllib.error.HTTPError as e:
    print('An HTTP error occurred:', e.code)
except urllib.error.URLError as e:
    print('A URL error occurred:', e.reason)
This code handles exceptions such as HTTP status code errors and connection errors, allowing appropriate error messages to be displayed when unexpected errors occur during request processing.

3. Parsing and Manipulating URLs

urllib.parse By using this module, you can easily parse URLs and manipulate query parameters. This section explains how to parse URLs and extract their components, and covers URL encoding and decoding.

URL Parsing

urlparse() By using this function, you can retrieve the individual components of a URL (scheme, host, path, etc.).
from urllib.parse import urlparse

url = 'https://example.com/path/to/page?name=python&lang=ja'
parsed_url = urlparse(url)

print('Scheme:', parsed_url.scheme)
print('Host:', parsed_url.netloc)
print('Path:', parsed_url.path)
print('Query:', parsed_url.query)
This code parses a URL, extracts each component, and displays them.

4. Practical Use Cases

Here, as practical examples using urllib, we explain how to perform web scraping and integrate with APIs. Through these examples, learn advanced uses of urllib and apply them in real projects.

Basics of Web Scraping

Web scraping is the technique of automatically retrieving information from websites to collect data. Here, we show how to fetch web page content with urllib and parse the HTML using the BeautifulSoup library.
import urllib.request
from bs4 import BeautifulSoup

# Send a GET request to the specified URL
url = 'https://example.com'
response = urllib.request.urlopen(url)
html = response.read().decode('utf-8')

# Parse the HTML using BeautifulSoup
soup = BeautifulSoup(html, 'html.parser')

# Extract specific elements
title = soup.title.string
print('Page title:', title)

# Get other elements (e.g., all links)
for link in soup.find_all('a'):
    print(link.get('href'))
In this example, we first retrieve the HTML of the URL specified by urllib and parse it with BeautifulSoup. You can extract specific elements like page titles and links, allowing you to efficiently collect the information you need through scraping.
Note: When scraping, check the website’s terms of service and ensure you do so using permitted methods.

How to Integrate with APIs

An API (Application Programming Interface) is a mechanism for applications to exchange data. Many web services offer RESTful APIs, and you can access them and retrieve data using urllib. Here we show how to make API requests with urllib and obtain and parse response data in JSON format.
import urllib.request
import json

# Specify the API URL
url = 'https://api.example.com/data'

# Send the API request
response = urllib.request.urlopen(url)

# Read the response in JSON format
data = json.loads(response.read().decode('utf-8'))
print('Retrieved data:', data)
In this example, the data retrieved from the API is parsed as JSON. Because JSON consists of key–value pairs, the retrieved data can be treated like a dictionary, making it easy to work with.

5. Considerations and Best Practices

When making web requests or API calls with urllib, there are several important considerations and best practices. Keep the following points in mind to build a reliable application.

Setting Timeouts

If a server’s response is delayed, your program may end up waiting for a long time. To prevent this, it’s common to set a timeout for requests. By setting a timeout, the request will raise an error if there’s no response within a specified time, allowing you to proceed to the next operation.
import urllib.request
from urllib.error import URLError, HTTPError

url = 'https://example.com'

try:
    # Set the timeout to 10 seconds
    response = urllib.request.urlopen(url, timeout=10)
    html = response.read().decode('utf-8')
    print(html)
except HTTPError as e:
    print('HTTP error occurred:', e.code)
except URLError as e:
    print('URL error occurred:', e.reason)
except Exception as e:
    print('An unexpected error occurred:', e)
In this example, the request will automatically time out if there is no response within 10 seconds. Timeouts are an important setting for improving an application’s reliability.

Implementing Retry Logic

Network communication isn’t always stable. Therefore, it’s recommended to introduce a retry mechanism that retries requests when they fail. Retries are especially useful for handling temporary network outages or errors caused by short-lived server overload.
import urllib.request
from urllib.error import URLError
import time

url = 'https://example.com'
max_retries = 3  # Set the number of retries
retry_delay = 5  # Retry interval (seconds)

for attempt in range(max_retries):
    try:
        response = urllib.request.urlopen(url)
        html = response.read().decode('utf-8')
        print(html)
        break  # Exit the loop on success
    except URLError as e:
        print(f'Request failed (attempt {attempt + 1}/{max_retries}): {e.reason}')
        if attempt < max_retries - 1:
            time.sleep(retry_delay)  # Wait before retrying
        else:
            print('Reached the maximum number of retries')
In this code, it retries up to three times, waiting 5 seconds between each retry. Implementing retries makes it easier to handle temporary network issues.

6. Summary

urllib is part of the Python standard library and a handy tool for making HTTP requests and working with URLs. Throughout this article, you learned about GET/POST requests, URL parsing, scraping, and integrating with APIs. urllib requires no additional installation and can be used by anyone right away, so it’s highly recommended for anyone considering web application development. By actually trying it out yourself, you’ll gain a deeper understanding of how to use urllib.
年収訴求