The World of Python Programming

Python Queue Tutorial: Master FIFO, LIFO, and Priority Queues with Examples

Multithreading & Asynchronous
目次

1. What is a Queue in Python?

Basic Concept of a Queue

A queue is a type of data structure that follows the FIFO (First In, First Out) principle. This means that the first element added is the first one retrieved. This mechanism is widely used in computer science and programming, making queues an essential tool for efficient data processing. For example, queues are commonly used in situations such as:
  • Task Scheduling: Execute tasks in the order they were started.
  • Buffering: Store streaming data in a queue and process it sequentially.
  • Inter-thread Communication: Manage the order of data when multiple threads process information simultaneously.
Python’s standard library provides the queue module, a powerful tool that makes queue operations easy to implement. This module includes an internal locking mechanism to safely exchange data between threads.

2. Use Cases of Queues in Python

Common Applications of Queues

There are many scenarios where queues are used in Python. Some of the most useful cases include:
  • Task Scheduling: An efficient way to handle multiple tasks sequentially. For example, when a web server receives a large number of requests, it can add them to a queue and process them one by one, ensuring efficient use of resources.
  • Data Buffering: Useful in streaming processes to temporarily store data until processing catches up. Examples include video streaming or real-time data processing.
  • Data Sharing Between Threads: Queues are an effective way to safely exchange data between different threads. They can be used to distribute tasks across threads in multithreaded programs.

3. Overview of the queue Module

Available Classes

The Python queue module provides three main classes. Below are their features and usage:
  1. Queue (FIFO Queue)
    • The most basic queue where the first added item is retrieved first. Implements the FIFO (First In, First Out) principle.
    • Example:
    import queue q = queue.Queue() q.put("task1") q.put("task2") print(q.get()) ## Outputs "task1"
  2. LifoQueue (LIFO Queue)
    • Behaves like a stack, retrieving the last added item first. Implements the LIFO (Last In, First Out) principle.
    • Example:
    import queue q = queue.LifoQueue() q.put("task1") q.put("task2") print(q.get()) ## Outputs "task2"
  3. PriorityQueue (Priority Queue)
    • Retrieves items based on priority. Lower values indicate higher priority.
    • Example:
    import queue q = queue.PriorityQueue() q.put((1, "task1")) q.put((3, "task3")) q.put((2, "task2")) print(q.get()) ## Outputs "(1, 'task1')"
These classes should be chosen based on the specific situation.

4. Implementing a FIFO Queue

Basic Usage

The FIFO queue is the most common type. It can be easily implemented using queue.Queue. Here’s a simple example:
import queue

## Create a FIFO queue
q = queue.Queue()

## Add elements to the queue
q.put("apple")
q.put("banana")
q.put("cherry")

## Retrieve elements from the queue
while not q.empty():
    print(q.get())
This code retrieves elements in the order "apple", "banana", "cherry". The empty() method ensures the loop continues until the queue is empty.

Practical Example

For instance, when a web server processes incoming requests, it can enqueue each request and handle them sequentially. In such cases, a FIFO queue works effectively.

5. Advanced Queue Operations

Queue Methods

The Python queue module provides several useful methods for efficient queue operations. Some of the most important are:
  1. qsize()
    • Returns the number of items in the queue. Useful for checking if it’s empty.
    • Example:
    q = queue.Queue() q.put("task1") print(q.qsize()) ## Outputs 1
  2. empty()
    • Checks if the queue is empty. Returns True or False.
    • Example:
    q = queue.Queue() print(q.empty()) ## Outputs True
  3. full()
    • Checks if the queue is full. Only applicable when maxsize is set.
    • Example:
    q = queue.Queue(maxsize=2) q.put("task1") q.put("task2") print(q.full()) ## Outputs True
  4. put(item)
    • Adds an item to the queue. By default, block=True, which may block until space is available.
    • Example:
    q = queue.Queue() q.put("task1")
  5. get()
    • Retrieves an item from the queue. If empty and block=True, it waits until an item is available.
    • Example:
    q = queue.Queue() q.put("task1") task = q.get() print(task) ## Outputs "task1"

6. Exception Handling with Queues

Queue Exceptions

The queue module provides exceptions to handle errors efficiently during queue operations:
  1. queue.Full
    • Raised when calling put() on a full queue.
    • Example:
    try: q.put("task", block=False) except queue.Full: print("Queue is full")
  2. queue.Empty
    • Raised when calling get() on an empty queue.
    • Example:
    try: task = q.get(block=False) except queue.Empty: print("Queue is empty")
These exceptions are particularly important for blocking operations, ensuring the program does not crash unexpectedly.

7. Using Queues in Python Multithreading

Task Management in Multithreading

The queue module is especially useful in multithreaded environments. Queues allow safe data sharing and efficient task distribution between threads. Example:
import queue
import threading

## Create a queue
q = queue.Queue()

## Define a worker thread
def worker():
    while True:
        item = q.get()
        print(f"Processing: {item}")
        q.task_done()

## Start a thread
threading.Thread(target=worker, daemon=True).start()

## Add tasks to the queue
for item in range(5):
    q.put(item)

## Wait for all tasks to finish
q.join()
print("All tasks completed")
In this program, multiple threads can simultaneously process tasks from the queue until all tasks are done, avoiding data conflicts while enabling efficient parallel processing.

8. Using Bounded Queues

What is a Bounded Queue?

A bounded queue has a fixed maximum size. This is useful for avoiding resource overuse in specific scenarios. For instance, when a web server handles大量 requests, setting a limit prevents overload. Key features include:
  1. Behavior when full: When the queue is full, adding new items triggers one of two behaviors:
  • Reject new items: Additional items are not accepted.
  • Overwrite old items: The oldest item is removed to make space for a new one.
  1. Resource Management: Helps manage resources like memory and CPU efficiently.

Example

import queue

## Create a bounded queue
q = queue.Queue(maxsize=3)

## Add items
q.put("task1")
q.put("task2")
q.put("task3")

## Adding another item raises an exception
try:
    q.put_nowait("task4")
except queue.Full:
    print("Queue is full")
In this example, since the queue size is limited to 3, adding a fourth item raises a queue.Full exception.

9. Conclusion

The Python queue module is a powerful tool for efficient data management, parallel processing, and thread communication. Using FIFO, LIFO, and Priority Queues allows flexible handling of different scenarios. Additionally, exception handling and bounded queues enhance error control and resource efficiency. If you’re working with complex data processing in Python, leveraging these features is highly recommended.