Complete Guide to Handling Japanese Text in Python

1. Introduction

What you’ll learn in this article

• Basic setup for handling Japanese in Python
• Common issues in Japanese text processing and how to solve them
• How to use Python libraries useful for Japanese processing
• How to apply Japanese in real-world use cases

Why use Python to handle Japanese?

Purpose of this article

2. Basics of Japanese Character Encodings in Python

What is a character encoding?

• UTF-8: An international standard. Used in many environments and recommended as the default encoding in Python.
• Shift_JIS: An encoding that was widely used in Japan. Found in older systems and some datasets.
• EUC-JP: An encoding that was once used in Japanese environments.
• ISO-2022-JP: An encoding mainly used for email.

Handling character encodings in Python

Examples of encoding settings in Python

1. Declaring the encoding in source code

## -*- coding: utf-8 -*-
print("Hello, Python!")

2. Specifying the encoding when reading and writing files

with open("example.txt", "r", encoding="utf-8") as file:
    content = file.read()
    print(content)

with open("example.txt", "w", encoding="utf-8") as file:
    file.write("Hello, Python!")

3. Encoding and decoding operations

text = "Hello"
encoded_text = text.encode("utf-8")
print(encoded_text)  ## b'こんにちは'

decoded_text = encoded_text.decode("utf-8")
print(decoded_text)  ## Hello

Preventing garbled text in Python

1. Use UTF-8 Since Python uses UTF-8 by default, choosing UTF-8 whenever possible is the safest option.
2. Specify the encoding explicitly Always declare the encoding when performing file operations or interacting with external systems.
3. Add error handling If errors occur during encoding or decoding, you can handle them by specifying error-handling behavior.

try:
    with open("example.txt", "r", encoding="utf-8") as file:
        content = file.read()
except UnicodeDecodeError as e:
    print(f"Garbled text occurred: {e}")

Summary

3. How to Handle Japanese in Python

How to Manipulate Japanese Strings

Basic Operations on Japanese Strings

greeting = "Hello"
name = "Mr. Sato"
message = greeting + ", " + name
print(message)  ## Hello, Mr. Sato

word = "Python"
repeated = word * 3
print(repeated)  ## PythonPythonPython

Getting the Length of Japanese Strings

text = "Hello"
print(len(text))  ## 5

text = "Hello"
byte_length = len(text.encode("utf-8"))
print(byte_length)  ## 15

Slicing Japanese Strings

text = "Hello, Python"
substring = text[0:5]  ## Get the first 5 characters
print(substring)  ## Hello

Searching and Replacing Japanese Using Regular Expressions

Searching Japanese Strings

import re

text = "The weather is nice today, isn't it."
pattern = "weather"
match = re.search(pattern, text)
if match:
    print(f"Found: {match.group()}")  ## Found: weather

Replacing Japanese Strings

import re

text = "The weather is nice today, isn't it."
new_text = re.sub("weather", "climate", text)
print(new_text)  ## The climate is nice today, isn't it.

Issues Specific to Japanese and How to Handle Them

Handling Half-width and Full-width Characters

import unicodedata

text = "ＡＢＣ123"
normalized = unicodedata.normalize("NFKC", text)
print(normalized)  ## ABC123

Removing Spaces and Newlines

text = "  Hello  
"
cleaned_text = text.strip()
print(cleaned_text)  ## Hello

Character Classification in Japanese

import re

text = "Today is December 15, 2024."
kanji = re.findall(r"[一-龥]", text)
print(kanji)  ## ['今', '日', '年', '月', '日']

Summary

4. Introduction to Japanese-supporting Libraries and How to Use Them

Popular Libraries for Japanese Support

MeCab

Installation

pip install mecab-python3

Basic Usage

import MeCab

text = "The weather is nice today."
mecab = MeCab.Tagger()
parsed = mecab.parse(text)
print(parsed)

今日    Noun,Adverbial,*,*,*,*,今日,キョウ,キョー
は      Particle,Binding,*,*,*,*,は,ハ,ワ
天気    Noun,General,*,*,*,*,天気,テンキ,テンキ
が      Particle,Case,General,*,*,*,が,ガ,ガ
良い    Adjective,Independent,*,*,Adjective-Auo,Base,良い,ヨイ,ヨイ
です    AuxiliaryVerb,*,*,*,Special-Desu,Base,です,デス,デス
。      Symbol,Punctuation,*,*,*,*,。,。,。
EOS

Janome

Installation

pip install janome

Basic Usage

from janome.tokenizer import Tokenizer

text = "The weather is nice today."
tokenizer = Tokenizer()
for token in tokenizer.tokenize(text):
    print(token)

今日    Noun,Adverbial,*,*,*,*,今日,キョウ,キョー
は      Particle,Binding,*,*,*,*,は,ハ,ワ
天気    Noun,General,*,*,*,*,天気,テンキ,テンキ
が      Particle,Case,General,*,*,*,が,ガ,ガ
良い    Adjective,Independent,*,*,Adjective-Auo,Base,良い,ヨイ,ヨイ
です    AuxiliaryVerb,*,*,*,Special-Desu,Base,です,デス,デス
。      Symbol,Punctuation,*,*,*,*,。,。,。

SudachiPy

Installation

pip install sudachipy sudachidict_core

Basic Usage

from sudachipy.tokenizer import Tokenizer
from sudachipy.dictionary import Dictionary

text = "The weather is nice today."
tokenizer = Dictionary().create()
mode = Tokenizer.SplitMode.C
for token in tokenizer.tokenize(text, mode):
    print(token.surface())

今日
は
天気
が
良い
です
。

How to Choose a Library

Summary

5. Input and Output of Japanese Data

Reading and Writing Files

Reading Text Files

with open("example.txt", "r", encoding="utf-8") as file:
    content = file.read()
    print(content)

Writing to Text Files

with open("example.txt", "w", encoding="utf-8") as file:
    file.write("Hello, I'm working with files in Python.")

Handling CSV Files

Reading CSV Files

import csv

with open("example.csv", "r", encoding="utf-8-sig") as file:
    reader = csv.reader(file)
    for row in reader:
        print(row)

Writing CSV Files

import csv

data = [["Name", "Age", "Address"], ["Sato", "30", "Tokyo"], ["Suzuki", "25", "Osaka"]]

with open("example.csv", "w", encoding="utf-8-sig", newline="") as file:
    writer = csv.writer(file)
    writer.writerows(data)

Handling JSON Files

Reading JSON Files

import json

with open("example.json", "r", encoding="utf-8") as file:
    data = json.load(file)
    print(data)

Writing JSON Files

import json

data = {"Name": "Sato", "Age": 30, "Address": "Tokyo"}

with open("example.json", "w", encoding="utf-8") as file:
    json.dump(data, file, ensure_ascii=False, indent=4)

Working with Databases

Storing and Retrieving Japanese Data with SQLite

import sqlite3

## Connect to the database
conn = sqlite3.connect("example.db")
cursor = conn.cursor()

## Create table
cursor.execute("CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT, age INTEGER)")
conn.commit()

## Insert Japanese data
cursor.execute("INSERT INTO users (name, age) VALUES (?, ?)", ("Sato", 30))
conn.commit()

## Retrieve Japanese data
cursor.execute("SELECT * FROM users")
rows = cursor.fetchall()
for row in rows:
    print(row)

conn.close()

Notes and Best Practices

Consistent Encoding

Adding Error Handling

try:
    with open("example.txt", "r", encoding="utf-8") as file:
        content = file.read()
except FileNotFoundError as e:
    print(f"File not found: {e}")

Choosing Libraries Based on Data Format

Summary

6. Tips and Best Practices for Working with Japanese

Key Considerations When Working with Japanese

Causes of Garbled Characters and Prevention

Main Causes

• The file encoding and the program’s settings do not match.
• The encoding differs when interacting with external systems (e.g., databases, APIs).

Prevention Measures

1. Always use UTF-8 UTF-8 is the standard character encoding that supports multiple languages. Use UTF-8 for all file operations and system settings.
2. Explicitly specify the encoding Always specify the encoding when reading/writing files or communicating with external systems.

with open("example.txt", "r", encoding="utf-8") as file:
    content = file.read()

Japanese-specific Text Processing

Unifying Full-width and Half-width Characters

import unicodedata

text = "Ｐｙｔｈｏｎ　２０２４"
normalized = unicodedata.normalize("NFKC", text)
print(normalized)  ## Python 2024

Classification of Japanese Characters

import re

text = "Today is December 15, 2024."
kanji = re.findall(r"[一-龥]", text)
print(kanji)  ## ['今', '日', '年', '月', '日']

Interacting with External Data

Database Character Encoding Settings

CREATE DATABASE example_db CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

Best Practices for Working with Japanese

Improve Code Readability

• Use comments and documentation to clarify the intent of the code.
• Write function and variable names in English.

Leverage Japanese-specific Libraries

from janome.tokenizer import Tokenizer

text = "The weather is nice today."
tokenizer = Tokenizer()
for token in tokenizer.tokenize(text):
    print(token.surface, token.part_of_speech)

Verify Japanese Processing with Unit Tests

def test_japanese_encoding():
    text = "Hello"
    encoded = text.encode("utf-8")
    assert encoded.decode("utf-8") == text

Implement Robust Error Handling

try:
    with open("example.txt", "r", encoding="utf-8") as file:
        content = file.read()
except UnicodeDecodeError as e:
    print(f"Garbled text error: {e}")

Summary

7. Summary and Next Steps

Recap of this article

1. Introduction

• Explained that Python has strong multilingual support and makes it easy to process Japanese text.

1. Basics of Japanese character encoding in Python

• Covered the importance of character encodings, why UTF-8 is recommended, and how to specify encodings.

1. How to handle Japanese in Python

• Reviewed basic Japanese string operations and examples of searching and replacing with regular expressions.

1. Overview and usage of Japanese-specific libraries

• Introduced how to use libraries suited for Japanese-specific processing, such as MeCab and Janome.

1. Input and output of Japanese data

• Learned how to read and write data in formats such as text, CSV, and JSON.

1. Cautions and best practices when handling Japanese

• Explained measures to prevent mojibake (garbled text), ways to handle Japanese-specific issues, and best practices for efficient work.

Python’s advantages in Japanese text processing

• Flexible multilingual support: Built-in Unicode support reduces the risk of mojibake.
• Rich library ecosystem: MeCab, Janome, SudachiPy, etc., can handle Japanese-specific challenges.
• Simple code structure: Achieve Japanese processing with concise, readable code.

Next steps

Start a practical project

• Try building simple applications or analysis projects using Japanese data.
• Examples: frequency analysis of Japanese text; text classification using morphological analysis.

Apply libraries in practice

• Explore the libraries introduced here in more depth and experiment with their use in different scenarios.
• Example: Build a Japanese chatbot using MeCab.

Try multilingual processing beyond Japanese

• With Python, you can support other languages as well. Work on developing multilingual applications.

Further your Python learning

• Not limited to Japanese processing, learning other areas of Python (data analysis, web development, machine learning, etc.) can broaden your skill set.

Hone your troubleshooting skills

• Resolving real encoding errors and mojibake issues will help you develop practical skills.

Resources for further learning

• Official Python documentation: https://docs.python.org/ja/
• MeCab official site: https://taku910.github.io/mecab/
• SudachiPy official documentation: https://github.com/WorksApplications/SudachiPy
• Books and other resources on Japanese natural language processing: ” NLP 100 Exercises” and others.

In closing