Python Mouse Control with PyAutoGUI – Image, Click, JP Input

1. Introduction (Benefits and Use Cases of Automating Mouse Operations with Python)

• Automating routine data entry or simple tasks that involve repeatedly clicking the same spot.
• Periodic checks of web pages or applications and capturing screenshots.
• Grinding, rerolling, or any repetitive pattern-based actions within games.
• Using it as part of batch processing in business software or RPA (Robotic Process Automation).

2. Prerequisite Knowledge and Preparation

pip install pyautogui
pip install pyperclip pillow

import pyautogui
print(pyautogui.position())

3. Basics of Mouse Operations You Can Do with Python

3-1. How to Move the Mouse Cursor

import pyautogui
pyautogui.moveTo(100, 200)

pyautogui.moveTo(500, 300, duration=2)

pyautogui.moveRel(50, 0)  # move right 50px

3-2. Click Operations (Single/Double/Right Click)

pyautogui.click(400, 400)

pyautogui.doubleClick(400, 400)      # double click
pyautogui.rightClick(400, 400)       # right click
pyautogui.click(400, 400, button='right')  # right click (also works via argument)
pyautogui.click(400, 400, clicks=3, interval=0.5)  # triple click with 0.5‑second interval

3-3. Drag‑and‑Drop Operations

# Drag from (300, 300) to (600, 600) over 2 seconds
pyautogui.moveTo(300, 300)
pyautogui.dragTo(600, 600, duration=2)

pyautogui.dragRel(100, 0, duration=1)  # drag right 100px from current location

3-4. Scrolling with the Mouse Wheel

pyautogui.scroll(500)   # scroll up 500px
pyautogui.scroll(-300)  # scroll down 300px

4. Automatic Clicking Using Coordinate Retrieval and Image Recognition

4-1. How to Retrieve Screen Coordinates

import pyautogui
input("Move the mouse cursor to the position where you want to get the coordinates and press the Enter key:")
print(pyautogui.position())

pyautogui.displayMousePosition()

4-2. Identifying Click Positions with Image Matching (Image Recognition)

import pyautogui

# Search the screen for a location matching button.png and click its center
location = pyautogui.locateCenterOnScreen('button.png', confidence=0.8)
if location is not None:
    pyautogui.click(location)
else:
    print("Image not found.")

• Make the screenshot image match the actual display’s resolution and size as closely as possible
• Differences in OS or screen scaling can cause mismatches
• If the button’s color or design changes, it may be hard to recognize

5. Keyboard Automation Integration Techniques

5-1. Text Input and Special Key Operations

import pyautogui

pyautogui.write('Hello, world!')          # Input alphanumeric characters
pyautogui.press('enter')                  # Press the Enter key
pyautogui.hotkey('ctrl', 'a')             # Send Ctrl+A (select all)
pyautogui.hotkey('ctrl', 'c')             # Ctrl+C (copy)
pyautogui.hotkey('ctrl', 'v')             # Ctrl+V (paste)

5-2. Tips for Automating Japanese Input

import pyautogui
import pyperclip
import time

text = "Hello, Python automation!"
pyperclip.copy(text)                           # Copy the string to the clipboard
pyautogui.hotkey('ctrl', 'v')                 # Paste with Ctrl+V
time.sleep(0.2)                               # Wait a short moment just in case
pyautogui.press('enter')                      # Confirm with the Enter key

5-3. Example of Mouse and Keyboard Integration

import pyautogui
import pyperclip
import time

# Click the input field
pyautogui.click(500, 350)

# Paste Japanese and press Enter
pyperclip.copy("Test message")
pyautogui.hotkey('ctrl', 'v')
time.sleep(0.2)
pyautogui.press('enter')</> In this way, combining mouse actions with keyboard input enables more advanced automation and a wider range of tasks. In the next chapter, we will present practical sample code that uses these techniques.

6. Frequently Used Practical Sample Code Collection

6-1. Script for Automatic Clicking at Specified Coordinates

import pyautogui
import time

# Click the coordinates (400, 500) 10 times with a 2‑second interval
for i in range(10):
    pyautogui.click(400, 500)
    time.sleep(2)

6-2. Find and Click a Button Using Image Recognition

import pyautogui
import time

for i in range(5):
    location = pyautogui.locateCenterOnScreen('button.png', confidence=0.8)
    if location is not None:
        pyautogui.click(location)
        print("Clicked the button")
    else:
        print("Button not found")
    time.sleep(1)

6-3. Switching Between Multiple Windows and Automating Operations

import pyautogui
import time

# Switch windows with Alt+Tab, then click
pyautogui.hotkey('alt', 'tab')
time.sleep(0.5)
pyautogui.click(300, 300)

6-4. Coordinating Mouse and Keyboard Operations

import pyautogui
import pyperclip
import time

# Click the input field and paste Japanese text
pyautogui.click(600, 400)
pyperclip.copy("Automation test message")
pyautogui.hotkey('ctrl', 'v')
time.sleep(0.2)
pyautogui.press('enter')

7. Advanced Tips & Troubleshooting

7-1. Checklist for When Clicks Don’t Register

• Window Foregrounding If the target window or application isn’t displayed in the foreground, clicks may not register as expected. Using pyautogui‘s hotkey() to send Alt+Tab and bring the window to the front can be effective.
• Coordinate Drift & Screen Resolution Differences If the screen resolution or scaling settings differ between development and production environments, the specified coordinates can be offset. Unify the environment or combine with image recognition to increase flexibility in coordinate targeting.
• Image Recognition Not Working Even slight differences in resolution or color of button or icon images can cause recognition failures. Re-capture the image or adjust the confidence value and try again.

7-2. Looping & Periodic Automated Clicks

import pyautogui
import time

try:
    while True:
        pyautogui.click(400, 500)
        time.sleep(60)  # Click every minute
except KeyboardInterrupt:
    print("Manually stopped")

7-3. Safety Measures During Automation (PAUSE and FAILSAFE)

• pyautogui.PAUSE A setting that automatically inserts a pause after every pyautogui function call. It helps align with human interaction and stabilizes actions.

import pyautogui
pyautogui.PAUSE = 0.5  # Pause 0.5 seconds between all actions

• pyautogui.FAILSAFE Moving the mouse to the top-left corner of the screen forces the script to stop (enabled by default).

import pyautogui
pyautogui.FAILSAFE = True  # Enable safety mechanism (default)

7-4. Differences & Considerations Across OSes (Windows/Mac/Linux)

• Be aware of shortcut key differences (e.g., Ctrl vs. Cmd).
• Image recognition accuracy and window management conventions can vary by OS.
• Some features or key inputs may only be supported on certain OSes.

8. Advancing to GUI Tools, Batch Processing, and Executable Packaging

8-1. Creating a GUI Tool (Using PySimpleGUI)

import PySimpleGUI as sg
import pyautogui
import time

layout = [
    [sg.Text('Click coordinates (X, Y):')],
    [sg.InputText('400', key='X'), sg.InputText('500', key='Y')],
    [sg.Button('Start clicking'), sg.Button('Exit')]
]

window = sg.Window('Mouse Auto-Click Tool', layout)

while True:
    event, values = window.read()
    if event in (None, 'Exit'):
        break
    if event == 'Start clicking':
        x = int(values['X'])
        y = int(values['Y'])
        pyautogui.click(x, y)
        sg.popup('Clicked')
window.close()

8-2. Batch Processing and Running Multiple Actions Together

@echo off
python my_mouse_script.py
pause

8-3. Creating an Executable (Packaging with PyInstaller for Distribution)

pip install pyinstaller

pyinstaller --onefile my_mouse_script.py

9. Summary and Future Development

• Time savings and quality improvement through automating repetitive and simple tasks
• Diverse applications by combining with image recognition and keyboard input automation
• Stable operation is achievable by considering troubleshooting measures and safety design
• Scalability to shareable formats such as GUI applications and exe files

10. FAQ (Frequently Asked Questions and Answers)

Q1. Does it work on Windows, macOS, and Linux?

Q2. How to handle Japanese input that doesn’t work?

Q3. What to do when image recognition clicks fail?

• Whether the screenshot image matches the actual button or icon in size and color
• If the screen scaling or resolution has changed
• Adjust the confidence value (e.g., 0.8 or 0.9) to improve recognition
• Whether the button or icon design has changed

Q4. Why does the script sometimes stop or throw an error while running?

Q5. How can I adjust click speed and the interval between actions?

Q6. Can you automate tasks other than mouse actions (keyboard, screen capture, etc.)?