Handwritten Data Input: A Task That Always Required Human Eyes
The problem our team faced was surprisingly simple. Some of the order forms sent by overseas buyers were handwritten PDFs or scanned images. Shipping addresses, special requests, and other information were written with pens on printed forms.
Initially, someone had to open these images, "read" them, and manually type the information into Google Sheets. With about 200 to 300 scans per month, it consumed a significant amount of time.
"Could a bot read these for us?"
Can Claude Code Give a Robot "Eyes"?
One of Claude Code's strengths is that it can understand images. Not just simple character recognition (OCR), but also grasp the context of the entire image.
Our approach was:
1. **Monitor Upload Folder**: Automatically detect when scanned images arrive in a specific folder on the Windows device
2. **Call Claude Code**: Send the image to the Claude API
3. **Context-Based Extraction**: Not just recognize characters, but understand "this field is a shipping address," "this is a special request," and similar semantic meanings
4. **Auto-Fill Google Sheets**: Automatically place extracted data in the correct columns
Actual Implementation Steps (Follow Along)
Step 1: Prepare Claude API Key
First, request an API key at the [Claude API Console](https://console.anthropic.com).
Step 2: Set Up Python Environment
pip install anthropic google-auth-oauthlib google-auth-httplib2 google-api-python-client pillow
Step 3: Basic Script (Image Reading)
import anthropic
import base64
from pathlib import Path
def read_handwriting_image(image_path):
"""Extract handwritten data from image"""
# Encode image as Base64
with open(image_path, "rb") as img:
image_data = base64.standard_b64encode(img.read()).decode("utf-8")
client = anthropic.Anthropic(api_key="YOUR_API_KEY")
# Send image and question to Claude
message = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
messages=[
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": image_data,
},
},
{
"type": "text",
"text": """Extract the following information from this image.
- Shipping Address
- Recipient Name
- Contact Information
- Special Requests
Return in JSON format."""
}
],
}
],
)
return message.content[0].text
# Usage Example
result = read_handwriting_image("order_scan.jpg")
print(result)
Step 4: Auto-Fill Google Sheets
Parse the results from the above script and add them to Google Sheets, and you're done. (Google Sheets API integration details can be found in other tutorials)
Actual Changes We Saw
• **Time Saved**: 6 to 8 hours per month
• **Accuracy**: Handwriting recognition rate around 92% (not perfect, but only requires verification)
• **Reduced Human Error**: Significant decrease in typos and missing information that occurred during manual data entry
Limitations and What We Learned
Not all handwriting can be recognized. Particularly:
• When handwriting is very faint or small
• When multiple languages are mixed
In these cases, the bot flags uncertainty, and the team does manual verification using a "hybrid" approach.
Sometimes "good enough automation" is more practical than perfect automation.
Next Experiment: Learning Handwriting Patterns
With slight refinements to Claude Code, we should be able to learn specific buyer's handwriting patterns and further improve accuracy. We'll share that in the next article.