Featured image of post Fix character encoding issues
Data Wrangling Problems

Fix character encoding issues

A solution pattern for repairing text broken by Shift JIS and UTF-8 mismatches

Use this pattern when Japanese text becomes garbled after opening a CSV, or when a GIS or spreadsheet app fails to read the file correctly.

Solution

Identify the original encoding first, then convert the file into the encoding expected by the next tool. A fast converter makes it easy to validate both directions before sharing or importing the data.

Tool

Shift JIS ⇄ UTF-8 Converter
dataprep.jp
Shift JIS ⇄ UTF-8 Converter
Convert character encoding between Shift JIS and UTF-8
change-character-encoding.dataprep.jp

Good fit for

  • Preparing CSV files for distribution
  • Fixing import errors in Japanese datasets
  • Moving files between different systems