• mb_check_encoding()
• mb_convert_encoding()

To check if $string is in UTF-8 encoding, we call mb_check_encoding() like this:

if (mb_check_encoding($string, "UTF-8")) { // do_something(); }

To convert $string, which is currently Shift-JIS, to UTF-8, we call mb_convert_encoding() like this:

$convertedString = mb_convert_encoding($string, "UTF-8", "Shift-JIS);

A convenient feature of mb_convert_encoding() is that you can generalize the function by adding a list of character encodings to convert from. This can come in very handy if you want to convert all Japanese multibyte string encodings to UTF-8, or something else. There are actually 18 Japanese-specific multibyte encodings (that I know of), not including all the Unicode variants like UTF-8, UTF-16, etc. A lot of them come from the Japanese mobile phone carriers.

Let’s put all of this together and check if a string is UTF-8, and if it’s not, meaning it is one of the other 18 Japanese encoding types, let’s convert it to UTF-8.

if (!mb_check_encoding($string, "UTF-8")) {

   $string = mb_convert_encoding($string, "UTF-8",
      "Shift-JIS, EUC-JP, JIS, SJIS, JIS-ms, eucJP-win, SJIS-win, ISO-2022-JP,
       ISO-2022-JP-MS, SJIS-mac, SJIS-Mobile#DOCOMO, SJIS-Mobile#KDDI,
       SJIS-Mobile#SOFTBANK, UTF-8-Mobile#DOCOMO, UTF-8-Mobile#KDDI-A,
       UTF-8-Mobile#KDDI-B, UTF-8-Mobile#SOFTBANK, ISO-2022-JP-MOBILE#KDDI");
}

To deal with this, know that each character in Unicode has a hexadecimal code point. For example, the code point for the hiragana あ is 3042, and this is designated by U+3042. This code point can be used in a regular expression like this: \x3042. This will match a hiragana あ. This is very useful for programmers who must code pattern matching for Japanese on a system where they cannot input or display Japanese text, or have the know-how to do it (See some of my great Japanese input posts if you need to know how!).

Additionally, some flavors of regular expressions have what are known as Unicode block properties, or Unicode scripts. These are pre-defined blocks of regex Unicode character classes. Hiragana, katakana, and kanji are included in the block properties—very convenient if you need the full-script match in your regular expression.

With this basic knowledge, the following is a thorough list of different Japanese character classes and the various Japanese regular expressions that match those character classes. And further down, a few programming examples showing them in use.

Hiragana

Unicode code points regex: [\x3041-\x3096]
Unicode block property regex: \p{Hiragana}

ぁ あ ぃ い ぅ う ぇ え ぉ お か が き ぎ く ぐ け げ こ ご さ ざ し じ す ず せ ぜ そ ぞ た だ ち ぢ っ つ づ て で と ど な に ぬ ね の は ば ぱ ひ び ぴ ふ ぶ ぷ へ べ ぺ ほ ぼ ぽ ま み む め も ゃ や ゅ ゆ ょ よ ら り る れ ろ ゎ わ ゐ ゑ を ん ゔ ゕ ゖ  ゙ ゚ ゛ ゜ ゝ ゞ ゟ

Katakana (Full Width)

Unicode code points regex: [\x30A0-\x30FF]
Unicode block property regex: \p{Katakana}

゠ ァ ア ィ イ ゥ ウ ェ エ ォ オ カ ガ キ ギ ク グ ケ ゲ コ ゴ サ ザ シ ジ ス ズ セ ゼ ソ ゾ タ ダ チ ヂ ッ ツ ヅ テ デ ト ド ナ ニ ヌ ネ ノ ハ バ パ ヒ ビ ピ フ ブ プ ヘ ベ ペ ホ ボ ポ マ ミ ム メ モ ャ ヤ ュ ユ ョ ヨ ラ リ ル レ ロ ヮ ワ ヰ ヱ ヲ ン ヴ ヵ ヶ ヷ ヸ ヹ ヺ ・ ー ヽ ヾ ヿ

Kanji

Unicode code points regex: [\x3400-\x4DB5\x4E00-\x9FCB\xF900-\xFA6A]
Unicode block property regex: \p{Han}

漢字 日本語 文字 言語 言葉 etc. Too many characters to list.

This regular expression will match all the kanji, including those used in Chinese.

Kanji Radicals

Unicode code points regex: [\x2E80-\x2FD5]

⺀ ⺁ ⺂ ⺃ ⺄ ⺅ ⺆ ⺇ ⺈ ⺉ ⺊ ⺋ ⺌ ⺍ ⺎ ⺏ ⺐ ⺑ ⺒ ⺓ ⺔ ⺕ ⺖ ⺗ ⺘ ⺙ ⺚ ⺛ ⺜ ⺝ ⺞ ⺟ ⺠ ⺡ ⺢ ⺣ ⺤ ⺥ ⺦ ⺧ ⺨ ⺩ ⺪ ⺫ ⺬ ⺭ ⺮ ⺯ ⺰ ⺱ ⺲ ⺳ ⺴ ⺵ ⺶ ⺷ ⺸ ⺹ ⺺ ⺻ ⺼ ⺽ ⺾ ⺿ ⻀ ⻁ ⻂ ⻃ ⻄ ⻅ ⻆ ⻇ ⻈ ⻉ ⻊ ⻋ ⻌ ⻍ ⻎ ⻏ ⻐ ⻑ ⻒ ⻓ ⻔ ⻕ ⻖ ⻗ ⻘ ⻙ ⻚ ⻛ ⻜ ⻝ ⻞ ⻟ ⻠ ⻡ ⻢ ⻣ ⻤ ⻥ ⻦ ⻧ ⻨ ⻩ ⻪ ⻫ ⻬ ⻭ ⻮ ⻯ ⻰ ⻱ ⻲ ⻳
⼀ ⼁ ⼂ ⼃ ⼄ ⼅ ⼆ ⼇ ⼈ ⼉ ⼊ ⼋ ⼌ ⼍ ⼎ ⼏ ⼐ ⼑ ⼒ ⼓ ⼔ ⼕ ⼖ ⼗ ⼘ ⼙ ⼚ ⼛ ⼜ ⼝ ⼞ ⼟ ⼠ ⼡ ⼢ ⼣ ⼤ ⼥ ⼦ ⼧ ⼨ ⼩ ⼪ ⼫ ⼬ ⼭ ⼮ ⼯ ⼰ ⼱ ⼲ ⼳ ⼴ ⼵ ⼶ ⼷ ⼸ ⼹ ⼺ ⼻ ⼼ ⼽ ⼾ ⼿ ⽀ ⽁ ⽂ ⽃ ⽄ ⽅ ⽆ ⽇ ⽈ ⽉ ⽊ ⽋ ⽌ ⽍ ⽎ ⽏ ⽐ ⽑ ⽒ ⽓ ⽔ ⽕ ⽖ ⽗ ⽘ ⽙ ⽚ ⽛ ⽜ ⽝ ⽞ ⽟ ⽠ ⽡ ⽢ ⽣ ⽤ ⽥ ⽦ ⽧ ⽨ ⽩ ⽪ ⽫ ⽬ ⽭ ⽮ ⽯ ⽰ ⽱ ⽲ ⽳ ⽴ ⽵ ⽶ ⽷ ⽸ ⽹ ⽺ ⽻ ⽼ ⽽ ⽾ ⽿ ⾀ ⾁ ⾂ ⾃ ⾄ ⾅ ⾆ ⾇ ⾈ ⾉ ⾊ ⾋ ⾌ ⾍ ⾎ ⾏ ⾐ ⾑ ⾒ ⾓ ⾔ ⾕ ⾖ ⾗ ⾘ ⾙ ⾚ ⾛ ⾜ ⾝ ⾞ ⾟ ⾠ ⾡ ⾢ ⾣ ⾤ ⾥ ⾦ ⾧ ⾨ ⾩ ⾪ ⾫ ⾬ ⾭ ⾮ ⾯ ⾰ ⾱ ⾲ ⾳ ⾴ ⾵ ⾶ ⾷ ⾸ ⾹ ⾺ ⾻ ⾼ ⾽ ⾾ ⾿ ⿀ ⿁ ⿂ ⿃ ⿄ ⿅ ⿆ ⿇ ⿈ ⿉ ⿊ ⿋ ⿌ ⿍ ⿎ ⿏ ⿐ ⿑ ⿒ ⿓ ⿔ ⿕

Katakana and Punctuation (Half Width)

Unicode code points regex: [\xFF5F-\xFF9F]

｟ ｠ ｡ ｢ ｣ ､ ･ ｦ ｧ ｨ ｩ ｪ ｫ ｬ ｭ ｮ ｯ ｰ ｱ ｲ ｳ ｴ ｵ ｶ ｷ ｸ ｹ ｺ ｻ ｼ ｽ ｾ ｿ ﾀ ﾁ ﾂ ﾃ ﾄ ﾅ ﾆ ﾇ ﾈ ﾉ ﾊ ﾋ ﾌ ﾍ ﾎ ﾏ ﾐ ﾑ ﾒ ﾓ ﾔ ﾕ ﾖ ﾗ ﾘ ﾙ ﾚ ﾛ ﾜ ﾝ ﾞ

Japanese Symbols and Punctuation

Unicode code points regex: [\x3000-\x303F]

、 。 〃 〄 々 〆 〇 〈 〉 《 》 「 」 『 』 【 】 〒 〓 〔 〕 〖 〗 〘 〙 〚 〛 〜 〝 〞 〟 〠 〡 〢 〣 〤 〥 〦 〧 〨 〩 〪 〫 〬 〭 〮 〯 〰 〱 〲 〳 〴 〵 〶 〷 〸 〹 〺 〻 〼 〽 〾 〿

Miscellaneous Japanese Symbols and Characters

Unicode code points regex: [\x31F0-\x31FF\x3220-\x3243\x3280-\x337F]

ㇰ ㇱ ㇲ ㇳ ㇴ ㇵ ㇶ ㇷ ㇸ ㇹ ㇺ ㇻ ㇼ ㇽ ㇾ ㇿ
㈠ ㈡ ㈢ ㈣ ㈤ ㈥ ㈦ ㈧ ㈨ ㈩ ㈪ ㈫ ㈬ ㈭ ㈮ ㈯ ㈰ ㈱ ㈲ ㈳ ㈴ ㈵ ㈶ ㈷ ㈸ ㈹ ㈺ ㈻ ㈼ ㈽ ㈾ ㈿ ㉀ ㉁ ㉂ ㉃
㊀ ㊁ ㊂ ㊃ ㊄ ㊅ ㊆ ㊇ ㊈ ㊉ ㊊ ㊋ ㊌ ㊍ ㊎ ㊏ ㊐ ㊑ ㊒ ㊓ ㊔ ㊕ ㊖ ㊗ ㊘ ㊙ ㊚ ㊛ ㊜ ㊝ ㊞ ㊟ ㊠ ㊡ ㊢ ㊣ ㊤ ㊥ ㊦ ㊧ ㊨ ㊩ ㊪ ㊫ ㊬ ㊭ ㊮ ㊯ ㊰ ㊱ ㊲ ㊳ ㊴ ㊵ ㊶ ㊷ ㊸ ㊹ ㊺ ㊻ ㊼ ㊽ ㊾ ㊿
㋀ ㋁ ㋂ ㋃ ㋄ ㋅ ㋆ ㋇ ㋈ ㋉ ㋊ ㋋  ㋐ ㋑ ㋒ ㋓ ㋔ ㋕ ㋖ ㋗ ㋘ ㋙ ㋚ ㋛ ㋜ ㋝ ㋞ ㋟ ㋠ ㋡ ㋢ ㋣ ㋤ ㋥ ㋦ ㋧ ㋨ ㋩ ㋪ ㋫ ㋬ ㋭ ㋮ ㋯ ㋰ ㋱ ㋲ ㋳ ㋴ ㋵ ㋶ ㋷ ㋸ ㋹ ㋺ ㋻ ㋼ ㋽ ㋾
㌀ ㌁ ㌂ ㌃ ㌄ ㌅ ㌆ ㌇ ㌈ ㌉ ㌊ ㌋ ㌌ ㌍ ㌎ ㌏ ㌐ ㌑ ㌒ ㌓ ㌔ ㌕ ㌖ ㌗ ㌘ ㌙ ㌚ ㌛ ㌜ ㌝ ㌞ ㌟ ㌠ ㌡ ㌢ ㌣ ㌤ ㌥ ㌦ ㌧ ㌨ ㌩ ㌪ ㌫ ㌬ ㌭ ㌮ ㌯ ㌰ ㌱ ㌲ ㌳ ㌴ ㌵ ㌶ ㌷ ㌸ ㌹ ㌺ ㌻ ㌼ ㌽ ㌾ ㌿ ㍀ ㍁ ㍂ ㍃ ㍄ ㍅ ㍆ ㍇ ㍈ ㍉ ㍊ ㍋ ㍌ ㍍ ㍎ ㍏ ㍐ ㍑ ㍒ ㍓ ㍔ ㍕ ㍖ ㍗ ㍘ ㍙ ㍚ ㍛ ㍜ ㍝ ㍞ ㍟ ㍠ ㍡ ㍢ ㍣ ㍤ ㍥ ㍦ ㍧ ㍨ ㍩ ㍪ ㍫ ㍬ ㍭ ㍮ ㍯ ㍰ ㍱ ㍲ ㍳ ㍴ ㍵ ㍶  ㍻ ㍼ ㍽ ㍾ ㍿

Alphanumeric and Punctuation (Full Width)

Unicode code points regex: [\xFF01-\xFF5E]

！ ＂ ＃ ＄ ％ ＆ ＇ （ ） ＊ ＋ ， － ． ／ ０ １ ２ ３ ４ ５ ６ ７ ８ ９ ： ； ＜ ＝ ＞ ？
＠ Ａ Ｂ Ｃ Ｄ Ｅ Ｆ Ｇ Ｈ Ｉ Ｊ Ｋ Ｌ Ｍ Ｎ Ｏ Ｐ Ｑ Ｒ Ｓ Ｔ Ｕ Ｖ Ｗ Ｘ Ｙ Ｚ ［ ＼ ］ ＾ ＿
｀ ａ ｂ ｃ ｄ ｅ ｆ ｇ ｈ ｉ ｊ ｋ ｌ ｍ ｎ ｏ ｐ ｑ ｒ ｓ ｔ ｕ ｖ ｗ ｘ ｙ ｚ ｛ ｜ ｝ ～

Japanese RegEx Code Examples

Find all hiragana in a text string

// PHP
$pattern = "/[\x{3041}-\x{3096}]/u";
preg_match_all($pattern, $text, $matches);
print_r($matches);

# Perl
if ($text =~ m/[\x{3041}-\x{3096}]/) { print $text; }

Remove all hiragana from a text string

//PHP
$pattern = "/\p{Hiragana}/u";
$text = preg_replace($pattern, "", $text);

# Perl
$text =~ s/\p{Hiragana}//g;

Remove everything but Kanji

// PHP
// \P{Han} matches everything other than kanji
$pattern = "/\P{Han}/u";
$text = preg_replace($pattern, "", $text);

Note: In PHP and Perl, the Unicode code block regular expression is written with curly braces around the hexadecimal codes. So the regex of \x3041 becomes \x{3041} and so on.

Note: In Perl you have to make sure you have Unicode set up properly to get regular expressions to work over Japanese. You may also have to run perl with the -CS options (perl -CS) to get rid of any Wide character in print warnings. See http://ahinea.com/en/tech/perl-unicode-struggle.html for more information.

Students of Japanese often wonder what kanji they should learn. It can be hard to identify what kanji is most important. And even between subject matters what kanji is more important will differ.

To help with this, I’ve created a kanji concordance application in PHP to create a list of kanji and their usage counts in descending order.

Example

If you had this Japanese text:

私の名前はマークです。私はテキサス大学を卒業しました。すしが大好きです。

The kanji concordance would generate a list that looked like this:

2 私
2 大
1 名
1 前
1 学
1 卒
1 業
1 好

The kanji 私 and 大 are both used twice, so they are at the top of the list with the number 2 for the usage count. The rest of the kanji are used once and show a usage count of 1.

Kanji Concordance Code Explanation

The first thing we do in PHP is set the language locale with the setlocale() function. This is always good practice when dealing with language-related applications.

setlocale(LC_ALL, "ja_JP.utf8");

The LC_ALL parameter sets the locale for all categories, and the ja_JP.utf8 parameter sets the language and locale to Japanese/Japan in Unicode UTF-8.

Next, we will need some string of Japanese text that we want to examine and create our kanji concordance from. In our simple example we will use a hard-coded string. But in a real application we would probably dynamically input the string from some source.

$string = "私の名前はマークです。私はテキサス大学を卒業しました。私はすしが大好きです。
           私は漢字が好きです。";

Once we have our input string, we need to strip it of everything but the kanji, since that is all we are interested in. Japanese text can have hiragana, katakana, English characters, and various punctuation. If we remove all of those, we’ll be left with just the kanji. We will define a regular expression to match these unwanted characters, and then replace them with nothing.

$pattern = "/[a-zA-Z0-9０-９あ-んア-ンー。、？！＜＞： 「」（）｛｝≪≫〈〉《》【】
            『』〔〕［］・\n\r\t\s\(\)　]/u";

This regular expression pattern is fairly straight forward. a-zA-Z0-9 matches the English alphanumeric characters. We also match the double-byte numbers with ０-９. あ-ん will match all the hiragana, and ア-ン will match all of the katakana. Finally, we match the various punctuation marks and other special characters we can expect to find. The u at the end of the pattern is a pattern modifier that tells PHP that this pattern is Unicode UTF-8. I’ve probably left out some punctuation characters but for our example purposes this will do. (We can actually do a much simplier regex than this. For a throrough discussion of regular expressions for Japanese text, see this post on Japanese regex.)

We will use the regular expression search and replace function preg_replace() to match our input string against the regex pattern to remove the unwanted characters.

$kanjiString = preg_replace($pattern, "", $string);

The first parameter, $pattern, is the regular expression pattern to match against. The second paramter “” is an empty string that we use to replace the regex matches. We match an unwanted non-kanji character and replace it with nothing—in other words, we delete it. The last parameter $string is the input string of Japanese to match against the regex pattern and remove everything but the kanji.

The variable $kanjiString now contains only the kanji characters from our original input string.

// $kanjiString = "私名前私大学卒業私大好私漢字好";

Our next step is to split up all the kanji characters and insert them into an array. We will do this in one step with the split by regular expression function preg_split().

$kanjiArray = preg_split("//u", $kanjiString, -1, PREG_SPLIT_NO_EMPTY);

The first parameter “//u” is a regular expression that will match everything, and the u pattern modifer argument puts it in Unicode match mode. The second parameter $kanjiString is the input string to match against the regular expression. The third parameter -1 is the limit parameter, and -1 indicates no limit. This means it will parse the entire string. The final parameter is the PREG_SPLIT_NO_EMPTY flag. This flag sets it so only non-empty items will be returned.

Now that we have an array full of individual kanji, we want to count them to get our kanji usage numbers. The array_count_values() function will count all the values of our input array, and return a new array with those values and their usage count.

$countedArray = array_count_values($kanjiArray);

With our array of kanji and their usage counts, we just need to sort them in reverse order with the arsort() function.

arsort($countedArray);

Our counted_array now contains a list of all the kanji used from our input string in order of their usage counts. In other words, we have successfully built a kanji usage count concordance.

The final step is to iterate through the array and display our concordance to the screen. We will do this with a simple foreach loop over our array.

foreach ($countedArray as $kanji => $count) {
   echo "$count $kanji <br/>";
}

Our kanji usage count concordance will display like this:

4 私
2 好
2 大
1 漢
1 字
1 業
1 学
1 名
1 前
1 卒

There we go. We have a list of all the kanji used and in order from most used to least used. As we can see, 私 seems to be a pretty important kanji. Better put it on your list to study.

In this example our Japanese input string was hard coded, but we can easily expand this code to take in input from a file or even screen scrape a Web site and see what their most used kanji are. With a large enough input sample, we can get a pretty good list of kanji and usage counts for our concordance.

PHP Source Code

Here is the full source code for the kanji count usage concordance in PHP that we built.

<?php
setlocale(LC_ALL, "ja_JP.utf8");

$string = "私の名前はマークです。私はテキサス大学を卒業しました。私はすしが大好きです。
           私は漢字が好きです。";

$pattern = "/[a-zA-Z0-9０-９あ-んア-ンー。、？！＜＞： 「」（）｛｝≪≫〈〉《》【】
            『』〔〕［］・\n\r\t\s\(\)　]/u";
$kanjiString = preg_replace($pattern, "", $string);

$kanjiArray = preg_split("//u", $kanjiString, -1, PREG_SPLIT_NO_EMPTY);

$countedArray = array_count_values($kanjiArray);
arsort($countedArray);

foreach ($countedArray as $kanji => $count) {
   echo "$count $kanji <br/>";
}
?>

Prerequisites

• YaST software repositories are configured properly.

Setup Procedure

Click on the Kickoff Application Launcher.

On the Computer tab, click Install/Remove Software.

On the Search tab, search for anthy.

In the search results window showing the matching packages, select the anthy and ibus-anthy packages.

Press the Accept button on the bottom right of the window.

YaST will now download, install, and configure the anthy packages.

Do the same for ibus. Open Install/Remove Software, search for ibus, and select the package for ibus. Press Accept to install.

Click on the Kickoff Application Launcher, and from the Leave tab, click Restart to restart openSUSE with the new configuration.

After restarting, log back in.

You will now have the IBus input method framework keyboard icon in the bottom panel.

Right click the IBus input method framework keyboard icon and click on Preferences.

On the Input Method tab, select Japanese → Anthy from the dropdown menu.

Press the Add button to add Japanese Anthy input method, and then press Close.

Open up a text editor or any application with a text input window, and click on the IBus input method framework icon and select Japanese – Anthy.

The IBus input method framework keyboard icon will change to the Anthy Aち icon.

You can now type in Japanese.

Click the Anthy Aち icon to select between the various Japanese input modes.

That’s it. Setting up Japanese input on openSUSE 12.1 is not very difficult. When you try to type Japanese, make sure the cursor is in a text box in an application, or you may get an error saying No input window. 日本語入力方法を楽しんでください。

Linux Mint 12 Japanese IME Setup Procedure

Click on the Mint Menu and navigate to Other → Software Manager.

In the Software Manager, search for ibus.

Select ibus.

Click Install.

In the Authentication Required dialog box, enter your system password and press Authenticate.

Sotware Manager will now download and install IBus in the background.

While IBus is installing, search for anthy.

Select ibus-anthy and click Install.

In the Authentication Required dialog box, enter your system password and press Authenticate.

Software Manager will now download and install ibus-anthy in the background.

When the activity bar on the bottom shows 0 ongoing actions, installation is complete.

Close Software Manager.

From the Mint Menu, navigate to System Tools → System Settings.

Open Language Support.

Note: If language support was not installed during the Mint install process you may get a pop up dialog indicating that the language support is not installed completely. In that case, select Install to install the language support. In the Authentication Required dialog box, enter your system password and press Authenticate. The Applying changes screen will display and show the installation progress. When the language support has been fully installed, the Language Support screen will display.

On the Language Support screen, select Install / Remove Languages….

Scroll down and check Japanese, and then press Apply Changes.

On the Language Support screen, press the Keyboard input method system: drop down and select ibus.

Then press Close.

Click on the Mint Menu and select System Tools → IBus.

You should now have the little IBus keyboard icon displayed somewhere on the right side of your Gnome top panel.

Click on the IBus keyboard icon and select Preferences.

On the IBus Preferences screen, go to the Input Method tab.

Press the Select an input method dropdown and select Japanese → Anthy.

Press Add on the IBus Preferences screen to add the Anthy Japanese input method.

Open a text application like Text Editor. While the cursor is in the text field, press the Ibus keyboard icon in the top panel and select Japanese – Anthy.

The Japanese Anthy toolbar should appear and you can now type in Japanese. Place the cursor in a text input application like Text Edit and try to type in Japanese.

That’s all there is to it. Linux Mint is known to be a very easy to use distribution, but it takes quite a few more steps to install Japanese input than the latest versions of Ubuntu or Fedora.

Note: I had issues with the Anthy toolbar not appearing and instead showed this icon which usually means no input window found. But, I could still type in Japanese in this mode, so no worries if this happens to you.

Note: I had trouble when trying to add Japanese on the Install / Remove Languages screen. It worked fine in the Gnome 2 version of Mint 12 I installed in a virtual machine, but it gave a Software database is broken error message in the Gnome 3 version of Mint 12 I installed on a physical laptop. I tried reinstalling twice but I kept getting the same problem. It may have been a problem with the laptop because I also had issues when trying to install drivers for the wireless card. I had no issues with Japanese input on Mint in a VM.

ALC has many advanced search options similar to search engines like Google that you can use to refine your search queries. Let’s take a look at some of these search options.

Basic Search Options

And Search (Word1 Word2)

Search for phrases containing two or more search terms in the results. The search results will contain all the search terms.

Instructions: Put a space between each search term to be included in the search result.

Example: 野球 サッカー

Example: up down

Or Operator [Standalone] (Word1 | Word2)

Search for phrases containing one or more search terms in the results. The search results will contain at least one of the search terms.

Instructions: Put a | (vertical bar) between the search terms.

Example: 製造装置 | 製造設備

Example: USPS | FedEx

Or Operator [Within Phrase] (Word1 | Word2)

Search for different variations of phrases containing one of the terms in the parenthesis.

Instructions: Put a | (vertical bar) between the search terms that are inside of ().

Example: (ケーキ|ピザ)を食べます

Example: do (one’s | my | your | his | her | its | our | their) best

Exact Phrase Search (“Phrase”)

Search for an exact phrase.

Instructions: Put the phrase within double quotes “”.

Example: “open source software”

Advanced Search Options

Designating Number of Words In Between (Word {#} Word)

Specify a certain number of words between search terms.

Instructions: Put the number of words you want to appear between words in braces. For a specific number of words, put one number, like {2}. For a range of possibilities, put the end limits in braces, like {1,3}.

Example: make {2} request

This example will find phrases like make a personal request that have two words between make and request, but will not find phrases like make a request that only have one word in between.

Example: thank you {2,4} cooperation

Search All Conjugations ([Word])

Search for all variations of an English word such as verb conjugations and plurals, etc.

Instructions: Put the variable word in brackets [].

Example: “[go] the distance”

This example finds all forms of the word go, including the past tense went the distance. Notice we put the entire search query in quotes to find the full phrase.

Example: [take] pictures of

This example fines take, takes, taking, took, etc.

Terms to Exclude (-Word)

Exclude certain translations from your search results. Useful to narrow your focus when there are multiple translations for a word.

Instructions: Put a dash – before the word to exclude.

Example: サッカー -soccer

This example will find examples of the word 「サッカー」 that exclude the American translation of soccer, and finds those examples that use football instead.

Example: diet -国会

This example will exclude the Japanese governmental body the Diet. This is useful if you are looking for food and diet related translations.

Multiple Search Options

You can combine search options for really advanced search queries.

Example: “[take] (my | our | your | his | her) picture -can”

This example uses the exact phrase quotes, the conjugation search [], the or operator within a phrase, and the not operator to remove phrases containing the word can.

Searching ALC can often find hundreds of translations. These advanced search options are easy to use and can help narrow down what you are looking for.

For more information, refer to the ALC Help – Basic Usage, Help – High Level Usage, and Search Tips pages. These ALC help pages are all in Japanese.

Fedora still uses the IBus keyboard input method system and uses the Anthy Japanese input method for the Japanese keyboard input, so it will be a familiar process to set up and use if you have done it on earlier Fedora Linux distributions.

For previous versions of Fedora, refer to:

• Fedora 15 (Gnome 3)
• Fedora 14 (Gnome 2)

Fedora 16 Japanese IME Setup Procedure

To start, open Activities from the Top Panel.

In the Search Box, type Input Method and select the Input Method Selector.

In the Input Method Selector screen, select Use IBus (recommended).

Press the Preference… link to the right of Use IBus (recommended) to open the IBus Preferences screen.

On the Input Method tab, check the Customize active input methods check box.

Press the Select an input method dropdown and select Show all input methods.

Press the Select an input method dropdown once again and now select Japanese → Anthy.

Press the Add button, and then press Close.

You must log out for the changes to take effect, so press the Log Out button on the Input Method Selector screen.

When you log back in you will now have the IBus input method framework button on the Gnome top panel (It looks like a small keyboard). This is the button to change input modes. Open a text editor such as gedit or some other application with a text input window.

Press the IBus input method framework button and select Japanese – Anthy.

The keyboard icon has now changed to Aち, which shows the letter A and the hiragana character chi, which probably is trying to get something close the the pronunciation of Anthy while indicating Japanese/English input modes.

You should now be able to type in Japanese.

Use the Anthy Aち button to toggle between Japanese, English, and other Japanese IME modes.

Note: I did not have to log out and log back in for the changes to take effect to allow me to type in Japanese in Firefox. However, there may be applications that cannot take advantage of the IME changes until after logging out.

Note: If you get the message No input window when you try to select Japanese Anthy, make sure you have the mouse cursor in an application with a text input box, such as a text editor or a Web browser.

That’s it. You should be able to type in Japanese now. Setting up Japanese IME input on Fedora Linux is simple and very similar to previous versions of Fedora.

Fortunately, CakePHP and PHP itself provide us with some easy mechanisms to provide translations and localize our code without much effort. You do not have to make copies of HTML or PHP files. Everything will be done with the PHP files you already have, and the translated text strings will dynamically be inserted at render time, ready for the user in their localized language and format.

In this example, we will localize a CakePHP application that was written in English into Japanese. Let’s assume we have a menu for e-mail functions that we want to localize. This example assumes CakePHP version 2.0 (But earlier and perhaps later versions of CakePHP will work in a similar manner).

Wrapping Translatable Text in __() Functions

The first step in the localization process is to identify the text strings that will need to be translated, and replace them with CakePHP’s localized string __() function. Supposing our menu looked like the following:

<ul>
   <li>Send</li>
   <li>Reply</li>
   <li>Forward</li>
   <li>Delete</li>
</ul>

We would wrap each text string inside the __() function like as follows:

<ul>
   <li><?php echo __('Send') ?></li>
   <li><?php echo __('Reply') ?></li>
   <li><?php echo __('Forward') ?></li>
   <li><?php echo __('Delete') ?></li>
</ul>

The __() function identifies these strings as translatable text that will differ by language locale and uses the text within the __() function as the message ID. If we define the translations for a certain language, those translations will appear in place of these functions. If we do not define the translations for that language, the text within the __() function will display instead by default.

Creating the Localized PO Files

The next step is to create the PO files which will contain the translations for each language to be dynamically inserted in each of the __() functions. CakePHP has two ways you can do this: automatically using the console shell; or manually.

Using the I18N Shell

CakePHP has some console shell programs that you can run on the command line, including one to generate the PO file to use as the original language source file for translations. In our case it will be a file with all the English text strings.

To run the i18n shell command, type the following on the Linux command line in your CakePHP application directory:

./Console/cake i18n extract

Then follow the onscreen menu.

The shell command will examine all of your application files for instances of the __() function and generate a PO file for the original source language that you can use to create the PO files for each of the translations you are going to use.

Creating the PO Files Manually

If you want to do this manually—for example you don’t have many translatable text strings like in our example—you can create the PO files by hand in a text editor.

First we will create the original source language English version here:

/app/Locale/eng/LC_MESSAGES/default.po

The default.po file will have this format:

msgid "ID"
msgstr "STRING"

Where msgid is the ID within the __() function; msgstr is the localized translation that should appear as output.

Our full English source PO file will look like this:

msgid "Send"
msgstr "Send"

msgid "Reply"
msgstr "Reply"

msgid "Forward"
msgstr "Forward"

msgid "Delete"
msgstr "Delete"

To create the Japanese localized version, we copy the English PO file to the Japanese directory, and then replace the English strings in the msgstr field with the Japanese translations. (If you have a large application being localized into dozens of languages, it is at this point that you send the PO files to a language service provider to translate the localized strings.)

Our localized PO file with Japanese translations will go here:

/app/Locale/jpn/LC_MESSAGES/default.po

Our final localized Japanese PO file will look like this.

msgid "Send"
msgstr "送信"

msgid "Reply"
msgstr "返信"

msgid "Forward"
msgstr "転送"

msgid "Delete"
msgstr "削除"

That’s it. The translations for English and Japanese will display appropriately for the proper locale. If you want to add translations for other languages, you do the same process and put the new PO file in the directory that corresponds with that language code. CakePHP uses the ISO 639-2 standard for naming locales. Follow that standard for naming your localized directories. Make sure you save these files at UTF-8.

Detecting and Changing Languages and Locales

Having the translations ready is nice, but you still have to detect and change to the proper language in your PHP code to get the translations to appear. Detecting the user’s language is tricky. You could do it in JavaScript or try to use something like the following PECL function:

locale_accept_from_http($_SERVER['HTTP_ACCEPT_LANGUAGE']);

In either case, you have to trust the user agent to report back the right language.

Another option is to simply have a button or menu for the user to select their language. Flag icons usually work well for this. Then you can set the language manually. In CakePHP it is easy to do with the CakeSession class:

CakeSession::write('Config.language', 'jpn);

Finally, you also have to set the locale in PHP. Since you already know this from however you determined the language above, you can use PHP’s setlocale function to do this. This is important for localization of date, time, money, and numeric separator formats among others.

setlocale("LC_ALL", "ja_JP.utf8");

That’s all there is to localizing a CakePHP application with proper translations and other locale-specific customizations.

Setup Procedure

To start, select Dash home from the Unity Launcher.

From the Dash home, select More Apps.

From the Installed menu area, select See more results.

Scroll down and select Language Support.

On the Language tab of the Language Support screen, press Install / Remove Languages…

On the Installed Languages screen, scroll down to Japanese and check Installed, and then press Apply Changes.

Enter your password on the Authenticate screen.

It will take a few moments to download and install the Japanese IME packages.

Back on the Language Support screen, select ibus for the Keyboard input method system, and then press Close.

Once again select Dash home from the Unity Launcher.

From the Dash home, select More Apps.

From the Installed menu area, select See more results.

Scroll down and select Keyboard Input Methods.

You may get a pop up message saying Keyboard Input Methods (IBus Daemon) has not been started. Do you want to start it now? Select Yes.

On the Input Method tab of the Ibus Preferences screen, press Select an input method and select Japanese → Anthy.

Press Add and then press Close.

The Ibus keyboard icon will now display on the top panel.

Open up any application with a text box such as Tomboy Notes and place the cursor in the text box.

Press the Ibus keyboard icon on the tap panel and select Japanese-Anthy.

The Ibus keyboard icon will now change to the Anthy Aち icon.

That’s it. You can now type in Japanese in Ubuntu 11.10. 簡単にできますね。

The biggest benefit a translation memory can bring you is providing you with a 100% match and eliminating any translation work for that sentence. Best practices say you should always proofread your translations, even if it is from a 100% TM match—although it is hardly ever done. With Japanese, however, you check your 100% matches because the translations may not be accurate for reasons we will discuss.

Plurals

Japanese does not have different singular and plural forms of nouns the same way English does. There are specific instances where a plural-like form is used, but these are the exception rather than the norm. Let’s look at a simple example:

ねじを取り外す。

You could translate this sentence two different ways:

• Remove the screw.
• Remove the screws.

Which is correct? Well, that depends on how many screws there are. In Japanese, this one sentence covers both instances. Suppose your translation memory had only this translation pair in the database:

• JA: ねじを取り外す。
• EN: Remove the screw.

If the sentence you are currently translating matches the Japanese, but in this present context there are multiple screws, the matching 100% translation is not correct.

This shows why context is important—even more so in Japanese. And if you use software like SDL Trados or some other CAT tool that only provides you with an XLIFF file, you may not have the surrounding images and context to know whether there is one or many screws.

How can we remedy this for the next person that uses our translation memory? We can definitely save a new translation for this sentence, and our TM now looks like this.

• JA: ねじを取り外す。
• EN: Remove the screw.
• EN: Remove the screws.

This is fine for the translator—they can cycle through the multiple translations and select the best one, assuming they know the context to be able to pick the right one. On the other hand, this is not ideal for the person paying for the translation. Generally only 100% matches are done for free or at a greatly reduced price. When duplicate translations exists for a single source segment, SDL Trados and other software will flag this with some sort of penalty so it will be less than a 100% match, often a 99% match, which will cost more to translate.

The best way to deal with this, and the hardest to implement, is for the original Japanese language authors to write with context, knowing that their documents will be the source language for translation. Ideally, there should be multiple versions of the Japanese sentence. For example

• JA: ねじ(１本)を取り外す。
• EN: Remove the screw.

• JA: ねじ(２本)を取り外す。
• EN: Remove the two screws.

• JA: ねじ(３本)を取り外す。
• EN: Remove the three screws.

If the source Japanese text is written with specific contextual information, this solves the problem and there will not be any ambiguous 100% hits in the translation memory. Unfortunately, original source texts are hardly ever written with translation in mind.

Capital and Lowercase Letters

Japanese similarly does not have an equivalent of capital and lowercase letters. A hiragana is a hiragana and a kanji character is a kanji character. In English, usually only the first word of a sentence is capitalized. This is called sentence capitalization. However, titles, headings, etc. have all the major words capitalized. This is called heading capitalization.

Japanese will have the same exactly sentence whether it is a title/heading sentence or it is a normal sentence in the text body. English will have two variations. One with heading capitalization, and one with sentence capitalization. Similar to the ambiguity with plurals, we have ambiguity with capitalization. If we only have the heading capitalization style sentence in the translation memory, that will hit a 100% match when the same Japanese sentence appears in the text body, but the corresponding English 100% match will have the wrong capitalization.

Unlike the pluralization problem, there is no clear fix to avoid the capitalization problem. There is no simple and obvious way we could rewrite the original Japanese text to have multiple variations for heading and sentence style contexts. In these instances, it is important to verify all 100% matches in the translation memory for the proper context.

Sentences with No Subject or Object

This is something uniquely Japanese: sentences with no subject. This is completely normal in Japanese—and absolutely unheard of in English. Sentences can also have direct object verbs with no object whatsoever. There is nothing wrong with sentences without subjects or objects in Japanese. The problem, however, is when translating these sentences. It is difficult without proper context. Now, consider translating with a translation memory, and you can begin to understand the complexity of the situation.

Consider how you would translate this Japanese sentence:

終わったら、取り外す。

This sentence has no subject and no object. In context it is probably clear what the meaning is, but by itself it is all sorts of vague. Let’s imagine two completely different, but totally reasonable translations for this sentence:

• When it’s done, remove it.
• When you are finished, take apart the pieces

Both of these are reasonable translations in two completely different contexts. However if the first English translation is registered in the translation memory and it came up as a 100% match, it would be totally wrong if the context were the second sentence.

Context is everything when translating ambiguous Japanese sentences. But a translation memory does not preserve that context. Even if you know what came before and what comes afterwards, that still may not be enough to know the full context of the original Japanese meaning. Even though you are getting a 100% match in the translation memory, instead of being just a little wrong such as singular/plural or capitalization mistakes, it may be completely wrong in terms of meaning!

Same Words Written Differently

In English we have many words and expressions that have the same meaning, and therefore, we can say the same thing many different ways. But Japanese takes this a step further: you can write the same word many different ways!

For example, the word screw could be written: ねじ、ネジ、ﾈｼﾞ、螺子. That’s four different ways to write the same word.

Another example, the word install could be written: 取り付ける、取付ける、取りつける、とりつける. Again, that is four different ways to write the same word, and we didn’t even consider other forms such as です・ます調 or 敬語.

Now, take these two words and construct the same sentence, and look at the number of possibilities you have to say the same exact thing with the same exact words, only written differently.

• ねじを取り付ける。
• ねじを取付ける。
• ねじを取りつける。
• ねじをとりつける。
• ネジを取り付ける。
• ネジを取付ける。
• ネジを取りつける。
• ネジをとりつける。
• ﾈｼﾞを取り付ける。
• ﾈｼﾞを取付ける。
• ﾈｼﾞを取りつける。
• ﾈｼﾞをとりつける。
• ﾈ螺子を取り付ける。
• ﾈ螺子を取付ける。
• ﾈ螺子を取りつける。
• ﾈ螺子をとりつける。

That is 16 different possibilities! Now imagine your translation memory only has one of these variations registered in the database. When you come across the exact same sentence, but only written differently, you will not get a 100% match, even though you have basically the exact same sentence in one form or another right in front of you. And this sentence is so short, you might not even get any match at all if the hit percentage is set high.

Author variation and style guides conformance are very important in the original source language to prevent these kinds of problems. This is big issue in itself that I’ll take up in another article.

Japanese is very different from English, and when translating, you have to take into greater account the textual context and other issues. And this becomes even more so when dealing with a translation memory containing Japanese as a source language.

Translation memory software such as SDL Trados is very useful, and can be used to great benefit even in Japanese. However, you must be aware of these kinds of issues and double check all of your translations, especially your 100% matches.