Archive for December, 2011

Kanji Usage Count Concordance in PHP

Wednesday, December 7th, 2011

A concordance is a list of all words used in a document, Web site, or publication, and some additional useful information about those words. A useful concordance in translation and localization is a list of the most frequently used words. This can be used to identify important terms that should be picked up for a glossary. It can also be used by students of a foreign language to identify important vocabulary words they should dedicate time to study.

Students of Japanese often wonder what kanji they should learn. It can be hard to identify what kanji is most important. And even between subject matters what kanji is more important will differ.

To help with this, I’ve created a kanji concordance application in PHP to create a list of kanji and their usage counts in descending order.

Example

If you had this Japanese text:

私の名前はマークです。私はテキサス大学を卒業しました。すしが大好きです。

The kanji concordance would generate a list that looked like this:

2 私
2 大
1 名
1 前
1 学
1 卒
1 業
1 好

The kanji 私 and 大 are both used twice, so they are at the top of the list with the number 2 for the usage count. The rest of the kanji are used once and show a usage count of 1.

Kanji Concordance Code Explanation

The first thing we do in PHP is set the language locale with the setlocale() function. This is always good practice when dealing with language-related applications.

setlocale(LC_ALL, "ja_JP.utf8");

The LC_ALL parameter sets the locale for all categories, and the ja_JP.utf8 parameter sets the language and locale to Japanese/Japan in Unicode UTF-8.

Next, we will need some string of Japanese text that we want to examine and create our kanji concordance from. In our simple example we will use a hard-coded string. But in a real application we would probably dynamically input the string from some source.

$string = "私の名前はマークです。私はテキサス大学を卒業しました。私はすしが大好きです。
           私は漢字が好きです。";

Once we have our input string, we need to strip it of everything but the kanji, since that is all we are interested in. Japanese text can have hiragana, katakana, English characters, and various punctuation. If we remove all of those, we’ll be left with just the kanji. We will define a regular expression to match these unwanted characters, and then replace them with nothing.

$pattern = "/[a-zA-Z0-90-9あ-んア-ンー。、?!<>: 「」(){}≪≫〈〉《》【】
            『』〔〕[]・\n\r\t\s\(\) ]/u";

This regular expression pattern is fairly straight forward. a-zA-Z0-9 matches the English alphanumeric characters. We also match the double-byte numbers with 0-9. あ-ん will match all the hiragana, and ア-ン will match all of the katakana. Finally, we match the various punctuation marks and other special characters we can expect to find. The u at the end of the pattern is a pattern modifier that tells PHP that this pattern is Unicode UTF-8. I’ve probably left out some punctuation characters but for our example purposes this will do. (We can actually do a much simplier regex than this. For a throrough discussion of regular expressions for Japanese text, see this post on Japanese regex.)

We will use the regular expression search and replace function preg_replace() to match our input string against the regex pattern to remove the unwanted characters.

$kanjiString = preg_replace($pattern, "", $string);

The first parameter, $pattern, is the regular expression pattern to match against. The second paramter “” is an empty string that we use to replace the regex matches. We match an unwanted non-kanji character and replace it with nothing—in other words, we delete it. The last parameter $string is the input string of Japanese to match against the regex pattern and remove everything but the kanji.

The variable $kanjiString now contains only the kanji characters from our original input string.

// $kanjiString = "私名前私大学卒業私大好私漢字好";

Our next step is to split up all the kanji characters and insert them into an array. We will do this in one step with the split by regular expression function preg_split().

$kanjiArray = preg_split("//u", $kanjiString, -1, PREG_SPLIT_NO_EMPTY);

The first parameter “//u” is a regular expression that will match everything, and the u pattern modifer argument puts it in Unicode match mode. The second parameter $kanjiString is the input string to match against the regular expression. The third parameter -1 is the limit parameter, and -1 indicates no limit. This means it will parse the entire string. The final parameter is the PREG_SPLIT_NO_EMPTY flag. This flag sets it so only non-empty items will be returned.

Now that we have an array full of individual kanji, we want to count them to get our kanji usage numbers. The array_count_values() function will count all the values of our input array, and return a new array with those values and their usage count.

$countedArray = array_count_values($kanjiArray);

With our array of kanji and their usage counts, we just need to sort them in reverse order with the arsort() function.

arsort($countedArray);

Our counted_array now contains a list of all the kanji used from our input string in order of their usage counts. In other words, we have successfully built a kanji usage count concordance.

The final step is to iterate through the array and display our concordance to the screen. We will do this with a simple foreach loop over our array.

foreach ($countedArray as $kanji => $count) {
   echo "$count $kanji <br/>";
}

Our kanji usage count concordance will display like this:

4 私
2 好
2 大
1 漢
1 字
1 業
1 学
1 名
1 前
1 卒

There we go. We have a list of all the kanji used and in order from most used to least used. As we can see, 私 seems to be a pretty important kanji. Better put it on your list to study.

In this example our Japanese input string was hard coded, but we can easily expand this code to take in input from a file or even screen scrape a Web site and see what their most used kanji are. With a large enough input sample, we can get a pretty good list of kanji and usage counts for our concordance.

PHP Source Code

Here is the full source code for the kanji count usage concordance in PHP that we built.

<?php
setlocale(LC_ALL, "ja_JP.utf8");

$string = "私の名前はマークです。私はテキサス大学を卒業しました。私はすしが大好きです。
           私は漢字が好きです。";

$pattern = "/[a-zA-Z0-90-9あ-んア-ンー。、?!<>: 「」(){}≪≫〈〉《》【】
            『』〔〕[]・\n\r\t\s\(\) ]/u";
$kanjiString = preg_replace($pattern, "", $string);

$kanjiArray = preg_split("//u", $kanjiString, -1, PREG_SPLIT_NO_EMPTY);

$countedArray = array_count_values($kanjiArray);
arsort($countedArray);

foreach ($countedArray as $kanji => $count) {
   echo "$count $kanji <br/>";
}
?>

Japanese Input on OpenSUSE Linux 12.1 (KDE 4.7)

Tuesday, December 6th, 2011

Setting up Japanese input IME (日本語入力方法) on openSUSE Linux 12.1 is not difficult, but it requires a little know-how of what packages need to be installed. It only takes a few minutes to download all the files and get it set up. Once installed and configured, you will be able to type in Japanese in your Linux applications. If you’ve used the previous 11.4 version of openSUSE, it’s exactly the same, although some icons look a little different.

Prerequisites

  • YaST software repositories are configured properly.

Setup Procedure

Click on the Kickoff Application Launcher.

On the Computer tab, click Install/Remove Software.

On the Search tab, search for anthy.

In the search results window showing the matching packages, select the anthy and ibus-anthy packages.

Press the Accept button on the bottom right of the window.

YaST will now download, install, and configure the anthy packages.

Do the same for ibus. Open Install/Remove Software, search for ibus, and select the package for ibus. Press Accept to install.

Click on the Kickoff Application Launcher, and from the Leave tab, click Restart to restart openSUSE with the new configuration.

 

After restarting, log back in.

You will now have the IBus input method framework keyboard icon in the bottom panel.

Right click the IBus input method framework keyboard icon and click on Preferences.

On the Input Method tab, select Japanese → Anthy from the dropdown menu.

Press the Add button to add Japanese Anthy input method, and then press Close.

Open up a text editor or any application with a text input window, and click on the IBus input method framework icon and select Japanese – Anthy.

The IBus input method framework keyboard icon will change to the Anthy Aち icon.

You can now type in Japanese.

Click the Anthy Aち icon to select between the various Japanese input modes.

That’s it. Setting up Japanese input on openSUSE 12.1 is not very difficult. When you try to type Japanese, make sure the cursor is in a text box in an application, or you may get an error saying No input window. 日本語入力方法を楽しんでください。

Japanese Input on Linux Mint 12 Lisa

Saturday, December 3rd, 2011

This tutorial will show you how to set up and install Japanese input method IME (日本語入力方法) on Linux Mint 12 Lisa so you can type in Japanese. Linux Mint is quickly becoming one of the more popular Linux distributions. Linux Mint 12 comes in a Gnome 2 and Gnome 3 variety. This tutorial works for either version, however, the menus look a little different in Gnome 2.

Linux Mint 12 Japanese IME Setup Procedure

Click on the Mint Menu and navigate to Other → Software Manager.

In the Software Manager, search for ibus.

Select ibus.

Click Install.

In the Authentication Required dialog box, enter your system password and press Authenticate.

Software Manager will now download and install IBus in the background.

While IBus is installing, search for anthy.

Select ibus-anthy and click Install.

In the Authentication Required dialog box, enter your system password and press Authenticate.

Software Manager will now download and install ibus-anthy in the background.

When the activity bar on the bottom shows 0 ongoing actions, installation is complete.

Close Software Manager.

From the Mint Menu, navigate to System Tools → System Settings.

Open Language Support.

Note: If language support was not installed during the Mint install process you may get a pop up dialog indicating that the language support is not installed completely. In that case, select Install to install the language support. In the Authentication Required dialog box, enter your system password and press Authenticate. The Applying changes screen will display and show the installation progress. When the language support has been fully installed, the Language Support screen will display.

On the Language Support screen, select Install / Remove Languages….

Scroll down and check Japanese, and then press Apply Changes.

On the Language Support screen, press the Keyboard input method system: drop down and select ibus.

Then press Close.

Click on the Mint Menu and select System Tools → IBus.

You should now have the little IBus keyboard icon displayed somewhere on the right side of your Gnome top panel.

Click on the IBus keyboard icon and select Preferences.

On the IBus Preferences screen, go to the Input Method tab.

Press the Select an input method dropdown and select JapaneseAnthy.

Press Add on the IBus Preferences screen to add the Anthy Japanese input method.

Open a text application like Text Editor. While the cursor is in the text field, press the Ibus keyboard icon in the top panel and select Japanese – Anthy.

The Japanese Anthy toolbar should appear and you can now type in Japanese. Place the cursor in a text input application like Text Edit and try to type in Japanese.

That’s all there is to it. Linux Mint is known to be a very easy to use distribution, but it takes quite a few more steps to install Japanese input than the latest versions of Ubuntu or Fedora.

Note: I had issues with the Anthy toolbar not appearing and instead showed this icon which usually means no input window found. But, I could still type in Japanese in this mode, so no worries if this happens to you.

Note: I had trouble when trying to add Japanese on the Install / Remove Languages screen. It worked fine in the Gnome 2 version of Mint 12 I installed in a virtual machine, but it gave a Software database is broken error message in the Gnome 3 version of Mint 12 I installed on a physical laptop. I tried reinstalling twice but I kept getting the same problem. It may have been a problem with the laptop because I also had issues when trying to install drivers for the wireless card. I had no issues with Japanese input on Mint in a VM.