Character Encoding 101

August 12th, 2008 Amir Posted in Guides, Website tips |

127 Characters or less

European characters can also be encoded with a single byte (8 bits) per character. Older version of Windows used to do that. The mapping between the character code and its graphical representation is called the code page. If your application doesn’t support Unicode characters, it can still be localized, but would run properly only in one language. That’s because the code page on a PC can only be set to one value at a time.

More than 127 characters

Supposing you want your application to speak several languages, you’d need more than 127 different characters. This could be the case if you want a single build to support several languages. It could also happen if you just want to support an Asian language that has more than 127 characters to start with (Japanese, Chinese, Korean, etc).

So, what do you do?

The best alternative would be to use Unicode characters. The remaining question is how to encode Unicode strings in the application.

Just a small reminder: your source files are stored byte by byte. If you’ve got Unicode strings, they must be encoded as bytes. Then, during run-time, they would be decoded back to Unicode.

Different encodings for Unicode characters

Try this text document with non-English characters. Open it with Notepad (which comes standard in Windows). You’ll see some texts in different languages (English, Spanish and Hebrew). Now, click on File->Save as…

The save dialog asks you which encoding to use. The default is UTF-8, which is what this file was saved in. If you try switching it to ANSII and saving you’ll get an error message. That’s because the file’s characters don’t fit into the 127 ANSII character set.

Notepad can encode the file as ANSI, UTF-8, Big Endian Unicode (Motorola PowerPC) and Little Endian Unicode (Intel Pentium). Try saving the file using different encoding and open it in a binary or byte editor. You’ll see the same text encoded using different ways.

UTF-8 stands for Unicode Transformation Format (8). It uses sequences of bytes, of different lengths, to encode different characters. The nice thing is that English characters map to themselves. This means that plain text English text is valid UTF-8.

Bonus round - character encoding and markup languages

Supposing you’ve got some Unicode text, encoded as bytes that needs to be included in a markup languages (like HTML, XHTML or XML), what’s the right order of things?

Here’s the answer:

Unicode text <> Byte encoding <> Markup escape <> Markup

So, if you’re going to place Unicode texts in an HTML file, this is what you’ll do:

1. Encode the text as UTF-8.
2. Apply the HTML escape (for example, replace < with &lt; and & with &amp;).
3. Wrap inside the HTML tags

You’ll also place the encoding meta data in the HTML head section, so that the parser (browser) later knows how to decode it:

<meta http-equiv=content-type content="text/html; charset=UTF-8">

When the browser gets the HTML, it will do the same thing, in reverse:

1. Parse the HTML and extract texts
2. Undo the HTML escape
3. Decode the byte string as UTF-8 and produce Unicode text

The same idea applies for other escape techniques. If you’re encoding a URL, URL encoding will be required.

Let’s talk about it

What’s your experience with different character encoding? Any issues you had to tackle?