Code Set Overview

To understand code sets, it is necessary to first understand character sets. A character set is a collection of predefined characters based on the specific needs of one or more languages without regard to the encoding values used to represent the characters. The choice of which code set to use depends on the user's data processing requirements. A particular character set can be encoded using different encoding schemes. For example, the ASCII character set defines the set of characters found in the English language. The Japanese Industrial Standard (JIS) character set defines the set of characters used in the Japanese language. Both the English and Japanese character sets can be encoded using different code sets.

The ISO2022 standard defines a coded character set as a set of precise rules that defines a character set and the one-to-one relationship between each character and it's bit pattern. A code set defines the bit patterns that the system uses to identify characters.

A code page is similar to a code set with the limitation that a code-page specification is based on a 16-column by 16-row matrix. The intersection of each column and row defines a coded character.

The following code sets are supported:

• Support for industry-standard code sets is provided. The ISO8859 family of code sets provides a range of single-byte code set support that includes Latin-1, Latin-2, Cyrillic, Arabic, Greek, Hebrew, and Turkish countries. The IBM-eucJP code set is the industry-standard code set used to support the Japanese locale. The IBM-eucKR code set is the industry-standard code set used to support Korean countries. The IBM-eucTW code set is the industry-standard code set used to support Traditional Chinese countries. The IBM-eucCN code set is the industry-standard code set used to support countries using Simplified Chinese. The UTF-8 code set is a Universal Transformation Format of Unicode/ISO10646 used to support multiple languages at once (including Simplified Chinese, Traditional Chinese, and Chinese characters used in Japanese and Korean).

• ISO8859-15 standard codeset is a replacement standard for the existing ISO8859-1 codeset that is currently in use by the western European locales, the United States, and Canada. The need for a new codeset came about as a result of the introduction of the Euro currency unit, and the need for European countries to be able to do business transactions using the Euro. In addition, ISO8859-15 contains 7 additional characters for the French and Finnish languages.

• Support is also provided for the personal computer (PC) based code sets IBM-850, IBM-856, IBM-943, IBM-932, and IBM-1046. IBM-850 is a single-byte code set used to support Latin-1 countries (U.S., Canada, and Western Europe). IBM-856 is a single-byte code set used to support Hebrew countries. IBM-943 and IBM-932 are multibyte code set used to support the Japanese locale. IBM-1046 is a single-byte code set used to support Arabic countries.

• IBM-1129 is a single-byte code set used to support Vietnamese.

• TIS-620 is a single-byte code set used to support Thai.

• IBM-1124 is a single-byte code set used to support Ukrainian.

• Full Unicode support is provided via the UTF-8 code set for ALL languages and territories supported by AIX. The UTF-8 code set is a Universal Transformation Format of Unicode/ISO10646 used to support multiple languages at once. The UTF-8 code set provides the most complete solution for use in environments where multiple languages and alphabets must be processed. The Unicode/UTF-8 codeset also provides full support for the common European currency (Euro).

• IBM-1252 codeset support is provided as a compatibility option for users who require a single byte codeset environment containing the Euro currency symbol. The structure of the IBM-1252 codeset is identical to the industry standard codeset ISO8859-1, except that additional graphic characters are added in the ISO control characters range from 0x80 through 0x9F. The Euro currency symbol is located at hexadecimal value 0x80 in the IBM-1252 codeset.

For more information on code sets, refer to these articles:

• ASCII Characters
• Code Set Strategy
• Code Set Structure
• ISO Code Sets
• IBM PC Code Sets

ASCII Characters

The following sections describe the 7-bit ASCII characters.

ASCII Characters in the Unique Code-Point Range

The following table lists the ASCII characters in the unique code-point range. These characters are in the range 0x00 through 0x3F.

Other ASCII Characters

The following table lists the 7-bit ASCII characters that are not in the unique code-point range. These characters are in the range 0x40 through 0x7F.

Code Set Strategy

Prior to Version 3.2, IBM-850 and IBM-932 were the only supported code sets. Version 3.2 enhanced the system code set support by adding code sets that are based on International Organization for Standardization (ISO) and industry-standard code sets. It is suggested that users use to these new code sets. The ultimate goal is to provide industry-standard code sets that satisfy the data processing needs of users.

Support for the IBM-850 codeset will be removed in future releases of AIX. Users who are currently using IBM-850 based locales should strongly consider use of the corresponding industry standard ISO8859-1 based locale. For example, users of the French IBM-850 locale (Fr_FR) should use the French ISO8859-1 locale (fr_FR).

Each locale in the system defines which code set it uses and how the characters within the code set are manipulated. Because multiple locales can be installed on the system, multiple code sets can be used by different users on the system. While the system can be configured with locales using different code sets, all system utilities assume that the system is running under a single code set.

Most commands have no knowledge of the underlying code set being used by the locale. The knowledge of code sets is hidden by the code set-independent library subroutines (NLS library), which pass information to the code set-dependent subroutines.

Because many programs rely on ASCII, all code sets include the 7-bit ASCII code set as a proper subset. Since the 7-bit ASCII code set is common to all supported code sets, its characters are sometimes referred to as the portable character set.

The 7-bit ASCII code set is based on the ISO646 definition and contains the control characters, punctuation characters, digits (0-9), and the English alphabet in uppercase and lowercase.

Code Set Structure

This code set table shows the general format for all code sets:

Each code set is divided into two principal areas:

The first two columns of each code set are reserved by International Organization for Standardization (ISO) standards for control characters. The terms C0 and C1 are used to denote the control characters for the Graphic Left and Graphic Right areas, respectively.

Note: The IBM PC code sets use the C1 control area to encode graphic characters.

The remaining 6 columns are used to encode graphic characters. Graphic characters are considered to be printable characters, while the control characters are used by devices and applications to indicate some special function.

Control Characters

Based on the ISO definition, a control character initiates, modifies, or stops a control operation. A control character is not a graphic character, but can have graphic representation in some instances. The control characters in the control points table are present in all supported code sets and the encoded values of the control characters are consistent throughout the code sets.

Graphic Characters

Each code set can be considered to be divided into one or more character sets, such that each character is given a unique coded value. The ISO standard reserves six columns for encoding characters and does not allow graphic characters to be encoded in the control character columns.

The internationalization of AIX is based on the assumption that all code sets can be divided into any number of character sets.

Single-Byte and Multibyte Code Sets

Code sets that use all 8 bits of a byte can support European, Middle Eastern, and other alphabetic languages. Such code sets are called single-byte code sets. This provides a limit of encoding 191 characters, not including control characters.

Languages that require more than 191 characters use a mixture of single-byte characters (8 bits) and multibyte characters (more than 8 bits). The system is capable of supporting any number of bits to encode a character.

ISO Code Sets

These code sets are based on definitions set by the International Organization for Standardization (ISO).

• ISO646-IRV
• ISO8859-1
• ISO8859-2
• ISO8859-5
• ISO8859-6
• ISO8859-7
• ISO8859-8
• ISO8859-9
• ISO8859-15
• IBM-eucJP
• IBM-eucTW
• Big5
• IBM-eucKR
• ISO10646 UCS-2 (Unicode)
• UTF-8 (UCS Transformation Format)

ISO646-IRV

The ISO646-IRV code set defines the code set used for information processing based on a 7-bit encoding. The character set associated with this code set is derived from the ASCII characters.

ISO8859 Family

ISO8859 is a family of single-byte encodings based on and compatible with other ISO, American National Standards Institute (ANSI), and European Computer Manufacturer's Association (ECMA) code extension techniques. The ISO8859 encoding defines a family of code sets with each member containing its own unique character sets. The 7-bit ASCII code set is a proper subset of each of the code sets in the ISO8859 family.

While the ASCII code set defines an order for the English alphabet, the Graphic Right (GR) characters are not ordered according to any specific language. The language-specific ordering is defined by the locale.

Each code set includes the ASCII character set plus its own unique character set. The ISO8859 encoding table shows the ISO8859 general encoding scheme.

Code Set ISO8859-1

The figure summarizes the available symbols and shows the layout of Code Set ISO8859-1.

Code Set ISO8859-2

The figure summarizes the available symbols and shows the layout of Code Set ISO8859-2.

Code Set ISO8859-5

The figure summarizes the available symbols and shows the layout of Code Set ISO8859-5.

Code Set ISO8859-6

The figure summarizes the available symbols and shows the layout of Code Set ISO8859-6.

Code Set ISO8859-7

The figure summarizes the available symbols and layout of Code Set ISO8859-7. This code set is made up of an ASCII character set plus its own unique character set.

Code Set ISO8859-8

The figure summarizes the available symbols and shows the layout of Code Set ISO8859-8.

Code Set ISO8859-9

The figure summarizes the available symbols and layout of Code Set ISO8859-9. This code set is made up of an ASCII character set plus its own unique character set.

Code Set ISO8859-15

The figure summarizes the available symbols and shows the layout of Code Set ISO8859-15.

Extended UNIX Code (EUC) Encoding Scheme

The EUC encoding scheme defines a set of encoding rules that can support one to four character sets. The encoding rules are based on the ISO2022 definition for the encoding of 7-bit and 8-bit data. The EUC encoding scheme uses control characters to identify some of the character sets. The EUC encoding table shows the basic structure of all EUC encoding.

The term EUC denotes these general encoding rules. A code set based on EUC conforms to the EUC encoding rules but also identifies the specific character sets associated with the specific instances. For example, IBM-eucJP for Japanese refers to the encoding of the Japanese Industrial Standard characters according to the EUC encoding rules.

The first set (CS0) always contains an ISO646 character set. All of the other sets must have the most significant bit (MSB) set to 1 and can use any number of bytes to encode the characters. In addition, all characters within a set must have:

• Same number of bytes to encode all characters
• Same column display width (number of columns on a fixed-width terminal)

All characters in the third set (CS2) are always preceded with the control character SS2 (single-shift 2, 0x8e). Code sets that conform to EUC do not use the SS2 control character other than to identify the third set.

All characters in the fourth set (CS3) are always preceded with the control character SS3 (single-shift 3, 0x8f). Code sets that conform to EUC do not use the SS3 control character other than to identify the fourth set.

IBM-eucJP

The EUC for Japanese is an encoding consisting of single-byte and multibyte characters. The encoding is based on ISO2022, Japanese Industrial Standard (JIS), and EUC definitions.

The IBM-eucJP code set consists of the following character sets:

The IBM-eucJP code set is also capable of supporting:

The IBM-eucJP code set is encoded as follows:

• CS0 maps JISX0201 Graphic Left characters starting at the 0x00 position.
• CS1 maps the JISX0208 character set starting at the 0xa1xa1 position.

The positions 0xf5a1 through 0xfefe (940 characters) in CS1 are reserved as primary user definable character areas.

• CS2 maps the JISX0201 Graphic Right starting at the 0x8ea1 position.
• CS3 is capable of mapping JISX0212 starting at the 0x8fa1a1 position.

The positions 0x8ff5a1 through 0x8ffefe in CS3 (940 characters) are reserved as secondary user definable character areas.

• The positions 0x8feea1 through 0x8ff4fe in CS3 (658 characters) are reserved for future system use. Therefore, users should not use this area.

IBM-eucCN

The EUC for the Simplified Chinese language is an encoding consisting of characters that contain 1 or 2 bytes. The EUC encoding is based on ISO2022, GB2312 as defined by the People's Republic of China, and multibyte character definitions unique to the manufacturer.

The current GB2312 defines 6,763 Simplified Chinese characters and 682 symbols. The IBM-eucCN is based upon a concept of one plane containing up to 94x94 characters. The encoding values of these characters range from 0xa1a1 to 0xfefe.

The GB2312 is mapped into the CS1 of EUC. Specifically, the IBM-eucCN consists of the following character sets:

a2a1 -- a2b0    a1e3 -- a2e4  a1ef -- a2f0
a2fd -- a1fe    a4f4 -- a4fe  a5f7 -- a5fe
a6b9 -- a6c0    a6d9 -- a6fe  a7c2 -- a7d0
a7f2 -- a7fe    a8bb -- a8c4  a8ea -- a9a3
a9f0 -- affe    a7fa -- d7fe  f8a1 -- fedf

GBK

GBK stands for Guo (national) Biao (Standard) Kuo (Extension). GBK expands the national "Industry GB" definition to contain all 20, 902 Han Characters defined in Unicode and additional DBCS symbols defined in Big-5 code (Traditional Chinese PC defacto standard). Restated, GBK defined all DBCS characters and symbols in use on both sides of the Taiwan Strait. Currently, GBK is a Normative Annex of GB13000 (PRC Unicode Standard) and is being positioned as an interim step for migration to Unicode.

IBM-eucTW

The EUC for the Traditional Chinese language is an encoding consisting of characters that contain 1, 2 and 4 bytes. The EUC encoding is based on ISO2022, the Chinese National Standard (CNS) as defined by the Republic of China and multibyte character definitions unique to the manufacturer.

The current CNS defines 13,501 Chinese characters and 684 symbols. The IBM-eucTW is based upon a concept of 15 planes, each containing up to 8836 (94x94) characters. The encoding values of these characters range from 0xa1a1 to 0xfefe. Characters have presently been defined for only 4 of the planes, with the other planes being reserved for future expansion.

The 15 planes are mapped into the CS1 and CS2 of EUC, with the CS2 of EUC consisting of 14 planes. Specifically, the IBM-eucTW consists of the following character sets:

Planes 3-11 are expected to occupy positions 0x8ea3xxxx to 0x8eabxxxx. Planes 14-15 are expected to occupy positions 0x8eaexxxx to 0x8eafxxxx.

Big5

The Traditional Chinese big5 locale, Zh_TW, code set is the most commonly used code set in the PC field which is used to support countries using Traditional Chinese.

Big5 code set defines 13056 characters and 1004 symbols. It includes 684 symbols in CNS11643.192, as well as 325 IBM unique symbols.

Code Range for Big5 Locale:

IBM-eucKR

The EUC for the Korean language is an encoding consisting of single-byte and multibyte characters. The encoding is based on ISO2022, Korean Standard Code set and EUC definitions.

The Korean EUC code set consists of two main character groups:

• ASCII (English)
• Hangul (Korean characters)

The Hangul code set includes Hangul and Hanja (Chinese) characters. One Hangul character can be comprised of several consonants and vowels. However, most Hangul words can be expressed in Hanja. Each Hanja character has its own meaning and is more specific than Hangul.

The IBM-eucKR consists of the following character sets:

IBM PC Code Sets

IBM PC code sets are the code sets originally supported on the IBM PC systems and AIX. The IBM PC code sets assign graphic characters to the Control One (C1) control area. Applications that depend on these control characters can not support these code sets.

The ASCII characters are encoded with the most significant bit (MSB) zero in positions 0x20-0x7e. The extended Latin 1 combined with the IBM PC unique characters sets make up the extended set of characters which are encoded in positions 0x80-0xfe. The personal computer encoding table shows the location of the control, ASCII, and extended characters for the IBM-850 code set.

The IBM PC unique character set includes the following:

IBM-850

The figure summarizes the available symbols and shows the layout of Code Set IBM-850.

IBM-856

The figure summarizes the available symbols and shows the layout of Code Set IBM-856.

IBM-921

The figure summarizes the available symbols and shows the layout of Code Set IBM-921.

IBM-922

The figure summarizes the available symbols and shows the layout of Code Set IBM-922.

IBM-943 and IBM-932

Each of the Japanese IBM PC code sets are an encoding consisting of single-byte and multibyte coded characters. The encoding is based on the IBM PC code set and places the JIS characters in shifted positions. This is referred to as Shift-JIS or SJIS.

IBM-943 is newer code set for the Japanese locale than IBM-932. IBM-943 is a compatible code set for the Japanese Microsoft Windows environment. This code set is known as 1983 ordered shift-JIS. The difference between IBM-932 and IBM-943 is as follows

• Old JIS sequence (1978 ordered) is applied for IBM-932 while new JIS sequence (1983 ordered) is applied for IBM-943.
• NEC selected characters are added to IBM-943.
• NEC's IBM selected characters are added to IBM-943.

The IBM-932 consists of the following character sets:

The IBM-943 consists of the following character sets:

The first byte of each character is used to determine the number of bytes for a given character. The values 0x20-0x7e and 0xa1-oxdf are used to encode JISX0201 characters, with exceptions. The positions 0x81-0x9f and 0xe0-0xfc are reserved for use as the first byte of a multibyte character. The JISX0208 characters are mapped to the multibyte values starting at 0x8140. The second byte of a multibyte character can have any value. The Shift-JIS table shows where these characters are located on the code set.

The figure shows the DBCS part of IBM-943.

The figure shows the DBCS part of IBM-932.

IBM-1046

The figure summarizes the available symbols and shows the layout of Code Set IBM-1046.

IBM-1124

The figure summarizes the available symbols and shows the layout of Code Set IBM-1124.

IBM-1129

The figure summarizes the available symbols and shows the layout of Code Set IBM-1129.

TIS-620

The figure summarizes the available symbols and shows the layout of Code Set TIS-620.

UCS-2 and UTF-8

AIX provides a set of codesets that address the needs of a particular language or a language group. None of the codesets represented in the ISO8859 family of codesets, the PC codesets, nor the Extended Unix Code (EUC) codesets allow the mixing of characters from different scripts. With ISO8859-1, you can mix and represent the Latin 1 characters (languages principally spoken in the U.S., Canada, Western Europe, and Latin America). ISO8859-2 covers Eastern European languages; ISO8859-5 covers Cyrillic, ISO8859-6 covers Arabic, ISO8859-7 covers Greek, ISO8859-8 covers Hebrew, ISO8859-9 covers Turkish, IBM-eucJP covers Japanese, IBM-eucKR covers Korean, IBM-eucTW covers Simplified Chinese. The point is that none of the above codesets covers all of the languages.

The International Organization for Standardization (ISO) has addressed the limited language coverage by codesets by adopting Unicode as the encoding for the 2-octet form of the ISO10646 Universal Multiple-Octet Coded Character Set (UCS-2). The 32-bit form of ISO10646 is known as UCS-4 for 4-octet form. AIX has adopted the 16-bit form of ISO10646 and uses the standard label "UCS-2" to describe this encoding.

Although UCS-2 is ideal for an internal process code, it is not suitable for encoding plain text on traditional byte-oriented systems, such as AIX. Therefore, the external file code is X/Open's File System Safe UCS Transformation Format (FSS-UTF). This transformation format encoding is also known as UTF-8, and "UTF-8" is the label that is used for this encoding on AIX.

ISO10646 UCS-2 (Unicode)

Universal Coded Character Set (UCS) is the name of the ISO10646 standard that defines a single code for the representation, interchange, processing, storage, entry, and presentation of the written form of all the major languages of the world.

UCS has the following key objectives:

• Improve interoperability between systems in an open environment.
• Simplify development of internationalized products.
• Provide a base for multilingual applications.
• Make more characters available.

ISO10646 defines canonical character codes with a length of 32 bits. This provides code numbers for over 4 billion characters. When used in canonical form to represent text, the coding is referred to as UCS-4 for Universal Coded Character Set 4-byte form.

The code values from 0x0000 through 0xFFFF of ISO 10646 can be represented by a uniform character encoding of 16 bits. When used in this form to represent text, these codes are referred to as UCS-2, for Universal Character Set 2-octet form. This range is also called the Basic Multilingual Plane (BMP) of ISO10646. The standard is arranged so that the most useful characters, covering all major existing standards worldwide, are assigned within this range.

The character code values of UCS-2 are identical to those of the Unicode character encoding standard published by the Unicode Consortium.

UCS-2 defines codes for characters used in all major written languages. In addition to a set of scientific, mathematic, and publishing symbols, UCS-2 covers the following scripts:

• Arabic
• Armenian
• Bengali
• Bopomofo
• Cyrillic
• Devanagari
• Georgian
• Greek
• Gujarati
• Gurmukhi
• Hangul
• Chinese Hanzi
• Hebrew
• Hiragana
• International Phonetic Alphabet (IPA)
• Katakana
• Japanese Kanji
• Kannada
• Korean Hanja
• Laotian
• Latin
• Malayalam
• Oriya
• Tamil
• Teluga
• Thai
• Tibetan

The ability of AIX to display characters in the scripts mentioned above is limited to the availability of fonts. AIX provides bitmap fonts for most of the major languages of the world, as well as a Unicode based scalable TrueType font. Use of this font requires the TrueType font rasterizer for AIX, which is a separately installable feature.

UCS-2 encodes a number of combining characters, also known as non-spacing marks for floating diacritics. These characters are necessary in several scripts including Indic, Thai, Arabic, and Hebrew. The combining characters are used for generating characters in Latin, Cyrillic, and Greek scripts. However, the presence of combining characters creates the possibility for an alternative coding for the same text. Although the coding is unambiguous and data integrity is preserved, the processing of text that contains combining characters is more complex. To provide conformance for applications that choose not to deal with the combining characters, ISO10646 defines the following three implementation levels:

UTF-8 (UCS Transformation Format)

X/Open has developed a transformation format for UCS designed for use in existing file systems. The original name of this transformation is FSS-UTF, but it is expected to be registered by ISO as UTF-8. UTF-8 is expected to become the standard transformation method, for situations where UCS is not practical. The intent is that UCS will be the process code for the transformation format, which is usable as a file code.

UTF-8 has the following properties:

• It is a superset of ASCII, in which the ASCII characters are encoded as single-byte characters with the same numeric value.

• No ASCII code values occur in multibyte characters, other than those which represent the ASCII characters.

• Conversion to and from UCS is simple and efficient.

• The first byte of a character indicates the number of bytes to follow in the multibyte character sequence and cannot occur anywhere else in the sequence.

The UTF-8 encodes UCS values in the 0 through 0x7FFFFFFF range using multibyte characters with lengths of 1, 2, 3, 4, 5, and 6 bytes. Single-byte characters are reserved for the ASCII characters in the 0 through 0x7f range. These all have the high order bit set to 0. For all character encodings of more than one byte, the initial byte determines the number of bytes used, and the high-order bit in each byte is set. Every byte that does not start with the bit combination of 10xxxxxx, where x represents a bit that may be 0 or 1, is the start of a UCS character sequence.

The UCS value is just the concatenation of the x bits in the multibyte encoding. When there are multiple ways to encode a value (for example, UCS 0), only the shortest encoding is legal.

The following subset of UTF-8 is used to encode UCS-2:

This subset of UTF-8 requires a maximum of three (3) bytes.

Related Information

Low Function Terminal (LFT) Subsystem Overview in AIX Kernel Extensions and Device Support Programming Concepts.

National Language Support Overview for Programming, Locale Overview for Programming, Converters Overview for Programming, Input Method Overview, Keyboard Overview.

National Language Support Overview for System Management, National Language Support Overview for Devices, and Locale Overview for System Management in AIX Version 4.3 System Management Guide: Operating System and Devices.

The iconv command.