To write a program that will compile and run without modification on Linux, Windows, and a variety of other platforms is a matter of abstracting the techniques listed above used by each platform. Linux uses both multi-byte and wide character encodings. Windows uses wide characters however it is important to note that Windows does not support a UTF-8 locale so if Unicode is desired wide character strings are the only option. Most other Unix and Unix-like systems support multi-byte strings as well as possibly wide character strings to different degrees. Programs written using the technique described here will still premit using runtime defined codepages using the standard setlocale(3) mechanism.
The Tchar Type
The idea behind this technique is to use a typedef for the character type that resolves to either plain char or wchar_t. In this way the character type identifier does not change in the source code.
#ifdef USE_WCHAR
typedef wchar_t tchar;
#else
typedef unsigned char tchar;
#endif
Abstracting String Functions
In addition to the character type, all functions that operate on it will need to be abstracted with macros that reference tchar rather than char or wchar_t. Consider the strncpy function. It uses the plain char type. Fortunately the major string functions have a wide character equivalent that usually has the same signature but accepts the wchar_t type.
char *strncpy(char *dest, const char *src, size_t n);
wchar_t *wcsncpy(wchar_t *dest, const wchar_t *src, size_t n);
From the above signatures it can be seen that the only difference is the character type. The number, order, and meaning of the parameters are the same. This permits the function to be abstracted with macros as follows:
#ifdef USE_WCHAR
#define tcsncpy wcsncpy
#else
#define tcsncpy strncpy
#endif
To use this function is now a matter of substituting all instances of strncpy or wcsncpy with tcsncpy. Depending on how the program is compiled, code that uses these functions will support wide character or multi-byte strings (but not both at the same time). See the Text Module API Documentation for a complete list of macros in text.h.
There are of course many other functions that operate on strings. Fortunately most standard C library function have wide character versions that are reasonably consistent about identifier names. An identifier that begins with str will likey have a wide character version that begins with wcs. Other functions like vswprintf are not so obvious and depending the the system being used there will certainly be omissions or incompatablities (e.g. the vsnprintf counterpart wide character function is vswprintf without the n even though it accepts an n parameter). If a function does not have a man page or if the compiler issues a warning it does not necessarily mean the function does not exist on your system. For example, with the GNU C library it may be necessary to specify C99 or define _XOPEN_SOURCE=500 to indicate a UNIX98 environment is desired. Check your C library documentation (e.g. /usr/include/features.h). Check the POSIX documentation on the OpenGroup website. On my RedHat Linux 7.3 system the wcstol and several other conversion functions are not documented. It is necessary to specify -std=c99 or define -D_ISOC99_SOURCE with gcc to trigger it to export that symbol.
Variable Width Encodings
Unicode on Unix and Unix-like systems is supported using UTF-8. On Microsoft Windows UTF-16LE is used. As explained previously these are variable width encodings. Each character can occupy a variable number of bytes in memory. The question is; when does this require special processing in your code?
A good example of when UTF-8 string handling requires special hanlding is when each character needs to be examined individually. Consider the example of caseless comparison of two strings. They cannot simply be compared element by element. Each character must be decoded to their wide character value and converted to upper or lowercase for the comparison to be valid. Below is just such a function:
/* Case insensitive comparison of two UTF-8 strings
*/
int
utf8casecmp(const unsigned char *str1, const unsigned char *str1lim,
const unsigned char *str2, const unsigned char *str2lim)
{
int n1, n2;
wchar_t ucs1, ucs2;
int ch1, ch2;
mbstate_t ps1, ps2;
memset(&ps1, 0, sizeof(ps1));
memset(&ps2, 0, sizeof(ps2));
while (str1 < str1lim && str2 < str2lim) {
if ((*str1 & 0x80) && (*str2 & 0x80)) { /* both multibyte */
if ((n1 = mbrtowc(&ucs1, str1, str1lim - str1, &ps1)) < 0 ||
(n2 = mbrtowc(&ucs2, str2, str2lim - str2, &ps2)) < 0) {
PMNO(errno);
return -1;
}
if (ucs1 != ucs2 && (ucs1 = towupper(ucs1)) != (ucs2 = towupper(ucs2))) {
return ucs1 < ucs2 ? -1 : 1;
}
str1 += n1;
str2 += n2;
} else { /* neither or one multibyte */
ch1 = *str1;
ch2 = *str2;
if (ch1 != ch2 && (ch1 = toupper(ch1)) != (ch2 = toupper(ch2))) {
return ch1 < ch2 ? -1 : 1;
} else if (ch1 == '\0') {
return 0;
}
str1++;
str2++;
}
}
return 0;
}
This is a fairly pathological example. In practice this is probably as difficult as it gets. For example, if the objective is to search for a certain ASCII character such as a space or '\0' termniator, it is not necessary to decode a Unicode value at all. It might even be reasonable to use isspace and similar functions (but probably not ispunct for example). This will require some experimenting and research.
Another example of when using a variable width encoding requires special handling in your code is when calculating the number of bytes from the string required to occupy at most a certin number of dispay positions in a terminal window. In this case it is necessary to convert each character to it's Unicode value and then use the wcwidth(3) function. When the total of values returned by wcwidth(3) equals or exceeds the desired number of columns the number of bytes traversed in the substring is known.