Unicode, UTF等字符编码摘记

两个遵守相同规范的编码标准
unicode 3.0（最新版本5.0）和ISO-10646。从Unicode2.0开始，unicode采用了与ISO 10646-1相同的字库和字码。ISO-10646也叫做UCS (Universal Character Set)。

几个术语：
UTF: Unicode/UCS Transformation Format

UTF-16: 16位编码。基本上是Unicode的双字节编码，额外附加空间用于非常用字符和未来扩充需要（一般很少用到），常用字符在0-0xFFFF，包括扩充空 间的范围是0-0×10FFFF，所以最长编码位数是21位。关于扩充空间，在ISO-10646有相对应的定义。由于它是一个变长码，与CPU字序有关 （例如“汉”字的Unicode编码是6C49。那么写到文件里时，究竟是将6C写在前面，还是将49写在前面？如果将6C写在前面，就是big endian。如果将49写在前面，就是little endian。），最节省空间，所以常作为网络传输的外码。UTF-16是Unicode的preferred encoding。

UTF-8: 由于UTF-16直接就是Unicode编码，没有变换，包含了0×00在编码内，这个在操作系统内（C语言）中有特殊意义（和ASCII不兼容？），会 引起问题，所以有时候需要采用UTF-8编码对Unicode的直接编码做一些变换。UTF-8对ASCII不作变换，进行8位编码，其他字符做变长编 码，每个字符1-3个字节。与CPU字序无关，可以在不同平台之间交流。

UCS-2: 与UTF-16基本一样。

UCS-4: 4字节编码，目前是在UCS-2前加上2个全零的byte。

内码：内码是指操作系统内部的字符编码。早期操作系统的内码是与语言相关的.现在的Windows在内部统一使 用Unicode，然后用代码页适应各种语言,“内码”的概念就比较模糊了。微软一般将缺省代码页指定的编码说成是内码，在特殊的场合也会说自己的内码是 Unicode，例如在GB18030问题的处理上。

字符集：charcterset 字符的集合，例如Unicode是一种字符集。

字符编码：Encoding 如何将二进制数据识别为字符的编码，一种编码表示的字符是有限的，常常一种编码设计为表示一种字符集。例如UTF-8，UTF-16是两种字符编码，它们能够表示Unicode字符集的所有字符。

中国国标编码:

GB 13000: 完全等同于ISO 10646-1/Unicode 2.1, 今后也将随ISO 10646/Unicode的标准更改而同步更改.

GBK: 对GB2312的扩充, 以容纳GB2312字符集范围以外的Unicode 2.1的统一汉字部分, 并且增加了部分unicode中没有的字符.

GB
18030-2000: 基于GB 13000, 作为Unicode 3.0的GBK扩展版本, 覆盖了所有unicode编码,
地位等同于UTF-8, UTF-16, 是一种unicode编码形式. 变长编码, 用单字节/双字节/4字节对字符编码.
GB18030向下兼容GB2312/GBK.
GB 18030是中国所有非手持/嵌入式计算机系统的强制实施标准.

Update (20061114): ISO 8859-1:

ISO/IEC 8859-1，又称Latin-1或“西欧语言”，是国际标准化组织内ISO/IEC 8859的第一个8位字符集。它以ASCII为基础，在空置的0xA0-0xFF的范围内，加入192个字母及符号，借以供使用变音符号的拉丁字母语言使用。

其他：

UCS只是规定如何编码，并没有规定如何传输、保存这个编码。例如“汉”字的UCS编码是6C49，我可
以用4个ascii数字来传输、保存这个编码；也可以用utf-8编码:3个连续的字节E6 B1
89来表示它。关键在于通信双方都要认可。UTF-8、UTF-7、UTF-16都是被广泛接受的方案。UTF-8的一个特别的好处是它与ISO-
8859-1完全兼容。UTF是“UCS Transformation Format”的缩写。

所谓代码页(code page)就是针对一种语言文字的字符编码。例如GBK的code page是CP936，BIG5的code page是CP950，GB2312的code page是CP20936。

　　Windows中有缺省代码页的概念，即缺省用什么编码来解释字符。例如Windows的记事本打开了一个文本文件，里面的内容是字节流：BA、BA、D7、D6。Windows应该去怎么解释它呢？

　
　是按照Unicode编码解释、还是按照GBK解释、还是按照BIG5解释，还是按照ISO8859-1去解释？如果按GBK去解释，就会得到“汉字”
两个字。按照其它编码解释，可能找不到对应的字符，也可能找到错误的字符。所谓“错误”是指与文本作者的本意不符，这时就产生了乱码。

　　答案是Windows按照当前的缺省代码页去解释文本文件里的字节流。缺省代码页可以通过控制面板的区域选项设置。记事本的另存为中有一项ANSI，其实就是按照缺省代码页的编码方法保存。

　　Windows的内码是Unicode，它在技术上可以同时支持多个代码页。只要文件能说明自己使用什么编码，用户又安装了对应的代码页，Windows就能正确显示，例如在HTML文件中就可以指定charset。

趣事：

　　“endian”这个词出自《格列佛游记》。小人国的内战就源于吃鸡蛋时是究竟从大头(Big-Endian)敲开还是从小头(Little-Endian)敲开，由此曾发生过六次叛乱，一个皇帝送了命，另一个丢了王位。

　　我们一般将endian翻译成“字节序”，将big endian和little endian称作“大尾”和“小尾”。

Resources:
1. 对字符编码与Unicode,ISO 10646,UCS,UTF8,UTF16,GBK,GB2312的理解 国际化支持 USENIX.CN - powered by Sinoprise Technology Lab （有比较详细的介绍）

2. 无废话XML

3. 简要解释UCS、UTF、BMP、BOM等名词

4. 中文编码处理(1) －－ 编码与字符集

5. ISO 8859-1

注：文中参考的不全是官方或权威资料，难免有错误，仅作学习用，本人对文中错误不负任何责任，并欢迎改正错误。

XML Processing With Java Overview

There’re basically two ways of processing xml with Java. One is the DOM way, that is tree-structure based way, and the other way is the SAX way that is event-driven stream based way. However, the bad thing is that there’re pros and cons for both ways, and the good thing is that we can use one of them in different situation to meet different needs.

The DOM way
DOM, Document Object Model, is the standard specification released by w3c consortium. It is a tree like structure which represents the structure of a XML document and is what what we often first parse a XML document into before we do any manipulation to it. It is quite intuitive for most programmers to manipulate. With it we can easily get what we want from a XML document, element names, attributes, values of elements, etc. But the price to pay is that before any manipulation we have to read the entire xml document and parse it into a DOM object during which everything must be stored in memory. This is inefficient and sometimes impossible, especially for extremely large documents. By the way, besides DOM, there’re some unofficially object models in use, such as JDOM, XOM, DOM4J and so on.

The SAX way
It is a stream like and event-based way. We can processing a document while we’re reading it. It is a very flexible but more complicated way than the DOM way. It is flexible because the SAX stream can be redirected to other process or document. It is complicated because the event handler (usually the DefaultHandler or ContentHandler) must be first written and then registered with the Parser (alternatively reader, or something like that). And there’re other disadvantages. Because it is processed like a stream, it is impossible to make changes to it or move backward to the data stream. But it is possilbe to make some simple structure (not the data itself) changes by using xsl transformation. In general, the SAX way is much faster than the DOM way.

What make up the “XML Processing”
So-called XML processing or sometimes called parsing consists of several aspects or procedures.
Validation
Data Modification and Retrieve
Transformation
Data Query

Examples

Higher Level Application
What are mentioned above are only those basic aspects about xml processing. Seen from a more global perspective, there’re many other higher level application of xml or xml processing.

Published temporarily and remains further refinement.

Always prototyping your new features that are to be added

Wow, this new feature is awesome. It’s super cool!. I woner if we could add it our product. Ok, I’m gonna do it right now.

But wait…

First develop a prototype with this new feature added. Then see how this feature functions within your product, how it collaborate with other features. If these all are just fine then change your artifacts to add it to your product extensively and “aggressively”. This would be safer.

Never do so-called feasibility analysis just in your mind or just on a paper with your pen. You’re cheating yourself. In this way you would be very excited with your analysis results at most time and say to your colleagues those words written in the first paragraph of this post.

We experienced this cheating-ourself process. And it turns out to be that this new feature actually doesn’t function as we expected but meantime we have to keep those codes modified during adding this new feature for not wasting more time removing them and bringing new risks even though we know they’re useless. This may be a lesson you can learn from.

It is noteworthy that the beta policy of most web2.0 applications goes to extremes in prototyping new features and that developping iteratively is the essence of contemporary software development.

One post of my development diary series…to be continued.

Technorati : beta, programming, software development, software engineering, tech
Del.icio.us : beta, programming, software development, software engineering, tech

Help me with automating KDE environment settings

I have pasted this on some tech forums, but no one seems to be willing to help me. So I would like to also paste it here. Any help or even any response making no sence would be very appreciated.

我刚接触Linux不久，现在碰到要做这个，请各位高手支支招。

主要分为几个部分：
1. Desktop shortcut, background, etc;
2. Kicker (Panel), start menu, custom menu, etc; and
3. Konqueror, Konsole, etc.

由于KDE采用 Cascading Configuration Files的结构 （至少KDE3.1+是这样），有针对所有用户的设置和用户自定义设置，主要的配置文件分别在/usr/share/config， ~/.kde/share/config。我现在采取的策略主要是写了一个脚本用定制的标准桌面环境的配置文件去覆盖 ~/.kde/share/config下面的配置文件（其实我把整个share文件夹都覆盖了）。

现在问题是其他的都似乎没有什么大问题，但是覆盖的kickerrc（配置上文提到的第二个部分中的kicker）文件没有作用，无论是在 /usr/share/config还是在~/.kde/share/config下面，kickerrc里面的设置没法被应用，而且通过手动去修改 Panel（比如增减Applet）后relog in kde session会用修改的配置覆盖掉~/.kde/share/config/kickerrc。似乎kickerrc只是反映当前的panel的设置而 不是系统根据kickerrc去配置当前的panel。

我在网上查了很多资料没找到原因，不知道这里有没有人知道。

更新：我用的是Redhat 3企业版（update几忘了，等查到再来更新：）。

How to capture key combination press in java

Though it may appear extremely easy to someone who has mastered it, I wanna paste some useful code here both for myself to keep a note and for those who haven’t ever addressed this problem to use as a guideline.

Code snippet (catch “ctrl+shift+`”):

public final static int CTRL_SHIFT_MASK =
KeyEvent.SHIFT_MASK | KeyEvent.CTRL_MASK;

if ((evt.getModifiers() & CTRL_SHIFT_MASK) != 0) {
if (inputEnabled) holdInput(true);

// Press “Ctrl+Shift+`” to toggle view only option.
if ((evt.getModifiers() & CTRL_SHIFT_MASK) == CTRL_SHIFT_MASK &&
keyCode == KeyEvent.VK_BACK_QUOTE) {
boolean viewOnly = !_vc.getConfigManager().isViewOnly();
_vc.getConfigManager().setViewOnly(viewOnly);
if (!viewOnly) {
inputRecorder.setPaused(false);
inputRecorder.setLastLogTime(System.currentTimeMillis());
} else {
inputRecorder.setPaused(true);
}
clearInput();
return;
}

Here key combination restricts to the pattern of modifier key(s) (ctrl, shift, alt) plus ordinary character key or only combination of modifier keys themselves.

The end.