HZ - A Data Format for Exchanging Files of	      Arbitrarily Mixed Chinese and ASCII characters 	      Fung F. Lee ~{@n7c7e~} (lee@umunhum.stanford.edu)			    Stanford Univeristy				     			     September 4, 1989				     1. IntroductionMost existing computer systems which can handle a text file of arbitrarilymixed Chinese and ASCII characters use 8-bit codes.  To exchange such textfiles through electronic mail on ASCII computer systems, it is necessary toencode them in a 7-bit format.  A generic binary to ASCII encoder is notsufficient, because there is currently no universal standard for such 8-bitcodes. For example, CCDOS and Macintosh's Chinese OS use different internalcodes.  Fortunately, there is a PRC national standard, GuoBiao (GB), forthe encoding of Chinese characters, and Chinese characters encoded in theabove systems can be easily converted to GB by a simple formula. (* The ROCstandard BIG-5 is outside the scope of this article.)HZ is a 7-bit data format proposed for arbitrarily mixed GB and ASCII textfile exchange.  HZ is also intended for the design of terminal emulatorsthat display and edit mixed Chinese and ASCII text files in real time.2. SpecificationThe format of HZ is described in the following.Without loss of generality, we assume that all Chinese characters (HanZi)have already been encoded in GB.  A GB (GB1 and GB2) code is a two bytecode, where the first byte is in the range $21-$77 (hexadecimal), and thesecond byte is in the range $21-$7E.A graphical ASCII character is a byte in the range $21-$7E. A non-graphicalASCII character is a byte in the range $0-$20 or of the value $7F.Since the range of a graphical ASCII character overlaps that of a GB byte,a byte in the range $21-$7E is interpreted according to the mode it is in.There are two modes, namely ASCII mode and GB mode.By convention, a non-graphical ASCII character should only appear in ASCIImode.The default mode is ASCII mode. In ASCII mode, a byte is interpreted as an ASCII character, unless a '~~' isencountered. The character '~~' is an escape character. By convention, itmust be immediately followed ONLY by '~~', '{' or '\n' (<LF>), with thefollowing special meaning.o The escape sequence '~~~~' is interpreted as a '~~'.o The escape-to-GB sequence '~~{' switches the mode from ASCII to GB.o The escape sequence '~~\n' is a line-continuation marker to be consumed  with no output produced.In GB mode, characters are interpreted two bytes at a time as (pure) GBcodes until the escape-from-GB code '~~}' is read. This code switches themode from GB back to ASCII.  (Note that the escape-from-GB code '~~}'($7E7D) is outside the defined GB range.)The decoding process is clear from the above description.The encoding process is straightforward. Note that an (ASCII) '~~' is alwaysencoded as '~~~~'. A sequence of GB codes is enclosed in '~~{' and '~~}'.3. Remarks & RecommendationsWe choose to encode any ASCII character except '~~' as it is, rather than asa two byte code, and we choose ASCII as the default mode for the followingreasons. The computer systems we use is ASCII based.  A HZ file containingpure ASCII characters (i.e. no Chinese characters) except '~~' is preciselya pure ASCII file. In general, the English (ASCII) portion of a HZ file isdirectly readable.The escape character '~~' is chosen not only because it is commonly used inthe ASCII world, but also because '~~' ($7E) is outside the defined range($21-$77) of the first byte of a GB code.In ASCII mode, other potential escape sequences, i.e., two byte sequencesbeginning with '~~' (other than '~~~~', '~~{', '~~\n') are currently invalid HZsequences. Hence, they can be used for future extension of HZ with totalupward compatibility.The line-continuation marker '~~\n' is useful if one wants to encode longlines in the original text into short lines in this data format withoutintroducing extra newline characters in the decoding process.There is no limit on the length of a line. In fact, the whole file could beone long line or even contain no newline characters. Any DECODER of this HZdata format should not and has no need to operate on the concept of a line.It is easy to write encoders and decoders for HZ. An encoder or decoderneeds to lookahead at most one character in the input data stream.Given the current mode, it is also possible and easy to decode a HZ datastream by scanning backward. One of the implication is that "backspaces"can be handled correctly by a terminal emulator.To facilitate the effective use of programs supporting line/page skips suchas "more" on UNIX with a terminal emulator understanding the HZ format, itis RECOMMENDED that the ENCODER (which outputs in HZ) sets a maximum linesize of less than 80 characters.  Since '\n' is an ASCII character, thesyntax of HZ then automatically implies that GB codes appearing at the endof a line must be terminated with the escape-from-GB code '~~}', and theline-continuation marker '~~\n' should be inserted appropriately. The priceto paid is that the encoded file size is slightly larger.It is important to understand the following distinction.  Note that theabove recommendation does NOT change the HZ format.  It is simply anencoding "style" which follows the syntax of HZ. Note that this "style" isnot built into HZ. It is an additional convention built "on top of" HZ.Other applications may require different "styles", but the same basic HZDECODER will always work. The essence of HZ is to provide such a flexiblebasic data format for files of arbitrarily mixed Chinese and ASCIIcharacters.4. ExamplesTo illustrate the "stylistic" issue of HZ encoding, we give the followingfour examples of encoded text, which should produce the same decodedoutput. (The recommendation in the last section refers to Example 2.)Example 1:  (Suppose there is no line size limit.)This sentence is in ASCII.The next sentence is in GB.~{<:Ky2;S{#,NpJ)l6HK!#~}Bye.Example 2:  (Suppose the maximum line size is 42.)This sentence is in ASCII.The next sentence is in GB.~{<:Ky2;S{#,~}~~{NpJ)l6HK!#~}Bye.Example 3:  (Suppose a new line is started whenever there is a mode switch.)This sentence is in ASCII.The next sentence is in GB.~~{<:Ky2;S{#,NpJ)l6HK!#~}~Bye.AcknowledgementEdmund Lai was the first one who brought my attention to this topic.Discussions with Ed, Tin-Fook Ngai, Yagui Wei and Ricky Yeung were veryhelpful in shaping the ideas in this article. Thanks to Tin-Fook for hiscareful review of the draft and numerous interesting suggestions.