The TeX FAQ

Frequently Asked Question List for TeX

Misc

TeX Engine development

While Knuth has declared that TeX will never change in any substantial way, there remain things that one might wish had been done differently, or indeed implemented at all. These have been explored in a number of engine developments, some of which have somewhat complex histories.

The first set of major additions to Knuth’s TeX were provided by the NTS project in ε-TeX: these additions are available in all current engines. pdfTeX added a range of new primitives, and many these have also been taken up by other engine developers.

Active projects

pdfTeX

pdfTeX was developed by Hàn Thế Thành, incorporating the ability to create PDF output directly into the TeX engine. Although originally developed in parallel, pdfTeX today includes the ε-TeX extensions as-standard. Development of pdfTeX has mostly stopped: changes are largely focussed on bug fixes. However, the central importance of pdfTeX as the “standard” engine in modern TeX distributions means further changes may be seen, particularly in the area of accessibility.

pdfTeX started as a topic for Hàn Thế Thành’s Master’s thesis, and seems first to have been published in TUGboat 18(4), in 1997 (though it was certainly discussed at the TUG’96 conference in Russia). While the world was making good use of “pre-releases” of pdfTeX, Thành used it as a test-bed for the micro-typography which was the prime subject of his Ph.D. research. Since Thành was finally awarded his Ph.D., day-to-day maintenance and development of pdfTeX 1.0 (and later) has been in the hands of a group of pdfTeX maintainers (which includes Thành); the group has managed to maintain a stable platform for general use.

XeTeX

XeTeX, originally developed by Jonathan Kew, a Unicode TeX engine which is able to load system fonts using the HarfBuzz library, which is accomplished by an extended syntax to the \font primitive. It builds on ε-TeX and also adds a range of new primitives useful for working with non-Latin languages. Otherwise, the process of typesetting is essentially the same as TeX’s. Like pdfTeX, major developments are not currently ongoing with the XeTeX codebase, though bug fixes and cross-compatibility with other engines do continue.

LuaTeX

As detailed above, development of pdfTeX is “in essence” complete. As pdfTeX development ran down, development of a new system, LuaTeX was started. Lua is a interpreter designed to be incorporated into other applications. LuaTeX consists of a TeX-like engine with a Lua interpreter “embedded” in it; the lua interpreter has access to many of the data structures used for typesetting, so that the programmer may also interpolate chunks of Lua code into their (La)TeX macros, or as “call-backs” for use when the TeX-like engine does certain operations.

This arrangement offers the prospect of a “semi-soft” typesetting engine: it will have its basic behaviour, but the user gets to redefine functionality if an idea occurs — there will be no need to persuade the world first, and then find a willing developer to work on the sources of of the distribution.

The LuaTeX project has pursued avenues that many of other projects have had in their sights, notably Unicode character representations and support for OpenType fonts. This included incorporating the extensions pioneered by Aleph. Current LuaTeX (v1.0.8) is considered close to feature-complete by the development team: at the time of writing, the possibility of a v2 development path has been suggested.

pTeX

pTeX is an old Japanese-specific extension of TeX82, which aims to support proper typesetting of Japanese text but only supports a limited character set, JIS X 0208 (6879 characters).

upTeX is developed as an extension of pTeX to support full Unicode characters. It also includes extensions to overcome the difficulties of pTeX in processing 8-bit Latin characters due to conflicts with legacy multibyte Japanese encodings.

𝜀-pTeX and 𝜀-upTeX are 𝜀-TeX extensions of pTeX and upTeX respectively. In the current release, some extensions derived from pdfTeX and Ω are also available.

Text taken from the Guide to pTEX for developers unfamiliar with Japanese.

Historical projects

ε-TeX

As part of the work of the NTS project, a number of extensions to Knuth’s TeX were developed. Many of these support programming in a way which makes tasks easier than in Knuth’s TeX, or indeed possible at all. Others extend the range of available registers in TeX or make more internal information available. Current TeX distributions are delivered with the ε-TeX extensions available in almost all formats other than tex (Knuth’s TeX) itself.

ExTeX

The ExTeX project was building on the experience of the many existing TeX development and extension projects, to develop a new TeX-like system. The system was developed in Java.

ExTeX was designed to accept Unicode input, and to implement all of TeX’s primitives, but with some marked as obsolete, and “modern” alternatives provided. Desirable extensions from ε-TeX, pdfTeX and Ω were identified for incorporation.

Omega and Aleph

Omega was developed as an extension of TeX, to use with multilingual texts, expressed in a variety of input encodings. Omega used 16-bit, Unicode-encoded, characters. It provided many innovative concepts, notably including the “translation process” that takes a character stream and transforms it according to various processes that may be internally specified, or be a separate program.

While Omega showed a lot of promise at its mid-1990s announcement, progress was slow, and development was essentially dead by the time that one of the original developers withdrew (taking with him a bunch of research students).

Before that distressing event, a separate thread of development had started, to produce a program called Aleph, which merged the facilities of ε-TeX into a stable Omega codebase and added other extensions. Aleph also proved an attractive platform for many people; but its development, too, has dried up.

Some of the work from these projects has been incorporated into LuaTeX.

New Typesetting System (NTS)

The NTS project set out to produce an advanced replacement for TeX, to provide a basis for developing such modifications: this “New Typesetting System” would share Knuth’s aims, but would implement the work in a modern way taking account of the lessons learned with TeX. While a first demonstrator NTS did appear, it wasn’t practically useful, and the project seems no longer active.

In parallel with this work, the NTS developed a set of extensions that can be used with a (“true”) TeX system. Such a modified system is known as an ε-TeX system, and the concept has proved widely successful.

FAQ ID: Q-enginedev
Tags: pdftexluatexxetex
Last updated: 2018-05-24