typst: Ignore linebreaks between CJK characters in source code

Typst currently support breaking a paragraph into several consecutive lines in source code, without introduce a newline in result PDF. The single newline in source code will just become a space. However, when working with scripts that do not using spaces between words, like Chinese and Japanese, we often not want to have the newline, nor the space.

In a paragraph like this:

中文
测试

In Chinese context, the most sensible result is 中文测试, without any spaces and newlines.

I propose implementing the following features in typst:

  1. Eliminate spaces between CJK characters in source code.
  2. Let trailing comment consume a linebreak in source code. This is because sometimes we have a inline frame and CJK detect will not work, so we need an opt-in method. I think the changed behavior can be minimal, comments will consume new line only if it has no space before it. For example, this code
    abc //
    def
    
    becomes abc def and
    abc//
    def
    
    becomes abcdef. Latin users often use the former one so they will not be affected.

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 47
  • Comments: 25 (16 by maintainers)

Most upvoted comments

+1

Yes, this feature is very desirable in East Asian context, especially when one wants to manually wrap lines in source file.

However, instead of removing space between CJK characters, I propose disabling automatic space insertion at linebreaks in source code depending on the lang or script parameter of the current text.

As for the syntax of opting-out this automatic space occasionally, I think this is another topic that needs further discussion (maybe in another issue).

@earmer You make a lot of good and interesting points. I am totally with you that supporting the world’s typography in all its variety and depth is a wonderful goal for Typst.

That said, I must also be pragmatic. For Typst to be a lasting success, both as an open-source project and a company, it requires adoption and this adoption depends on many other critical features. I agree that CJK in particular should be placed higher in the priorities in the near future, but the resources we can commit to internationalization efforts at this time are still quite limited. This will hopefully change in the future (as adoption increases our resources).

I would love to make the vision you present a reality, but I must ask for patience. As a first step, we can try making a solution to this particular issue here a priority for Typst 0.14.

+1 for the trailing comment

That would solve my issue at https://github.com/typst/typst/discussions/1173 and give users more control over line breaks.

Based on those cases, I don’t think we can do this purely in the parser, since we can only know if preceding or following text is CJK at runtime, and even then, show rules might affect it, so this is really a realization or layout effect.

I think the only thing we would need at the Parser stage is to add a new SingleNewline item to SyntaxKind that acts like SyntaxKind::Space, but preserves the fact of the single newline when added in the AST (double newlines already create a SyntaxKind::ParBreak).

Then evaluation can store if there was a single newline as a parameter in SpaceElem, and layout can check if both preceding and trailing text is CJK to determine if the space is kept.

Well, many behaviors in LaTeX are only documented in Chinese, so let me explain here.

What it is like in LaTeX

  1. The CTeX package bundle implements common Chinese typesetting practices. It determines how LaTeX compilers interpret spaces and \n.

  2. CTeX provides an option to control whether to keep spaces after Han characters in the generated PDF.

    An example main.tex:

    汉字 分词
    技术 English Latin
    
    • auto (default): remove the space if it is followed by another Han character; keep it if not.

      ⇒ 汉字分词技术 English Latin

    • (fallback): remove spaces generated by \n, but keep other literal spaces

      ⇒ 汉字 分词技术 English Latin

    • true: always keep

      ⇒ 汉字 分词 技术 English Latin

      In this case, we need to add % before \n. (% in LaTeX = // in typst) Otherwise, \n will generate an extra space, because LaTeX compiler takes a single \n as a space.

    • false: always remove

      ⇒ 汉字分词技术English Latin

  3. CTeX use different packages for different LaTeX compilers. Consequently, the option is not fully supported by all compilers…

    auto (default) (fallback) true false
    XeLaTeX ✅ ❌ ✅ ❌
    LuaLaTeX ❌ ✅ ❌ ❌
    upLaTeX ❌ ✅ ❌ ❌
    (pdf)LaTeX ✅ ❌ ✅ ✅

    (Many Chinese templates only support XeLaTeX and LuaLaTeX.)

References:

  • CTeX doc (Chinese), §5.3 space = ⟨true|false|auto⟩.
  • xeCJK doc (Chinese), §3.1CJKspace = ⟨true|false⟩.
  • Overleaf documents (English): Chinese, Japanese, Korean.
  • LuaTeX-ja doc (Japanese and English), §1.2 Linebreak after a Japanese character and Spaces related to Japanese characters, and §15 Linebreak after a Japanese Character.

+1

Well, it is not so difficult to remove spaces that generated by a \n after a Han character… If this behaviour is accepted, maybe I can organize it into a PR.

I’m sure most CJK users will be happy to see this new behavior. Looking forward to your PR!

We wouldn’t even necessarily need a new syntax kind since the node stores its text, though checking at runtime whether it contains a newline is probably a little more costly than doing it in the parser.

Well, it is not so difficult to remove spaces that generated by a \n after a Han character… If this behaviour is accepted, maybe I can organize it into a PR.

Implementation

The “(fallback)” behaviour mentioned in https://github.com/typst/typst/issues/792#issuecomment-2303677367 can be implemented by changing the function https://github.com/typst/typst/blob/d48293f1c6ff5cfb256946be4f814df851197bd7/crates/typst-syntax/src/ast.rs#L67-L80 to

        let mut could_ignore = false;
        self.0
            .children()
            .filter(move |node| {
                let kind = node.kind();
                // Ignore newline if directly after…
                let keep = !could_ignore || kind != SyntaxKind::Space;
                // …statements without semicolons,
                could_ignore = kind.is_stmt()
                // or texts ended with a Han character.
                || (kind == SyntaxKind::Text
                        && node.text().ends_with(|c: char| {
                            matches!(c.script(), Script::Han)
                                || matches!(c, ',' | '。' | ':') // and other Chinese punctuations
                        }));
                keep
            })
            .filter_map(Expr::cast_with_space)

and use unicode_script::{Script, UnicodeScript};

An example

#set page(height: auto)
#set text(lang: "zh", region: "CN", font: "Noto Serif CJK is fantastic, but spaces are more visible between tofu")

汉字 分词
技术 English Latin
Continue 汉字 
And
孔乙己

鲁镇的酒店的格局,是和别处不同的:
都是当街一个曲尺形的大柜台,
柜里面预备着热水,可以随时温酒。

鲁镇的酒店的格局,是和别处不同的: //
都是当街一个曲尺形的大柜台, //
柜里面预备着热水,可以随时温酒。

图片

Side effects

Only one test failed, and it is expected.

❌ text-chinese-basic (tests/suite/layout/inline/cjk.typ:4)
  mismatched rendering
    live      | tests/store/render/text-chinese-basic.png
    ref       | tests/ref/text-chinese-basic.png
2225 passed, 1 failed, 0 skipped

图片

Future work

If this is the right way to go,

I imagine at some point in the future Typst will inevitably add something like #pragma to control global options, such as whether to use system font. After that we can add options to manage white space behavior.

There aren’t any plans to that effect and I would like to avoid it. Can’t we retain the necessary information as a field of SpaceElem?

Also may I suggest that spaces added between CJK characters should not be rendered. This behavior is consistent with XeLaTeX and will make vim-based motion a lot easier.

Update: I’ve the following hack that achieves this behavior in most cases, though it struggles with spaces around CJK punctuation and line breaks. I’m unsure how to resolve this issue, which I believe may be related to #86.

#let cjkre = regex("\s*(\p{Han}+)\s*")
#show cjkre: it => it.text.match(cjkre).captures.at(0)

= 你好,我是 你的朋友。

“你好” check测试:
hello, this is a test!
世界 你好,
我很好。

Using the idea from #1173, I’ve made some changes that seem to handle both line breaks and spaces around CJK characters pretty well. I’m still new to typst and haven’t tested it beyond this example yet, though.

#let cjk-char = "[\p{Han},。;:!?‘’“”()「」【】…—]"
#let cjk-re = regex("\s*(" + cjk-char + ")\s*")
#show cjk-re: it => it.text.match(cjk-re).captures.at(0)
#show: rest => {
  let ends-with-cjk = it => {
    it != none and it.has("text") and it.text.ends-with(regex(cjk-char))
  }
  let last-item = none
  for item in rest.children {
    if item == [ ] and ends-with-cjk(last-item) {
      item = []
    }
    last-item = item
    item
  }
}

= 你好,我是 你的 朋友。

“你好” check测试:
hello, world! this is a test!
this
is good.
世界
你 好              啊,
我很好。

which gets rendered as:

你好,我是你的朋友。

 “你好”check 测试:hello, world! this is a test! 世界你好啊,我很好。 

Also may I suggest that spaces added between CJK characters should not be rendered. This behavior is consistent with XeLaTeX and will make vim-based motion a lot easier.

Update: I’ve the following hack that achieves this behavior in most cases, though it struggles with spaces around CJK punctuation and line breaks. I’m unsure how to resolve this issue, which I believe may be related to #86.

#let cjkre = regex("\s*(\p{Han}+)\s*")
#show cjkre: it => it.text.match(cjkre).captures.at(0)

= 你好,我是 你的朋友。

“你好” check测试:
hello, this is a test!
世界 你好,
我很好。

Is there any workaround to this problem for now?

Well…

Examining the recent discussions, particularly around newline behavior, reveals me a deeper issue. It’s not just about \n; it’s about the often-unacknowledged assumptions we’ve built into a tool intended for universal application. We aim for global typesetting, yet are we unintentionally projecting a rather… localized perspective?

Consider Typst’s ambition: to handle diverse texts – code, prose, poetry – across languages and cultures with millennia of distinct typographic traditions. We aspire to create a tool for typesetting the world, in all its textual richness. However, we risk inadvertently simplifying this vibrant diversity into a somewhat… homogenous set of defaults.

Offline typesetting? Remarkably universal. Pen, paper, ruler – tools understood across cultures. Even OpenGL provides pixel-perfect control, should one desire a truly… artisanal approach to typesetting. Typst and LaTeX excel by automating the necessary tedium. But here lies the inherent challenge: automation requires assumptions, and these assumptions, being human-created, inevitably carry cultural perspectives.

We see “despotic defaults” praised in tools like Black – 4 spaces, fixed line lengths. Perfectly reasonable for code, where consistency is often valued. But prose? The moment we encounter established standards like GB/T 15834 (China), JIS X 4051 (Japan), or CLreq (W3C), the limitations of a purely default-driven approach become apparent.

Typst, as a “Unicode product,” aims for borderless text. Yet Unicode’s strength – unifying punctuation like 。, ., and » – paradoxically highlights a weakness: cultural reduction through categorization. #set text(lang: "zh") becomes a rather… optimistic attempt to encapsulate millennia of Chinese, Japanese, and Korean typographic nuance. Three distinct systems, grouped as “CJK” – it’s akin to suggesting “Europe is a single cultural entity” because of some shared script origins. Indeed, even the UK, the birthplace of English, has recently redefined its relationship with the European Union, further illustrating the complexities of assumed unity.

Let’s consider \n. In English code, “newline = whitespace, double newline = paragraph” is functional. But in Chinese poetry, a line break is a breath, a pause, an integral part of meaning. In Japanese vertical text, it’s a rhythmic pivot (soft-wrapping in the middle of ございます would be… unconventional). In Korean, a syllabic unit. Typst’s algorithm? “Whitespace, paragraph.” Algorithms, inherently context-agnostic, are applying a singular logic to centuries of culturally evolved textual practices. Chinese users are then compelled to insert manual linebreaks for readability, and we address the resulting whitespace with… somewhat elaborate Regex solutions. Pragmatic, perhaps, but not entirely… elegant.

Regarding multi-paragraph quotes in Chinese, the closing quote is often reserved for the final paragraph – a deliberate textual silence between sections. Typst, with its uniform approach, applies quotes to every paragraph. The workaround? #hack(regex_replace: "”") or #manual(delete_every_second_quote). Because incorporating specific cultural typesetting conventions into a “global” tool is considered… “too specialized.” One might suggest, “Typst should respect user input!” Certainly. Then perhaps a subtle indication: warn: in Chinese typesetting, end-of-line quotes should be omitted. instead of enforcing a Western-centric default?

Spacing considerations extend beyond CJK. For numbers and units: “0.125kg”, “0.125 kg”, “0.125 kg”. In Chinese, a space is generally expected. However, cjk-latin-spacing may not apply because both “0.125” and “kg” are classified as “latin-ish.” We risk overlooking nuanced cultural spacing norms by relying on overly broad categories.

Numerous examples illustrate this point: Date formats (DD/MM/YYYY vs. MM/DD/YYYY). Number formatting (thousands separators – commas, periods, spaces). Quotation styles (single, double, guillemets). Paper sizes (ISO vs. Letter). Subtle but significant variations impacting readability and cultural appropriateness.

Dismissing GB/T 15834, which shapes the reading experience of billions, as a “local preference” is akin to describing the Pacific Ocean as “a moderately sized body of water.” Or dismissing HTTPS because “most websites are primarily static content.”

For Typst to truly realize its potential, it must embrace cultural pluralism as a core architectural principle. Not through if lang == "zh" workarounds, but by designing locales as fundamental primitives—essential building blocks: #set typesystem(lang: "ja", rules: jis_x_4051). This isn’t feature bloat; it’s about building a tool that genuinely accommodates the rich diversity of human textual expression. Otherwise, we may find ourselves perpetually patching cultural gaps with regex and manual adjustments, pursuing the idealized, yet ultimately limited, vision of “one engine to typeset all texts”—a vision that, upon closer inspection, reflects a rather… particular cultural viewpoint.

Let’s prioritize making #set typesystem(rules: ...) a practical reality. This isn’t merely a desirable feature; it’s foundational for Typst to become a genuinely global, culturally aware, and widely adopted typesetting solution.

Recognizing that Typst is a developing project with a dedicated team, it’s crucial to place culturally sensitive text handling, starting with robust locale support, high on the roadmap. Implementing #set typesystem(rules: ...) as a core primitive is not just a feature request, but a fundamental step. Let’s discuss concrete actions to achieve this, ensuring Typst’s long-term success and global relevance. But I belive, finally one day in the future, we will have a typesetting engine that even supports #set text(lang: "ts") (not for TypeScript, but for Tangut script).

To achieve this, the first thing that we should do, is allowing more features that \n can play in source, to adapt its meaning based on cultural context.

I hope Typst will make greater achievements in text processing.