typst: Ignore linebreaks between CJK characters in source code
Typst currently support breaking a paragraph into several consecutive lines in source code, without introduce a newline in result PDF. The single newline in source code will just become a space. However, when working with scripts that do not using spaces between words, like Chinese and Japanese, we often not want to have the newline, nor the space.
In a paragraph like this:
中文
测试
In Chinese context, the most sensible result is 中文测试, without any spaces and newlines.
I propose implementing the following features in typst:
- Eliminate spaces between CJK characters in source code.
- Let trailing comment consume a linebreak in source code.
This is because sometimes we have a inline frame and CJK detect will not work, so we need an opt-in method. I think the changed behavior can be minimal, comments will consume new line only if it has no space before it.
For example, this code
becomesabc // defabc defand
becomesabc// defabcdef. Latin users often use the former one so they will not be affected.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 47
- Comments: 25 (16 by maintainers)
+1
Yes, this feature is very desirable in East Asian context, especially when one wants to manually wrap lines in source file.
However, instead of removing space between CJK characters, I propose disabling automatic space insertion at linebreaks in source code depending on the
langorscriptparameter of the current text.As for the syntax of opting-out this automatic space occasionally, I think this is another topic that needs further discussion (maybe in another issue).
@earmer You make a lot of good and interesting points. I am totally with you that supporting the world’s typography in all its variety and depth is a wonderful goal for Typst.
That said, I must also be pragmatic. For Typst to be a lasting success, both as an open-source project and a company, it requires adoption and this adoption depends on many other critical features. I agree that CJK in particular should be placed higher in the priorities in the near future, but the resources we can commit to internationalization efforts at this time are still quite limited. This will hopefully change in the future (as adoption increases our resources).
I would love to make the vision you present a reality, but I must ask for patience. As a first step, we can try making a solution to this particular issue here a priority for Typst 0.14.
+1 for the trailing comment
That would solve my issue at https://github.com/typst/typst/discussions/1173 and give users more control over line breaks.
Based on those cases, I don’t think we can do this purely in the parser, since we can only know if preceding or following text is CJK at runtime, and even then, show rules might affect it, so this is really a realization or layout effect.
I think the only thing we would need at the Parser stage is to add a new
SingleNewlineitem toSyntaxKindthat acts likeSyntaxKind::Space, but preserves the fact of the single newline when added in the AST (double newlines already create aSyntaxKind::ParBreak).Then evaluation can store if there was a single newline as a parameter in
SpaceElem, and layout can check if both preceding and trailing text is CJK to determine if the space is kept.Well, many behaviors in LaTeX are only documented in Chinese, so let me explain here.
What it is like in LaTeX
The CTeX package bundle implements common Chinese typesetting practices. It determines how LaTeX compilers interpret spaces and
\n.CTeX provides an option to control whether to keep spaces after Han characters in the generated PDF.
An example
main.tex:auto(default): remove the space if it is followed by another Han character; keep it if not.⇒
汉字分词技术 English Latin(fallback): remove spaces generated by
\n, but keep other literal spaces⇒
汉字 分词技术 English Latintrue: always keep⇒
汉字 分词 技术 English LatinIn this case, we need to add
%before\n. (%in LaTeX =//in typst) Otherwise,\nwill generate an extra space, because LaTeX compiler takes a single\nas a space.false: always remove⇒
汉字分词技术English LatinCTeX use different packages for different LaTeX compilers. Consequently, the option is not fully supported by all compilers…
auto(default)truefalse(Many Chinese templates only support XeLaTeX and LuaLaTeX.)
References:
space = ⟨true|false|auto⟩.CJKspace = ⟨true|false⟩.+1
I’m sure most CJK users will be happy to see this new behavior. Looking forward to your PR!
We wouldn’t even necessarily need a new syntax kind since the node stores its text, though checking at runtime whether it contains a newline is probably a little more costly than doing it in the parser.
Well, it is not so difficult to remove spaces that generated by a
\nafter a Han character… If this behaviour is accepted, maybe I can organize it into a PR.Implementation
The “(fallback)” behaviour mentioned in https://github.com/typst/typst/issues/792#issuecomment-2303677367 can be implemented by changing the function https://github.com/typst/typst/blob/d48293f1c6ff5cfb256946be4f814df851197bd7/crates/typst-syntax/src/ast.rs#L67-L80 to
and
use unicode_script::{Script, UnicodeScript};An example
Side effects
Only one test failed, and it is expected.
Future work
If this is the right way to go,
',' | '。' | ':'should be replaced byis_cjk_punctuation/is_cjk_left_aligned_punctuation/… here https://github.com/typst/typst/blob/d48293f1c6ff5cfb256946be4f814df851197bd7/crates/typst/src/layout/inline/shaping.rs#L114#set text(ignore-???-newline: false), as in #2334 .There aren’t any plans to that effect and I would like to avoid it. Can’t we retain the necessary information as a field of SpaceElem?
Using the idea from #1173, I’ve made some changes that seem to handle both line breaks and spaces around CJK characters pretty well. I’m still new to typst and haven’t tested it beyond this example yet, though.
which gets rendered as:
Also may I suggest that spaces added between CJK characters should not be rendered. This behavior is consistent with XeLaTeX and will make vim-based motion a lot easier.
Update: I’ve the following hack that achieves this behavior in most cases, though it struggles with spaces around CJK punctuation and line breaks. I’m unsure how to resolve this issue, which I believe may be related to #86.
Is there any workaround to this problem for now?
Well…
Examining the recent discussions, particularly around newline behavior, reveals me a deeper issue. It’s not just about
\n; it’s about the often-unacknowledged assumptions we’ve built into a tool intended for universal application. We aim for global typesetting, yet are we unintentionally projecting a rather… localized perspective?Consider Typst’s ambition: to handle diverse texts – code, prose, poetry – across languages and cultures with millennia of distinct typographic traditions. We aspire to create a tool for typesetting the world, in all its textual richness. However, we risk inadvertently simplifying this vibrant diversity into a somewhat… homogenous set of defaults.
Offline typesetting? Remarkably universal. Pen, paper, ruler – tools understood across cultures. Even OpenGL provides pixel-perfect control, should one desire a truly… artisanal approach to typesetting. Typst and LaTeX excel by automating the necessary tedium. But here lies the inherent challenge: automation requires assumptions, and these assumptions, being human-created, inevitably carry cultural perspectives.
We see “despotic defaults” praised in tools like Black – 4 spaces, fixed line lengths. Perfectly reasonable for code, where consistency is often valued. But prose? The moment we encounter established standards like GB/T 15834 (China), JIS X 4051 (Japan), or CLreq (W3C), the limitations of a purely default-driven approach become apparent.
Typst, as a “Unicode product,” aims for borderless text. Yet Unicode’s strength – unifying punctuation like
。,., and»– paradoxically highlights a weakness: cultural reduction through categorization.#set text(lang: "zh")becomes a rather… optimistic attempt to encapsulate millennia of Chinese, Japanese, and Korean typographic nuance. Three distinct systems, grouped as “CJK” – it’s akin to suggesting “Europe is a single cultural entity” because of some shared script origins. Indeed, even the UK, the birthplace of English, has recently redefined its relationship with the European Union, further illustrating the complexities of assumed unity.Let’s consider
\n. In English code, “newline = whitespace, double newline = paragraph” is functional. But in Chinese poetry, a line break is a breath, a pause, an integral part of meaning. In Japanese vertical text, it’s a rhythmic pivot (soft-wrapping in the middle of ございます would be… unconventional). In Korean, a syllabic unit. Typst’s algorithm? “Whitespace, paragraph.” Algorithms, inherently context-agnostic, are applying a singular logic to centuries of culturally evolved textual practices. Chinese users are then compelled to insert manual linebreaks for readability, and we address the resulting whitespace with… somewhat elaborate Regex solutions. Pragmatic, perhaps, but not entirely… elegant.Regarding multi-paragraph quotes in Chinese, the closing quote is often reserved for the final paragraph – a deliberate textual silence between sections. Typst, with its uniform approach, applies quotes to every paragraph. The workaround?
#hack(regex_replace: "”")or#manual(delete_every_second_quote). Because incorporating specific cultural typesetting conventions into a “global” tool is considered… “too specialized.” One might suggest, “Typst should respect user input!” Certainly. Then perhaps a subtle indication:warn: in Chinese typesetting, end-of-line quotes should be omitted.instead of enforcing a Western-centric default?Spacing considerations extend beyond CJK. For numbers and units: “0.125kg”, “0.125 kg”, “0.125 kg”. In Chinese, a space is generally expected. However,
cjk-latin-spacingmay not apply because both “0.125” and “kg” are classified as “latin-ish.” We risk overlooking nuanced cultural spacing norms by relying on overly broad categories.Numerous examples illustrate this point: Date formats (DD/MM/YYYY vs. MM/DD/YYYY). Number formatting (thousands separators – commas, periods, spaces). Quotation styles (single, double, guillemets). Paper sizes (ISO vs. Letter). Subtle but significant variations impacting readability and cultural appropriateness.
Dismissing GB/T 15834, which shapes the reading experience of billions, as a “local preference” is akin to describing the Pacific Ocean as “a moderately sized body of water.” Or dismissing HTTPS because “most websites are primarily static content.”
For Typst to truly realize its potential, it must embrace cultural pluralism as a core architectural principle. Not through
if lang == "zh"workarounds, but by designing locales as fundamental primitives—essential building blocks:#set typesystem(lang: "ja", rules: jis_x_4051). This isn’t feature bloat; it’s about building a tool that genuinely accommodates the rich diversity of human textual expression. Otherwise, we may find ourselves perpetually patching cultural gaps with regex and manual adjustments, pursuing the idealized, yet ultimately limited, vision of “one engine to typeset all texts”—a vision that, upon closer inspection, reflects a rather… particular cultural viewpoint.Let’s prioritize making
#set typesystem(rules: ...)a practical reality. This isn’t merely a desirable feature; it’s foundational for Typst to become a genuinely global, culturally aware, and widely adopted typesetting solution.Recognizing that Typst is a developing project with a dedicated team, it’s crucial to place culturally sensitive text handling, starting with robust locale support, high on the roadmap. Implementing
#set typesystem(rules: ...)as a core primitive is not just a feature request, but a fundamental step. Let’s discuss concrete actions to achieve this, ensuring Typst’s long-term success and global relevance. But I belive, finally one day in the future, we will have a typesetting engine that even supports#set text(lang: "ts")(not for TypeScript, but for Tangut script).To achieve this, the first thing that we should do, is allowing more features that
\ncan play in source, to adapt its meaning based on cultural context.I hope Typst will make greater achievements in text processing.