js-lingui: `lingui extract-experimental` bugs

An update here:

Extracted message id incorrect when Trans children has plain text follow JSX element, space is losted

I created a separate discussion, please add your ideas there.

Dependency tree related files (AboutText.tsx/Developers.tsx) do not extracted in reproduction, but Switcher.tsx did

This one is fixed in https://github.com/lingui/js-lingui/pull/1867 separate test case was added

Wow, ok. Let’s check than, does this change #1867 helps? (you can build a next branch by yourself following these instructions.

The problem is, i don’t have a big enough project to play with. If you can point me to the one of the opensource projects which may reproduce the performance issue, i will try to debug performance on my own.

On 4.7.1

Catalog statistics for pages/_i18nEntry.tsx:
┌────────────────┬─────────────┬─────────┐
│ Language       │ Total count │ Missing │
├────────────────┼─────────────┼─────────┤
│ de-DE          │     472     │   472   │
│ en-US (source) │     472     │    -    │
│ en-GB          │     472     │   472   │
│ en-PL          │     472     │   472   │
│ fr-FR          │     472     │   472   │
│ ja-JP          │     472     │   15    │
└────────────────┴─────────────┴─────────┘

pnpm --filter=app-dashboard lang:extract  128.83s user 9.58s system 134% cpu 1:42.79 total

On Next

Catalog statistics for pages/_i18nEntry.tsx:
┌────────────────┬─────────────┬─────────┐
│ Language       │ Total count │ Missing │
├────────────────┼─────────────┼─────────┤
│ de-DE          │     472     │   472   │
│ en-US (source) │     472     │    -    │
│ en-GB          │     472     │   472   │
│ en-PL          │     472     │   472   │
│ fr-FR          │     472     │   472   │
│ ja-JP          │     472     │   15    │
└────────────────┴─────────────┴─────────┘

pnpm --filter=app-dashboard lang:extract  15.20s user 2.39s system 198% cpu 8.874 total

So yeah, you could call that an improvement 😅

Hi @semoal i see following tasks:

1. Move macro step to each file before it would be processed by esbuild. (Right now babel with macro applied at the last step together with extractor plugin on the whole bundle). This could be done quite easily by writing esbuild plugin or taking one from existing.
2. [Optional] Instead of applying babel and babel’s macro version on each file, it’s better to apply SWC + lingui-plugin. I expect a noticeable time overhead if point 1 would be implemented, using SWC instead of babel should cut the time. Unfortunately, this requires changes in lingui-swc plugin, since it doesn’t create /*i18n*/ comments which are required by extractor.
3. In some discord discussions someone reported that esbuild trying to resolve scss file, I don’t remember how it is implemented now, but i suppose we should mark as “external” every non js/js-like file for simplicity.

I think this 3 is essential to at least unblock following adoption of the extractor.

From my side, i can help with SWC Rust plugin, and can explain how extractor works now.

After basic functionality become stable we can do performance optimizations:

1. [Multithreading] Run extraction process on each bundle in the separate worker
2. Move new extractor to separate package (to avoid people download and install esbuild & swc if they don’t use it)

Because it doesn’t take any preprocess step (esbuild) and consume sources directly.

Interesting, thanks for report, that probably related to useLingui and this refactoring I will take a look.

Extracted message id incorrect when Trans children has plain text follow JSX element, space is losted

Fixed: https://github.com/lingui/js-lingui/pull/1882

How far @thekip are we of converting this feature in something stable? Probably if you detail which tasks are pending I can help on my free time 😃 We noticed that our catalogs are getting large and this splitting by page would be amazing

I considered that solution in the beginning, even look into that exact library. Solution with esbuild i like more because:

1. it’s doing tree shaking, more chances that catalog will not contain unnecessary strings. If your project uses tons of barrel files (index.{ts,js} with re-exports) this deps tree resolution become useless without tree-shaking.
2. it’s much faster
3. it’s much easier to use, setup and maintenance.

I still think we need to continue with esbuild. The problems mentioned in this issue are fixable, i just don’t have a time for that currently.

Actually, some spaces are deleted intentionally. You can read unit-tests to understand the cases. The issue would be if the final line would be different between what left by macro in the code and what was extracted to catalog.

So use extract-experimental in production seems not a good idea recently.

Yes, it was created as PoC, and actually there not match attention to it, so i could not get enough feedback to make it production ready.

On the other hand, how about the first question: Space is losted when plain text follow JSX element under Trans.

I didn’t investigate it, but i think the problem is similar, when code is processed by esbuild it gets reprinted and space might be dropped.