In this series, I relate my quest for the perfect internationalization system (i18n) for hybrid frameworks, namely Next.js and Astro. This story involves nasty technical concepts, but also teapots 🫖, so it's probably worth the read.

The end goal is to improve the Devographics translation system, which powers the State of JavaScript, CSS, HTML, React and GraphQL surveys. It is used both by the Next.js survey app and the new Astro results app which is not released yet.

You are reading the third part of this series. You can read part 1, "Tokens are all you need", here. Part 2, "Server Components, we have a deeply nested problem" is available there. Part 3, "From tokens to token expressions", this way.

Part 4 : Optimized client-side translations

Our goal is to collect the tokens used by client components for a given page. We can use this list of tokens to construct an optimized dictionary to be sent to the browser.

In the previous article, we have proposed a simplified syntax:

// In HomeButton.tsx
"use client";
function HomeButton() {
  return (
    <button onClick={doSomething}>
      <T token="home.button" />
    </button>
  );
}
export const tokens = ["home.button"];

This syntax expects some additional work from the developer, and leads to repetitions as the token appears twice in the code. Let's craft an helper function to make to process more pleasurable and scalable.

At this stage, I don't want to involve bundle magic, for a good reason: the Next.js bundler turbopack doesn't have plugins yet.

Also writing bundler plugin is time-consuming and my only experience with bundlers is crafting a tiny Webpack plugin that was rewriting local imports to an absolute path, somewhere around 2017.

The teapot function

So I want to figure a better API for declaring tokens used by a client component.

Here are the constraints :

• must not rely on build-time magic
• must provide a decent developer experience
• should not affect the way we write React components, so we can easily ditch this syntax when we have a smarter bundler plugin to automatically extract tokens

Boil some water to 100°C, let infuse a few minutes and here's the result:

// In HomeMessage.tsx
import { SomeChild, tokens } from "./SomeChild"

export const { T, tokens } = teapot([
    "home.button",
    ...SomeChildTokens
] as const)


function HomeButton() {
    return (
        <button>
            <T token="home.button" />
            <SomeChild />
        </button>
    )
}

Well, it doesn't look that simple, but trust me it's great. This syntax works like so:

• Instead of exporting an array of tokens directly, we pass the array to a teapot function.
• The teapot function generates a type-safe T component: it will detect token props that are not properly listed in the array.
• The as const is there to get this type-safety. Maybe we can get rid of it but this is beyond my TypeScript knowledge.
• We reexport the resulting tokens array, so it can be consumed server-side to construct an optimized client-side dictionary
• We need to include the tokens from each children, progressively bubbling up tokens. Yeah that's not great, but the best I could do without bundler magic.

The teapot🫖 makes T🍵: that's impossible to forget! When you call this function in 100+ components, cuteness of the API plays a role.

After experimenting this idea on a dozen components, it seems to work. The as const part is really the most annoying bit but can probably be eliminated in the future.

Notice that the syntax of the React component itself is not affected at all. If we change the system later on, we can just drop the call to teapot altogether, the HomeButton component will stay exactly the same.

This system can feel cumbersome but it's much cleaner than shoving 200ko of translations in a React context and read that from client components. It works without bundle magic, so it's agnostic to the underlying framework or build system.

This idea worked out-of-the box in Astro, but, to my great dismay, not in Next.js.

Let's explain why and figure some alternative solutions.

In Next.js, exported variables can't cross the client boundary freely

In Next.js, server components can import client components from files marked with the "use client", but not values.

"use client"
// ✅ This can be used in a server component
function ClientComponent() {
	return (...)
}
// ❌ This cannot be used in a server component
export const { tokens } = teapot(["home.title"])

To be precise, the array can be imported, but not processed. It has to be passed as-is to another client component, we can't do anything with the value during the server-side data fetching step.

We wanted to use this list of tokens to build the client-side optimized translation dictionary, but it's not possible.

We can't modify the array, we can't iterate over it using map, includes or filter.

// In the page component
import { HomeButton, tokens } from "./HomeButton";
export async function HomePage() {
  const bigDictionary = await fetchLocale();
  // ❌ Will throw an Error
  const clientSideDictionary = filterTokens(bigDictionary, tokens);
  return (
    <I18nContext dictionary={clientSideDictionary}>
      <HomeButton />
    </I18nContext>
  );
}

I've opened a GitHub ticket to get more insights. There is probably a good reason for this behaviour, so we shouldn't expect it to change in a foreseeable future. Serializing the array to a string doesn't work, freezing it doesn't work.

Separate token files

Because of the limitation described in the previous section, the list of tokens used by client components have to live in separate files, that are NOT marked with the "use client" directive.

This way they can be imported by the client component that display the translations, and by the server components that uses the list to provide the translations.

This is a bit similar to CSS modules, the file structure would be:

HomeButton.tsx
HomeButton.module.css
HomeButton.tokens.ts

The tokens file simply exports an array:

// In HomeButton.tokens.ts
// as const allows type-safety
export const tokens = ["home.button"] as const

Usage in a client component:

// In HomeButton.tsx
import { tokens } from "HomeButton.tokens";
// teapot provides type-safety and cuteness
const { T } = teapot(tokens);
function HomeButton() {
  return (
    <button>
      <T token="home.button" />
      <SomeChild />
    </button>
  );
}

Usage in the server component that builds the translation dictionary:

// In the page server component
import { HomeButton } from "./HomeButton";
import { tokens } from "./HomeButton.tokens";
export async function HomePage() {
  const bigServerDictionary = await fetchLocale();
  // ✅ Now it works
  const smallClientDictionary = 
      filterTokens(bigServerDictionary, tokens);
  return (
    <I18nContext dictionary={smallClientDictionary}>
      <HomeButton />
    </I18nContext>
  );
}

Dependency tree and results

The main limitation of this approach is handling dependencies. You have to list them explicitly in each .tokens file and keep them in sync with the main component.

// In Layout.tokens.ts
// We have to keep in sync the list of dependencies,
// based on Layout.tsx imports
import { tokens as tokensHeader } from "./Header.tokens";
export const tokens = ["general.skip_to_content", ...tokensHeader];

We finally have a system that works for Next.js, here is the resulting client-side context that contains the filtered dictionary. It's an array of size 3, containing only the tokens used by the layout.

We have a similar context for the page (in Next layout and pages have their own lifecycle), also optimized.

That's much better than the initial 800 tokens!

Conclusion : i18n rendering must be optimized at framework-level

Four articles later, we finally have an end-to-end internationalization system properly optimized for Astro and Next.js App Router.

Our "index.html" for the survey app page is now ~9.5kb, versus ~23kb for the version currently in production.

For server components, the HTML file size is not exactly optimal because text content have to repeated twice (see explanations in this thread). I couldn't prove a perf improvement CPU-wise, but maybe because I am bad at profiling.

Was this 50% optimization of the HTML file size worth the hassle? I don't know, but the ride was great.

The more I work on this issue, the more I feel that translations rendering should be a feature provided at framework-level.

It is very difficult to implement a solid filtering system in user-land, without relying on some external bundler magic.

I18n routing should however be doable in user-land, Next.js approach with middleware is exemplary in this regard. I am specifically talking about rendering translations here, a problem which could be rephrased as personalization of the leaves of a rendering tree.

Let's hope for a brighter future! In the meantime our .tokens file format is not so bad and our teapot🫖🍵 function makes it as convenient as possible.

My next focus will be "Translator Experience": figuring missing or unused tokens, and allowing external contributors to propose translations in their language.

A bientôt !