Skip to content

cometkim/unicode-segmenter

Folders and files

NameName
Last commit message
Last commit date

Latest commit

ย 

History

305 Commits
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 

Repository files navigation

unicode-segmenter

NPM Package Version NPM Downloads Integration codecov CodSpeed Badge LICENSE - MIT

A lightweight implementation of the Unicode Text Segmentation (UAX #29)

  • Spec compliant: Up-to-date Unicode data, verified by the official Unicode test suites and fuzzed with the native Intl.Segmenter, and maintaining 100% test coverage.

  • Excellent compatibility: It works well on older browsers, edge runtimes, React Native (Hermes) and QuickJS.

  • Zero-dependencies: It doesn't bloat node_modules or the network bandwidth. Like a small minimal snippet.

  • Small bundle size: It effectively compresses the Unicode data and provides a bundler-friendly format.

  • Extremely efficient: It's carefully optimized for runtime performance, making it the fastest one in the ecosystemโ€”outperforming even the built-in Intl.Segmenter. Its flat lookup tables also keep the memory footprint far smaller than other libraries.

  • TypeScript: It's fully type-checked, and provides type definitions and JSDoc.

  • ESM-first: It primarily supports ES modules, and still supports CommonJS.

Note

unicode-segmenter is now e18e recommendation!

Unicodeยฎ Version

Unicodeยฎ 17.0.0

Unicodeยฎ Standard Annex #29 - Revision 47 (2025-08-17)

APIs

Entries for Unicode text segmentation.

And matchers for extra use cases.

Export unicode-segmenter/grapheme

Utilities for text segmentation by extended grapheme cluster rules.

Example: Get grapheme segments

import { graphemeSegments } from 'unicode-segmenter/grapheme';

[...graphemeSegments('aฬeฬoฬˆฬฒ\r\n')];
// 0: { segment: 'aฬ', index: 0, input: 'aฬeฬoฬˆฬฒ\r\n' }
// 1: { segment: 'eฬ', index: 2, input: 'aฬeฬoฬˆฬฒ\r\n' }
// 2: { segment: 'oฬˆฬฒ', index: 4, input: 'aฬeฬoฬˆฬฒ\r\n' }
// 3: { segment: '\r\n', index: 7, input: 'aฬeฬoฬˆฬฒ\r\n' }

Example: Split graphemes

import { splitGraphemes } from 'unicode-segmenter/grapheme';

[...splitGraphemes('#๏ธโƒฃ*๏ธโƒฃ0๏ธโƒฃ1๏ธโƒฃ2๏ธโƒฃ')];
// 0: #๏ธโƒฃ
// 1: *๏ธโƒฃ
// 2: 0๏ธโƒฃ
// 3: 1๏ธโƒฃ
// 4: 2๏ธโƒฃ

Example: Count graphemes

import { countGraphemes } from 'unicode-segmenter/grapheme';

'๐Ÿ‘‹ ์•ˆ๋…•!'.length;
// => 6
countGraphemes('๐Ÿ‘‹ ์•ˆ๋…•!');
// => 5

'aฬeฬoฬˆฬฒ'.length;
// => 7
countGraphemes('aฬeฬoฬˆฬฒ');
// => 3

Note

countGraphemes() is a small wrapper around graphemeSegments().

If you need it more than once at a time, consider memoization or use graphemeSegments() or splitGraphemes() once instead.

If counting is on your hot path, use the standalone unicode-segmenter/grapheme-counter entry instead.

Example: Build an advanced grapheme matcher

graphemeSegments() exposes some knowledge identified in the middle of the process to support some useful cases.

For example, knowing the Grapheme_Cluster_Break category at the beginning and end of a segment can help approximately infer the applied boundary rule.

import { graphemeSegments, GraphemeCategory } from 'unicode-segmenter/grapheme';

function* matchEmoji(str) {
  for (const { segment, _catBegin } of graphemeSegments(input)) {
    // `_catBegin` identified as Extended_Pictographic means the segment is emoji
    if (_catBegin === GraphemeCategory.Extended_Pictographic) {
      yield segment;
    }
  }
}

[...matchEmoji('1๐ŸŒท2๐ŸŽ3๐Ÿ’ฉ4๐Ÿ˜œ5๐Ÿ‘')]
// 0: ๐ŸŒท
// 1: ๐ŸŽ
// 2: ๐Ÿ’ฉ
// 3: ๐Ÿ˜œ
// 4: ๐Ÿ‘

Or build even more advanced one like an Unicode-aware TTY string width utility.

Export unicode-segmenter/grapheme-counter

A standalone counter for extended grapheme clusters.

The result is identical to countGraphemes() of unicode-segmenter/grapheme, but it runs the boundary rules directly without allocating segment objects or slicing strings, so counting is 4+ times faster and zero heap usage.

It is a separate entry so that counting doesn't have to carry the segmenter code, and vice versa.

Example: Count graphemes fast

import { countGraphemes } from 'unicode-segmenter/grapheme-counter';

countGraphemes('๐Ÿ‘‹ ์•ˆ๋…•!');
// => 5

Note

The counter module contains duplicate code from graphemeSegments().

If you want a smaller bundle size and performance multipliers are not very important, use the unicode-segmenter/grapheme module instead.

Full Segmenter is already optimized enough.

Export unicode-segmenter/intl-adapter

Intl.Segmenter API adapter (only granularity: "grapheme" available yet)

import { Segmenter } from 'unicode-segmenter/intl-adapter';

// Same API with the `Intl.Segmenter`
const segmenter = new Segmenter();

Export unicode-segmenter/intl-polyfill

Intl.Segmenter API polyfill (only granularity: "grapheme" available yet)

// Apply polyfill to the `globalThis.Intl` object.
import 'unicode-segmenter/intl-polyfill';

const segmenter = new Intl.Segmenter();

Export unicode-segmenter/emoji

Utilities for matching emoji-like characters.

Example: Use Unicode emoji property matches

import {
  isEmojiPresentation,    // match \p{Emoji_Presentation}
  isExtendedPictographic, // match \p{Extended_Pictographic}
} from 'unicode-segmenter/emoji';

isEmojiPresentation('๐Ÿ˜'.codePointAt(0));
// => true
isEmojiPresentation('โ™ก'.codePointAt(0));
// => false

isExtendedPictographic('๐Ÿ˜'.codePointAt(0));
// => true
isExtendedPictographic('โ™ก'.codePointAt(0));
// => true

Export unicode-segmenter/general

Utilities for matching alphanumeric characters.

Example: Use Unicode general property matchers

import {
  isLetter,       // match \p{L}
  isNumeric,      // match \p{N}
  isAlphabetic,   // match \p{Alphabetic}
  isAlphanumeric, // match [\p{N}\p{Alphabetic}]
} from 'unicode-segmenter/general';

Runtime Compatibility

unicode-segmenter uses only fundamental features of ES2015, making it compatible with most browsers.

To ensure compatibility, the runtime should support:

If the runtime doesn't support these features, it can easily be fulfilled with tools like Babel.

React Native Support

Since Hermes doesn't support the Intl.Segmenter API yet, unicode-segmenter is a good alternative.

unicode-segmenter is compiled into small & efficient Hermes bytecode than other JavaScript libraries. See the benchmark for details.

Comparison

unicode-segmenter aims to be lighter and faster than alternatives in the ecosystem while fully spec compliant. So the benchmark is tracking several libraries' performance, bundle size, and Unicode version compliance.

unicode-segmenter/grapheme vs

JS Bundle Stats

Name Unicodeยฎ ESM? Size Size (min) Size (min+gzip) Size (min+br) Size (min+zstd)
unicode-segmenter/grapheme 17.0.0 โœ”๏ธ 9,009 5,171 2,497 2,249 2,544
unicode-segmenter/grapheme-counter 17.0.0 โœ”๏ธ 7,912 4,852 2,353 2,120 2,396
unicode-segmenter/grapheme + grapheme-counter 17.0.0 โœ”๏ธ 9,985 5,489 2,571 2,310 2,612
graphemer 15.0.0 โœ–๏ธ ๏ธ 410,435 95,104 15,752 10,660 15,911
grapheme-splitter 10.0.0 โœ–๏ธ 122,254 23,682 7,852 4,802 6,753
@formatjs/intl-segmenter* 17.0.0 โœ–๏ธ 268,301 176,759 45,988 31,701 45,370
unicode-segmentation* 15.1.0 - 56,529 52,439 24,108 17,343 24,375
Intl.Segmenter* - - 0 0 0 0 0
  • @formatjs/intl-segmenter handles grapheme, word, and sentence, but it's not tree-shakable.
  • unicode-segmentation size contains only minimum WASM binary and its bindings to execute benchmarking. It will increases to expose more features.
  • Intl.Segmenter's Unicode data depends on the host, and may not be up-to-date.
  • Intl.Segmenter may not be available in some old browsers, edge runtimes, or embedded environments.

Hermes Bytecode Stats

Name Bytecode size Bytecode size (gzip)*
unicode-segmenter/grapheme 19,721 10,874
unicode-segmenter/grapheme-counter 18,210 10,069
unicode-segmenter/grapheme + grapheme-counter 20,667 11,371
graphemer 134,085 31,770
grapheme-splitter 63,942 19,165
@formatjs/intl-segmenter 329,547 136,751
  • The installation size contains compressed assets.

Memory Stats

Retained memory per library after segmenting the benchmark corpus, measured by yarn memory-stats:grapheme in isolated Node.js processes (median of 5, GC-stabilized deltas).

Name JS heap External*
unicode-segmenter/grapheme 209 kB 42 kB
graphemer 2.32 MB -
grapheme-splitter 570 kB -
@formatjs/intl-segmenter 1.95 MB -
unicode-segmentation* 214 kB -
Intl.Segmenter* 86.5 kB -
  • "External" is the extra ArrayBuffer memory relative to the ~40 kB runtime baseline shared by every row; unicode-segmenter keeps its lookup tables in typed arrays, which live there instead of the JS heap.
  • unicode-segmentation's WASM linear memory and Intl.Segmenter's ICU data are allocated by native code and invisible to both columns.

Runtime Performance

Here is a brief explanation, and you can see archived benchmark results.

Performance in Node.js/Bun/Deno: unicode-segmenter/grapheme has best-in-class performance.

Performance in Browsers: The performance in browser environments varies greatly due to differences in browser engines, which makes benchmarking inconsistent, but:

  • Still significantly faster than other JavaScript libraries.
  • Generally outperforms the built-in in the most browser environments, except the Firefox.

Performance in React Native: unicode-segmenter/grapheme is still faster than alternatives when compiled to Hermes bytecode. It's ~4x faster than graphemer and ~34x faster than grapheme-splitter.

Performance in QuickJS: unicode-segmenter/grapheme is the only usable library in terms of performance.

Instead of trusting these claims, you can try yarn perf:grapheme directly in your environment or build your own benchmark.

Acknowledgments

LICENSE

MIT

About

A lightweight implementation of the Unicode Text Segmentation (UAX #29)

Topics

Resources

License

Stars

105 stars

Watchers

3 watching

Forks

Contributors