[feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer by nishant94 · Pull Request #64667 · apache/doris

nishant94 · 2026-06-22T05:53:45Z

What problem does this PR solve?

Issue Number: #64646

Related PR: None

Problem Summary:
Doris has no Japanese-aware tokenizer for the inverted index. Japanese text has no spaces between words, so the existing parsers can't segment it and MATCH / MATCH_PHRASE on Japanese columns end up with poor recall and precision.

This PR adds a built-in kuromoji parser for Japanese, in the same style as the existing Chinese IK analyzer. It's opt-in per column:

 INDEX content_idx (`content`) USING INVERTED
 PROPERTIES("parser" = "kuromoji", "parser_mode" = "search");

After indexing, MATCH, MATCH_PHRASE and TOKENIZE() run against the segmented Japanese terms.

How it works:

Native C++ under be/src/storage/index/inverted/analyzer/kuromoji/, so there's no JVM on the indexing path. KuromojiAnalyzer / KuromojiTokenizer mirror the IK analyzer/tokenizer, with a Viterbi cost-model segmenter over the IPADIC connection-cost matrix.
- The dictionary is a process-wide singleton loaded once from ${inverted_index_dict_path}/kuromoji. An offline converter compiles raw IPADIC into a compact C++ runtime format (double-array trie + cost matrix + char/unknown tables) at build time, so no binary blob is committed.
- search (default), normal and extended modes are supported. No thrift/proto changes — parser and mode ride as strings in the index properties.

Dictionary source is mecab-ipadic-2.7.0-20070801 (NAIST-2003 license, the same lexicon Lucene kuromoji uses).

Release note

Support Japanese text tokenization in the inverted index via a new kuromoji parser (PROPERTIES("parser"="kuromoji")), with search/normal/extended modes.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)

  CREATE TABLE test_jp (
    id BIGINT,
    content TEXT,
    INDEX idx_content (content) USING INVERTED
      PROPERTIES("parser" = "kuromoji", "parser_mode" = "search")
  ) ENGINE=OLAP
  DUPLICATE KEY(id)
  DISTRIBUTED BY HASH(id) BUCKETS 1
  PROPERTIES("replication_num" = "1");

  INSERT INTO test_jp VALUES
    (1, '東京都に住んでいます'),
    (2, '日本語の形態素解析エンジン');

  -- search-mode decompounding: 東京都 also matches 東京
  SELECT id FROM test_jp WHERE content MATCH '東京';          -- expect: 1
  SELECT id FROM test_jp WHERE content MATCH_PHRASE '形態素解析'; -- expect: 2

  -- inspect segmentation directly
  SELECT TOKENIZE('東京都に住んでいます', '"parser"="kuromoji","parser_mode"="search"');

Behavior changed:
- No.
- Yes. It adds a new opt-in kuromoji parser. Existing parsers and their output are unchanged; the new behavior only applies to indexes that explicitly set parser="kuromoji".
Does this need documentation?
- No.
- Yes. PR Link to Doris-Website.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-06-22T05:53:50Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

nishant94 · 2026-06-22T06:23:33Z

run buildall

yiguolei · 2026-06-22T06:27:35Z

@nishant94 have you tried icu analyzer? because I think icu could handle many different languages.

nishant94 · 2026-06-22T06:31:13Z

@nishant94 have you tried icu analyzer? because I think icu could handle many different languages.

@yiguolei The ICU Analyzer is not good as the Kuromoji. There is huge difference between icu and kuromoji when it comes to morphology of the Japanese words. So I think it worth it adding this new parser.

BiteTheDDDDt · 2026-06-22T07:02:18Z

Is the code under be/src/storage/index/inverted/analyzer/kuromoji entirely original or derived from other projects? Perhaps we need to clarify the situation regarding this part.

nishant94 · 2026-06-22T07:29:43Z

Is the code under be/src/storage/index/inverted/analyzer/kuromoji entirely original or derived from other projects? Perhaps we need to clarify the situation regarding this part.

This is original code but it is modeled on Apache Lucene's kuromoji.

hello-stephen · 2026-06-22T07:56:03Z

FE UT Coverage Report

Increment line coverage 44.44% (4/9) 🎉
Increment coverage report
Complete coverage report

nishant94 · 2026-06-22T09:58:03Z

run buildall

hello-stephen · 2026-06-22T12:00:26Z

BE UT Coverage Report

Increment line coverage 82.40% (791/960) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.51% (21439/39327)
Line Coverage	38.17% (205347/537919)
Region Coverage	34.16% (161044/471416)
Branch Coverage	35.14% (70517/200651)

hello-stephen · 2026-06-22T15:49:21Z

BE UT Coverage Report

Increment line coverage 84.10% (836/994) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.50% (21433/39329)
Line Coverage	38.13% (205092/537920)
Region Coverage	34.11% (160793/471446)
Branch Coverage	35.11% (70468/200678)

hello-stephen · 2026-06-22T16:58:18Z

BE Regression && UT Coverage Report

Increment line coverage 83.85% (462/551) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.11% (28441/38375)
Line Coverage	58.02% (309954/534209)
Region Coverage	54.69% (258833/473301)
Branch Coverage	56.10% (112608/200725)

hello-stephen · 2026-06-22T17:05:28Z

FE Regression Coverage Report

Increment line coverage 66.67% (6/9) 🎉
Increment coverage report
Complete coverage report

nishant94 · 2026-06-23T06:16:44Z

run buildall

hello-stephen · 2026-06-23T11:38:51Z

FE UT Coverage Report

Increment line coverage 44.44% (4/9) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-23T11:55:51Z

FE Regression Coverage Report

Increment line coverage 35.29% (6/17) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-23T14:39:39Z

BE UT Coverage Report

Increment line coverage 84.10% (836/994) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.72% (21523/39332)
Line Coverage	38.18% (205493/538169)
Region Coverage	34.17% (161179/471738)
Branch Coverage	35.13% (70561/200832)

hello-stephen · 2026-06-23T14:44:04Z

BE Regression && UT Coverage Report

Increment line coverage 83.85% (462/551) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.19% (28466/38371)
Line Coverage	58.03% (310151/534436)
Region Coverage	54.77% (259367/473580)
Branch Coverage	56.13% (112755/200875)

nishant94 · 2026-06-24T05:18:18Z

run buildall

hello-stephen · 2026-06-24T06:46:43Z

FE UT Coverage Report

Increment line coverage 44.44% (4/9) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-24T10:23:58Z

BE UT Coverage Report

Increment line coverage 84.20% (842/1000) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.58% (21476/39348)
Line Coverage	38.09% (205045/538313)
Region Coverage	34.07% (160754/471838)
Branch Coverage	35.04% (70387/200890)

hello-stephen · 2026-06-24T14:28:27Z

BE Regression && UT Coverage Report

Increment line coverage 84.02% (468/557) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.22% (28491/38387)
Line Coverage	58.10% (310621/534591)
Region Coverage	55.02% (260603/473686)
Branch Coverage	56.29% (113100/200937)

- Introduced a new configuration option `enable_kuromoji_analyzer` to toggle the Kuromoji analyzer functionality. - Updated unit tests to validate the behavior of the Kuromoji analyzer when enabled and disabled. - Modified tests to enable the Kuromoji analyzer for specific test cases.

- Updated the namespace for Kuromoji components from `doris::segment_v2::kuromoji` to `doris::segment_v2::inverted_index::kuromoji` across multiple files for better organization and clarity.

…ictionary

- Updated the CMake configuration to ensure the required Kuromoji dictionary files are present at build time, failing the build if any are missing. - Modified the KuromojiAnalyzer and KuromojiTokenizer to throw exceptions when the dictionary is not loaded, preventing silent fallbacks to per-codepoint tokenization. - Improved error handling and validation in the dictionary loading process to ensure robust operation. - Updated unit tests to validate the new behavior, ensuring that missing dictionaries trigger appropriate errors.

- Replaced the `ending_at` vector with `end_head` and `end_next` for better memory management and performance during node processing. - Updated node addition and traversal logic to utilize the new data structures, enhancing the segmenter's efficiency in handling word segmentation.

- Changed the values in the `unk.per_category[CAT_DEFAULT]` entry from `{5, 5, 4769, "unk-default"}` to `{2, 2, 4769, "unk-default"}` to correct the test setup.

- Modified CMake configuration to conditionally include the Kuromoji dictionary files only for non-test builds (MAKE_TEST=ON). - Adjusted the custom target for generating the Kuromoji dictionary to reflect the new conditional behavior, ensuring it remains a manual target during unit-test builds.

- Added checks for empty trie and out-of-range category mappings in the Kuromoji dictionary. - Updated tests

- Added logic to return the Kuromoji search mode based on the analyzer property. - Updated unit tests accordingly.

- This enhancement ensures that the necessary Kuromoji dictionary source is available for builds, improving the setup process for users.

- Updated test cases in `test_japanese_analyzer.groovy` to use query-time (qt_) assertions for better readability and maintainability.

- Introduced a penalty calculation mechanism for segmenting words based on the analyzer mode, improving the accuracy of word segmentation. - Added caching for the current same-category run's byte end to optimize processing of grouped unknown words, reducing unnecessary rescans.

- Moved mecab-ipadic staging into a conditional block based on the BUILD_BE flag.

- Refactored the logic for setting the Kuromoji analyzer mode to ensure it defaults to the search mode when the parser mode is empty or set to coarse granularity.

- Introduced sorting of CSV file paths before processing to ensure consistent order. - Enhanced error handling during lexicon parsing, including detailed error messages for failed parses. - Added sorting of BuilderWord entries by cost and IDs.

- Implemented a new utility function to decode the first UTF-8 code point from a given string view, enhancing character processing capabilities. - Updated the KuromojiTokenizer to skip unknown tokens based on their character category, improving tokenization accuracy.

- Introduced a new executable `kuromoji_build_dict` to compile the UTF-8 mecab-ipadic source into binary files. - Added custom commands and targets to generate the Kuromoji dictionary, ensuring it is built as part of the overall project unless in test mode.

- Updated the CMake configuration to prepend the Kuromoji JVM library directory to the DYLD_LIBRARY_PATH and LD_LIBRARY_PATH environment variables during the dictionary generation process.

- Updated CMake configuration to set DYLD_LIBRARY_PATH and LD_LIBRARY_PATH for better library management during dictionary generation. - Improved README documentation for manual dictionary regeneration steps. - Refactored the Kuromoji dictionary loading logic to cache only successful loads, enhancing error handling. - Added assertions in the Japanese analyzer tests to ensure proper configuration and cleanup of the Kuromoji analyzer state.

- Replaced `qt_` queries with direct SQL calls and assertions to validate results in the Japanese analyzer test suite.

- Updated the build script to allow compilation of the Kuromoji dictionary when the COMPILE_BENCH flag is set. - Modified the dictionary building logic to write temporary files before renaming them atomically, improving reliability during the build process. - Ensured that the mecab-ipadic staging in the third-party build script uses a temporary directory for safer file operations.

nishant94 · 2026-08-11T04:07:07Z

run buildall

hello-stephen · 2026-08-11T10:03:01Z

FE Regression Coverage Report

Increment line coverage 7.89% (6/76) 🎉
Increment coverage report
Complete coverage report

yiguolei · 2026-08-11T10:50:07Z

/review

github-actions

Request changes: three distinct correctness/build-isolation issues remain; the inline comments give concrete triggers and fix/test directions.

Critical checkpoint conclusions:

Goal/proof: the PR adds an opt-in Kuromoji/IPADIC analyzer across DDL, index writes, MATCH-family queries, and TOKENIZE. The main Japanese morphology path is wired and tested, but Unicode lowercasing, malformed-byte handling, and build-output provenance are not correct.
Scope/focus: the 51-file change is broad but cohesive across FE metadata, BE runtime, dictionary compilation/packaging, and tests. No additional review focus was supplied, so the complete scope was reviewed.
Concurrency: runtime dictionary publication is mutex-protected and successful mappings are immutable; analyzer/tokenizer state is instance-local. Independent build trees can race or reuse the shared generated dictionary directory (inline).
Lifecycle/static initialization: factory, load, cache, mmap, and error lifecycles were traced. No new static-initialization issue was found; existing failure-cache and mmap-accounting concerns already have live threads.
Configuration: the mutable default-off gate reaches writer, query, and TOKENIZE construction paths. No distinct gate-refresh issue was found beyond existing threads.
Compatibility: no wire enum or storage-format version is added, but string metadata crosses FE/BE and rolling upgrades. Existing custom-name, default-mode, analyzer-validation, and mixed-version concerns were not duplicated.
Parallel paths: writer, indexed and slow MATCH/phrase paths, and TOKENIZE were traced. The lowercasing and malformed-byte findings affect the shared analyzer path.
Conditional checks: mode, property, gate, parser, loader, and dictionary validation branches were reviewed; no additional distinct conditional-check defect remains.
Test coverage: synthetic BE, FE validation, real-dictionary, and regression paths were inspected. Unicode case, invalid-byte, and two-build-tree provenance coverage is missing as described inline; other missing coverage already has live comments.
Test results: the changed expected errors and query assertions were inspected. Existing deterministic-output and config-restoration concerns were suppressed as duplicates. No build or test was run in this review-only environment.
Observability: load and validation failures log or propagate errors; no new distinct logging/metric gap was found. Existing mmap accounting feedback was not repeated.
Transactions/persistence: catalog property persistence/replay was traced and no transaction/EditLog-specific defect was found.
Data writes: index-term generation and dictionary publication/package installation were reviewed. Wrong shared-build provenance can package incompatible terms (inline); existing multi-file atomic-publication feedback was not duplicated.
FE-BE variables: analyzer/parser/mode properties use existing metadata and thrift maps; DDL, replay, writer, indexed/slow query, and TOKENIZE send/receive paths were checked.
Performance: lattice/OOV complexity, copies, cache locking, and mmap cost were reviewed. Known performance concerns already have live threads; no new distinct issue was found.
Other issues: licensing, vendoring, formatting, registration, and documentation changes were inspected; no further distinct issue remains.

Review completion: converged in round 3. All final normal and risk-focused passes returned NO_NEW_VALUABLE_FINDINGS after validating the three inline findings.

…d validation - Enhanced the `decode_utf8` function to include additional validation for continuation bytes, ensuring proper handling of malformed UTF-8 input. - Updated the `first_codepoint` function in `KuromojiTokenizer` to align with the new decoding logic, improving consistency in character processing. - Introduced a Unicode-aware lowercasing function that utilizes ICU for better text normalization, ensuring compatibility with other analyzers.

nishant94 requested review from BiteTheDDDDt, airborne12 and zclllyybb as code owners June 22, 2026 05:53

nishant94 force-pushed the feat/kuromoji-japanese-analyzer branch from 389fcfb to b79db3c Compare June 22, 2026 09:57

morningman self-assigned this Jun 23, 2026

yiguolei reviewed Jun 24, 2026

View reviewed changes

Comment thread be/src/storage/index/inverted/analyzer/analyzer.cpp

nishant94 force-pushed the feat/kuromoji-japanese-analyzer branch from db0ee69 to 06b4ef6 Compare June 24, 2026 03:59

nishant94 requested a review from yiguolei June 24, 2026 05:18

yiguolei reviewed Jun 26, 2026

View reviewed changes

Comment thread be/dict/kuromoji/README.md

nishant94 added 23 commits August 11, 2026 09:36

fix indentation issues

78d1293

Refactor Kuromoji namespace to inverted_index

2a777f7

- Updated the namespace for Kuromoji components from `doris::segment_v2::kuromoji` to `doris::segment_v2::inverted_index::kuromoji` across multiple files for better organization and clarity.

Update README.md to include Apache License information for Kuromoji d…

36a10e9

…ictionary

Fix comment formatting

65bfa26

Fix test case for Kuromoji dictionary

060b662

- Changed the values in the `unk.per_category[CAT_DEFAULT]` entry from `{5, 5, 4769, "unk-default"}` to `{2, 2, 4769, "unk-default"}` to correct the test setup.

Enhance Kuromoji dictionary validation and error handling

58d766e

- Added checks for empty trie and out-of-range category mappings in the Kuromoji dictionary. - Updated tests

Enhance inverted index parser mode handling

bf0e0a9

- Added logic to return the Kuromoji search mode based on the analyzer property. - Updated unit tests accordingly.

Add mecab-ipadic staging in build script

f7dd828

- This enhancement ensures that the necessary Kuromoji dictionary source is available for builds, improving the setup process for users.

Refactor Japanese analyzer tests for clarity and consistency

6abdb7e

- Updated test cases in `test_japanese_analyzer.groovy` to use query-time (qt_) assertions for better readability and maintainability.

Refactor mecab-ipadic staging logic in build script

fc2d00e

- Moved mecab-ipadic staging into a conditional block based on the BUILD_BE flag.

Update Kuromoji analyzer mode handling in inverted index

e5bc484

- Refactored the logic for setting the Kuromoji analyzer mode to ensure it defaults to the search mode when the parser mode is empty or set to coarse granularity.

Fix BE UT (MacOS) workflow

b330e3b

- Updated the CMake configuration to prepend the Kuromoji JVM library directory to the DYLD_LIBRARY_PATH and LD_LIBRARY_PATH environment variables during the dictionary generation process.

Refactor Japanese analyzer tests to use inline assertions

0bf9cc3

- Replaced `qt_` queries with direct SQL calls and assertions to validate results in the Japanese analyzer test suite.

nishant94 force-pushed the feat/kuromoji-japanese-analyzer branch from c6e4849 to 5bda577 Compare August 11, 2026 04:06

github-actions Bot requested changes Aug 11, 2026

View reviewed changes

Comment thread be/CMakeLists.txt

Comment thread be/src/storage/index/inverted/analyzer/kuromoji/KuromojiTokenizer.cpp

Comment thread be/src/storage/index/inverted/analyzer/kuromoji/kuromoji_viterbi.cpp

nishant94 and others added 2 commits August 18, 2026 17:48

Merge branch 'master' into feat/kuromoji-japanese-analyzer

1eb364d

Uh oh!

Conversation

nishant94 commented Jun 22, 2026

What problem does this PR solve?

Release note

Uh oh!

hello-stephen commented Jun 22, 2026

Uh oh!

nishant94 commented Jun 22, 2026

Uh oh!

yiguolei commented Jun 22, 2026

Uh oh!

nishant94 commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BiteTheDDDDt commented Jun 22, 2026

Uh oh!

nishant94 commented Jun 22, 2026

Uh oh!

hello-stephen commented Jun 22, 2026

FE UT Coverage Report

Uh oh!

nishant94 commented Jun 22, 2026

Uh oh!

hello-stephen commented Jun 22, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 22, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 22, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jun 22, 2026

FE Regression Coverage Report

Uh oh!

nishant94 commented Jun 23, 2026

Uh oh!

hello-stephen commented Jun 23, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 23, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

nishant94 commented Jun 24, 2026

Uh oh!

hello-stephen commented Jun 24, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jun 24, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 24, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

nishant94 commented Aug 11, 2026

Uh oh!

hello-stephen commented Aug 11, 2026

FE Regression Coverage Report

Uh oh!

yiguolei commented Aug 11, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

nishant94 commented Jun 22, 2026 •

edited

Loading