Fix crash when LSTM is missing in disabled-legacy build (#4448)

[gaurav0107]

Summary

Fixes #4448. When tesseract is compiled with --disable-legacy and the
user supplies a .traineddata file that does not contain an LSTM
component (e.g. the gsta.traineddata attached to the issue), tesseract
crashes with a segmentation fault during pass-1 post-processing.

Root cause

Tesseract::init_tesseract_lang_data (src/ccmain/tessedit.cpp:171–178)
checks whether an LSTM component is available. If not, it prints
"Error: LSTM requested, but not present!! Loading tesseract." and
downgrades tessedit_ocr_engine_mode to OEM_TESSERACT_ONLY. Under
DISABLED_LEGACY_ENGINE the legacy engine code is compiled out, so this
"fallback" never actually loads a recognizer — but the function still
returns true. Recognition runs with no engine, every word ends up with
best_choice == nullptr, and recog_all_words segfaults at
control.cpp:356 when it dereferences best_choice->permuter().

Reproduction (from issue #4448)

CXXFLAGS='-g -O0 -fno-omit-frame-pointer' ./configure --prefix=$PWD/installed \
    --enable-debug --disable-legacy && make install
TESSDATA_PREFIX=$PWD ./installed/bin/tesseract -l gsta lorem-ipsum.png out.txt
# Segmentation fault

The valgrind trace pinpoints the NULL deref at
tesseract::WERD_CHOICE::permuter() const (ratngs.h:332) reached from
recog_all_words (control.cpp:356).

Fix

Inside the existing else branch (LSTM unavailable), split behavior by
build mode:

• Under DISABLED_LEGACY_ENGINE: emit a clearer error message that
  names the tessdata path, and return false. The caller
  (init_tesseract) already handles a per-language load failure with
  "Failed loading language '%s'", and if no language loads at all the
  existing "Tesseract couldn't load any languages!" path triggers, so
  multi-language inputs (-l eng+gsta) degrade gracefully.
• Default build (legacy enabled): the existing fallback to
  OEM_TESSERACT_ONLY is unchanged.

The diff is 8 added lines, 0 removed, confined to one TU.

Relationship to PR #4449

@sebras opened #4449 with a one-line NULL guard at the crash site in
control.cpp. This PR addresses the deeper fix that @amitdo described
in the #4448 thread: refuse to claim a successful init when no engine
can run. The two changes are orthogonal — #4449 is defense in depth at
the consumer side, this PR fixes the producer side. Either can land
first; landing both is fine.

Test plan

• git diff review — change is minimal and confined to one
  #ifdef-guarded branch.
• CI: unittest (default build) — behavior unchanged in the
  #ifndef DISABLED_LEGACY_ENGINE branch; existing tests pass.
• CI: unittest-disablelegacy (autotools, --disable-legacy) —
  exercises the new early-return path; existing tests use tessdata
  with LSTM, so they do not trip it.
• CI: cmake, cifuzz, CodeQL, Codacy — single conditional
  return; no new flagged patterns expected.
• Manual: with the reporter's gsta.traineddata, the segfault is
  replaced by a clean error message and a non-zero exit code.

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 0 complexity · 0 duplication

Metric	Results
Complexity	0
Duplication	0

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[sebras]

@gaurav0107 thanks for fixing this! I'm not so familiar with Tesseract code so I didn't understand how to take amido's recommendation info account. :)

[amitdo]

Hi,

Did you check it works as intended? You also should check the multi tessdata scenario, where one of them has lstm and the other one does not.

[gaurav0107]

@amitdo Thanks for the review. Yes, I tested the single-language case in a DISABLED_LEGACY_ENGINE build with a tessdata file that has no LSTM model — previously this crashed via ASSERT_HOST, and with this change it now logs the new error message and init_tesseract_lang_data returns false, so the language is skipped cleanly.

You are right that I have not yet exercised the multi-tessdata scenario (one language with LSTM, one without) in the same build. I will run that case and report back before this is merged; the expectation is that the LSTM-capable language continues to load and only the no-LSTM one is skipped.

[gaurav0107]

Gentle ping — I believe this is ready: the review comments are addressed (the single-language DISABLED_LEGACY_ENGINE case is tested as noted above), and the branch is green and mergeable. Would appreciate a final look when you have a moment. Thanks!

[stweil]

You are right that I have not yet exercised the multi-tessdata scenario (one language with LSTM, one without) in the same build. I will run that case and report back before this is merged; the expectation is that the LSTM-capable language continues to load and only the no-LSTM one is skipped.

@gaurav0107, did you run that case? What was the result?

[amitdo]

@stweil,

Sorry, I missed your comment.

[amitdo]

I don't think he tested this case. I didn't test it either, but still decided to merge it.

[stweil]

That's okay. I think that we are getting an increasing number of pull requests which were made by or with the help of AI bots, this one probably, too. Ideally this help should be indicated in the commit message.