tesseract: tesstrain.sh script exits with error

Dear @Shreeshrii ,

thank you very much for your help! The training worked beautifully after rewrapping the corpus (I wrote a short script in Python 3, as it works beautifully with utf-8 documents; you can find it here)

I had to rewrap the corpus by 35 characters per line. The widths of 10, 20, 30, 40, 50, 60 and 70 characters per line did not work. But this is another issue, I think.

To be honest I both followed and disregarded your advice about using training scripts for Tesseract4 only at the same time. As a matter of fact, I need to use Tesseract3, thus I tested the old training scripts first. Nevertheless, I created a Docker container with Ubuntu Bionic and ran the training script for and with Tesseract4. It worked as well as with Tesseract3.

Hence we can regard the rewrapping of a corpus file as an official workaround for this issue. Shall I edit the wiki pages about training Tesseract3 and Tesseract4?

valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes --log-file=./valgrind-out.txt /usr/local/bin/text2image --fontconfig_tmpdir=/tmp/font_tmp.opjpN7f94T --fonts_dir=…/.fonts --strip_unrenderable_words --leading=32 --char_spacing=0.0 --exposure=-1 --outputbase=/tmp/tmp.tH84CXo6fq/akk/akk.CuneiformOB.exp-1 --max_pages=0 --font=CuneiformOB --text=./langdata/akk/corpus-12pt.txt

valgrind-out.txt

~/tessdata_akk$ valgrind --tool=memcheck /usr/local/bin/text2image --fontconfig_tmpdir=/tmp/font_tmp.opjpN7f94T --fonts_dir=../.fonts --strip_unrenderable_words --leading=32 --char_spacing=0.0 --exposure=-1 --outputbase=/tmp/tmp.tH84CXo6fq/akk/akk.CuneiformOB.exp-1 --max_pages=0 --font=CuneiformOB --text=./langdata/akk/corpus-12pt.txt
==3121== Memcheck, a memory error detector
==3121== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==3121== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==3121== Command: /usr/local/bin/text2image --fontconfig_tmpdir=/tmp/font_tmp.opjpN7f94T --fonts_dir=../.fonts --strip_unrenderable_words --leading=32 --char_spacing=0.0 --exposure=-1 --outputbase=/tmp/tmp.tH84CXo6fq/akk/akk.CuneiformOB.exp-1 --max_pages=0 --font=CuneiformOB --text=./langdata/akk/corpus-12pt.txt
==3121==
Rendered page 0 to file /tmp/tmp.tH84CXo6fq/akk/akk.CuneiformOB.exp-1.tif
Rendered page 1 to file /tmp/tmp.tH84CXo6fq/akk/akk.CuneiformOB.exp-1.tif
Rendered page 2 to file /tmp/tmp.tH84CXo6fq/akk/akk.CuneiformOB.exp-1.tif
Rendered page 3 to file /tmp/tmp.tH84CXo6fq/akk/akk.CuneiformOB.exp-1.tif
Rendered page 4 to file /tmp/tmp.tH84CXo6fq/akk/akk.CuneiformOB.exp-1.tif
cluster_text.size() == start_byte_to_box.size():Error:Assert failed:in file ../../../src/training/stringrenderer.cpp, line 546
==3121==
==3121== Process terminating with default action of signal 5 (SIGTRAP)
==3121==    at 0x101D7844: ERRCODE::error(char const*, TessErrorLogCode, char const*, ...) const (errcode.cpp:84)
==3121==    by 0x10027487: tesseract::StringRenderer::ComputeClusterBoxes() (stringrenderer.cpp:546)
==3121==    by 0x10028617: tesseract::StringRenderer::RenderToImage(char const*, int, Pix**) (stringrenderer.cpp:806)
==3121==    by 0x10015E1B: Main() (text2image.cpp:632)
==3121==    by 0x1000A0FB: main (text2image.cpp:736)
==3121==
==3121== HEAP SUMMARY:
==3121==     in use at exit: 141,636,503 bytes in 11,076 blocks
==3121==   total heap usage: 258,218 allocs, 247,142 frees, 1,652,783,976 bytes allocated
==3121==
==3121== LEAK SUMMARY:
==3121==    definitely lost: 7,168 bytes in 11 blocks
==3121==    indirectly lost: 12,207 bytes in 474 blocks
==3121==      possibly lost: 1,514 bytes in 21 blocks
==3121==    still reachable: 141,613,430 bytes in 10,559 blocks
==3121==                       of which reachable via heuristic:
==3121==                         length64           : 80 bytes in 2 blocks
==3121==                         newarray           : 1,568 bytes in 18 blocks
==3121==         suppressed: 0 bytes in 0 blocks
==3121== Rerun with --leak-check=full to see details of leaked memory
==3121==
==3121== For counts of detected and suppressed errors, rerun with: -v
==3121== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
Trace/breakpoint trap

@zdenop The first post about issue has the info you want.

Attachments:

I attached all files used: akktrain.zip.

The fonts are hosted here, but for the sake of completeness the .ttf-files are included in the archive; they shoud be moved to /usr/share/fonts.

@wincentbalin had a workaround the problem too…

Hence we can regard the rewrapping of a corpus file as an official workaround for this issue. Shall I edit the wiki pages about training Tesseract3 and Tesseract4?

@stweil : can you provide font and text that failed for you?

Issue #765 is a duplicate.

Currently the WER is around 10 per cent, but sometimes I got it lower.

I looked at the LSTM training results that I have. They have CER of less than 10% and WER of 15%.

See note above code section that cause crash:

  // There is a subtle bug in the cluster text reported by the PangoLayoutIter
  // on ligatured characters (eg. The word "Lam-Aliph" in arabic). To work
  // around this, we use text reported using the PangoGlyphIter which is
  // accurate.
  // TODO(ranjith): Revisit whether this is still needed in newer versions of
  // pango.

Is this with tesseract 3.05? Training for legacy engine?

I had done a training run for LSTM but didn’t test it. I will share it in a day or two, I am traveling now.

If you can make your test images and ground truth available some place, I can check accuracy too.

On Wed, 19 Sep 2018, 20:53 Wincent Balin, notifications@github.com wrote:

What kind of accuracy are you getting with the akk traineddata with tesseract3?

Currently the WER is around 10 per cent, but sometimes I got it lower.

The program I use takes random words from the wordlist, creates texts from it and saves the text into an image using text2image. Then tesseract is used to recognize the text back and the WER of the result in comparison to the original text is calculated.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tesseract-ocr/tesseract/issues/1781#issuecomment-422845120, or mute the thread https://github.com/notifications/unsubscribe-auth/AE2_owq8wFdtZIlOIjSlTHb65rpmAAqyks5ucmFrgaJpZM4VR-Rj .

What kind of accuracy are you getting with the akk traineddata with tesseract3?

Currently the WER is around 10 per cent, but sometimes I got it lower. I think it requires some tinkering.

The program I use takes random words from the wordlist, creates texts from it and saves the text into an image using text2image. Then tesseract is used to recognize the text back and the WER of the result in comparison to the original text is calculated.

https://en.wikipedia.org/wiki/Cuneiform_script

It will be hard to get a good accuracy on >1900 years old material.

we can regard the rewrapping of a corpus file as an official workaround for this issue.

OK

The widths of 10, 20, 30, 40, 50, 60 and 70 characters per line did not work. But this is another issue, I think. I had to rewrap the corpus by 35 characters per line.

I think 35 characters per line is dependent on the size of akkadian characters in the fonts that were used. Don’t think that will be the case globally for all languages.

As a matter of fact, I need to use Tesseract3, thus I tested the old training scripts first.

Sure. My suggestion was geared more towards using tesseract4.

What kind of accuracy are you getting with the akk traineddata with tesseract3?

On Thu, Aug 2, 2018 at 2:01 AM Wincent Balin notifications@github.com wrote:

Dear @Shreeshrii https://github.com/Shreeshrii ,

thank you very much for your help! The training worked after rewrapping (I wrote a short script in Python 3, as it works beautifully with utf-8 documents; you can find it here https://gist.github.com/wincentbalin/85707ce703b1e4bc0737ed569fb16bea), but I had to rewrap the corpus by 35 characters per line.

The widths of 10, 20, 30, 40, 50, 60 and 70 characters per line did not work. But this is another issue, I think.

To be honest I both followed and disregarded your advice about using training scripts for Tesseract4 only at the same time. As a matter of fact, I need to use Tesseract3, thus I tested the old training scripts first. Nevertheless, I created a Docker container with Ubuntu Bionic and ran the training script for and with Tesseract4. It worked as well as with Tesseract3.

Hence we can regard the rewrapping of a corpus file as an official workaround for this issue. Shall I edit the wiki pages about training with Tesseract3 and Tesseract4?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tesseract-ocr/tesseract/issues/1781#issuecomment-409711949, or mute the thread https://github.com/notifications/unsubscribe-auth/AE2_ozf7-j7YdFFuBuPcW_6EdZexO-Ckks5uMhAkgaJpZM4VR-Rj .

–

भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

The issue #765 seems to be (roughly) related to this one.

@ stweil This assert seems to be related to the size of training_text line. Is it possible to have a more descriptive error message, if that is indeed the case.

cluster_text.size() == start_byte_to_box.size():Error:Assert failed:in file stringrenderer.cpp, line 541

I used notepad++ on windows. Yes, it displays correctly after changing encoding to utf-8.

I think the problem is being caused by extra long lines. I saved file as utf-8 and split the long lines to a smaller size and it seems to be working ok.

Make sure you are using the new version of tesstrain.sh (uninstall the version from tesseract 3).

file corpus-12pt.txt recognizes UTF-8 encoding. Which software do you use to look at the text?

If you choose the right encoding, this image should appear:

Hello @Shreeshrii,

I ran the tesstrain.sh with the same options under Ubuntu Bionic (in a Docker container) and got the same results, as well as the attached coredump.

The version information is

tesseract 4.0.0-beta.1
 leptonica-1.75.3
  libgif 5.1.4 : libjpeg 8d (libjpeg-turbo 1.5.2) : libpng 1.6.34 : libtiff 4.0.9 : zlib 1.2.11 : libwebp 0.6.1 : libope
njp2 2.3.0

 Found AVX2
 Found AVX
 Found SSE

P.S.: the language-specific.sh script is attached too.

Please try with latest version of tesseract.