New bug when test samples

[timeconscious]

When we have large number of samples, there is a new error as follows:

"Assertion failed: (segments_without_special_tokens[n] > 0), function split, file src/npylm/sentence.cpp, line 131.
zsh: abort python3 viterbi.py --input-filename test.txt --working-directory "

We use the UTF-8 type of samples. Could you give us some suggestions how to figure this problem out?

[musyoku]

there are probably empty lines in your text file

[timeconscious]

Thank you for your relpy. The bug is solved according to your suggestion.