makeDictionary doesn't allow creating big dictionaries

[marmistrz]

I generated a words.xml and then ran the gradle build task. Then makeDictionary randomly fails:

:makeDictionary
Reading words from input /home/marcin/proj/LanguagePack/dictionary/words.xml
Will store output files under /home/marcin/proj/LanguagePack/src/main/res/raw
Deleting previous versions...
Nodes = 2152096
:makeDictionary FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':makeDictionary'.
> 4194305

Ran it again:

:makeDictionary
Reading words from input /home/marcin/proj/LanguagePack/dictionary/words.xml
Will store output files under /home/marcin/proj/LanguagePack/src/main/res/raw
Deleting previous versions...
Nodes = 3466568
:makeDictionary FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':makeDictionary'.
> java.lang.ArrayIndexOutOfBoundsException (no error message)

Once again, with stacktrace:

:makeDictionary
Reading words from input /home/marcin/proj/LanguagePack.priv/dictionary/words.xml
Will store output files under /home/marcin/proj/LanguagePack.priv/src/main/res/raw
Deleting previous versions...
Nodes = 4781040
:makeDictionary FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':makeDictionary'.
> java.lang.ArrayIndexOutOfBoundsException (no error message)

* Try:
Run with --info or --debug option to get more log output.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':makeDictionary'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:84)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:55)
        at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:61)
        at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
        at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:88)
        at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:45)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:51)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
        at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:233)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:215)
        at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.processTask(AbstractTaskPlanExecutor.java:74)
        at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.run(AbstractTaskPlanExecutor.java:55)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.process(DefaultTaskPlanExecutor.java:32)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:113)
        at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:37)
        at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
        at org.gradle.execution.DefaultBuildExecuter.access$000(DefaultBuildExecuter.java:23)
        at org.gradle.execution.DefaultBuildExecuter$1.proceed(DefaultBuildExecuter.java:43)
        at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)
        at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
        at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:30)
        at org.gradle.initialization.DefaultGradleLauncher$4.run(DefaultGradleLauncher.java:197)
        at org.gradle.internal.Factories$1.create(Factories.java:25)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:91)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:53)
        at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:194)
        at org.gradle.initialization.DefaultGradleLauncher.access$200(DefaultGradleLauncher.java:36)
        at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:118)
        at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:112)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:91)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:63)
        at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:112)
        at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:98)
        at org.gradle.launcher.exec.GradleBuildController.run(GradleBuildController.java:66)
        at org.gradle.tooling.internal.provider.ExecuteBuildActionRunner.run(ExecuteBuildActionRunner.java:28)
        at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
        at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:41)
        at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:26)
        at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:75)
        at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:49)
        at org.gradle.tooling.internal.provider.ServicesSetupBuildActionExecuter.execute(ServicesSetupBuildActionExecuter.java:44)
        at org.gradle.tooling.internal.provider.ServicesSetupBuildActionExecuter.execute(ServicesSetupBuildActionExecuter.java:29)
        at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:67)
        at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
        at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:47)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
        at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:26)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
        at org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:34)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
        at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:74)
        at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:72)
        at org.gradle.util.Swapper.swap(Swapper.java:38)
        at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:72)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
        at org.gradle.launcher.daemon.server.exec.LogAndCheckHealth.execute(LogAndCheckHealth.java:55)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
        at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:60)
        at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
        at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:72)
        at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
        at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:50)
        at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:293)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
        at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
Caused by: java.lang.ArrayIndexOutOfBoundsException

Using gradle 3.2.1, with the 2.2.0 Android gradle plugin.

[marmistrz]

What's interesting, when I limit the words.xml to 49 words, no exception is thrown. My words.xml has 652988 words.
I suspect that one word causes the whole thing to crash (I'm unable to filter all the weird ones)

[marmistrz]

Ok, I see that my frequencies are much bigger than 255. May this be the source of a problem?

I managed to get a more meaningful stack trace: http://pastebin.com/HwiYeb0e

[marmistrz]

Ok, I see, the culprit is the following line: https://github.com/AnySoftKeyboard/AnySoftKeyboardTools/blob/master/makedictionary/src/main/groovy/com/anysoftkeyboard/tools/makedictionary/MakeBinaryDictionary.java#L244

The number of words in a dictionary seems to be a hard limit.

What's interesting, I only have 652988 words in my words.xml

[marmistrz]

Another update: if I have 1024 * 1024 / 4 words in my words.xml, makeDictionary succeeds. If I have 1024 * 1024 / 2 of them - it crashes

[kertase]

I can confirm this issue and it has been existing for a while now see: #4

I think that the dictionary size limit is somehow hard coded to 4mb=4,194,304.

[kertase]

For your info, I tried changing the limit in the code as you can see here: https://github.com/kertase/AnySoftKeyboardTools
The dictionary gets build but app crashes with a dictionary of that size unfortunately so the limit is there for a reason and that is the app can’t manage a bigger dic.

Came across the same problem, hoping for an update

[PanderMusubi]

If this is a duplicate of #4 then please close this issue. Workaround for now is to limit the number of words.