Create a SizeUnit enum (similar to TimeUnit)

[jtnelson]

To help converting sizes across representations like GB, TB, etc. Then have a method that will return the best representation that is most human readable

private static String formatBytes(long bytes) {
        Preconditions.checkArgument(bytes > 0);
        double unit = 1000;
        if(bytes < unit) {
            return bytes + " B";
        }

        String[] units = { "KB", "MB", "GB", "TB", "PB", "EB" };
        int unitIndex = (int) (Math.log10(bytes) / Math.log10(unit)) - 1;
        unitIndex = Math.min(unitIndex, units.length - 1);

        // Calculate the value in the determined unit
        double value = bytes / Math.pow(unit, unitIndex + 1);

        // Format with up to 2 decimal places, removing trailing zeros
        if(value == (long) value) {
            return String.format("%d %s", (long) value, units[unitIndex]);
        }
        else {
            return String.format("%.2f %s", value, units[unitIndex])
                    .replaceAll("\\.00$", "").replaceAll("(\\.[0-9])0$", "$1");
        }
    }

Once this is implemented, replace the code in LargeTermIndexDeduplicator (in Concourse)