BUG: Logstash in Python 3, byte to string in handler_tcp.py

[israel-fl]

The following error is thrown when configuring logstash for Gunicorn.

Traceback (most recent call last):                                                                                          
  File "/usr/lib/python3.5/logging/handlers.py", line 621, in emit                                                          
    s = self.makePickle(record)                                                                                             
  File "~/Projects/env/lib/python3.5/site-packages/logstash/handler_tcp.py", line 25, in makePickle    
    return self.formatter.format(record) + b'\n'                                                                            
TypeError: Can't convert 'bytes' object to str implicitly                                                                   

Changing handler_tcp.py to:

from logging.handlers import DatagramHandler, SocketHandler
from logstash import formatter


# Derive from object to force a new-style class and thus allow super() to work
# on Python 2.6
class TCPLogstashHandler(SocketHandler, object):
    """Python logging handler for Logstash. Sends events over TCP.
    :param host: The host of the logstash server.
    :param port: The port of the logstash server (default 5959).
    :param message_type: The type of the message (default logstash).
    :param fqdn; Indicates whether to show fully qualified domain name or not (default False).
    :param version: version of logstash event schema (default is 0).
    :param tags: list of tags for a logger (default is None).
    """

    def __init__(self, host, port=5959, message_type='logstash', tags=None, fqdn=False, version=0):
        super(TCPLogstashHandler, self).__init__(host, port)
        if version == 1:
            self.formatter = formatter.LogstashFormatterVersion1(message_type, tags, fqdn)
        else:
            self.formatter = formatter.LogstashFormatterVersion0(message_type, tags, fqdn)

    def makePickle(self, record):
        return str.encode(self.formatter.format(record)) + b'\n'

Specifically the last line:

return str.encode(self.formatter.format(record)) + b'\n'

Encoding the string to bytes allows the library to run.

Didn't want to submit a pull request in case you or someone else had a better solution.

[SalahAdDin]

@israel-fl I have the same problem now, with python 3.6.

Which your solution?

[israel-fl]

@SalahAdDin I've modified the handler_tcp.py file to encode the string. You can actually copy and paste this block into your file

from logging.handlers import DatagramHandler, SocketHandler
from logstash import formatter


# Derive from object to force a new-style class and thus allow super() to work
# on Python 2.6
class TCPLogstashHandler(SocketHandler, object):
    """Python logging handler for Logstash. Sends events over TCP.
    :param host: The host of the logstash server.
    :param port: The port of the logstash server (default 5959).
    :param message_type: The type of the message (default logstash).
    :param fqdn; Indicates whether to show fully qualified domain name or not (default False).
    :param version: version of logstash event schema (default is 0).
    :param tags: list of tags for a logger (default is None).
    """

    def __init__(self, host, port=5959, message_type='logstash', tags=None, fqdn=False, version=0):
        super(TCPLogstashHandler, self).__init__(host, port)
        if version == 1:
            self.formatter = formatter.LogstashFormatterVersion1(message_type, tags, fqdn)
        else:
            self.formatter = formatter.LogstashFormatterVersion0(message_type, tags, fqdn)

    def makePickle(self, record):
        return str.encode(self.formatter.format(record)) + b'\n'

Alternatively, I've made a fork of this repo and updated it to work with Python3. You can run

pip3 install python3-logstash

to install it.

[SalahAdDin]

@israel-fl I made a PR, hahahaha

What's about https://github.com/eht16/python-logstash-async

[miso-belica]

I believe there is a duplicate report #70.

Also, I found PR #72 and #78. But according to #72 (comment) I guess the project is no longer active and maybe it'đ worth to try https://github.com/eht16/python-logstash-async or some fork.

[vklochan]

The project wasn't supported for some time. I'm going to review the issue this month.

[wizardion]

TypeError: Can't convert 'bytes' object to str implicitly

I believe this is an issue with Python 3.5/Python 3.6. Can you confirm that this problem still exists within the 3.7.x version?
Or probably, there are some suggestions about upgrading ulule/python-logstash-formatter#41 to 0.5.16 should resolve the issue.

[Chris2048]

Above may fix TCPLogstashHandler, but what about UDPLogstashHandler?

This appears to have been fixed in ce0df09 by adding the bytes-encoding in the formatter, but I wonder if it should be added to the makePickle fns instead? It makes it harder to modify the formatted by just using strings as the formatter is also encoding the bytes.

[sibiryoff]

TypeError: Can't convert 'bytes' object to str implicitly

I believe this is an issue with Python 3.5/Python 3.6. Can you confirm that this problem still exists within the 3.7.x version?
Or probably, there are some suggestions about upgrading ulule/python-logstash-formatter#41 to 0.5.16 should resolve the issue.

Problem still exists in 3.7. I'm using JSON-log-formatter.

[LombardiDaniel]

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/logging/handlers.py", line 673, in emit
    s = self.makePickle(record)
  File "/home/daniel/Connector/venv/lib/python3.10/site-packages/logstash/handler_tcp.py", line 25, in makePickle
    return self.formatter.format(record) + b'\n'
TypeError: can only concatenate str (not "bytes") to str

Problem still present with python 3.10.5

[nofacer]

3.11 with same issue as well