JSON Parse Failure on host Field (could not set field 'ip' on object)

[eren]

Using logstash 8.4.2, I get the following error from logstash when using a simple example with version=1 formatter.

[2022-09-22T12:25:39,507][ERROR][logstash.codecs.json][main][3d55bea51246de48d1dcce7bf9f91a050ba59de18eabf4f05de2ead10f5d45ae] 

JSON parse error, original data now in message field

{:message=>"Could not set field 'ip' on object ‘mymachine.local' to value '127.0.0.1'.This is probably due to trying to set a field like [foo][bar] = someValuewhen [foo] is not either a map or a string", 

:exception=>Java::OrgLogstash::Accessors::InvalidFieldSetException, :data=>"{\"@timestamp\": \"2022-09-22T09:25:39.434Z\", \"@version\": \"1\", \"message\": \"this is a warning\", \"host\": \"mymachine.local\", \"path\": \"/Users/eren/src/log/test.py\", \"tags\": [], \"type\": \"logstash\", \"level\": \"WARNING\", \"logger_name\": \"__main__\"}"}

The root cause of the problem seems to be Could not set field 'ip' on object 'betelgeuse.local' to value '127.0.0.1’.. It looks like logstash tries to add ip to host field, however this field is not a dictionary and is sent by logstash-python as a string.

python-logstash/logstash/formatter.py

Lines 117 to 125 in a795733

	class LogstashFormatterVersion1(LogstashFormatterBase):
	def format(self, record):
	# Create message dict
	message = {
	'@timestamp': self.format_timestamp(record.created),
	'@version': '1',
	'message': record.getMessage(),
	'host': self.host,

When I remove L125 and do not send host field, everything works perfectly. self.host is populated in the base class and it’s understandable that FQDN may be sent. However, If we don’t send host field at all, logstash already adds ip field. Effectively, this code block with default settings is useless:

python-logstash/logstash/formatter.py

Lines 18 to 21 in a795733

	if fqdn:
	self.host = socket.getfqdn()
	else:
	self.host = socket.gethostname()

I do not know how host field is used in the earlier versions of logstash, or if it’s useable at all. However, sending host field in the log message breaks logstash 8.4.2 and it cannot properly write the log. I simply removed sending host field and started using the library with a fork until the issue is resolved.

I’m reporting this issue in the hopes that google will index. I spent a whole day trying to understand logstash, google answers, cryptic error messages etc. If you are seeing this error, simply remove sending host field and you should be good to go until a fix is released.

Best,
Eren

[phoerious]

I can confirm this, but it only happens with the UDP handler and version=1. TCP with both versions and UDP with version=0 works.

[phoerious]

TBH, the message design could also use a general cleaning up to adhere to the ECS spec. logger_name should be log.logger, level should be log.level, host should be log.syslog.hostname etc.

[dailytabs]

Ran into this with UDP+v1. Documented examples don't work.