What to do with the Log Guide?

[xpdota]

This discussion was mostly in discord, but I want to get it documented here since it's easier to follow long-running conversations.

Currently, I have a fork of the log guide: https://github.com/xpdota/guide-scratch/blob/master/3pty/LogGuide.md

It would be nice if there were a better central log resource. Currently, the Cactbot log guide is the de-facto resource, but it has a couple issues:

1. Since portions are auto-genned from Cactbot code, it means contributors may need to set up an entire dev environment depending on what kind of change they're making.
2. There was also a problem in the past of not being able to document new fields added to existing lines, because it breaks line parsing for CN/KR until they get the patch.
3. If a field, especially one that doesn't currently have a descriptive name, is renamed, this may break stuff in Cactbot, so it would need the ability to have "alternate" field names or some other backwards compatibility mechanism.
4. The fields currently lack type and representation info (e.g. whether a field is base-10 or base-16).
5. Since they also lack descriptions, adding a new field or clarifying an existing field typically requires manual changes to the log guide anyway.
6. Some fields, such as the flags/value pairs on 21/22, or the trios on 38/42, should almost always be treated as an array of things, rather than querying specific indices (since it makes the logic a lot more brittle even in circumstances where it tends to work).

So, in terms of what to do, there's two options that come to mind:

1. Significantly improve the field defs in Cactbot, including type info (as well as how to convert it, e.g. b10 vs b16, or 1.23 vs 0x3F9D70A4), array types, alt names, and whatever else might be needed. Then, improve the auto-gen with all of that. The advantage is that you might be able to pull more type safety into the actual line parsing as well as reduce boilerplate.
2. Split out the Cactbot-specific guidance from the more general log information. Don't auto-gen the generalized guide. From the standpoint of the change I've made, the auto-gen doesn't cover the parts that require the most effort anyway (like the mess that is ability effects), so it really wouldn't be a problem to just have me or someone else write those portions manually. In addition, it

I think the ideal way would be to represent fields in a table, like this:

Line 21

Index	Data Type	Representation	Cactbot Name*	Description
0	Line Number	Base 10	N/A	Line Number
1	Timestamp	yyyy-MM-dd etc	timestamp	The time at which the event occurred
2	Entity ID	Base 16	sourceId	The caster of the ability
...	...	...	...	...
(8, 10, ...22)	Four Bytes	Base 16	flags1...flags8	For each pair, this determines the type of effect as well as extra effect-specific flags
(9, 11, ...23)	Four Bytes	Base 16	value1...value8	For each pair, this effect-specific value determines the main "value", e.g. damage taken or which status effect was applied

* If going with option 1

I think there's also value in exploring if there's a nice way of representing fields that "go together" (e.g. ability ID + name, or target ID + name + xyz/heading + HP + etc.

[quisquous]

It would be nice if there were a better central log resource. Currently, the Cactbot log guide is the de-facto resource, but it has a couple issues:

I'm not really sure what the advantage is to moving the log guide somewhere else. You list a lot of things, but all these issues (except for number 3) are already solved or are issues that would need to be solved no matter where the log guide lives. And, number 3 has straightforward solutions that we've discussed in discord some. (I'll re-mention them below.)

If the log guide moved somewhere else, you'd need to set up linting, autogenerating, and how to keep cactbot data in sync with whatever the log guide was using for autogenerating (possibly via npm package). This seems very do-able, and maybe it's worth doing that work to have log resource in a different place.

Having the log definitions as data and not in manual text makes it easier to write tools that use that. The log guide is one of them, and you could use that same information to make some new hypothetical web log viewer with mouseover on fields, etc etc. My opinion is that the log guide should come from netlog_defs data and be autogenerated as much as possible.

If you're proposing "it'd be nice to have a log guide somewhere else where you don't have to do autogeneration or linting", then I think that's the wrong approach.

1. Since portions are auto-genned from Cactbot code, it means contributors may need to set up an entire dev environment depending on what kind of change they're making.

As I mentioned above, I think autogenerating as much as possible and linting the log guide are both good things to do, and so a dev environment will be an issue no matter where it lives. I think it'd be nice to autogenerate your table proposal, for instance.

2. There was also a problem in the past of not being able to document new fields added to existing lines, because it breaks line parsing for CN/KR until they get the patch.

This has been fixed by valarnin and is no longer an issue.

3. If a field, especially one that doesn't currently have a descriptive name, is renamed, this may break stuff in Cactbot, so it would need the ability to have "alternate" field names or some other backwards compatibility mechanism.

valarnin and I have both discussed approaches to make this possible with you in discord. I think you could add a "descriptive name" field to netlog_defs and use that during log generation as an override to the cactbot field name. I don't think this is a big deal to implement at all, and would also solve the "flag/value" and "trio" issue below.

Alternatively, if netlog_defs moves elsewhere, this renaming could be added to cactbot. Or, if netlog_defs is imported from cactbot into a log guide that's moved elsewhere, then the renaming could be added there.

4. The fields currently lack type and representation info (e.g. whether a field is base-10 or base-16).

This sounds like a nice thing to add to the netlog_defs structure.

5. Since they also lack descriptions, adding a new field or clarifying an existing field typically requires manual changes to the log guide anyway.

This can be added to netlog_defs too, and then tables can be auto-generated as you describe below. The tables you propose are a good idea. If you had descriptions and types for fields, I think they would be a good thing to auto-generate.

So, in terms of what to do, there's two options that come to mind:

1. Significantly improve the field defs in Cactbot, including type info (as well as how to convert it, e.g. b10 vs b16, or 1.23 vs 0x3F9D70A4), array types, alt names, and whatever else might be needed. Then, improve the auto-gen with all of that. The advantage is that you might be able to pull more type safety into the actual line parsing as well as reduce boilerplate.

This is what I would prefer and is what I have said to you previously. I think type information and better names would make for a better guide overall.

2. Split out the Cactbot-specific guidance from the more general log information. Don't auto-gen the generalized guide. From the standpoint of the change I've made, the auto-gen doesn't cover the parts that require the most effort anyway (like the mess that is ability effects), so it really wouldn't be a problem to just have me or someone else write those portions manually. In addition, it

There's not really "cactbot-specific guidance" in the guide. I don't understand this proposal.

It's not important to me that the field names in the guide match what cactbot is using. (There could be a cactbot-specific field list where they differ.) My opinion is largely that the guide is auto-generated as much as possible to reduce manual work and reduce mistakes especially as more and more logs get added over time.