Japanese Text

[ryan-jeffrey]

When using the adapter I get a lockup then pushing japanese text. The text is "ｽﾄﾛｰｸｴﾝﾄﾞ軸あり", the whole line I'm sending is "2026-01-07T22:32:50.8567651Z|systemCondition|FAULT|1111|||ｽﾄﾛｰｸｴﾝﾄﾞ軸あり".

While experimenting with the adapter I changed all the ASCIIEncoding instances to UTF8Encoding instances. I included the relevant agent output files:

ascii-adapter-output.txt
utf8-adapter-output.txt
mtconnectoutput.txt

The difference is with the following lines:

For the UTF8Encoding version I get this returned:

<Fault dataItemId="systemCondition" timestamp="2026-01-07T23:16:17.1481955Z" sequence="80" nativeCode="1111" type="SYSTEM">&#xFF7D;&#xFF84;&#xFF9B;&#xFF70;&#xFF78;&#xFF74;&#xFF9D;&#xFF84;&#xFF9E;&#x8EF8;&#x3042;&#x308A;</Fault> , those escaped unicode characters are the correct characters for the text I passed in. The issue is that the adapter seems to lock up after sending this, but maybe that's an issue with the adapter?

For the unaltered agent I get this:

<Fault dataItemId="systemCondition" timestamp="2026-01-07T23:26:19.8203954Z" sequence="80" nativeCode="1111" type="SYSTEM">????????????</Fault>
But the adapter does not lock up.

I also included the telnet output for the adapter when run with the UTF8 encodings just in case that's useful.

[wsobel]

Thanks a lot for the files. I'll take a look at this after I finish the WebSocket testing.

[wsobel]

Started working on this in a branch. Will push once I have a repeatable test.

[wsobel]

I have been trying to decode the characters in the file: "ｽﾄﾛｰｸｴﾝﾄﾞ軸あり" and I have been unable to determine what coding system they are in. I looked online and the only explanation I could find is it could be something called Mojubake

(Japanese: 文字化け; IPA: [mod͡ʑibake], 'character transformation') is the garbled or gibberish text that is the result of text being decoded using an unintended character encoding.[1] The result is a systematic replacement of symbols with completely unrelated ones, often from a different writing system.

I will test with the text "2026-01-07T22:32:50.8567651Z|systemCondition|FAULT|1111|||ｽﾄﾛｰｸｴﾝﾄﾞ軸あり" and see what happens when the adapter receives the text and it flows through the Agent. I will try to fix any issue that may be introduced on our end. Please check your encoding and streaming on your side as well to make sure your adapter is doing Mokibake.

I'm going to write some tests and get back soon.

[wsobel]

When I pass the data through the agent, I get the following:

      <ComponentStream component="Controller" name="Controller" componentId="cont">
        <Condition>
          <Fault dataItemId="lp" nativeCode="1111" sequence="35" timestamp="2026-01-07T22:32:50.856765Z" type="LOGIC_PROGRAM">ｽﾄﾛｰｸｴﾝﾄﾞ軸あり</Fault>
        </Condition>
      </ComponentStream>

[wsobel]

I just tested with the agent hooked up to a simulator with the following input to the adapter:

2026-01-07T22:32:50.8567651Z|systemCondition|FAULT|1111|||ｽﾄﾛｰｸｴﾝﾄﾞ軸あり

The result I received was:

<?xml version="1.0" encoding="UTF-8"?>
<MTConnectStreams xmlns:m="urn:mtconnect.org:MTConnectStreams:2.4" xmlns="urn:mtconnect.org:MTConnectStreams:2.4" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mtconnect.org:MTConnectStreams:2.4 http://schemas.mtconnect.org/schemas/MTConnectStreams_2.4.xsd">
    <Header creationTime="2026-01-10T17:15:31Z" sender="myshkin.local" instanceId="1768065180" version="2.7.0.1" deviceModelChangeTime="2026-01-10T17:13:01.050394Z" bufferSize="131072" nextSequence="30" firstSequence="1" lastSequence="29"/>
    <Streams>
        <DeviceStream name="Agent" uuid="1986788b-d764-548f-a848-2319742705e0">
            <ComponentStream component="Adapter" name="shdr://127.0.0.1:7878" componentId="_127.0.0.1_7878">
                <Samples>
                    <AssetUpdateRate dataItemId="_127.0.0.1_7878_asset_update_rate" sequence="11" statistic="AVERAGE" timestamp="2026-01-10T17:13:01.054311Z">UNAVAILABLE</AssetUpdateRate>
                    <ObservationUpdateRate dataItemId="_127.0.0.1_7878_observation_update_rate" duration="10" sequence="29" statistic="AVERAGE" timestamp="2026-01-10T17:15:31.078379Z">0</ObservationUpdateRate>
                </Samples>
                <Events>
                    <AdapterSoftwareVersion dataItemId="_127.0.0.1_7878_adapter_software_version" sequence="12" timestamp="2026-01-10T17:13:01.054343Z">UNAVAILABLE</AdapterSoftwareVersion>
                    <AdapterURI dataItemId="_127.0.0.1_7878_adapter_uri" sequence="9" timestamp="2026-01-10T17:13:01.054244Z">SHDR://127.0.0.1:7878</AdapterURI>
                    <ConnectionStatus dataItemId="_127.0.0.1_7878_connection_status" sequence="25" timestamp="2026-01-10T17:15:01.083769Z">ESTABLISHED</ConnectionStatus>
                    <MTConnectVersion dataItemId="_127.0.0.1_7878_mtconnect_version" sequence="13" timestamp="2026-01-10T17:13:01.054375Z">UNAVAILABLE</MTConnectVersion>
                </Events>
            </ComponentStream>
            <ComponentStream component="Agent" name="Agent" componentId="agent_1986788b">
                <Events>
                    <AssetChanged assetType="UNAVAILABLE" dataItemId="agent_1986788b_asset_chg" sequence="6" timestamp="2026-01-10T17:13:01.054151Z">UNAVAILABLE</AssetChanged>
                    <AssetCountDataSet count="0" dataItemId="agent_1986788b_asset_count" sequence="7" timestamp="2026-01-10T17:13:01.054182Z">UNAVAILABLE</AssetCountDataSet>
                    <AssetRemoved assetType="UNAVAILABLE" dataItemId="agent_1986788b_asset_rem" sequence="3" timestamp="2026-01-10T17:13:01.054064Z">UNAVAILABLE</AssetRemoved>
                    <Availability dataItemId="agent_avail" sequence="23" timestamp="2026-01-10T17:13:01.05469Z">AVAILABLE</Availability>
                    <DeviceAdded dataItemId="device_added" hash="mziVcdPxc3FK1ultMgsyQH+udMw=" sequence="22" timestamp="2026-01-10T17:13:01.054659Z">000</DeviceAdded>
                    <DeviceChanged dataItemId="device_changed" sequence="5" timestamp="2026-01-10T17:13:01.054125Z">UNAVAILABLE</DeviceChanged>
                    <DeviceRemoved dataItemId="device_removed" sequence="4" timestamp="2026-01-10T17:13:01.054099Z">UNAVAILABLE</DeviceRemoved>
                </Events>
            </ComponentStream>
        </DeviceStream>
        <DeviceStream name="LinuxCNC" uuid="000">
            <ComponentStream component="Controller" componentId="cont">
                <Events>
                    <Execution dataItemId="exec" sequence="20" timestamp="2026-01-10T17:13:01.054592Z">UNAVAILABLE</Execution>
                    <ControllerMode dataItemId="mode" name="mode" sequence="19" timestamp="2026-01-10T17:13:01.054559Z">UNAVAILABLE</ControllerMode>
                </Events>
            </ComponentStream>
            <ComponentStream component="Device" name="LinuxCNC" componentId="d">
                <Events>
                    <Availability dataItemId="a" name="avail" sequence="14" timestamp="2026-01-10T17:13:01.054406Z">UNAVAILABLE</Availability>
                    <AssetChanged assetType="UNAVAILABLE" dataItemId="d_asset_chg" sequence="15" timestamp="2026-01-10T17:13:01.05444Z">UNAVAILABLE</AssetChanged>
                    <AssetCountDataSet count="0" dataItemId="d_asset_count" sequence="17" timestamp="2026-01-10T17:13:01.054497Z">UNAVAILABLE</AssetCountDataSet>
                    <AssetRemoved assetType="UNAVAILABLE" dataItemId="d_asset_rem" sequence="16" timestamp="2026-01-10T17:13:01.054469Z">UNAVAILABLE</AssetRemoved>
                </Events>
                <Condition>
                    <Fault conditionId="1111" dataItemId="syscond" nativeCode="1111" sequence="26" timestamp="2026-01-10T17:15:02.658177Z" type="SYSTEM">ｽﾄﾛｰｸｴﾝﾄﾞ軸あり</Fault>
                </Condition>
            </ComponentStream>
        </DeviceStream>
    </Streams>
</MTConnectStreams>

I think the problem is most likely in the adapter and how it is writing to the stream. I have created a test just to make sure this does not break in the future. If I can help more, please tell me how.

[wsobel]

This will also regress on the different platforms to make sure the behavior is the same.