Bug: Can we access the Raw data of Tap 1 & 2 on this API (Tap Freeze)?

[EloiStree]

Hello guys,

In my Unity 2019 application. when I call the method
void StartRawSensorMode(string tapIdentifier, int deviceAccelerometerSensitivity, int imuGyroSensitivity, int imuAccelerometerSensitivity);

The Tap 2 start to freeze and then disconnect.

In the doc I read that I could be able to use Sensors (for finger with Tap 1 and all for Tap 2)
https://tapwithus.atlassian.net/wiki/spaces/TD/pages/792002574/Tap+Strap+Raw+Sensors+Mode
But that not the case from what I observe.

Does it works on your side ?
Did you test this functionality in this API ?

Just to check if is me the problem.

I keep you tuned if I find the why of this bug.

(Firmware is 20400 maybe but I am on the last update.)

[EloiStree]

Update:
Apparently when I add a listener to the message received, I can see that I received message from you Java app for around 22s seconds.

public delegate void JavaMessageReceived(string msg);
	public static JavaMessageReceived m_rawReceived;


  private void onRawSensorDataReceived(String rsArg)
    {
		m_rawReceived(rsArg);
		if (OnRawSensorDataReceived != null)
        {
			string[] argParts = rsArg.Split(ARGS_SEPERATOR);
            if (argParts.Length == 2)
            {
				RawSensorData data = RawSensorData.makeFromString(argParts[1], "^");
                if (data != null)
                {
                    OnRawSensorDataReceived(argParts[0], data);
                }
            }
		}
    }

[EloiStree]

The last 2000 char:

TAP RawSensor Data receivedtimestamp: 350865, type: IMU, points: (-7507,5, -6265, -18865), (730,17, -333,304, 535,336)|350865^IMU^-7507.5^-6265.0^-18865.0^730.17^-333.304^535.336
+TAP RawSensor Data receivedtimestamp: 350863, type: DEVICE, points: (812,5, -375, 562,5), (62,5, 968,75, 343,75), (-125, 937,5, 250), (0, 906,25, 312,5), (-125, 843,75, 562,5)|350863^DEVICE^812.5^-375.0^562.5^62.5^968.75^343.75^-125.0^937.5^250.0^0.0^906.25^312.5^-125.0^843.75^562.5
+TAP RawSensor Data receivedtimestamp: 350863, type: DEVICE, points: (812,5, -375, 562,5), (125, 937,5, 375), (-93,75, 937,5, 250), (0, 906,25, 343,75), (-125, 843,75, 562,5)|350863^DEVICE^812.5^-375.0^562.5^125.0^937.5^375.0^-93.75^937.5^250.0^0.0^906.25^343.75^-125.0^843.75^562.5
+TAP RawSensor Data receivedtimestamp: 350860, type: IMU, points: (-8680, -6737,5, -19810), (735,05, -351,36, 543,632)|350860^IMU^-8680.0^-6737.5^-19810.0^735.05^-351.36^543.632
+TAP RawSensor Data receivedtimestamp: 350856, type: IMU, points: (-9940, -7367,5, -20597,5), (736,636, -360,022, 546,072)|350856^IMU^-9940.0^-7367.5^-20597.5^736.636^-360.022^546.072
+TAP RawSensor Data receivedtimestamp: 350856, type: DEVICE, points: (812,5, -375, 562,5), (125, 937,5, 375), (-62,5, 937,5, 250), (62,5, 906,25, 343,75), (-62,5, 875, 562,5)|350856^DEVICE^812.5^-375.0^562.5^125.0^937.5^375.0^-62.5^937.5^250.0^62.5^906.25^343.75^-62.5^875.0^562.5
+TAP RawSensor Data receivedtimestamp: 350839, type: DEVICE, points: (781,25, -406,25, 593,75), (62,5, 937,5, 406,25), (-93,75, 1000, 281,25), (31,25, 937,5, 343,75), (-125, 843,75, 625)|350839^DEVICE^781.25^-406.25^593.75^62.5^937.5^406.25^-93.75^1000.0^281.25^31.25^937.5^343.75^-125.0^843.75^625.0
+TAP RawSensor Data receivedtimestamp: 350835, type: IMU, points: (-10832,5, -11112,5, -23502,5), (720,898, -380,762, 557,052)|350835^IMU^-10832.5^-11112.5^-23502.5^720.898^-380.762^557.052
+TAP RawSensor Data receivedtimestamp: 350831, type: IMU, points: (-11567,5, -11690, -23205), (722,728, -390,766, 550,0

[EloiStree]

Ok, I tried everything I can on my side (Unity).
I will wait to have feedback from the team before continuing on the RawSensor.
Ping me when you are available.

You can find my current code state here if you need.
(https://gitlab.com/eloistree/2020_04_20_TapUtilityOnAndroid)
(Commit of when is message is wrote)

[shahartap]

@EloiStree Hello,
Can you please try opening our TapManager app (Download it from PlayStore or AppStore if you don't have it installed yet). Pair your Tap to your phone and on the "Settings" screen in TapManager, enable "Developer Mode". This should allow you to use the RawSensorData.
Please let me know that helps. Sorry it took us time to get back to you. Will do anything I can to help you.

Regards,
Shahar Biran
TAP WITH US.

[EloiStree]

@shahartap I make some try for hours.
What you ask me is one of them.
(The maximum time I succeed to received the data is 22 secondes. In any of this test)
(I can try on an other phone but I don't think that will change something.)

[EloiStree]

@shahartap Stupid question but... Does it works on your side ?
If you create a Unity project on Android with your plug-in and set the code with "Start Raw Data"
with the Tap 2 updated to 2.4 on developer mode you succeed to read the date for more that 20+ seconds ?

[shahartap]

@EloiStree hello,
Yes it works. It’s been tested and been used on Hackathon event.
I would like to know more about the freeze. Does the app freeze and not responding ? Or is it just the Tap? Can you subscribed to all Tap events (tapped, Moused, airgestured) and see if you receive them after the freeze ? Are there any Tap apps running in the background ? After the freeze can you still use the Tap (without disconnecting it) ?

Also - I intend to look at your code , is the previous link you provided is updated ?

Thanks,
Shahar Biran

[EloiStree]

To be clear 95% of the API is working well: Tap, gesture, mouse...
And that amazing :)

The only part that bug is the access the device info Acc, Gyro, Finger Accelerometer.
As soon as I call "RawSensor" I can received the java info of the device in Unity.

void StartRawSensorMode(string tapIdentifier, int deviceAccelerometerSensitivity, int imuGyroSensitivity, int imuAccelerometerSensitivity);

Then after a little time the device seem to go in a sleep mode ( or bug/freeze I don't really know).
And Java stop sending information to Unity API. (The application continue to run almost normally)

( I don't think it is my project because it did try with only your code to be sure before writing this issue and the bug is there too. I quickly try on my seconds phone before going to sleep. )

Sorry to bother you with that.
But I really want to see what we can do in VR with the access of those data in Unity.

Kind regards,
Eloi

(PS: You can look at my code but it is going to be messy. So I will write as small script from your script to demonstrate)

[EloiStree]

Hope it help

A step by step video of how I produce the bug from this repository.

https://youtu.be/dx2uvCSUZHE

• 0-20 minutes, It is just me starting for zero to the bug.
• On Samsung S7: https://youtu.be/dx2uvCSUZHE?t=1412
• On Galaxy Note 4: https://youtu.be/dx2uvCSUZHE?t=1651
  New: It bug on the Galaxy Note 4 and two other old phone.
  But bug less on the Galaxy S7 as you can see in the video.

You can find here the code of your script a bit modify to allow triggering the Start Raw Data from a button and display the info on the screen.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;

public class TapInputDemoRawSensor : MonoBehaviour
{

    public Text LogText;
    public Text m_justRawdata;

    private ITapInput tapInputManager;

    private bool mouseHIDEnabled;

    private string connectedTapIdentifier = "";

    public int m_accelerometerSensibility=0;
    public int m_gyroSensibility=0;
    public int m_fingerSensibility=0;

    public void StartListeningToRawData() {
        tapInputManager = TapInputManager.Instance;
        if(connectedTapIdentifier.Length>0)
        tapInputManager.StartRawSensorMode(connectedTapIdentifier, m_accelerometerSensibility, m_gyroSensibility, m_fingerSensibility);
    }


    void Start()
    {

        tapInputManager = TapInputManager.Instance;

        tapInputManager.OnTapInputReceived += onTapped;
        tapInputManager.OnTapConnected += onTapConnected;
        tapInputManager.OnTapDisconnected += onTapDisconnected;
        tapInputManager.OnMouseInputReceived += onMoused;
        tapInputManager.OnAirGestureInputReceived += onAirGestureInputReceived;
        tapInputManager.OnTapChangedAirGestureState += onTapChangedState;
        tapInputManager.OnRawSensorDataReceived += onRawSensorDataReceived;
        tapInputManager.EnableDebug();
        tapInputManager.SetDefaultControllerWithMouseHIDMode(true);
        mouseHIDEnabled = false;

    }

    private void Log(string text)
    {
        if (LogText != null)
        {
            LogText.text += string.Format("{0}\n", text);
        }
        Debug.Log(text);
    }

    void onMoused(string identifier, int vx, int vy, bool isMouse)
    {
        Log("onMoused" + identifier + ", velocity = (" + vx + "," + vy + "), isMouse " + isMouse);
    }

    void onTapped(string identifier, int combination)
    {
        bool[] arr = TapCombination.toFingers(combination);
        Log("onTapped : " + identifier + ", " + combination);
    }

    void onTapConnected(string identifier, string name, int fw)
    {
        Debug.Log("onTapConnected : " + identifier + ", " + name + ", FW: " + fw);
        Log("onTapConnected : " + identifier + ", " + name);
        this.connectedTapIdentifier = identifier;
    }

    void onTapDisconnected(string identifier)
    {
        Debug.Log("UNITY TAP CALLBACK --- onTapDisconnected : " + identifier);
        Log("UNITY TAP CALLBACK --- onTapDisconnected : " + identifier);
        if (identifier.Equals(this.connectedTapIdentifier))
        {
            this.connectedTapIdentifier = "";
        }
    }


    void onAirGestureInputReceived(string tapIdentifier, TapAirGesture gesture)
    {

        Log("OnAirGestureInputReceived: " + tapIdentifier + ", " + gesture.ToString());
    }

    void onTapChangedState(string tapIdentifier, bool isAirGesture)
    {
        Log("onTapChangedState: " + tapIdentifier + ", " + isAirGesture.ToString());

    }

    void onRawSensorDataReceived(string tapIdentifier, RawSensorData data)
    {
        string rawInfo = "-";
        //RawSensorData Object has a timestamp, type and an array points(x,y,z).
        if (data.type == RawSensorData.DataType.Device)
        {
            // Fingers accelerometer.
            // Each point in array represents the accelerometer value of a finger (thumb, index, middle, ring, pinky).
            Vector3 thumb = data.GetPoint(RawSensorData.iDEV_THUMB);
            rawInfo += string.Format(" Finger Thumb {0}:{1}:{2} ", thumb.x, thumb.y, thumb.z);

            if (thumb != null)
            {
                // Do something with thumb.x, thumb.y, thumb.z
            }
            // Etc... use indexes: RawSensorData.iDEV_THUMB, RawSensorData.iDEV_INDEX, RawSensorData.iDEV_MIDDLE, RawSensorData.iDEV_RING, RawSensorData.iDEV_PINKY
        }
        else if (data.type == RawSensorData.DataType.IMU)
        {
            // Refers to an additional accelerometer on the Thumb sensor and a Gyro (placed on the thumb unit as well).
            Vector3 gyro = data.GetPoint(RawSensorData.iIMU_GYRO);
            rawInfo += string.Format(" Gyro {0}:{1}:{2} ", gyro.x, gyro.y,gyro.z) ;
            if (gyro != null)
            {
                // Do something with gyro.x, gyro.y, gyro.z
            }
            // Etc... use indexes: RawSensorData.iIMU_GYRO, RawSensorData.iIMU_ACCELEROMETER
        }
        // -------------------------------------------------
        // -- Please refer readme.md for more information --
        // -------------------------------------------------

        m_justRawdata.text = rawInfo + "\n" + m_justRawdata.text;
    }
}

[shahartap]

@EloiStree Hello,
Thank you for the effort of making a video!

I still can't identify the cause of the problem, and since we don't have the model of your mobile device, I have few more things to try remotely, if you will please:

Before everything - please change:
tapInputManager.SetDefaultControllerWithMouseHIDMode(true);
to
tapInputManager.SetDefaultControllerWithMouseHIDMode(false);

This will reduce overloading the event system.

a) As I saw from the video- you're not receiving Tap Inputs as well - Judging from the textview on screen that stopped updating. Does debug prints stops also??

b) After you stop getting updates - I want you to try and tap SHIFT (That would be your thumb, index and middle finger together) and tell me if you sense vibration on the Tap. That's of course while the tap is still paired to your mobile device and the unity app is still running.

c) If you have a logcat of somekind (android studio for example) It'd be helpful to send me the log, from when the app starts running until few seconds after it stopped getting updates.

Please understand, This is the first time we've been informed of such bug. And as we're dealing with android which Is a system of a large diverse group of mobile devices, we can't locally access most of them. So we're trying the best we can to remotely help.

Thanks,
Shahar Biran

[EloiStree]

@shahartap Won't try that today because I am busy. But I will try as soon as I can.

Please understand, This is the first time we've been informed of such bug. And as we're dealing with android which Is a system of a large diverse group of mobile devices, we can't locally access most of them. So we're trying the best we can to remotely help.

I am a developer too, I understand 😊. I know that some time 1+1 = a fucking long list of problems.

(I don't know how to do the logcat. And it seem lot's of effort to study to do it.
Do you want me to prepare android studio, the project and stuffs for a teamviewer session? )