Add utilities to convert KQL timespans to pandas timedeltas

[Copilot]

KQL returns timespans as strings: "hh:mm:ss.fffffff" for < 1 day, "d.hh:mm:ss.fffffff" for >= 1 day. Pandas to_timedelta() fails on the latter format with ValueError: expecting hh:mm:ss format, received: 1.00:00:00.

Changes

Added to msticpy.common.data_utils:

• parse_timespan(timespan) - Parses KQL timespan strings to pd.Timedelta

  • Handles both formats via regex pattern
  • String-based fractional second parsing to avoid floating-point precision loss
  • Supports negative timespans, None passthrough

• ensure_df_timedeltas(data, columns=None) - Converts DataFrame columns to timedelta64[ns]

  • Auto-detects timespan columns if not specified
  • Tries vectorized pd.to_timedelta() first, falls back to element-wise for large timespans
  • Mirrors ensure_df_datetimes() API

Tests:

• 29 tests covering edge cases (negative, zero, fractional precision, None/NaN, auto-detection)

Usage

from msticpy.common.data_utils import ensure_df_timedeltas

# After KQL query execution
df_result = qp_sentinel.exec_query(query=my_query)

# Convert timespan columns (auto-detects or specify columns)
df = ensure_df_timedeltas(df_result)

# Now timedelta operations work
df['duration'].mean()
df[df['duration'] > pd.Timedelta(hours=1)]

Design Notes

• Optional utility (not automatic conversion in drivers) to avoid per-row overhead on all queries
• Fractional seconds parsed as strings then converted to nanoseconds to preserve precision
• Regex pattern extracted as module constant for consistency between parsing and detection

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[Bug]: msticpy KQL timespans are not implicity converted to timedeltas</issue_title>
<issue_description>Describe the bug

KQL datetime types are implicity converted to a pandas/numpy datetime type but timespans are not converted and returned as strings.

To Reproduce

Setup:

import msticpy as mp


mp.init_notebook()
qp_sentinel = mp.QueryProvider('MSSentinel')
qp_sentinel.connect()

Issue:

q = '''
let TimeSpanSmall = 1ms;
print(TimeSpanSmall)
'''
df_timespan_small = qp_sentinel.exec_query(query=q)
print(df_timespan_small['print_0'].dtype)
print(type(df_timespan_small['print_0'].values[0]))

Output:

object
<class 'str'>

Expected behavior

timedelta64[ns]
<class 'numpy.timedelta64'>

Screenshots and/or Traceback

From a test notebook: msticpy_large_timespan_type_conversion_fails.zip

Environment (please complete the following information):

• Python Version: 3.11
• OS: Ubuntu
• Python environment: venv
• MSTICPy Version: 2.15

Additional context

It's arguable if this should be classified as a feature request vs bug. However, given the behavior for datetime64, I argue that the lack of consistency and convenience makes it a bug.

Another problem is that for larger timespan values, pandas does support handling the string format for type conversion to timedeltas.

• Smaller timespans (< 1 day) can be converted to pandas timedelata types.
• Larger timestamps (>= 1 day) cannot be readily converted to pandas timedelta types without a helper function.

E.g. an "ValueError: expecting hh:mm:ss format, received: 1.00:00:00" exception occurs for a 1 day timespan.

Demo in notebook: msticpy_large_timespan_type_conversion_fails.zip

Client wrapper function to help compensate.

from typing import Optional, Union
import re
import pandas as pd
import pprint


def parse_timespan(timespan: Optional[Union[str, pd.Timedelta]]) -> pd.Timedelta:
    """
    Parse a Kusto timespan-like formatted string into a pandas Timedelta object.

    - Timespans a day or larger are returned by Kusto as strings, e.g. "1.19:37:05.1697513", are not converted correctly into timedeltas which expects only hh:mm:ss formats of less than one day
    - Require at least "hh:mm:ss" to be present.
    - Return a ns resolution Timedelta object (match pandas default).
    """

    if isinstance(timespan, pd.Timedelta):
        return timespan

    # require at minium an "hh:mm:ss" format
    pattern = re.compile(r'(?P<sign>-?)(?:(?P<days>\d+)\.)?(?P<hours>\d+):(?P<minutes>\d+):(?P<seconds>\d+)(?P<frac_seconds>\.\d+)?')
    match = pattern.match(timespan)
    if match:
        match_groups = match.groupdict()
        sign = -1 if match_groups['sign'] == '-' else 1
        days = int(match_groups['days']) if match_groups['days'] else 0
        hours = int(match_groups['hours'])
        minutes = int(match_groups['minutes'])
        seconds = int(match_groups['seconds'])
        frac_seconds = float(match_groups['frac_seconds']) if match_groups['frac_seconds'] else 0
        nano_seconds = int(round(frac_seconds, ndigits=9) * 1e9)
        return sign * pd.Timedelta(days=days, hours=hours, minutes=minutes, seconds=seconds, nanoseconds=nano_seconds)
    else:
        raise ValueError(f"Invalid timespan format: {timespan}")


# Tests
parse_timespan_tests = dict(
    pos_small_frac = '00:00:00.001',
    neg_small_frac = '-00:00:00.001',
    zero_frac = '00:00:00.000',
    pos_large = '1.00:00:00',
    neg_large = '-1.00:00:00',
    pos_large_frac_round = '3.23:59:59.99999999999',
    neg_large_frac_round = '-3.23:59:59.99999999999'
)
parse_timespan_tests = {k: (v, parse_timespan(v)) for k, v in parse_timespan_tests.items()}
pprint.pp(parse_timespan_tests)
```</issue_description>

<agent_instructions>Can you investigate if there is an efficient fix for this? </agent_instructions>

## Comments on the Issue (you are @copilot in this section)

<comments>
<comment_new><author>@ianhelle</author><body>
I've looked at this a couple of times. 
We don't do any specific conversions for the LogtableRow content that we get from the AzureMonitor driver - which is just pandas conversion from JSON.

Not sure that I want to embark on this, since it's going add a significant per-row conversion penalty to every query, whether or not it has timedeltas in it or not.
I think if they are important for your case, it's probably better to add the wrapper around the returned results.

Do you know if azure-kusto-data handles timed...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes microsoft/msticpy#829

<!-- START COPILOT CODING AGENT TIPS -->
---

✨ Let Copilot coding agent [set things up for you](https://github.com/microsoft/msticpy/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot) — coding agent works faster and does higher quality work when set up for your repo.

[FlorianBracq]

That looks OK for me now!