Skip to content

Conversation

@joelklabo
Copy link

Adding tools to run unit tests on iOS.

The error reporting could be improved but it successfully indicates which test fails when run.

image

@joelklabo
Copy link
Author

#9

@cameroncooke
Copy link
Collaborator

Thanks for this will look at this later!

@joelklabo
Copy link
Author

Thank you for the project!

@cameroncooke cameroncooke requested a review from Copilot April 30, 2025 11:24
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces a new iOS unit testing tool to run tests on iOS simulators and improve error reporting for failing tests.

  • Extends the build response by adding a new rawOutput field.
  • Adds type definitions for rawOutput.
  • Implements and registers an iOS simulator test tool.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
src/utils/build-utils.ts Added rawOutput property to the success response for build actions.
src/types/common.ts Updated ToolResponse interface to include rawOutput.
src/tools/test_ios_simulator.ts Implemented iOS simulator test tool with test failure parsing logic.
src/index.ts Registered the new iOS simulator test tools.

text: `✅ ${platformOptions.logPrefix} ${buildAction} succeeded for scheme ${params.scheme}.`,
},
],
rawOutput: result.output + (result.error ? '\n' + result.error : ''),
Copy link

Copilot AI Apr 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Consider trimming trailing newlines from result.output before concatenation to avoid unintended extra newline characters in rawOutput.

Suggested change
rawOutput: result.output + (result.error ? '\n' + result.error : ''),
rawOutput: result.output.trimEnd() + (result.error ? '\n' + result.error : ''),

Copilot uses AI. Check for mistakes.
.split('\n')
.filter(l => /Test Case .* failed/.test(l))
.map(l => {
const m = l.match(/Test Case '(.*)' failed \((.*)\)/)!;
Copy link

Copilot AI Apr 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a non-null assertion with the regex match may cause a runtime error if the log format ever changes; consider performing a safe check before destructuring the match result.

Copilot uses AI. Check for mistakes.
@cameroncooke cameroncooke self-requested a review April 30, 2025 11:26
@cameroncooke
Copy link
Collaborator

@joelklabo I've had a quick scan and it looks good, I need to take a deeper look later and will get back to you. I hope to have some time this evening.

registerTool(
server,
'ios_simulator_test_by_name_workspace',
'Run tests for an iOS app on a simulator specified by name using a workspace',
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cameroncooke I think I should improve these if you have any tips. I see now that your other tools have much more detail here.

@cameroncooke
Copy link
Collaborator

@joelklabo I've added some tests to the example projects, run the linter and formatted and removed some unused imported symbols.

I think the executeXcodeBuild function needs refactoring to improve how it parses output to make it more robust and identify build failure vs test failure, though I'm not clear currently on how to identify this. I was hoping we could use exit codes but it seems the test failure and build failure exit code is the same (65) 🤦‍♂️

Screenshot 2025-04-30 at 21 22 12

executeXcodeBuild is already responsible for parsing warnings and errors, so I think it should also parse test result bundles. We can then parse them using xcrun xcresulttool and then return the test failures in the ToolResponse.Content field instead of exposing the rawOutput to the calling function.

@cameroncooke
Copy link
Collaborator

@joelklabo I've added some tests to the example projects, run the linter and formatted and removed some unused imported symbols.

I think the executeXcodeBuild function needs refactoring to improve how it parses output to make it more robust and identify build failure vs test failure, though I'm not clear currently on how to identify this. I was hoping we could use exit codes but it seems the test failure and build failure exit code is the same (65) 🤦‍♂️

Screenshot 2025-04-30 at 21 22 12 `executeXcodeBuild` is already responsible for parsing warnings and errors, so I think it should also parse test result bundles. We can then parse them using `xcrun xcresulttool` and then return the test failures in the `ToolResponse.Content` field instead of exposing the `rawOutput` to the calling function.

I think we can just look for the presence of an xcresultbundle when the test command returns code 65, if it is a build failure, then the bundle won't exist.

I'm not sure when I'm going to have time to look into this, I'm afraid. You're welcome to take a sta,b otherwise I'll get on it as soon as I can so we can get this PR merged in.

@cameroncooke
Copy link
Collaborator

I've not forgotten about this been super busy recently and only have a few hours a day to focus on this. I will get this merged in.

@joelklabo
Copy link
Author

Yeah, same here, I agree with your comments. I'll take a stab at it if I can find time today

@reftonull
Copy link

Just wanted to +1 this! Would love to see this added soon. Thanks for the great tools already, they work really well. Would love to help in any way I can.

@mmbabaev
Copy link

mmbabaev commented Jun 4, 2025

also waiting for this feature

@cameroncooke
Copy link
Collaborator

Thanks for your contribution, this has now been implemented, so I am closing this PR. Thanks again!

@github-project-automation github-project-automation bot moved this from Backlog to Done in Xcode MCP Improvements Jun 7, 2025
@joelklabo
Copy link
Author

Thanks for implementing it @cameroncooke !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants