[BUG] Arguments are not correctly passed from CLI to npm script (npm 7, Windows, Powershell)

[cyberbiont]

Environment:

• OS: Windows 10Pro 20H4
• Node: 16.0.0
• npm: 7.11.0

package.json:

...
"script": { 
    "git": "git",
    "help": "npm run git -- --help"
}

npm run help // I get help about Git (as intended)
npm run git -- --help from Powershell // I get help about npm run-script

As far as I can tell, the problem exists only in Powershell on Windows.
cmd and Git Bash work as intended.
Bash and pwsh on Linux also work as intended.

The problem is new to npm7 (I switched back to npm6 and it worked as intended).

[rikbrowning]

I have noticed the following:

running npm run test:unit -- TestName will pass the argument to the underlying command
running npm run test:unit -- -u TestName will not pass the -u but still passes TestName.

[holtalanm]

I am experiencing this same issue.

npm version: 7.19.0
node versiion: 16.4.0
powershell version: 5.1.19041.610
os: windows v10.0.19042.0

when running npm run test:unit -- -h, I get:

npm run-script

Run arbitrary package scripts

Usage:
npm run-script <command> [-- <args>]

Options:
[-w|--workspace <workspace-name> [-w|--workspace <workspace-name> ...]]
[-ws|--workspaces] [--if-present] [--ignore-scripts]
[--script-shell <script-shell>]

aliases: run, rum, urn

Run "npm help run-script" for more info

when running npm run test:unit -- -- -h, I get:

> vue-cli-service test:unit "-h"


  Usage: vue-cli-service test:unit [options] <regexForTestFiles>

  Options:

    --watch   run tests in watch mode

  All jest command line options are supported.
  See https://facebook.github.io/jest/docs/en/cli.html for more details.

It seems a workaround currently is to include an additional -- in the arguments to npm run when running on powershell.

can also confirm that this behavior is new to npm7, as I just upgraded from v6 recently.

[holtalanm]

oddly enough, but this behavior does not seem to occur when running within an Azure DevOps pipeline.

npm version: 7.18.1
os version: Microsoft Windows NT 10.0.19042.0
powershell version: 5.1.19041.610

I checked to see if the npm version was the difference, but after installing v7.18.1 of npm on my local machine, I still had the broken behavior. Looks like there may be something related to interactive vs non-interactive?

edit: doesn't look like -NonInteractive has any bearing on whether things work or not. I'll keep digging.

[holtalanm]

after completely uninstalling nodejs and all globally installed packages, then reinstalling the latest version of nodejs, this issue went away.

[AgainPsychoX]

It seems a workaround currently is to include an additional -- in the arguments to npm run when running on powershell.

Then it does not work in cmd or any other stuff, therefore its worse than it was (unless you are working alone on the local project and you want stick to powershell).

Also experiencing the issue, for now I will switch to using cmd.exe as shell, but it's painful sometimes.

[bottlezz]

It seems a workaround currently is to include an additional -- in the arguments to npm run when running on powershell.

Then it does not work in cmd or any other stuff, therefore its worse than it was (unless you are working alone on the local project and you want stick to powershell).

Also experiencing the issue, for now I will switch to using cmd.exe as shell, but it's painful sometimes.

triple dash works for me too "---" but still only powershell has this issue

[bruxisma]

Hello all, I've run into this issue again and I decided that I would get to the bottom of it and after a few hours of spelunking I've discovered the reason for why it fails in pwsh 7. I'll be putting an explicit detail for non-experts for the reason below and explain how I figured it out, and how it is failing, as it's a combination of several behaviors. Luckily, this can be fixed fairly easily and I should be able to open a PR to resolve this unless someone beats me to it. 🙂

So, strap in because this is going to be in depth. It's a small nightmare. Feel free to skip down to "the fix" if you don't care about the reasons. 🙂

The Explanation

To start with, pwsh itself treats -- special for its own syntax. This special character is treated like programs in other shells treat it: an "end of parameters" parameter. I first noticed something was up when I saw there is a slight difference in syntax highlighting in the terminal for this parameter. However this is barely noticeable if you have syntax highlighting enabled in your shell as the colors are simply the same as any other parameter but bold. I've attached a screenshot for sighted folks.

Notice how the -- is slightly brighter in the attached screenshot (I'm using Windows Terminal with a gruvbox theme). This same highlighting does not appear for other "special" parameters including the "stop parsing" --% token, which can't be used because when you run node in pwsh, it's executing a .ps1 file, and passing --% $args to node will of course error because it has no earthly idea what $args means.

This is further compounded by pwsh treating the execution of node as a native expression because of the & call operator, instead of something like Start-Process. Additionally because of rules regarding pwsh parsing command line arguments in a specific way is due to a result of how ProcessStart parses \, which in turn is a result of how the Win32 C function CommandLineArgvW parses command line arguments. If you're not aware, Win32 treats all arguments as a single string, and it is runtimes further up the chain that break them up into an array/list/sequence/etc.

So, when pwsh encounters --, it parses it out, like any other syntax. This brings us to the next issue: nopt, the node option parsing library. It only stops treating parameters as "done" once it sees a --, otherwise it parses all "valid" flags, shoves them into an object, and returns that. There is, at this point, no hope of retrieving or undoing the parse efforts to find these additional flags and they are removed from npm's CLI as a result and cannot be retrieved.

The Posit

So how do we get pwsh to ignore --? The answer is, in fact, terrible. If you simply use single quotes, you can "escape" the special parameter and it will be treated as normal: '--'. Yes, single quotes work just fine.Except, this only works on the command line. Attempting to use it inside of package.json will drop the --fix and then put quotes around -- directly. Instead, you must write \"--\" to properly escape the -- inside of package.json.

So how do we fix this? We can't let pwsh see the --. This means that if I have a variable $special = "--" and pass that in as a parameter, it will be passed through to npm correctly.

The Fix

My current plan for the fix is a hack but should be simple:

Check if -- was passed to the $MyInvocation.Line, find the parameter that came after it, and then insert a "--" into the $args array. We sadly can't detect the existence of a -- otherwise, but I'm confident we can at least work around this. 🙂

I'll be submitting a PR once I have a solution (and some tests!), so it might be a day or two before it's submitted. 😅

[bruxisma]

Brief update: While I have a working fix for just npm/npx, the issue is more to do with how cmd-shim currently generates the .ps1 file for npm. This is used to generate every .ps1 script, not just the NPM ones installed with node, but every "binary" installed by npm.

There may not be a fix for this without breaking the entire ecosystem beyond using --- for npm run (nopt parses any - of 2 or greater to be treated "as if" it were just -- on every platform) or waiting for Powershell 7.2 which has a new (breaking change) way of handling -- passed to native commands where they will pass through.

So much for my optimism regarding this being "easily" fixable 😔