Skip to content

Conversation

@StevenLangbroek
Copy link
Collaborator

@StevenLangbroek StevenLangbroek commented Nov 24, 2025

Hey! Opening Draft PR early to have a place to discuss direction early & continuously. Feel free to offer feedback at any point in time, I'll actively sollicit at key moments myself.

To do

  • Sketch public API (npx + docker based)
  • Implement basic test harness & cli
  • Implement hooks for test lifecycle

@changeset-bot
Copy link

changeset-bot bot commented Nov 24, 2025

⚠️ No Changeset found

Latest commit: 3eb9472

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@StevenLangbroek
Copy link
Collaborator Author

@ThisIsMissEm hey! I gathered my thoughts after our meeting, and wrote what I think would be an appropriate README for the conformance tests. Let me know if you have any feedback!

A note: I thought vitest supported tags, but they don't (jest does, but hard pass 😄 ). I think arg wrangling to abstract away a directory-based implementation makes sense, but am not super attached to it.

Copy link
Contributor

@ThisIsMissEm ThisIsMissEm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A fair few comments

@ThisIsMissEm
Copy link
Contributor

@StevenLangbroek I think for now we should probably just go with a docker release strategy, instead of npx. I'm not too keen on how you need to setup trusted publishing, so not publishing to npm seems safest.

@StevenLangbroek
Copy link
Collaborator Author

Still fighting @optique a bit, it's not so easy to have a disjoint union like this, where 3 of the reporters require an output-file arg, but the console reporter does not. Actively working on it, tbd. For now, you can run the tests with the (lovely) command FIRES_SERVER_URL=http://localhost:4444 pnpm --filter @fedimod/fires-conformance exec vitest

StevenLangbroek and others added 9 commits December 1, 2025 15:10
- Update license to AGPL-3.0, per reference server
Co-authored-by: Emelia Smith <ThisIsMissEm@users.noreply.github.com>
Signed-off-by: Steven Langbroek <steven.android@gmail.com>
- Remove npm mention until publishing has been figured out by maintainers
- Rename category to suite
- Remove mention of unnecessary test suites
- Add basic nodeinfo test
"@optique/run": "^0.7.0",
"bcp-47": "^2.1.0",
"jsonld": "^9.0.0",
"ts-node-maintained": "^10.9.6",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be a dev dep

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it? It's a dependency for running the tool, no, through cli.js?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope, as the distribution would be compiled.

expect(dataset).toHaveProperty('id');
expect(typeof dataset.id).toBe('string');

// name is required
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// name is required
// name is required
// TODO: support i18n

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ThisIsMissEm which properties support this behavior (i18n, field -> fieldMap) again?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • name -> nameMap
  • content -> contentMap
  • summary -> summaryMap

Copy link
Contributor

@ThisIsMissEm ThisIsMissEm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, I left a few comments

},
});

const data: any = await response.json();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can dry the above up a bit later

@dahlia
Copy link

dahlia commented Dec 1, 2025

Still fighting @optique a bit, it's not so easy to have a disjoint union like this, where 3 of the reporters require an output-file arg, but the console reporter does not. Actively working on it, tbd. For now, you can run the tests with the (lovely) command FIRES_SERVER_URL=http://localhost:4444 pnpm --filter @fedimod/fires-conformance exec vitest

@StevenLangbroek Hey, I'm the author of Optique. Thanks for trying it out!

You're right that this pattern is tricky to express with the current API. The or() combinator works, but it splits the choice() across separate parsers, which makes help text fragmented and error messages confusing when the user forgets a conditionally required option.

I've opened an issue to track a potential solution: dahlia/optique#49

The idea is a conditional() combinator that would let you write something like:

const parser = conditional(
  option("--reporter", choice(["console", "junit", "html", "json"])),
  {
    console: object({}),
    junit: object({ outputFile: option("--output-file", string()) }),
    html: object({ outputFile: option("--output-file", string()) }),
    json: object({ outputFile: option("--output-file", string()) }),
  },
  object({}) // default when --reporter is omitted
);

This would keep all reporter values in a single choice(), give better error messages like “--output-file is required when --reporter is junit”, and produce a proper discriminated union type.

This is high on my priority list, so I'm hoping to get it implemented soon. Feel free to chime in on the issue if you have thoughts on the API design!

@dahlia
Copy link

dahlia commented Dec 9, 2025

@StevenLangbroek The conditional() combinator is shipped with Optique 0.8.0!

@StevenLangbroek
Copy link
Collaborator Author

@dahlia hey! awesome, thanks so much! i'll get to work and let you know how it goes.

@StevenLangbroek
Copy link
Collaborator Author

@dahlia works a treat, thanks! small note about the API, the syntax for default behavior doesn't intuitively make sense for my use-case. I'd have expected the default to be passed as a string / "choice", e.g.:

const parser = conditional(
  option("--reporter", choice(["console", "junit", "html", "json"])),
  {
    console: object({}),
    junit: object({ outputFile: option("--output-file", string()) }),
    html: object({ outputFile: option("--output-file", string()) }),
    json: object({ outputFile: option("--output-file", string()) }),
  },
  'console' // this makes more sense to me than `object({})`
);

But I understand there might be reasons / constraints driving that. I also don't feel super strongly about it, just a thought.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants