Release v1.0.0 stable version of common (or subset of packages)

[bwplotka]

Hi 👋🏽

It's time to take common to a higher stability level. We damage the ecosystem by making breaking changes (I authored some 🙃 ) e.g. thanos-io/thanos#8162 (pulling some dep suddenly break another dep etc.).

• Why not doing this?
• Should we have a separate common/exp module with 0.x like we do in client_golang now for experimental packages (that we can move to stable mod one day?)
• Should we be more strict on merging new code (in terms of API flexibility, naming) given it will be hard to change?
• Do we need to do any major changes before 1.0?

If we need more data on how much problematic 0.x common version is, to motivate this, I can try to research exact numbers, let me know!

cc @aknuds1 @SuperQ

[aknuds1]

Should we have a separate common/exp module with 0.x like we do in client_golang now for experimental packages (that we can move to stable mod one day?)

I lack experience personally with having an exp module, but it sounds like it could be a good idea, to allow for experimentation.

Do we need to do any major changes before 1.0?

I'd like for the otlptranslator package to be put to use in intended clients, e.g. Prometheus and OTel Collector, before stabilizing. I'm not sure about its current API.

[SuperQ]

I don't think an exp module would be useful, because as soon as we depend on it the stability requirements are the same. "There's no such thing as a temporary patch".

We already do have good practices around breaking changes.

For example with promslog, we actually created a whole new package in order to keep the existing promlog package in place. We created a release and converted the whole ecosystem over. Then we marked the promlog package deprecated and cut another release. It wasn't until another couple releases that we removed the old promlog package. This was ~6 month process.

There is no preventing change. We must move our codebases forward. We can do our best to review changes to make sure interfaces are stable and well designed. But mistakes happen, design patterns change, and we need to publish a breaking change.

I don't think we need or want a 1.0 of this library.

How would this actually work? What's the day-2 plan? Would we keep release-1.x branches? How long would we support them for?

IMO we don't have the people-time to maintain several release branches over any time in order to support older major versions. We barely do this (LTS) for prometheus itself.

I also think this will just churn the major version of the module for no utility. And that's going to be even more painful for us, as well as downstream users.

Also, how do you define breaking? Just function signatures? Structs? Just function removals? There are lots of small things that could be breaking or deprecated. Any behavior change comes with concequences.

I think we need to improve our design and review quality.

Easy to say, hard to do. But that's where I'm at with what are the realistic ways we can improve the quality of life for downstream users of our code. Similar to how Linux has a "don't break userspace" creed, I think we need to consider our downstream users more when making changes.

[aknuds1]

Nice writeup @SuperQ :)

[aknuds1]

I think we need to improve our design and review quality.

@SuperQ @bwplotka Is there anything concrete we can do in this regard? As a concrete example, I think actually the otlptranslator package shouldn't have been merged into main before making a PoC of its integration into relevant client projects (Prometheus, OTel Collector, Mimir, Thanos, others?). Hammering out the integration in retrospect has me feeling like I'm catching up.

[SuperQ]

As a concrete example, I think actually the otlptranslator package shouldn't have been merged into main before making a PoC

Perhaps we shouldn't have it in common. We could/should reduce the friction to creating new repos for Go libraries that don't need to be in common.

The definition in the README.md is this:

This repository contains Go libraries that are shared across Prometheus
components and libraries. They are considered internal to Prometheus, without
any stability guarantees for external usage.

If a library is meant for external use, perhaps it needs to be split out.

[aknuds1]

If a library is meant for external use, perhaps it needs to be split out.

That's a good point. @ArthurSens WDYT?

[ArthurSens]

I'm not sure if Bartek created this issue just because of otlptranslator, but I'm happy to move it elsewhere, no problems!

Regarding releasing Common 1.0, I generally agree with what Ben mentions above. The problem I'm seeing here in this repo is that it does way too many things, and we have way too few hands to maintain such a generic codebase. Some things are not that important and will need a 1.0 release with all the complicated stability guarantees, but other parts have so much adoption that changing any part of it will break dozens of downstream projects.

It would make sense to create a separate repository and assign maintainers who show expertise in the most adopted parts of the codebase. I can take ownership of otlptranslator and help with the expfmt package, but I'm oblivious to what the other packages do or should do.

[aknuds1]

@bwplotka Should we require CI checks and reviews before being able to merge a PR? I notice they are currently not required.

[bwplotka]

Perhaps we shouldn't have it in common. We could/should reduce the friction to creating new repos for Go libraries that don't need to be in common.

Agree. Maybe we should offer a template Go repo so it's easier to create those. Also automation to release it (go releases, our CI GH actions propagation etc).

The definition in the README.md is this:

This repository contains Go libraries that are shared across Prometheus
components and libraries. They are considered internal to Prometheus, without
any stability guarantees for external usage.

If a library is meant for external use, perhaps it needs to be split out.

Let's talk about this more. (:

One could argue that this is no longer true ("considered internal to Prometheus"). There are pieces that leaks to external use, e.g.:

• Our unlucky global variable that controls validation (https://github.com/prometheus/common/blob/main/model/metric.go#L49). Changing it's default value, changes validation behaviour which absolutely breaks external behaviour of stable projects that imports common for this e.g. Prometheus, client_golang, Kubernetes, Otel SDK etc. (See https://github.com/prometheus/client_golang/releases/tag/v1.21.0 as the consequence).
• expfmt pieces for auto-generating _total suffixes, which client_golang maintainers were not aware of. It's so hidden that if common maintainers will decide to change at some point, it's very easy to miss on client_golang side. It might some change that might slip even through you review @SuperQ.
• I see many projects in ecosystem using promlog/promslog package for consistent logging. Not only Prometheus (e.g. Thanos).

It's just not possible to stop users from using useful common utilities outside of Prometheus 🙃

Second argument/question is.. should stable module depend on unstable module? How can we ensure stability if dependency can break anytime? Doesn't go mod tooling punish quite much deps with indirect deps that have breaking changes? It's ok for binaries maybe, but is it ok for libraries like client_golang?

How would this actually work? What's the day-2 plan? Would we keep release-1.x branches? How long would we support them for?

What do you do with Prometheus 1.x version? You keep it, don't support it. That's ok and enough. Let's give option for people to control when they have time to make manual changes to their code, not wake downstream users in random moments because common hiddently broke 4 layers down the upgrade for irrelevant pieces of code.

IMO we don't have the people-time to maintain several release branches over any time in order to support older major versions. We barely do this (LTS) for prometheus itself.

Yes. But equally we don't have people-time to deal with the common upgrade breaking other people code the moment they upgrade client_golang (e.g. https://github.com/prometheus/client_golang/releases/tag/v1.21.0) which suppose to be stable. It just feels like a ticking bomb.

Also, how do you define breaking? Just function signatures? Structs? Just function removals? There are lots of small things that could be breaking or deprecated. Any behavior change comes with concequences.

All of the above like client_golang is enduring for years, with all it's glory and flaming feedback when we break people by mistake 🙃 (or intentionally - sometimes you do break but you make it clear and loud, and you give mitigation plans).

[bwplotka]

One alternative is for client_golang to stop depending on prometheus/common which might be the desired outcome here. We could vendor what we need if API stability is not on the roadmap for this module.

I definitely agree common has so many different packages with variety of stability needs.

[bwplotka]

If anyone is interested on why stability of Go libs is essential, feel free to read a short story from @dims on k8s channel: https://kubernetes.slack.com/archives/CHGFYJVAN/p1741621071818569