429 - too many projects

[debuggerone]

Problem description

I am building an abstract package with interfaces for stuff like storage, cache, etc

Want to use that in my quicpro package so I can for example just accept a "storage" instead of a hardcoded solution for S3 or minio or local storage.

So I made 122 repos for the individual drivers for stuff like S3, minio, ... and wanted to reserve the pypi project names to have some sort of consistency before I implement the interfaces (got the interfaces for the different categories already).

How can I get rid of the 429 error?

[debuggerone]

    def load_storage_backend(self) -> Optional[StorageInterface]:
        """ Dynamically loads the storage backend from the config. """
        storage_config = self.config["storage"]
        class_path = storage_config["backend"]
        init_args = storage_config.get("init_args", {})

        try:
            module_name, class_name = class_path.rsplit(".", 1)
            module = importlib.import_module(module_name)
            storage_class: Type[StorageInterface] = getattr(module, class_name)

            if not issubclass(storage_class, StorageInterface):
                raise TypeError(f"{class_path} does not implement StorageInterface")

            return storage_class(**init_args)

        except (ImportError, AttributeError, TypeError) as e:
            logger.error("Failed to load storage backend: %s", e)
            return None

and then in a config file I could do stuff like this:

storage:
  backend: "core-storage-azure_blob.AzureBlobStorage"
  init_args:
    bucket_name: "my-session-tickets"

and change it to

storage:
  backend: "core-storage-s3.S3Storage"
  init_args:
    bucket_name: "my-session-tickets"

I thought that's a pretty solid approach for handling Vendor Lock-ins

[debuggerone]

tried again - was able to up a new package - but now getting a 403...

[merwok]

There are no reservations on PyPI!

[debuggerone]

That would have been for a few hours. It totally broke my flow.

[merwok]

I don’t understand how you can have a flow based on something that does not exist.

You want to publish a ton of packages: this is possible, if you publish each one as it is ready (with a sdist and/or wheel). I am not sure if PyPI will enforce rate limits on you.

Are you uploading manually (using twine for example) or automatically from github actions?

[woodruffw]

How can I get rid of the 429 error?

You're getting this because you hit one of PyPI's rate limits on simultaneous project creation. PyPI imposes these on a per-user and per-IP basis to prevent abuse.

In general, you shouldn't be relying on squatting to 'reserve' names on PyPI -- PyPI doesn't support this kind of reservation at all, and engaging in mass squatting is a violation of the AUP, even if your motives are good.

Practically speaking, I think you have two options here:

1. Create individual projects as you need them, instead of relying on squatting. You'll continue to hit rate limits at this volume.
2. Revisit your architectural assumptions -- instead of a distinct project name for each backend, you could use "extra" specifiers for each of your backends. In other words, foo-bar-backend would instead become foo-bar[backend]. This would require only one project for N backends instead of N for N.

[debuggerone]

I want to inject driver packages into my autocoding pipeline and smaller packages require less token.

So even if I could build foo-storage or foo-cache packages that would mean more complexity and also higher costs.

So I prefer to have foo-storage-s3 or foo-storage-local...

I think it will take about 14 days to build and test them all.

Anyways. I got them all now. The rate limits were reseted after some hours.

Beside that. The overall idea to have wrappers and a core lib with interfaces for all of them isn't too bad, is it?

[henryiii]

This would require only one project for N backends instead of N for N.

Extras just install packages, so this could be a user facing nicety, but you'd still have to have all the N packages.

[woodruffw]

Beside that. The overall idea to have wrappers and a core lib with interfaces for all of them isn't too bad, is it?

It doesn't seem that bad; I think a lot of projects use the namespace package pattern for this. However, I'll reiterate that you should not engage in name squatting just because it makes your AI application slightly simpler. If you genuinely need a lot of package names, you should publish them as you implement them rather than uploading stubs.

Extras just install packages, so this could be a user facing nicety, but you'd still have to have all the N packages.

Ah, right -- this is what I get for splitting my time too much between Python and Rust 🤦