What is the "hash" in the metadata dataset exacly, are you sure it is SHA256?

[venthur]

According to the docs, the metadata-dataset about every file uploaded to PyPI, i.e. the parquet files listed in https://github.com/pypi-data/data/raw/main/links/dataset.txt, contain a SHA256 hash. However, it is not described how the hash is calculated.

When trying to verify that you calculate the SHA256 over the respective file itself, i encountered some issues:

• your hash is too short for a SHA256, it has the same length as a SHA1 though
• however, when i calculate the SHA1 of a downloaded file, it does not match yours (neither does SHA256)
• two files in your dataset that have the same hash, also have the same SHA1 hash on my end, however, my and your hashes are different

Can you explain, which hash you are using and if you are hashing the contents of the file linked to via the path?

Thank you very much for the awesome dataset!

[orf]

Hey! Thanks for the kind words @venthur. Sorry about the late reply: I had these messages filtered out.

It's been a while since I looked at this project, but you're right: it's not a SHA256 hash. I dug into where I generate this, and for some reason I chose to use Oid::hash_object here, from libgit2.

That was... an unfortunate decision, and I can't see why I chose to do it that way. Parts of this project where pretty experimental, and I was pretty much learning Rust at the time.

So it's going to be a SHA1 hash of blob ${length}\0${content}. Which is actually so inconvenient.

I'll try and think of a way to rectify this.

[kam193]

Ah, it was an interesting choice :D

Do you maybe think about extending the current ingestion to include a real SHA256? It may not fix the old data, but would give us a growing way database for the future :)