Failed Trimming Links for 6-1-2024 wiki dump

[lennonc1atwit]

[INFO] Download date: 20240601
[INFO] Download URL: https://dumps.wikimedia.org/enwiki/20240601
[INFO] Output directory: dump

[WARN] Already downloaded sha1sums file
[WARN] Already downloaded redirects file
[WARN] Already downloaded pages file
[WARN] Already downloaded links file
[WARN] Already trimmed redirects file
[WARN] Already trimmed pages file

[INFO] Trimming links file

real    13m37.643s
user    13m43.964s
sys     3m58.226s

real    13m37.648s
user    13m43.967s
sys     3m58.229s

Tried using n1-highmem-16 (16 vCPU, 8 core, 104 GB memory) due to larger file sizes then when procedure was written (fails using recommended too), produces an empty gz temp file.

[jwngr]

Possibly related to #100.

[MrElyazid]

I am working on this issue, it seems that most of the scripts used to build the database need a re-write since the schema for the table pagelinks in the sql dump has changed ( now the ids are directly in the table, which is actually good ), and also some of the python scripts have some old python2 syntax, i will try to submit a PR when ready with proper scripts

[MysaaJava]

I already rewrote most of the scripts, and i will write a pull request soon.
You can check what i did on my fork of the repo, branch stuff (i should make a clean commit on main branch soon)

[jwngr]

Hello, thank you for the pull request! I appreciate you helping get this script working again after many years of it being broken. I'm hoping to have some time to take a look at this and run it myself over the Thanksgiving holiday. I doubt I'll find time before then. Wanted to say hi and thanks in the meantime.

[levrik]

Hi @jwngr. I wanted to say thank you for this project. It was a big help in what I needed for a side-project.

For this I didn't only rewrite the data preparation code in TypeScript but also the whole algorithm. I store all incoming/outgoing links in a custom compact binary format which I can instantly load to memory. For German Wikipedia that's around 1.3GB of data on drive and since I'm reading it on byte-level, it also does not consume any more in RAM than that. After some more optimizations I can find paths between pages locally on my M1 MacBook in 5-30ms. I was wondering if you're interested in that stuff, might want to port it to Python or whatever. Just let me know.