Expired chapters aren't getting removed

[valdotle]

@curche I think your commit #366 broke the removal of chapters. I tried to wrap my head around what the code is doing but I was having a hard time, so I can't fix it myself:

updated_chapters = [
         chapter
         for chapter in normalised_chapters
         if str(chapter.chapter_id) not in self._posted_chapters_ids
         and chapter.chapter_expire >= time_epoch_now
         and chapter.chapter_timestamp >= time_epoch_three_days_ago
]

Going by git blame, I think this is the relevant part.
From my understanding, the chapters in updated_chapters are getting updated? But we don't want to update newly created entries (assuming this is what str(chapter.chapter_id) not in self._posted_chapters_ids checks for). We don't want to update old chapters (or at least that's the idea to avoid old chapters getting reuploaded), hence the check whether the chapter has been uploaded within the past 3 days (chapter.chapter_timestamp >= time_epoch_three_days_ago), but why are we only updating chapters with an expiry date in the future?

Also, I was trying to figure out what caused the chapters to get reuploaded in the first place, but the api endpoint you updated doesn't work (assuming that's even relevant here).
Are old chapters becoming briefly unavailable, get deleted and when they become available again get reuploaded? Are M+ actively deleting and reuploading chapters? Or are the reader urls of existing chapters changing?

[curche]

Issue acknowledged. Will be bit busy so might be late to reply properly. Based on a cursory reading, here's some hopefully useful pointers

1. Try to collect some mangadex titles and see if there's more patterns. Tried a random page no in the group feed and found one interesting case https://mangadex.org/title/64ac1d0a-3170-4dbb-b24d-645a2f16ba8c/vigilante-boku-no-hero-academia-illegals (check the /viewer/[\d+], also might need to check the dedupe logic (see 2. below))
2. Check the parent repository https://github.com/publoader/publoader/blob/2ec07c5555993cf9944af94a90284db6aa9e9d02/publoader/extension_uploader.py#L114; Expired chapters are cleaned up in a two step process afaik.
3. The api url does work, because if not there'd be no new updates past work (its also what I use via the app daily while reading). Seems like the order of the query parameters matters, &format=json should go at the very end or not at all (in which case, it's raw protobuf content). That said, there's been some webview blocking, reuploading (see 1) and everything.

[valdotle]

1. Try to collect some mangadex titles and see if there's more patterns. Tried a random page no in the group feed and found one interesting case https://mangadex.org/title/64ac1d0a-3170-4dbb-b24d-645a2f16ba8c/vigilante-boku-no-hero-academia-illegals (check the /viewer/[\d+], also might need to check the dedupe logic (see 2. below))

The title doesn't reveal anything out of the ordinary, unfortunately. Chapter IDs are in ascending order, and the start/end timestamps make sense. I think we have to log would-be edits to figure out what actually changes, instead of doing guesswork based on the outcome...
The duplicate chapters are due to there being two titles (one re-edition...). I don't think the dupe checker is supposed to take care of this scenario.

2. Check the parent repository https://github.com/publoader/publoader/blob/2ec07c5555993cf9944af94a90284db6aa9e9d02/publoader/extension_uploader.py#L114; Expired chapters are cleaned up in a two step process afaik.

From a first look at the code you linked, that part only deletes chapters of titles that have chapters available on MD but not M+. That seems pretty useless tbh but shouldn't be affected by nor affect the current issue.
I'll have to keep searching for other things. I even found one that I think should take care of this, but it doesn't, apparently. It's a bit tricky to figure out which extension data gets passed under which name and stored in/retrieved from what attribute when/where...
Based on my current understanding, the list of updated chapters is used for determining whether a chapter needs to be created (new) or edited (updated)...

3. The api url does work, because if not there'd be no new updates past work (its also what I use via the app daily while reading). Seems like the order of the query parameters matters, &format=json should go at the very end or not at all (in which case, it's raw protobuf content). That said, there's been some webview blocking, reuploading (see 1) and everything.

Yeah, I figured I was doing something wrong. But for the url query param order to matter would be stupid beyond belief. Thankfully, it's not that bad, I just typo'd...

[curche]

But for the url query param order to matter would be stupid beyond belief. Thankfully, it's not that bad, I just typo'd...

Heh you'd be surprised how easily scrapers can be blocked

Anyways, I'll be busy for a bit more so might not be able to dig deeper soon. I can revert my previous problematic commit in case there's any alternative solutions for what it was trying to solve (some wednesdays, m+ does special offers where they let you read older premium chapters for free. For eg when an anime adaptation happens..).

Ofc a great alternative would be to just not delete said expired chapters and just mark them unavailable. While I'm not sure how effective it is, in the interest of everyone's time, it might make sense to get assistance from copilot/claude/whatever-is-trending... bunch of older commits were coded like that anyways so 🫠

[valdotle]

Ofc a great alternative would be to just not delete said expired chapters and just mark them unavailable.

I agree, something less destructive (either disabling/marking as unavailable) would be nice, but afaik that's done via the admin API, which is both undocumented and might require additional permissions the bot acc doesn't have 🤔