Add Kodi library support (without the need for MySQL)

[dagwieers]

Feature request

I'm submitting a proposal to ...

• new feature
• change an existing feature

Describe your request

We started implementing a solution that allows you to add Movies and Series to Kodi without the need of having a MySQL database or a complex setup. The implementation is quite straightforward.

Describe the use case

The way this works is that the user adds 2 new sources to Kodi, pointing to a Movies and TV shows listing, eg.

• plugin://plugin.video.netflix/library/movies
• plugin://plugin.video.netflix/library/tvshows

These sources are typical Virtual Directories (comprising ListItem objects) and will be indexed into Kodi's library.
This works both with Kodi setups using local databases, or a shared MySQL Videos library.

Development tips

Here is the implementation for:

• Streamz
• VRT NU

Specific details

See more information at:

• Support plugin sources for video library with metadata.local scraper + related improvements/fixes xbmc/xbmc#13566
• [VideoLibrary] Do not scan unsupported plugin sources + fix database cleaning xbmc/xbmc#14210
• https://github.com/AkariDN/plugin.video.testlib

[CastagnaIT]

seem interesting, but i have a lot of perplexities on this method and some leak of info
because currently the mysql database is used to share netflix data that cointains the exported catalogue and other data to be used between different devices
the method proposed is different and not provide support to exchange data

Then first point is: How to have a central point to exchange data on differents Kodi systems?
Theorizing to eliminate the use of shared data, mainly we lost two functionalities:
-remember audio/subtitle between devices
-watched status manual override between devices (used only for content within the add-on, but little interest..)
And raise a big problem:
i need to query/save in some way the videoid's data of exported items, and i do not understand how is possible to do whitout a shared database

Then second point is:
When you select e.g. a tv show from kodi library, after kodi call directly the add-on navigation, then:
-Where are stored the items(movies/tvshows) exported? i do not understand where pick the data if you browse from another device same path... (referred to data exchange)

Third:
The Kodi sections "Recently added episodes", "Unwatched tv shows".... are updated?
How many calls are made to the add-on when the Kodi library will be updated?
a call for every tvshows? for every season and/or episode? will open thousands of add-on instances?
this could be impossible to manage from NF due too high traffic

These are some of the perplexities
Could be perhaps an idea to replace STRM management code and cut off library management,
but this touch almost the entire code of the add-on there is too much work to do
at moment i don't know what to say

[dagwieers]

Then first point is: How to have a central point to exchange data on differents Kodi systems?

I have a shared MySQL database setup, and this method works out-of-the-box with such a setup. No need to configure anything.
It also shares te resumepoints and what you have watched. (This was working before for my own videos, but now it also works for add-on content).

When you select e.g. a tv show from kodi library, after kodi call directly the add-on navigation, then:

The items are stored in the Kodi library as plugin:// locations pointing to the add-on. So you play using the add-on, but from the Kodi Library itself.

The Kodi sections "Recently added episodes", "Unwatched tv shows".... are updated?

Yes, that also works out-of-the-box. It does require proper metadata (which you already have) and it probably would make sense to add all movies, but e.g. only the TV shows that the user has in his List.

but this touch almost the entire code of the add-on there is too much work to do

We implemented it in two add-ons and it was not a lot of work. The hardest part will probably be to add a unique hash based on the episodes in a TV show (so that Kodi does not scan every TV show every time, but only the ones that actually changed). If you already have a cache of your metadata (our add-ons do) then the overhead can be reduced. Granted Netflix is a lot bigger than our VOD services.

• VRT NU: 432 movies and TV shows (about 50 oneoff videos and 6200 episodes)
• VTM GO: 1390 movies and TV shows (about 500 movies and a few thousands episodes)

[dagwieers]

Wrt. shared settings for subtitles/audio that is not foreseen in Kodi's Video library, so that is not supported.

[CastagnaIT]

I have a shared MySQL database setup, and this method works out-of-the-box with such a setup

i use Mysql to store addon data (in a separated database) to be shared with other addon installations, as said before if i can not use it, will become complex many parts of the add-on must be rewritten and other will be broken, i do not see alternative method

shared settings for subtitles/audio

this is a feature provided from nf add-on not kodi, that need mysql (or similar db/store to share data) to be shared with all devices

The hardest part will probably be to add a unique hash based on the episodes in a TV show (so that Kodi does not scan every TV show every time, but only the ones that actually changed). If you already have a cache of your metadata (our add-ons do) then the overhead can be reduced.

e.g. some parts of the add-on require the ability to get the entire exported list as "videoid" then this have to be rewritten
in different way and require to touch different part of the add-on

The kodi library scan is the best candidate to be a killer NF account,
unlike many other add-ons that load a webpage and parserize the data, here we communicate through the official web API
and i can not manage all these requests, if i do it every time you will have the netflix account banned to until 48h
due to too high requests.
There is no way to understand if a tvshow has been updated by netflix then is needed every time request all data:
E.g. If you have exported 300 tvshow means:
300 (tvshow data) + 300 (list seasons) + (list episodes, per season, we theorize 5 season per tvshow) 1500
To update the library is needed about 2100 http requests!! this is not acceptable... (without takin in account of the paginated results that require more than 1 req to get the episodes list)
and you can't rely on the cache, otherwise you will get wrong updates

Currently library exports and updates are managed via "metadata" API,
this api allows you not to ban the account (immediately) (at max refuses to answer with http error)
but provide provides very few data (no images no authors, no plot etc...) then can not be used in this case...
In above example with metadata API to update the library is needed "only" about 300 http requests

From my POV the method proposed is not so simple to integrate

[CastagnaIT]

An example to try resolve this can be:
still use the "metadata" API to get updates, and query only the changed season/episodes lists
in this way should have:
300 (tvshows "metadata" that include seasons and ep.) + only the query's needed for the new episodes data
this decreases the number of requests, however, it will be slower than the current STRM management to update the lib.

to do this is needed to build a complete separated data elaboration

but still another problematic point:
Currently is possible set the NF Profile to be used to play videos with Kodi library,
(required to have right audio/subtitle languages and send watched data sync to nf)
now this new library method is a sort of hybrid,
to keep this feature is needed to add a new "play" endpoint to be used only with the "library sources endpoint"

there is a lot of thing to look around and add/modify...

But still not understand, things like example:

1. I add a movie from another Kodi device, how i can know that this movie has been added if i use a different Kodi device e.g. to do updates of episodes?
2. I do not understand how Kodi update the library (only manually or startup by settings?) and how schedule the update with this method

[dagwieers]

1. I add a movie from another Kodi device, how i can know that this movie has been added if i use a different Kodi device e.g. to do updates of episodes?

What do you mean "add a movie"? Please be more specific.

• Add a movie to "My list"
• Add a movie to the Kodi Library
• Add a movie to local storage on the device

I also do not understand why you mentioned then "updates of episodes".

The mechanism is quite simple, if you understand the mechanism, you understand the implications and limitations.

2. I do not understand how Kodi update the library (only manually or startup by settings?) and how schedule the update with this method

Kodi only has a setting to update on start-up (this will affect all sources), but we plan to have our own scheduler per add-on to periodically update the sources (with some random factor obviously). There is a JSON-RPC call for updating a source in the Library.

But there is also this project: https://github.com/robweber/xbmclibraryautoupdate

[CastagnaIT]

What do you mean "add a movie"? Please be more specific.

yes before some backgroud, because i do not know if i able to explain without misunderstandings:
Currently a user can export via context menu a tvshow that have an interest (so add to library movie/tvshow), this is saved as strm but also saved in the internal addon db (that can be shared via mysql) this allow me to know what movie/tvshow/season/espisode are exported from any device with nfaddon, and allow me to create a list of exported movies/tvshows from any device

This can be synched with My list but often user not Sync to My list and use it separately

So now with this method, i do not understand on how do the same thing, so e.g.:

• when i want export (add to library) a tvshow/movie what i should have to do with the code?
• where i read the list of exported tvshow/movies when the library is not sync to My List?

[dagwieers]

Most add-ons have the concept of "Follow" (usually for TV shows) or "Watch later" (usually for Movies, oneoffs or episodes). This is stored at the VOD service, so this works cross-device already.

My preference (and this could be made configurable) is to make this available through these plugin:// locations for Movies and TV shows as highlighted above, but list all Movies (and oneoffs) by default, and only list the followed TV series (which is a smaller subset).

This way the user-experience is not harmed, if a user adds something to "My List" it will automatically be listed in his Kodi Library, and new episodes or new movies would be listed there as well. (Automatically here could mean that the add-on would scan the newly added item immediately or at some later time schedules an update for everything)

For Netflix I would make it work like this, unless the total Movies listing is a concern as well, I don't know how often this changes. I understand that Netflix is different than our typical VOD service in scale. It is also possible that the user can select a sub-selection to make visible in the Kodi Library (e.g. only SciFi movies, or Children movies) so that it is somewhat more efficient.

That said, I have never used the current implementation in the Netflix add-on because it is fairly complex and it is specific to Netflix only. The benefit of the mechanism we are considering now would be standardized and supported by Kodi. We have 4 add-ons that are candidates for implementing this.

The only thing missing in Kodi is a way to share your add-on settings cross-device, which would make VOD service credentials and other specific settings to be synchronized. I know there are solutions already for this, but nothing out-of-the-box that is straightforward for normal users :-)

And that said, I understand you have invested in a more comprehensive implementation, and it will not be easy to move existing users to something new and different. I just wanted to give you the heads up that this exists (and existed since 2018 apparently). I don't think a lot of people knew this was possible.

[dagwieers]

So now with this method, i do not understand on how do the same thing, so e.g.:

• when i want export (add to library) a tvshow/movie what i should have to do with the code?

Ok, let's make it more clear. Netflix would have 2 plugin:// locations, one for Movies, one for TV shows.

• plugin://plugin.video.netflix/library/movies
• plugin://plugin.video.netflix/library/tvshows

Each return a list of ListItems, including all metadata (just like any other view in Netflix). This could already be the same view as your current My List, but it needs to seperate the Movies and TV shows (because the library handles them differently).

You can already start playing with this, with minimal changes:

• Add the paths to addon.xml

  <extension point="xbmc.python.pluginsource" library="addon.py">
    <provides>video</provides>
    <medialibraryscanpath content="movies">directory/video_list_sorted/myList/queue/</medialibraryscanpath>
  </extension>

• Add a new video source pointing to a virtual directory in Netflix, e.g. plugin://plugin.video.netflix/directory/video_list_sorted/myList/queue/
• Select Movies and Local information only when adding the source

Beware that you can only add one location per source, more than one is unsupported and will fail to work.

That is it, your My List movies (and unfortunately TV shows) will become part of your Kodi Library. And if you update the Library, new movies will be added.

• where i read the list of exported tvshows/movies when the library is not sync to My List?

You can find the list in Video / Sources once you have added it.

See the pull requests I linked to see how easy it could be to support this, even without removing the existing infrastructure.

[dagwieers]

Unfortunately I cannot use the ZIP file from your repository (anymore) to test this myself. The mysql-connector is no longer included.

[CastagnaIT]

thanks for the tips,
i think for now it's better to keep the STRM/Mysql support because of the big difference of use case
i will try to do some test with the lib endpoint next days