Technology & library reevaluation

[jwr1]

Interstellar's code base is growing steadily, and I think it would be a good time to reevaluate some of our technology & library choices, so here are just a few ideas I had (this is by no means definite, just some brainstorming).

Routing with Navigator 2.0

The Navigation and routing page in the Flutter docs gives a great overview on the different types of navigation that Flutter provides. ATM, we're using "Navigator 1.0", which is an imperative API that is extremely simple and easy to use: call Navigator.push to add a screen, and call Navigator.pop to remove a screen; nothing really special about it. Flutter also has "Navigator 2.0", which is a declarative API that defines screens based on page route paths, similar to how a website might have a "/user/johndoe" URL path.

With "Navigator 2.0", it is extremely recommended to use a package with it. The two packages I've found are go_router and auto_route. go_router is more popular and officially recommended in the Flutter docs, but auto_route uses code generation (which we conveniently already use) to provide a type safe API for switching routes.

If we want to support the web platform (which wouldn't be a bad idea), then we would definitely need to switch to "Navigator 2.0". Besides web support, we would also be able to set up deep linking if we thought that would be nice.

Storage

I've seen people recommend Isar and Hive in the past, but those are both long dead, and I don't want to migrate to something that's already unmaintained. I did read this interesting blog post detailing the best local databases for Flutter. I think I've narrowed it down to these three options though.

Drift (SQL)

Drift is a reactive library to store relational data in Dart and Flutter applications, built on top of SQLite.

Drift seems to be the best contender if we want a local database built on top of SQLite. Due to being SQL, it makes migrations extremely easy when needed, and we can also run any complex SQL query required in the app to fetch any data. One thing that's cool about Drift is that it's so popular that it has a bunch of different integrations from external packages, such as this in-app drift db viewer, which would be great as a debug tool.

sembast (NoSQL)

Yet another NoSQL persistent store database solution for single process io applications. The whole document based database resides in a single file and is loaded in memory when opened. Changes are appended right away to the file and the file is automatically compacted when needed.

This is what we're already using, so it already has a +1 due to not needing to migrate to it. It's good to rethink it, though.

Just some thoughts: In the past, I have found it a bit "clunky" to use; maybe I'm just making this up, but the API, especially for making search queries, just feels weird to use, but I probably just need to get used to it. I do find its lack of type safety in certain areas annoying too. Additionally, I'm not really sure how I feel about the whole database being loaded all at once in memory once the app starts. I'm sure our database would never be that big that we have to worry about it crashing people's phones, but I still feel like it's something to think about. Additionally, sembast doesn't have any migration mechanisms (correct me if I'm wrong); migration in general is a pain to handle, but it is something that an app has to deal with if it's around long enough. All I'm saying here is that it wouldn't hurt to consider our options; maybe we should stick with sembast, or maybe we should switch to something else.

Roll our own (Filesystem)

This is an option I would consider as seriously as the other options. If we rolled out our own system of data storage, we would have full control over it, and Interstellar would integrate seamlessly with it. What I'm imagining is that we just use the OS's own file system to store and organize everything. Each settings profile, for instance, could have its own file in a profiles directory. I've looked at Firefox's data storage before, and it similarly has a folder to store data for each of its profiles. Doing it like this would also make the data extremely easy to debug, share, export, import, and manipulate. I would imagine the file structure could look something like the following:

• interstellar.jwr.one/
  • entry (contains values needed for app startup, such as the selected account and profile)
  • profiles/
    • Main.json
    • Moderate.json
  • accounts/
    • kbin.earth.json
    • johndoe@kbin.earth.json

Of course, just using the filesystem wouldn't work (at least not easily) if we wanted to support web.

State management

Provider's been working pretty well for us for the most part, but we could always consider a more advanced state management library. The two big ones are Riverpod (which was actually developed by the creator of Provider) and BLoC. I've never used either of them though, so if we did want to switch, I wouldn't know which one to go for; I'd have to do some research.

---

Let me know your thoughts on all this @olorin99! I just wanted to put out some ideas to make the codebase more maintainable.

[olorin99]

For navigation is seems like a good idea to move to navigation 2.0. From a skim of the two suggested packages I like the look of auto_router more. Type safety is always nice to have.

In regards to storage what are the advantages for us using SQL? I believe most of what we store on disk is fairly simple data. I agree the search api is a little awkward in sembast but not sure if its worth migrating to another solution.

I also don't know much about the various state management solutions for flutter so can't say much for this.

[olorin99]

On second thoughts I'm all in on changing to SQL. Having structured data would be much easier to manage. Also it might just be me but exporting and importing the sembast database seems to have halved the storage used (4.7mb -> 2mb, mostly read posts). I don't think the compacting algorithm used is triggering since checking the file in a text editor shows a lot of duplicated entries. I guess just not enough relative to the total size to trigger the compaction.

[jwr1]

That's funny, I was literally just wondering if SQL would work better as I was reviewing your cache PR.

Timeframe wise, I think it would be best to save the migration for version 0.11 instead of this update, since it's already been so long since the last release and I'm still trying to work on Rules.

[olorin99]

Yeah makes sense to separate big feature release from big infrastructure changes too.

[olorin99]

Regarding #324, it looks like this is one of the cases which Riverpod helps with.

To handle user tags I think we would do something like this

// create a provider to fetch and handle the async state
@riverpod
Future<List<Tag>> fetchUserTags(Ref ref, String username) async {
    final database = ref.watch(databaseProvider); // assume database is also in a provider

    return query to fetch tags
}

...

Widget build(BuildContext context, WidgetRef ref) {
    // request tags
    final tagsValue = ref.watch(fetchUserTagsProvider(widget.user.name));
    // pattern match to handle loading and error states
    final tags = tagsValue.when(
        data: (data) => data,
        loading: () => [],
        error: (err) => [],
    );
...    

With this the tags value should always be valid and we don't have to worry about whether the widget is mounted or not. The widget will automatically be refreshed with the new values when the provider returns.

And considering Riverpod is pretty much Provider 2.0 it should be fairly straightforward to do the initial switch keeping the same behaviour of Provider and then make updates later to take advantage of the more advanced features of Riverpod.

[jwr1]

Wow yeah, your example does look interesting! I'm fine going with Riverpod, but I think I (or you) would want to do a little more research before totally ruling out BLoC (or other solutions). Just to ensure we select the solution that's best for us.

On a somewhat different (but similar) note, I wonder if there's any good way to remove the intermediate state we've created in AppController. I hate how we've basically duplicated what the database already contains, and so now throughout AppController we need to worry about keeping both the controller state and the database in sync. I've already noticed a case where they went out of sync, which caused a bug to appear only once the app was reloaded and the app state was re-pulled from the database (the bug with the empty account handle).

[olorin99]

I believe both Riverpod and BLoC automatically handle caching values from queries and so forth. So rather than managing the controller state ourselves we might be able to have it automatically reload the state whenever we trigger a state change.

So every time we call setProfile it automatically fetches the profile list from the database and then the in memory cache will always match the databse.

Will need to do some more research though to make sure it is possible.