The journey of building Timelines sync with Vapor

by Lukáš Petr

What makes syncing possible in Timelines is Timelines Cloud - a custom sync solution that I've built from the ground up.

It's been a long journey for me to figure out which way I want to go with sync, and after trying some pre-made solutions such as Ensembles or CloudKit, I ultimately decided to go with a custom solution - building both the client-side and server-side logic for syncing myself. While this was the most difficult option, and I spent way more time and energy on it than I originally envisioned, it has allowed me truly optimize it for my model schema, and to make it very efficient - in terms of minimal data transfer, and also in terms of reliability, performance, and security.

Some technical details

Behind the scenes, the app keeps track of changes occurring on your device and sends them to a central server. The server then notifies your other devices using silent push notifications that new content is available. Changes (inserts, updates, and removals) are contained in changesets, and those are intelligently compressed for better performance.

It is the app's responsibility to first download what's new on the server and then to merge that with what has happened locally, and then to upload that to the server. That way the server always contains a valid history of all the inserts, updates, and deletes that make up each user's dataset.

On the server, I'm using Swift and the Vapor framework. I'm also using Docker for managing deployment and PostgreSQL for storing and querying the changesets.

In terms of some of the specific challenges that came up along the way:

  • handling the case when new local changes happen while the app is downloading and processing changes from the server - in that case, these local changes need to be applied again to preserve the same ordering of changes both locally and on the server

  • merging local unsynced dataset with what's already on the server

    • for this, I first create a "history" of inserts from the local data, and then I am downloading everything from the server in batches, subtracting from these local inserts. Objects are identified by their unique identifiers, and last edit timestamps are used to identify the same versions of objects (or - if the local insert has a newer timestamp, it's instead turned into an update of what's already on the server). Then, what remains after this subtracting (updates or newer inserts) is uploaded to the server.

  • handling the case when a timer is started in the same category on two different devices. If this case wasn't handled properly, there would be a duplicate event. So, after the sync is finished I do a sanitation pass. This also handles the case if edits on different devices were made that would result in an event with a negative duration (its start date being later than the end date).

  • performance of fetching large datasets - both changesets uploaded to the server and changesets downloaded to the client are split into batches if they exceed a certain threshold. The app only needs to do a full sync once, and then just updates based on new changesets that were uploaded to the server since the last time the client synced. This is different from the peer-to-peer solutions that sometimes need to do a full sync of the complete dataset even after the original sync has finished, just to get data across devices into a consistent state

  • implementing the change tracking itself - fortunately, Core Data has great support for tracking changes, so that helped a lot.

  • making sure that state is saved and changes are processed in such a way that even if the app is killed halfway through the sync, it still functions correctly the next time the sync is resumed

  • tracking of changes happens at the level of individual attributes, to ensure that if one object is changed on multiple devices at the same time (but on each device a different field is edited), the resulting object after both devices have synced will contain both of these changes

Is Vapor truly production ready?

The concern many developers have with using Vapor is whether it's truly ready to be used in production. Speaking from my experience - yes, it is. Especially now with version 4, and its 7 years of development and maturing.

I've been running a server for managing iOS subscriptions in Timelines that's written in Vapor for two years now. The big advantage you get with Vapor is that you get to use Swift, Xcode, and generally, all the tooling that you are used to if you are an iOS developer. All the type safety and modern features of Swift are heavily used all throughout Vapor.

Another big reason why I was comfortable with using Vapor is that, under the hood, it uses SwiftNIO by Apple. And there is also the Swift Server Workgroup - a team of people both from Apple and from third-party frameworks working together to promote the use of Swift to build server-side applications.

Also, if not more important - Vapor has a vibrant community around it that is super responsive and helpful whenever an issue arises. And - maybe that's entirely subjective - but I find it cool and fascinating to be able to use Swift on the server. There is also something that drew me to go with the project that's been considered an "underdog" compared to Angular and the other more established Javascript or PHP frameworks. To be completely fair - I did run into a few problems that probably wouldn't have happened with other more mature frameworks, but whenever that happened it was always addressed very promptly by the Vapor team.

Acknowledgements

What applies here is that my sync solution truly stands on the shoulder of giants, as the saying goes. I wouldn't have been able to build it if it wasn't for the technologies available, and for the help of specific people, both my friends and people from the community.

I'd like to thank Drew McCormack for building Ensembles, a true changesets-based sync solution for Core Data. While I couldn't use it as a whole in the end, the way change tracking is implemented there was a big inspiration for building my own solution.

I'd like to thank Tim Cordon and the whole core team for their relentless work on Vapor. His talk at iOSDevUK in 2019 got me excited to seriously consider using Vapor for my app. Also, I really appreciate the timely help that I got from the Vapor community on Discord whenever I ran into a problem with Vapor or needed some general help with server-side development - you guys are the best!

I'd also like to thank Cyril Courtelier - my good friend and a very smart developer. He helped me at a critical time with the idea of basing the sync solution on how Git is doing it - that the server should reject incoming changes if there were new changes uploaded since the last time the app has synced with the server. That was a critical piece of the puzzle for me. He also helped me with coming up with sanitation rules to make sure that the resulting data is valid.

Big thanks to Russ Shanahan for his encouragement at a time when I was debating just quitting altogether because it all seemed too difficult and uncertain. His validation of the idea that having a clean history of inserts, updates, and removals stored on the server should make things more reliable was exactly what I needed at the time.

Also, big thanks to all the testers who had been using the sync for months and helped me refine it further.

Also, to all of you who have asked over the years how this is going and have encouraged me along the way - thank you, I really appreciate it!

Timelines 3.0 with this sync mechanism is available starting today. You can get it on the App Store here.