storaged - next evolution step of udisks2
In this blog post I would like to present you the storaged project you may have already heard of. It's the next evolution step of udisks2 with the goal to provide a fast, stable and scalable DBus API for storage configuration/management ranging from personal (laptop) to server (enterprise) use cases.
A bit of history
storaged started as a "complementary daemon" for udisks2 providing API for LVM (at that time heavily required by the Cockpit web management project ASAP). Later the LVM API part was converted into a udisks2 module and an ability to load modules was added to udisks2, but with no luck merging the changes to the upstream udisks2 codebase. The only way forward was to fork udisks2 and continue with the development in the forked project - named storaged. You can read more details about this in an explanatory email from Tomas Smetana 1.
Since then storaged has been getting a lot of attention and contributions including extra features like support for bcache, Btrfs and iSCSI (the client side) as well as new tests and memory and performance analysis. We believe storaged will become the storage configuration and management userspace API similarly to how NetworkManager became an API for networking. Not only for personal/desktop use cases, but also for enterprise solutions.
However, we realize that there's an existing API, the udisks2 API, covering a great portion of the existing features and thus after a short period of providing a new incompatible API storaged was changed to provide an extended version of the udisks2 API. So it can be used as a drop-in replacement of udisks2 with no changes required in the existing tools/libraries/daemons/... using the udisks2 API. New features have been being added to the original API in a backwards-compatible way. In fact, starting with Fedora 25 storaged is being used/installed in place of udisks2. 2
Goals for the future are:
- keep the API 100 % compatible to UDisks2,
- extend the test set and write regression tests covering ~100 % of the API,
- reduce resource consumption (memory, cpu),
- setup a CI/CD instance running tests on pull requests as well as on merged changes (ideally in multiple distributions in the future),
- separate the service logic from the storage logic (by using libblockdev 3),
- replace UDisks2 across GNU/Linux distributions
and of course much more when we are done with these.
But we don't want storaged to become a Fedora and/or RHEL specific project. And that's why I am writing this blog post on behalf of many people contributing to storaged so that we can coordinate our efforts and activities with the wider community and all stakeholders in order to fulfill the goal of wide adoption of storaged across various (all) GNU/Linux distributions and environments and to make sure there's little to no friction caused by our efforts.
So why should distributions and environments adopt storaged? As mentioned before, the API is 100% compatible with the UDisks2 API from both DBus and binary point of view while adding support for multiple extra storage technologies: Bcache, Btrfs, iSCSI, LSM, LVM (including thin provisioning) and Zram. This allows existing tools, libraries and frameworks using udisks2 to optionally and iteratively add support for the above technologies with no disruption to the existing functionality. Desktop environments can allow users to manage their storage deployments in an easier way, support mounting/unmounting LVs, btrfs subvolumes, creating snapshots (btrfs, LVM thin), etc. Also, storaged has already been getting better test coverage than udisks2 together with memory footprint and performance analysis and fixes/improvements.
The Cockpit web management tool/framework 4 is using storaged as its storage backend and relies on fixes and features added to storaged. So distributions that want to provide Cockpit with all its features to their users need storaged.
Unfortunately this doesn't come with no packaging work needed. Using libblockdev allows storaged to share code with other storage-oriented projects as well as to get rid of a lot of code taking care of the storage logic (which already exists in libblockdev), but it introduces a new set of dependencies -- the library and deps for the plugins required by storaged. There have already been some efforts and attempts to package the dependencies as well as storaged itself for other distributions. 5
We believe storaged is a great evolution step of udisks2 that has the potential to become a standard and versatile storage DBus API. A lot of work has already been done and lot more is yet to come soon. Backwards compatibility will be preserved as we improve stability and add new features.
What do you think about the above goals? Do you think GNU/Linux distributions should and will adopt storaged as a replacement for *udisks2"? Would you like your favorite distribution to do so? Or do you see a really bumpy road ahead? Please tell us what you think in the comments and if you know about somebody who should read this post and participate in the broader discussion, don't forget to let them know and send them the link!
libblockdev is a C library supporting GObject introspection for manipulation of block devices. It has a plugin-based architecture where each technology (like LVM, Btrfs, MD RAID, Swap,...) is implemented in a separate plugin, possibly with multiple implementations (e.g. using LVM CLI or the new LVM DBus API). http://rhinstaller.github.io/libblockdev/↩