Reporting and monitoring storage actions

Introduction

Two recent blog posts are focusing on reporting and monitoring of storage events. The first one is a proposal and a specification of a new approach for reporting storage-related events using structured logging (i.e. journal). The second one describes an implementation of two proof-of-concept tools - for reporting MD RAID events using structured logging; and for monitoring such events and displaying them as desktop notifications.

However, there are other things happening to storage than just failures and recoveries. The storage configuration is from time to time changed either by administrator(s) or automatically as a reaction to some trigger - e.g. when a thin pool's data or metadata parts are grown when its running out of space [1]. And there are components of the system, together with its users, that could/would benefit from getting the information about such changes.

Real-life example

Here's what happened to a colleague of mine a few weeks ago. They had a new laptop installed and they didn't like the name the Fedora installer used for the swap LV. So they renamed it. And guess what? The system didn't boot on next reboot. Could be obvious to more experienced users when pointed out like this, but I believe less experienced users have just no idea what happened. The /etc/fstab file needed to be updated when the LV was renamed to use the new name on the line defining which device should be used as swap. If they renamed the LV used for the root (/) file system the situation would be even worse because it would be not only /etc/fstab being out of date, but also the GRUB boot/configuration entries defining where to find the root file system. And if they renamed the VG with those "system" LVs? Boom!

Computer-assisted administration

Of course, it's the administrator's responsibility to do all the necessary changes at once (today's systems in many cases have one real human user who is thus also the administrator). As any other human being, the administrator, however, makes mistakes and forgets to do things. On the other hand the system can be improved to help the administrator make less mistakes. The best way to do so is through automation. If there was, in the situation described above, a tool monitoring such changes and reacting on them properly, thus updating the /etc/fstab file (and potentially also the GRUB configuration), everything would work just fine. The system would adapt to the change done by the human administrator. Sounds cool, doesn't it? Or, if the automated reaction wasn't possible or desirable, there could at least be some notification reminding the administrator that they need to do a couple more things.

Roadblocks

Having at least such notifications would be a big improvement. So what is the problem and reason why we don't already have anything like that? The same as with (no) reports for failures and recoveries - there's no source of such data/reports to monitor and get the necessary information from. Somebody may now think: "Wait, there's udev and the uevents it produces!". Sure! Well, kind of. Any idea what is going on here? (output from udevadm monitor):

UDEV  [192.247445] change   /devices/virtual/block/dm-0 (block)
UDEV  [192.294074] change   /devices/virtual/block/dm-0 (block)
UDEV  [192.306974] change   /devices/pci.../block/sda (block)
UDEV  [192.319068] change   /devices/pci.../block/sdb (block)

(the last two lines are ellipsized to fit)

This is how an LV rename looks like. So if there's a tool monitoring uevents, this is what it gets. And the best chances it has are to figure out that either "hey, /dev/dm-0 is an LV, let's update the information about it from LVM and see what has changed" or "hey, /dev/sda is a PV, let's update all the information about the particular VG and its LVs and see what has changed". Which is a total waste of resources and not really reliable approach. And things get much worse if more complicated changes happen. Any guesses on this?:

UDEV  [1215.630950] add      /devices/virtual/bdi/253:1 (bdi)
UDEV  [1215.634760] add      /devices/virtual/block/dm-1 (block)
UDEV  [1215.665616] change   /devices/virtual/block/dm-1 (block)
UDEV  [1215.687381] remove   /devices/virtual/bdi/253:1 (bdi)
UDEV  [1215.690728] remove   /devices/virtual/block/dm-1 (block)
UDEV  [1215.696313] remove   /devices/virtual/block/dm-1 (block)
UDEV  [1215.709090] add      /devices/virtual/bdi/253:1 (bdi)
UDEV  [1215.710842] add      /devices/virtual/block/dm-1 (block)
UDEV  [1215.740192] change   /devices/virtual/block/dm-1 (block)
UDEV  [1215.752385] remove   /devices/virtual/bdi/253:1 (bdi)
UDEV  [1215.760079] remove   /devices/virtual/block/dm-1 (block)
UDEV  [1215.762538] remove   /devices/virtual/block/dm-1 (block)
UDEV  [1215.795592] add      /devices/virtual/bdi/253:1 (bdi)
UDEV  [1215.800114] add      /devices/virtual/block/dm-1 (block)
UDEV  [1215.808450] add      /devices/virtual/bdi/253:2 (bdi)
UDEV  [1215.811551] add      /devices/virtual/bdi/253:3 (bdi)
UDEV  [1215.812843] add      /devices/virtual/block/dm-2 (block)
UDEV  [1215.813891] add      /devices/virtual/block/dm-3 (block)
UDEV  [1215.815970] add      /devices/virtual/bdi/253:4 (bdi)
UDEV  [1215.817625] add      /devices/virtual/block/dm-4 (block)
UDEV  [1215.843361] change   /devices/virtual/block/dm-1 (block)
UDEV  [1215.852085] change   /devices/virtual/block/dm-3 (block)
UDEV  [1215.870895] change   /devices/virtual/block/dm-2 (block)
UDEV  [1215.886220] change   /devices/virtual/block/dm-4 (block)
UDEV  [1216.018808] change   /devices/virtual/block/dm-0 (block)
UDEV  [1216.130786] change   /devices/pci.../block/sda (block)
UDEV  [1216.150838] change   /devices/pci.../block/sdb (block)

A linear LV was converted into a raid1 LV.

It's probably clear now that udev is not a great source to monitor to get information about changes. In this case it is only good for telling its listeners that something happened providing hints about with which device(s).

Moving forward

It is probably not going to be a suprise that a proposed solution for the above problem(s) is again going to be structured logging and journal as the source to monitor and get information from. Originally we thought that changes like device rename, resize, etc. could just be reported using the specification from the first blog post of this series. After all, if somebody or something renames a device, the device is renamed and so it is a state change. However, that's quite cumbersome and enforcing everything to be "twisted" like this just to fit into some "one for all"/"golden" specification, that is really not much generic, is not a good idea.

Especially when it's so cheap and easy to simply have another catalog entry (template) [2] specifying a report of generic storage related action. It can be as simple as this:

-- 73b25b99a72149318e4b7226d96ba1b4
Subject: Storage action reported by @SOURCE@
Defined-by: storage-action-reporting
Documentation: man:storage-action-reporting(7)
Documentation: man:@SOURCE_MAN@

@ACTION@ performed on @ENTITY@

where @ACTION@ would simply be rename (or even lvrename) or resize, etc. @ENTITY@ should of course be an entity the action was performed on, not limitted to just devices because for example a VG is not a device plus there are other things actions related to storage configuration can be performed on -- for example the files like /etc/fstab, /etc/crypttab, etc.

Together with the above fields used in the catalog entry, the source of the report should also add the PROPERTY, OLD_VALUE and NEW_VALUE fields if that makes sense. Plus a mandatory field PERSISTENT_ID which has to be a persistent (as much as possible) identifier of the ENTITY (uuid, wwid/wwn, serial,... for devices and file systems, paths for files, etc.). This extra value would simplify recognition/filtering of all reports related to the same entity. So something like lvrename system/root f25_root would be reported with the following fields:

MESSAGE_ID=73b25b99a72149318e4b7226d96ba1b4
SOURCE=lvm2
SOURCE_MAN=lvrename(8)
ACTION=rename
ENTITY=system/root
PROPERTY=name
OLD_VALUE=root
NEW_VALUE=f25_root
PERSISTENT_ID=rOAP7U-WbOw-0xu2-QQi2-VJQX-g0un-ZvPzoI

and some extra LVM-specific information that can simplify/improve processing of such reports:

LVM_VG=system
LVM_LV_UUID=rOAP7U-WbOw-0xu2-QQi2-VJQX-g0un-ZvPzoI
...

With a log entry like this, any tool monitoring the journal (and filtering out everything with different MESSAGE_ID s) can easily get all the necessary information and do whatever it wants to do with it.

Conclusions

The sections of this blog post describe a problem area and a specific real-life situation which seem to be much more complicated than necessary, together with a proposed way to improvements using a technique very similar to what was proposed in the previous two blog posts of this series. Again, the journal is leveraged as a common place for storing and retrieving information about storage related events. Only this time those events are not failures/recoveries or changes of some devices' states, but actions performed on any storage related entities. Getting such information seems to be very useful. One could say that almost as useful as the information about failures.

"Target audience" for such action reports seem to be tools helping administrators to do changes to their systems properly, tools showing users notifications for actions happening in the background and potentially also tools/daemons requiring the up to date data about the storage configuration in order to provide means to monitor or change the configuration (udisks, GNOME Disks, but also e.g. file managers mounting removable devices, etc.).

Whether such reports should replace uevents is an open question. udev has never been meant to be any big database of all information about all devices in the system and it wasn't designed to provide any high-level events. So using other mechanisms for sharing the information about storage related actions and changes could be a great step forward for all tools, daemons,... requiring such data. On the other hand, journal was (also) not designed to be such "message bus" so it is unclear if using it for these purposes would be a great idea and not just another misuse.

Anyway, logging all the actions in a both human and machine readable format definitely looks like a good idea and something many tools and users can benefit from.

Opinions? Suggestions? Please share them in the comments!

[1] and there's somewhere to grow in the VG, of course!
[2] more in the first post of this series
For anonymous comments click into the Start the discussion entry, fill in your name (or a nick) and check the I'd rather post as a guest checkbox. Anonymous comments with links might require approval by an administrator.