File System Layout
While KOS is based on Linux, and many parts of the filesystem resemble a typical Linux system, most of the user space filesystem is an immutable stack of layers that are assembled at boot time. This note explains the details of the KOS filesystem layout so that users can understand the purpose of the various filesystems and where to find data within the various directories.
Filesystems
A typical KOS node is made up of three distinct filesystems, each serving a particular purpose and containing specific data.
Root filesystem
This is the root Linux filesystem within KOS. This contains all the files and directories of a typical Linux distribution. Unlike a typical Linux distribution, this entire filesystem is virtual and any changes to this filesystem will not be persisted.
KOS is a fine grained immutable operating system. Every boot of KOS is driven by a manifest which enumerates all the components, in the form of KAB files, that make up the running system. Some of these KABs are layers, where a layer KAB contain a single squash filesystem image. These layer KABs are stacked into a union filesystem image during the boot process to construct a complete root filesystem to user space applications.
This ability to stack layers allows KOS to deliver entire user space applications, such as java and chrome as individual KABs. This means KOS has no need for a package manager and there is no install process for Linux applications as they simply appear in the filesystem when included in the underlying manifest.
The union filesytsem that represents the root filesystem also contains a small mutable overlay. It provides the appearance of mutability, but this exists to preserve changes to select configuration files across reboots. Any changes outside of these config files will not be persisted across a reboot. As the mutable overlay only expects to persist Linux configuration changes, is has very limited capacity. Attempting to copy large amounts of data into the root filesystem will quickly result in out-of-space errors until the device is rebooted.
The root filesystem provides several directories specific to KOS:
/mnt/extensions
Extensions are optional parts of the KOS java runtime which provide functionality for specific types of devices. For example, the dispense extension introduces nozzles, pumps, ingredients and other building blocks to facilitate the development of dispensers.
Extensions are loaded as part of the KOS boot process and are available to all applications loaded within the java runtime.
/mnt/logs
The logging infrastructure within KOS is designed to unify all logging and provide standardized tools for processing logs. All KOS standard components log to this directory.
The logs directory is a fuse filesystem that captures all writes and adds standardized high resolution timestamp information, adding a leading and trailing marker on each log entry. These entries effectively form a linked list within the log data, allowing rapid traversal of log data, the ability to integrate log data from multiple sources, as well as the ability to prune corrupt data from the end of log files. Processed log data is written to /mnt/datafs/logs/live.
/mnt/fuse
Within KOS, everything is packaged as a KAB file, and KOS provides the ability to mount a KAB into the native filesystem using fuse. This allows the content of a KAB to be visible to native code and external tools without ever expanding the contents of the KAB. This is the base directory used for any KAB mounts.
As KOS boots, every application in the manifest is automatically mounted in the fuse directory, and application code can then choose to mount additional KABs as needed. Mounting a KAB allows any native code within the KAB to be run natively in Linux. The KAB tooling within KOS also allows the owner, group and permissions of individual files and directories to be specified so that when the KAB is mounted, the contents have the desired permissions and ownership within Linux.
/mnt/storage
Unlike a typical desktop distribution, usb storage devices are not automatically mounted in KOS. The storage service detects insertion and removals of storage devices and applications can react to these events in real time, mounting and unmounting devices as needed.
As usb storage within embedded devices is typically used for transactional operations such as copying logs, installing updates and so on, KOS provide fine grained mount support and defaults these devices to synchronous mode. This allow processes that write to storage to be ready for removal immediately after the copy is complete. Also, by only mounting devices during reads and writes, the device can be safely removed at any time without causing hung mounts within Linux or file corruption on the device itself.
The storage directory is used for all dynamic mounts of storage devices, ensuring a consistent location for every device. In addition, each device insertion will result in a different mount directory, ensuring that references to previously inserted devices cannot accidentally impact future devices.
Image filesystem
Located at /mnt/imagefs, the image filesystem is a read-only partition that contains key boot related data such as all layer KABs and boot manifests. Unlike the root filesystem, this is an actual partition on disk. All interactions with imagefs are managed by KOS internal tools and user code never interacts directly with imagefs.
In addition to storing all layer KABs, kernels, and boot manifests, the image filesystem also stores the boot queue. This is a list of boot manifests in priority order and forms the basis of rollback and recovery. If a given image is not bootable, the boot system will invalidate the manifest and move to the next available manifest in the queue. Given the critical nature of imagefs, it remains mounted read-only to avoid accidental changes as well as minimize the changes of filesystem corruption.
Care must be taken when provisioning the image filesystem partition. It should be sufficiently large to hold the maximum expected number of releases. Since releases can share layers, the actual required space may be much lower than the worst case, but future releases which do not share layers may consume considerable additional space. The KOS installer, when faced with space constraints, will garbage collect lower priority releases to reclaim space.
Data filesystem
Located at /mnt/datafs, the data filesystem is a read-write partition which represents the primary storage area within KOS. While this entire filesystem is accessible to user developed software, there are a number of guidelines and best practices that should be followed to take advantage of all the features that KOS has to offer. The sections below will outline the layout of datafs and provide some insights into how users should use the filesystem most effectively.
/mnt/datafs/kabs
A KOS release is defined by a list of KABs, where each KAB is identified in a chain of manifests, and the manifests themselves are also KABs. The bootable components of a release, consisting of the kernel, root filesystem layers and so on, are stored in imagefs, but all other components are stored in the kabs directory in datafs. As the bootable portion of KOS tends to be a consistent size over time, this allows imagefs size requirements to be fairly predictable, allowing the partition to be sized with confidence. Applications and application data, which may grow to include large assets such as video content, exist in the unified datafs space, which tends to make up the bulk of available storage.
While imagefs enforces pre-boot guarantees about a particular bootable image, the post boot processes in KOS perform a similar verification of datafs components of the release, repairing or rolling back if needed.
/mnt/datafs/core
This directory is reserved for exclusive use by KOS internal services that require persistent storage. No use application code should write to this directory or rely on data in this directory as it is subject to change from one release to another.
/mnt/datafs/logs
As mentioned above, KOS provides a standardize logging interface via the /mnt/logs fuse filesystem. The system logger will then write the resulting normalized log data to various files and directories in this logs directory as described below:
/mnt/datafs/logs/live
As log data is written to the fuse logger, it is appended to files in the live directory. These can be tailed to view live log data and these files are periodically torn and compressed for archival.
/mnt/datafs/logs/archive
As live log files reach size thresholds or cross a day boundary, they will be compressed with gzip and placed into the archive directory. Files that remain in this directory beyond the configured archive duration will be purged automatically.
/mnt/datafs/logs/damaged
On every boot all the files in the live directory are analyzed for trailing corruption. KOS will automatically prune the corruption to allow the log file to continue to be written to. If any file is found to be damaged beyond repair, it is moved to the damaged directory until it is purged by age.
/mnt/datafs/node
Every board that runs KOS is referred to as a node, and every node has a nodeType. The nodeType of a primary node is the type of the entire device. As a node needs to store application data, and application data and configurations for one type of node may not be compatible with other types of nodes, KOS namespaces all application data by node type and this starts at the node directory. The layout of data within this directory is described below:
/mnt/datafs/node/<nodeType>
The root directory of all application storage. If a board switches node type, the data from the previous node type will have no effect no the new software.
/mnt/datafs/node/<nodeType>/database
This directory is reserved for internal KOS use. Any KOS services that store data that is specific to node type is located here, which ensures that it doesn’t not impact any new software if the node type of the board changes.
/mnt/datafs/node/<nodeType>/apps/<appId>
Application specific storage area for an application with the specified application id. This isolates storage between all applications. This is the directory returned from calling Application.getDataDir().
/mnt/datafs/node/<nodeType>/apps/<appId>/db
A directory reserved for database storage for a given application. Calls to Application.openDatabase() result in database files in this directory. These should only be managed by the database infrastructure within KOS and never accessed directly.
/mnt/datafs/mfg
A directory reserved for manufacturer specific data. This is typically used for manufacturing specific processes such as recording end of line device data, birth certificates for the device and so on. The contents and format of any data in this directory is defined by the manufacturer.
