Skip to main content

Flatcar Documentation Analysis

Introduction​

This document is an analysis of the effectiveness and completeness of the Flatcar open source software (OSS) project's documentation and website. It is funded by the CNCF Foundation as part of its overall effort to incubate, grow, and graduate open source cloud native software projects.

According to CNCF best practices guidelines, effective documentation is a prerequisite for program graduation. The documentation analysis is the first step of a CNCF process aimed at assisting projects with their documentation efforts.

Purpose​

This document was written to analyze the current state of Flatcar documentation. It aims to provide project leaders with an informed understanding of potential problems in current project documentation. A second implementation document, Implementation, outlines an actionable plan for improvement. A third document, Issues is an issues list of issues to be added to the project documentation repository. These issues can be taken up by contributors to improve the documentation.

This document:

  • Analyzes the current Flatcar technical documentation and website
  • Compares existing documentation against the CNCF’s standards
  • Recommends a program of key improvements with the largest return on investment

Scope of analysis​

The documentation discussed here includes the entire contents of the website, the technical documentation, and documentation for contributors and users on the Flatcar GitHub repository.

The Flatcar website and documentation are written in Markdown and are compiled using the Hugo generator with the Flatcar theme and served from a GitHub repository. The site's code is stored on the Flatcar GitHub repo.

In scope​

Out of scope​

Any Flatcar content that is not accessed by the documentation URL, https://www.flatcar.org/docs/latest, is outside the scope of this analysis. The FAQ and Blog are outside the scope, however an argument can be made to keep an up-to-date FAQ in the core documentation.

How this document is organized​

This document is divided into three sections that represent three major areas of concern:

  • Project documentation: concerns documentation for users of the Flatcar software, aimed at people who intend to use the project software
  • Contributor documentation: concerns documentation for new and existing contributors to the Flatcar OSS project
  • Website: concerns the mechanics of publishing the documentation, and includes branding, website structure, and maintainability

Each section begins with summary ratings based on a rubric with appropriate criteria for the section, then proceeds to:

  • Comments: observations about the existing documentation, with a focus on how it does or does not help Flatcar users achieve their goals.
  • Recommendations: suggested changes that would improve the effectiveness of the documentation.

The accompanying implementation document breaks the recommendations down into concrete actions that can be implemented by project contributors. Its focus is on drilling down to specific, achievable work that can be completed in constrained blocks of time. Ultimately, the implementation items are decomposed into a series of issues and entered as GitHub project-doc-website/issues.

How to use this document​

Readers interested only in actionable improvements should skip this document and read the issues list.

Readers interested in the current state of the documentation and the reasoning behind the recommendations should read the section of this document pertaining to their area of concern:

Examples of CNCF documentation that demonstrate the analysis criteria are linked from the [criteria] specification.

Recommendations, requirements, and best practices​

This analysis measures documentation against CNCF project maturity standards, and suggests possible improvements. In most cases there is more than one way to do things. Few recommendations here are meant to be prescriptive. Rather, the recommended implementations represent the reviewers' experience with how to apply documentation best practices. In other words, borrowing terminology from the lexicon of RFCs, the changes described here should be understood as "recommended" or "should" at the strongest, and "optional" or "may" in many cases. Any "must" or "required" actions are clearly denoted as such, and pertain to legal requirements such as copyright and licensing issues.

Project documentation​

Flatcar is an incubating project of CNCF. This means that the project should be developing professional-quality documentation alongside the project code.

CriterionRating (1-5)
Information architecture2 - Needs improvement
New user content2 - Needs improvement
Content maintainability2 - Needs improvement
Content creation processes meets2 - Needs improvement
Inclusive language3 - Meets standards

Comments​

The following sections contain brief assessments of each element of the project documentation.

The current Flatcar documentation navigation structure, the table of contents (TOC), defines the areas of knowledge needed to install and provision Flatcar, but it does not show the different paths for new users depending on their environment and expectations.

The following comments are in regard to the top-tier nodes in the current table of contents:

  • The top "Flatcar Container Linux" page contains references and links that appear to refer to an alternate version of the TOC. While its good to provide quick links, the user wonders whether the TOC node references the same content, or if the links in the overview (right side) are supplemental. On mobile the left-side TOC is not available, so the links in this overview serve as the navigation.

  • The "Installing" node contains the large "Cloud Providers" node, which might be better as top tier node. The same with "Bare Metal". The team agrees that "Community supported platforms" could be merged into the "Cloud Providers" node.

  • The installation node should address all the new user paths, providing an installation roadmap or strategy. The "Learning Series" node, introduced into the documentation recently, outlines the key steps for provisioning, configuring, and managing Flatcar throughout the life cycle.

  • The "Provisioning Tools" title is descriptive, but its unclear how other areas of the docs relate to this section.

  • The "Nebraska" node is about updates, but a top node should convey the functionality rather than a product name.

  • The "Setup and Operations" node casts too wide a net in its heading. How does "setup" differ from installation? The node contains several important content areas that should be more discoverable. For instance:

    • "Managing Clusters" might be better at a higher level because it's an initial evaluation in deploying Flatcar.
    • The Storage and Security nodes are typically at a higher level.
    • The systemd is about core OS management, which should be reflected in its title.

  • The "Container Runtimes" node is expected to focus on containers and clusters. An argument could be made to elevate "Getting Started with Kubernetes" and "High Availability Kubernetes" to its own node, as Flatcar has its optimization benefits for Kubernetes deployment.

  • The "Migrating from CoreOS" node might be better placed earlier or within an Installation node, as it is an installation scenario.

  • The "Reference" node contains expected look-up information such as "Constants and IDs" and "Supply chain security mechanisms" but these sections could be better placed:

    • "Integrations" could be incorporated into the Cloud Providers documentation.
    • "Developer Guides" contains conceptual content typically not found in a Reference section and would be better as a top-tier node.

  • The "How to Contribute" node is well-placed and has the expected content.

The documentation also includes an FAQ, accessible from the top banner of the home page. This FAQ has some content that could also be in the docs, such as historical context and how images are updated. Conversely, there should be a few top-of-mind installation and support FAQ items derived from the docs.

Code blocks are different from those in other documentation sets as they are not bordered, don't have a different fill background, don't have a copy button, and the language is not indicated. There is already a GitHub issue on this.

In several topics, new users might not be sure of the context for a given block of code. Is the Linux prompt on a VM, in a client computer, or in a CLI session with a cloud provider? Normally this can be reinforced by the narrative, such as starting a procedure with "In the VM window, use the following command to ..." or similar guidance.

In pages with code examples the narrative follows a conversational flow, introducing the steps such as "First do this", show a block of code, followed by "Now do this", followed by the next block of code. While at first this tone may seem refreshing from sterile nature of technical documentation, it inhibits learning because it inhibits scanning and the ability to reference the exact steps. It's easier to go back a numbered step in a procedure that to read the narrative again to find that step.

The pertinent areas of knowledge to install and run Flatcar have been identified and documented. They just need to be better organized. The next task is to structure the documentation so that it meets these objectives:

  • The node and topic titles provide expected guidance, such as "Getting Started", with nodes and sections organized for precise purposes, rather than headings that serve as broad categories such as "Setup and Operations".
  • The structure provides the expected flow and orientation so that users can identify a pathway for learning about and deploying Flatcar efficiently.
  • The structure provides effective navigation and identification of the tools and technologies so that users can identify the ones they need to learn about and use.

Information architecture​

The overall structure (pages/subpages/sections/subsections) of your project documentation. We evaluate on the following:

Is there high-level conceptual or About content?​

The current content is comprehensive on the key concepts needed for understanding the processes and technologies involved. Now the objective is to organize the concepts to map to the user the path for provisioning and configuring Flatcar.

  • Is the documentation feature complete?

    There are GitHub issues to document how to use Flatcar with particular providers and tools.

  • Are there step-by-step instructions (tasks, tutorials) documented for features?

    Procedures, mainly invoking bash commands, are introduced informally and do not follow the typical style of numbered steps that read "Use the following command to ..." verbiage that would clarify the context.

  • Are there any key features which are documented but missing task documentation?

    Yes, there are such features identified in GitHub issues.

  • Are tasks clearly named according to user goals providing a happy path for users to get their tasks accomplished?

    Not currently. The content areas are established but should be coordinated into a "Flatcar provisioning roadmap" for most user paths.

  • If the documentation does not suffice, is there a clear escalation path for users needing more help? (FAQ)

    The FAQ does not currently provide that guidance. Flatcar users of the GitHub Flatcar repository can find help and community support in the README file.

  • If the product exposes an API, is there a complete reference?

    There is a Developer Guide for building Flatcar Container Linux from source and/or in modifying the OS. The SDK is not an API with function calls, but rather a kit of scripts and tools.

New user content​

New users are the most avid users of documentation, and need content specifically for them. We evaluate on the following:

  • Is “getting started” or similar clearly labeled?

    The "Learning Series" is a well-documented getting started guide. There is a Heading 3 heading "Getting started" in the overview.

  • Is installation documented step-by-step?

    Procedures are not formal step-by-steps, but rather sufficient descriptions of the purpose and result of running the provided code example.

    Even if the new user does not know anything about the technologies, the guidance still works but would be helped greatly with numbered steps.

  • If needed, are multiple OSes documented?

    Users are typically running a Linux machine, or a Virtual Machine running on Windows or macOS. It would be beneficial to acquaint the user with any specific client guidance, particularly when installing tools and images. For example, brew may not install all components one gets from directly installing binaries.

  • Do users know where to go after reading the getting started guide?

    The Learning Series section provides sufficient content to get Flatcar instances running. It would be good to have listings of next steps for the various deployment scenarios.

  • Is your new user content clearly signposted on your site’s homepage or at the top of your information architecture?

    Other than being a top level node in the TOC, no.

  • Is there sample code or other example content that can easily be copy-pasted?

    Yes, most pages have code samples, but currently the UI does not show code example blocks with copy buttons. The code is simply in a different font. There is a GitHub issue to fix this.

Content maintainability & site mechanics​

As a project scales, concerns like localized (translated) content and versioning become large maintenance burdens, particularly if you don’t plan for them.

We evaluate on the following:

  • Is your documentation searchable?

    Yes, the home page has a search bar where any results populate a dropdown for selection.

  • Are you planning for localization/internationalization?

    There are currently no plans for localization.

  • Do you have a clearly documented method for versioning your content?

Being an incubating project, the content is not versioned at this time.

Content creation processes​

Documentation is only as useful as it is accurate and well-maintained, and requires the same kind of review and approval processes as code.

We evaluate on the following:

  • Is there a clearly documented (ongoing) contribution process for documentation?

    Yes. There is a 'How to contribute' section with guidance for using the GitHub repository, formatting, and style.

  • Does your code release process account for documentation creation & updates?

    Not yet a formal process, but the team regularly updates content.

  • Who reviews and approves documentation pull requests?

    Maintainers delegate doc approval to experienced code contributors.

  • Does the website have a clear owner/maintainer?

    Yes, the GitHub repository has a Maintainers document.

Inclusive language​

Creating inclusive project communities is a key goal for all CNCF projects.

We evaluate on the following:

  • Are there any customer-facing utilities, endpoints, class names, or feature names that use non-recommended words as documented by the Inclusive Naming Initiative website?

The 175 hits were "master", "disable", "abort", and "man in the middle". Of those only "abort" would necessitate a fix on eight occurrences.

  • Does the project use language like "simple", "easy", etc.?

    Yes, there are about 53 hits of "easy" to replace. The word "simple" is used, but the context is a simpler piece of code or process rather than a task being simple.

Recommendations​

The Flatcar documentation is evolving into comprehensive and understandable guidance. Technical savvy readers are well suited to gather the information they need. New users can still find their path, but they need more context of the technologies and tools involved, why they're needed, and the environments they're used in.

Information architecture​

At a high level, we recommend organizing the Flatcar documentation into three segments:

  • The first segment is a Getting Started guide.
  • The second segment comprises several chapters or guides devoted to operations that span several categories organized by functionality.
  • The last segment contains supporting content such as the Reference and Contributor Documentation.

These recommendations are intended to be malleable; as always, the documentation should serve the needs of the project, and the maintainers and contributors are the best arbiters of what changes are appropriate to the project.

  • Create a Getting Started node with the following sections:

    • Create an "Roadmap" page for users that enables them to determine how they want to deploy containers with Flatcar. The starting points for the discussion are whether the user comes from an on-prem or cloud environment and many users will use both. This purpose of this page is to provide users with a procedure for installation.
      • Prerequisites regarding client operating systems including running VMs from macOS and Windows.
      • Prerequisites for the Quickstart.
      • Any guidance on installing binaries directly or using programs like brew.

  • For the middle functional nodes of the TOC, devise the categories of tools, technologies, products, and terms pertinent areas for Flatcar. While these could be under a broader umbrella, such as Operations, nine or headings in the navigation bar is not too much, and speeds up discovery. Given a list of about 60 tools, technologies, etc., Copilot devised the following categories:

    • Virtualization and Hypervisors
    • Cloud and Hosting Platforms
    • Bare Metal Booting and Installation
    • Security and Compliance
    • File systems
    • System Services and OS Internals
    • Provisioning Configuration and Automation
    • Container and Orchestration

  • Update the top Overview page to accommodate an introduction to the overall documentation to guide users into determining their happy path to accomplish their Flatcar objectives.

  • Add at least one architectural diagram that depicts a Flatcar container, clusters, capabilities or other ideas for diagrams.

  • Sections like "Virtual Machines" and "Cloud Providers" have subtopics where common tasks and concepts could be discussed in the overview, leaving the subtopics with simpler procedures and minimal repetition.

  • CNCF suggests that all CNCF doc maintainers run check with this tool that measures how well AI agents can read, navigate, and use a documentation site using this tool: https://afdocs.dev/

    One of the main ways to improve AI capabilities is by creating an index file to reside at the root of the repository that contains links to key sections, essentially a high level index. AI agents look for this file to navigate the site.

    Visit https://llmstxt.org to learn about the index file. Consult with the Flatcar team to determine key sections and check in the file.

Content maintainability and site mechanics​

  • Edit each procedural topic into a formal How-to topic with numbered steps. While the conversational flow is inviting, it inhibits scanning, predictability, and troubleshooting when needed to reference a particular code block. No need to go granular, but at least have then main steps numbered.

  • For all code blocks, clearly indicate where the code is to be run. Preceded each step with clauses such as the following:

    • On your local machine, use the following command to...
    • Inside the Flatcar instance, ...
    • SSH into the instance and then run ...

  • Reformat all code blocks to have a copy button. Having to manually select code can pick up unwanted characters and is also more difficult on mobile devices.

Inclusive language and tone​

  • Consider replacing the eight occurrences of "abort".
  • Remove occurrences of "easy" that could be construed as ableist or condescending; that is, where they imply that a task should be trivial for the user.

Contributor documentation​

Flatcar is an incubating project of CNCF. This means that the project should be developing professional-quality documentation alongside the project code.

CriterionRating (1-5)
Communication methods documented3 - meets standards
Beginner friendly issue backlog2 - needs improvement
“New contributor” getting started content2 - needs improvement
Project governance documentation4 - meets standards

Comments​

The following sections contain brief assessments of each element of the Contributor Documentation rubric.

Communication methods documented​

One of the easiest ways to attract new contributors is making sure they know how to reach you.

The Flatcar team casts a wide net for gathering feedback and contributions. Their effort spans documentation, code, community platforms, Flatcar apps, blogs, presentations, workshops, bug fixing events, developer challenges, and web design workshops.

The team provides links to aggregated GitHub issues for newcomers and advanced users to work on.

The team encourages users to create a Flatcar app, a reference implementation showing how to run a specific service on Flatcar, as a compelling way learn Flatcar and help the user community.

We evaluate on the following:

Beginner-friendly issue backlog​

We evaluate on the following:

  • Are docs issues well-triaged?

    Yes, this URL shows doc issues being tracked in GitHub: https://github.com/flatcar/Flatcar/issues?q=state%3Aopen%20label%3Akind%2Fdocs

  • Is it easy for new contributors to make contributions (i.e. a “good first issue” label)?

    The Contributing Guide, https://github.com/flatcar/Flatcar/blob/main/CONTRIBUTING.md, linked from the repository README file welcomes new contributors. The "Finding issues" subsection has these helpful links:

    • Good first issue - Displays pages of coding and documentation tasks marked good first issue in DevOps. These issues have guidance to help newcomers with the task.
    • Help wanted issues - suitable for non-core maintainers.

    Each documentation page has these two links at the bottom:

    • Edit this page - link opens the page for editing in GitHub if a fork exists, otherwise shows the option to fork the repository.
    • File documentation issue - link opens a new GitHub issue.

  • Are issues well-documented (i.e., more than just a title)?

    Yes, most issues are thoroughly described with detailed proposed solutions. See the Finding Issues section in the Contributing document in the repository.

  • Are issues maintained for staleness?

    Yes. Confident the team has freshness as a continuing objective.

New contributor getting started content​

Open source is complex and projects have many processes to manage that. Are processes easy to understand and written down so that new contributors can jump in easily?

We evaluate on the following:

  • Do you have a community repository or section on your website?

    Not currently.

  • Is there a document specifically for new contributors/your first contribution?

    Yes. There is a How to contribute section at the bottom of the navigation tree, and contains guidance on making pull requests in the Flatcar GitHub repository. Also included is guidance style and formatting with links to style guides.

    In addition to the website, there is guidance for contributors on the README page of the Flatcar GitHub repository, in Participate and Contribute section.

  • Do new users know where to get help?

    That would be Flatcar's Slack and Matrix chat channels.

    • Matrix (preferred): #flatcar:matrix.org
    • Slack: #flatcar (Kubernetes Slack)
    • GitHub Discussions: flatcar/Flatcar/discussions
    • Mailing List (Users): #flatcar-Linux-user

    These and other resources are listed in the Onboarding guide.

Project governance documentation​

One of the CNCF’s core project values is open governance.

We evaluate on the following:

  • Is project governance clearly documented?

    Yes, the Flatcar Project Governance document projects values of openness, fairness, community and participation. For an incubating project, Flatcar is laying the foundation for vigorous community participation.

Recommendations​

The Flatcar team has a robust Community outreach effort and seems to have good participation. All the criteria have been met for an incubating project. As the project evolves, the following recommendations should be considered:

  • Put a link to community participation portal content on the main Flatcar.org website.
  • Include direct links to documentation issues in the Finding Issues section of the Contributing guide.

Communication methods documented​

Information is well documented and discoverable from the repository's Contributing and Onboarding guides linked to from the README file.

Beginner-friendly issue backlog​

Suggest adding a link to documentation issues in the Contributing guide's Finding Issues section.

New contributor getting started content​

The team plans to expand the guidance in How to contribute section on the website to parallel the guidance in the repository's Contributing guide as appropriate.

Project governance documentation​

The Flatcar Project Governance document is comprehensive, and appears well-maintained and up-to-date.

Website and infrastructure​

The analysis for this section is not complete.

Flatcar is an incubating project of CNCF. This means that the project should be developing professional-quality documentation alongside the project code.

Criterion[Rating (1-5)]
Single-source for all files[rating (1-5)]
Meets min website req. (for maturity level)[rating (1-5)]
Usability, accessibility, and design[rating (1-5)]
Branding and design[rating (1-5)]
Case studies/social proof[rating (1-5)]
SEO, Analytics, and site-local search[rating (1-5)]
Maintenance planning[rating (1-5)]
A11y plan & implementation[rating (1-5)]
Mobile-first plan & implementation[rating (1-5)]
HTTPS access & HTTP redirect[rating (1-5)]
Google Analytics 4 for production only[rating (1-5)]
Indexing allowed for production server only[rating (1-5)]
Within site / local search[rating (1-5)]
Account custodians are documented[rating (1-5)]

Comments​

AUTHOR NOTE: make any overall comments about the Website and documentation infrastructure here.

The following sections contain brief assessments of each element of the Website and documentation infrastructure rubric.

AUTHOR NOTE: for each heading below, discuss how well the in-scope items meet these criteria. Keep in mind that much of the website infrastructure criteria depend on the tools (static site generator, website framework and hosting, analytics tools, etc.) and processes (project CI, release procedures, governance, etc.) used to produce the documentation. (Criteria are copied from criteria.md)

Single-source requirement​

Source files for all website pages should reside in a single repo. Among other problems, keeping source files in two places:

  • confuses contributors
  • requires you to keep two sources in sync
  • increases the likelihood of errors
  • makes it more complicated to generate the documentation from source files

Ideally, all website files should be in the website repo itself. Alternatively, files should be brought into the website repo via git submodules.

If a project chooses to keep source files in multiple repos, they need a clearly documented strategy for managing mirrored files and new contributions.

Minimal website requirements​

Listed here are the minimal website requirements for projects based on their maturity level, either incubating or graduated. (These are the only two levels for which a tech docs analysis can be requested.)

CriterionIncubating RequirementGraduated Requirement
[Website guidelines]All guidelines satisfiedAll guidelines satisfied
Docs analysis (this)Requested through CNCF service deskAll follow-up actions addressed
Project doc: stakeholdersRoles identified and doc needs documentedAll stakeholder need identified
Project doc: hostingHosted directlyAll Hosted directly
Project doc: user docsComprehensive, addressing most stakeholder needsFully addresses needs of key stakeholders

Usability, accessibility and devices​

Most CNCF websites are accessed from mobile and other non-desktop devices at least 10-20% of the time. Planning for this early in your website's design will be much less effort than retrofitting a desktop-first design.

  • Is the website usable from mobile?
  • Are doc pages readable?
  • Are all / most website features accessible from mobile -- such as the top-nav, site search and in-page table of contents?
  • Might a mobile-first design make sense for your project?

Plan for suitable accessibility measures for your website. For example:

  • Are color contrasts significant enough for color-impaired readers?
  • Are most website features usable using a keyboard only?
  • Does text-to-speech offer listeners a good experience?

It is up to each project to set their own guidelines.

Branding and design​

CNCF seeks to support enterprise-ready open source software. A key aspect of this is branding and marketing.

We evaluate on the following:

  • Is there an easily recognizable brand for the project (logo + color scheme) clearly identifiable?
  • Is the brand used across the website consistently?
  • The website’s typography clean and well-suited for reading?

Case studies/social proof​

One of the best ways to advertise an open source project is to show other organizations using it.

We evaluate on the following:

  • Are there case studies available for the project and are they documented on the website?
  • Are there user testimonials available?
  • Is there an active project blog?
  • Are there community talks for the project and are they present on the website?
  • Is there a logo wall of users/participating organizations?

SEO helps users find your project, and it's documentation, and analytics helps you monitor site traffic and diagnose issues like page 404s. Intra-site search, while optional, can offer your readers a site-focused search results.

We evaluate on the following:

  • Analytics:
    • Is analytics enabled for the production server?
    • Is analytics disabled for all other deploys?
    • If your project used Google Analytics, have you migrated to GA4?
    • Can Page-not-found (404) reports easily be generated from your site analytics? Provide a sample of the site's current top-10 404s.
  • Is site indexing supported for the production server, while disabled for website previews and builds for non-default branches?
  • Is local intra-site search available from the website?
  • Are the current custodian(s) of the following accounts clearly documented: analytics, Google Search Console, site-search (such as Google CSE or Algolia)

Maintenance planning​

Website maintenance is an important part of project success, especially when project maintainers aren’t web developers.

We evaluate on the following:

  • Is your website tooling well-supported by the community (i.e., Hugo with the Docsy theme) or commonly used by CNCF projects (our recommended tech stack?)
  • Are you actively cultivating website maintainers from within the community?
  • Are site build times reasonable?
  • Do site maintainers have adequate permissions?

Other​

  • Is your website accessible via HTTPS? Yes
  • Does HTTP access, if any, redirect to HTTPS? Yes

Recommendations​

AUTHOR NOTE: Write general recommendations based on the comments from the previous section.

Single-source requirement​

Minimal website requirements​

Usability, accessibility and devices​

Branding and design​

Case studies/social proof​

SEO, Analytics and site-local search​

Maintenance planning​

Other​

References and notes​

Rating values​

The numeric rating values used in this document are as follows

  1. Not present
  2. Needs improvement
  3. Meets standards
  4. Meets or exceeds standards
  5. Exemplary