ocis/docs/helpers/README.md

Docs Helpers

• Introduction
• Output Generated
• Process for Admin Docs
• Working With Branches
• Service-Dependent Envvars
  • Generate Envvar Docs From Config Structs
  • Deprecation Process
• Global Envvars
• Special Envvars
  • Special Scope Envvars
  • Extended Envvars
    • Extract Extended Envvars
  • Generate Extended Envvar Docs
• Doc Tasks for New Releases
• Backporting
• Old Processes

Introduction

This readme file describes how envvars are managed internally for the documentation process.

docs/helpers contains a go program named main.go which creates docs by extracting information from the code using additional go programs.

In ./docs/helpers run: go run . --help
This will give you an overview of available commands.

Individual steps (programs) can be called manually if needed. Note that not all programs are called automatically on purpose, see the Doc Tasks for New Releases below. main.go is used by make docs-generate (or make -C docs docs-generate when running manually from the repos root) which is triggered by the CI or can be called manually. It calls the other required programs and has these main responsibilities for automatic runs:

• Generate docs for envvars in config structs including deprecations if there are any.
• Extract and generate docs for extended envvars that are not mentioned in config structs (aka "rogue" envvars).
• Extract and generate docs for global envvars which occur in multiple services.
• Create docs/service/<service-name>/_index.md from service/<service-name>/README.md files while keeping the existing _index.md if the service README.md has not been created so far. Also see the important note at docs README.

Note that whenever you run make docs-generate, a file named env_vars.yaml will appear to have changed content. Usually, this change can safely be dropped and no merge is required. BUT, before a new production release ocis is generated, this file neds close attention, manual intervention and changes need to be merged. See the developer documentation for more details.

Output Generated

• The generated yaml files can be found at: docs/services/_includes when running locally respectively in the docs branch after the CI has finished.
• The generated adoc files can be found at: docs/services/_includes/adoc when running locally respectively in the docs branch after the CI has finished.
• The file name for global envvars is named: global_configvars.adoc.
• The file name for extended envvars is named: extended_configvars.adoc.
• A file named docs/helpers/env_vars.yaml containing envvar changes gets updated if changes have been identified.
• A file named docs/helpers/extended_vars.yaml containing changes for extended envvars gets updated if changes have been identified.
  Note, if changes appear, this file needs manual treatment before committing, see Extended Envvars below.

Process for Admin Docs

Whenever a build is triggered from the ocis Admin docs or any other admin-related documentation, the files generated in this ocis repository are included in the build process and added in the manner defined by the admin documentation. The updated documentation will then appear on the public admin documentation page. The same is true for ocis production releases.

Notes

• When docs relevant data will be generated in ocis, they will be written into the docs/servcies/... folder structure, but they are in the master branch. When merging, an automated process will move them into the docs branch.
  When running make commands locally, the relocation is not done and files reside on the generated location!

• The above is the reason you should have an up-to-date docs branch before deriving a docs-stable-x.y branch from docs - though backports can be done.

• .adoc file generation

  • Service related adoc files are autogenerated and saved in /docs/services/<service-name> but will be relocated by each merge into the docs branch into a subfolder named services/_include/adoc/<service-name.adoc>.
  • Release based envvar changes are saved during manual generation in /docs/services/general-info/env-var-deltas/<filename> but will be copied on each merge into the doc branch into a subfolder named services/_include/adoc/env-var-deltas/<filename>.

• Exclude paths for Hugo

  • The paths defined above for .adoc files are excluded from parsing by Hugo. The source of what to exclude is defined in the config.yaml file located in owncloud.github.io. Any folder containg .adoc files must be added there else make -C docs docs-serve or CI will fail reporting an adoc related parsing error. (We could also add an asciidoc parser to Hugo as alternative...)

Backgroud Information
The admin docs rely on the existence of the following branches in the ocis repository. Note that the admin docs define which ocis branch is accessed via the attributes in the antora.yml file in the respective admin docs branch.

• docs
  This reflects ocis master and is referenced from the admin docs from the master branch showing as next in the documentation.
• docs-stable-x.y
  This reflects a published ocis release and is referenced from the admin docs via the corresponding admin docs branch such as 7.2.

Because of this, branching and parametrizing admin docs occurs after branching an ocis release with its necessary branches! If you branch admin docs before the required ocis branches are available, you must set the ocis source branch to import data from to docs and reconfigure afterwards.

Working With Branches

The following is valid for envvars and yaml files related to the doc process:

• When filing a pull request in the ocis master branch relating to docs, CI runs make docs-generate and copies the result into the docs branch of ocis. The docs branch is taken as base for the developer documnetation and as reference for the admin docs.
• When running make docs-generate locally, the same output is created as above but it stays in the same branch where the make command was issued.

In both cases, make docs-generate removes files in the target folder _includes to avoid remnants. All content is recreated.

On a side note (unrelated to the docs branch), deployment examples have their own branch related to an ocis stable version to keep the state consistent, which is necessary for the admin documentation.

Service-Dependent Envvars

For each service available, a file named like <service name>_configvars.adoc is created containing a:

• table on top defining deprecated envvars - if applicable
• table containing all envvars with their name, type, default value and description

The table with deprecations is always printed in the final adoc file even if there are none, but is rendered in the docs build process only if the HasDeprecations value is set. This value is automatically handed over via the .adoc file. The template file can be found at docs/templates/ADOC.tmpl.

Generate Envvar Docs From Config Structs

Generates docs from a template file, mainly extracting "env" and "desc" tags from the config structs.

Templates can be found in docs/helpers folder. (Same as this README). Check the .tmpl files.

Deprecation Process

For details on deprecation see the deprecating-variables documentation in the dev docs.

Global Envvars

True global envvars are identified by checking if they are available in more than one service. The table created is similar to the service-dependent envvar table but additionally contains a column with all service names where this envvar occurs. The output is rendered in list form where each item is clickable and automatically points to the corresponding service page. The template file can be found at docs/templates/ADOC_global.tmpl.

If global envvars does not appear in the list of globals, before checking if the code works, do a manual search in the ocis/services folder with grep -rn OCIS_xxx if the envvar in question appears at least twice. If the envvar only appears once, the helpers code works correct.

IMPORTANT
A global envvar may not appear in the globals list though it is named OCIS_xxx. This can be the case if a developer decides for a global name but it is currently not used in more than one service!

Special Envvars

Special Scope Envvars

These envvars cant be managed automatically and need a full manualy process. See the Special Scope Envvars documentation in the dev docs.

Extended Envvars

For a description and their handling see the Extended Envvars documentation in the dev docs.

Extract Extended Envvars

The grep command parses the code, looking for os.Getenv and passes these contents to a yaml file along with the following information:

// Variable contains all information about one rogue envvar
type Variable struct {
	// These field structs are automatically filled:
	// RawName can be the name of the envvar or the name of its var
	RawName string `yaml:"rawname"`
	// Path to the envvar with linenumber
	Path string `yaml:"path"`
	// FoundInCode indicates if the variable is still found in the codebase.
	FoundInCode bool `yaml:"foundincode"`
	// Name is equal to RawName but will not be overwritten in consecutive runs
	Name string `yaml:"name"`

	// These field structs need manual filling:
	// Type of the envvar
	Type string `yaml:"type"`
	// DefaultValue of the envvar
	DefaultValue string `yaml:"default_value"`
	// Description of what this envvar does
	Description string `yaml:"description"`
	// Do not export this envvar into the generated adoc table
	Ignore bool `yaml:"do_ignore"`

	// For simplicity ignored for now:
	// DependendServices []Service `yaml:"dependend_services"`
}

This yaml file can and must be edited manually to add descriptions, default values, etc.

IMPORTANT: Some value are automatically filled by the program and mujst not be changed. See the developer documentation for more details.

Generate Extended Envvar Docs

The process further picks up the yaml file generated in the Extract Rogue Envvars step and renders it to an adoc file (a table is created) using a go template. The template file for this step can be found at docs/templates/ADOC_extended.tmpl.

Doc Tasks for New Releases

For details see the Release Process for Envvars documentation in the dev docs.

Backporting

The ocis repo contains branches which are necessary for the documentation. The docs branch is related to changes in master, necessary for owncloud.dev and the admin docs referencing master content when it comes to envvars and yaml files.

Cases for a backport can be a typo in an envvar description you want to have fixed in a stable branch too or a file was created after the stable branch was set up but needs to be available in that branch.

When a new stable ocis release (branch) is published, like stable-7.2, an additional branch (including CI) is set up manually by the dev team for referencing docs content like docs-stable-7.2 - related to envvars and yaml files only - and added to the CI.

In case it is necessary to transport a change from master to a stable branch like docs-stable-7.2, you must backport the original changes that will create that file to the stable-7.2 branch. The CI will then take care of creating the results in the target docs-stable-7.2.

If the change is expected to have a bigger impact on documenation, you can locally run make -C docs docs-generate in the respective branch containing the changes or independently in the stable-x.y branch after merging to see if there are additional actions necessary and changed files may need to get checked in.

Old Processes

The generation of delta files for markdown could also be done using a go helper. This should not be done anymore because:

1. It needs that the production tag is already created for the new release and contains all data.
2. The outcome has a different look and feel compared the new process.

For the sake of documentation because the code still extists, here is the description:

• Because env_vars.yaml has been cleaned up as part of the update tasks, we can rely on its actuality for the branches to be compared.

  • Create delta files for added, removed and deprecated envvars. To do so type:
    go run . env-var-delta-table and use as parameter the versions you want to compare. Example: v7.1.0 v7.2.0.
  • List and check the files created in ./docs/helpers/output/env-deltas/. The markdown files created contain a table with dev relevant data. Any other files created are not relevant and can safely be deleted.
  • Create a branch and move the markdown files from ./docs/helpers/output/env-deltas/ to ./docs/services/general-info/env-var-deltas/. The markdown files will be consumed by dev docs from this location.

• Commit all changes, create a PR and merge. Dev docs is now up-to-date.