# Docs Helpers

   * [Introduction](#introduction)
   * [Output Generated](#output-generated)
   * [Process for Admin Docs](#process-for-admin-docs)
   * [Working With Branches](#working-with-branches)
   * [Service-Dependent Envvars](#service-dependent-envvars)
      * [Generate Envvar Docs From Config Structs](#generate-envvar-docs-from-config-structs)
      * [Deprecation Process](#deprecation-process)
   * [Global Envvars](#global-envvars)
   * [Special Envvars](#special-envvars)
      * [Special Scope Envvars](#special-scope-envvars)
      * [Extended Envvars](#extended-envvars)
         * [Extract Extended Envvars](#extract-extended-envvars)
      * [Generate Extended Envvar Docs](#generate-extended-envvar-docs)
   * [Doc Tasks for New Releases](#doc-tasks-for-new-releases)
   * [Backporting](#backporting)
   * [Old Processes](#old-processes)

## Introduction

This readme file describes how envvars are managed internally for the documentation process.

`docs/helpers` contains a go program named `main.go` which creates docs by extracting information from the code using additional go programs.

In `./docs/helpers` run: `go run . --help`\
This will give you an overview of available commands.
  
Individual steps (programs) can be called manually if needed. Note that not all programs are called automatically on purpose, see the [Doc Tasks for New Releases](#doc-tasks-for-new-releases) below. `main.go` is used by `make docs-generate` (or `make -C docs docs-generate` when running manually from the repos root) which is triggered by the CI or can be called manually. It calls the other required programs and has these main responsibilities for automatic runs:

- Generate docs for envvars in config structs including deprecations if there are any.
- Extract and generate docs for `extended` envvars that are not mentioned in config structs (aka "rogue" envvars).
- Extract and generate docs for `global` envvars which occur in multiple services.
- Create `docs/service/<service-name>/_index.md` from `service/<service-name>/README.md` files while keeping the existing `_index.md` if the service README.md has not been created so far. Also see the important note at [docs README](../README.md).

Note that whenever you run `make docs-generate`, a file named `env_vars.yaml` will appear to have changed content. Usually, this change can safely be dropped and no merge is required. BUT, before a new production release ocis is generated, this file neds close attention, manual intervention and changes need to be merged. See the [developer documentation](https://owncloud.dev/general-info/envvars/new-release-process/) for more details.


## Output Generated

- The generated yaml files can be found at: `docs/services/_includes` when running locally respectively in the `docs branch` after the CI has finished.
- The generated adoc files can be found at: `docs/services/_includes/adoc` when running locally respectively in the `docs branch` after the CI has finished.
- The file name for global envvars is named: `global_configvars.adoc`.
- The file name for extended envvars is named: `extended_configvars.adoc`.
- A file named `docs/helpers/env_vars.yaml` containing envvar changes gets updated if changes have been identified.
- A file named `docs/helpers/extended_vars.yaml` containing changes for extended envvars gets updated if changes have been identified.\
Note, if changes appear, **this file needs manual treatment** before committing, see [Extended Envvars](#extended-envvars) below.

## Process for Admin Docs

Whenever a build is triggered from the [ocis Admin](https://github.com/owncloud/docs-ocis) docs or any other admin-related documentation, the files generated in this ocis repository are included in the build process and added in the manner defined by the admin documentation. The updated documentation will then appear on the public [admin documentation](https://doc.owncloud.com/ocis/next) page. The same is true for ocis production releases.

**Notes**
* When docs relevant data will be generated in ocis, they will be written into the `docs/servcies/...` folder structure, but they are in the master branch. When merging, an automated process will **move** them into the `docs` branch.\
**When running make commands locally, the relocation is not done and files reside on the generated location!**

* The above is the reason you should have an up-to-date docs branch before deriving a `docs-stable-x.y` branch from docs - though backports can be done.

* .adoc file generation
  * Service related adoc files are autogenerated and saved in `/docs/services/<service-name>` but will be _relocated_ by each merge into the docs branch into a subfolder named `services/_include/adoc/<service-name.adoc>`.
  * Release based envvar changes are saved during manual generation in `/docs/services/general-info/env-var-deltas/<filename>` but will be _copied_ on each merge into the doc branch into a subfolder named `services/_include/adoc/env-var-deltas/<filename>`.

* Exclude paths for Hugo
  * The paths defined above for .adoc files are excluded from parsing by Hugo. The source of what to exclude is defined in the `config.yaml` file located in [owncloud.github.io](https://github.com/owncloud/owncloud.github.io/). Any folder containg .adoc files must be added there else `make -C docs docs-serve` or CI will fail reporting an adoc related parsing error. (We could also add an asciidoc parser to Hugo as alternative...)

**Backgroud Information**\
The admin docs rely on the existence of the following branches in the ocis repository. Note that the admin docs define which ocis branch is accessed via the attributes in the `antora.yml` file in the respective admin docs branch.

* `docs`\
This reflects ocis master and is referenced from the admin docs from the master branch showing as `next` in the documentation.
* `docs-stable-x.y`\
This reflects a published ocis release and is referenced from the admin docs via the corresponding admin docs branch such as `7.2`.

Because of this, branching and parametrizing admin docs occurs **after** branching an ocis release with its necessary branches! If you branch admin docs before the required ocis branches are available, you must set the ocis source branch to import data from to `docs` and reconfigure afterwards. 

## Working With Branches

The following is valid for envvars and yaml files related to the doc process:

* When filing a pull request in the ocis master branch relating to docs, CI runs `make docs-generate` and copies the result into the `docs` branch of ocis. The `docs` branch is taken as base for the [developer documnetation](owncloud.dev) and as reference for the [admin docs](https://doc.owncloud.com/ocis/next/).
* When running `make docs-generate` _locally_, the same output is created as above but it stays in the same branch where the make command was issued.

In both cases, `make docs-generate` removes files in the target folder `_includes` to avoid remnants. All content is recreated.

On a side note (unrelated to the `docs` branch), [deployment examples](https://github.com/owncloud/ocis/tree/master/deployments/examples) have their own branch related to an ocis stable version to keep the state consistent, which is necessary for the admin documentation.

## Service-Dependent Envvars

For each service available, a file named like `<service name>_configvars.adoc` is created containing a:

* table on top defining deprecated envvars - if applicable
* table containing all envvars with their name, type, default value and description

The table with deprecations is always printed in the final adoc file even if there are none, but is rendered in the docs build process only if the `HasDeprecations` value is set. This value is automatically handed over via the .adoc file. The template file can be found at `docs/templates/ADOC.tmpl`.

### Generate Envvar Docs From Config Structs

Generates docs from a template file, mainly extracting `"env"` and `"desc"` tags from the config structs.

Templates can be found in `docs/helpers` folder. (Same as this `README`). Check the `.tmpl` files.

### Deprecation Process

For details on deprecation see the [deprecating-variables](https://owncloud.dev/services/general-info/envvars/deprecating-variables/) documentation in the dev docs.

## Global Envvars

True global envvars are identified by checking if they are available in more than one service. The table created is similar to the service-dependent envvar table but additionally contains a column with all service names where this envvar occurs. The output is rendered in list form where each item is clickable and automatically points to the corresponding service page. The template file can be found at `docs/templates/ADOC_global.tmpl`.

If global envvars does not appear in the list of globals, before checking if the code works, do a manual search in the `ocis/services` folder with `grep -rn OCIS_xxx` if the envvar in question appears at least twice. If the envvar only appears once, the helpers code works correct.

**IMPORTANT**\
A global envvar may not appear in the globals list though it is named `OCIS_xxx`. This can be the case if a developer decides for a global name but it is currently not used in more than one service!

## Special Envvars

### Special Scope Envvars

These envvars cant be managed automatically and need a full manualy process. See the [Special Scope Envvars](https://owncloud.dev/general-info/envvars/special-envvars/) documentation in the dev docs.

### Extended Envvars

For a description and their handling see the [Extended Envvars](https://owncloud.dev/general-info/envvars/special-envvars/) documentation in the dev docs.

#### Extract Extended Envvars

The `grep` command parses the code, looking for `os.Getenv` and passes these contents to a yaml file along with the following information:
```golang
// Variable contains all information about one rogue envvar
type Variable struct {
	// These field structs are automatically filled:
	// RawName can be the name of the envvar or the name of its var
	RawName string `yaml:"rawname"`
	// Path to the envvar with linenumber
	Path string `yaml:"path"`
	// FoundInCode indicates if the variable is still found in the codebase.
	FoundInCode bool `yaml:"foundincode"`
	// Name is equal to RawName but will not be overwritten in consecutive runs
	Name string `yaml:"name"`

	// These field structs need manual filling:
	// Type of the envvar
	Type string `yaml:"type"`
	// DefaultValue of the envvar
	DefaultValue string `yaml:"default_value"`
	// Description of what this envvar does
	Description string `yaml:"description"`
	// Do not export this envvar into the generated adoc table
	Ignore bool `yaml:"do_ignore"`

	// For simplicity ignored for now:
	// DependendServices []Service `yaml:"dependend_services"`
}
```

This yaml file **can and must be edited manually** to add descriptions, default values, etc.

**IMPORTANT**: Some value are automatically filled by the program and mujst not be changed. See the developer documentation for more details.

### Generate Extended Envvar Docs

The process further picks up the `yaml` file generated in the `Extract Rogue Envvars` step and renders it to an adoc file (a table is created) using a go template. The template file for this step can be found at `docs/templates/ADOC_extended.tmpl`.

## Doc Tasks for New Releases

For details see the [Release Process for Envvars](https://owncloud.dev/general-info/envvars/new-release-process/) documentation in the dev docs.

## Backporting

The ocis repo contains branches which are necessary for the documentation. The `docs` branch is related to changes in master, necessary for owncloud.dev and the admin docs referencing master content when it comes to envvars and yaml files.

Cases for a backport can be a typo in an envvar description you want to have fixed in a stable branch too or a file  was created after the stable branch was set up but needs to be available in that branch.

When a new stable ocis release (branch) is published, like `stable-7.2`, an additional branch (including CI) is set up manually by the dev team for referencing docs content like `docs-stable-7.2` - related to envvars and yaml files only - and added to the CI.

In case it is necessary to transport a change from master to a stable branch like `docs-stable-7.2`, you must backport the original changes that will create that file to the `stable-7.2` branch. The CI will then take care of creating the results in the target `docs-stable-7.2`.

If the change is expected to have a bigger impact on documenation, you can locally run `make -C docs docs-generate` in the respective branch containing the changes or independently in the `stable-x.y` branch after merging to see if there are additional actions necessary and changed files may need to get checked in.

## Old Processes

The generation of delta files for markdown could also be done using a go helper. This should not be done anymore because:

1. It needs that the production tag is already created for the new release and contains all data.
2. The outcome has a different look and feel compared the new process.

For the sake of documentation because the code still extists, here is the description:

* Because `env_vars.yaml` has been cleaned up as part of the update tasks, we can rely on its actuality for the branches to be compared.
  * Create delta files for added, removed and deprecated envvars. To do so type:\
    `go run . env-var-delta-table` and use as parameter the versions you want to compare. Example: `v7.1.0 v7.2.0`.
  * List and check the files created in `./docs/helpers/output/env-deltas/`. The markdown files created contain a table with dev relevant data. Any other files created are not relevant and can safely be deleted.
  * Create a branch and move the markdown files from `./docs/helpers/output/env-deltas/` to `./docs/services/general-info/env-var-deltas/`. The markdown files will be consumed by dev docs from this location.

* Commit all changes, create a PR and merge. Dev docs is now up-to-date.    
