feat: Config reference documentation autogeneration #2033

ilyakuz-db · 2024-12-18T18:24:19Z

Changes

Documentation autogeneration tool. This tool uses same annotations_*.yml files as in json-schema

Result will go there and there

Tests

Manually

eng-dev-ecosystem-bot · 2024-12-18T20:50:54Z

Test Details: go/deco-tests/12401106836

…erence-doc-autogen

github-actions · 2025-01-03T12:53:29Z

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

PR number: 2033
Commit SHA: c6703c13637c0d85e6754e4c802f38052304a5d6

Checks will be approved automatically on success.

…erence-doc-autogen

ilyakuz-db · 2025-01-13T16:12:49Z

bundle/internal/docs/markdown.go

+	md "github.com/nao1215/markdown"
+)
+
+const (


These templates are in this repo temporary to simplify the process of docs updating

This can be further improved by using include declarations in doc repo so we could generate only meaningful part here and store the wrapper file in docs repo

ilyakuz-db · 2025-01-13T16:18:55Z

bundle/internal/schema/annotations.go

 type annotationHandler struct {
 	// Annotations read from all annotation files including all overrides
-	parsedAnnotations annotationFile
+	parsedAnnotations annotation.File


Moved annotations to separate package but yaml files kept as is to avoid messy conflicts. Will move them right before merging

Can also be done in a follow-up PR.

…erence-doc-autogen

ilyakuz-db · 2025-01-14T12:10:20Z

@pietern @shreyas-goenka @andrewnester this one is ready for review if anyone want to take a look!

More content updates to annotations will be introduced in follow-up PRs

pietern

Did you consider checking in the Markdown output as well?

That can serve as both a check that the contents is OK and that it doesn't inadvertently drift when changes to the schema or annotations are made.

pietern · 2025-01-15T10:47:39Z

bundle/internal/annotation/main.go

main.go is typically used for a func main() entry point.

Please rename/split this into file.go, types.go, etc, to be more intuitive.

pietern · 2025-01-15T10:49:30Z

bundle/internal/annotation/main.go

+ * github.com/databricks/cli/bundle/config.Bundle:
+ *  	cluster_id:
+ *      description: "Description"
+ */


I know it is copied, but we use regular line comments also for blocks everywhere else (//).

pietern · 2025-01-15T10:56:20Z

bundle/internal/docs/README.md

+
+1. Install [Golang](https://go.dev/doc/install)
+2. Run `go mod download` from the repo root
+3. Run `make docs` from the repo


The vendor target can be a dep of make docs, which removes a step here.

pietern · 2025-01-15T11:02:51Z

bundle/internal/docs/markdown.go

+    * - [volume](#volume) (<UC>)
+      - ✓
+      - [Volume object](/api/workspace/volumes/create)
+`


If this is a literal, you can include it under a templates directory and read it directly.

That's easier than working around backticks and quoting issues inline.

Separate question, why is the list of resources not generated?

Good suggestion!

Separate question, why is the list of resources not generated?

There are some custom columns in the table, I decided to keep as is to give some flexibility for intro

pietern · 2025-01-15T11:04:22Z

bundle/internal/docs/markdown.go

+	for _, node := range nodes {
+		m = m.LF()
+		if node.TopLevel {
+			m = m.H2(node.Title)


What's the value of this dependency if you can also write output += "## " + node.Title?

It was useful for more complex things as tables but then I realised that we use some custom markup

I'll try to remove this dep

Removed the dependency, added smaller custom renderer with the same interface

pietern · 2025-01-15T11:06:57Z

bundle/internal/docs/nodes.go

+	return isCustomField
+}
+
+func extractNodes(prefix string, props map[string]*jsonschema.Schema, refs map[string]jsonschema.Schema, customFields map[string]bool) []rootProp {


Same as above.

pietern · 2025-01-15T11:07:09Z

bundle/internal/docs/refs.go

+	return true
+}
+
+func resolveAdditionaProperties(v *jsonschema.Schema) *jsonschema.Schema {


pietern · 2025-01-15T11:07:38Z

bundle/internal/schema/annotations.go

 type annotationHandler struct {
 	// Annotations read from all annotation files including all overrides
-	parsedAnnotations annotationFile
+	parsedAnnotations annotation.File


Can also be done in a follow-up PR.

pietern · 2025-01-15T11:08:59Z

go.mod

@@ -34,6 +34,8 @@ require (
 	gopkg.in/yaml.v3 v3.0.1
 )

+require github.com/nao1215/markdown v0.6.0


Deps should have an entry in the NOTICE file + vetted to be compatible with the license.

Are you sure you want to use it?

Removed extra deps to avoid that

pietern · 2025-01-15T11:12:54Z

bundle/internal/docs/nodes.go

+		return nodes[i].Title < nodes[j].Title
+	})
+	return nodes
+}


It's very hard to tell what this function does and how it is supposed to work. Please include a few more comments with how you expect it to work, and if possible even a testcase or two to make sure it does work the way you assume. As is we need to run it and check the output to confirm this.

Added few test cases and re-arranged the code with more comments to highlight important parts

ilyakuz-db · 2025-01-15T15:22:54Z

Did you consider checking in the Markdown output as well?

That can serve as both a check that the contents is OK and that it doesn't inadvertently drift when changes to the schema or annotations are made.

You mean to remove from .gitignore, right?

denik · 2025-01-16T08:56:05Z

Makefile

 schema:
 	go run ./bundle/internal/schema ./bundle/internal/schema ./bundle/schema/jsonschema.json

+docs: vendor


docs: vendor

We don't add this implicitly anymore to other targets. If you're on CI, you can always specify both with "make vendor docs".

BTW, do you intended to run it on CI?

BTW, do you intended to run it on CI?

Currently the only use case of using that tool is to checkout CLI repo and run locally

We don't add this implicitly anymore to other targets

By implicitly you mean using make deps or in general?

I mean the make entry should be:

docs: ...

no dependency on vendor

denik · 2025-01-16T08:59:23Z

Makefile

 schema:
 	go run ./bundle/internal/schema ./bundle/internal/schema ./bundle/schema/jsonschema.json

+docs: vendor
+	go run ./bundle/internal/docs ./bundle/internal/schema ./bundle/internal/docs


Why is this part if /bundle/internal/ ?

internal is meant for code that is not meant to be used outside of the package, but this is clearly intended to be used outside the package, that's why it's in Makefile.

How about placing this at top level? /docsgen/ ?

I see you're following existing structure, so it's a more of a question for @pietern

I see you're following existing structure,

Yes I reused same approach as with json schema generation which is used in .codegen.json along with other internal scripts

ilyakuz-db added 2 commits December 18, 2024 17:07

tmp: Documentation autogeneration script

8c58d48

fix: Custom table markup changes

e27cc9a

ilyakuz-db temporarily deployed to test-trigger-is December 18, 2024 18:24 — with GitHub Actions Inactive

ilyakuz-db added 2 commits December 18, 2024 19:59

fix: Styling issues

fe4c6b8

fix: Remove openAPI docs

cfa2be3

ilyakuz-db temporarily deployed to test-trigger-is December 18, 2024 19:53 — with GitHub Actions Inactive

ilyakuz-db temporarily deployed to test-trigger-is December 18, 2024 19:54 — with GitHub Actions Inactive

ilyakuz-db added 2 commits December 18, 2024 21:12

fix: Remove extra headings

10e0d27

fix: Small readme

64fa2bf

ilyakuz-db temporarily deployed to test-trigger-is December 18, 2024 20:50 — with GitHub Actions Inactive

ilyakuz-db added 5 commits December 23, 2024 12:25

Merge branch 'main' of github.com:databricks/cli into feat/config-ref…

40c4b3a

…erence-doc-autogen

feat: Add examples to the docs

820dd5f

chore: Extract annotation package

460a455

feat: More explicit type for arrays

541c3e3

feat: Support for resources

2aadfcb

ilyakuz-db requested review from pietern, andrewnester, shreyas-goenka and denik as code owners January 2, 2025 17:24

fix: Updated styles

c6703c1

ilyakuz-db added 6 commits January 7, 2025 16:52

fix: Styling

fe6ba76

fix: Description of root types with additional properties

c355fbf

docs: Add override for volume spec

f9278c2

fix: Missing array types

6c5268a

fix: Sync annotations

bad77bd

Merge branch 'main' of github.com:databricks/cli into feat/config-ref…

954ef76

…erence-doc-autogen

ilyakuz-db temporarily deployed to test-trigger-is January 10, 2025 19:36 — with GitHub Actions Inactive

ilyakuz-db added 2 commits January 10, 2025 21:01

fix: Add field-name for some properties

5fb4fa0

fix: Few links updates

ad81e1f

ilyakuz-db temporarily deployed to test-trigger-is January 13, 2025 14:17 — with GitHub Actions Inactive

ilyakuz-db added 2 commits January 13, 2025 15:27

fix: Schema update

c88498e

fix: Allow empty fields

3e0d232

ilyakuz-db temporarily deployed to test-trigger-is January 13, 2025 14:28 — with GitHub Actions Inactive

fix: Circular types

2591172

ilyakuz-db temporarily deployed to test-trigger-is January 13, 2025 15:00 — with GitHub Actions Inactive

ilyakuz-db changed the title ~~[WIP] feat: Config reference documentation autogeneration~~ feat: Config reference documentation autogeneration Jan 13, 2025

ilyakuz-db commented Jan 13, 2025

View reviewed changes

ilyakuz-db added 2 commits January 14, 2025 11:52

Merge branch 'main' of github.com:databricks/cli into feat/config-ref…

f684afe

…erence-doc-autogen

fix: Apps fixes after merge

1e41e61

ilyakuz-db temporarily deployed to test-trigger-is January 14, 2025 12:00 — with GitHub Actions Inactive

pietern reviewed Jan 15, 2025

View reviewed changes

fix: Add vendor step to make docs

14b2c86

ilyakuz-db added 9 commits January 15, 2025 16:23

fix: Stage output changes

c74ab75

fix: Split annotations/main.go in 2 files

1b73105

fix: Move templates to separate files

e5497f7

fix: Typo

33bd3b8

fix: Remove extra dependency

b91bb4d

fix: More documentation for internal types

14721de

fix: Remove type inconsistency in maps

26466d2

fix: Rename fields to make it more clear

d27282c

fix: More ducmentation + tests

13181fc

denik reviewed Jan 16, 2025

View reviewed changes

ilyakuz-db requested a review from pietern January 16, 2025 11:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Config reference documentation autogeneration #2033

feat: Config reference documentation autogeneration #2033

ilyakuz-db commented Dec 18, 2024 •

edited

Loading

eng-dev-ecosystem-bot commented Dec 18, 2024

github-actions bot commented Jan 3, 2025

ilyakuz-db Jan 13, 2025

ilyakuz-db Jan 13, 2025

pietern Jan 15, 2025

ilyakuz-db commented Jan 14, 2025

pietern left a comment

pietern Jan 15, 2025

pietern Jan 15, 2025

pietern Jan 15, 2025

pietern Jan 15, 2025

ilyakuz-db Jan 15, 2025

pietern Jan 15, 2025

ilyakuz-db Jan 15, 2025

ilyakuz-db Jan 16, 2025

pietern Jan 15, 2025

pietern Jan 15, 2025

pietern Jan 15, 2025

pietern Jan 15, 2025

ilyakuz-db Jan 16, 2025

pietern Jan 15, 2025

ilyakuz-db Jan 16, 2025

ilyakuz-db commented Jan 15, 2025 •

edited

Loading

denik Jan 16, 2025

ilyakuz-db Jan 16, 2025

denik Jan 16, 2025

denik Jan 16, 2025

ilyakuz-db Jan 16, 2025

feat: Config reference documentation autogeneration #2033

Are you sure you want to change the base?

feat: Config reference documentation autogeneration #2033

Conversation

ilyakuz-db commented Dec 18, 2024 • edited Loading

Changes

Tests

eng-dev-ecosystem-bot commented Dec 18, 2024

github-actions bot commented Jan 3, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilyakuz-db commented Jan 14, 2025

pietern left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilyakuz-db commented Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilyakuz-db commented Dec 18, 2024 •

edited

Loading

ilyakuz-db commented Jan 15, 2025 •

edited

Loading