[TT-13365] Create json schema script for Bento config validation

[buraksezer]

User description

PR for https://tyktech.atlassian.net/browse/TT-13365

generate_bento_config_schema.go generates a JSON schema for the Input/Output resources we support.

Simply,

go run generate_bento_config.go

It'll generate a bento-config-schema.json file in the current working folder. You can also set an output path via -output-path <string> parameter.

How to add a new Input/Output resource

1- Import the related component for its side effects, for example if you want to produce a JSON schema that supports redis component, you can import it like the following:

_ "github.com/warpstreamlabs/bento/public/components/redis"

2- Add its name to supportedItems slice. You should know that some components exposes different input/output sources For example, components/kafka exposes kafka and kafka_franz. You need to dig into the Bento's codebase to understand which input/output is exposed by a component.

Importing a small number of components was preferred instead of importing components/all because importing all components results in a huge definitions/processor object and there is no way to know which processor are used by the components we support.

This tool assumes that we support the exact same components in both input and output sections.

---

PR Type

enhancement

---

Description

• Introduced a new Go script generate_bento_config_schema.go to generate a JSON schema for Bento configuration validation.
• The script parses and manipulates JSON data to create a schema file, with error handling for file operations and JSON parsing.
• Instructions are included for adding new Input/Output resources by importing components and updating the supportedItems slice.
• The script outputs a bento-config-schema.json file, with an option to specify a custom output path.

---

Changes walkthrough 📝

Relevant files

Enhancement

generate_bento_config_schema.go Add Go script for generating Bento config JSON schema        apidef/streams/bento/schema/generate_bento_config_schema.go Added a new Go script to generate JSON schema for Bento config validation. Implemented functions to parse and manipulate JSON schema data. Included error handling for file operations and JSON parsing. Provided instructions for adding new Input/Output resources. +269/-0

---

💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

[buger]

💔 The detected issue is not in one of the allowed statuses 💔

Detected Status	Open	❌
Allowed Statuses	In Dev,In Code Review,Ready for Testing,In Test,In Progress,In Review	✔️

Please ensure your jira story is in one of the allowed statuses

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Error Handling The script uses a custom error handling function that exits the program immediately. This approach might not be suitable for all use cases, especially where more graceful error handling and logging are required. Hardcoded Values There are multiple hardcoded values for JSON keys and file paths which could be externalized to configuration files or environment variables for better flexibility and maintainability. Code Duplication The function printErrorAndExit is repeatedly called in various parts of the code. Consider refactoring to use more centralized error handling to reduce redundancy and improve maintainability.

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Score
Possible bug	Add error handling after jsonparser.ArrayEach to catch and return errors Handle potential errors from jsonparser.ArrayEach in insertDefinitionKind function to ensure robust error handling. apidef/streams/bento/schema/generate_bento_config_schema.go [126] _, err := jsonparser.ArrayEach(anyOfItems, func(value []byte, dataType jsonparser.ValueType, offset int, err error) { ... }) +if err != nil { + return err +} Suggestion importance[1-10]: 8 Why: Proper error handling after jsonparser.ArrayEach is crucial to catch and handle any errors that occur during the iteration. This prevents errors from being silently ignored, improving the robustness of the error handling.	8
Enhancement	Improve error message formatting in error handling Replace the use of fmt.Fprint with fmt.Fprintf to properly format the error message before sending it to os.Stderr. apidef/streams/bento/schema/generate_bento_config_schema.go [45] -_, _ = fmt.Fprint(os.Stderr, err) +_, _ = fmt.Fprintf(os.Stderr, "%v\n", err) Suggestion importance[1-10]: 7 Why: Using fmt.Fprintf allows for better formatting of the error message, making it clearer and more consistent. This change enhances the readability and maintainability of error logging.	7
Performance	Optimize string comparison by converting key to string once outside the loop Consider using a more efficient method for string comparison instead of string(key) == property inside loops to enhance performance. apidef/streams/bento/schema/generate_bento_config_schema.go [73-81] +keyStr := string(key) for _, property := range properties { - if string(key) == property { + if keyStr == property { ... } } Suggestion importance[1-10]: 6 Why: Converting the key to a string once outside the loop reduces the number of conversions needed, which can enhance performance, especially when dealing with large data sets.	6
	Enhance file writing performance by using a buffered writer Use a buffered writer in saveFile function to improve file writing performance. apidef/streams/bento/schema/generate_bento_config_schema.go [193] -_, err = file.Write(buf.Bytes()) +writer := bufio.NewWriter(file) +_, err = writer.Write(buf.Bytes()) +if err == nil { + err = writer.Flush() +} Suggestion importance[1-10]: 6 Why: Using a buffered writer can improve the performance of file writing operations by reducing the number of system calls, which is beneficial for writing large files or frequent write operations.	6

[github-actions]

API Changes

--- prev.txt	2024-11-05 12:21:25.142776900 +0000
+++ current.txt	2024-11-05 12:21:18.662744120 +0000
@@ -5519,6 +5519,9 @@
 
 type ValidatorKind string
 
+# Package: ./apidef/streams/bento/schema
+
+
 # Package: ./certs
 
 package certs // import "github.com/TykTechnologies/tyk/certs"

[buger]

💔 The detected issue is not in one of the allowed statuses 💔

Detected Status	Open	❌
Allowed Statuses	In Dev,In Code Review,Ready for Testing,In Test,In Progress,In Review	✔️

Please ensure your jira story is in one of the allowed statuses

[sonarcloud]

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint