diff --git a/README.rst b/README.rst index f19801fc..aaadf023 100644 --- a/README.rst +++ b/README.rst @@ -297,34 +297,32 @@ This example generates a `uint8_t` property value. A different type may be speci Calculated Values ----------------- -With ConfigDB, if an attribute name is prefixed with ``@`` then it will be evaluated as a simple expression. See https://github.com/SmingHub/Sming/blob/develop/Tools/Python/evaluator/README.md for details. - -Such expressions are parsed during loading of a schema. During the build process, copies are written to ``out/{SOC}/{build}/ConfigDB/schema`` to assist with debugging and development. - -.. note:: - - Variable names correspond to environment variables. - The build system is not aware of variable dependencies, so it may be necessary to perform a manual `clean` or `configdb-rebuild` to pick up any changed values. +Within a ConfigDB schema, if an attribute name is prefixed with @ then the attribute value will be evaluated and the result used as the actual value. +The expression must be given as a string, with variable names corresponding to environment variables. +See :doc:`/_inc/Tools/Python/evaluator/README` for details. +The ``.cfgdb`` schema are pre-processed on every build and the source files regenerated automatically if there is a change. +The pre-processed schema can be found in ``out/ConfigDB/schema/``, together with a *summary.txt* file. An example is included in the test application: .. code-block:: json - "properties": { + "simple-string": { + "type": "string", "@default": "SIMPLE_STRING" } -During loading, the attribute value is evaluated in python and the result stored in `default`. The value `SIMPLE_STRING` must be available in the environment - an error occurs if not found. +The pre-processed schema will contain a ``default`` attribute with the contents of the environment variable ``SIMPLE_STRING``. +An error will be given if named variable is not present in the environment. To test this, build and run as follows: .. code-block:: bash - make clean SIMPLE_STRING="donkey2" make -j run -The test application now fails as the value has changed - "donkey" is expected. +The test application now fails as the schema default value has changed - "donkey" is expected. .. note:: @@ -338,21 +336,28 @@ JSON does not support extended number formats, such as `0x12`, so this mechanism .. code-block:: json - "properties": { + "simple-int": { + "type": "integer", "@default": "8 + 27", "minimum": 0, "@maximum": "0xffff" } -Array defaults -~~~~~~~~~~~~~~ +Array values +~~~~~~~~~~~~ + +If a calculated attribute value is an array, then each element is evaluated separately. +Arrays may contain a mixture of types, but only string values will be evalulated: others will be passed through unchanged. -Array defaults may contain a mixture of types: .. code-block:: json - "properties": { + "simple-array": { + "type": "array", + "items": { + "type": "integer" + }, "@default": [ "0x12", 5, @@ -361,7 +366,34 @@ Array defaults may contain a mixture of types: ] } -Only string values will be evalulated, the others will be passed through unchanged. + +Dictionary values +~~~~~~~~~~~~~~~~~ + +To conditionally select from one of a number of options, provide a dictionary as the calculated attribute value. +The first key which evaluates as *True* is matched, and the corresponding value becomes the value for the property. +If none of the entries matches, an error is raised. + +In this example, the default contents of the *pin-list* array is determined by the targetted SOC. +The final *"True": []* ensures a value is provided if nothing else is matched. + +.. code-block:: json + + "pin-list": { + "type": "array", + "items": { + "type": "integer", + "minimum": 0, + "maximum": 255 + }, + "@default": { + "SMING_SOC == 'esp8266'": [ 1, 2, 3, 4 ], + "SMING_SOC == 'esp32c3'": [ 5, 6, 7, 8 ], + "SMING_SOC == 'esp32s2'": [ 9, 10, 11, 12 ], + "SMING_SOC in ['rp2040', 'rp2350']": [ 13, 14, 15, 16 ], + "True": [] + } + } Store loading / saving @@ -375,11 +407,11 @@ This can be overridden to customise loading/saving behaviour. The :cpp:func:`ConfigDB::Database::getFormat` method is called to get the storage format for a given Store. A :cpp:class:`ConfigDB::Format` implementation provides various methods for serializing and de-serializing database and object content. -Currently only **json** is implemented - see :cpp:class:`ConfigDB::Json::format`. +Currently only **json** is implemented - see :cpp:member:`ConfigDB::Json::format`. Each store is contained in a separate file. The name of the store forms the JSONPath prefix for any contained objects and values. -The :sample:`BasicConfig` sample demonstrates using the stream classes to read and write data from a web client. +The :sample:`Basic_Config` sample demonstrates using the stream classes to read and write data from a web client. .. important:: diff --git a/component.mk b/component.mk index 550c1a92..1debb393 100644 --- a/component.mk +++ b/component.mk @@ -9,31 +9,39 @@ ifneq (,$(COMPONENT_RULE)) CONFIGDB_GEN_CMDLINE := $(PYTHON) $(COMPONENT_PATH)/tools/dbgen.py COMPONENT_VARS := APP_CONFIGDB_DIR -APP_CONFIGDB_DIR ?= $(PROJECT_DIR)/$(OUT_BASE)/ConfigDB +APP_CONFIGDB_DIR := $(PROJECT_DIR)/out/ConfigDB COMPONENT_INCDIRS += $(APP_CONFIGDB_DIR) COMPONENT_APPCODE := $(APP_CONFIGDB_DIR) COMPONENT_VARS += CONFIGDB_SCHEMA CONFIGDB_SCHEMA := $(wildcard *.cfgdb) +CONFIGDB_JSON := $(patsubst %.cfgdb,$(APP_CONFIGDB_DIR)/schema/%.json,$(CONFIGDB_SCHEMA)) + +##@ConfigDB + +.PHONY: configdb-preprocess +configdb-preprocess: ##Pre-process .cfgdb into .json + $(Q) $(CONFIGDB_GEN_CMDLINE) --preprocess --outdir $(APP_CONFIGDB_DIR) $(CONFIGDB_SCHEMA) + CONFIGDB_FILES := $(patsubst %.cfgdb,$(APP_CONFIGDB_DIR)/%.h,$(CONFIGDB_SCHEMA)) CONFIGDB_FILES := $(CONFIGDB_FILES) $(CONFIGDB_FILES:.h=.cpp) -COMPONENT_PREREQUISITES := $(CONFIGDB_FILES) +COMPONENT_PREREQUISITES := configdb-preprocess $(CONFIGDB_FILES) -$(CONFIGDB_FILES): $(CONFIGDB_SCHEMA) +$(CONFIGDB_FILES): $(CONFIGDB_JSON) $(MAKE) configdb-build .PHONY: configdb-build -configdb-build: $(CONFIGDB_SCHEMA) +configdb-build: $(CONFIGDB_SCHEMA) ##Parse schema and generate source code $(vecho) "CFGDB $^" $(Q) $(CONFIGDB_GEN_CMDLINE) --outdir $(APP_CONFIGDB_DIR) $^ .PHONY: configdb-rebuild -configdb-rebuild: configdb-clean configdb-build +configdb-rebuild: configdb-clean configdb-build ##Force regeneration of source code .PHONY: configdb-clean -configdb-clean: - $(Q) rm -f $(CONFIGDB_FILES) +configdb-clean: ##Remove generated files + $(Q) rm -rf $(APP_CONFIGDB_DIR)/* clean: configdb-clean diff --git a/test/modules/Enum.cpp b/test/modules/Enum.cpp index 999668e4..607aa77a 100644 --- a/test/modules/Enum.cpp +++ b/test/modules/Enum.cpp @@ -76,6 +76,44 @@ class EnumTest : public TestGroup REQUIRE(!range.contains(badColor)); REQUIRE_EQ(range.clip(badColor), Color::blue); } + + /* + * RAW enums without ctype override + */ + TEST_CASE("Raw enums") + { + // These properties contain a value index, not the value itself + auto numIndex = root.getNum(); + int num = TestConfigEnum::numType.values()[numIndex]; +#ifdef SMING_RELEASE + REQUIRE_EQ(numIndex, 4); + REQUIRE_EQ(num, 25); +#else + REQUIRE_EQ(numIndex, 5); + REQUIRE_EQ(num, 45); +#endif + + auto wordIndex = root.getWord(); + String word = TestConfigEnum::wordType.values()[wordIndex]; + REQUIRE_EQ(wordIndex, 2); + REQUIRE_EQ(word, "brown"); + } + + TEST_CASE("Conditional enum") + { + String s; + for(auto v : TestConfigEnum::pinType.values()) { + s += v; + s += ','; + } +#if defined(ARCH_ESP8266) + REQUIRE_EQ(s, "1,2,3,4,"); +#elif defined(ARCH_HOST) + REQUIRE_EQ(s, "50,51,52,55,"); +#else + REQUIRE_EQ(s, "0,"); +#endif + } } }; diff --git a/test/test-config-enum.cfgdb b/test/test-config-enum.cfgdb index 1b05e565..9a387c49 100644 --- a/test/test-config-enum.cfgdb +++ b/test/test-config-enum.cfgdb @@ -5,9 +5,12 @@ "color": { "$ref": "#/$defs/Color" }, + "pin": { + "$ref": "#/$defs/Pin" + }, "num": { "type": "integer", - "default": 25, + "@default": "25 if SMING_RELEASE else 45", "enum": [ 15, 37, @@ -100,6 +103,43 @@ "green", "blue" ] + }, + "TestMode": { + "type": "string", + "ctype": "Color", + "@enum": { + "SMING_RELEASE": [ + "silent", + "error" + ], + "1": [ + "silent", + "error", + "debug", + "info" + ] + } + }, + "Pin": { + "comment": "Valid pin numbers depend on selected SOC", + "type": "integer", + "@enum": { + "SMING_SOC == 'esp8266'": [ + 1, + 2, + 3, + 4 + ], + "SMING_ARCH == 'Host'": [ + 50, + 51, + 52, + 55 + ], + "True": [ + 0 + ] + } } } } \ No newline at end of file diff --git a/tools/dbgen.py b/tools/dbgen.py index 7ed693b0..2c4d36ae 100644 --- a/tools/dbgen.py +++ b/tools/dbgen.py @@ -569,6 +569,7 @@ def data_size(self): @dataclass class Database(Object): schema: dict = None + calcprops: dict[str, Any] = None object_defs: dict[Object] = field(default_factory=dict) external_objects: dict[Object] = field(default_factory=dict) strings: StringTable = StringTable() @@ -660,31 +661,55 @@ def make_static_initializer(entries: list, term_str: str = '') -> list: def load_schema(filename: str) -> Database: def evaluate(expr: Any) -> Any: + if isinstance(expr, dict): + for k, v in expr.items(): + if evaluate(k): + return v + raise ValueError("No choice match") if isinstance(expr, list): return [evaluate(v) for v in expr] if isinstance(expr, str): return evaluator.run(expr) return expr + calc_props: dict[str, Any] = {} + + def calculate_props(props: dict, path: str): + new_props = {} + keys_by_id = {} + for key, value in props.items(): + if key.startswith('@'): + new_key = key[1:] + new_path = f'{path}/{new_key}' + try: + new_value = evaluate(value) + except Exception as e: + raise ValueError(f'{new_path} in "{filename}"') from e + calc_props[new_path] = new_value + key = new_key + value = new_value + new_props[key] = value + id = make_identifier(key) + if not id: + raise ValueError(f'Invalid key "{key}"') + key_conflict = keys_by_id.get(id) + if key_conflict: + raise ValueError(f'Key "{key}" conflicts with "{key_conflict}"') + keys_by_id[id] = key + props.clear() + props.update(new_props) + for k, v in props.items(): + if isinstance(v, dict): + calculate_props(v, f'{path}/{k}') + + '''Load JSON configuration schema and validate ''' - def parse_object_pairs(pairs): - d = {} - identifiers = set() - for k, v in pairs: - if k.startswith('@'): - v = evaluate(v) - k = k[1:] - id = make_identifier(k) - if not id: - raise ValueError(f'Invalid key "{k}"') - if id in identifiers: - raise ValueError(f'Key "{k}" produces duplicate identifier "{id}"') - identifiers.add(id) - d[k] = v - return d with open(filename, 'r', encoding='utf-8') as f: - schema = json.load(f, object_pairs_hook=parse_object_pairs) + schema = json.load(f) + + calculate_props(schema, '') + try: from jsonschema import Draft7Validator v = Draft7Validator(Draft7Validator.META_SCHEMA) @@ -696,7 +721,7 @@ def parse_object_pairs(pairs): except ImportError as err: print(f'\n** WARNING! {err}: Cannot validate "{filename}", please run `make python-requirements` **\n\n') schema_id = os.path.splitext(os.path.basename(filename))[0] - db = Database(None, schema_id, None, schema_id=schema_id, schema=schema) + db = Database(None, schema_id, None, schema_id=schema_id, schema=schema, calcprops=calc_props) databases[schema_id] = db return db @@ -1616,6 +1641,7 @@ def main(): parser.add_argument('cfgfiles', nargs='+', help='Path to configuration file(s)') parser.add_argument('--outdir', required=True, help='Output directory') + parser.add_argument('--preprocess', action="store_true", help='Pre-process and generate .json only') args = parser.parse_args() @@ -1623,11 +1649,38 @@ def main(): os.makedirs(schema_out_dir, exist_ok=True) for f in args.cfgfiles: - print(f'Loading "{f}"') + if not args.preprocess: + print(f'Loading "{f}"') db = load_schema(f) filename = os.path.join(schema_out_dir, f'{db.name}.json') - with open(filename, 'w') as f_schema: - json.dump(db.schema, f_schema, indent=2) + try: + with open(filename, 'r') as f_schema: + old_schema_content = f_schema.read() + except FileNotFoundError: + old_schema_content = None + new_schema_content = json.dumps(db.schema, indent=2) + if new_schema_content != old_schema_content: + with open(filename, 'w') as f_schema: + f_schema.write(new_schema_content) + + filename = os.path.join(schema_out_dir, 'summary.txt') + with open(filename, 'w') as f: + print('Calculated properties', file=f) + print('=====================', file=f) + empty = True + for db in databases.values(): + if not db.calcprops: + continue + empty = False + print(f'\n{db.name}', file=f) + for k, v in db.calcprops.items(): + print(f' {k} = {v}', file=f) + if empty: + print('None.', file=f) + + # If parse-only requested, we're done + if args.preprocess: + return for db in databases.values(): print(f'Parsing "{db.name}"')