Cartesian – generator for complex configurations

TLDR; you define test-case 'dimensions' (box arch, compiler, etc) and this will be expanded to a Cartesian product of test cases. You can then make specific exclusions by defining functions that return false on certain dimension combinations.

Alternatively (in python):

  dimensions = [boxes, compilers, testcases]
  
  def excl(box, compiler, testcase):
      """Test exclusions.
      """
      ## don't run loadtest when RAM is low
      if compiler == 'loadtest' and box.ram < 8:
          return True
  
      return False
  
  
  for c in itertools.product(*dimensions):
      if excl(*c):
          continue
      run_testcase_with_params(c)

An alternative way to generate the final JSON from the input parts would be to use Jsonnet. http://jsonnet.org/. The benefits include support for using on the command line or from multiple programming languages (although, I guess not pure JS) as well as supporting more diverse generator functions than just the Cartesian product.

The `is()` function, or other system info, could be injected via Jsonnet's external variable feature.

Interesting project. I wonder what is the relation to approaches that allow explicit composition of enumerators, such as [1]? It seems that this approach could provide a nice interface for generators that need to be constructed explicitly. (This would be interesting to explore. By the way, I am the developer of the SciFe enumerator framework.)

[1] https://github.com/kaptoxic/SciFe