Command-Level Templating

[RiSKeD]

I think we would benefit significantly from supporting templating at the command level. Right now, there’s no clean way to pass only parts of the arguments to individual submodules, or to reuse the same arguments across multiple modules. The current model forces all arguments to be handed off verbatim to a single module, which limits flexibility and reduces the potential use-cases of each module.

My proposal is to introduce an args list in the command configuration. This list would define the command’s arguments and make them available to any submodule. For example, imagine we want to copy a non-static file to the dutagent and then execute it there. Using the existing copy module (purely as a placeholder) we could pass arguments in a templated way: $X would refer to the X-th argument of the list and could be used multiple times across modules.

The args list would also double as metadata for user-facing help: argument descriptions, prompts, etc.

For backward compatibility, we keep the current main option as-is. So, a command may:

• Specify its own args list (new behavior),

• Use main to pass all raw args directly to a single module (current behavior), or

• Specify neither, meaning the command does not accept any arguments.

      flash-write:
        desc: |
           Copy file to dutagent and execute it.
        args: 
           - flash-file: "Firmware Binary to flash"
        modules:
          # Step 1: Copy file from client to dutagent
          - module: copy
            args: {flash-file}
          # Step 2:  Execute it on the dutagent
          - module: shell
            options:
              path: "/usr/bin/python3"
            args:
              - "{flash-file}"
              - "test123"

[AtomicFS]

Yeah, maybe. But feels a bit redundant to yaml anchoring.

[RiSKeD]

Maybe i am missing smth. but i don't think this is related to yaml anchoring. I want to pass the runtime args given by a user into a predefined template. YAML anchoring would be more of a reuse of static values.

[RiSKeD]

When defining a command that wraps a module (e.g. ssh), you sometimes want to forward all or part of the caller's arguments down to the module rather than binding them explicitly. Or allow the remaining arguments to be passed to another module.
@jenstopp and me thought up two candidate syntaxes for this.

Approach 1

Arguments are forwarded by referencing special implicit variables inside the args list using the existing ${...} template syntax:

${args} - expands to the full argument list passed to the command (replacing main)
${remaining} - expands to the arguments left over after all named parameters have been consumed

  exec:
    desc: "SSH exec"
    cmds:
      ssh:
        desc: "Holla"
        uses:
          - module: ssh
            args: ["${args}"]
            with:
              host: enigma
              user: oscar
              privatekey: ...

mixed:
  desc: "Flash firmware and exec ssh"
  args:
	  - name: duration
	    desc: "Duration to press GPIO Pin 1"
    - name: firmware-path
      desc: "Path to firmware binary"
  modules:
    - module: file
      args: ["${firmware-path}", "/tmp/fw.bin"]
    - module: gpio-button
      args: ["${duration}"]
        with:
          pin: 1
          activelow: true
    - module: flash
      args: ["/tmp/fw.bin"]
    - module: time-wait
      args: ["500"]
    - module: ssh
      args: ["${remaining}"]
        with:
          host: enigma
          user: oscar
          privatekey: ...

Approach 2

Instead of using template variables, argument forwarding is expressed as dedicated keys on the module:

forward_args: true - forward all arguments to the module (replacing main)
rest_args: true - forward only the arguments remaining after named parameters are consumed

  exec:
    desc: "SSH exec"
    cmds:
      ssh:
        desc: "Holla"
        uses:
          - module: ssh
            forward_args: true
            with:
              host: enigma
              user: oscar
              privatekey: ....

mixed:
  desc: "Flash firmware and exec ssh"
  args:
	  - name: duration
	    desc: "Duration to press GPIO Pin 1"
    - name: firmware-path
      desc: "Path to firmware binary"
  modules:
    - module: file
      args: ["${firmware-path}", "/tmp/fw.bin"]
    - module: gpio-button
      args: ["${duration}"]
        with:
          pin: 1
          activelow: true
    - module: flash
      args: ["/tmp/fw.bin"]
    - module: time-wait
      args: ["500"]
    - module: ssh
      rest_args: true
        with:
          host: enigma
          user: oscar
          privatekey: ...

Reviewing this now myself i would prefer approach 2. It has the benefit of being more explicit and not using any magic templating values (or reserved words for that matter). What do you think?

[AtomicFS]

That is not what I meant :D

Do not integrate jinja2, or any other templating mechanism other than those in YAML specification (anchoring for example).

Instead just embrace 3rd party tools, like jinja2 or other things, and make sure they can be used. And hint on their use in documentation. Do not integrate, just enable ;)

[RiSKeD]

Well, not sure how you would use jinja2 to parse the arguments given by a user. Sure, you can use jinja2 once to template the configuration in a simpler way but this can also be done right now. Or do you want to be able to use jinja2 in the config directly and the server parsing the input to generate a new one with each command?

[AtomicFS]

Oh wait, sorry, I got confused and in my mind was talking about the FWCI configs xD

As I am working on the dutctl testing with FWCI, in my mind those are very close topics easy to confuse :D

Sorry about that. Please disregard the comments about jinja2 :D

Essentially:

Personally, I prefer the approach 2, if I had to pick one. Keep in mind that in some cases the ordering of arguments also changes between Linux distributions (and maybe versions), as such the approach 1 would not be very predictable.

But personally, I am not really sure if this is even needed. When I started using Jinja2 templates in the dutctl testing, I also used jijnja2 to handle these things for me.

Maybe you should just embrace jinja2 templating as the "go-to" approach, instead of trying to get similar feature-set into dutctl.

[RiSKeD]

I've reduced the complexity a bit more with a new approach. Instead of having a difference between remaining_args and forward_args. forward_args handles both cases. It essentially forwards all the args to a module which are not mapped by the templating to a positional argument:

```yaml
run-command:
  desc: "Run a shell command on the device"
  modules:
    - module: shell
      forwardArgs: true

dutctl run my-device run-command ls -la /tmp
# shell receives: ["ls", "-la", "/tmp"]

When combined with command-level templating, the first N runtime args (one per declared args entry) are consumed for template substitution. The remaining args are forwarded:

flash-and-run:
  desc: "Flash firmware, then run a command on the device"
  args:
    - name: firmware-file
      desc: "Path to firmware binary"
  modules:
    - module: file
      args: ["${firmware-file}", "/tmp/fw.bin"]
    - module: flash
      args: ["/tmp/fw.bin"]
    - module: shell
      forwardArgs: true

[RiSKeD]

This will replace main and can be used in combination with the templating. Should give us the most flexibility while keeping it very simple.

[RiSKeD]

#284

[jenstopp]

Done in #264