Skip to main content

sdf run

The sdf run compiles and runs a dataflow. The command expects a dataflow.yaml file in the current directory. The command loads the dataflow, compiles all components, and starts a process to run the dataflow —all artifacts are stored in the .sdf directory and should not be manually changed.

After provisioning the command enters an interactive shell that allows you to interact with the dataflow.

sdf run command​

The run command has several options:

$ sdf run -h
Run dataflow (requires: dataflow.yaml)

Usage: sdf run [OPTIONS]

Options:
      --ui
          Start Development Studio [env: UI=]
  -p, --port <PORT>
          Port for web server to listen on [env: PORT=] [default: 8000]
  -e, --env <ENV_LIST>
          (Optional) environment variables to be passed to each operator They
          should be passed using key=value format Eg. sdf run -e foo=bar -e
          key=value -e one=1
      --skip-running
          when set, it will skip running the service
      --build-profile <BUILD_PROFILE>
          [default: release]
      --dev
          set runtime to use dev mode [env: DEV=]
      --prod
          set runtime to use production mode [env: PROD=]
      --force-update
          Force update

Where:

  • --ui start graphical represetation of the project in a web browser. The default port is 8000.
  • --port sets the port for the web server to listen on
  • --env sets environment variables to be passed to operators
  • --skip-running - compiles components and exists without running the dataflow
  • --build-profile - sets the build profile
  • --dev - sets runtime to apply dev specific parameters
  • --prod - sets runtime to apply prod specific parameters
  • --force-update - forces the update of the project dependencies

In addition, the run command opens interactive shell.

run interactive shell​

The interactive shell allows you to look-up runtime object, such as state.

>> help
SDF - Stateful Dataflow

Usage: <COMMAND>

Commands:
  show  Show or List states. Use `show state --help` for more info
  exit  Stop interactive session

show state​

Show states or show state for given namespace and key.

>> show state -h
List all states or show state for given key

Usage: show state [OPTIONS] [NAMESPACE]

Arguments:
  [NAMESPACE]  namespace of state

Options:
      --key <KEY>        key of state
      --filter <FILTER>  filter regex

Where:

  • --key and --filter refines the result.

Examples​

Run command​

Navigate to the directory with dataflow.yaml file, and run the command:

$ sdf run

The command generates all components required and starts a process to run the dataflow.

Show all available objects​

Use the default command to show ojects:

>> show state
 Namespace                                Keys  Type
 filter-service/sentences/metrics         1     table
 filter-service/filter-questions/metrics  1     table
 filter-service/sentences/topic.offset    1     offset

Show state for a key​

Show the detailed information:

>> show state filter-service/filter-questions/metrics
 Key    Window  succeeded  failed
 stats  *       2          0