orc-intro

Let's start with something really simple

Create a new object of type "Workflow" 

Some simple workflow will be created. 

You can press "One-time run" button and see it is running on the "Runs" tab.

Let's make something more complicated

In order to interact with YT from a workflow we need to provide a user token. Create a new secret store first: 

Create a new record in this store: save your YT token with key token. Save the secret.


Now let's create a workflow. There are two ways to do that: via UI and via CLI.

Creating workflow via UI

Just create a new workflow via Create object -\> Workflow button. You will get a simple workflow consisting of one step. Let's make it a bit more interesting. Replace its config with the following one, but use your secret store's path from the previous step in steps[0].secrets[0].value_ref:

{
    "_format_version": "0.1",
    "triggers": [
        {
            "trigger_type": "cron",
            "params": {
                "cron_expression": "*/30 * * * *"
            }
        }
    ],
    "steps": [
        {
            "step_id": "step_1",
            "task_type": "docker",
            "task_params": {
                "docker_image": "ghcr.io/tractoai/notebook-kernel-default:2024-12-27-14-31-09-307b3bc68",
                "command": "YT_TOKEN=$YT_SECURE_VAULT_YT_TOKEN yt start-query yql \"SELECT 42 as tweeenveertig\" \>&2"
            },
            "secrets": [
                {
                    "key": "YT_TOKEN",
                    "value_src_type": "secret_store",
                    "value_ref": "//path/to/your/secret/store:token"
                }
            ]
        },
        {
            "step_id": "step_2_1",
            "task_type": "docker",
            "task_params": {
                "docker_image": "ghcr.io/tractoai/notebook-kernel-default:2024-12-27-14-31-09-307b3bc68",
                "command": "echo \"first child\" \>&2"
            },
            "depends_on": ["step_1"]
        },
        {
            "step_id": "step_2_2",
            "task_type": "docker",
            "task_params": {
                "docker_image": "ghcr.io/tractoai/notebook-kernel-default:2024-12-27-14-31-09-307b3bc68",
                "command": "echo \"second child\" \>&2"
            },
            "depends_on": ["step_1"]
        }
    ]
}

On the secret's page you can see that a new delegation has been created: it allows the workflow to use the secret. You can revoke the delegation on that page (and it will break the workflow).

Now we can return to the workflow's page. Press "One time run" button to launch the workflow. Open the "Run" tab to see your new run.

And after some time the workflow must successfuly finish 

Creating workflow via CLI

Install orchestracto client:

!pip install orchestracto-client

Here is our workflow in YAML format:

---
_format_version: '0.1'
triggers:
- trigger_type: cron
  params:
    cron_expression: "*/30 * * * *"
steps:
- step_id: step_1
  task_type: docker
  task_params:
    docker_image: ghcr.io/tractoai/notebook-kernel-default:2024-12-27-14-31-09-307b3bc68
    command: YT_TOKEN=$YT_SECURE_VAULT_YT_TOKEN yt start-query yql "SELECT 42 as tweeenveertig" \>&2
  secrets:
  - key: YT_TOKEN
    value_src_type: secret_store
    value_ref: "//path/to/your/secret/store:token"
- step_id: step_2_1
  task_type: docker
  task_params:
    docker_image: ghcr.io/tractoai/notebook-kernel-default:2024-12-27-14-31-09-307b3bc68
    command: echo "first child" \>&2
  depends_on:
  - step_1
- step_id: step_2_2
  task_type: docker
  task_params:
    docker_image: ghcr.io/tractoai/notebook-kernel-default:2024-12-27-14-31-09-307b3bc68
    command: echo "second child" \>&2
  depends_on:
  - step_1

Save it on your computer.

Prepare env:

export YT_PROXY=planck.yt.nebius.yt
export YT_TOKEN=...

Then run

orc workflow update --wf-path //path/to/workflow/on/yt --from-file /local/path/to/workflow/config.yaml

This command will upload config on YT and register it in orchestracto. Now we can create a new run:

orc run create --wf-path //path/to/workflow/on/yt

The command will print ID of the run. You can use it to obtain run info:

!orc --format json run get --wf-path //home/yt-team/dmifedorov/some_new_workflow --run-id ab3b0734-da174153-8bc6c6be-ced658db | jq .stage

or logs:

!orc run get-logs --wf-path //home/yt-team/dmifedorov/some_new_workflow --run-id ab3b0734-da174153-8bc6c6be-ced658db | tail -n 5

You can also list runs:

!orc workflow get-runs --wf-path //home/yt-team/dmifedorov/some_new_workflow --limit 3