Skip to Content
Raydo

Build a deduplication workflow

Deduplicate four sample rows locally and compare the result with the expected rows.

This example uses the local Data transform and a test project. It needs no external capability or source file.

  1. Choose Workflows → Start delivery → Create blank flow. If the draft opens with example read and delivery steps, remove them. Add Data transform. If you keep the Goal / brief step, connect it to the transform.
  2. In Workflow → Start Form / Run Input, replace the default topic field with a required array named rows.
  3. Select the transform. In its input mappings, add a target items pointing to $.run.input.rows. If you want the mapping in the reference picker, declare items as an array in the step’s expected input fields.
  4. Set the processing method to Remove duplicates, key to id, input source to /items, and output name to uniqueRows. The transform spec under the step’s advanced settings should have these values.
{
  "schemaVersion": "raydo.data_transform.v1",
  "operation": "dedupe",
  "source": "/items",
  "key": "id",
  "outputKey": "uniqueRows"
}
  1. Use Review & Publish and complete the release reason, approval comment, and approver signature to activate the flow in the test workspace.
  2. Choose Start Run, select the test project, and enter the following run input.
{
  "rows": [
    {"id": "A01", "name": "Sample A"},
    {"id": "A01", "name": "Duplicate A"},
    {"id": "B02", "name": "Sample B"},
    {"id": "B02", "name": "Duplicate B"}
  ]
}

In Runs & Approvals, open the transform’s step output. result.uniqueRows should contain Sample A and Sample B. Deduplication keeps the first occurrence of each key. To use the array in a downstream step, map $.nodes.<actual-transform-node-id>.output.result.uniqueRows after connecting the steps. A missing key can cause rows to collapse together, so inspect the upstream rows before deploying the transform.

See the data processing pattern and data and flow nodes.

Build a deduplication workflow