Transformations is one of RudderStack's key features. It allows you to leverage custom JavaScript functions to transform the events in a destination-specific format. These transformations can be used across your Event Stream, Warehouse Actions, and Cloud Extract pipelines, and help you implement a variety of use-cases, like:

• Filtering and/or sampling events.

• Enriching events by implementing static logic or by leveraging an external API.

• Cleaning data by removing unnecessary bits.

• Data masking or removing sensitive PII information in the events to ensure data privacy.

• Data aggregation / rolling-up at a micro-batch level.

• Implementing external actions on the events in the stream by means of an API.

Here's a quick 2-minute walkthrough of this feature:

Creating a New Transformation in RudderStack

• Log into your RudderStack dashboard.

• Click on the Transformations link from the left panel in the dashboard, as shown:

• Click on the Create New option as shown:

• Next, assign a name for this new transformation. Enter the transformation function's code in the Transformation window, as shown:

• Add your transformation code within the transformEvent function in the Transformation window. You can also add other functions and call them from within transformEvent.

• RudderStack also gives you the ability to test your transformation function with the Run Test option as shown below:

• In case you want to perform any aggregation/roll-up operation on a micro batch of events, use the transformBatch feature, as shown:

/***  
* This function gets executed on a batch of events before it gets pushed to a destination
* events    => JSON list of events sent to rudder
* metadata (optional) => Javascript function which can be used to access metadata of
                        the given event by calling metadata(event)  
* After all the transformations are done, the final event that 
* needs to be pushed to the destination should be returned by this function
***/
export function transformBatch(events, metadata) {      
  return events
}

Libraries

One of the key features of RudderStack transformations is the ability to reuse code written for a transformation in other transformations as well. For this, RudderStack lets you create libraries or functions that can be reused in different transformations.

To create a library, follow these steps:

• Click on the Transformations Link to see the Libraries section. Use Create New option to add new libraries.

• Add the library's Name and include the custom functions that you need to reuse across all other transformations as shown:

• You can add more functions under a single library, as shown:

Using Libraries in Transformations

To use the libraries in your existing transformation, simply take the name of your library and convert it into camel case without spaces - this becomes your library handle.

As an example, if your library name is is rudder email, then the library handle would be isRudderEmail.

Let's say you want to import a function called rudderEmail, which returns true for the emails from a specific domain (e.g. RudderStack) and false otherwise, from the is rudder email library. Also, you want to use this function to filter the events that don't have the email address of the specified domain.

We can implement this in the following way:

import { rudderEmail } from 'isRudderEmail'
export function transformEvent(event) {
    const email = event.context && event.context.traits && event.context.traits.email;
    if (email) {
      if (!rudderEmail(email)) return;
    }
    return event;
}

On running a test, an example event not having the specified email domain is filtered out, as shown below:

Accessing Metadata

RudderStack injects a function metadata(event) into your transformations as an argument. This allows you to access the event metadata variables that help you customize your transformations.

The following properties, if available, are present in the metadata response:

Since you may not need the metadata in every transformation, you can optionally access it by including it in your function signature as an argument wherever required.

An example of this is as shown below:

export function transformEvent(event, metadata){
  const meta = metadata(event);
  event.sourceId = meta.sourceId
​
  return event;
}

External API Requests

You can make any number of external API requests in your transformer functions and use the response to enrich your events data. RudderStack injects an asynchronous fetch(url) function into your transformations. It makes an API call to the given URL and returns the response in a JSON format.

Examples of how to use the fetch function in transformations are shown below:

export async function transformEvent(event) {
    const res = await fetch("any_api_endpoint");
    event.response = JSON.stringify(res);
    return event;
}

export async function transformEvent(event) {
    const res = await fetch("post_url", {
        method: "POST", // POST, PUT, DELETE, GET, etc.
        body: JSON.stringify(event),
        headers: {
         "Content-Type": "application/json;charset=UTF-8",
        },
    });
    event.response = JSON.stringify(res);
    return event;
}

export async function transformEvent(event) {
    const res = await fetch("post_url", {
        method: "POST", // POST, PUT, DELETE, GET, etc.
        headers: {
         "Content-Type": "application/json;charset=UTF-8",
         "Authorization": "Bearer <your_authorization_token>"
        },
        body: JSON.stringify(event),
    });
    event.response = JSON.stringify(res);
    return event;
}

Debugging with logs

You can access logs while running a test by including log() function in transformation code wherever required.

An example of this is as shown below:

export function transformEvent(event, metadata) {
  const meta = metadata(event);
  event.sourceId = meta.sourceId
​
  log("Event Name is", event.event, ";", "Message Id is", event.messageId);
  log("Source Id is", meta.sourceId);
​
  return event;
}

Upon running a test on above code, you can see above logs in Logs section as shown below:

Limits

You should take into account the memory and time limits when adding a new user transformation. Each invocation of the user transformation should not exceed these limits.

The limits are:

Contact Us

If you want to know more about transformations and libraries in RudderStack, feel free to contact us. You can also talk to us on our Slack channel.