Export Flex Insights Data with Twilio Functions

Time to read: 6 minutes

May 13, 2024
Written by
Bry Schinina
Twilion
Reviewed by
Niti Patel
Twilion
Paul Kamp
Twilion

Export Flex Insights Data with Twilio Functions

Twilio Flex Insights is the out-of-the-box reporting and analytics tool for Twilio Flex. You can create reports with a user-friendly interface and export data from the UI or the API.

However, interacting with the Flex Insights API differs from other Twilio APIs. For example, authentication is a two-step process, fetching data involves strategic retries, and dynamic filtering requires intricate data mappings.

In this tutorial, you will learn how to simplify and streamline your interactions with the Flex Insights API using Twilio Serverless Functions to build a custom Node.js wrapper. By the end, you’ll have a single, secured API endpoint that receives requests to export reports with date filters and returns raw CSV data.

Ready to get started?

To follow along with this tutorial, you will need:

Get started with the Twilio Serverless Toolkit

The Serverless Toolkit , a CLI tool, helps you develop Twilio Functions locally and deploy them to Twilio Functions & Assets. The best way to work with the Serverless Toolkit is through the Twilio CLI.

If you don't have the Twilio CLI installed yet, run the following commands from your terminal to install it and the Serverless Toolkit:

npm install twilio-cli -g
twilio login
twilio plugins:install @twilio-labs/plugin-serverless

Then, create a new Serverless Service called serverless-flex-insights and navigate to the new directory:

twilio serverless:init serverless-flex-insights --empty
cd serverless-flex-insights

Configure the Service

For this solution, you’ll need a way to make HTTP requests. Numerous packages enable network requests with Node.js. This tutorial uses the axios library for simplicity.

From the /serverless-flex-insights directory, install the lone dependency.

npm i axios --save

Finally, create a new directory called helpers in the functions directory. This will store the helper functions you’ll create in the next few steps.

mkdir functions/helpers

Now that your environment is prepared, you’re ready to start exploring the Flex Insights API.

Create axios Wrapper Function

Interacting with the Flex Insights API requires one (or both) of the following authentication tokens to be present in the request headers:

  • X-GDC-AuthSST (Super-Secure Token)
  • X-GDC-AuthTT (Temporary Token)

In this step, you’ll make a helper function to automatically include the necessary tokens for each network request.

Create a new file in the /helper directory called request.private.js and update it with the following code:

const axios = require("axios")

//axios request customized for Flex Insights
exports.httpRequest = async (method, path, auth, params) => {
    const options = {
        method: method,
        url: path,
        headers: {
            'Content-Type': 'application/json'
        }
    }

    if (params) {
        if (method.toUpperCase() === "GET") {
            options.params = params
        } else {
            options.data = params
        }
    }
    if (auth.sstoken) {
        options.headers['X-GDC-AuthSST'] = auth.sstoken
    }
    if (auth.ttoken) {
        options.headers['X-GDC-AuthTT'] = auth.ttoken
    }

    try {
        //return response from request
        return await axios(options)
    }
    catch (error) {
        let formattedError = JSON.stringify({
            message: error.message,
            status: error.response?.status,
            data: error.response?.data
        })

        throw Error(formattedError)
    }
}

The exported httpRequest function expects the following parameters:

  • method - String - HTTP request method
  • path - String - request url
  • auth - Object - Flex Insights sstoken and/or Flex Insights ttoken
  • params - Object - request parameters

It returns the responsefrom the request to the Flex Insights API.

In the next step, you will use this httpRequest function to generate the Flex Insights authentication tokens.

Flex Insights Authentication

Authentication requires two API requests to get tokens and a third optional API request to invalidate the tokens before their typical expiry.

Super-Secure Token

First, you need a SST ( Super Secured Token), which can be minted with the username and password credentials you created in the prerequisite steps.

SuperSecure Tokens (SST) are long-lived and have a default lifetime of two weeks.

Temporary Token

Once you have the SST, you can use it to generate a short-lived TT (Temporary Token).

Temporary Tokens (TT) are valid for 10 minutes, and must be included in any subsequent requests to the Flex Insights API.

Delete Super-Secured Token

An optional third request can be made to deactivate the tokens before their default expiry. Delete the SST to invalidate both tokens.

Create Token Helper Function

Let’s create a helper Function to efficiently handle the above three requests.

Make a new file in the /helper directory called token.private.js containing the following code:

//import custom request function
const { httpRequest } = require(Runtime.getFunctions()['helpers/request'].path)

//set Flex Insights endpoint as global variable
const domain = `https://analytics.ytica.com/gdc/account`

//get SST
exports.getSuperSecureToken = async (credentials) => {
    const loginUrl = `${domain}/login`
    const params = {
        postUserLogin: {
            login: credentials.username,
            password: credentials.password,
            remember: 0,
            verify_level: 2
        }
    }
    const loginAuth = await httpRequest("POST", loginUrl, {}, params)

    //return user profile and SST
    return {
        user: loginAuth.data.userLogin.profile.split('/').pop(),
        sstoken: loginAuth.data.userLogin.token,
    }
}

//get TT
exports.getTmpToken = async (sstoken) => {
    const tokenUrl = `${domain}/token`
    const tokenAuth = { sstoken }
    const token = await httpRequest("GET", tokenUrl, tokenAuth)

    //return TT
    return token.data.userToken.token
}

//invalidate tokens
exports.deleteSST = async (auth, ttoken) => {
    const logoutUrl = `${domain}/login/${auth.user}`
    const logoutAuth = { sstoken: auth.sstoken, ttoken: ttoken }
    await httpRequest("DELETE", logoutUrl, logoutAuth)
}

This module exports three functions, which will be used in a later step:

  • getSuperSecureToken
  • getTmpToken
  • deleteSST

Continue to the next section to learn about filtering and exporting report data.

Report Helper Functions

Create a Date filter Function

Each kind of filter within Flex Insights (Date, Time Interval, Queue, etc.) maps to a unique “identifier” within your Flex Insights environment.

Finding the identifier for your needs can be a bit tricky. For more guidance on finding the specific identifier you need, take a look at our mapping global identifiers documentation.

In this example, you will implement a Date filter, which has the following identifier:

"date.date.mmddyyyy"

Learn how to use this identifier in the next step.

Filter Function - filter.private.js

Make a new file in the /helper directory called filter.private.js containing the following code:

//import custom request function
const { httpRequest } = require(Runtime.getFunctions()['helpers/request'].path)

//get the Date filter
exports.getDateFilter = async (ttoken, workspaceId) => {
    const filterUrl = `https://analytics.ytica.com/gdc/md/${workspaceId}/identifiers`
    const params = {
        identifierToUri: ["date.date.mmddyyyy"]
    }
    const auth = { ttoken }
    const date = await httpRequest("POST", filterUrl, auth, params)

    // return date filter uri
    return date.data.identifiers[0].uri
}

This function will make a request to Flex Insights to acquire the correlating uri for the Date identifier. We will use this in the next step when configuring the request to fetch the Report data.

Get Report CSV

Next, let’s create a helper Function for exporting the report object (with the Date filter configured) and downloading the CSV contents.

Make a new file in the /helper directory called report.private.js containing the following code:

//import helper functions
const { getDateFilter } = require(Runtime.getFunctions()['helpers/filter'].path)
const { httpRequest } = require(Runtime.getFunctions()['helpers/request'].path)

//get parameters for report export
const getReportExportConfig = async (ttoken, workspaceId, reportId, dateFilter) => {
    return JSON.stringify({
        report_req: {
            report: `/gdc/md/${workspaceId}/obj/${reportId}`,
            context: {
                filters: [
                    {
                        uri: await getDateFilter(ttoken, workspaceId),
                        constraint: {
                            type: "interval",
                            from: dateFilter.startDate,
                            to: dateFilter.endDate
                        }
                    }
                ]
            }
        }
    })
}

//get the report export
const getReportExport = async (ttoken, workspaceId, reportId, dateFilter) => {
    const exportUrl = `https://analytics.ytica.com/gdc/app/projects/${workspaceId}/execute/raw`
    const params = await getReportExportConfig(ttoken, workspaceId, reportId, dateFilter)
    const exportAuth = { ttoken }
    const reportExport = await httpRequest("POST", exportUrl, exportAuth, params)

    //return the report export uri
    return reportExport.data.uri
}

//get CSV data from export
const getCsvData = async (ttoken, reportExport, reportId) => {
    const csvUrl = `https://analytics.ytica.com${reportExport}`
    const csvAuth = { ttoken }
    const reportCsvObj = await httpRequest("GET", csvUrl, csvAuth)

    switch (reportCsvObj.status) {
        case 200:
            return reportCsvObj.data //CSV data
        case 202:
            //report is not ready - try again
            return await getCsvData(ttoken, reportExport, reportId)
        default:
            throw Error(JSON.stringify({ message: "Something went wrong getting report CSV data.", status: reportCsvObj.status }))
    }
}

module.exports = {
    getReportExport,
    getCsvData
}

getReportExportConfig function

Returns the request parameters needed to fetch the report with the proper Date filter applied.

getReportExport function

Gets an export uri from Flex Insights. This uri contains query parameters that allow for the data to be exported with applied filters.

getCsvData function

Uses the export uri to fetch the report as CSV data.

Handle 202 Response

For large or complex reports, Flex Insights may require additional processing time. In these cases, the API will return a 202 response and the request must be retried until the API returns a 200.

This solution will keep retrying by recursively calling the getCsvData function until a 200 response is received–or the Twilio Serverless function times out.

Twilio Serverless Functions have an execution time limit of 10 seconds. Executions that surpass this threshold will terminate with a HTTP 504 timeout response and throw a 82002 error in the Twilio logs. Potential solution: Adjust your Flex Insights Report to contain less data or less complex custom metrics.

Secure Function with Authentication

It’s important to protect your public Function and any sensitive data that can be exposed from unauthorized requests. This final helper function enhances the security of your public endpoint by processing requests only when they include a valid Flex Insights username and password.

Authentication is validated by attempting to generate a Super-Secure Token with the supplied credentials.

Validation helper Function

Make a new file in the /helper directory called validator.private.js containing the following code:

//import helper function
const { getSuperSecureToken } = require(Runtime.getFunctions()['helpers/token'].path)

//set error message
const error = JSON.stringify({
    message: "Invalid or missing Flex Insights credentials",
    status: 401
})

//extract basic auth credentials
const getCredentials = (authorization) => {
    const auth = authorization.split(" ")

    if (auth[0].toLowerCase() !== 'basic') { 
        throw Error(error)
    }

    let bufferStr = Buffer.from(auth[1], "base64").toString('utf8')
    const credentials = bufferStr.split(":")
    return {
        username: credentials[0],
        password: credentials[1]
    }
}

//validate auth by creating SST
exports.validator = async (authorization) => {
    if (!authorization) { 
        throw Error(error)
    }

    const credentials = getCredentials(authorization)
    return await getSuperSecureToken(credentials)
}

This Function uses the getSecureToken helper created earlier, and ensures authorization is supplied and in the Basic Authentication format.

It exports the final validator function that returns the SST or throws a 401 error.

Create Public API Endpoint

Finally, it is time to assemble all the helper functions into the final, public API endpoint.

Make a new file in the /functions directory called getReport.js containing the following code:

//import helper functions 
const { validator } = require(Runtime.getFunctions()['helpers/validator'].path)
const { getTmpToken, deleteSST } = require(Runtime.getFunctions()['helpers/token'].path)
const { getReportExport, getCsvData } = require(Runtime.getFunctions()['helpers/report'].path)

exports.handler = async (context, event, callback) => {
    //set up response
    const response = new Twilio.Response()
    response.appendHeader('Access-Control-Allow-Origin', '*')
    response.appendHeader('Access-Control-Allow-Methods', 'OPTIONS, POST, GET')
    response.appendHeader('Access-Control-Allow-Headers', 'Content-Type')

    //get request variables
    const { WorkspaceId, ReportId, DateStart, DateEnd } = event

    if (WorkspaceId && ReportId && DateStart && DateEnd) { //verify parameters were received 
        try {
            //validate credentials - return object with user profile_id and SST
            const auth = await validator(event.request.headers.authorization)

            //get temporary token
            const ttoken = await getTmpToken(auth.sstoken)

            //get report export
            const reportExport = await getReportExport(ttoken, WorkspaceId, ReportId, { startDate: DateStart, endDate: DateEnd })

            //get report csv
            const reportCsv = await getCsvData(ttoken, reportExport, ReportId)

            //invalidate the tokens
            await deleteSST(auth, ttoken)

            //determine the size of the CSV data
            const size = Buffer.byteLength(reportCsv, 'utf8')

            //compare CSV size to Functions limit (4MB)
            if (size < (4 * 1024 * 1024)) {    //1MB = 1024 * 1024 bytes
                //return CSV in response
                response.appendHeader('Content-Type', 'text/csv')
                response.setBody(reportCsv)
            } else { //CSV is too large
                //return a link to download CSV
                const downloadLink = { downloadCsvLink: `https://analytics.ytica.com${reportExport}` }
                response.appendHeader('Content-Type', 'application/json')
                response.setBody(downloadLink)
            }
        }
        catch (e) {
            let error = JSON.parse(e.message)
            response.appendHeader('Content-Type', 'application/json')
            response.setBody(error)
            response.setStatusCode(error.status)
        }
    }
    else {
        response.setBody("Missing request parameters.")
        response.setStatusCode(400)
    }

    return callback(null, response)
}

Request variables and headers are accessed via the event object’s properties:

  • WorkspaceId
  • ReportId
  • DateStart
  • DateEnd
  • request.headers.authorization

This code goes through each step needed to obtain the report CSV data:

  1. Get SST
  2. Get TT
  3. Get report export
  4. Get CSV data
  5. Return CSV or a link if data is too large

Deploy the Service

Run the following command from the root directory (/serverless-flex-insights) to deploy the Function:

twilio serverless:deploy

Copy the deployed Function URL for the getReport Function.

https://YOUR_DOMAIN_NAME.twil.io/getReport

Test the new API with a Report

Flex Insights Identifiers

Get the Flex Insights identifiers for your environment:

  1. Log in to the Analytics Portal with the credentials you created in one of the prerequisite steps

  2. Copy and store the Workspace ID from the URL for use later (see image below)

    1. Example: k4wauvc30rsj26tw4phjct3zyi6lif5c

  3. Navigate to the Reports tab from the top menu options

  4. Select one of the Reports

  5. Copy and store the Object ID/Report ID from the URL (see image below)

    1. Example: 3035643

Where to find the Flex Insights ID
Where to find the Flex Insights ID

Make HTTP Request

Make an HTTP request to your newly deployed public Function endpoint. Below is an example for how to test with cURL:

curl -u 'USERNAME:PASSWORD' 'https://YOUR_DOMAIN_NAME.twil.io/getReport?WorkspaceId=k4wauvc30rsj26tw4phjct3zyi6lif5c&ReportId=3035643&DateStart=2023-12-01&DateEnd=2023-12-30'

Replace USERNAME and PASSWORD with your credentials.Replace serverless domain (YOUR_DOMAIN_NAME) with your deployed Function Service domain.

Replace WorkspaceId (k4wauvc30rsj26tw4phjct3zyi6lif5c) and ReportId (3035643) with the values that correspond to your Flex Insights environment.

CSV Data

Note the returned CSV data and rejoice in a job well done.

"Queue","Date","Interval 15 Minutes","Channel","Direction","Offered Contacts","Average Handle Time","Abandons","Service Level","Average Speed of Answer","Volume Handled"
"Everyone","12/02/2023","17:30","Call","Inbound","4","80.5000000000000000",,"4","4.0000000000000000","4"
"Everyone","12/03/2023","14:15","Call","Inbound","1",,"1",,,
"Everyone","12/13/2023","14:30","Call","Inbound","4","294.2500000000000000",,"4","3.2500000000000000","4"

Conclusion

In this tutorial, you learned how to retrieve data with dynamic date filters from the Flex Insights API using Twilio Serverless Functions. You can now access your organization’s operational insights with more ease and flexibility.

To learn more about the Flex Insights Analytics Portal, consider going through the Twilio Training course, Flex Insights Advanced: Custom Reporting.

To learn about adding custom data to your Flex Insights, take a look at our guide for enhancing the data captured in your historical reports.

Bry is a developer and educator who enjoys making useful and intuitive software solutions. She works as Tech Lead and Sr. Technical Account Manager at Twilio, solving complex problems and helping organizations succeed in their digital endeavors. She can be reached at bschinina [at] twilio.com.

Related Posts

Related Resources

A newspaper article

Twilio Docs

From APIs to SDKs to sample apps

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
An Open book

Resource Center

The latest ebooks, industry reports, and webinars

Learn from customer engagement experts to improve your own communication.
User group reactions

Ahoy

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.