How To Use Recording Add-ons in Java

In this guide we’ll cover how to combine Twilio Add-ons with Programmable Voice to use data from Twilio's partners in your Spring MVC web application. Let's get started!

What’s an Add-On?

Twilio Add-ons are pre-integrated third party services that you can use with your Twilio application. Add-ons enhance the responses your application receives from Twilio by including additional data from Twilio’s partners.

In this guide we’ll cover how to combine Twilio Add-ons with Programmable Voice to use data from Twilio's partners in yourSpring MVC web application. The sample code in this guide will use the Twilio Java SDK and the Spring MVC framework. Let's get started.

Loading Code Samples...
Language
{
  "status":"successful",
  "message":null,
  "code":null,
  "results":{
    "ibm_watson_speechtotext":{
      "request_sid":"XRc82d4197bbc51d5982588ae367365a51",
      "status":"successful",
      "message":null,
      "code":null,
      "payload":[
        {
        "content_type":"application/json",
        "url":"https://api.twilio.com/2010-04-01/Accounts/AC597d02bf69de79f8f411f289ac211a0d/Recordings/REab85c1c2860471abf040b545c4e5aa40/AddOnResults/XRc82d4197bbc51d5982588ae367365a51/Payloads/XH364bd4514b1377fd8844cd9793416131/Data"
        }
      ],
      "links":{
        "add_on_result":"https://api.twilio.com/2010-04-01/Accounts/AC597d02bf69de79f8f411f289ac211a0d/Recordings/REab85c1c2860471abf040b545c4e5aa40/AddOnResults/XRc82d4197bbc51d5982588ae367365a51",
        "payloads":"https://api.twilio.com/2010-04-01/Accounts/AC597d02bf69de79f8f411f289ac211a0d/Recordings/REab85c1c2860471abf040b545c4e5aa40/AddOnResults/XRc82d4197bbc51d5982588ae367365a51/Payloads",
        "recording":"https://api.twilio.com/2010-04-01/Accounts/AC597d02bf69de79f8f411f289ac211a0d/Recordings/REab85c1c2860471abf040b545c4e5aa40"
      }
    }
  }
}
This is a sample of the payload your application would receive when using the IBM Watson Speech to Text add-on.
Sample data from the IBM Watson Text to Speech Add-on

This is a sample of the payload your application would receive when using the IBM Watson Speech to Text add-on.

Here’s an example that uses the IBM Watson Speech to Text Add-on. The generated transcription will be pushed to the application through a callback from Twilio.


The IBM Watson Speech add-on can convert Twilio audio recordings into text using machine intelligence to combine information about grammar and language structure.

Loading Code Samples...
Language
{
  "user_token": "XRf951c9b0bce23930750991099e553b0a",
  "results":[
    {
      "results":[
        {
          "keywords_result":{},
          "alternatives":[
            {
            "timestamps":[
              ["I", 0.04, 0.28],
              ["love", 0.28, 0.71],
              ["coding", 0.74, 1.25]
            ],
            "confidence":0.782,
            "transcript":"I love coding "
            }
          ],
          "final":true
        }
      ],
      "result_index":0
    }
  ],
  "id":"b0e7a660-bcb6-11e6-8b1b-192a5f5dc12f",
  "event":"recognitions.completed_with_results"
}
This is the sample analysis that the Add-on partner provides which is included in the URL of the payload above.
Sample recording analysis from the IBM Watson Speech to Text Add-on

This is the sample analysis that the Add-on partner provides which is included in the URL of the payload above.

See the Add-ons Catalog for the complete list of available add-ons.

Enabling Add-ons in the Twilio Console

The first stop in using an add-on is your Twilio Console. Log in and navigate to the Add-ons page within the Marketplace section.

Add-ons catalog

Then choose on the add-on you want to enable and click the “Install” button. After accepting the Terms of Service, it will be installed on your account. In this example, we’ll use the IBM Watson Speech to Text Add-on.

IBM Watson Speech to Text install Add-on

Once installed, select the checkbox for the recording method you are using (see the different recording methods here). In our example to follow, we would only need the "Record Verb Recordings".  Set the “Callback URL” and “Callback Request Method” as appropriate for your public server URL (instructions below). Pro Tip: you can quickly inspect what Twilio will send to your callback URL using tools like RequestBin before you actually set up your server.

IBM Watson Speech to Text Configure tab

 Now we’re ready to use the extra data from our add-on in our Flask application.

Using Add-on data in your application

Twilio will include data from all of your add-ons in an “AddOns” field on HTTP requests it makes to your application. The following example demonstrates how to access add-on data within your Spring MVC application.

Loading Code Samples...
Language
SDK Version:
  • 7.x
import com.jayway.jsonpath.DocumentContext;
import com.jayway.jsonpath.JsonPath;
import com.jayway.jsonpath.ReadContext;
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;
import com.mashape.unirest.http.exceptions.UnirestException;
import com.twilio.twiml.*;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;

import java.util.List;
import java.util.Map;

public class RecordingController {

    private static final String ACCOUNT_SID = System.getenv("TWILIO_ACCOUNT_SID");
    private static final String AUTH_TOKEN = System.getenv("TWILIO_AUTH_TOKEN");

    @RequestMapping(value = "/voice",
            produces = "application/xml",
            method = {RequestMethod.GET, RequestMethod.POST})
    public ResponseEntity<String> voice() throws TwiMLException {
        // Start our TwiML response
        VoiceResponse voiceResponse = new VoiceResponse.Builder()
                .say(new Say
                        .Builder("Hi! I want to know what do you think about coding.")
                        .build())
                .record(new Record.Builder().maxLength(10).action("/recording").build())
                .hangup(new Hangup())
                .build();

        return okResponse(voiceResponse);
    }

    @RequestMapping(value = "/recording",
            produces = "application/xml",
            method = {RequestMethod.GET, RequestMethod.POST})
    public ResponseEntity<String> recording(@RequestParam("RecordingUrl") String recordingUrl) throws TwiMLException {
        System.out.println(recordingUrl);

        // Start our TwiML response
        VoiceResponse voiceResponse = new VoiceResponse.Builder()
                .say(new Say
                        .Builder("Thanks for howling... take a listen to what you howled.")
                        .build())
                .play(new Play.Builder(recordingUrl).build())
                .say(new Say.Builder("Goodbye.").build())
                .build();

        return okResponse(voiceResponse);
    }
    @RequestMapping(value = "/recording",
            produces = "application/xml",
            consumes = "application/json",
            method = {RequestMethod.POST, RequestMethod.PUT})
    public ResponseEntity<String> callback(@RequestParam("AddOns") String addOns) throws UnirestException {
        // If the Watson Speech to Text add-on found nothing, return immediately
        ReadContext addonsContext = JsonPath.parse(addOns);
        Map<String, Object> results = addonsContext.read("$.results");
        if(!results.containsKey("ibm_watson_speechtotext")) {
            return new ResponseEntity<>(
                    "Add Watson Speech to Text add-on in your Twilio console", HttpStatus.OK);
        }

        // Retrieve the Watson Speech to Text add-on results
        String payloadUrl = addonsContext.read("$.results['ibm_watson_speechtotext'].payload[0].url");
        HttpResponse<String> response = Unirest
                .get(payloadUrl)
                .basicAuth(ACCOUNT_SID, AUTH_TOKEN)
                .asString();

        DocumentContext speechToTextContext = JsonPath.parse(response.getBody());
        List<String> transcripts = speechToTextContext
                .read("$.results[0].results[*].alternatives[0].transcript");

        return new ResponseEntity<>(String.join("", transcripts), HttpStatus.OK);
    }

    private ResponseEntity<String> okResponse(VoiceResponse voiceResponse) throws TwiMLException {
        return new ResponseEntity<>(voiceResponse.toXml(), getHttpHeaders(), HttpStatus.OK);
    }

    private HttpHeaders getHttpHeaders() {
        final HttpHeaders httpHeaders = new HttpHeaders();
        httpHeaders.setContentType(MediaType.APPLICATION_XML);
        return httpHeaders;
    }
}
Use the IBM Watson Speech to Text add-on to get a transcription of a recording
Use IBM Watson Speech to Text Add-on

Use the IBM Watson Speech to Text add-on to get a transcription of a recording

This application gets the transcription for the recorded call from IBM Watson Speech to Text Add-on we configured in the Twilio console. You can find a detailed specification of the data structure for an individual add-on within that add-on's page in the catalog.

And that’s it! Try giving your application a call to see your add-on in action. If you've never received a Twilio phone call with Java before, check out our guide on that topic.

Where to next?

We focused on the IBM Watson Speech to Text Add-on in this guide, but you can just as easily use any of the add-ons available in the catalog in your Twilio Console.

Check out the full Add-Ons API Reference documentation if you want to learn more about how add-ons work, or even how to publish your own.

Hector Ortega
David Prothero
Andrew Baker
Kevin Whinnery
Stephen Wai
Rob Spectre
Agustin Camino
Ricky Robinett

Need some help?

We all do sometimes; code is hard. Get help now from our support team, or lean on the wisdom of the crowd browsing the Twilio tag on Stack Overflow.

1 / 1
Loading Code Samples...
{
  "status":"successful",
  "message":null,
  "code":null,
  "results":{
    "ibm_watson_speechtotext":{
      "request_sid":"XRc82d4197bbc51d5982588ae367365a51",
      "status":"successful",
      "message":null,
      "code":null,
      "payload":[
        {
        "content_type":"application/json",
        "url":"https://api.twilio.com/2010-04-01/Accounts/AC597d02bf69de79f8f411f289ac211a0d/Recordings/REab85c1c2860471abf040b545c4e5aa40/AddOnResults/XRc82d4197bbc51d5982588ae367365a51/Payloads/XH364bd4514b1377fd8844cd9793416131/Data"
        }
      ],
      "links":{
        "add_on_result":"https://api.twilio.com/2010-04-01/Accounts/AC597d02bf69de79f8f411f289ac211a0d/Recordings/REab85c1c2860471abf040b545c4e5aa40/AddOnResults/XRc82d4197bbc51d5982588ae367365a51",
        "payloads":"https://api.twilio.com/2010-04-01/Accounts/AC597d02bf69de79f8f411f289ac211a0d/Recordings/REab85c1c2860471abf040b545c4e5aa40/AddOnResults/XRc82d4197bbc51d5982588ae367365a51/Payloads",
        "recording":"https://api.twilio.com/2010-04-01/Accounts/AC597d02bf69de79f8f411f289ac211a0d/Recordings/REab85c1c2860471abf040b545c4e5aa40"
      }
    }
  }
}
{
  "user_token": "XRf951c9b0bce23930750991099e553b0a",
  "results":[
    {
      "results":[
        {
          "keywords_result":{},
          "alternatives":[
            {
            "timestamps":[
              ["I", 0.04, 0.28],
              ["love", 0.28, 0.71],
              ["coding", 0.74, 1.25]
            ],
            "confidence":0.782,
            "transcript":"I love coding "
            }
          ],
          "final":true
        }
      ],
      "result_index":0
    }
  ],
  "id":"b0e7a660-bcb6-11e6-8b1b-192a5f5dc12f",
  "event":"recognitions.completed_with_results"
}
SDK Version:
  • 7.x
import com.jayway.jsonpath.DocumentContext;
import com.jayway.jsonpath.JsonPath;
import com.jayway.jsonpath.ReadContext;
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;
import com.mashape.unirest.http.exceptions.UnirestException;
import com.twilio.twiml.*;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;

import java.util.List;
import java.util.Map;

public class RecordingController {

    private static final String ACCOUNT_SID = System.getenv("TWILIO_ACCOUNT_SID");
    private static final String AUTH_TOKEN = System.getenv("TWILIO_AUTH_TOKEN");

    @RequestMapping(value = "/voice",
            produces = "application/xml",
            method = {RequestMethod.GET, RequestMethod.POST})
    public ResponseEntity<String> voice() throws TwiMLException {
        // Start our TwiML response
        VoiceResponse voiceResponse = new VoiceResponse.Builder()
                .say(new Say
                        .Builder("Hi! I want to know what do you think about coding.")
                        .build())
                .record(new Record.Builder().maxLength(10).action("/recording").build())
                .hangup(new Hangup())
                .build();

        return okResponse(voiceResponse);
    }

    @RequestMapping(value = "/recording",
            produces = "application/xml",
            method = {RequestMethod.GET, RequestMethod.POST})
    public ResponseEntity<String> recording(@RequestParam("RecordingUrl") String recordingUrl) throws TwiMLException {
        System.out.println(recordingUrl);

        // Start our TwiML response
        VoiceResponse voiceResponse = new VoiceResponse.Builder()
                .say(new Say
                        .Builder("Thanks for howling... take a listen to what you howled.")
                        .build())
                .play(new Play.Builder(recordingUrl).build())
                .say(new Say.Builder("Goodbye.").build())
                .build();

        return okResponse(voiceResponse);
    }
    @RequestMapping(value = "/recording",
            produces = "application/xml",
            consumes = "application/json",
            method = {RequestMethod.POST, RequestMethod.PUT})
    public ResponseEntity<String> callback(@RequestParam("AddOns") String addOns) throws UnirestException {
        // If the Watson Speech to Text add-on found nothing, return immediately
        ReadContext addonsContext = JsonPath.parse(addOns);
        Map<String, Object> results = addonsContext.read("$.results");
        if(!results.containsKey("ibm_watson_speechtotext")) {
            return new ResponseEntity<>(
                    "Add Watson Speech to Text add-on in your Twilio console", HttpStatus.OK);
        }

        // Retrieve the Watson Speech to Text add-on results
        String payloadUrl = addonsContext.read("$.results['ibm_watson_speechtotext'].payload[0].url");
        HttpResponse<String> response = Unirest
                .get(payloadUrl)
                .basicAuth(ACCOUNT_SID, AUTH_TOKEN)
                .asString();

        DocumentContext speechToTextContext = JsonPath.parse(response.getBody());
        List<String> transcripts = speechToTextContext
                .read("$.results[0].results[*].alternatives[0].transcript");

        return new ResponseEntity<>(String.join("", transcripts), HttpStatus.OK);
    }

    private ResponseEntity<String> okResponse(VoiceResponse voiceResponse) throws TwiMLException {
        return new ResponseEntity<>(voiceResponse.toXml(), getHttpHeaders(), HttpStatus.OK);
    }

    private HttpHeaders getHttpHeaders() {
        final HttpHeaders httpHeaders = new HttpHeaders();
        httpHeaders.setContentType(MediaType.APPLICATION_XML);
        return httpHeaders;
    }
}