APIGateway Apache VTL Mapping Templates

I guess this article will come in handy for those writing one of these templates for the first time. You will find plenty of documentation about Apache VTL , and some examples. But those are pretty basic. So this article is focused on a particular use case where we need to transform a generic response including JSON body into XML.

var mappingTemplate = require('api-gateway-mapping-template')

var vtl = `
#set($cdrMap = $input.path('$'))
#set($accountSidPos = $context.resourcePath.indexOf('{AccountSid}') + 13)
#set($sidPos = $context.resourcePath.indexOf('{Sid}') - 2)
#set($cdrXmlRoot = $context.resourcePath.substring($accountSidPos, $sidPos))
<RestcommResponse>
   <$cdrXmlRoot>
    #foreach($cdrFieldKey in $cdrMap.keySet())
       #set($cdrField = $cdrMap.get($cdrFieldKey))
       #set($cdrFieldKeyArray = $cdrFieldKey.split("\_"))
       ## Single line becuase of VTL loop/set restrictions.**
       <#foreach($cdrFieldKeyPart in $cdrFieldKeyArray)#set($firstLetter = $cdrFieldKeyPart.charAt(0).toUpperCase())#set($fixedCase = "$firstLetter$cdrFieldKeyPart.substring(1)")$fixedCase#end>
       #if ($cdrField.toString() != "[object Object]")
          $cdrField
       #else
          #foreach($subFieldKey in $cdrField.keySet())
             #set($subField = $cdrField.get($subFieldKey))
             #set($subFieldKeyArray = $subFieldKey.split("\_"))
             ##skip the eval function included in the map.**
             #if ($subFieldKey != "eval")
             <#foreach($subFieldKeyPart in $subFieldKeyArray)#set($firstLetter = $subFieldKeyPart.charAt(0).toUpperCase())#set($fixedCase = "$firstLetter$subFieldKeyPart.substring(1)")$fixedCase#end>
                $subField
             </#foreach($subFieldKeyPart in $subFieldKeyArray)#set($firstLetter = $subFieldKeyPart.charAt(0).toUpperCase())#set($fixedCase = "$firstLetter$subFieldKeyPart.substring(1)")$fixedCase#end>
             #end
          #end
       #end
       </#foreach($cdrFieldKeyPart in $cdrFieldKeyArray)#set($firstLetter = $cdrFieldKeyPart.charAt(0).toUpperCase())#set($fixedCase = "$firstLetter$cdrFieldKeyPart.substring(1)")$fixedCase#end>

    #end
   </$cdrXmlRoot>
</RestcommResponse>
`;

var payload = '{"sid":"IDbb44ef8c2ebf4d4599d0378a2a8a2ea8-CAd9e08ede8b604e739ba7d0bd477fdf49","instance_id":"IDbb44ef8c2ebf4d4599d0378a2a8a2ea8","parent_call_sid":"IDbb44ef8c2ebf4d4599d0378a2a8a2ea8-CAd9e08ede8b604e739ba7d0bd477fdf50","conference_sid":"CF01652ab2101c469c98d49eb65493c4c5","date_created":"Mon, 12 Oct 2020 17:12:47 +0000","date_updated":"Mon, 12 Oct 2020 17:12:47 +0000","account_sid":"ACecbe82bfd2609b6b809aa741a4879c2d","to":"11118801","from":"22228801","phone_number_sid":"ACecbe82bfd2609b6b809aa741a4879c2d","status":"completed","start_time":"2020-10-12T17:12:47.553Z","end_time":"2020-10-12T17:27:47.553Z","duration":893,"ring_duration":7,"direction":"inbound","answered_by":"Ans-By","api_version":"2012-04-24","forwarded_from":"Forwarded-From","caller_name":"CallerName-Unknown","uri":"/2012-04-24/Accounts/ACecbe82bfd2609b6b809aa741a4879c2d/Calls/IDbb44ef8c2ebf4d4599d0378a2a8a2ea8-CAd9e08ede8b604e739ba7d0bd477fdf49","subresource_uris":{"recordings":"/2012-04-24/Accounts/ACecbe82bfd2609b6b809aa741a4879c2d/Calls/IDbb44ef8c2ebf4d4599d0378a2a8a2ea8-CAd9e08ede8b604e739ba7d0bd477fdf49/Recordings","notifications":"/2012-04-24/Accounts/ACecbe82bfd2609b6b809aa741a4879c2d/Calls/IDbb44ef8c2ebf4d4599d0378a2a8a2ea8-CAd9e08ede8b604e739ba7d0bd477fdf49/Notifications"}}';
var context = {"resourcePath" : "/api/2012-04-24/Accounts/{AccountSid}/Calls/{Sid}"};
var params = {"path": {}, "queryString":{"jander" : "clander"}, "header" : {}};

var result = mappingTemplate({template: vtl, payload: payload, context: context, params:params})
console.dir(result);

So, that’s the whole template including a testing environment based on this unofficial tool I found. Notice this template is quite generic and has little details about the input or output models. This is done on purpose, as opposed to most of the examples you will find out there, where the data model is hard coded in the template. The reason is that our particular model represents a Telco CDR that is expected to be enriched over and over again. We don’t want to hard code the CDR model in the template too much, otherwise we will need to revisit the template every time we add a new field. Also, we have plenty of different CDR types, with slightly different fields.

So, Let’s go step by step. My template is expecting a JSON body from a Lambda function. As you can see we start the template engine with the payload the lambda function is supposed to print, and the context. This is simulating the same runtime environment we will have in the actual APIGateway.

var result = mappingTemplate({template: vtl, payload: payload, context: context, params:params})

The first Set directive will load the whole JSON body into a VTL Map called “cdrMap”. Is based on APIGateway standard template functions. The ‘$’ is a JSON path expression that will map the whole JSON body.

#set($cdrMap = $input.path('$'))

Next directives will figure out the XML root element of the model based on the input request path. This is based in a convention we established in our own API. We use some java String methods to extract the label from the path based on some Path Parameters the APIGateway has defined.

#set($accountSidPos = $context.resourcePath.indexOf('{AccountSid}') + 13)
#set($sidPos = $context.resourcePath.indexOf('{Sid}') - 2)
#set($cdrXmlRoot = $context.resourcePath.substring($accountSidPos, $sidPos))

Then we start looping over the JSON structure. This is the most interesting part. We use foreach directive over the map we got initially.

#foreach($cdrFieldKey in $cdrMap.keySet())
       #set($cdrField = $cdrMap.get($cdrFieldKey))

In our particular JSON to XML transformation we need to convert JSON snake case field names, to Pascal case format. For example, a JSON field called “phone_number_sid” is expected to be named “PhoneNumberSid” in XML. For this we use String methods again. This part looks and feels quite hacky, and it s. VTL loops and set directives have some restrictions, so we need to be careful on how we use them. First we split the orginal JSON field name by the “_” separator. Then we create a foreach loop to do the case translation. We take first later from each part, and turn it to upper case.

#set($cdrFieldKeyArray = $cdrFieldKey.split("\_"))
## Single line becuase of VTL loop/set restrictions.**
<#foreach($cdrFieldKeyPart in $cdrFieldKeyArray)#set($firstLetter = $cdrFieldKeyPart.charAt(0).toUpperCase())#set($fixedCase = "$firstLetter$cdrFieldKeyPart.substring(1)")$fixedCase#end>

Another hack I found was to check whether the JSON field was a simple type, or yet another Map I have to process to create the nested structure. This condition seems to work as I expected to find whether the current JSON field is a Map or not. If it’s not a Map, then we simply print the field value. Otherwise we go into a new foreach loop level. I must say this is the only trick I found working, in spite of trying all the threads I found based on checking the Java class, which were not working properly.

#if ($cdrField.toString() != "[object Object]")

The foreach loop for the JSON nested element is just a copy of the parent one. The reason is copy pasted is because APIGateway doesn’t seem to support VTL macros, that could otherwise encapsulate this logic in a single block, and potentially support any nesting level through recursion. Unfortunately this is not supported, and this template only supports one level of nesting, which it’s fine in this particular example, but will be hard to maintain if your JSON structure have more intricate hierarchies.

#foreach($subFieldKey in $cdrField.keySet())
   #set($subField = $cdrField.get($subFieldKey))
   #set($subFieldKeyArray = $subFieldKey.split("\_"))
   ##skip the eval function included in the map.**
   #if ($subFieldKey != "eval")
   <#foreach($subFieldKeyPart in $subFieldKeyArray)#set($firstLetter = $subFieldKeyPart.charAt(0).toUpperCase())#set($fixedCase = "$firstLetter$subFieldKeyPart.substring(1)")$fixedCase#end>
      $subField
   </#foreach($subFieldKeyPart in $subFieldKeyArray)#set($firstLetter = $subFieldKeyPart.charAt(0).toUpperCase())#set($fixedCase = "$firstLetter$subFieldKeyPart.substring(1)")$fixedCase#end>
   #end
#end

And that’s it for now. I hope this usecase provide you more advanced and meaningful example on how to use Amazon APIGateway to map response.