Reading TAP 3.12 (ASN1) Files with Dataverse

When I first worked at Lavastorm, many years ago now, it was most commonly used for Revenue Assurance and Fraud Analytics in Telecommunications companies. Today, the same visual mapping of data processes allows analysts from all industries rapidly build data models and apply analytics to data from many disparate sources.

Revenue Assurance commonly requires that you acquire data as close to the source as possible, the objective being that you would identify any discrepancy in how business rules had been applied to the data as early as possible in the data value chain. The consequence being that downstream processes, including billing, would use this information and if errors existed this could result in a financial loss or gain, rework and/or impact customer experience.

That’s where Dataverse’s ability to read complex data types, and build your own complex data readers, can come in handy. It allows you to quickly take a vendor or standard specifications (such as the GSMA’s TAP specification for sharing data about roaming customers) and read the data, converting it into records.

Pre-requisites:

ASN.1

Abstract Syntax Notation 1 is a common format for Telecoms network vendors to use when creating files to log call information to be processed later. At a high level ASN.1 is like XML in that it is self describing and can contain tree link record structures. The main difference being that instead of pointy brackets it uses bits and bytes to describe the data. The specifications provided by the vendors usually consist of two key items: ASN.1 specification that describes the structure of the data and further details that describe additional encoding. In this article we focus primarily on decoding ASN.1 data structure rather than configuring each field type that can either be very simple or quite complex. This functionality is available and fully documented (http://www.lavastorm.com/assets/5.0-LDR-User-Manual.pdf).

Dataverse ASN.1 Nodes

Dataverse provides some quite powerful data acquisition nodes that will allow you to configure complex binary data as long as you have the specification (Data Reader). It also comes with some pre-configured “helpers” for complex data types like ASN.1 (e.g. ASN1 File – which we will demonstrate here, and Blocked ASN1 File, Convert ASN1, etc).

The ASN1 File node allows you to simply select the file, copy and paste the ASN1 specification and specify the top level element/data type to be able to extract all the records from the ASN1 file.

Step by Step Instructions:

  1. Start Up Dataverse and create a new project “Create -> Data Flow”
    • From the Import menu, select Data Flows or Nodes
  2. Drag a folder node onto the canvas and configure it to read from the directory your TAP files are in
  3. Select the transform node and connect it to the directory node, configure the “script” to add “emit ‘FileName’.str() as FileName” – currently need to convert from Unicode to String data type for the ASN1 File node.
  4. Search for the ASN1 File node, enter “asn” and then find and drag the ASN1 File node to the canvas and connect it to the transform node
  5. Now configure the ASN1 File node:
    1. Set “EncodingRules” to “BER” – most ASN1 Files will be BER encoding and the TAP files will use Basic Encoding Rules.
    2. Copy and paste the ASN1 Specification from the file TD.57_v32.3.pdf into the “Asn1Specification” – if the specification is well formed you can just cut and paste, but typically there will be some minor issues to address (like I had to with the TAP 3.12 specification that has the field lengths for some fields defined after the actual data type with a “- (SIZE (0-10))” where this is not actually part of the specification and either needs to be removed or have the dash placed with two dashes to ensure they are read as comments)
    3. Specify the “TopLevelType”, set it to “DataInterChange” – this is usually a field/record type that is at the top level of the data structure and contains all the sub records that you will want to keep or connect together
    4. Set the “FileNameFieldExpr” to “FileName” as we have defined earlier from the transform node
    5. You should be able to leave all the other nodes to their default values
  6. Run the three nodes to be presented with the output
    1. If the node errors, it will most likely refer to a line in the Asn1Specification parameter and you should look up the line in question review and adjust until you no longer get any more errors – I used this approach to find all the “- (SIZE(1-10))” values and comment them out earlier. You should not have this issue if you are using the pre-formatted text file
  7. Inspect the output
    1. The output represents the different records and sub-records that must still be stitched together depending on your requirements. One of the benefits of this approach is that you only need to join the records and sub-records required for your analysis.
    2. In the example I’ve provided at the end, I’ve filtered to identify the “record_transferBatch_callEventDetails_mobileOriginatedCall” and identified the fields that are not null to demonstrate the output

I’ve put this together to provide a quick overview of the functionality provided by Dataverse because it quickly allows you to inspect your ASN1 data. There are still however two steps to complete to finish the reader:

  1. Join the records and sub-records together, and
  2. Optionally now that you’ve inspected the data use a similar approach to generate a Data Reader Input XML and Data Reader Output XML that can be enriched to optionally read some of the additional data that is not encoded using basic ASN1 types. This is a much more complicated process and I recommend ensuring you can’t just manage with what you have before you proceed down this path. If you do, reach out to me at [email protected] to see if we can help.

Download the completed Dataverse Data Flow and validate your roaming data (change file extension from zip to lna after downloading).

Leave a Reply