AaC Evolution

This is a continuation of previous article about Agile Architecture. I’m finally using the term Architecture As Code, as it seems to take traction in my community, and it is a fair statement since code usually means: Change Management, Verifiable, Testable, Automation, Continuos Integration, Continuos Deployment.

Since the last article, a couple of improvements have been added. I’m also happy to announce that after moving to a new company, the process shows good results again (I will elaborate more later).

The New Process

New Architecture Process

As shown in picture a couple of new building blocks have been added; CI Pipeline, Query DB, Documentation Repository. Also some tooling, and practices have been added: Asciidoc documentation, diagram convention and legends, meta data annotation…

I will go through these improvements one by one.

From Markdown to Asciidoc

Asciidoc is de facto standard for technical documentation, and is a single spec instead of multiple flavors. At the same time there is good compatibility with Git repositories and both Github and Gitlab will render content without any special configuration (or premium plugins). Find a comparison here.

Here is a list of main benefits:

• Cross references: Asciidoc allows to reference different pages around creating a similar experience as HTTP and the web. Having all content linked together encourages proper modularization and provides better browsing experience for the users.
• Keywords: Asciidoc allows to provide metadata that will categorize your content. This can be used to feed existing search engines (Confluence Label Search), or create custom ones ( check Query DB section later).
• Extensibility: Asciidoc delivers a pluggable solutions where you can hook custom tools in a predefined extension points ( pre-processor , tree processor, post processor,
• Exporting: Asciidoc has built-in support to export to HTML or PDF. It also provides extensibility to provide your own exporting solution, and the community has extensive support to all sort of formats already. Being able to export your architecture is essential to comply with customer enforcement processes.

Diagram Conventions

In order to enrich diagrams and be able to visually convey more information about the architecture and its details, a convention has been defined. This is leveraging Plantuml capability to define constants and provide legend.

diagram convention example

The convention is heavily influence by Telco Industry where traditionally solution have been separated both physically and logically into Management, Control and User Planes as explained in this article.

• Management Plane Links: This is using the dotted Plantuml line style. All links related to provisioning, configuration, monitoring, or not directly related to real control or user plane will be defined as Management.
• Control Plane Links: This is an established convention borrowed from Telco industry, where links related to control or signalling are using dashed lines to create a quick contrast over user plane links.
• User Plane Links: User/Data plane will use Bold Plantuml style to reflect the inherit high bandwidth of links in this category.
• Encryption in-transit: Links with encryption (IPSec, VPN, TLS, ALE..) will use red color to convey the information is being encrypted. Unencrypted links will use green color.
• Public Exposure: Elements exposed to public network such as Internet will be using a red square. This usually means that the solution should enforce proper protection on this demarcation points( firewall, anti DoS/spoofing…)

This convention can be defined in a separate Plantuml/text file that will be included in all diagrams:

!define public_expose red
!define encrypted red
!define unencrypted #77D245
!define user_plane bold
!define control_plane dashed
!define mng_plane dotted

legend bottom right
    Line Convention:
    PUBLIC EXPOSE public_expose
    ENCRYPTED encrypted
    UNENCRYPTED green
    USER PLANE user_plane
    CONTROL PLANE control_plane
    MNG PLANE mng_plane
end legend

The resulting diagram on text view looks like this:

@startuml
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
!include https://raw.githubusercontent.com/jaimecasero/jaimecasero.github.io/master/1nce_plantuml_skin.txt

title Level 1: System Context diagram
skinparam linetype ortho


System_Ext(iot_device, "IOT Device", $sprite="robot")
System_Ext(iot_app, "IOT APP", $sprite="robot2")
System_Ext(messaging_app, "Msg App", $sprite="robot2")

actor customer
System_Ext(customer_backend, "Customer Backend")
System_Ext(sim_manufacturer, "SIM Manufacturer")
System_Ext(sm_dp, "SM-DP", "RSP Data Preparation")
System_Ext(sm_sr, "SM-SR", "RSP Secure Routing")

cloud mobile_network {
}

cloud GRX_IPX

cloud ent_network
cloud internet

System( ng, "NG Platform", "Allows data connectivity and  SMS services for IoT devices")

iot_device -[#encrypted]-> mobile_network
mobile_network <-[#encrypted]-> GRX_IPX
GRX_IPX <-[#encrypted]-> ng: control,user[Diameter,SS7, GTP]
sim_manufacturer <<-[#encrypted;mng_plane]->> ng: input output [sFTP]

customer -[#encrypted;mng_plane]->> ng: prov,monitor [HTTPs]
customer_backend -[#encrypted;mng_plane]->> ng: prov,monitor [HTTPs]

ng -[#encrypted;user_plane]-> ent_network: data[IPSec,VPN]
ng -[user_plane]-> internet: data[IPv4]

ng <<-[#encrypted;mng_plane]->> sm_dp: ES2 [HTTPs]
ng <<-[#encrypted;mng_plane]->> sm_sr: ES4 [HTTPs]
ent_network -[#encrypted;user_plane]-> iot_app: iot_event[HTTPs]
internet -[#encrypted;user_plane]-> iot_app: iot_event[HTTPs]
internet <<-[#encrypted;control_plane]->> messaging_app: messaging [HTTPs]


@enduml

As we can see the usage of constants to specify the visual effect also provides useful meaning in the text view. This will be important during Merge/Pull request and review process, since the Git diff will be able to highlight difference if some link goes from un-encrypted to encrypted for example.

Diagram Look And Feel

To be honest UML, and PlantUML specifically, doesn’t provide nice diagrams by default. In the Cloud Era, the plain standard UML Icons are no longer appealing.

In order to improve the presentation of diagrams several improvements have been added.

Diagram Skin

This is a simple text file you can include in all your diagrams. You can use Plantuml Skinparam command to customize color and use your own company branding style. This file should be hosted in a public repository available through regular URL. Next snippet shows how the skin file looks like.

skinparam default {
    FontName SansSerif
    FontSize 17
    FontColor #007aff
    FontStyle bold
}

skinparam sequence {
    ArrowColor #77D245
    ActorBorderColor #194A7D
    ActorBackgroundColor #194A7D
}

skinparam interface {
    BorderColor #194A7D
    BackgroundColor #194A7D
}

C4 Model Icons

Thanks to Plantuml extensibility there is a nice C4 Model library available here. It will really improve the presentation of your diagrams.

C4 Model Icons showcase

AWS icons

There is also a very good AWS Plantuml lib available here. This will really boost you container level diagrams as shown in next picture.

aws icons showcase

Metadata Annotation

As explained in Asciidoc section, Keywords allow to inject metadata annotations for your documents. This also reduce the template free text content, delivering more focused content.

NFR Annotations

Each NFR is defined by using a set of metadata which establish the template:

• Level: MANDATORY, HIGHLY_RECOMMENDED, OPTIONAL. This is the requirement level.
• Risk: HIGH, MEDIUM, LOW . This is the risk associated to not complying with the NFR.
• Priority: HIGH, MEDIUM, LOW. This is the priority to implement the NFR in relation to other NFRs. This is useful for corresponding component team to plan their backlog for NFR compliance.
• Domain: SECURITY, OBSERVABILITY, PERFORMANCE, AVAILABILITY,MAINTAINABILITY, PORTABILITY, RELEASE, SCALABILITY. NFR are classified in different sets called domains

Next snippet shows how NFR is annotated with keywords.

= NFR-01-0001 Recoverability

:keywords: NFR_REQ, NFR_LEVEL_MANDATORY, NFR_RISK_HIGH, NFR_PRIORITY_HIGH, NFR_DOMAIN_AVAILABILITY, MILESTONE_FUT

== Meta Data

{keywords}

== Description

Service instances should be disposable and ready for accidental terminations.
Instances should guarantee fast startup procedure and hold stateless workloads.
Graceful shutdown/session transition to be handled outside of instance, for example: via logical state machine (queues, …), orchestration tool (K8s, …)
Specific stateful components in U-plane needs a workaround solution - to be defined in 2024/2025

Container Record Annotations

Container is anything that needs to be running in order to implement a component. This includes data stores, balancers, gateways, and service logic containers. Low level infrastructure such as VPC, Transit Gateway, NATs gateway are not included as part of this repository. Each container is defined by using a set of metadata which establish the template:

• Role: BALANCER GATEWAY SERVICE_LOGIC DATA_STORE. It helps to understand the main function of the container.
• Vendor: 1NCE AWS XXX. It tells whether the Container is delivered either in-house by 1nce, or AWS, or another third party.
• Infra Ownership: FULLY_MANAGED OWNED. It tells whether the container infrastructure is provided by 1NCE hence OWNED, or is provided by a third party.
• Public exposure: INTERNET_EXPOSURE GRX_IPX_EXPOSURE NO_EXPOSURE. It tells whether the container is exposed to some external network. This is a hint to evaluate NFR applicability.
• Deployment Region: CENTRAL LOCAL_BREAKOUT. It tells whether the component is designed to be deployed in the central EU region, or replicated in different regions for Local Breakout purposes.
• Module: Values are specific to solution. Identifies which module within the NG platform solution this container belongs to
• Component: Values are specific to solution. Identifies which component within the NG platform solution this container belongs to.

Next Snippet shows how Container Record is annotated with Keywords.

= API ALB
:keywords: CONTAINER_RECORD, MODULE_1CORE, COMPONENT_SDM, ROLE_BALANCER, VENDOR_AWS, EXPOSURE_NONE, REGION_CENTRAL, INFRA_FULLY_MANAGED

== Meta Data
{keywords}

* <<./container_level_2.adoc#, SDM Traffic Diagram>>
* <<./management_level_2.adoc#, SDM Management Diagram>>

== Description
AWS ALB to distribute load among api prov instances

CI Pipeline

The pipeline has mainly 5 stages; check, test, release, metadata, and publish.

• Check: For now simple lint over the commit message is implemented, ensuring Conventional Commit is honoured. This stage may hold other basic checks (NFR contain keyword, container record contain keyword…)
• Test: All Asciidocs are checked with Confluence Publisher Plugin. There is a Convert-Only/Dry-Run option that will check docs are ok (syntax, cross references, diagram includes…)
• Release: here the Semantic Versioning script is used to bump the version accordingly, and save it as a Tag in Git. Only on main branch
• Metadata: A set of simple bash scripts are extracting metadata from NFR and Container Records and upload them to the Query Database. Only on main branch.
• Publish: The Confluence Publisher plugin is used again to actually publish all the content into Confluence. Only on main branch.

The pipeline is configured with environment variables as shown in next picture:

Pipeline configuration

Query Database

This Database based on AWS DocumentDB/MongoDB allows run customizable queries over the metadata extracted from the Git repo. Both simple and aggregated queries are possible based on this simple Data Model:

architecture data model

The infrastructure is following the templated published by AWS here.

The Lambda function was modified to accommodate using the Git repo Tag version as the name of the DocumentDB logical database. This allows to save all the metadata of the different versions, and potentially run queries across different architecture versions.

Following query shows containers under given component which are provided in house:

https://p6arptw924.execute-api.eu-central-1.amazonaws.com/play/ngarch/latest/containers?filter=%7B%22keywords%22%3A%20%7B%22%24all%22%3A%5B%22COMPONENT_SDM%22%2C%22VENDOR_1NCE%22%5D%7D%7D

[
    {
        "_id": "6565c4956192d36044a0172e",
        "id": "diameter_proxy",
        "keywords": [
            "CONTAINER_RECORD",
            "MODULE_1CORE",
            "COMPONENT_SDM",
            "ROLE_GATEWAY",
            "VENDOR_1NCE",
            "EXPOSURE_NONE",
            "REGION_CENTRAL",
            "INFRA_OWNED"
        ],
        "nfrs": [
            {
                "nfr": "NFR-01-0001",
                "applicability": "APPLICABLE",
                "implementation": "PARTIALLY_IMPLEMENTED"
            },
            {
                "nfr": "NFR-01-0002",
                "applicability": "APPLICABLE",
                "implementation": "IMPLEMENTED"
            },

Predefined Reports

Now all metadata and NFR assessment information is in a database, it’s easy to run some queries and provide a couple of predefined reports per components. These reports help at understanding the current NFR gaps for a given component, considering every component is usually made up of tenths of containers, and it takes a lot of browsing to gather this information.

Next picture shows an example of report auto generated from database during the CI pipeline, and published as document in Confluence to be consumed by internal users (project managers, architects, devs…)

report example

Documentation Publishing

Architecture Documentation plays a major role in Delivery Organization to provide visibility of changes, and creates powerful knowledge base for Solution Designers, Product Manager/Owners, Developers to understand how the full solution works as a whole.

Thanks to the Confluence Publisher Plugin all the git repo is automatically converted to HTML and pushed to Confluence via API:

confluence publish example

Only the changed content is updated, and there is a correlation between the Confluence page history and the Git Tag:

confluence history correlates with git tags

Asciidoc Keywords are converted into Confluence Labels, and available in Search to filter content. The following query provides containers with public exposure under specific module:

https://confluence.atlassian.net/wiki/search?text=Description&spaces=1CP&ancestor=3924852831&type=page&labels=exposure_internet%2Cmodule_1cmp

confluence label search

1nce Journey

I joined 1nce 6 months ago. The Enterprise Architecture Team is formed by dedicated Solution Architects assigned to specific teams for support. At this moment there are 5 members with plans to grow 2 more positions in the short term.

The team is similar to the Architecture Guild with following role:

• The architects are converting customer needs and demands in to meaningful NFR definition for Delivery teams.
• Architects help Product Owners to define and prioritize backlog when NFR gaps are detected in the solution.
• Tracking the implementation status and updating documentation accordingly.
• Technology Selection and Alignment for incoming Customer demands.

I started to work on formalizing AaC repository with the team starting on May, and we manage to baseline half of the solution in 1 month. Then we managed to evaluate 13 ADRs, where 4 ADRS were actually merged into the main branch, and 7 have been already rejected.

There are activities to complete the baseline for the whole solution, and other internal products will follow.

The repository is visible across the organization from CTO and Portfolio Manager to Product Managers and Developers. Feedback has been captured to improve content ( diagram convention, per component reports, confluence publishing…). External consumers such as Customers and Providers have started to get in contact with the content providing very positive feedback, and realizing its value to ease technical discussions.

Conclusion And Next Steps

As shown in previous sections significant improvements has been delivered over the original approach towards a real As Code approach:

• CI Pipeline
• Documentation publishing
• API Exposure
• Predefined Reports

Although this iteration is major, I foresee much room for improvements yet, and so here is a list of ideas that we are planning to implement:

• Separate Links from Containers, and apply metadata annotation and NFR assessment as a completely separate entity.
• Implement NFR automatic assessment following CNCF approach, replacing manual evaluation as much as possible

All resources to start your own AaC repository can be found here.