Felix Tegelkamp

In this ongoing project, we are actively working on modernizing and optimizing our AWS infrastructure, refining DevOps processes, and incorporating cutting-edge technologies to enhance the efficiency of our .NET applications. The project focuses on utilizing AWS CDK (Cloud Development Kit) and Terraform for infrastructure provisioning and updates, implementing streamlined DevOps pipelines, containerizing existing workloads, and designing new data workflows using AWS Lambda, Step Functions, and Athena.

Key Objectives:

Infrastructure Modernization:

We leverage AWS CDK and Terraform to define and provision infrastructure as code (IaC).

Our version-controlled templates ensure reproducibility and scalability of infrastructure components.

DevOps Pipeline Optimization:

We have established robust CI/CD pipelines for our .NET applications, automating build, test, and deployment processes.

Containerization of Workloads:

Existing .NET applications are containerized using Docker, providing a consistent environment across development, testing, and production.

Containers are orchestrated using Amazon ECS for improved scalability and resource utilization.

Serverless Architecture for Data Workflows:

We design and implement serverless data workflows using AWS Lambda for event-driven processing.

AWS Step Functions are utilized to orchestrate complex workflows, ensuring reliability and maintainability.

AWS Athena is implemented for ad-hoc querying of data stored in Amazon S3 within Step Function workflows, providing a serverless and cost-effective solution for data analysis.

Data storage and retrieval mechanisms are optimized to enhance query performance.

In August 2022, a renowned energy company embarked on a journey to transform its legacy systems and processes. They approached us with the vision to automate their on-premise workloads, enhance security, and provide real-time, comprehensive system monitoring.

As a Cloud Solutions Architect, we began this transformation by creating a Landing Zone architecture in Azure using the hub and spoke model with Terraform as Infrastructure as Code (IaC) tool. This was based on Microsoft Best Practices (Cloud Adoption Framework), with the hub hosting shared Azure services accessible to workloads hosted on the spokes. The hub network also became the pivotal node for cross-premise connections.

One of our key successes was automating their software development life cycle. We implemented robust CI/CD pipelines that automated unit and integration testing, static code analysis, package auditing, and deployment to different stages upon manual confirmation.

The next critical step was to replace on-premise workloads with serverless cloud applications. We leveraged Azure Functions (with c# and .Net6 Framework) and Logic Apps to achieve this. As a result, the system could automatically check and process email terminations and feed data into downstream on-premise systems.

To ensure thorough monitoring and logging, we utilized Azure Monitor, Application Insights, Log Analytics, and Diagnostic Settings. We were able to log platform logs for all VNets and the Virtual Gateway into a centralized Log Analytics Workspace. Additionally, we configured Application Insights for every Azure Function and Logic App, providing essential metrics such as availability and consumption.

Moreover, we established alarms for the Logic App to notify about unsuccessful workflow runs, providing real-time updates on any system issues.

With technologies like Azure Virtual Gateway, Azure Vnets, Azure Private Resolver, Azure Firewall, Azure DevOps, Azure Functions, Azure Logic Apps, Azure API Management, Azure Active Directory, Azure Private Zones, Azure DNS Forwarding Rulesets, c#, .Net 6, the project was a great success.

The company now enjoys a higher degree of automation, robust security, and continuous monitoring that optimizes their operations and strengthens their foothold in the energy sector. The successful collaboration with us resulted in significant operational efficiencies, a step towards an even more sustainable future.

A public sector client was facing challenges with efficiently scaling their IT infrastructure and ensuring high-level security across their management and regional networks. The existing infrastructure lacked the agility to cater to different administrative units while meeting their unique security demands.

Solution:

Hired as a Cloud Solutions Architect, we undertook a transformative journey to address these concerns and optimize the IT landscape. Here’s how:

1. Landing Zone Architecture:

Developed a robust architecture in AWS using the Transit Gateway as a central hub. This design facilitated the scalable connection of various administrative and regional networks, emphasizing the need for cryptographically secure connections beyond TLS1.2.

2. Application Architecture:

Crafted a flexible application architecture allowing tenants to utilize either shared or dedicated services. This dynamic design ensured that the individual security requirements of various units were met, in line with standards set by the BMI.

3. Infrastructure Automation:

Leveraged Terraform for the automated provisioning of infrastructure for different tenants. This encompassed setting up a Kubernetes cluster, deploying an Istio Service Mesh using Helm Charts, database instance provisions, and even streamlined the management of database user credentials within Kubernetes namespaces.

4. CI/CD Pipelines:

Integrated CI/CD pipelines through GitLab and Artifactory. This led to the efficient deployment of containerized applications built on Spring Boot into the Kubernetes cluster. With features like automated unit and integration tests, image scans with Aqua Trivy, and dedicated release pipelines, development cycles were optimized, and software quality was ensured.

5. Logging and Monitoring:

Implemented a holistic logging system with the ELK Stack and comprehensive monitoring using the Prometheus/Grafana stack. With autoscaling at both pod and node levels based on resource metrics, the system’s performance was consistently maintained at optimum levels. Crucial metrics, such as resource consumption and alarm thresholds, were constantly monitored, ensuring quick detection and resolution of potential issues.

Technologies Used:

Gitlab, Terraform, Kubernetes, Helm, Artifactory, Java Spring Boot, PostgreSQL, ElasticSearch, Kibana, Logstash, Prometheus, Grafana.

Outcome:

Thanks to these state-of-the-art solutions, the public sector client now benefits from a scalable, highly secure, and agile IT infrastructure. The flexibility of the new system means that various administrative units can operate seamlessly, with each unit’s unique requirements catered for. Automated processes have not only reduced manual intervention but have also significantly improved efficiency and reliability. With advanced logging and monitoring, system health is constantly under watch, reducing downtimes and ensuring a smooth user experience.

Feedback:

The client expressed immense satisfaction with the transformation. The agility, security, and efficiency brought by these changes have immensely benefited their operations. They acknowledged that the architectural decisions made were pivotal in meeting their unique needs and ensuring sustainable growth.