Developer Experience at American Airlines

A Product Technical Lead Perspective

American Airlines has been heading down the path of a technology transformation. Various groups have identified problems across American related to how our developers work on a daily basis and during the pandemic, unique opportunities arose to make the experiences easier for our application teams.

How it started

Like many large companies, American has a typical IT Service Management tool where teams go in and enter information for service requests. The teams then wait for the submitted requests to be manually fulfilled. These service requests are not workflows but usually one request being performed at a time. In the end, launching an application often entailed waiting months to get the proper infrastructure in place following numerous service requests.

Many teams found a need to build portals to collect information from customers and automate processes with the collected information. The IT Service Management tool has these capabilities, but for many reasons, teams found the need to build their own. We had several groups working on the same problem for their own use creating fragmentation and duplication of efforts, which is not a good use of resources and is not a sustainable model.

We had competing products attempting to be the frontdoor to IT services. Between home grown solutions and vendor provided solutions, it was not always clear what web portal to enter service requests into. We did not want to abandon any work that had already been done; so our objective was to instead piece together and create abstraction layers where the tooling behind the scenes no longer mattered.

In addition to service requests, many teams also had their own way of creating apps with no shared patterns or knowledge among organizations. Our coaching group created CLI tools that they used during engagements to help create these patterns, but this was not a scalable solution to the problem. The coaching group desired a way to take the guess work out of what the approved patterns are and combine required service requests into a single workflow.

We wanted the create app workflow to make it easy on our application teams to bootstrap new apps and takes the guess work out of approved patterns while providing batteries out of the box!

This is where a group of developers turned to the InnerSource model to help us grow our platform internally and work with groups to contribute to a single goal.

What we chose

During the early days of the pandemic, our coaching team was unable to engage with customers as they do face-to-face, so time opened up for their team to experiment with Spotify’s Backstage. Due to leadership changes and an emphasis on automation from our Chief Information Officer (CIO), open source became a key operating objective. More time combined with leadership support of open source allowed the team to explore the possibilities of the Backstage ecosystem.

Backstage was discovered after its initial alpha launch and quickly utilized by a team to support American’s IT. We looked to the open source project to provide a solid base for an enterprise framework that we could build upon. Backstage brought the catalog, which American has been looking for a solution for quite some time as well as a standard way of blueprinting application code.

Backstage was branded as Runway at American Airlines, and a full-time team was built around it as the experiment continued.

Where we are at

Contributions and InnerSource

The Runway product is expanding constantly. We have teams contributing from across American Airline’s IT through ideas, requests, and development. We grew from a small handful of developers to multiple squads assigned to the project.

The core development of the Runway platform is handled by the squads assigned to our Developer Experience product and their responsibilities include training and support to other contributing squads. We have groups that have never coded before the start of the journey to providing self-service to their customers, and we have seen these groups picking up the platform quickly. For contributing squads, we encourage microservice development to power the backend of the Runway platform which come in many flavors.

As Runway continues to expand, teams are moving their antiquated service request forms out of the swivel-chair mindset into automated requests and eventually workflows to reduce time to delivery as well as effort required.

Create App

While we have automated service requests into workflows, the ultimate goal of the Developer Experience platform at American is ‘Create App’, a core offering as part of the Backstage platform. Spotify’s offering for ‘Create App’ fulfilled our coaching organizations’ desire to provide a standard library of templates for our developers to choose from.

Over the years our coaching organization has come up with command line tools to fill the gap of blueprints, however it never felt super easy and straight forward to use and it was primarily utilized during coaching engagements. Now through our developer experience platform, our engineers have a simple way to utilize blueprints through their browser.

Now our app teams have the ability to launch applications to the public internet with security and observability within just 6 minutes; a process that used to take months.

Moving along

We have spent months automating the infrastructure creation and bootstrapping behind the scenes to get Kubernetes clusters from Terraform files to fully bootstrapped clusters in under 20 minutes using GitHub and GitHub Actions. We have worked across organizations to gain input and InnerSource our contributions, so other teams can create clusters easily. While Developer Experience would like teams to use shared clusters for many reasons, some teams may want their own clusters so we are looking to make an easy-to-use blueprint to support those teams as well as Bring Your Own Cluster to Runway.

In addition to the automation of cluster work, app teams can click into our developer portal and have a running app with a public ingress in under six minutes; batteries included. Our app teams at this time have monitoring of their app and infrastructure, SSL certificates, Kuma service mesh, ability to access namespaces through Rancher, a GitHub repo and workflows with unit testing, global traffic management through our content delivery network and security provider, and logging. We want app teams to focus on their coding instead of putting time into infrastructure needs.

Challenges

We have faced many challenges as a group, but our journey continues down the runway.

Early Adopters

Backstage was released as a public FOSS offering on March 16th, 2020, and American adopted the project on 0.1.1-alpha.4 on May 1st, 2020. Design changes occurred in the Backstage ecosystem causing technical debt to accumulate.

At one point, due to changes in direction with Backstage, we questioned whether it was worth it as a team to continue with Backstage and laid out options to abandon ship. Due to the overall positives, we continued with Backstage.

Backstage specific skillsets

Typescript and React are not universally known at American, as we have chosen other languages and frameworks in the past. Being an airline, we accumulate a lot of legacy practices and technology, so Backstage was very different from what our teams were used to.

Project to Product Mindset

With new leadership and the pandemic hitting, our strategic direction and mindset shifted. Teams were reorganized to make automation a priority and there was a renewed focus to build up skillsets. Our team members grew in new ways that prepared them for a future of less shuffling of paperwork and instead empowering teams through self-service automation. We’ve seen groups who have never programmed before become excited to provide new offerings to their customers.

Snowflakes

One of the originally described problems was patterns, as identified by our coaching organization. Our application teams have the freedom to build how they want, but that causes variations on the ways we build our applications. By using Runway blueprints, a Backstage term, our application teams can work from a similar base and build from there.

Infrastructure Offerings

One of our strategic goals in the next year is multi-cloud and to accomplish this, Kubernetes was selected as our primary orchestration engine for application workloads. This departs from previous messaging where many of our teams were told Platform as a Service such as App Services was our future. Kubernetes allows our apps to be fluid in their placement with our cloud partners, while App Services is stuck in Azure. We are not forcing a shift for teams, rather we are making it extremely easy to utilize our offerings. It will take time for application teams to warm up to our offerings as well as time for those offerings to mature. We are working closely with a few teams to enhance our offerings while they utilize them.

Future

Teams continue to find value in our InnerSource platform. From teams building in access control for Data Lake to the creation of new Virtual Machines, every team can benefit in building within the Runway platform. These teams will continue with the Minimal Viable Product (MVP) approach to deliver value in small bite-sized pieces which then can be worked into larger workflows.

We will continue to grow and mature our offerings for application teams as this is just the start. With Create App 2.0, we are looking to provide teams with monorepos for their frontend, backend, and supporting infrastructure. Our users can click to get a frontend, backend, and database or other needed services all launched within minutes and ready to use. We will continue improving our devkit where our InnerSource Runway developers can utilize shared complex functionality through small blocks of code. The message of Runway will continue to make its way through our teams and workshops on using Runway to launch applications to our clusters will start.

With a flexible, solid, and enterprise-ready base that Backstage provides, we can continue integrating backends together to abstract complexities so that developers can focus on their application code. We look forward to continuing the development of Runway.