The legacy challenge…

Legacy mainframe applications are usually excluded when it comes to DevOps transformation. After all, it’s a daunting task to modernize millions of lines of code written in the 1970s to the present day while the system is still monolithic. Avoiding these applications, though, can significantly hamper the effectiveness of overall IT delivery efficiency. What does it take to implement DevOps for the mainframe so that it can mesh with the rest of the DevOps-enabled environments in the organization and help deliver better business results?

This was the challenge facing American with their Flight Operating System (FOS) application, a mission-critical TPF mainframe system that manages the airline’s flight operations and crew management.

Embracing digital innovation and acquiring the ability to respond quickly to market demand are key IT objectives for most organizations in the race to create and sustain competitive advantage. American’s IT program is guided by Tech 2.0 goals to achieve these objectives, including “Transforming the way IT is delivered” and “Ensuring the systems are reliable.” The state of DevOps metrics is being used as the barometer for product teams to determine whether American is delivering value faster and improving quality.

Many organizations are looking to modernize their mainframes. That said, despite the availability of DevOps tools on the mainframe, application developers struggle with compliance processes, lack of application modularity, manual processes and inconsistent development times. These challenges are hampering innovation.

While mainframe development has lagged behind in the adoption of DevOps practices, we don’t see the mainframe disappearing altogether. The mainframe is still a staple of many industries, outperforming cloud solutions in reliability, scalability and processing speed. Within the airline industry alone, mainframes manage 4 billion passenger flights each year across the globe. The implications of moving the mainframe into the DevOps space can have a significant impact on the airline industry’s ability to deliver faster for its customers. A variety of tools focus on the z/OS mainframe but little has been done in the z/TPF mainframe space.

At American, the FOS applications on TPF mainframe had not embraced DevOps and had become siloed and left out of modern tooling and technologies. From a talent perspective, the TPF mainframe relied on a few niche developers, and only had limited appeal to new developers, who saw it as an antiquated environment that was difficult to learn. Instead they expected continuous and distributed development and continuous integration. To transform development practices, DXC Technology and American worked together to introduce DevOps into FOS and to train FOS users to embrace distributed source code in a five-stage process, as follows:

1. Confirm the scope.

While modern applications could adapt to DevOps transformation, the applications running on the TPF mainframe continued to be supported with traditional development methods, primarily due to these reasons:

👉 There were (and are) no ready-made or off-the-shelf DevOps solutions for the TPF environment. TPF is a mission-critical, special and unique operating system. It morphed from what was originally called Airline Control Program, a software package written specifically for airlines in the 1960s. It continues to be a central solution for many airlines, global distribution systems (GDS), credit card companies, hotels and train systems. It was a proprietary solution with fewer options in the DevOps solution space than the more common z/OS mainframe environment.

👉 Principally, mainframe systems are averse to frequent changes, and the TPF environment is no different. However, the core principles of DevOps are about embracing frequent changes and collaborative teams.

Despite these challenges, DXC teams ensured that service delivery continued to remain at expected levels for the highly critical, high-availability FOS applications.

The software development life cycle in a mainframe environment has historically been handled through programmer discipline and agile practices. If we could supplement this with DevOps — which improves quality at speed — we could then build a culture of trust, leading to an acceptance of frequent changes. It must be noted, however, that “speed” in a mainframe environment does not mean releasing code 20 times a day. Even one release per week is a perfect vision for this environment.

Changes of this magnitude include not only the tools used but also the way we think about building and deploying software. One major shift is changing the view of the mainframe and the legacy development processes. Historically, mainframe development and deployment processes were considered inseparable. If you were developing on a mainframe, you had to use a specific set of tools in a certain way, and it could not be changed without major intervention. In the shift to DevOps, we are separating out development and deployment processes (and the associated code) from the actual mainframe infrastructure. As we decouple the code, build scripts and deploy scripts from the infrastructure, we can start to do what distributed systems have been able to do for years in the realm of DevOps.

It is vital to be clear from the outset about what you are looking to achieve with DevOps in the mainframe, why it matters to your enterprise and how you can be sure that it will deliver. In this instance, the immediate scope was to determine how we could bring a DevOps toolchain into the FOS environment.

2. Characterize the problem.

The following challenges were uncovered during the American hangar session (more commonly referred to as a DevOps Dojo) for one squad working on the FOS system:

👎 The code base for FOS applications is hosted on an environment with restrictive access. Only DXC programmers had access to the program files — not the architect or coaches, nor American team members. It was hard for everyone to understand what was going on with the code, forcing dependency on the programmers.

👎 Code is hosted in a centralized Concurrent Versions System (CVS) repository — a source control system that would not work with pipeline tools or support decentralized code development.

👎 Code changes could not easily be merged between squad members or other squads.

👎 Code integration and version conflicts were handled through developer discipline, often late in the development cycle.

👎 All tests had to live in a separate code repository.

3. Architect and implement the solution.

As a next step, we proceeded through a proof of concept to ascertain the art of the possible. The DXC Modernized Apps and DevOps offering guides users in designing the DevOps tool chain for the mainframe environment (Figure 1). The project initially focused on distributed code control, which is the most impactful of all the treatments.

Figure 1. DevOps tool chain for the mainframe environment

FOS applications were hosted in the zTPF environment, version control of the source code was handled through CVS hosted in the zo’s environment, and development took place in the zLinux environment using the TPF toolkit. The high-level architectural change that DXC introduced was to include the American enterprise GitHub that would host the zTPF source code and enable distributed development (Figure 2).

Figure 2. American’s high-level DevOps architectural diagram  

4. Validate the results.

The critical success factors were to ensure:

✔️ Feasibility of managing code integration in GitHub

The requirement was narrowed down to a specific functionality within FOS, and the respective set of programs alone was moved from CVS to the American Enterprise GitHub. The typical code development and integration use cases were tested using the source code in the GitHub and found to pass without any major challenges.

✔️ Feasibility of ensuring that the production promotion cycle is not impacted

To avoid complicating and thereby risking the production promotion cycle, the process of having the source code in the CVS repository was retained. So, when the code is ready for deployment, it will be moved to the CVS repository via the TPF toolkit, and the current process of production promotion will take over.

With the successful validation of the proof of concept, we now have a minimum viable product that could be planned for a rollout.

5. Next steps: Scale up the proof of concept.

Several fundamental changes needed to be in place before we could roll out the solutions. Adding tools does not always solve problems. It can instead complicate the existing development process. A fundamental change in the mindset and practices of software development must be implemented before the introduction of new tools. A rollout would have to be a series of planning steps that need to be driven cohesively.

1. Socialization of the solution

Hackathons, sandbox experiments, roadshows and solution peer reviews are typically the best approaches to ensure that the solution is socialized, validated and continuously improved.

2. Learning journey

Without training, the new methodologies and tools will fail to realize the promise of faster and better software deliveries. The change could be extremely complex and perplexing for developers who are used to traditional software development life cycles. American and DXC are investing in training and coaching staff throughout the transition period — from high-level concepts to implementation specifics. This approach is creating opportunities to cross-train resources in DevOps concepts, tools and solutions.

3. Production implementation

Once we have removed the barriers of technology, culture and skills gap, we are ready to move the solution to production.  

In closing

DevOps is fundamentally changing and improving how development and operations are done today. The initial change of embracing distributed version code will set the stage for embracing further DevOps “treatments.” We consciously commenced with the source code handling of FOS applications to make small, meaningful changes around continuous integration. We then measured the results and continued the iterations until we could drive greater automation in the development cycle. It may take some effort to integrate the Dev and Ops processes, but eventually it will help create an agile and scalable system that is ready for rapid change and growth.

DevOps team

Penny Appleton | Hari Sathya | Javier Rodriguez | John Elgin | Lloyd Rabalais |

James Rutherford | Nikhil Singhal | Ahmed Riaz | Prateek Gupta | Mehul Sutaria |