Agile Project Management in an SOA Strategy
Can agile project management help in an SOA strategy? In this post we will analyze this issue, in my opinion one of the main forgotten issues when we approach an ICT strategy based on SOA.
Tabla de Contenidos
Project management is a key aspect to projects’ success, this is obvious. Their correct planning, in time and in human and technical resources, as well as a good monitoring of the project, risk management, etc., are vital for our project to reach its objectives.
What I do not find so obvious, or at least I do not find much debate about it, is how project management must adapt to a Service Oriented strategy. Although it’s not only a question of adapting: it is also to take advantage of the benefits it brings.
We will begin by clearly situating the context: we are implementing a Service Orientation strategy in our organization and, after a successful pilot (not without difficulties), we have an Initial Service Catalog, a published SOA Governance Policy, a Governance Office leading the Strategy, and an Interoperability Office responsible for the development and maintenance of the Service Catalog, and the deployment and monitoring of the infrastructure.
We are therefore on the verge of extending the strategy to the rest of the information ecosystem. Beware, dangerous curves ahead.
We have already talked several times about the pillars of an SOA strategy: Governance, Low Coupling, Events vs. request / response (ie asynchronous messaging vs. synchronous messaging), Service Contracts (Services Catalog), Infrastructure (sizing, configuration and Monitoring), and the rest of the SOA principles recognized in the industry (Service Autonomy, Service Composition, Service Reusability, Service Abstraction and Service Discoverability).
Deploying an SOA strategy in an organization is something like tackling a “project of projects”. Actually this is a familiar concept in ICT management for decades, because any organization that has a suitable degree of maturity in this area, key to its objectives, must periodically manage a Project Plan.
But the “project of projects” that is adopting an SOA strategy implies something more. Because these projects require coordination and hierarchical management under the governance office that leads the strategy. In an SOA strategy, it is very rare to find a project that only affects one system, one team, one supplier company and one project manager. Usually, at least two, and quite often, more than two are involved.
An SOA strategy involves changing some well-established patterns in the practices of many teams and companies, and this is neither immediate nor easy. Many of the projects will not be evolutionary from a functional point of view, but of SOA reengineering, that is to say: to adapt the current systems with their current functionalities to the new strategy.
At the same time, the organization is also developing lines of action in the areas of communications, servers, training, process improvement, attention to incidents, users support, etc., etc.
Typically, there is a higher level department to which the status of projects must be reported, especially key projects, or the return of some investment, or the result of some management that affects certain systems or equipments.
You may be thinking that effective ICT governance has always been responsible for coordinating and overseeing an organization’s entire Project Plan, which is true. But they have traditionally been isolated projects that affect a single system and software development team (in addition to other cross-cutting areas related to infrastructures, communications, etc.), and have usually been undertaken independently, raising follow-up reports to those responsible of the Project Plan, that allowed to manage risks and to make decisions about that plan.
In an SOA strategy, on the other hand, interoperability takes on more prominence, and project management must take into account other issues that are not so familiar or so immediate from a traditional point of view.
Well, once we’ve set the context, let’s continue.
When there is a Service Catalog available for reuse by other systems, projects and teams working at different speeds begin to coexist. If the organization maintains the traditional approach to project management, based on the cascade model (planning, analysis, design, coding, testing, deployment), we will soon find projects with several months of duration that will find difficult to fit with the availability of the services of the Catalog. In addition, these traditional methodologies have well-known drawbacks: difficulty adapting to changes and high probability of obtaining obsolete needs over time.
This speed offset mentioned above happens because the Interoperability Office (IO) is a specialized team dedicated to maintaining the Service Catalog. And this catalog is the “Swiss Army Knife”, where all systems should look to reuse existing services for their requirements, and, with the IO, to incorporate new services of interest to the organization. So when you identify the need to evolve an existing service or create a new one, it will be available in days or a few weeks at most (assuming the IO is correctly sized). The new service or the new version of the service will be available long before the project that needs it has reached the construction phase.
In a complex information ecosystem, this dynamic will lead to important conflicts of versions over time, conflicts with deployments, testing phases, piloting, etc. In the end, we will see an increasing inefficiency of the IO, a slowed growth of the Service Catalog (with frequent and long blockages), a drag on the extent of the strategy (which will begin to see how certain projects end up out of its scope), and most likely, a decline in the quality of the results that are being achieved.
In short, our bosses will not see all those benefits promised by an SOA strategy, and will come to consider withdrawing their trust, reducing investment, commitment and involvement, pushing for results, etc., etc. Everything headed for disaster and back again.
But do not spread panic. There’s a solution.
To avoid this, fortunately, we have today the methodologies of agile project management. And in my opinion, they are the perfect travel companions for the extension of a Service Orientation strategy. I would dare include them as a necessary ingredient in the SOA strategy.
Although there are several of these methodologies, SCRUM is the best known and most widely used. I believe that it would be a good idea to include within the service-oriented interoperability strategy the adoption of this methodology for the projects management, at least from the extension stage of the strategy, in order to have the greatest flexibility and capacity to adapt information systems to the evolution of the Service Catalog. As we have seen in the previous section, the extension phase of an SOA strategy is, by its very nature, an agile phase, with a very high capacity to generate rapid releases in short periods of time. Therefore, the application of agile project management methodologies is especially appropriate.
In SCRUM there are two roles that are fundamental:
– Product Owner, who is usually the one who exposes the requirements (usually someone from the organization).
– SCRUM Master or facilitator, responsible for “clearing the way” to the technical team resolving the blockades, dependencies with third parties, resolving doubts, etc.
Along with these roles, there are some concepts that are especially important in my opinion:
– Product Backlog, which is the list of requirements that will be solved in the project. These requirements will be distributed in the different iterations, usually prioritizing the ones that are most beneficial with the least effort, and taking into account the velocity of the team (a revisable measure of the capacity of tasks that the team can perform in an iteration).
– Sprint, which is each of the iterations in which a subset of the Product Backlog (called Sprint Backlog) is attacked. Each sprint results in a release, which is shown to the Product Owner in a demo, and is ready for deployment. The iterations are usually short, between 2 and 4 weeks, although it is not uncommon to find references to sprints of 2 and up to 3 months (which to me seems excessive).
There are many things in SCRUM that I like, but here it is not a matter of detailing the SCRUM methodology or my particular analysis. The daily synchronization sessions of the Sprint and the Retrospective seem like a wise move that shows how much SCRUM approaches the reality of the projects, and of course here we also consider it. But I insist, let’s return to the focus of the post: agile project management in an SOA strategy.
AGILE PROJECT MANAGEMENT IN AN SOA STRATEGY WITH SCRUM.
And how does SCRUM fit into an SOA strategy? Well, in my opinion, it fits great. However, bearing in mind that we are dealing with a “project of projects”, a “SCRUM of SCRUMs” should be considered here.
If it sounds complex it is because it is complex. But nothing to be afraid of!.
And given the inherent complexity of an SOA strategy, we are not adding more complexity. We are adding techniques that will simplify and alleviate the coordination of the projects to be executed during the extension of the strategy.
Let’s see how.
The Project Plan that emerges in the extension of an SOA strategy will mix functional requirements with SOA reengineering requirements. Most often, all projects include both types of requirements, and it may happen that some projects consist exclusively of SOA re-engineering requirements.
The first key aspect to using SCRUM in our scenario is obvious: all Product Owner and all SCRUM Master must be perfectly aware and know the strategy, the rules and the Service Catalog.
The second one is in the opposite direction: the Governance Office and the IO should take into account the peculiarities of SCRUM in project management when designing procedures and policies. It does not make sense to design long, complex or very bureaucratic procedures, when the very philosophy of the projects is agility and speed in obtaining results.
Third, it will be necessary for a Product Owner of the Governance Office to coordinate the Product Backlogs of the projects and the rest of Product Owners. The person responsible for the IO would be the Product Owner of the Services Catalog. This team would adopt the same approach as the rest of the teams to facilitate the synchronization of releases.
One of the features of SCRUM is the daily Sprint synchronization meeting. In this quick session, standing up precisely to speed it up, the most immediate goals of the day are set for the team, and put on the table the blocks, doubts or dependencies that the SCRUM Master must clear . Well, it would be desirable for an SOA Strategy to hold a daily session that would bring together all the SCRUM Masters involved in the projects, so that the Governance Office could speed up the resolution of problems and blockages raised by the different teams.
Another of the most effective features of SCRUM is holding Retrospective sessions at the end of each Sprint. These meetings aim to review how the Sprint has worked, what things need to be enhanced, improved, changed or corrected for all team roles. And indeed, the next thing I raise is to also have a Retrospective session at the Governance Office level with the Product Owner and SCRUM Master of the projects.
Depending on the size of the organization and the variety of concurrent projects, the duration of Sprints must be reasonably estimated. Maybe less than a month can be too stressful, too many meetings, and more than a month in duration for each Sprint may be too much, favoring a too slow progress.
Have I gone mad? 84 meetings a month? Not counting follow-up meetings, committees, etc., etc. Keep calm…
Let’s think a little:
– A shorter sprint would not reduce the number of meetings, on the contrary: there would be more retrospective meetings, because the daily synchronization sessions would remain the same.
– A longer sprint would not reduce daily synchronization sessions, but retrospective meetings. However, these would lose effectiveness, be more dense and require more effort to obtain really efficient improvement measures. And I insist: we would reduce global meetings minimally.
– The weight of this amount of meetings seemingly exaggerated is carried out by the daily synchronization sessions of each project, where as we mentioned previously the SCRUM Master meets with his team to outline the tasks of the day, and put on the table the obstacles that must be cleared by the facilitator. But these meetings, following the SCRUM philosophy, are anything but traditional: they are held standing, should last no more than 15 minutes, and are one of the main engines of the agile methodology. And if a technique gives you agility, it should be used, right?
– Where it would be necessary to consider if it makes sense to meet is in all those traditional meetings that are held in the projects, these highly unproductive sessions, where a few well qualified professionals to advance the projects sit for a few hours to conclude that they have to do another meeting, to re-schedule some project, to ask this or that, or to send an e-mail reminding something, etc, etc … a meeting where most of the attendees do not open their mouth, or are only affected by what it is said in 10 minutes at most … those meetings followed by the writing and review of a record … how many hours of work do these meetings go? Is not this lot of hours highly unproductive that should be addressed?.
THE LAST OBSTACLE AND ITS SOLUTION.
To everything that was said about how to integrate SCRUM as an agile project management methodology in the adoption of an SOA Strategy, I would add another ingredient related to agility in ICT management and the ability of an organization to put into operation its solutions and services. In my experience, I have encountered significant delays and conflicts in this final stage, despite all the work done to have a release ready for deployment and production.
I’m referring to DevOps.
DEVOPS AND THE SOA STRATEGY
And what is DevOps? You may have heard the term lately but I bet nobody has ever stopped to explain what it is (if I’m wrong, then I’m glad I did not bet anything concrete). Without diverting the focus of the post, and to place it in the context that interests us, we will try to fix the concept.
DevOps is born from the merger of “Development” and “Operations”. Actually the term does not do justice to the breadth and variety of teams and profiles it encompasses. It is frequent to find in the definition of DevOps three related disciplines and active participants in the design, development, testing, and deployment of information systems: Development, Quality (QA) and Operations.
What DevOps pursues is to extend the benefits of agile methodologies to the stages after the user approval to the final tests. Traditionally the development teams have had their goal in that milestone: user testing. But for the organization, at that point it is not where everything ends, but where it begins. And we find usually with important lags and problems in the coordination of packaging, versioning, deployment, and support.
As we have said, DevOps is closely related to Agile methodologies, and it is especially necessary and efficient in scenarios where frequent releases with little functional load are pursued, which quickly add value to the organization (just like SCRUM). The automation of testing, and by extension, the generation of the corresponding package to deploy, error free, as well as the implication of the Operations profiles in the development lifecycle, are two of the most outstanding features of DevOps, and allow much better alignment of the production and support to the operation after the pure development of the project, and ultimately, substantially improve the quality of the result of all this work.
In an SOA strategy, where we seek to extend the strategy with the least possible number of blocks, we have proposed the incorporation of agile methodologies for agile project management, thus allowing the release of well coordinated improvement packages and extension of the Services Catalog.
In this context, DevOps is the icing on the cake: it drastically reduces “post-production” problems, improving attendance and support capacity, the quality of the implementations, the time required to install new versions, etc.
And, finally, an additional and collateral detail to all this, which I do want to underline as very important for the success of any project (be it the implementation of an SOA strategy, like any other project), and that is in the DNA of the underlying philosophy of agile methodologies. The professionals who participate in the projects, dedicate many hours, energy, effort and illusion, to what we do. Getting our work to end users and citizens in optimal conditions, in time and form, is an enormously motivating element. On the other hand, to see that effort is lost along the way is an enormously demotivating element.
And it’s no secret that motivated teams are more productive and effective teams.
If you liked this article, share it on your social networks. If you are interested or want to add or clarify something, or just comment on the content, do not hesitate to leave a comment here below.
And thanks for your time!