Building a successful platform team at CROZ
Real-life experience in applying Team Topologies patterns
By Ivan Krnic, Director of Engineering at CROZ
Ivan Krnic
Overview
CROZ is a professional services company. This means that we do not build our own products. Instead, we use our technical, domain, and organizational expertise to help our clients build their products and achieve their business goals.
With the speed of innovation higher than ever and the technology more complex than ever, we realized that our existing teams couldn’t keep tabs on every moving part in the new technology stack that was emerging in the cloud. Looking for a different way to structure teams and coordinate work, we have found our inspiration in four fundamental team topologies and their interaction modes from the book Team Topologies.
A year after introducing the platform team and establishing interactions with the rest of the organization, we found that the new team structure promotes collaboration and knowledge sharing in a better way, relieves cognitive load from existing teams enabling them to focus on delivering value to our clients, and serves as additional leverage for organizational improvement initiatives.
How did it all start?
Ever since the company experienced rapid growth in terms of people and projects, we have used the concept of long-lived and cross-functional teams leveraging benefits such as decentralized work management, elimination of bottlenecks, increased efficiency, and better knowledge sharing.
What our teams continuously aspire to is to own as much of the delivery process and supporting infrastructure as possible in order to reduce dependencies and increase the flow of features. Therefore, after initial help from the Internal IT team, our teams have always owned their virtual machines with test environments and took care of all runtime components such as application servers and databases.
Figure 1. The initial structure of our development teams
The advent of cloud native development brought a host of new concepts and technologies into the picture, putting additional pressure on our teams and their technical capabilities. With the speed of innovation higher than ever and the technology more complex than ever, we realized that our existing teams cannot keep tabs on every moving part in this new technology stack, particularly the cloud runtime platform.
The concept of a platform team is something that we needed at CROZ even before we knew that something like that even existed.
The cloud native approach changed everything and introduced new concepts such as cloud runtime, containers, and container orchestrators. All of that deeply affected existing practices regarding configuration management, deployment pipelines, deployment strategies, and vulnerability scanning, just to name a few. These additional moving parts were the straw that broke the camel's back. It became impossible for our teams to simultaneously deliver new features and maintain this new infrastructure stack. From our personal experience, we knew that teams were a powerful organizational concept and a key to solving this problem. Still, we were circling around, trying to apply it correctly to the new situation.
Right about that time I heard about a book called Team Topologies and loaded an audio version on my mobile phone the 11th day after it was published. It suggested a different way to form and operate teams in an organization. The concept that stuck the most with me was the notion of a platform team, and it goes like this: if we have feature teams (the authors call them stream-aligned teams to acknowledge the existence of value streams in the organization) that deliver value to the end users, let us not overburden those teams with the additional cognitive load of maintaining the platform, because the platform is becoming more and more complex every day.
Rather, let us build the platform and provide it as a service to stream-aligned teams. This way, stream-aligned teams can focus on delivering value to the end users - what they do best in the organization. The team that is providing the platform as a service is called the platform team. By doing this, we are splitting the cognitive load between stream-aligned teams that are using the platform and the platform team that is maintaining the platform and evolving it further as a product.
Splitting the cognitive load this way improved the efficiency of our teams. Once our stream-aligned teams didn't need to focus so much on the platform, but rather use it as something that works and is provided as a service, they found it much easier to focus on their everyday work - delivering value to the end users.
Figure 2. Introducing the platform team
What is the platform at CROZ?
There is no single definition of what a platform is. It depends on the organization. The platform at CROZ is based on the Red Hat OpenShift container platform. Our platform team maintains this Kubernetes-based layer, but also some of the services and features above it that are necessary for stream-aligned teams. This includes monitoring tools, observability tools, and some security features. Also, let us not forget self-service features because what we want is the least possible friction between stream-aligned teams and the platform team. If stream-aligned teams need a particular resource, it would be best if they could get it through some kind of a self-service platform. In our vision, a self-service portal is a component that complements Kubernetes-layer and brings additional value to the whole platform. In that sense, the platform is larger than just the runtime engine. It also encompasses a self-service component that provides a user interface and APIs for the stream-aligned teams.
Figure 3. Platform components at CROZ
As a professional services company, we work with various clients to solve their business problems. Different problems require different technology stacks. Consequently, not all products that we build are using the same technology stack. Our clients use the technology that best suits their needs.
Therefore, our internal platform aims to be the greatest common divisor that covers the needs of all these technology stacks. For us, it is a continuous effort to strike a perfect balance of standardizing as many components as possible by putting them in the platform, but also giving stream-aligned teams enough flexibility to build the best possible solution for every client by letting them use components that best serve the need. Standardization is always good. It helps with knowledge sharing, onboarding new people, and debugging problems. But standardizing too many components by putting them in the platform imposes additional constraints on the types of solutions that stream-aligned teams can produce.
One could ask who decides what frameworks and libraries should become part of the platform. Apart from stream-aligned teams that deliver features to the clients, there is also an interesting concept of enabling teams. The role of enabling teams at CROZ is to perform R&D activities. These teams spin-off to the side and try new technologies and concepts. Once they figure them out and understand where the value in them is, they get back and share insights with other stream-aligned teams. For example, one of our enabling teams was exploring the value and benefits of using Groovy programming language in our specific context. If enabling teams discover something that is of value to all teams, then we consider including it in the platform. In our case, enabling teams enabled (pun intended!) incremental evolution of the platform.
Figure 4. Enabling teams helping stream-aligned teams
Who is driving the platform evolution?
At CROZ, initiatives are coming from two directions. The first direction is from the platform team. Since the platform team lives with the platform every day, they know its strengths and weaknesses. They recognize room for improvement. The platform team also keeps an eye on new trends and suggests new capabilities that could help stream-aligned teams. But the final decision is made together.
The second direction is from stream-aligned teams. We consider this direction even more important because the sole reason the platform team is building the platform is to help the stream-aligned teams. In a way, the platform team recognizes and treats stream-aligned teams as their clients. And in a true product management fashion, they are the ones who need to take care of the platform and talk with stream-aligned teams to understand their needs, struggles, and challenges. In essence, they need to learn in which way to further evolve the platform and bring the maximum value to the stream-aligned teams.
This dynamic is something that took us a little bit longer to figure out. It is not something that came naturally in our case. We have traditionally pushed platform evolution more from the bottom up. As a result, the platform was on a brink of being disconnected from the stream-aligned teams because people in the platform teams were implementing some ideas that were not in line with the needs of the stream-aligned teams.
What we had missed here - making it a good lesson for other organizations - is to start treating the platform as a product. And the only way to treat something as a product is to recognize who we’re building it for. Who are our personas? We need to understand the customer journey of these people and how the platform can help them. It does not differ that much from typical product management work. All the activities and techniques that we would use if we were to design and deliver a product like a web application are also applicable here. This comes naturally while we’re building a web application, but it doesn't come naturally when we're building a platform. It is something that we need to do consciously. Once we made that mental switch of treating the platform as a product, things started to move in the right direction. This little shift in perspective made the biggest change for us.
How does the platform team operate?
Maintaining and evolving the platform is conceptually no different than building a web application. Therefore, the process is the same.
In both cases, we have a backlog of features we want to implement, and we strive to have a cross-functional team that can ideally handle all the tasks without any hand-offs to the other teams. In the platform team case, the skills needed to deliver value are a bit different. For example, we don't have typical business analysts proficient in a particular business domain. We do have people that own the platform vision, but these people are not business analysts. They are product managers coming from the technical area. The roles in the platform team are a bit different. We have system administrators with network and security skills, people with strong development skills, people proficient in Linux and Kubernetes, and people with automation experience.
Figure 5. Platform team during weekly check-in
You will notice this is still a cross-functional team - a group of people that has all the skills and knowledge necessary to deliver their product - in this case, the platform. They do work incrementally, talk a lot with their customers, aim at short feedback loops, and are always in the contact with their clients. The platform team grows just like any other, so regular retrospectives are also in place. In that sense, how the platform team operates is pretty much the same as any other stream-aligned team.
In our case, the platform team also operates as an enabling team when it comes to helping other teams onboard the platform and embrace technical practices that are supporting DevOps culture.
Figure 6. The platform team doubling as an enabling team
Sociotechnical aspects of an organization
Most of the platform team members have been with us for a long time. These are the people that know all the nuances of our delivery process. When some issues need to be resolved or something needs to be built for a specific team, their experience with the technology, but also the years they spent working with people in stream-aligned teams, help us deliver features more efficiently.
It mostly comes down to the relationships between the platform team and the stream-aligned teams. Although these two types of teams are independent, it proved particularly useful to have good relationships between them and to have them work together and collaborate on evolving the platform. This ties back to sociotechnical aspects of an organization, and how to structure teams in an organization for them to collaborate efficiently, which is, no doubt, the most important thing that you can achieve in an organization.
How does the conversation in teams and between teams happen?
There are many ways how to spark up the conversation between like-minded people. A concept that proved most efficient for us is the Community of Practice. The Community of Practice is an informal group of peers gathered around a common topic that is of interest to everyone. Group members share knowledge, experience and skills, and work together toward a common goal. Due to its nature, the Community of Practice is a highly inclusive group of people. Diversity is welcomed and needed since it introduces different perspectives to the same problem and helps find optimal solutions.
How do you start a Community of Practice? Look around you and pick something you’re very passionate about that is worth improving. Do you feel like your organization could benefit from a better testing process? Find your like-minded peers and recruit them into a Community of Practice for Testing. Work together on improving the testing process. Define your working agreement. Everything is acceptable: you can come together every morning over coffee, or later during the day, or after work over beer. The only important thing is that participation is voluntary, and members are genuinely interested and motivated.
Figure 7. A Community of Practice for DevOps
This is how we started our Community of Practice for improving the platform and the delivery process. Today, our community is hosting weekly get-togethers where community members are exchanging ideas and tracking improvement initiatives. We also hold monthly visibility events called #Flashnews that are targeted at the whole organization to additionally elevate the visibility of platform evolution plans.
Where do we go from here?
Understanding the concept of cognitive load and relationships between different types of teams initiated a positive dynamic in our organization. Not only do we have a viable blueprint of how our teams should be structured, but the structure itself also promotes collaboration and knowledge sharing. Splitting the cognitive load relieved much of the pressure from stream-aligned teams and enabled them to focus better on delivering value to end users.
The Community of Practice is gathering people of all roles around the same goal: optimizing the software delivery process. And the platform is catalyzing this change. The platform is simultaneously serving both as a means of improvement and as a mirror that transparently shows how well our process is set up. It enables us to improve the process, but it is also the first one to give us the feedback that we’re doing something wrong. In that sense, the platform drives further technical and process improvements in the whole organization.
Thinking in terms of four fundamental team topologies and three core interaction modes cleared up some ambiguities around roles and responsibilities in the delivery process. This enabled teams to focus on what they love and do best, motivating them to further build their skills. It also made the role of every team member in the delivery process transparent. No other past initiative produced such engagement among people.
It is difficult to improve the process if you do not have the right organizational and technical practices in place. And it is equally difficult to improve these practices if you do not have a tool to gauge your efforts. Structuring the organization around four fundamental team topologies and treating the internal platform as a first-class citizen gives us additional leverage on our improvement journey. We will continue to use this momentum in the following improvement initiatives.
Follow Ivan Krnic:
Follow CROZ: