Demands for robust connectivity and business system agility are greater than ever. We want the latest information available across all systems and devices at all times. The tolerance we once had for waiting on a nightly batch integration to update data across our systems is diminishing rapidly. Everything must be connected. And, by the way, those connections need to be secure, reliable, and scalable.
Being highly connected is already the norm on our personal devices and social media. You can tweet a message about the dinner you’re having or post a picture of your cat chasing a toy, and everyone in your network around the world can see it immediately on whatever device they happen to be using. So why is this level of connectivity not the norm in business applications? Can you imagine telling Facebook or Instagram users that they will only receive updates from their friends once or twice per day when a scheduled integration happens? No way. The bar (and the expectation) has been set on having information at your fingertips. Now the same expectation is happening for business applications.
Business Applications and Point-to-Point Integration
Integration for business applications is not usually as plug-and-play as integration for personal device apps for many reasons. Sometimes that is blamed on the complexity of the applications and business processes from one organization to another. This is definitely part of the challenge, but even more significant is the fact that many organizations have had a culture of building single-use, point-to-point integrations for a specific application because that is how the implementation team could get the application implemented quickly. When many of these applications were first implemented, the integration options were limited. Plus, there was rarely an organizational strategy in place that helped standardize how integration was done from one application to another. An application team would find out what was needed to bring data into its new application and push data out of it. Usually, the path of least resistance was picked to accomplish this, meaning a direct integration from one application to another. This point-to-point approach is synonymous with hard coding. It is not flexible and cannot accommodate changes. When you multiply this point-to-point approach by the hundreds of integrations your organization probably has running today, you get a mess of spaghetti integrations and end up dealing with problems that limit the team’s efficiency and effectiveness like the following:
- Your ability to upgrade or replace your core systems is hindered by the effort and cost associated with changing the existing point-to-point integrations.
- There is uncertainty of how any changes to your applications may impact underlying integrations, which you will discover the hard way when you can’t take an order the next day.
- The IT department is constantly in maintenance mode, babysitting individual integrations of disparate standards and quality.
- The IT team is faced with a backlog of requested integration changes that will take a long time to complete (longer than the business wants to wait).
- There is a lack of visibility into what integrations exist and what they really do.
- IT and business users have no insight into what has successfully been integrated and what has not.
- A lack of central security management creates ongoing risk that one or more the integrations may be exposing unencrypted sensitive data (don’t worry, an auditor will be happy to help you find those things!).
I could list dozens more, but you get the idea.
The ability to be flexible and adapt to change is more important than ever for organizational success, meaning many organizations are faced with an integration challenge they must overcome. It’s time for connectivity to be seen as a strategic, competitive aspect of an organization’s business application ecosystem instead of just a few line items on the project plan for an application implementation. The good news is that most business applications now offer more advanced capabilities for integration. The key is to devise an integration strategy at the organizational level and enforce its use across all integration needs for applications, devices, and data.
There Is a Better Way to Do Integration
To get away from the point-to-point model, it’s important to look at integration not just from the perspective of one application or one department, but at a holistic level. Think of the analogy of building a new house in an undeveloped area. The builder of that house may think about laying pipes to the nearest reservoir miles away for water and electrical lines to the nearest powerplant for electricity. Sounds logical, right? Sure, if you are the only house there. Just like you have many business applications, there would likely be multiple houses being built. What if each new house in the area started from scratch like this? Each different house builder would be reinventing the wheel, incurring more cost, risking doing things improperly, and setting up a maintenance nightmare. Builders solved that problem. They looked at the big picture and laid out infrastructure (water pipes, electrical cables, gas lines, etc.) to allow all houses in an area to tap into common utility services that meet all safety and scalability specifications and are easily accessible. Those common services adhere to standards, allowing electricity or water to be easily consumed by any type of building or house. This seems like a no-brainer, right? The same concept exists for the integration of business systems in an organization. Look at what is needed, build a common infrastructure that can be leveraged by all groups, and put your governance in place to keep things running smoothly. This is the concept of centralized integration and the reason integration platforms continue to grow in popularity.
With centralized integration, an integration platform is implemented independently of any business application. This platform handles the integration needs across your applications. Each of your applications interacts with the integration platform, but not directly with other applications. Your application-specific integration tools are used to get data in and out of your application, but they should only interact with the integration platform, not with other applications since that would go back to the point-to-point model.
Integration Platforms, Web Services, and Application Programming Interfaces (APIs)
An integration platform typically offers sophisticated connectivity capabilities to connect to all kinds of applications, databases, queues, services, or whatever data sources and destinations you have. In addition, it offers some level of message-based transformation, data mapping, and routing functionality.
There are many integration platforms available on the market, including both licensed and open source tools. This blog is not meant to evaluate or compare integration platforms, but rather to emphasize their value. Some reside in the Cloud (Integration Platform as a Service, or “iPaaS”), some run in your own data center or private Cloud, and some can run in hybrid architectures for added flexibility. You should pick an architecture based on what will be most appropriate for your needs and goals. It is wise to consult with the vendor or other experienced integration platform advisors to select an architecture that aligns with your needs and goals. It is definitely not a one-size-fits-all model.
These integration platforms can connect to almost anything and expose Services or APIs. I won’t dive too deeply into the technical details in this post, but generally, services and APIs refer to either SOAP (simple object access protocol) or REST (representational state transfer) Web Services. Web Services are standards-based functions that are typically called via HTTP or HTTPS to fetch data or perform a task. They have evolved as a common standard across platforms that simplify how we can interact with so many different applications, systems, and data stores. Not only can the integration platforms expose these services for others to call using standards, they also provide powerful connectivity tools so the services can interact with applications and data stores of all types, from files to databases to mainframes to other external services. API Management is a common term and refers to applying policies to services to limit who calls them, how often they can be called, and more. API Management tools can also track usage and provide metrics and analytics. These are all very useful and will be discussed more in future blogs.
Having an integration platform in the middle addresses the issues presented by point-to-point integrations and adds significant value as described below:
- Reduced dependency on specific applications and versions of applications. Upgrades to packaged applications, interacting with a new destination system, or completely switching one application with another are very costly and time-consuming in a point-to-point architecture. With an integration platform in the middle and proper integration design, applications are not “hard-coded” to each other, so the changes are reduced to routing and mapping within the integration platform and nothing on the related applications needs to change. This is the concept of application virtualization.
- Services and APIs that are consumable, discoverable, and application agnostic. Similar to application virtualization, this feature describes how an integration platform can help make the underlying applications irrelevant to those who need to interact with them. Exposing a Customer service, for example, allows any application, device, or system to work with customer data (query, create, read, update, delete) by simply calling the Customer service and not worrying about where that data actually resides and how to get it. All that is needed is to find and be given access to that service or API. Most integration platforms also provide a repository that can be used to find existing services easily and promote reuse.
- Centralized management, monitoring, and auditing. Having all integration transactions going through a central hub greatly enhances the options for managing a viewing status of integrations. As you have more integrations, this becomes more and more valuable. Scaling, deploying, and monitoring services in one place means less overhead and fewer skill sets. Audit logging can also be consolidated and viewed or easily fed into log management tools like Splunk or ELK. These capabilities are valuable for audits, troubleshooting, and health management of your integrations.
- Improved data integrity and reliability. When various outside systems are creating their own point-to-point integrations into an application instead of reusing a common service, there is a chance for inconsistency in how data is queried or updated. That can lead to inconsistency across systems. Also, it’s more difficult to keep many point-to-point integrations working properly as opposed to fewer, centrally managed integrations—meaning reliability is higher.
- Faster time to market on new initiatives from reuse. The availability of reusable services and APIs serves as a set of existing tools for central IT to leverage when developing new solutions or responding to requests from lines of business. Aside from reuse, the central platform allows for stronger governance and more streamlined processes in development and testing lifecycles.
- Stronger governance processes. Any time consistency and standardization of technology and approach are in place, governance of the associated people, process, and technology becomes much easier to implement and maintain over time.
- Lines of business enablement and removing IT as a bottleneck. When IT focuses on building components that can be reused, the lines of business can leverage those components without needing to wait on new components to be built. Departments that need certain access to data for new initiatives can now possibly meet those needs with services already available in the library.
- Innovation. When services are available for others to consume (assuming they are granted access), it creates an environment of innovation within an organization. It’s the same environment that led to so many disrupter companies in recent years (Uber is a great example) that have been able to innovate based on existing services and APIs that were available. This same concept can drive innovation internally when APIs become available and people start thinking of innovative ways to improve how they do business.
- Technology modernization and standardization. Staying current on technology allows an organization to leverage technical capabilities and efficiencies. Integration platforms focus on offering the most robust and modern features for integration. This opens options for architecting new and improved integration solutions around not just application integration, but also other modern technologies like Internet of Things (IoT), Big Data, Cloud, Artificial Intelligence, Machine Learning, Blockchain, and more. These new technologies don’t work well without close connectivity with existing systems and data.
Applying Centralized Integration to PeopleSoft
Like many packaged applications, PeopleSoft provides a set of integration tools to help simplify getting data into and out of PeopleSoft in a supported manner. These tools include PeopleCode, Integration Broker, Application Package Classes, and Component Interfaces. While these are powerful tools that can be leveraged for integration, they are still part of the PeopleSoft application and should not be considered a central integration platform. They provide a lot of value in that they represent a supported method from the vendor for getting data in and out of that specific application. These tools should be used to interact with the integration platform, but not other applications directly. Also, they often provide advanced integration features that can be leveraged including event-driven capabilities, change-data-capture, and posting messaging out to a queue or web service. This enhances the integration capabilities specific to that application. For example, PeopleSoft could utilize PeopleCode and Integration Broker to publish out specific changes to an employee record as soon as they occur. These changes are sent by Integration Broker to the integration platform where the information can be sent to other applications, initiate other business processes, or perform any other required tasks. The importance of keeping that integration platform in there is to avoid the point-to-point spaghetti problem discussed earlier. Even though PeopleSoft has tools capable of interacting directly with other applications, using them for that purpose would create point-to-point, tightly coupled integrations that we are trying to avoid. The integration platform virtualizes PeopleSoft and brings all the benefits of centralized monitoring, security, and more.
PeopleSoft’s tools offer some good options for capturing events and publishing messages out to the integration platform for near-real-time integration. Unfortunately, those capabilities remain largely untapped with many PeopleSoft customers. Web services are becoming more popular in the PeopleSoft world, but there remains a large number of PeopleSoft customers who use database links, file import/exports, and direct database queries for integration. Batch integrations continue to be widely used as opposed to the more real-time options available. There are scenarios that justify scheduled batch integrations; however, Sierra-Cedar believes that the new norm should be to make things real-time and justify any exceptions. Most lines of business are starting to expect that type of immediate integration, so they know that their data is current and accurate when processes are run and decisions are made. PeopleSoft customers should start looking into more robust integration models to position themselves for the future. These capabilities are available in PeopleSoft, and when paired with an integration platform, the benefits of centralized integration can start to be realized.
One particular driving factor that has contributed to PeopleSoft customers shifting to this new integration mindset is moving PeopleSoft to the Cloud. This is now a more common strategy among PeopleSoft customers and is described in last month’s “PeopleSoft in the Cloud” webinar presented by Larry Horvath as part of our PeopleSoft Playbook webinar series. If there is an integration layer between PeopleSoft and any other application or system that interacts with PeopleSoft, moving PeopleSoft to the Cloud is greatly simplified. The services and APIs are simply repointed to the new location, and the other applications just continue calling them as usual.
Start with an Integration Strategy
If you think these integration concepts would benefit your organization, your first step is to start building a case. This blog includes some of the considerations around the negative aspects of point-to-point integrations that could lead to higher costs, slower time to market on new initiatives, less application agility, and an overall disadvantage from a competitive standpoint. As with any initiative, the costs and benefits should be weighed and aligned with key organizational goals that this technology could enable. It is also wise to voice any value you see from this approach to your IT department. As stated earlier, this type of strategic integration initiative is best driven by central IT and not a specific application team or department. That said, there is no need to move in a big-bang manner to this type of integration. Starting with one application or even just a set of services or APIs to show the value makes a lot of sense and is a great approach to making this change. If you need assistance determining what a future-state integration strategy should look like at your organization or how to make it a reality, seek assistance from experienced teams. To help build out a vision and roadmap, Sierra-Cedar offers customized strategy workshops ranging from three days to several weeks.
Optimal Integration Approaches for Your PeopleSoft Playbook
Investing in modern integration approaches makes sense for organizations as agility and time to market become more and more critical to success. PeopleSoft customers have a great opportunity to leverage the tools available within PeopleSoft and align them with a centralized integration platform. To join a presentation on these topics, please attend my webinar—click on the banner below to register!
After you register, reach out to us to talk more about establishing an integration strategy as part of your enterprise blueprint that will help you adopt a more agile, reliable, scalable, secure, and manageable integration architecture.