The Cloud is great, isn’t it? It opens so many doors, removes some major barriers when it comes to organizations getting a system up and running quickly, and it… well… also poses some questions to those who are new to it. This blog post will deal with one very common question asked by both business and IT users: how do I handle integration with applications that are in the Cloud? In an attempt to stay consistent with other blog postings in this series, I will address the business user audience as opposed to the IT team. Plus, most of the online content you can find today focuses on technical architectures while leaving the business user’s head spinning with acronyms. So this one is for the non-techies. Also, for the purposes of this blog post, Cloud applications will refer to SaaS (Software as a Service) applications that reside in the public Cloud.
Obviously, every organization is different and solutions and recommendations depend on specific needs, priorities, strategies, and even preferences. This blog is intended to provide some general information and guidelines, but it should not be considered a one-size-fits-all recommendation. If you are interested in discussing an integration strategy specific to your organization, let us know and we will help. The message I want everyone to take from this blog is that with the right know-how, you can get robust, secure, scalable, and proven integration between your Cloud and on-premise applications today.
In an attempt to make this as relevant as possible, I’ll structure this blog post based on common questions I’ve received from business users about Cloud integration. I’ll use the first three questions to explain some concepts that will be helpful when I get to the last three questions.
- How do I get data in and out of Cloud applications?
- Is real-time integration possible with Cloud applications?
- What is a service bus, and do I need one? What’s the difference between a service bus in the Cloud and a service bus on premise?
- What does basic Cloud-to-Cloud integration look like?
- What does basic on-premise-to-Cloud integration look like?
- I have a bunch of applications both in the Cloud and on premise. How do I integrate them all?
1. How do I get data in and out of Cloud applications?
Get ready for a consulting answer… it depends! Every Cloud application determines how its users can integrate with it and whether it will provide options for file export/import, Web Services, or even prebuilt integrations with other common applications. While there are some standards that are growing in popularity, the fact of the matter is that you need to see what is available from the Cloud application you are implementing, because that is what you will be able to use. The vendor has probably documented the best approach for integration to its application.
This section will explain some common integration options for how to get data in and out of your Cloud applications. You’ll notice these are not much different than what you are used to seeing from on-premise applications, but Cloud applications typically provide a more robust Web Services or API framework for you to use.
Web Services or APIs (Application Programming Interface). Most SaaS applications offer Web Services or APIs to get data in and out in a secure and supported way. The best practice is typically to use a service bus to call these Web Services, although you can call them from custom code as well. A service bus is an integration tool that essentially accepts a message containing data and transforms that data into one or more other formats based on what the destination applications need. It then delivers the message to one or more destination applications in the proper format for each application. Since that logic and the delivery of the message all reside within the service bus, it is easier to troubleshoot problems and is less intrusive on the source and destination applications because the need for custom code in each application is minimized. In addition, most enterprise service buses (you might have heard the acronym ESB) have a console of some kind that provides centralized visibility and monitoring of all integrations. When you have 30, 100, or 500+ integrations flowing at your organization, being able to see everything consolidated in one place is critical for auditing and monitoring.
Direct database interaction. If no Web Services or APIs are available (which is rare, especially for SaaS applications), then we would see if there is a way to communicate directly to the application’s data or to its proprietary functions. With SaaS applications, direct access to the data is typically not possible—but there are some cases where it is made available, so I included it in this section. Direct data access should only be used for reading data, and then only when you are certain you know exactly how to get the data you need. For updating or adding data, you should always use a Web Service or API so you do not inadvertently impact the integrity of your application by making an invalid data change.
Built-in point-to-point integration offerings from one or more of your Cloud applications. Many Cloud applications anticipate that you will need to integrate with some other common applications, so they attempt to do the work for you and offer that integration out of the box (or with some basic configuration). These can be good or bad. Obviously, a supported solution built by the vendor means you do not need to design, build, and support the integration yourself. On the other hand, you are relying on the vendor to keep the integration current and stable as their application—and the applications to which they integrate—change over time. Most Cloud apps that offer these integrations are pretty good at that, so we usually recommend using these prebuilt integrations when they are available as long as they meet your needs. After all, why reinvent the wheel if the SaaS vendor has already done the work for you? One other downside is if you have another centralized integration solution in order to consolidate management of your integrations in one place, these integrations will not conform to that and you’ll have to check status of these prebuilt integrations separately. This inconvenience is often acceptable since it is still a time saver and someone else is supporting it.
Traditional file-based integration. Everyone has a comfort level with flat files and FTP. It’s been used for decades, and yes, it is often (but not always) still an option for Cloud applications. One system writes a file with data in it. That file is sent via FTP or Secure FTP to another server where the destination application imports it. This typically requires some expertise on both applications to make sure the right data is exported at the right time and the data is properly mapped in to the correct fields in the destination application. The downside is that it is three distinct activities that often make troubleshooting difficult, i.e. did the problem occur in the export of system A, in the transfer of the file, or in the import to system B? Often these three activities are set up by three different individuals with different skill sets, each pointing the finger at the other two when a problem occurs. Integration in the last 7–10 years has steered away from this approach since it is not centralized and often embeds the logic within the source and destination systems, making changes to the integration difficult and upgrades to the applications more complicated. Plus, if the applications each require a proprietary skill set, you’d better have all those different skills available on your team.
2. Is real-time integration possible with Cloud applications?
Yes, real-time (or near real-time, which is probably more accurate given that there is always some time required for updates to get to another destination) integration is common with Cloud applications. In fact, it is often easier to accomplish in a Cloud environment since so many applications provide Web Service interaction, as opposed to file directories or interface tables that tend to take more time and are more suitable for batch operations. Using a service bus is the best way to accomplish real-time integration because it can take that update and very efficiently deliver it to one or more target systems. The key is how the change is initiated, which requires a short explanation of event-driven concepts.
Event-driven integration. Some applications have the ability to recognize when a certain event occurs (such as booking an order, changing a candidate from Offered status to Hired status, or saving a new address to a customer record) and then performing some action. For real-time integrations, this is extremely helpful since this action can initiate an integration to send those changes to other applications. If this capability does not exist in the application, the options are to periodically check for changes (not very efficient) or send batch updates to capture all changes since the last batch was run (not very timely). If you only need the changes integrated periodically (once per day, for example), then event-based capabilities are not needed. Given the nature of business applications today and mobile capabilities, many business requirements dictate that the latest information is always available so there is no chance of making decisions based on old information.
3. What is a service bus, and do I need one? What’s the difference between a service bus in the Cloud and a service bus on premise?
A service bus is a central “hub” through which integrations can be routed from one application to another. It is a high-speed delivery tool that has ways of interacting with almost any application, database, or file to get data or deliver data. Ideally, the applications will have Web Services or APIs available, which are very easy for the service bus to utilize. The service bus also centralizes and standardizes the way information flows from application to application because all data goes through the bus, thus simplifying auditing and monitoring. When using a service bus, all applications interact with the bus, not other applications directly. This makes it easier to upgrade or replace any of your applications because the changes needed to map to a new version or application are all done in the service bus. All of the applications that previously shared data with that old application are now sharing it with the new application. If your applications all directly interacted with each other (what we call “point-to-point” integration), you would need to go into the code for each of those applications and change all of the references and mappings in all of them to move from the old target application to the new one. That is not a fun activity.
A service bus provides exceptional value for an organization that needs to integrate business-critical data and has many integration points. Centralizing the integration maintenance, monitoring, auditing, and security are very beneficial, and service buses promote the use of common services in your organization and a Service-Oriented Architecture. For example, you can expose a service that is called “getEmployee” which fetches an employee record from your master HR system based on some search criteria and returns the result. If you have many applications, Websites, mobile apps, and business processes that at some point need to get an employee record, wouldn’t you prefer to create a single service that you know has the right logic to get the employee data, is secure, and can be managed so any changes only need to be done in one place? Or would you rather each of those entities that need employee data figure out their own way to get it and hope they are all consistent? Reusable services greatly simplify not only data consistency, but also data security.
Along the same lines, these services that are exposed can be taken a step further in terms of security and support and made available to the Internet for others to use (such as Google exposing its service for getting driving directions based on starting and ending locations). Many organizations are not quite ready for this, but it is a way to empower others to use your service and you can even monetize it if you want. Services exposed like this are typically referred to as APIs. These same APIs can be used for building mobile apps that need to securely tap into logic or systems you have internally. So, long story short, unless you just have a couple of integration points, odds are you would get significant value out of a Service-Oriented Architecture and the core behind that is the service bus.
In response to the second question, a service bus hosted in the Cloud and offered on a subscription-pricing model to customers is often called an iPaaS (Integration Platform as a Service) solution. These solutions require very little on-premise infrastructure if you are integrating an on-premise application, or none at all if you are only connecting Cloud apps to other Cloud apps. An on-premise service bus is software you would license (unless it is open source, which we do not recommend for enterprise integration) and implement in your company data center. If the great majority of your integrations are between Cloud apps, then an iPaaS solution could be a great model. If you have six on-premise applications you need to integrate with each other and only one Cloud application to integrate, then an on-premise service bus might be preferable. Just keep in mind that you want to avoid round-trips through your firewall as much as possible for security and performance reasons. If most of your integration is between applications in the Cloud, then it’s best to have your service bus in the Cloud. If on-premise, then it’s best to have your service bus on-premise. These scenarios are outlined more clearly in the following sections, as well as a consideration if you have a large amount of integration BOTH between Cloud applications AND between on-premise applications.
4. What does basic Cloud-to-Cloud integration look like?
Since both applications are in the Cloud, we’ll make the assumption here that an iPaaS integration solution is used. Remember, iPaaS is a service bus in the Cloud, so in this example, the Source and Target applications, as well as the service bus, are all in the Cloud. This type of architecture allows some smaller organizations to get enterprise-level applications and services that would not be possible without the Cloud because of hardware, maintenance, and license costs. Because iPaaS solutions can connect to on-premise applications, one on-premise application is also shown connected via the service bus for good measure.
5. What does basic on-premise-to-Cloud integration look like?
The example below uses a service bus that is on premise to integrate between the on-premise and Cloud applications. Since it appears that most integration is happening between applications on premise in this graphic, the on-premise service bus makes the most sense. Again, based on specific requirements, this approach is not right for everyone, but it gives an example of an on-premise service bus integrating to the Cloud. Note that either the on-premise or the Cloud application could be the source and the other the target (bi-directional).
6. I have a bunch of applications both in the Cloud and on premise. How do I integrate them all?
If your organization has a ton of traffic in the Cloud AND a ton of traffic on premise, it does not make sense to force all integration through the Cloud or on-premise service bus. In this case, a service bus in the Cloud can facilitate all Cloud-to-Cloud integration and the service bus on premise can facilitate all on-premise integration. When an integration needs to go between Cloud and on premise, the two buses simply interact directly. This means only one source outside the firewall will be interacting with the on-premise service bus (which simplifies security), and each bus serves as the central hub for its respective on-premise or Cloud applications, keeping performance and control optimal.
Here are some thoughts I’ll leave you with when it comes to figuring out how Integration in the Cloud will look at your organization:
- Make sure to include some time for integration strategy planning during the planning of any new application, whether it is in the Cloud or not. Also, make sure there is someone who understands service buses in that planning—far too often, the integration strategy is dictated by application experts who consider integration an afterthought that really is not a priority to them implementing their application.
- Don’t be afraid to make the integration portion of your project a parallel project in itself. Since the integration solution could potentially be used for other projects and applications, separating it from a specific project sometimes allows for a broader vision with integration so other projects benefit as well.
- Don’t just recreate the data flows you had with your old applications when you go to the Cloud. Look closely at your business processes and think about how the processes should work in terms of real-time versus batch, visibility into the status of integrations, exception handling, business logic, and anything else that might add effectiveness or efficiency to your business process. For example, who says only IT should see that all invoices were successfully integrated? Let your AR team see that information, too, so they can take action immediately rather than relying on IT to inform them first! Start removing those dependencies on IT where you can.
- If you have integration coming from the Cloud to on-premise applications, make sure there is a solid security plan in place. These architectures have been defined and are reliable, although they are a little more technical than I wanted to cover in this post. Still, any time you are letting an outside entity access something on your network, you need to make sure you are protected. There is much less risk if the integrations all originate on premise.
This blog post hopefully educates business users about some of the nuances of Integration in the Cloud. Many organizations and users have a fear of the unknown about Cloud, and while doing business outside of your network and data center does require proper know-how and expertise, it is a very cost-effective and powerful way to let your organization focus its time and resources on your business plan instead of the technology behind the scenes.