eBonding Design Considerations and Good Practices

As mentioned previously, the example I’m working from is a bi-directional application to application integration, meaning that the systems are integrating application records throughout the lifecycle of that application’s workflow. For example, an Incident in system X that integrates with a ServiceNow Incident and exchanges updates throughout the life of both incidents, regardless of who has ownership of the resolution. Many know this concept as “eBonding”. Simply put, this is integration of both data and process, where what data is exchanged, and when, are as a result of process and may also influence process.

The technical designs I’ve outlined above work very well for eBonding, and are in fact designed to work with this practice. In addition to the technical aspects, here are other considerations when designing an integration solution for eBonding:

• Both systems have to agree on the field mappings and data types. (No different than any other integration.)
• Both systems have to agree when mapped fields can be updated. This is especially important for things like the ServiceNow “state” field, which either controls or is controlled by workflow. In our Incident example, often the only states that are allowed to be set by the integration are canceled or resolved. Other states may change in the other system but aren’t automatically updated by the integration as it may affect workflows, SLAs, etc. Rather, information may be included in a work note so each system is aware of activity in the other, but the process is not potentially adversely affected by it.
• The integration needs to include mapping translations for field values that don’t match in usage across the systems. For example, if ServiceNow uses Priorities 1-4 and System X uses Severity 1-10, you’ll need to create a mapping matrix to map System X’s Severities into ServiceNow’s Priorities, and vice versa. (Also consider States, Categories, etc.)
• You’ll need to consider how Reference fields get populated and integrated, but I’ll discuss that more in the Good Practice Tips.

I’m including diagrams from the AT&T Incident eBonding I built for ServiceNow below. It details the integration flow for two scenarios: A “Proactive” incident initiated by AT&T, and a “Reactive” incident sent to AT&T. In both scenarios AT&T is the owner of the incident – responsible for the resolution, as the use case is AT&T owns the customer’s network and in the incident is network related.

Note the listing of updateable fields, and when, as well as uni and bidirectional flows of data.

The keys to a successful eBonding integration are the discussion of, and agreement on, the data what and when that will flow between the systems, and the rigorous test planning and testing of all lifecycle scenarios. These are vital to ensure you don’t break existing internal processes already developed and running in your ServiceNow environment.

Other Good Practice Tips

In addition to the primary design considerations outlined above, I recommend the following:

• While security is of the utmost importance, and is often the thing customers think about first, try to design and build your integration without the security layer, or use the most basic security possible. This allows you to prove out the design and confirm the connectivity first, and assumes you have sub-production environments to develop and test in. Security can almost always be layered in as a second step. This eliminates a layer to troubleshoot as you iterate your development.
• You’ll need to consider and account for integrating ServiceNow Reference fields *. As you know, these are fields that are stored as sys_ids in the integrating ServiceNow record, which is not likely to mean anything to the external system. Here are some guidelines for integrating Reference fields:
  • Consider if there is value in having the Reference data tables stored and maintained wholly in each system, so each is aware of the full dataset and mapping is an easier exercise. (There good reasons to do this, and reasons it’s often either impossible or a bad idea.)
  • Ensure that both systems have a field that uniquely identifies the reference in both systems. For example, for users records, email address may suffice.
  • Ensure that field data is included in the bidirectional payloads
  • Use the “Reference value field name” in your Web Service Import Set Transform Map Entry to use this field to choose the right ServiceNow reference record (using our out of box functionality again!)
  • Set up your Outbound Field Mapper to map the ServiceNow field to System X field, so that the external system doesn’t get the sys_id
  • And for goodness sake don’t try to use display value strings as unique identifiers!
• I suspect integrations that don’t use REST (or even SOAP) could use the same approach I’ve outlined. Even a file-based export could work the same, save for the nature of the outbound and/or inbound payloads.
• Wherever possible, the outbound integrations from ServiceNow should be run asynchronously. This is a general good practice with all integrations. For example, if the integration is triggered via a Business Rule, the Business Rule should be set to async if at all possible. This way the end user and the system (UI) do not wait on the integration to move forward, and the integration runs as system resources are available to it. The exception is if there is a business requirement for the system to wait on the integration, e.g. the end user is expecting to get a result back from the external system before proceeding. There are also technical reasons this can be a challenge: For example, you cannot run an async Business Rule on a comment or work note addition.
• Only use a Scripted Web Service if the inbound payload will not be in a name:value format that can easily map into a staging table, and rather requires scripting logic to manage the payload before injecting it into a ServiceNow record. Consider this a “last resort” in most cases.

Some of these points could warrant their own article; hopefully this article triggers your design thoughts and gives you ideas about how to manage your integrations.

Conclusion

Since its early days ServiceNow has had integration technologies built into, and fundamental to, the platform. Many a system has been integrated into ServiceNow in all shapes and flavors. While all kinds of new tools inside and outside of ServiceNow have attempted to simplify integrations, the “good old” ways still work when no other options exist (or existing options don’t quite fit the bill).

Captain: You can have the easy way, Luke… Or you can have it the hard way… It’s all up to you. – Cool Hand Luke

If you work in a ServiceNow environment in 2023, it’s more than likely you’ve got it integrated with other systems. Given ServiceNow’s place in the market, it’s unlikely that an instance is running in an environment small enough or segregated enough to not need to be integrated with other systems. At the very least, you’re likely getting your core data from somewhere outside of ServiceNow, and hopefully not through a manual import. (Who wants to keep up with that effort?) You may be using a “good old” LDAP integration, or you may be using middleware, or an Integration Hub pre-built solution. Regardless of the solution, I’m going to use the rest of this article to talk about good practices for how integrations should be designed in ServiceNow so that your applications, and indeed the platform as a whole, are protected from possible integration chaff, and so they can be easily extended by non-coders when the need arises. I’ll primarily cover custom Web Service integrations, with the intention that if you understand how to design these kinds of integrations the knowledge translates well to all integrations.

In Part 1 of this article, I’ll delve into Inbound and Outbound design considerations, and in Part 2, I’ll cover considerations for a true eBonding type integration as well as other general tips I’ve learned through the years building integrations.

A quick bit of “curriculum vitae” to establish my bona-fides: I’ve been doing ServiceNow integrations since 2011; I was one of the early ServiceNow Professional Services consultants to delve into integration work. I developed one of the first AT&T eBonding integrations and gave the code and configuration to ServiceNow development to leverage as a packaged offering. I also built the ServiceNow side of the Workday to ServiceNow connector for Workday (the company). I’ve focused primarily on SOAP (early on) and REST based Web Service integrations. I also helped build the first iteration of the Perspectium DataSync tool.

Baseline Knowledge

This article assumes the reader has a baseline knowledge of how to do integrations, both in general and in ServiceNow. It also assumes you have knowledge of the various ways that ServiceNow does, or can do, integrations “out of the box”.

The examples in this article are based on Diagram 1. The example is a bi-directional application to application integration and includes the following:

• The REST protocol with JSON payloads
• A Web Service Import Set to stage the inbound data
• An integration with a Task-based application in ServiceNow
• A field mapping table to manage inbound and outbound data updates

Inbound: Default to Using a Staging Table

You should always default to staging the inbound data in Web Service Import Sets (WSIS). These are nothing more than Import Set tables with a slightly extended API. (I’ve honestly never needed anything more than the standard API calls when using these tables.) Here are the reasons these tables should be the place to integrate into ServiceNow:

• Staging the data insulates your application tables from data issues with inbound integrations. This allows you to build both data and logic safeguards into your integration. External systems can use the WSIS Table APIs to inject data into ServiceNow, where it waits to be transformed into application or core-specific data. Transform logic can ensure that bad or malformed data doesn’t make its way into your SN processes, preventing potential SN instance issues.
• Staged data in WSIS can be transformed like any import set. This means SN administrators who may not be familiar with Web Services or integration design in general can still configure transform maps and transform logic. Many of the future changes to the integration can be handled by anyone who can maintain transform maps.
• Staged data can be used for troubleshooting integration issues: If there’s an issue with the integration after the inbound request has reached ServiceNow, the import set record serves as an auditable trace of the raw data received. Often issues can be solved by a review of this data, e.g “Hey we agreed you’d send the data in format YYYY-MM-DD and you sent MM-DD-YY.” Clever developers will set up ways to store raw JSON payloads and integration messages (errors etc) in the Import Set record.

Things to note with this approach:

• For most situations, you’ll need to ensure the integrating system receives the unique identifier of the record created by the transform, not the import set record. Recent versions of ServiceNow’s Import Set API appear to do this inherently in the JSON response.
• The import set will need to be set up not to use the default ServiceNow system delete property of 7 days if you want to be able to trace issues older than this.

ServiceNow documentation can show you how to achieve both of these.

For folks reading this who have become skeptical because Integration Hub doesn’t take this approach, I learned this approach from ServiceNow employee #1 (those who know, know). My standard tact is to believe those who created the platform over “johnny come latelies”.

Outbound: Create Extensible Field Mappers Instead of Writing Code

While WSIS are standard ServiceNow functionality, this recommendation is my good practice, and one I’ve espoused for all platform development. The goal is to build solutions that write code once, and build configurations that are extendable for future changes – ones that can be managed by non-coders. Think of it like a custom Transform Map for your integration. In the diagram above, this is the “X Request Map” at the middle bottom. In its simplest form, the table contains the following:

1. The table and field of the application in ServiceNow
2. The table and field of the integrating system
3. The nature of the integration: Inbound, Outbound or Bidirectional
4. An active flag

For #1, you can use the Table Name and Field Name dictionary field types. (The latter is dependent on the former; e.g. choose the Field from the Table selected) (Dictionary types).

Numbers 1 and 2 tell the integration the tables and fields from both systems map to each other. Conceptually, exactly like a Transform map, although one of the table\field combinations in this case is from the external, integrating system. Number 3 is a choice drop-down with options for Inbound, Outbound, and Bi-directional. Finally, an Active flag (Number 4) tells the integration is this a mapping that is currently used.

In Diagram 1, the bottom right side of the image shows how and where this is used. In many integrations I’ve seen, the creation of an outbound request from ServiceNow is done with pure code: the payload is built out via code, and changing the payload requires changed code. My suggested approach is to use Business Rules to trigger the request – when an update to an integrated record occurs, trigger the initiation of an outbound request. I use a Script Include function to build the request, so that it can be called from multiple places. Most importantly, I use the mapping table to determine what fields should be sent, and the field names to get the values from the ServiceNow record. The process flow is:

1. The integrated ServiceNow record is updated
2. A Business Rule running on that record’s table determines if the update needs to trigger the outbound integration
3. The Business Rule code calls a Script Include function, passing it the current GlideRecord
4. The Script Include function queries the mapping table, filtering on active, type=outbound, table is the current table
5. The Script Include function loops through the result, pulling the values for the external system fields and the GlideRecord field values, building an outbound name:value pair payload
6. The Script Include function triggers an outbound REST message and attaches the payload
7. The Script Include function processes the response as desired

Important Note: Wherever possible, the Business Rules should be run async. There is more on this in part 2.

If this is built correctly, the major benefit is that future updates to the integration can be completed with updates to the Mapping table, rather than with code. A true low-code, no-code solution!

More to come in Part 2.

“A tree with a rotten core cannot stand.” — Aleksandr Solzhenitsyn

In the fitness world, and in fact the physical human world, your core is the central part of your body. It includes the pelvis, lower back, hips and stomach. Exercises that build core strength lead to better balance and steadiness, also called stability. Stability is important whether you’re on the playing field or doing regular activities. In fact, most sports and other physical activities depend on stable core muscles.

As ServiceNow has moved further towards being a product company and less a platform company, it’s easy to lose sight of the aspects of the system that are core to its functionality and its value. If you’re solely focused on products*, it’s akin to building big arms and shoulders, and large calves and thighs, but ignoring your back, abs and glutes. Eventually you’ll be a Popeye-ish figure, unable to balance because you’re both disproportionate and “weak in the middle”. In this article, we’ll discuss what’s core to ServiceNow, what benefits having a good core provides, and how to build and maintain this core.

*Products, or Applications and Application Suites, are things like ITSM, CSM, HRSM, ITBM, ITOM, and their components. These are the things ServiceNow builds, markets and licenses on top of the core platform.

What is the ServiceNow Core?

There are several aspects to the core platform. I’ve highlighted some of these in a previous post, primarily the development components that make up all the applications. In this post, I’m going to focus on the core data which in the least drives consolidated reporting, and as I’ll elucidate later, the best gives a full insight into how your business is running.

From a data perspective, the core data are the tables that can be seen in the Organization application in the left navigation menu.

The main tables are:

• Companies
• Departments
• Locations
• Business Units
• Cost Centers
• Users
• Groups

Note: Vendors and Manufacturers are Companies with particular attributes, not unique tables.

If you look at the schema map for any of these tables, you’ll see how many tables reference these records. For example, the Department table is referenced 746 times in my largely out-of-the-box PDI. Most of these are the CMDB, and indeed, it is hard to use the schema visualization ServiceNow provides because of the number of CMDB tables it needs to draw to represent the schema. If you look at the dictionary references to Department, there are still 36 entries.

However this is just part of the usage of this data. Consider the dot-walking use cases for Department. (If you don’t understand what dot-walking means, please refer to ServiceNow documentation.) Since Department is a field on User record, everywhere a User reference exists, Department can be used by dot-walking to it. Looking at the dictionary references to User, in my PDI there are 784 non-CMDB fields across the platform. So this is 784 places you can inherently filter on or group by Department by dot-walking from the User reference field.

Because the in-platform schema is overwhelmed by the CMDB, I drew a diagram of just how these core tables tie together:

Note: Depending on your organization, you may not need all these tables populated. Smaller organizations may not distinguish between Departments, Business Units and Cost Centers.

Building and Maintaining Your Core

Experienced system administrators and ServiceNow developers are familiar with these tables and this data. What I’ve often found is there’s an effort during the initial implementation to populate the required tables, then the maintenance is lacking and data becomes stale or messy.

Here are some common examples:

• The data is imported once and either an integration or an ongoing process for maintaining the data isn’t implemented
• The company does a re-organization and user departments, cost centers, business units aren’t updated
• Unique identifiers aren’t determined for the core records and subsequent imports create duplicates
• Companies treat users like tickets – they just need to be able to login, have the correct roles, and “life is good”

Here’s a small example I ran into recently: A customer had done a series of User imports from other systems without clearly identifying and marking a unique field. An integration was built from Salesforce using the email address as an identifier of a Requester (User) in ServiceNow. An issue was reported after we went live because the Requester was incorrect. The root cause was there were three active user records with the same email address and the system picked the first one sorted by sys_id. This issue was not previously identified because the two bad records weren’t being used by actual users.

In these scenarios, core data quickly becomes utilitarian and not useful for broader service management insight or improvements.

My recommendations for implementing and maintaining good core data are as follows:

1. Identify sources of truth and system(s) of record for core data. This is an organizational best practice that certainly applies to ServiceNow as well. It’s rare that ServiceNow is or should be the source of truth or system of record for core data, perhaps other than local User Accounts, Groups and Group Memberships. For example, Active Directory is often the system of record for users across the enterprise. As an organization, identify these systems and implement integrations to receive data from these systems of record.
2. Identify and implement unique identifiers for data records across systems. Akin to my example above, and assuming you’ve done step 1, before importing data from a system of record you need to determine the unique identifier from the source system. Ensure that ServiceNow has this field in the destination table (and import set table), and set up your transform maps or other integration logic to coalesce on this field’s data. This is critical to ensuring duplicate records are not created.
3. Set up your imports and transforms to ensure core references are populated. You’ll need to order your transformations so that references to core table records on other core tables are populated – see the table diagram above. For example, in order to set the Location on the User record, the Location table needs to be populated first. However, if you want to use the “Contact” field on the Location records, you’ll need the Users in place. The reality is you’ll need to do multiple transformations or scripting to handle this circular logic. (Challenge yourself and try multiple transformations!)
4. Use the Production (“PROD”) instance as the system of record for core data across ServiceNow instances. Within your ServiceNow environment, PROD should be the system of record for this core data; sub-production environments will get their core data from PROD via clones. You can and should use sub-production for testing core data integrations, but the data itself should come from PROD. This includes Groups and Group Memberships wherever possible, save for a one-off when development requires a new Group that cannot exist in PROD prior to release. (Think about this as you do development – often a Group can be created in PROD without impact to process.) Using PROD as the system of record for this data means you have matching sys_ids of these records across your environments, and references to this data will not break in clones or code and configuration promotions. It is fine and expected to create additional core data records in sub-production instances – test users in your TEST instance for example – but use PROD as your source of truth.

Benefits of a Strong Core

For experienced system administrators and ServiceNow developers who are aware of and/or follow good practices, I haven’t mentioned anything they don’t already know. Sometimes it’s a matter of time and execution rather than knowledge. But what is sometimes not known is why this is important other than having good, clean data in your systems. What the larger benefits from having this data correct and available are.

It is hard to generalize all the benefits into clean, succinct bullet points. What you can do is move past the ideas of “number of tickets open/closed” and SLAs. Here are examples of the use of good core data, and hopefully it will trigger your imagination to think how it might apply to your organization and its business needs:

• Using the Department and Cost Center data tied to the User References on task-based application records, you can see what organizations are delivering services to what organizations, and use this data for charge-back accounting. For example, IT has completed 500 Incidents for Sales, or HR has fulfilled 300 service requests for Finance. With timekeeping and cost accounting, this data could be used to flow-down into cross-department accounting.
• Analyze trends of types of Incidents and Service Requests by Location (again by requesting User). This analysis could reveal Incident types that could be converted to Problems that are localized but could be avoided in the future.
• Group by core data points to determine if certain organizations or locations could benefit from a new or modified service (Incident deflection?)
• Align new hires and intra-company moves with Department so that standard packages can be pre-ordered (rather than asking each time). For example, the Sales Department employees always get certain access, software and hardware; this could be aligned with the Department so that when a new hire is requested who is part of Sales, the access, software and hardware can be automatically ordered.

For further reading, I did detail an example of what level of service can be provided, reported and analyzed when your core data is complete and used in a previous blog: Tier 5 Service Management.

Hopefully these examples trigger your own ideas about using referential core data to improve insight and improvements to your own organization.

Conclusion

At a cursory level, it is fairly obvious why having good data in ServiceNow is beneficial: clean is always better than messy. However, there’s more benefit than just cleanliness. Having accurate, up-to-date core data can help take your Service Management to “the next level” – both understanding what is occurring in your organization at a deeper level, and being able to make informed judgments about how to deliver Services that maximize benefit and minimize human effort. So start with your roots – good core data – and cultivate the ideas and features that will make your Service Management bloom.

In journalism, there’s the concept of the Five W questions whose answers are fundamental to getting the information needed:

• Who
• What
• When
• Where
• Why

I want to talk about what I call the “Three Ws of Development” in the ServiceNow realm. These three are: When, Where and Why. We’re going to skip the questions “Who” and “What”. Why? Because “who” is a question for hiring managers, recruiting, and resource vetting. And “what” is (too often) the focus of most if not all training and documentation. Do you need to get the current user’s ID on the client side? Check the API – that’s the “what”. Instead, I want to focus on some areas of development consideration that I feel are often neglected, and I’ll explain each and try to put them in context.

Most everyone in the ServiceNow world knows the basic system architecture, which is the same for almost all cloud-based applications:

On the ServiceNow side, there’s an Application Server that stores the compiled code, the files (images, etc.) needed by the application, and delivers content to the user’s browser when requested. The App Server connects to a SQL Database that stores all the data records, and in ServiceNow’s case, all the configurations and code customizations created by both ServiceNow and customers.

Now consider a “normal” transaction between these entities. We’ll use one that’s fundamental to ServiceNow: Accessing and updating an Incident record. The following shows all the component parts in this transaction:

1. Client makes a call to the server to get a specific Incident record
2. Query Business Rules determine if certain records should be restricted from view
3. Application queries the database for that record from the incident and related tables
4. Database returns required data to the application server
5. ACLs are evaluated and applied
6. Display Business Rules run logic and place data on the scratchpad (to use client-side)
7. Application Server provides browser with form, data, and all configurations based on 4-6

8. onLoad UI Policies and Client scripts run
9. User manipulates data, onChange UI Policies and Client scripts run
10. UI Actions run client-side, server-side, and sometimes both (first client, then server)
11. On Save, client sends form data to the Application Server
12. Before Business Rules manipulate record data prior to saving to the Database
13. Application Server sends data to the Database
14. After Business Rules run on any table (current, previous still available)

This is a broader “order of execution” list than ServiceNow provides in their documentation, which deals strictly with Business rules.

So how does this apply to our Ws discussion? Let’s discuss:

Where

In considering where to develop your configurations and customizations, you will almost always get better performance having them run on the Application Server rather than the client’s browser session. Observe the middle section of the diagrams above and the components that live in it. For example, when a user accesses a record, if you need to run logic and have the result available on in the client’s session, it is faster performance-wise to run the logic in a display Business Rule and place the result in the scratchpad rather than run the logic in a Client script, particularly if the latter would require querying the database after the record has been accessed and loaded in the client browser session.

Another important use case is security. All security should be done in ACLs, and only supplemented with UI Policies and scripts where needed. ACLs are the only true security, all other methods are posing as security and can usually be bypassed. For example, let’s say you have a Social Security Number field on a User record that should only be available (visible) to Human Resources, and should not be editable by anyone (it feeds from an HR/HCM system). This field should be secured with field-level read ACL for users with an HR role, another field-level read ACL marked false for all other users, and field-level write ACL marked false for all users. If you were to use a data dictionary level read-only marker, this could be bypassed by scripts running as the system. If you were to use a UI Policy or a Client Script to make it visible and/or read-only, this could be bypassed by list views and edits, server-side scripts, and Integration APIs.

In keeping with the last idea, it is also good practice to mimic your client-side logic on the server-side. For example, if you don’t want to force a field to be mandatory for every save, but want to run client side logic that prevents the save of the record without the mandatory field in a particular scenario, you should also create a server-side Business Rule that aborts the save and messages the user about the mandatory field. This way, your logic is enforced in list edits and server-side record manipulation, and not just in form views.

• List edits bypass “standard” Client Scripts and UI Policies
• External systems integrate directly with the Application Server

Note: You may have noticed that I haven’t mentioned List configurations like List Edit client scripts. These are also usable to “fix” the List edit issues mentioned, but it doesn’t fix server-side logic and integrations.

When

Going hand-in-hand with the Where is the When of development. Specifically, consideration should be paid to when in the lifecycle of the full transaction the development is best to exist. Consider the following Business Rule scenarios:

• You need to access information client-side that is not available on the current record being accessed but can be ascertained using the current record (e.g. querying information from another table using field data from the current record. The correct time (when) to run this logic is in a display Business Rule and place the needed information in the scratchpad.
• You need to manipulate field data for the current record before the record is saved to the database; for example, you need to multiply two integer fields from the current record and store the product in a total field on the same record. The correct time (when) to execute this is in a before Business Rule.
• You need to manipulate field data for a different record using the current record data, and either need to access the “previous” object, or you need the manipulation to be reflected in the user’s updated client session view. For example, you are updating a related record to the current record and the update needs to be reflected in a related list of the record form the user is currently viewing. The correct time (when) to execute this is in an after Business Rule.
• You need to manipulate field data for a different record using the current record data but it doesn’t need to be reflected in the user’s updated client session view, OR you need to trigger an integration whose updates don’t need to be reflected in the user’s updated client session view. The correct time (when) to execute this is in an async Business Rule. Async Business Rules create a Schedule Job that gets queued and executed as system resources become available.

Why

There are many other scenarios an experienced system admin/developer can think of. A key is to understand all the component parts of the transaction diagrams above; the goal is to configure and develop in the correct place, both where and when. Why? The benefits of developing in the correct time and place are:

• Performance: Following the when and where guidance will ensure the best system performance. Running logic on the server is almost always faster than on the client. Running logic asynchronously means the client, and therefore the user, means they don’t have to wait for the transaction to complete. Each time the transaction can avoid another database query means faster execution and less wait time.
• Security: Developing measures that secure fields and records, both visibility and editability (R and U in CRUD), in the correct place, means your instance is actually secure, versus appearing to be secure to an end user. After all, most if not all security breaches do not come from a human clicking and typing.
• Maintainability: Broader maintainability comes from following good and best practices. Consider a ServiceNow health check – what is the likely result of a system audit if you follow the practices suggested above versus using whatever means are available to create solutions.

Conclusion

ServiceNow provides many means to an end. Sadly, much of the documentation and training does not cover which the correct means to the end; rather, it simply details, “If you need to do X, go here and do this”. It doesn’t say why you should do it this way, or what the overall implications are to doing it that way. What I’ve tried to do is give you a baseline of how the system works, so you can understand where and when the correct places are to do X. Understand the baseline, and stop and think about it before everything you develop. I promise your platform, and hopefully your work life, will be better for doing so.

Happy developing!