The perceived
performance of your ServiceNow instance is made up of these components.
Ø Application Server
response: Time for the application server to
process a request and render the resultant page
Ø Network latency
and throughput: Time for the
network to pass your request to the server and the response back
Ø Browser rendering
and parsing: Time for your browser to render the
HTML and parse/execute Javascript
Ø Instance Cache: The amount of system resources available for
processing
This document
will outline basic troubleshooting steps to try to isolate the source of slow
response times.
1.
Poor Network Performance: A fairly common scenario, users may
experience poor network performance especially when there are multiple locations,
in an on-premise installation, where the users must VPN in or where additional
devices such as firewalls or proxies come into play.
Symptoms
- One or more users will report intermittent degraded
performance
- If multiple users report issue it may only occur at
specific times or locations
- If a single user reports issue it may only happen
at specific locations
Verifying The Issue
- Logout, completely clear cache, delete all cookies
and try logging in again
- Log in from a different browser
- Log in from a different machine on the same network
If the issue
still persists it may be an issue with the network. To verify:
1.
Navigate to Transaction
Logs (All User)
2.
Find all transactions for a
particular users session
3.
Find all
transaction where the Client response time or Network time is
excessive (>5000ms)
4.
If there are excessively
high Network times then those should be flagged and you
should look for additional logs around the same time
5.
If additional log entries
are found this trend will likely point to a network issue
6.
If there are excessively
high Client response times you should compare this time
to the other “time” fields (excluding Browser Time)
7.
If the sum of the other
time fields does not closely match the Client response time then it is
likely a network issue.
Once you have
determined you have a network issue it is necessary to look for trends to
isolate the problem.
·
Do affected users have the
same location or network?
·
Is the issue isolated to a
specific time of day?
·
Do any users access
ServiceNow through a proxy?
·
Is the issue intermittent
or consistent?
Potential Root Causes
- Overloaded or malfunctioning networking equipment –
This would almost certainly affect other applications as well.
- Overloaded proxy – If users access ServiceNow
through a proxy, is the machine correctly configured and will it support
the current load?
- Insufficient bandwidth.
- Poor wireless signal.
Fixes
- Unfortunately for networking issues there is not
much that can be done from ServiceNow. Infrastructure may need to be
upgraded or reconfigured to support the load.
2. Mis-Configured ACL: Usually occurring for
non-admin users or those with specific roles, mis-configured ACLs can impact
forms but especially lists where ACLs run multiple times.
Symptoms
- One or more users will report excessively long load
times for lists or forms.
Verifying The Issue
- Navigate to Transaction Logs (All User).
- Find all transactions for a particular users
session.
- Find all transaction where the ACL time is
excessive (>100ms).
- For ACLs
even a fraction of a second is a very long time.
- Search for any other transactions for the same URL.
- If the same URL has a trend of long ACL times there
may be a mis-configured ACL.
Validate the ACL is mis-configured
- Based on the URL you should be able to find the
affected table. Usually it will be in the name however if it is not a
standard List or Form page (for instance a UI Page) you may need to look
through the contents of the page to see the source table(s).
- View the ACLs for the table.
- Look for any ACLs with Scripted Conditions.
- Verify the Script does not have nested for loops or
any other violations of best practice.
Potential Root Causes
- Poorly written Scripted ACLs
- Excessive number of records being accessed
Fixes
- Scripts should be re-written to be as efficient as
possible.
- Lists and queries should be properly limited where
necessary.
- Additional filters can be added to lists to cut
down on the number of records returned.
- Potentially re-structuring tables or columns to
simplify the script.
3. Mis-Configured Business Rule: A
very common scenario is when Business Rules are not written correctly or
several rules are written poorly and stack up to create a poor user experience.
Most commonly when a user creates or updates but this can occur any time a
business rule runs with the exception of asynchronous types.
Symptoms
- One or more users will report excessively long wait
times saving or loading a record.
Verifying The Issue
- Navigate to Transaction Logs (All User).
- Find all transactions for a particular users
session.
- Find all transaction where the Business rule
time is excessive (>2000ms).
- Find all transaction where the Business rule
count is excessive (>10).
- Search for any other transactions for the same URL.
- If the same URL has a trend of long Business rule
times or counts there may be an issue with the BRs.
Validate incorrect Business Rules
- Open the Business Rules for the particular table.
- Look for Business Rules which match the conditions
reported by user (when saving, when creating).
- Look for all After or Before rules (Async rules will
not apply since they do not stop the user session).
- Verify there are not an excessive number or rules.
- While having many Business Rules is not bad or
uncommon the efficiency of those rules needs to increase with the number.
Otherwise many rules that are somewhat slow can add up to a long wait time
for end users.
- Verify the content of the Business Rules.
- Like ACLs you should avoid making an excessive
amount of queries or nested queries in a BR.
Potential Root Causes
- Multiple poorly written Business rules
- Example: 10 or more rules which each take 250ms
will end up 2.5 seconds for the user, this is only a part of the process
though so the over all time may end up being excessive.
- Single poorly written Business rule
- A single rule taking more than 1000ms will directly
impact the users experience
Fixes
- Scripts should be re-written to be as efficient as
possible.
- Minimizing the number of GlideRecord queries and
number of results returned.
- Potentially moving some scripts to asynchronous
where possible
4. Under-performing Node: A
very uncommon scenario but also one that is not fixable except by ServiceNow.
This will usually involve ServiceNow restarting and possibly decommissioning
the node.
Symptoms
- Multiple users will report excessive wait times performing
any action in the system
- Issue will usually be intermittent
- Issue will occur more frequently at peak times
Verifying The Issue
- Navigate to Transaction Logs (All User).
- Find all transactions where the Response Time is
excessive (>5000ms).
- Add the System ID column
- Verify that no other cause (Network, BR, ACL) is
affecting the performance.
- This may manifest as a longer Session wait
time, Semaphore Wait time or SQL time
- You may also try to eliminate transactions that are
expected to take longer, such as those with larger outputs (Output length)
- You may also eliminate those requests which affect
multiple System IDs
- If there is a node (System ID) which
is constantly under-performing and where the same requests execute quickly
on other nodes it may be isolated to that node.
- You may open a HI server ticket to confirm the node
is not functioning correctly
Potential Root Causes
- If the issue consistently happens even after
restarting the node hardware may be defective
- Many other causes are unknown especially since we
do not have access to the back-end
Fixes
- Ask ServiceNow to restart the node via HI ticket or
Support
5.Browser Performance: A very common scenario especially for environments with
older browser (specifically Internet Explorer). Poor performance is almost
exclusively limited to IE but there are scenarios where Chrome and Firefox will
not work correctly.
Symptoms
- One or more subset of users will report a
performance issue especially loading forms or custom pages
- Issue will be consistent and easily reproduced
- Issue will only occur on a certain browser or
machine
Verifying The Issue
Before verifying check that the user is
actually using a Browser on their desktop or laptop and it is not related to a
mobile device.
- Navigate to Transaction Logs (All User).
- Add the User Agent column.
- Find all the Transactions for a particular users
session who is reporting the issue.
- OR Find all transactions where the Response Time is
excessive (>5000ms).
- Eliminate any transactions which can be explained
by other issues.
- Try to correlate any poor performance to a
particular User Agent.
- Look up the Users User Agent string
to determine which browser they are using. UserAgentString.com
Validate the browser is not running in “Compatibility Mode”
- For Internet Explorer ask the User to verify their
version by going to “About Internet Explorer”
- If the user reports a version which does NOT match
the User Agent reported in the transaction log they may have compatibility
mode on which should be disabled.
Validate the Browser is supported
- Older versions of Internet Explorer (<10)
generally do not perform well enough to have a good User Experience in
ServiceNow
- Versions of Internet Explorer (<9) may not work
at all
Validate extensions are not interfering
- Ask the user to disable extensions OR run an
instance of the browser in private browsing mode (this usually disables
all extensions)
- If the issue does not persist with extensions
disabled review the extensions to see if any may be causing an issue
- Disable extensions one a time to verify.
Validate no errors are being reported
- Ask the user to verify there are no errors in the
JavaScript console. Most browsers have a developer console which will
output all JavaScript errors.
- Believe it or not some errors are expected but an
excessive amount especially those which report missing resources could be
affecting the user experience.
Potential Root Causes
- Unsupported Browser.
- Browser running in compatibility mode.
- Browser extension impacting performance.
- GPO pushing browser setting which impacts
performance.
- GPO pushing system setting which prevents caching.
- Cache disabled through browser setting.
- Browser unable to reach resources – this may also
manifest as the UI appearing strange.
Fixes
- Upgrading to the latest browser version.
- Disabling Compatibility Mode.
- Removing or resolving extension or browser configuration.
- Verifying network configuration (firewall or proxy)
allows users machine to retrieve correct resources.
6. Excessive and Incorrect Client
Side Scripts: Another
common scenario is for the user to see poor performance due to too much form automation
on the client side. Offloading some work to Business rules or re-architecting
it for performance will usually resolve these.
Symptoms
- One or more users will report an issue with forms
taking a long time to load.
- Users may reports jittery or unresponsive form when
changing values.
-
Verifying The Issue
- Navigate to Transaction Logs (All User).
- Find all transactions for the table or form where
the issue is reported which have an excessive (>2000ms) Client script or UI Policy time
- Note that these values will only include
those scripts run when loading.
- If script time is consistently long there may be an
issue with the number or efficiency of the script.
Validate the script is impacting performance
- For either/both UI Policies and Client Scripts disable
all scripts.
- Enable each script testing the form.
- Take note of the load time. If a single script
increases the load time dramatically there may be an issue with the
script.
- If each script or policy increases the load
slightly resulting in a long load time, there may be an issue with the
quantity of scripts.
Validate the form layout
- Validate there is not an excessive amount of
fields,tabs, related lists or embedded lists on the form.
- Validate there are not custom Formatters present.
- If there are an excessive amount of controls on a
form it will directly impact the load time for the user.
Potential Root Causes
- Too many Scripts or Policies.
- Poorly written Scripts.
- Excessive amount of AJAX transactions in scripts.
- Some AJAX is OK, but while those transactions are
occuring synchronously the user is unable to do anything.
- Synchronous AJAX should only be used when the
result is NECESSARY for the user to proceed.
- Too many fields present on form.
- Too many lists or embedded lists.
Fixes
- Re-writing Scripts to be more efficient.
- Refactoring or combining UI Policies.
- Changing synchronous AJAX to asynchronous where
possible.
- Changing some form automation to run as Business
rules where possible.
- Removing fields or creating additional views to
only show necessary fields.
- Removing unnecessary fields or lists.
7. Incorrect User
Exception: A
perfectly valid expectation is that every page will respond quickly but not
every page in ServiceNow is equal. Sometimes 5, 10 or even 30 seconds is
reasonable when the system is crunching 1000s or rows to display in a report.
Symptoms
- User reports performance issue for particular
pages.
Verifying The Issue
- Navigate to Transaction Logs (All User).
- Find all the transactions for the users session.
- Find any excessively long transactions (>5000).
- Take note of the URL.
- If the user is loading
home.do
or a report it may be valid to expect wait
times greater than 5
seconds.
Potential Root Causes
- User is loading a Report or Dashboard.
- User is loading too many records on a list.
Fixes
- Explain to the user the wait times are valid.
- Reconfigure the report to have additional
conditions.
- Reset the user preference to display less rows.
- Change the users Home page if possible.