Command Validation Tool

Command Validation Tool

What is the problem?

There is no tool available, which allows users to run commands on range of IP Address(es). In the absence of such tool, the validation of commands on IP address(es) against the credentials present in the instance do not happen. In case of discovery failure, where certain commands did not execute as expected, majority of the times users are asked to perform such validations by running the command from the virtual machine where MID server is installed.

What are we doing?

We have implemented an application, which allows users to run command(s) on a range of IP addresses belonging a certain operating system. Users also get to choose the MID server selection method for this operation. The output of this operation tells whether the execution is successful or not.

All commands are available via a Command List, which is populated daily by parsing all the discovery patterns present in the instance. The Command List contains commands belonging to Shell, WMI and SNMP protocols.

How it is being resolved?

With the introduction of the Command Validation Tool, which supports most popular operation types and protocols, users can run commands to validate them from within their ServiceNow instance.

Command Validation Tool supports the following operations:

1. Shell Command
2. WMI Query
3. SNMP Query (Tabular and Scalar)
4. SNMP Walk
5. HTTP GET (authenticated via Basic Authentication credentials)

Following MID selection methods are available:

1. Specific MID Server
2. Auto Select MID Server
3. Try All Eligible MID Servers

Users can provide IPv4 addresses as input in the following manner:

1. Comma separated IP Addresses
2. IP Range
3. IP Network
4. Mix of all the above

Every command validation results with the following information:

1. Status: Success/Failure
2. State: pending/processing/done
3. Result: Response from device/Reason for failure
4. Credentials used for command validation
5. In case of failure:
   1. ServiceNow Discovery specific error code
   2. Steps to troubleshoot based on the error code

Requirements and Use Cases:

1. Troubleshooting Discovery Failures: In case of failures in Discovery, Command Validation Tool can be used to re-run failed commands on the corresponding IP addresses using the corresponding MID servers. This can be used to detect the following:

1. Authentication issue
2. Authorization/Permission Issue
3. No valid credentials present
4. IP not reachable from MID
5. Command with wrong syntax
6. Problems with MID configuration

2. Onboarding New Infrastructure to ServiceNow Pattern Based Discovery: Command Validation Tool can be used to onboard new infrastructure as it gives the ability to validate all commands in a chosen pattern. By doing that, users can make sure that the new added IP addresses are discovered via Patterns without major failures.

3. Bulk Execution of a Command: Command Validation Tool can be used to run any command (as long as it is available via the Command List) on a range of IPs to check for a certain configuration.

Note: The application supports Domain Separation.

Feature Flow:

Command Validation Tool is available via the "Pattern Designer Enhancements" store app.

To get Command Validation Tool, first install Pattern Designer Enhancements via the list of plugins:

After the application is installed, once you refresh and search for ‘Command Validation Tool’ in the navigation menu, you will notice the relevant menu items:

Under Command Validation Tool, notice 4 modules:

1. Create New: To create a new Command Validation Task, which represent a single execution entity of Command Validation.
2. Command Validation Tasks: To see the list of tasks
3. Command Validation Task Results: To see the list of Command Validation Task Results.
4. Commands List: Contains all commands (of type SHELL, WMI and SNMP) parsed from patterns.

Creating a new Command Validation Task:

Clicking ‘Create New’ opens the following form:

The following information in required:

1. Mid Server Selection Method: Choose either of the 3 options which are:
   1. Specific mid server: you will be required. To choose the specific MID Server via the MID Server field.
   2. Auto Select mid server: first checks if the give IP has an affinity for a certain MID server by querying dscy_credentials_affinity table. If there is no affinity, it then selects the MID server whose configurational parameters have desired IP Range and Application set to include ‘Discovery’.
   3. Try all eligible mid servers: will select all MID servers which has affinity to the given IP Address and those whose configurational parameters have desired IP Range and Application set to include ‘Discovery’.

2. OS Class: Concerned OS (CMDB CI) class to which the IP addresses belongs. Users are expected to provide the correct OS Class, as this information is used by MID Server to decide which ports and protocol implementations to consider, when executing the command. Note that in case the wrong OS Class is provided, command execution will fail. Example: Choose Linux Server (cmdb_ci_linux_server) as OS Class when all IPs you entered as input are Linux servers.

3. Command Type: The type of command, you want to execute on the IP addresses. It can be Shell, WMI query, SNMP query, SNMP Walk or HTTP GET.

Notes:

1. Choose Command Type in such a way that it can be run on the chosen OS Server. The chosen IP Addresses should belong to the chosen OS Type. For example: you can’t run WMI Query on Linux Server.
2. HTTP Get command type only works when the credentials of API call is present as Basic Auth credentials. Token based credentials are not yet supported for this operation. This is also conveyed via field messages.

4. IP Addresses: Provide IP4 addresses as input in the following manner:
   4. Comma separated IP Addresses
   5. IP Range (1.1.1.1-1.1.1.100)
   6. IP Network (1.1.1.1/28)
   7. Comma separated IP Addresses, IP Range, and IP Network.
   8. IP and port (for HTTP Get command only)

Notes:

1. Users can provide IP Addresses, which were not yet discovered via discovery and are not available in CMDB. In this case, the customer should have proper credentials for the IP address, stored in the credentials table, for the command executions to be successful.
2. The IP range and IP network will only work when you have ‘ParseIPRanges’ script-includes imported to your instance. Please refer to the Script include attached to this KB.

5. Commands: the list of commands to execute on the specified IP addresses, which can only be chosen from the Command List table. Clicking on search icon will show only those commands from Command List table as per the chosen ‘Command Type’ and ‘Applicable OS Class’ field.

Notes:

1. The field message for ‘Commands’ tells you when the Command List was last updated, or if it is empty then how to populate it.
2. The syntax in which the command will be present in the Command List table is given below:

* Even though the tool has the capability to run ‘SNMP Walk’ and ‘HTTP GET’, the Command List table currently does not support them.

6. Validate All Pattern Commands: Enable this checkbox if you want to run all the commands from a specific pattern against the specified IP(s). Once the user checks this, the ‘Commands’ field will be replaced by the ‘Pattern’ field and the Command validation task will be executed depending on the chosen pattern rather than commands.
7. Skip Failover Command Steps: This checkbox gets enabled, when ‘Validate All Pattern Command’ checkbox is enabled. Checking this, would cause all skipping of all failover commands when command validation task is run to validate all pattern commands.
8. Pattern: This field is hidden by default and shows only when ‘Validate All Pattern Commands’ checkbox is enabled. All the commands present in the selected pattern will be run against the specified IP(s). Currently, only Infrastructure and Application patterns are available to be chosen, as the Command List is populated with commands from these pattern types only.
9. Select Hardware: Enabling this checkbox, will bring in a new field called ‘hardware’ and ‘Ip Addresses’ field will become hidden. The user can directly choose the Cis on which command validation needs to be run, rather than providing the ip addresses.
10. Hardware: This field is hidden by default and will be visible when ‘Select Hardware’ checkbox is enabled. This field lets you select the hardware Cis as per the chosen OS class directly on which command validation is intended to run.
11. Priority: The task with lower value will given high priority for execution by mid servers. The scheduler will pick the high priority Command Validation Task results record in pending state first.
12. Credential: Specify a specific credential to use for validating the commands via the command validation task. This is optional. If not chosen, then the MID Server selects the proper credential to use based on affinity and order.

Note: This feature is only supported from Utah release and onwards, and supports Shell, WMI and SNMP operations.

Once done, you can save the record, or immediately click on ‘Run Command Validation’ button to start command validation execution.

After clicking the ‘Run Command Validation’ button, Command Validation Task Result records are created for every provided command, IP address and MID Server.

For example: for 3 IP addresses, 2 commands and 1 MID Server configured in the task, 6 task results records are created.

Command Validation Task Result records are used to track the progress of Command Validation Task execution.

State of the Command Validation Task Result Records:

1. Pending: Yet to be picked by scheduler and dispatched to MID server
2. Processing: Scheduler has picked the record and dispatched to MID server for execution
3. Done: The execution of command is completed, and result is created.

Task progress and status can also be monitored within the task record.

Schedule Command Validation Probe launch Job:

The responsibility of this scheduled job to pick all pending Command Validation Task Results records, in a First-In-First-Out basis, on a regular interval (by default the scheduler is configured to run every 15 seconds) and send their content in batches (which is by default 10) to ECC queue via the Command Validation Probe. Users can customize the batch size and frequency as per their environment and instance and MID configuration.

Use the following system property to configure the batch size: command.validation.probe.batch.count

Command List Table:

This is introduced in Feb 2023 release.

The table holds shell commands, WMI queries and SNMP queries present inside all Infrastructure and Application patterns, including their shared libraries and extension sections.

From Aug 2023 release, the command list will also hold commands which contains temporary variables inside them. The temporary variables inside command start with ‘$’ and ends with ‘#’.

The Command List acts as an Allow-List and is the pool of commands which customer can select while creating the Command Validation Task.

The Command List is also used when the user chooses to perform ‘Validate All Pattern Commands’, as the Command Validation Tool takes all the commands in the Command List belonging to the selected pattern.

The Command List table is delivered empty. It’s populated by one of two methods:

1. A scheduled job, named “CVT-Populate CmdList and Mappings”, which runs once per day to keep the table up to date. Any changes made to patterns, either a step with new command is added or an existing step is deleted or modified, are being reflected by the end of the day, and no stale records are remained.
2. Using the ‘Refresh Command List’ button present in the list view of the Command List table, for cases the user made some changes and wants the command list to immediately reflect them. Below is the screenshot of how the command list looks, with the ‘Refresh Command List’ button on top. Note that the population process takes a few minutes to complete.

Notes:

1. The population process for the Command List only parses for commands of type SHELL, WMI and SNMP.

Cancel Command Validation Task:

This ability is introduced in February 2023 store release.

Once a command validation task is submitted, "Cancel Command Validation Task" button appears in the Command Validation Task Results related list, at the bottom of the Command Validation Task form. Clicking the button cancels all Command Validation Task Result records in pending state by changing their state to "cancelled". For those records, Command Validation Probe will not launch.

Note that command validation task results records in processing state are not cancelled since they're already being processed by a MID Server till they’re eventually completed.

List Pattern Command:

When the user opens the sa_pattern table and performs right click on any pattern, then they will see a new option ‘List Pattern Commands’, clicking on the same, will redirect to the Commands List table, where all the commands belonging to the chosen pattern will be listed. The user can further export the list.

When the user opens a pattern record in the sa_pattern table, they will see a new UI action, with the same name as mentioned above , clicking on that will redirect to the Commands List table, where all the commands belonging to the chosen pattern will be listed. The user can further export the list.

Show Commands Action:

‘Show Command’ related link is visible only when the ‘Validate All Command Patterns’ field is checked, and the task record is saved. Clicking on ‘Show Commands’ shows the list of commands that is configured to get executed as per the selected pattern in the Command Validation Task. This should help users to figure out what all commands are executed once command validation is submitted.

Refresh Temporary Variable mappings:

In case the user chooses commands from Command list which contains temporary variables, the  mentioned UI action will populate all the temporary variables in the related table ‘Temporary Variable Mappings’. The temporary variables are populated per IP/host, as the value of the temporary variables varies per host. Every record populated in the ‘Temporary Variable Mappings’ table will contain the temporary variables, step name, patter name and IP address. Only 'value' field will be populated with ‘No Value’ as the initial value. The user needs to fill the 'value' field only. On adding/removing IP addresses or commands selected and on saving the task record and clicking on the UI action will add/remove records in Temporary Variables Mappings table as per IP and Command specified. In conclusion, the user needs to make sure to use the Ui action, if the commands chosen has temporary variables present in them.

Note:

1. 
   
   1. Commands with temporary variables will only run for comma separated IP addresses and  specifying IP ranges or IP network as input is not supported.
   2. When running CVT with ‘Validate All Pattern Commands’, the commands containing temporary variable will be skipped and will not run. Only static commands will be run.
   3. A Command with temporary variables will only run when values for all the temporary variables are specified in the Temporary Variable Mappings table.

Troubleshooting Failed Command Execution

Command Validation Tool gives users steps to troubleshoot in case command execution fails. Once Command Validation Task Result ends status “Failure”, opening the record's form exposes a related link by the name ‘Open Suggestions to troubleshoot failure’.

When clicked, it opens a dialog box with suggestions on how to troubleshoot this failure. This suggestion is mapped to the error code present in the Command Validation Task Result record. Check the screenshot below.

Note that mappings of error codes to suggestions are delivered with the base system. There might be cases where error codes are not mapped to any suggestions. The message which appears in the dialog in such cases reflects that.

Working Example:

In the screenshot below, we configured Command Validation Task to run two shell commands, ‘date’ and ‘hostname’, on an IP address which belongs to a Linux server.

We chose a specific MID Server to run on and didn’t choose a specific credential – we let the MID Server to choose one for us.

The execution of both commands succeeded, and for each we’re able to see the result.

Release Information

Release: November 2023 Store

Application version: 3.3.0

Previous version: 3.1.0

Note: This is the fifth release of this scoped application.