6 minute read
The Istanbul release included the Automated Testing Framework. This is the first time that the platform itself has included any significant mechanism for automated testing. There are third party solutions such as Cerna’s CapIO but none actually included in the product until Istanbul. Let’s take a look at this framework in depth to see what it includes.
Tests: A list of all Tests cases defined on the system. A Test is basically one transaction from beginning to end and is composed of a series of Test Steps.
Suites: A list of all the Suites. A Suite is a set of Tests that form a logical unit, such as all the tests for a given sprint or feature. Suites and Tests have a many to many relationship.
Test Results: The outcome of the previous runs of individual tests.
Suite Results: The outcome of previous runs of test suites.
Client Test Runner: Opens a window for the client test run worker. This must be open in order for any test with a client side component to execute.
Test Run Queue: The list of tests that are waiting for a client test worker to pick them up.
Suite Run Queue: The list of suites waiting for a client test worker.
Properties: A shortcut to the properties that control execution and verbosity of test runs. Rarely or never should these be enabled in production.
Step Configurations: This is the list of individual actions that compose tests. More details on these later, but the short version is that these can be client or server, an action to take or an assertion of a value.
Step Environments: These are the environments in which a test step can be run. At this time, the choices are Client (called b UI’) or Server.
Test Templates: These are sets of frequently co-occurring tests that can be used as a basis for creating new tests.
Step Configuration Categories: These are related to the Step Environments in that they currently break them down between Client (called b Form’ here) and Server.
Table Cleanup: This controls the retention policies for the results created by automated tests. It is defined in seconds and defaults to 30 days.B
The most basic unit of the whole framework is the Test Step, defined in Step Configurations. Istanbul ships with 19 predefined Step Configurations, 10 on the Server side and 9 on the Form (Client) side. These Step Configurations are a mix of actions necessary to exercise functionality (Impersonate, Record Insert, Set Field Values) and those that assert against expected states and values. Either can result in a test failure; an assert if the specified condition is false and an action if the results don’t match the configured expectation (again, more later on this.)
These Step Configurations are provided with the system in Istanbul. It is possible for end users to create their own Step Configurations in cases where the provided steps do not have the desired capabilities. The vision for the system is that each release will include an ever larger pool of Step Configurations but at this point it is possible to run into edges of functionality. In subsequent blog posts we will examine how to do that.
The real unit of work of the Automated Test Framework is the Test. Typically one would configure this to encapsulate a single transaction or unit of work of the system. We’ll walk through one of the example Tests that is pre-built to get a feel for how this works. We will look at “Multi-Batch Test.”
Step 1: Impersonate User - This will typically be the first step of any automated test. This is to force the session into using a specific user, rather then the user executing the tests. There could conceivably be a set of test users with different roles or groups in order to verify operation for every permutation of roles. In that case, the tests may be run a number of times with different users and different expected values and states.
Step 2: Open a New Form - This will be a commonly used test step for exercising typical Form based transactions. The configuration requires specifying the Table for which the form is opened, and optionally a view can be specified.
Step 3: Field State Validation - This validates the UI state of the Glide Form elements. This is only verifes the state, not any values for this step configuration (there exists a Field Values Validation step.) This is accomplished by defining sets of elements that should be in any given state. The options are Visible/Not Visible, Read Only/Not Read Only and Mandatory/Not Mandatory. Because this test will be executed by a Client Test Runner, client-side code will execute so any UI Policy defined will execute.
Step 4: Set Field Values - This step sets the values on the form. Again, since this is being executed client-side in an actual running browser, if these value changes trigger changes in the states of Fields, these can be asserted on in subsequent steps to verify those changes. This test step sets the values, but does not submit the form. That comes next.
Step 5: Submit Form - This actually executes the form submission as if the “Submit” button was clicked. This will attempt to submit the form, and can then assert nothing, assert that the form was successfully submitter or assert that the submission was successfully cancelled by client-side script. The test will return a success state if the configured expectation was met.
Step 6: Run Server Side Script - In this example, the test step exists to show how the server-side step in the middle forces the client-side steps to break into two batches of client execution. In practice these steps can be used to run any available scripting, so they can manipulate data and also execute Jasmine test scripts. There are examples of the latter in provided example tests as well.
Steps 7 through 9 - These are repeats of the form process on a different table.
One thing to note is that these tests is that they can use dynamic data that carries from step to step. On applicable test step configuration elements, there will be an icon to the right that allows for results of previous steps to be used. For example, we submitted the form in Step 5. In the Set Field Values of Step 8, we could use the table and/or sys_id of that resulting record in this form. This would allow for creating associations in Reference fields and later asserting the relationship existed.
A reasonable amount of functionality of the ServiceNow interface can be exercised and asserted via these provided Test Steps. In particular, when the test is to verify form based activity of loading forms, manipulating values and submitting data and then asserting results, much of this can be accomplished with only the pre-built steps. In subsequent posts, we will explore how to extend this by creating custom Step Configurations.
In the next post, we will explore the actual running of these Tests and organizing them into Suites and the configurations thereof. There is a lot to dig into here, so expect many more posts in the topic in the near future. As always, if you have questions or interesting tests you have created, feel free to reply to this post or email me at dave.slusher@servicenow.com. On to testing adventures!
Share this post
Twitter
Facebook
Reddit
LinkedIn
Email