New York: Automated Test Framework Changes

#New York Release , #Automated Test Framework

Dave Slusher

4 minute read

As we discuss the new features of New York, it is only natural I should discuss first one of my great loves in the platform - the Automated Test Framework. Every release since it was introduced expands the capabilities of this tool and New York is no exception. Here are the highlights but the full list is here.

Parallel Testing and Mutually Exclusive Tests

In previous releases, ATF ran effectively a single thread of testing. When there was a queue of tests, the next test would run on the system and if it required a client-side step the next available client-side test runner would pick it up. With the New York release, testing is now run in parallel. When running large suites, tests will be run in parallel until the testing consumes roughly half of the resources of the instance.

This does imply that the automated testing design principle of isolating them and building them to run independently is even more critical. The system will automatically build in mutal exclusion rules when the tests are acting on the same record. The developer also has some control over this as well. Tests can identify sets of other tests with which they are mutually exclusive, which will prevent them from running at the same time.

Related List for Mutally Exclusive Tests

Add Mutually Exclusive Test

Create a User Test Step

Previously, most ATF tests began with a step in which the test impersonated a user on the system with the roles and groups to be tested. As of New York, that strategy has been deprecated in favor of using the Create User test step.

The Create User test step asks for a first and last name, roles and groups for the user and transparently handles all the other details. This allows for better isolation of the tests from the data of the system. Previously, if the user impersonated for the test had roles changed on the system, it might break the tests. Now, tests created with the Create User steps are independent of the data that exists on the system.

Request an Instance: New York

After the user is created by the test step, that record is available to subsequent test steps via the Data Pill Picker. If you want to create a record or otherwise use the data of the newly created user, they are available to the rest of the test.

Select the User from Data Pill

Roll back in Browser Sessions

There is a change to the way roll back works in New York. When activity is being recorded from the client-side test runner to rolll back, it now records all activity from all browser tabs. It is very important that you don’t do any modification to the system during the period from the same browser session that your client-side test is running from. These changes were brought about in order to speed up test execution. When verifying the activity came from the test frame, the system had to wait for the end of it because the page may be refreshing itself. Now all activity is recorded for rollback from the session, whether or not it came from the frame with the test runner.

Overall this is a positive change to increase the speed at which tests run, which helps you out in proportion to the number of tests you have to run. When you are actively developing, however, be sure not to develop from the same browser and session the test runner is executing a test or you will get behavior you don’t like. Always running the client-side test runner from an incognito window might be a pretty good practice now.

Attachment Test Steps

Some functionality of the system requires that an attachment exists, such as a business rule that enforces that an attachment exists before a record can change state. Now this functionality is easier to test. There are two flavors of this - one that uploads via the form for a UI based client-side test and one that uses the server API to upload. In either strategy, the test will wait for the attachment to be uploaded before the test proceeds to the next step.

Conclusion

With each release, the amount of system logic that can be tested by ATF increases. Whether testing for basic system operation at upgrade time or using ATF to actively develop functionality, it is a tool that improves each release.

Enjoy exploring and please share your thoughts below.

Comments