As we discuss the new features of New York, it is only natural I should discuss first one of my great loves in the platform - the Automated Test Framework. Every release since it was introduced expands the capabilities of this tool and New York is no exception. Here are the highlights but the full list is here.
Parallel Testing and Mutually Exclusive Tests
In previous releases, ATF ran effectively a single thread of testing. When there was a queue of tests, the next test would run on the system and if it required a client-side step the next available client-side test runner would pick it up. With the New York release, testing is now run in parallel. When running large suites, tests will be run in parallel until the testing consumes roughly half of the resources of the instance.
This does imply that the automated testing design principle of isolating them and building them to run independently is even more critical. The system will automatically build in mutal exclusion rules when the tests are acting on the same record. The developer also has some control over this as well. Tests can identify sets of other tests with which they are mutually exclusive, which will prevent them from running at the same time.
Create a User Test Step
Previously, most ATF tests began with a step in which the test impersonated a user on the system with the roles and groups to be tested. As of New York, that strategy has been deprecated in favor of using the Create User test step.
The Create User test step asks for a first and last name, roles and groups for the user and transparently handles all the other details. This allows for better isolation of the tests from the data of the system. Previously, if the user impersonated for the test had roles changed on the system, it might break the tests. Now, tests created with the Create User steps are independent of the data that exists on the system.
After the user is created by the test step, that record is available to subsequent test steps via the Data Pill Picker. If you want to create a record or otherwise use the data of the newly created user, they are available to the rest of the test.
Roll back in Browser Sessions
There is a change to the way roll back works in New York. When activity is being recorded from the client-side test runner to rolll back, it now records all activity from all browser tabs. It is very important that you don’t do any modification to the system during the period from the same browser session that your client-side test is running from. These changes were brought about in order to speed up test execution. When verifying the activity came from the test frame, the system had to wait for the end of it because the page may be refreshing itself. Now all activity is recorded for rollback from the session, whether or not it came from the frame with the test runner.
Overall this is a positive change to increase the speed at which tests run, which helps you out in proportion to the number of tests you have to run. When you are actively developing, however, be sure not to develop from the same browser and session the test runner is executing a test or you will get behavior you don’t like. Always running the client-side test runner from an incognito window might be a pretty good practice now.
Attachment Test Steps
Some functionality of the system requires that an attachment exists, such as a business rule that enforces that an attachment exists before a record can change state. Now this functionality is easier to test. There are two flavors of this - one that uploads via the form for a UI based client-side test and one that uses the server API to upload. In either strategy, the test will wait for the attachment to be uploaded before the test proceeds to the next step.
With each release, the amount of system logic that can be tested by ATF increases. Whether testing for basic system operation at upgrade time or using ATF to actively develop functionality, it is a tool that improves each release.
Enjoy exploring and please share your thoughts below.