Balancing Old Tricks with New Feats: AI-Powered Conversion From Enzyme to React Testing Library at Slack

Slack’s engineering team recently undertook a massive effort to convert over 15,000 frontend unit and integration tests from Enzyme to React Testing Library (RTL). This blog post summarizes the key points from their approach and insights on using AI to facilitate this conversion.

Table of Contents

  1. Introduction
  2. Challenges with Enzyme
  3. Automating the Conversion
  4. Abstract Syntax Tree (AST) Transformations
  5. Large Language Models (LLMs) Transformations
  6. Hybrid Approach: AST + LLM
  7. Evaluation and Impact
  8. Conclusion

Introduction

With the move to React 18, Slack found it necessary to transition from Enzyme, which lacked support for React 18, to RTL. This conversion involved significant engineering hours, prompting the need for an automated solution.

  • converting more than 15,000 Enzyme test cases, which translated to more than 10,000 potential engineering hours
  • one important requirement for our conversion was achieving 100%-correct transformations

Author: Sergii Gorbachov Senior Software Engineer at Vancouver Canada.

Challenges with Enzyme

Enzyme’s lack of support for React 18 and the extensive use of its methods in Slack’s codebase necessitated a switch to RTL. The transition required converting numerous test cases, which posed a significant challenge.

Automating the Abstract Syntax Tree (AST) Conversion

To automate the conversion process, Slack’s team explored Abstract Syntax Tree (AST) transformations and Large Language Models (LLMs) independently, and then combined these approaches for a more effective solution.

  • Collaborating with the DevXP AI team at Slack
  • opted for a pragmatic approach:
    • relatively satisfactory conversions for the most common cases, success rate of 45% automatically converted code
    • while resorting to manual intervention for the more complex scenarios

Attempt-1, Large Language Models (LLMs) Transformations

LLMs, like Anthropic’s Claude 2.1, were integrated into the workflow to aid the conversion. However, results varied significantly, with success rates between 40-60%. The AI’s inconsistency highlighted the complexities of the conversion task.

  • approach: created the prompts and sent the test code along with them to our recently-implemented API endpoint

Attempt-2, improved, Hybrid Approach: AST + LLM

Combining AST and LLM approaches, Slack developed a hybrid pipeline that improved the conversion success rate to 80%. This approach utilized the strengths of both methodologies to achieve more accurate and reliable conversions.

Kind of like a RAG-variation to me…

we reviewed our workflows and integrated most of this relevant information into our conversion pipeline. This is our final pipeline flowchart:

Hybrid Approach Pipeline

LLM control with prompts and AST

Options comparison

  1. thumb-down: universal prompt for all requests => but, led to a significant increase in the complexity of requests made to the LLM
  2. adopted: streamline the process by formulating a prompt with the most critical instructions

Instructions consisted of three parts:

  1. introduction and general context setting,
  2. main request (10 explicit required tasks and seven optional),
    • control the output of the LLM was the utilization of AST transformations.
    • Instead of solely relying on prompt engineering, we integrated the partially converted code and suggestions generated by our initial AST-based codemod. The
  3. followed by the instructions on how to evaluate and present the results

copy-paste below from original post

1 . Context setting:

`I need assistance converting an Enzyme test case to the React Testing Library framework.
I will provide you with the Enzyme test file code inside <code></code> xml tags.
I will also give you the partially converted test file code inside <codemod></codemod> xml tags.
The rendered component DOM tree for each test case will be provided in <component></component> tags with this structure for one or more test cases "<test_case_title></test_case_title> and <dom_tree></dom_tree>".`

2 . Main request:

`Please perform the following tasks:
1. Complete the conversion for the test file within <codemod></codemod> tags.
2. Convert all test cases and ensure the same number of tests in the file. ${numTestCasesString}
3. Replace Enzyme methods with the equivalent React Testing Library methods.
4. Update Enzyme imports to React Testing Library imports.
5. Adjust Jest matchers for React Testing Library.
6. Return the entire file with all converted test cases, enclosed in <code></code> tags.
7. Do not modify anything else, including imports for React components and helpers.
8. Preserve all abstracted functions as they are and use them in the converted file.
9. Maintain the original organization and naming of describe and it blocks.
10. Wrap component rendering into <Provider store={createTestStore()}><Component></Provider>. In order to do that you need to do two things
First, import these:
import { Provider } from '.../provider';
import createTestStore from '.../test-store';
Second, wrap component rendering in <Provider>, if it was not done before.
Example:
<Provider store={createTestStore()}>
<Component {...props} />
</Provider>
Ensure that all 10 conditions are met. The converted file should be runnable by Jest without any manual changes.

Other instructions section, use them when applicable:
1. "data-qa" attribute is configured to be used with "screen.getByTestId" queries.
2. Use these 4 augmented matchers that have "DOM" at the end to avoid conflicts with Enzyme
toBeCheckedDOM: toBeChecked,
toBeDisabledDOM: toBeDisabled,
toHaveStyleDOM: toHaveStyle,
toHaveValueDOM: toHaveValue
3. For user simulations use userEvent and import it with "import userEvent from '@testing-library/user-event';"
4. Prioritize queries in the following order getByRole, getByPlaceholderText, getByText, getByDisplayValue, getByAltText, getByTitle, then getByTestId.
5. Use query* variants only for non-existence checks: Example "expect(screen.query*('example')).not.toBeInTheDocument();"
6. Ensure all texts/strings are converted to lowercase regex expression. Example: screen.getByText(/your text here/i), screen.getByRole('button', {name: /your text here/i}).
7. When asserting that a DOM renders nothing, replace isEmptyRender()).toBe(true) with toBeEmptyDOMElement() by wrapping the component into a container. Example: expect(container).toBeEmptyDOMElement();`

3 . Instructions to evaluate and present results:

`Now, please evaluate your output and make sure your converted code is between <code></code> tags.
If there are any deviations from the specified conditions, list them explicitly.
If the output adheres to all conditions and uses instructions section, you can simply state "The output meets all specified conditions."`

Evaluation and Impact

Slack’s automated conversion tool achieved an adoption rate of approximately 64%, saving significant developer time. Evaluations showed that 80% of the converted content met quality standards, and around 22% of developer time was saved across 2,300 test cases.

Evaluation and Impact

Conclusion

The integration of AI and traditional AST methodologies proved effective in automating the conversion from Enzyme to RTL. This approach not only saved time but also maintained high-quality standards in the converted tests. As Slack continues to innovate, these methods offer promising potential for future projects.

For more detailed insights, you can read the full article on Slack Engineering Blog: Balancing Old Tricks with New Feats: AI-Powered Conversion From Enzyme to React Testing Library at Slack.