The Evolution of Artera’s Quality Assurance Strategy

The story of Artera’s quality assurance strategy has followed the needs and maturity of Artera itself. There are many stories yet to be told in finer detail, but a high-level retrospective of the relationship between the company’s phases and the expectations of quality assurance will highlight the evolution of this pair while contextualizing the lessons we have learned along the way.

The Beginning

At the beginning of my tenure five years ago, Artera (at the time known as Well Health) staffed a dozen or so engineers and was very much in the “start-up mode mindset.” By this, I mean that we had a strong focus on delivering new functionality quickly, often in response to a specific sale. The engineers were excellent at writing and delivering new features yet secondary functions like operational excellence and release sanity were not a primary focus.

Back then, there were no dedicated quality assurance professionals, few automated tests, and a dependence upon the expertise of a few individuals to report issues. The issues that customers reported were, appropriately, handled urgently, but often at the cost of on-going work that would be in service of the company’s growth. It was for this very reason that Artera brought me on to build the quality assurance department. The challenge ahead of us in engineering was multifaceted. I had questions like, “How do I demonstrate the value of a dedicated quality assurance professional in an organization that has historically only had software engineers dedicated to making the product? How do I get everybody on board to test more rigorously and more often? What is the correct path forward for the company’s quality strategy?”

Over the course of this phase, we updated our processes to allow for exploratory testing and automated testing to occur before marking items as done, authored a suite of API tests (the first of which being pictured above), and hired a small team of testers and test engineers. Similar to the organization at the time, quality assurance focused on working quickly while tending to the noisiest sources of issues. While this was necessary and sufficient for the first year of quality assurance’s existence, the company would soon embark on its next phase.

Hyper-Growth Phase

The company would soon explode with growth, as many tech companies did in 2020. The engineering organization nearly tripled in size in the span of a year. The product organization would become its own, distinct entity. Quality assurance would similarly triple in size over this span. As a whole, Artera had cemented itself in the marketplace and it was at this moment our opportunity to grow the business quickly surfaced.

Artera’s move into the hyper-growth phase both necessitated, and was serviced by, quality assurance’s own growth. It became clear at the beginning of this era of the company that everybody needed to take responsibility for quality, not just those with “test” in their titles. As such, new strategies developed in this time. Software engineers significantly improved the automated tests in their purview (unit, component, and integration tests) by making them faster, more comprehensive, and easier to write. Product Managers supplied clearer requirements and worked more closely with us on the technical side of the house. The efforts of the Quality Assurance Analyst had stopped scaling so we trained them to be Software Engineers in Test and stopped hiring those whose sole responsibility was to manually test the application.

All of these efforts were in service of the company’s growth. Because of our collective focus on scalability and expansion, we were able to quickly spin up new product teams that delivered new functionality that is core to the platform to this day. These new product teams were delivering code with the highest quality and quickest velocity to date. Put shortly, a focus on quality was a key component in driving successful business outcomes for Artera.

Mature Growth Phase

Starting in December of 2021, Artera’s step into the “mature growth” phase is marked by three important events in the “Innovation Organization” as we now call it. Firstly, the company heavily invested in DevOps. Secondly, the quality assurance team created our first true end-to-end tests. Finally, new technical leadership came in. These three events together would prove to be the foundations for our current technological excellence.

Up until this point, the DevOps organization hovered somewhere between one and three people. Because of limited staffing, these individuals were often firefighting instead of proactively improving the platform. Together with the new leadership in place, we more than tripled the size of the team which enabled us to vastly improve the underlying systems, release cadence, and reliability of the product.

As these systems improved, so did the quality strategy. The team wrote end-to-end tests that gave us full confidence in the code we were writing. Those tests were then configured to execute in the cloud, obviating issues and results to anybody who needed the tests. As such, forward progress became easier and our velocity increased. This increased velocity kicked off a virtuous cycle because it necessitated a faster release process and this faster release process necessitated better, faster, more-comprehensive tests.

This virtuous cycle can be observed through our release metrics. The team began inspecting the successes and failures of our releases in June of 2023. Since then, our data-driven approach has guided us towards tackling important problems that demonstrate real value to the company. The following graph depicts the percentage of releases that did not go out because of an issue in an end-to-end test.

Simply, in June of 2023, 10% of releases failed because of an issue with our tests. These failures were a source of stress for everybody! Engineers were frustrated their code was not going out in a timely manner and the business depended upon new development to drive revenue goals. We took this problem seriously and at the time of writing, the current percentage is 0%.

All of this is to say that everything and everybody grew together in this mature growth phase. Gone are the days of quarterly releases that were difficult to get across the finish line. Today, we release ten times a day, a feat unthinkable in December of 2021 when this phase started! Importantly, this maturity was not the sole effort of any individual or one initiative. It took the business investing heavily in engineering excellence, new leadership to create room for us to grow, and a quality assurance strategy that met the needs of the business. This investment was only possible because we proved the value of good quality, and continue to do so. Because of this continued focus and investment in quality, Artera went without a product-related severity-1 issues for 17 months.

Present

I am very proud of the state of quality at Artera. The teams take quality concerns very seriously and consistently demonstrate a sense of ownership.

We extensively test every single code change that we make. Notably, we purposefully avoid directing to the teams how to run their quality assurance program. Our philosophy in the Innovation Organization is that the teams are empowered to work how they would like. This philosophy-driven approach has ensured that each team is engaged with what, why, and how they test. Furthermore, by allowing teams to take different approaches, we benefit from a constant innovation of the tools and processes used to assure quality. These innovations are then shared amongst the other teams for potential implementation. The best ideas find their way to be used across the organization because of their own merit, not because leadership mandated its usage.

Generally, the quality program looks the same for each team because of our ability to innovate and adopt the best practices. Tactically, the teams will include a QA professional in all agile ceremonies (refinement, sprint planning, standup, etc.), develop a sufficient test plan for each story, divvy up the responsibilities of that test plan in an efficient manner, write all of the necessary tests and commit them with the product code, engage in exploratory testing, and monitor the releases for unforeseeable issues. If things go wrong, the teams are excellent at adapting their practices to avoid making the same mistakes again.

Future

The future of quality assurance at Artera will be about learning from our lessons, listed below, while expanding what and how we test. With our elimination of severe product issues, the organization is exploring further maturity in how we test. We will focus on testing the systems for scalability, making our integrations more extensible, and improving the speed of the pipelines. Furthermore, there are departments within the organization, even outside of the Innovation Organization, that stand to benefit from the five years we have been improving quality!

Lessons Learned

• Use the preferred programming language of the organization

At my two previous jobs I worked in Python for test automation. Here at Artera the programming language we use is Typescript (when I first started we were still on Javascript). Had I originally leveraged Javascript when developing the automated tests I would have benefited from increased collaboration because the barrier to entry for the software engineers would have been smaller. Instead, I was one of two people at the company fluent in Python and this forced me to take on a lot myself. Notably, this is a “lesson learned” as we have actually learned from this lesson! When we set off to modernize the end-to-end suite in 2021, the first decision we made was to use Typescript. The aforementioned boon to collaboration is not simply a hypothetical but a proven benefit!

• Lead with the execution

If I had to do it again from scratch, I would have first developed an automated testing pipeline instead of the tests themselves. The reason for this is simple: we would have built the habit of taking the automated tests seriously while also ensuring that each and every test immediately provides value. The immediacy of the value from tests in an automated pipeline is very motivating!

• Build the tests as you build the product

You might already have a backlog of tests you would like to write. If your organization can take time to address this tech debt, do so! The realities of business goals might get in the way of your initiative, as it did for me. Artera had great success in improving our quality assurance program when the quality assurance goals aligned with the business goals. When the automated tests were first developed, I naively tried to disappear into a corner and just tackle the backlog of tests that the engineering organization needed. This worked for a week or two but I quickly lost momentum when new development needed the attention of the quality assurance team. The QA team shifted its mindset to one of “build the tests as we go” and found that we could both backfill tests and cover new development by strategically including the backfill work alongside new initiatives.

• Take the “Build tests as you go” mindset further: do not delineate the difference between “Software Development” and “Software Testing”

Artera is a quality-first organization, but this evolved with time. At first, the team had traditional notions of “testers test” and “engineers create.” Presently, there is no such delineation. Artera is an organization that now fully embodies the notion of everybody owns quality. Product Managers write high-quality requirements while collaborating often with the software engineers. Software engineers are accountable for excellent code and unit tests that support this code. Software Engineers in Test leverage their quality-assurance-mind to foresee issues by planning each ticket’s testing, creating end-to-end or integration tests, facilitate exploratory testing, and assist in a ticket’s hand off. Because a focus on quality is ingrained into every step of the software development lifecycle, there is no distinction between “development” and “testing.” Testing is a continuous activity within development, just as programming is!

• Focus on perfect tests

Yes, perfect. The whole organization must trust that every test that is written and executed is both necessary and sufficient. A test that fails incorrectly cannot be trusted. Superfluous tests aren’t necessary and waste valuable time and resources. You wouldn’t deploy production, user-facing code that only works 98% of the time. Treat your tests with same, or more, rigor!

Now, the intention here is not to lambast the team when something is not perfect. Rather, the focus on perfect tests is to create a culture that takes testing seriously. Think of this notion as a mathematical limit: the team will approach whatever line is drawn in the sand. By making the line “perfection,” you do not limit the organization by drawing a less-stringent line.

Artera builds perfect tests by subjecting them to a three-step process. Firstly, the new or changed tests must not introduce regression issues so we run all affected tests when we make changes. Secondly, any new or changed tests must go through the “Consistency Checker,” as we call it. This is a special job that exercises the test many times, currently 20 times, to ensure that the test is resilient. Thirdly, all changes are subject to a peer review process to ensure that the code itself is of the highest quality.

• Constantly highlight the value of quality

A traditional view of quality assurance is that it’s a department or set of processes that is a necessary business cost. This view is limited yet very easy to accept because QA’s contributions do not have a direct throughline to the new features being developed by the software engineers, product managers, and designers. To counteract this reductive notion, ensure that everybody in the business truly understands the value that quality assurance adds to the business. Find areas where quality assurance improved the customer experience, helped an engineer find a bug early in the software development life cycle, or redirected the course of a project away from a looming pitfall. Then, express these moments in true-to-life dollars if you must. Suddenly, what was once perceived as a cost is now a shrewd business investment! Successfully doing this motivates the company to continually invest in valuable initiatives in the name of quality.

---

About Artera:

*Artera is a SaaS digital health leader redefining patient communications. Artera is trusted by 800+ healthcare systems and federal agencies to facilitate approximately 2.2 billion messages annually, reaching 100+ million patients. The Artera platform integrates across a healthcare organization’s tech stack, EHRs and third-party vendors to unify, simplify and orchestrate digital communications into the patient’s preferred channel (texting, email,  IVR, and webchat), in 109+ languages. The Artera impact: more efficient staff, more profitable organizations and a more harmonious patient experience.*

Founded in 2015, Artera is based in Santa Barbara, California and has been named a Deloitte Technology Fast 500 company (2021, 2022, 2023), and ranked on the Inc. 5000 list of fastest-growing private companies for four consecutive years. Artera is a two-time Best in KLAS winner in Patient Outreach.

For more information, visit www.artera.io.

Artera’s blog posts and press releases are for informational purposes only and are not legal advice. Artera assumes no responsibility for the accuracy, completeness, or timeliness of blogs and non-legally required press releases. Claims for damages arising from decisions based on this release are expressly disclaimed, to the extent permitted by law.