Tipping the Scales of CSV Maturity Models

Tipping the Scales of CSV Maturity Models

The landscape of testing and CSV maturity models in Life Sciences is complex and varied. There are a number of reasons for this which we have discussed in other blog posts (subscribe, won’t you?), so I don’t want to get into that too much here, but it’s topic that is filled with nuance, individual requirements, legacy mindsets, and technical and process uncertainty. Sometimes, assessing where you are as an organization is just as important as understanding how to get where you are trying to go.

In a recent presentation on “The Evolution of Paperless Validation” that Tx3’s Dori-Gonzalez Acevedo did at a virtual KENX conference, this was a topic that she covered very effectively and comprehensively. Essentially, most organizations have a Testing Maturity Model that they follow and have goals around, as well as existing practices and processes around CSV. Sometimes, unfortunately, those two models are not always aligned, and not being on the same page is extremely detrimental to the overall health and effectiveness of an organization’s Quality, Validation, IT, and Business units. 

By the way, even though I credit her at the bottom, I would like to just take a second to note Dori as the real mastermind behind this content. Everything you are about to read from here on out is content of her authoring. I’m just converting it to a blog post, so all credit is due to Dori. I would highly recommend connecting with her on LinkedIn here.

Let's dive in, shall we?

To start, many of you probably know the TMMi (Testing Maturity Model Integration) has been around for many years and that it helps to define and benchmark an organization’s testing methodology. As a testing methodology matures within an organization, so do the tools and team members involved in the process, and a point becomes reached where quality metrics achieve a state wherein processes can really begin to become optimized.

It’s important to note that the Testing Maturity Model exists regardless of the testing method used. Whether your organization is operating within a waterfall, agile, or hybrid approach, the testing model focuses on how well your organization is positioned to conduct quality testing. While this is useful, it’s not the only piece of the puzzle in Life Sciences. There is a regulatory compliance component that must be accounted for, and yet there often isn’t a lot of discussion around a validation maturity model as it relates to integrating with a testing maturity model.

In response to this, at Tx3 we have developed a validation maturity model that helps our clients easily assess the health of their validation practices relative to their testing maturity. Through this, we are able to benchmark key elements to assess the current state of things, as well as to develop an executable plan to achieve optimized processes and move up the various levels of the maturity model we will discuss here.

Basically, there are 5 levels of validation maturity, and no doubt your organization falls into one of these. Identifying where you are is a crucial step for getting your teams aligned and figuring out a move forward-path. Let’s review each of these levels and discuss what we view the desired end state to be.

Validation Level 1: Initial

Tipping the Scales of CSV Maturity-4

Validation Level 1: Initial. The starting point. This is where many teams started their journey and it’s where many teams still are today. There are a number of factors that contribute to this, but what we need to look at is how to move beyond this phase. 

Here at Validation Level 1, an organization is operating under 100% waterfall testing with waterfall-based validation models in place and everything is tracked and managed on paper. It is extremely controlled by toll gates or controlled phases. The testing itself is completely manual and conducted on paper with wet signatures. It is document-centric, generally using scanned paper, which is ultimately deposited and stored in some type of repository, whether that be physical or electronic.

With each subsequent change control, all documents are reopened, evaluated, modified, and retested. Rinse, wash, and repeat. As you can imagine, this equates to a lot of paper, numerous manually enforced checkpoints, and a lot of physical signatures that are required.

At this level, there is a heavy price to be paid with the resources and time that need to be allocated to facilitate this type of process. While it’s very robust and controlled, which can be utilized to moderate success with stagnant products or applications that do not undergo change frequently, it is extremely rigid. The trouble with this is that rigid things can easily break or even be ignored.

Ultimately, Level 1 is not scalable and comes with a lack of transparency as well. Where does all the information live? How, and where, and with who does all the information lie? It’s on paper somewhere (assuming the lack of governance and manual effort hasn’t resulted in human error and deviation), but someone needs to search for and review the required information, which makes for long, tedious, and time-consuming audits.

That being said, Level 1 is an essential starting place for many teams. However, it should not be a place to stay. Although it doesn’t necessarily have to be the starting point, if it is, it should still never be the end goal of any organization.

Let’s weight the scales a bit further in the right direction…..

Validation Level 2: Defined

Tipping the Scales of CSV Maturity-1At Level 2: Defined, we start to see some forward progress. Here, there is still a clear and documented Level 1 process in place that remains very document-centric, but a shift has been made to managing that in a more electronic format. The move to Level 2 and a more electronic-based document process is often a fairly organic one that makes everyone’s life a little bit easier.

Typically, this is accomplished in haste by just dumping final documents into a PDF and subsequently into some type of document management system. Even though the documents are editable for revision control, teams are still left maintaining “final” PDF versions of documents, meaning those changes still have to be manually tracked. Someone still has to review the data and review the manual tracing of all that documentation within the data to ensure all components and relationships are present - this is very prone to error. While the electronic format does help a bit in an audit, it doesn’t advance the data input and the review process itself.

Level 2, much like Level 1, is still often considered far too rigid. Further, the volume of application change controls that are required by organizations today will quickly outpace this level. The limited technology solutions deployed at Level 2 is also not aligned with ever-evolving business needs. In addition, the implementation of risk methodology that is typically needed can’t be deployed here.

Ultimately, Level 2 is often quickly seen as limited in both function and efficacy, moving organizations to make the push to Validation Level 3: Blended.

Validation Level 3: Blended

Once organizations reach Validation Level 3: Blended, we have found that they can remain here for quite some time while delivering both high levels of quality, with moderate ease of compliance management. This is what we define as a balance between paper-based and paperless validation methodologies. 

At this level, we also begin to see data-driven application lifecycle management tools deployed that can easily accommodate both controlled electronic records, and electronic signatures. We now also see the ability to utilize flexible, risk-based methodologies applied throughout all levels of testing. There is also now an ability to handle both waterfall and hybrid approaches to testing and validation which will certainly satisfy the document-centric validation model.

However, a challenge as we see teams move into this phase is that the shift from Level 2 to Level 3 is often driven by IT or Technical groups as they start to get a taste of the benefits of electronic processes found in Level 2. The problem becomes that at that level, the demand for additional technology solutions quickly outpaces the existing platforms that are in place there.

As a result, it’s important to take into account that if organizations don’t start taking the appropriate steps to move to Level 3, it can start to pose business issues like seeing ad-hoc electronic signature solutions, rogue spreadsheets, non-standard and siloed tools popping up in parts of your organization, and even shadow IT groups. To avoid this, moving into Level 3 needs to be planned and executed effectively across the organization.

On the side of effective and cohesive (rather than rogue) technology rollout and utilization, the implementation of application lifecycle management tools becomes necessary in order to adequately deliver and support hybrid testing methodologies. While there is still typically a mix of paper-based and electronic documents for validation plans or summary reports, the difference is at Level 3, things like testing assets, requirements, and specifications have truly become electronic and paperless.

As I mentioned, many organizations can stay at this level for many years, honing their proficiencies, and maximizing their processes, tools, and their people. Yet while Level 3 makes IT and Quality happy most of the time, there is still room for growth and efficiency gains on both the testing and validation side of things, and this is where we start to see technology further tip the scales and enable more effective processes.

Validation Level 4: Transformed

In order to move to Level 4: Transform, the organization now implements a paperless solution with very few document-centric deliverables remaining. For the most part, it can now adjust to the rapid changes in technology, applications, testing tools, and regulatory changes. This is what enables organizations to start shifting towards piloting agile and test automation frameworks - even though some groups may still be hanging on to a sole, few documents as long as possible. 

Now, Technical groups are pushing Quality groups to stretch beyond their comfort zones. This in turn will drive the need to shift to more complex testing and change control frameworks. This is also where many organizations will start to see a split, or begin working in more of a bi-modal fashion where agile or DevOps teams start to break away from the standard processes and start piloting new groups of technologies.

The primary concern here is that the validation process will no longer be able to keep up with the rapid changes that become required to stay on pace with application development and testing needs. To avoid this becoming the case, Quality groups will need to shift their mindset and processes to support a fully data-driven validation model across the enterprise in order to overcome the final hurdle to the next step.

Validation Level 5: Optimized

At Validation Level 5: Optimized, an organization has finally achieved a fully paperless solution. Not just paperless, however. The important distinction at this level is that electronic documents have also been left behind and we now see validation deliverables being managed and achieved in a fully data-driven methodology. This is enabling the full utilization of best-of-breed technologies, coupled with robust risk-based approaches, and test automation frameworks - all based now on data-driven and agile methodologies.

In Level 5, Quality and Validation groups need to trust the data, understand the data, and be able to interpret the data. They need to be regularly educated in both technology and processes, and this needs to be advocated and mandated by strong leadership. Achieving this level will require a cultural mindset shift in order to move away from more legacy-based approaches to software testing and validation. Ongoing, concerted effort across the enterprise is crucial. 

On the flip side, IT and Technical groups need to commit to the Four Eyes principle to ensure that data integrity is truly intact. ALCOA Plus principles are still required, will always be required, and will still require some level of human intervention despite being electronic or automated. It is also important that IT organizations acknowledge and plan for the fact that not all testing can be automated. There will always be elements of manual testing. More importantly, however, is that the content matters most. The data itself matters most, and this can never be forgotten.

The back and forth between Level 4 and Level 5 is the hardest and most difficult to achieve, as it requires a complete top-down and bottom-up embrace of a cultural shift in validation and testing that is filled with new technologies, methodologies, and processes.

To sum up, moving an organization’s validation methodology through these phases is not something that is accomplished overnight, nor should that be the goal. This is a process that can take years to effectively implement and should be strategized and executed in a phased approach. The planning and rollout need to include considerations for all of the following:

  1. People/Culture: The folks that will actually be doing the work. Personnel will need to be enabled and trained early and often on the new tools they will be interacting with and the processes that they will need to adhere to. This is also where the shift in cultural mindset will need to be addressed.
  2. Process: Process and methodology shifts will need to be implemented to support the changes that will be required from both a testing and compliance standpoint, as well as to support the new tools that will be an integral component.
  3. Technology: This is the effective implementation, utilization, and standardization of tools to support the shift in methodologies, as well as enable the systematic reproduction and governance of defined processes.

Of course, all of this is a component of a larger discussion. Luckily, that is a conversation that the SME’s here at Tx3 have with our clients and prospective clients on a daily basis and we regularly help teams set up and execute a strategy to move through these levels.

As a starting point, we recommend folks complete our SDLC Modernization Assessment Questionnaire. Through this activity, we will evaluate the current state of your SDLC and validation practices, and will then build and supply a strategic roadmap for achieving a desired end state which we will review with your team.

You can complete the assessment questionnaire via this link:

Take the Assessment Questionnaire

If you would like to learn about some of the broader points on this topic, please make sure to view a recent presentation that Dori Gonzalez-Acevedo did on the “Evolution of Paperless Validation”. What we covered in over the course of this blog was a great component of that conversation, but she also addressed a client use case, went into more detail on testing and validation maturity models, as well as discussed the planning phase that needs to occur for a successful shift.

That presentation can be viewed here:

Thanks again to Dori Gonzalez Acevedo for putting this great content together and making this blog possible. Again, if you didn't connect with her via the link at the beginning of the post, reach out and connect to her on LinkedIn here.

Share to: