Expedoc

Engineering studies have been a tool our engineers use to troubleshoot problems, identify the root cause then verify a potential solution. The problem - the work is performed using informal methods using undocumented hardware/software (not reproducible) and cannot be used as objective evidence in the Design History File for our medical device.

After all this work, the engineers have a high level of confidence in their solution but then have to construct procedures to formally test the solution versus established acceptance criteria. The testing is rarely performed to same degree of rigor. Their intuition is that they are repeating the work they already completed. Management shares their frustration.

Our solution is to establish an Engineering Analysis Procedure. The FDA guidance identifies inspection, analysis and test as three acceptable methods for verifying a design. The procedure identifies three roles in the verification activity - Subject Matter Expert (author of the analysis), a Technical Expert (to peer review the acceptance criteria) and Quality Assurance Engineer (to ensure the procedure is followed).

After the subject matter expert completes the research needed to identify the root cause of the problem, he/she identifies the type of analysis to be performed (e.g. comparative, margin, etc), the baselines for the inputs to the analysis, the acceptance criteria and traceability to the requirement/specification used as a basis for the acceptance criteria. Each engineering analysis is uniquely identified using methods similar to those for identifying document and its revision.

For comparative analysis, we baseline the what is common between two systems/modules (apples to apples) and then baseline the differences.

The acceptance criteria must be complete, correct, unambiguous and not conflict with its design input requirements/specifications. In addition to the Pass/Fail criteria, the subject matter expert can add sample sizes, and number of runs (e.g. power up a system and ungracefully remove power 20 times).

Prior to performing the data collection for the analysis, the Subject Matter Expert, and Technical Expert approve the method to be used, the baselines for the analysis and verify the acceptance criteria. The Quality Assurance Engineer approves the protocol after verifying acceptance criteria is established and the procedure is being followed.

Data collection is performed by qualified personnel using the baselines and criteria established in the protocol. Individuals participating in the verification are identified using their signature and date.

The analysis of the data by the Subject Matter Expert results in a Pass or Fail result based on the acceptance criteria.

The form includes a place to record action items/resolutions. Design Change Request may be used to resolve issues identified during the verification.

When completed that results are approved by the Subject Matter Expert and the Quality Assurance Engineer.

An interesting lesson from our trail use of this procedure - Engineers' assumptions (what we know to be right) resulted in a Fail of one of our analyses. The benefit of the procedure is that we could adjust our approach to the reality of the situation - instead of using weights from the spec sheets for a comparative, we used actual weights of components - and quickly perform another analysis (Rev B).

To ensure this procedure was valid, we used the regulatory requirements applicable to design verification activities as goals for the process. Whether you are performing a test, an analysis or a peer review of a specification, the goals of the process and requirements for objective evidence are exactly the same.

This week, I was researching the usage requirements for validating an automated test suite. In question: Is an automated test that was not used for electronic record storage or electronic signature required to comply with any part of Part 11. The language in Part 11 that led to this question was that the tool is generating the test results and printing them.

Based on the following section in the FDA's guidance document, Part 11 requirements do not apply:

Guidance for Industry: Part 11, Electronic Records; Electronic Signatures -Scope and Application

III. DISCUSSION

B. Approach - Scope of Part 11

Narrow Interpretation of Scope

We understand that there is some confusion about the scope of part 11. Some have understood the scope of part 11 to be very broad. We believe that some of those broad interpretations could lead to unnecessary controls and costs and could discourage innovation and technological advances without providing added benefit to the public health. As a result, we want to clarify that the Agency intends to interpret the scope of part 11 narrowly.

Under the narrow interpretation of the scope of part 11, with respect to records required to be maintained under predicate rules or submitted to FDA, when persons choose to use records in electronic format in place of paper format, part 11 would apply. On the other hand, when persons use computers to generate paper printouts of electronic records, and those paper records meet all the requirements of the applicable predicate rules and persons rely on the paper records to perform their regulated activities, FDA would generally not consider persons to be "using electronic records in lieu of paper records" under §§ 11.2(a) and 11.2(b). In these instances, the use of computer systems in the generation of paper records would not trigger part 11.

Early in the project, the Incoming Inspection Review (IIR) procedure became a valuable tool when we integrated it into our design supplier management practices. It is a valuable tool because it establishes an expectation with our design partner that their payment are tied to our acceptance of the agreed deliverables. We established our acceptance criteria, including both functional requirements and quality system requirements, in the Statement of Work.

We use the IIR as gate for accepting deliverables from our design suppliers including: software designs documents; source code; software integration test procedures; and test results reports. Our contract with the design supplier tied our staged payments to each of their packages of deliverables. A bonus/penalty payment structure rewards our design supplier when their package was accepted earlier than scheduled, but then there was a cost if a package was accepted late. Action items from the IRR were prioritized so it was clear which action items had to be resolved to ensure acceptance. The incoming inspection reports were kept as evidence as required to demonstrate compliance with FDA 21 CFR 820.50, Purchasing Controls.

Initially, our top management was concerned that none of our four candidates for the work would accept this approach. When put into practice, three of our four candidates accepted our proposal.

For Reference: In section FDA 21 CFR 820.50, Purchasing Controls, the FDA requires medical device manufacturer to meet the following goals:

Each manufacturer shall establish and maintain procedures to ensure that all purchased or otherwise received product and services conform to specified requirements.

(a)Evaluation of suppliers, contractors, and consultants. Each manufacturer shall establish and maintain the requirements, including quality requirements, that must be met by suppliers, contractors, and consultants. Each manufacturer shall:
(1) Evaluate and select potential suppliers, contractors, and consultants on the basis of their ability to meet specified requirements, including quality requirements. The evaluation shall be documented.
(2) Define the type and extent of control to be exercised over the product, services, suppliers, contractors, and consultants, based on the evaluation results.
(3) Establish and maintain records of acceptable suppliers, contractors, and consultants.

(b)Purchasing data. Each manufacturer shall establish and maintain data that clearly describe or reference the specified requirements, including quality requirements, for purchased or otherwise received product and services. Purchasing documents shall include, where possible, an agreement that the suppliers, contractors, and consultants agree to notify the manufacturer of changes in the product or service so that manufacturers may determine whether the changes may affect the quality of a finished device. Purchasing data shall be approved in accordance with 820.40.

For introductory training courses on FDA regulations and FDA guidance, visit the FDA's training website

We ensure hardware testing produces valid results by managing its unique challenges. Hardware is subject to the environment, wear and damage.

Although the following steps may seem time consuming, our investment has be paid in multiples as we have virtually eliminated the time we wasted hunting ghosts in the system - invalid defects.

Training is our first step toward establishing control over the hardware. There are two parts to the training: electronic and assembly. The electronic training ensures all engineers, that handle the electronic components and printed circuit boards, are trained or otherwise qualified on procedures that minimized the probability of damaging components via electrostatic discharge (ESD) or electromagnetic interference (EMI). Manufacturing procedure training ensures the engineers understand how to use manufacturing work instructions to perform rework and to assemble modules and systems. For systems and modules for used for design validation, any changes made to the system must be made using detailed manufacturing work instructions or rework procedures.

Step 2: The lab is fitted to support the training. For handling the electronics, ESD mats, and static straps are installed on specific benches. The components and PSBs are stored in packaging and bins that protect them from static electricity and electromagnetic interference (EMI). For the mechanical activities, the lab has most of the tools and calibrated tools (e.g. torque wrenches) used in manufacturing. When a tool is too expensive, manufacturing engineering will perform the rework for us.

Step 3: We use configuration logs to track controlled components (e.g. a pneumatic hardware component that performs at a specific limit of the tolerance range), printed circuit boards, modules and systems. In the logs, we track changes (e.g. jumpers added, etc) made to the hardware from its as-build design, and we track activities that may have caused wear (e.g. HALT, EMI/EMC testing, reliability testing, etc). The logs provide the engineerings with the information needed to use the hardware that will give them valid test results.

Step 4: We segregate components, PCBs, modules, and systems into three groups: Sandbox, verification ready and validation ready. The budget of hardware was sufficient to provide engineerings with the hardware needed for prototyping, performing design verification activities and performing design validation activities.

Hardware for prototyping is segregated but not under any formal controls. It is the engineers' responsibility to follow the ESD procedures to ensure their prototyping is produces effective results.

Hardware for verification has a configuration logs and must be checked out of a locked storage locker. Plastic tamper-evident seals must be removed for engineers to have access to change modules and systems. Each seal is placed in the configuration log preceding its log entry. Configuration managers and quality engineers are responsible for witnessing changes made to the modules and system then logging the changes is sufficient detail that a qualified engineering could reproduce the change. After the rework is completed, a new seal is put into place. The new seal's unique identifier is logged. This provides us with a clear audit trail.

Hardware for validation begins with the same controls as the hardware for verification with the exception that all rework must be done using reviewed and approved rework instructions or manufacturing work instructions. Prior to performing any validation activities, the environmental conditions are recorded to ensure our testing is within the limits of our specified operating environment.

Finally, prior to each design verification or design validation activity, a test readiness review checklist is used to ensure we meet all of our requirements for valid testing. The two related items on the checklist confirm the configuration of the hardware is valid and that the required support equipments' calibration dates are current.

Note: Because established calibration of support equipment is a regulatory requirement, all lab equipment is managed to a strict calibration schedule. Due to the diligence of our calibration manager, I doubt I could find one uncalibrated piece of support equipment in the lab.