REAL WORLD TESTING RESULTS REPORT 2024

GENERAL INFORMATION

Plan Report ID Number: 20231121all
Developer Name: AllegianceMD Software, Inc
Product Name(s): Veracity
Version Number(s): 9.1
Product List (CHPL) ID(s):  15.02.05.2672.ALLE.01.01.1.220117
Developer Real World Testing Plan Page URL:  https://allegiancemd.com/real-world-testing
Developer Real World Testing Results Report URL:  https://allegiancemd.com/real-world-testing

CHANGES TO ORIGINAL PLAN

STANDARDS UPDATES  

SUMMARY OF TESTING METHODS AND KEY FINDINGS

Test Plan:
The test cases that were included actions by various user types to capture the required data and workflows. In most scenarios, real world patient data was used to confirm compliance with the things such as successful transmission statuses for some certification criteria requirements. Some standards were tested via manual inspection and/or with ONC-recommended test tools.

Test Methods:
User actions are sent to an event  distributed event streaming platform. User events are recorded. This method is superior to picking certain clients for testing as it provides real time data across all clients and in case of error, it provides error logs to help us trouble shoot any issue with the system.

Key Milestones

Challenges faced during testing

Key challenges necessitating changes to the test plans included a small userbase exacerbated by significant percentage of customers using Veracity 9.1 either not participating in government payers or utilizing the low-volume exception to MIPS/QPP participation, or accepting Medicare payment penalties as opposed to the labor of fully utilizing the tested functions of the software.

RWT Measure #1. Number of Transition of Care

Associated Criteria: 315(b)(1)
Care Coordination: 170.315 (b)(1): Transitions of Care

Testing Methodology: Reporting/Logging

Measurement Description:
This measure is tracking and counting how many C-CDAs are created and successfully sent or received from the EHR Module to or from a 3rd party during a transition of care event over the course of a given interval.

Received:
Care Settings:
Nephrology, Pain Medicine, Pediatrics, Internal Medicine, Family Medicine, Sleep Medicine, Neurology, Podiatrist, Surgery, Psychiatry, OBGYN, Infectious Disease, Endocrinologist.
Testing Results:
Clinics Queried: 48
Reporting Interval: 05/01/2024 to 05/31/2024
Reporting Interval: 31 days
Total C-CDA Received for all Clinics: 153
Denominator = Total no. of Incoming CCDA  with no error – Total no. of Invalid Incoming CCDA
Denominator = 153
Numerator = Total no. of  successful Incoming CCDA from Denominator
Numerator = 153
Receive: The total percentage of successful incoming CCDA: 100%

Sent:
Care Settings:
Podiatry, Rheumatology, Urology, Pain Medicine, Surgery, Nephrology, Family medicine, Behavioral Health
Testing Results:
Clinics Queried: 9
Reporting Interval: 05/01/2024 to 05/31/2024
Reporting Interval: 31 Days
Total C-CDA Received for all Clinics: 9
Denominator = Total no. of sent CCDA sent with no error – Total no. of Invalid CCDA
Denominator = 35
Numerator = Total no. of Sent CCDA from Denominator
Numerator = 35
Receive: The total percentage of successful sent CCDA: 100%

Analysis and Key Findings
Our clients do not regularly share data through C-CDA files so we have few records of exchanged C-CDAs. Clients usually use this service to receive from hospital health systems.
Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the time range and care settings.

Relied Upon Software
Phimail

RWT Measure #2. Clinical information reconciliation and incorporation

Associated Criteria: 315(b)(2)
Care Coordination: 170.315(b)(2) Clinical information reconciliation and incorporation

Testing Methodology: Reporting/Logging

Measurement Description:
Demonstration of incorporating problem list, medication list and allergy medication list reconciliation from discrete problems, medications and medication allergies parsed from a C-CDA in referral providers environment. Verfity the successful reconciliations of parsed discrete data in the referrall provider’s environment into the problem list, medication list and allergy list in the clinical summmary.

Care Settings:
Ambulatory Surgery

Testing Results
Clinics With Records: 1
Reporting Interval: 07/01/2024 – 07/31/2024
Number of C-CDAs: 12

Testing Metric/Measurement:
Number of C-CDAs: 12

Denominator = Total no. of Incorporated CCDA sent with no error – Total no. of Invalid CCDA
Denominator = 12
Numerator = Total no. of Incorporated CCDA from Denominator
Numerator = 12
Receive: The total percentage of successful imported CCDA: 100% 

Analysis and Key Findings
Our clients do not regularly share data through C-CDA files so we have few records of Direct exchange C-CDAs. One clinic in the clinics queried was heavily using the CCDA to import patients.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the date range and care settings.

Relied Upon Software
None

RWT Measure #3. Number of Rx Messages Successfully Sent

Associated Criteria: 315(b)(3)
Care Coordination: 170.315(b)(3) Electronic prescribing

Testing Methodology: Reporting/Logging

Measurement Description:
This measure is tracking and counting how many NewRx and CancelRx electronic prescriptions were created and successfully sent from the EHR Module to a pharmacy destination over the course of a given interval.

Care Settings:
Behavioral Health, Cardiology, Family, Fertitlity, Hormone Replacement, Infectious Disease, Internist, Neurologist, OB/GYN, Pain Doctor, Podiatry, Rheumatology, Sleep Doctor, Surgeon, Urgent Care, Palliative Medicine, Endocrinology, Nephrology, Urgent Care, Pediatrics, Dermatology, Plastic and Reconstructive, Psychiatry.
.

Testing Results:
Clinics Queried: 168
Reporting Interval: 07/01/2024 – 07/14/2024
Total Electronic Prescriptions Sent for all Clinics: 35,722

Denominator = Total no. of NEWRX sent  – Total no. of Invalid NEWRX
Denominator = 38,720 – 2,998= 35,722
Numerator = Total no. of NEWRX sent with no error
Numerator = 35,722
The total percentage of successful NEWRX sent: 100% 

Analysis and Key Findings:
Electronic prescribing is a very popular and widely used feature in our EHR. 

Non-Conformities or Errors Discovered:
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the date range and care settings.

Relied Upon Software
None

RWT Measure #4. Number of Patient Batch Exports

Associated Criteria: 315(b)(10)
Care Coordination: 170.315(b)(10) Data export

Testing Methodology: Reporting/Logging

Measurement Description:
This measure is tracking and counting how many batch exports of C-CDAs were successfully performed by the EHR Module over the course of a given interval.

Care Settings:

OB/GYN, Family care

Testing Results:
Clinics Queried: 1
Reporting Interval: 06/01/2024-12/31/2024

Single export:
Single Export Clinics: 1
Denominator: (Total no. of Downloaded Data Exports + Total no. of shared Data exports) – Total no. of
Data exports failed due to wrong data entry by users.
Denominator: 2+1=3
Numerator: Total no. of successful Manual Data exports + Total no. of Successful Automated Data
Numerator: 3
Exports percentage completed without any errors.: 100% 

Mass Export:
Mass Export Clinics: 2
Denominator: (Total number of full export)
Denominator: 2
Numerator: Total no. of successful  Data exports
Numerator: 2
Exports percentage completed without any errors.: 100% 

Analysis and Key Findings:
The results show that only three clinics used the export function: one clinic performed a single export, and two clinics executed a mass export. A “single export” refers to the process of exporting data for an individual patient, while a “mass export” is typically initiated by the clinic when transitioning to a different system.

Non-Conformities or Errors Discovered:
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the care settings. 

Relied Upon Software
None

RWT Measure #5. Clinical quality measures – record and export

Associated Criteria: 315(C)(1)
Clinical Quality Measures: 170.315(c)(1)—record and export

Testing Methodology: Reporting/Logging

Measurement Description
Demonstration of the ability to export patient data recorded in the EHR for a given patient population in a QRDAI format. 

Care Settings:
Geriatric Medicine, Family Medicine, Internist, Infectious Disease, Primary Care

Testing Results:
Number of Clinics With Records: 7
Reporting Interval: 07/01/2024 – 12/31/2024
Number of QRDA1 batches exported: 628
Denominator: Total no. of attempts to generate QRDA 1.
Denominator: 628
Numerator: Total no. of successful attempts to generate QRDA 1.
Numerator: 628
Total % of successful exports of Valid QRDA 1 = [Numerator/ Denominator]* 100
Total % of successful exports of Valid QRDA 1 = 100%

Analysis and Key Findings
Because we are certified in nearly all ONC eCQMs, our clients have a wide variety of options to choose from. While CMS will accept the top six scoring eCQMs for their MIPS Quality Reporting, users can submit more than 6, however because of the size of the clinics, many clinics choose claim based or optout all together.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the reporting interval and care settings. The date range was extended due to low number of clinics.

Relied Upon Software
None 

RWT Measure #6. Clinical quality measures – import and calculate

Associated Criteria: 315(c)(2)
Clinical Quality Measures: 170.315(c)(2)—import and calculate

Testing Methodology: Reporting/Logging

Measurement Description
Demonstration of the ability to calculate quality measures for the patient data and measures.

Care Settings:

Behavioral Health, Surgery, Endocrine, Pain Management, Neurology, Family Medicine, Internist, Nephrology, Pediatrics, Podiatry, Orthopedics, 

Testing Results:
Number of Clinics: 45
Reporting Interval: 07/01/2024 – 07/14/2024
Denominator: Total no. of QRDA 1 patient files imported into EHR – Invalid user action.
Denominator: 1280 
Numerator: Total no. of unique patients from Denominator who are considered for calculating
Numerator: 1280
Total % of successful patients QRDA 1 files imported and calculated = [Numerator/ Denominator]* 100
Total % of successful patients QRDA 1 files imported and calculated = 100%

Analysis and Key Findings
Because we are certified in nearly all ONC eCQMs, our clients have a wide variety of options to choose from. While CMS will accept the top six scoring eCQMs for their MIPS Quality Reporting, users can submit more than 6.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the date range and care settings.

Relied Upon Software
None

RWT Measure #7. Clinical quality measures – QRDA-III Report

Associated Criteria: 315(c)(3)
Clinical Quality Measures:  170.315(c)(3)—report

Testing Methodology: Reporting/Logging

Measurement Description
Demonstration of the ability to generate QRDA 3 files.

Care Settings:
Endocrine, Rheumatology, Nephrology

Testing Results:
Number of Clinics: 4
Reporting Interval: 07/01/2024 – 09/30/2024
Number Of QRDA-III Exported : 5
Denominator: Total no. of QRDA-III patient files imported into EHR – Invalid user action.
Denominator: 4
Numerator: Total no. of unique patients from Denominator who are considered for calculating
Numerator: 4
Total % of successful patients QRDA-III files imported and calculated = [Numerator/ Denominator]* 100
Total % of successful patients QRDA-III files imported and calculated = 100%

Analysis and Key Findings
Clinics record CQM results and submit QRDA-III files to CMS the following year. The number of clinics exported to QRDA-III was below average due to the reporting period wasn’t on the due dates of CMS submission.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the reporting interval and care settings. The date range was extended due to low number of clinics performing QRDA3 export.

Relied Upon Software
None

RWT Measure #8. View, download, and transmit to 3rd party

Associated Criteria: 315(e)(1)
Patient Engagement: 170.315(e)(1) View, download, and transmit to 3rd party

Testing Methodology: Reporting/Logging

Measurement Description
Login to patient portal to view clinical summary, create and make a valid C-CDA available to download or transmit to 3rd party.

Care Settings:
Podiatry, Family, OB/GYN, Surgery, Endocrine, Fertility, Sleep Medicine, Oncology, Rheumatology, Pediatrics, Behavioral Health, Cardiology, Weight Loss, Pain Medicine,  Medical Spa, Neurology.

Testing Results:
Number of Clinics: 69
Reporting Interval: 07/01/2024 – 07/14/2024
Number Of Download or View or Transmit CCDA : 989
Denominator = Total no. patients who attempted to download or view or transmit CCDA – Data Entry/Authentication Errors .
Denominator= 989 – 7=982
Numerator = 982
view or download or transmit is performed successfully.
Total Percentage = [(Numerator)(Denominator)]*100
Total Percentage = 100%

Analysis and Key Findings
Some patients login and click view CCDA and download it or transmit to a 3rd party. However this is rarely used by patients. Some clinics ask their patients to login to the portal and download the CCDA.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the care settings.

Relied Upon Software
MyPortal.MD

RWT Measure #9. Transmission to public health agencies-immunization registry

Associated Criteria: 315(f)(1)
Public Health: 170.315(f)(1) Transmission to immunization registries

Transmission to immunization registries Measure 1:
Testing Methodology: Reporting/Logging

Measurement Description
Demonstration of the ability to add an immunization to a patient and generate a VXU message for an administered immunization and transmit it via HL7 2.5.1 to a public health agency successfully. Verify the immunization log of successful VXU message transmissions.

Care Settings:
Pediatrics, Family Medicine.

Testing Results:
Number of Clinics With Records: 2
Reporting Interval: 07/01/2024 – 07/14/2024
Number Of transmissions : 38
Denominator: Total no. of Vaccination information transmitted – Errors due to invalid data entry
Denominator: 38-0=38
Numerator: Total no. of Vaccination information transmitted successfully
Numerator:38
Total %:  [Numerator/Denominator] * 100
Total%: 100%

Vaccine Hx and Vaccine Forecast Measure: 2: Total percentage of successful requests executed to get
Vaccine history and Vaccine forecast.

Transmission to immunization registries Measure 2:
Testing Methodology: Manual

Measurement Description:
Total percentage of successful requests executed to get Vaccine history and Vaccine forecast.

Care Settings:
Test family practice with non-real patient data.

Testing Results:
Number of Clinics With Records: 1
Reporting Interval: 12/01/2024 – 12/01/2024
Number Of Files : 1
Denominator: : Total percentage of successful requests executed to get Vaccine history and Vaccine forecast.
Numerator: Total no of valid messages manually tested
Denominator:1
Numerator:1
Total %:  [Numerator/Denominator] * 100
Total%: 100%

Analysis and Key Findings
Few clinics are using the feature. If the end user uses the state inventory, lot# has to be accurate. No clinics are using the forcast feature

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the  care settings . We had to manual test measure 2 due to no usage reported by any client.

Relied Upon Software
None

RWT Measure #10. Transmission to Public Health Agencies

Associated Criteria: 315(f)(5)
Public Health: 170.315(f)(5) Transmission to public health agencies — electronic case reporting

Testing Methodology: Manual

Measurement Description
Demonstration of the ability to generate a transmission to public health agencies depending on a table of trigger codes per encounter.

Care Settings:
Test family practice with non-real patient data.

Testing Results:
Number of Clinics With Records: 3
Reporting Interval: 07/08/2024 – 07/08/2024
Number Of Files : 3

Analysis and Key Findings
We don’t have any clinics using this feature. A manual test was performed. We have a system that can query encounters for trigger codes. Once the encounter was flagged, an eICR CCDA was generated. Since we implemented FHIR, We are utilizing eCRNow for 2024.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the care settings. 

Relied Upon Software
None

RWT Measure #11. Application access – patient selection

Associated Criteria: 315(g)(7)
Application Programming Interfaces: 170.315(g)(7) Application access— patient selection

Testing Methodology: Reporting/Logging

Measurement Description
The data required for the measure is collected for the duration of 15 days as mentioned in the schedule. Using the data logs we collect information as mentioned below to calculate the accuracy and successful responses percentage. Using these logs we can monitor if Veracity is able to accept the API key and secret. When the API receives a request from another software component or service containing sufficient patient information to identify the patient, the API returns a patient ID 

Care Settings:
Family Medicine

Testing Results:
Number of Clinics With Records: 1
Reporting Interval: 11/15/2024 – 11/30/2024
Number Of Clinics : 1
Total no. of API requests received for the patients: 7
Total no. of invalid API requests for the patients: 4
Total no. of successful of valid API requests for the patients: 7
Total no. of successful API responses from Veracity: 100%

Analysis and Key Findings
Very few facilities are using this feature.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the care settings.

Relied Upon Software
None

RWT Measure #12. Application access – all data request

Associated Criteria: 315(g)(9)
Application Programming Interfaces: 170.315(g)(9) Application access— all data request 

Testing Methodology: Reporting/Logging

Measurement Description
Demonstration of a patient’s ability to make a data request through API for one or more data elements from the Common Clinical Data Set.

Care Settings:
Family Medicine

Testing Results:
Number of Clinics With Records: 1
Reporting Interval: 07/01/2024 – 07/14/2024
Number Of Queries : 511
Denominator = (Total no. of requests for All Patient Data – Total no. of Invalid API requests)
Denominator = 511
Numerator = Total no. of successful responses given to requests in Denominator
Numerator = 511
Total % of Successful Transmissions = (Numerator/Denominator) X 100
Total % of Successful Transmissions = 100%

Analysis and Key Findings
Very few facilities are using this feature. Only two were found. We are hoping for better utilization with g (10) roll out.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the care settings.

Relied Upon Software
None

RWT Measure #13. Direct Project

Associated Criteria: 315(h)(1)
Electronic Exchange: 170.315(h)(1) Direct Project

Testing Methodology: Reporting/Logging

Measurement Description
This use case is tracking various measurements associated with Direct messaging successfully
sent and received with the EHR Module and a 3rd party over the course of a given interval. We
are tracking and measuring both the number of Direct messages sent, the unique destinations
Direct messages were sent to, and how many Direct messages were received.

Care Settings:
Family Medicine, Pediatrics, Surgery,  OBGYN, Rheumatology, Geriatric, Infectious Disease, Pain Medicine, Nephrology, Behavioral Health

Testing Results:
Number of Clinics With Records: 25
Reporting Interval: 01/01/2024 – 01/14/2024
Number of Messages: 42
Denominator: Total no. of Outgoing and incoming Transmissions initiated from Veracity
Denominator: 42
Numerator: Total no. of Outgoing and incoming wrapped Transmissions was successful
Numerator: 42
Total % of Successful Outgoing and incoming Transmissions = [Numerator/Denominator] * 100
Total % of Successful Outgoing and incoming Transmissions = 100%

Analysis and Key Findings
Our clients do not regularly share data through C-CDA files so we have few records of Direct exchange C-CDAs. Some clinics use third-party stand-alone apps to send direct messages.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the care settings.

Relied Upon Software
Phimail

RWT Measure #14. FHIR

Associated Criteria: 315(g)(10)
Standardized API for patient and population services: 170.315(g)(10)
Testing Methodology: Manual using inferno

Measurement Description
The data logs of API requests and responses are collected during the data logging schedule of 60 days. This data will prove that EHR has the real time working modules to receive and respond to API requests in FHIR format.

Care Settings:
Test family practice with non-real patient data using inferno.

Testing Results:
Number of Clinics With Records: 1
Reporting Interval: 07/08/2024 – 07/08/2024
Denominator: 1
Numerator: 1
Tests passed with inferno testing
Analysis and Key Findings
We don’t have any client that implemented FHIR API. We had to resort to manual testing.

Reason for Manual Testing:
Given our small client base, as of the end of May, no clients had implemented the FHIR API. This necessitated a shift to manual testing, which requires significant preparation time. The adjustment was unavoidable due to the absence of 3rd party FHIR-compatible client systems capable of consuming FHIR messages from our client FHIR server. Our attempts to engage with clients and third-party vendors revealed a consistent lack of FHIR-capable applications. While some third-party providers expressed interest in integrating with clinics, most relied on REST APIs without FHIR support. Despite extensive outreach and discussions, the lack of FHIR capability persisted, preventing us from moving forward with production testing.
We wish that the ONC fund the development for a patient-friendly mobile application to promote the adoption of FHIR for accessing health data across multiple vendors. This application would empower patients to seamlessly manage their health information and could be recommended by our clients to their patients. Additionally, such an initiative would encourage third-party developers to create FHIR-compatible applications, fostering a more robust ecosystem of interoperable healthcare solutions.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the care settings and date range.

Relied Upon Software
none