REAL WORLD TESTING RESULTS REPORT 2023

GENERAL INFORMATION

Plan Report ID Number: 20221208all
Developer Name: AllegianceMD Software, Inc
Product Name(s): Veracity
Version Number(s): 9.1
Product List (CHPL) ID(s):  15.02.05.2672.ALLE.01.01.1.220117
Developer Real World Testing Plan Page URL:  https://allegiancemd.com/real-world-testing
Developer Real World Testing Results Report URL:  https://allegiancemd.com/real-world-testing

CHANGES TO ORIGINAL PLAN

STANDARDS UPDATES (INCLUDING STANDARDS VERSION ADVANCEMENT PROCESS (SVAP) AND UNITED STATES CORE DATA FOR INTEROPERABILITY (USCDI))

SUMMARY OF TESTING METHODS AND KEY FINDINGS

Test Plan:
The test cases that were included actions by various user types to capture the required data and workflows. In most scenarios, real world patient data was used to confirm compliance with the things such as successful transmission statuses for some certification criteria requirements. Some standards were tested via manual inspection and/or with ONC-recommended test tools.

Test Methods:
User actions are sent to an event  distributed event streaming platform. User events are recorded. This method is superior to picking certain clients for testing as it provides real time data across all clients and in case of error, it provides error logs to help us trouble shoot any issue with the system.

Key Milestones

Challenges faced during testing

Key challenges necessitating changes to the test plans included a small userbase exacerbated by significant percentage of customers using Veracity 9.1 either not participating in government payers or utilizing the low-volume exception to MIPS/QPP participation, or accepting Medicare payment penalties as opposed to the labor of fully utilizing the tested functions of the software.

Change from Original Plan:

We decided to implemet a universal event driven logging system using a distributed event streaming platform across all of our clinics. This is a better plan than the original plan where we would contact or survey individual clinics. The event driven logging system also allows us to regulary query the logs and identify any issues or errors universally across all our clinics. 

Change of Care settings

Due to the the unpredictability of usage by certain care settings and the small user base available. We had to change the care settings to reflect the actual usage at the time of testings. There should be no impact as all care settings get the same copy of the software.

Change of Date ranges

Due to the the unpredictability of usage during certain ranges and the small user base available. We changed the date range or extends it to reflect the actual usage.

RWT Measure #1. Number of Transition of Care

Associated Criteria: 315(b)(1)
Care Coordination: 170.315 (b)(1): Transitions of Care

Testing Methodology: Reporting/Logging

Measurement Description:
This measure is tracking and counting how many C-CDAs are created and successfully sent or received from the EHR Module to or from a 3rd party during a transition of care event over the course of a given interval.

Received:
Care Settings:
OBGYN, Internist, Rheumatology, Family medicine, Geriatric Medicine, Vascular, Pediatrics, Cardiology, Infectious Disease, Pain Medicine, Nephrology, Behavioral Health
Testing Results:
Clinics Queried: 18
Reporting Interval: 07/01/2023 to 07/14/2023
Reporting Interval: 12 months
Total C-CDA Received for all Clinics: 51
Denominator = Total no. of Incoming CCDA received with no error – Total no. of Invalid Incoming CCDA
Denominator = 51
Numerator = Total no. of Incoming CCDA from Denominator
Numerator = 51
Receive: The total percentage of successful incoming CCDA: 100%

Sent:
Care Settings:
Podiatry, Sleep Medicine, Family medicine, Internist, Medical Spa, Pain Medicine, Pediatrics, Behavioral Health, Cardiovascular
Testing Results:
Clinics Queried: 18
Reporting Interval: 01/01/2023 to 01/14/2023
Reporting Interval: 12 months
Total C-CDA Received for all Clinics: 18
Denominator = Total no. of Incoming CCDA sent with no error – Total no. of Invalid CCDA
Denominator = 32
Numerator = Total no. of Sent CCDA from Denominator
Numerator = 32
Receive: The total percentage of successful sent CCDA: 100% 

Analysis and Key Findings
Our clients do not regularly share data through C-CDA files so we have few records of exchanged C-CDAs. Clients usually use this service to receive from hospital health systems.
Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the time range and care settings.

Relied Upon Software
Phimail

RWT Measure #2. Clinical information reconciliation and incorporation

Associated Criteria: 315(b)(2)
Care Coordination: 170.315(b)(2) Clinical information reconciliation and incorporation

Testing Methodology: Reporting/Logging

Measurement Description:
Demonstration of incorporating problem list, medication list and allergy medication list reconciliation from discrete problems, medications and medication allergies parsed from a C-CDA in referral providers environment. Verfity the successful reconciliations of parsed discrete data in the referrall provider’s environment into the problem list, medication list and allergy list in the clinical summmary.

Care Settings:
Ambulatory Surgery

Testing Results
Clinics With Records: 1
Reporting Interval: 07/01/2023 – 07/14/2023
Number of C-CDAs: 148

Testing Metric/Measurement:
Number of C-CDAs: 148

Denominator = Total no. of Incorporated CCDA sent with no error – Total no. of Invalid CCDA
Denominator = 148
Numerator = Total no. of Incorporated CCDA from Denominator
Numerator = 148
Receive: The total percentage of successful imported CCDA: 100% 

Analysis and Key Findings
Our clients do not regularly share data through C-CDA files so we have few records of Direct exchange C-CDAs. One clinic in the clinics queried was heavily using the CCDA to import patients.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the date range and care settings.

Relied Upon Software
None

RWT Measure #3. Number of Rx Messages Successfully Sent

Associated Criteria: 315(b)(3)
Care Coordination: 170.315(b)(3) Electronic prescribing

Testing Methodology: Reporting/Logging

Measurement Description:
This measure is tracking and counting how many NewRx and CancelRx electronic prescriptions were created and successfully sent from the EHR Module to a pharmacy destination over the course of a given interval.

Care Settings:
Behavioral Health, Cardiology, Family, Fertitlity, Hormone Replacement, Infectious Disease, Internist, Neurologist, OB/GYN, Pain Doctor, Podiatry, Rheumatology, Sleep Doctor, Surgeon, Urgent Care, Palliative Medicine, Endocrinology, Nephrology, Urgent Care, Pediatrics, Dermatology, Plastic and Reconstructive, Psychiatry.
.

Testing Results:
Clinics Queried: 218
Reporting Interval: 07/01/2023 – 07/14/2023
Total Electronic Prescriptions Sent for all Clinics: 37,131

Denominator = Total no. of NEWRX sent  – Total no. of Invalid NEWRX
Denominator = 37,131- 2326= 34,805
Numerator = Total no. of NEWRX sent with no error
Numerator = 34,805
The total percentage of successful NEWRX sent: 100% 

Analysis and Key Findings:
Electronic prescribing is a very popular and widely used feature in our EHR. 

Non-Conformities or Errors Discovered:
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the date range and care settings.

Relied Upon Software
None

RWT Measure #4. Number of Patient Batch Exports

Associated Criteria: 315(b)(6)
Care Coordination: 170.315(b)(6) Data export

Testing Methodology: Reporting/Logging

Measurement Description:
This measure is tracking and counting how many batch exports of C-CDAs were successfully performed by the EHR Module over the course of a given interval.

Care Settings:
Behavioral Health, Family Medicine, Pain Medicine, Plastic Surgery, 

Testing Results:
Clinics Queried: 5
Reporting Interval: 07/01/2023-07/14-2023
Total C-CDA exported for all Clinics: 20

Denominator: (Total no. of Manual Data Exports + Total no. of Automated Data exports) – Total no. of
Data exports failed due to wrong data entry by users.
Denominator: 20
Numerator: Total no. of successful Manual Data exports + Total no. of Successful Automated Data
Numerator: 20
Exports percentage completed without any errors.: 100% 

Analysis and Key Findings:
This test reveals that most of the clinics surveyed don’t utilize the batch exporting feature. The export result was only for five clinics.

Non-Conformities or Errors Discovered:
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the care settings.

Relied Upon Software
None

RWT Measure #5. Clinical quality measures – record and export

Associated Criteria: 315(C)(1)
Clinical Quality Measures: 170.315(c)(1)—record and export

Testing Methodology: Reporting/Logging

Measurement Description
Demonstration of the ability to export patient data recorded in the EHR for a given patient population in a QRDAI format. 

Care Settings:
Geriatric Medicine, Family Medicine, Internist, Infectious Disease, Primary Care

Testing Results:
Number of Clinics With Records: 5
Reporting Interval: 07/01/2023 – 12/31/2023
Number of QRDA1 batches exported: 31
Denominator: Total no. of attempts to generate QRDA 1.
Denominator: 31
Numerator: Total no. of successful attempts to generate QRDA 1.
Numerator: 31
Total % of successful exports of Valid QRDA 1 = [Numerator/ Denominator]* 100
Total % of successful exports of Valid QRDA 1 = 100%

Analysis and Key Findings
Because we are certified in nearly all ONC eCQMs, our clients have a wide variety of options to choose from. While CMS will accept the top six scoring eCQMs for their MIPS Quality Reporting, users can submit more than 6, however because of the size of the clinics, many clinics choose claim based or optout all together.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the reporting interval and care settings. The date range was extended due to low number of clinics.

Relied Upon Software
None 

RWT Measure #6. Clinical quality measures – import and calculate

Associated Criteria: 315(c)(2)
Clinical Quality Measures: 170.315(c)(2)—import and calculate

Testing Methodology: Reporting/Logging

Measurement Description
Demonstration of the ability to calculate quality measures for the patient data and measures.

Care Settings:
Internist, Podiatry, Nephrology, Surgery, Primary Care, Endocrine, Pain Medicine, Fertility, Urgent Care, Family Medicine,  Endocrinology, Weigh Loss, Pediatrics, Dermatology, Cardiology, Infectious Disease, Psychiatry

Testing Results:
Number of Clinics: 39
Reporting Interval: 07/01/2023 – 07/14/2023
Denominator: Total no. of QRDA 1 patient files imported into EHR – .
Denominator: 535
Numerator: Total no. of unique patients from Denominator who are considered for calculating
Numerator: 535
Total % of successful patients QRDA 1 files imported and calculated = [Numerator/ Denominator]* 100
Total % of successful patients QRDA 1 files imported and calculated = 100%

Analysis and Key Findings
Because we are certified in nearly all ONC eCQMs, our clients have a wide variety of options to choose from. While CMS will accept the top six scoring eCQMs for their MIPS Quality Reporting, users can submit more than 6.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the date range and care settings.

Relied Upon Software
None

RWT Measure #7. Clinical quality measures – import and calculate

Associated Criteria: 315(c)(3)
Clinical Quality Measures:  170.315(c)(3)—report

Testing Methodology: Reporting/Logging

Measurement Description
Demonstration of the ability to generate QRDA 3 files.

Care Settings:
Endocrine, Rheumatology, Primary Care

Testing Results:
Number of Clinics: 3
Reporting Interval: 07/01/2023 – 09/30/2023
Number Of QRDA-III Exported : 5
Denominator: Total no. of QRDA-III patient files imported into EHR.
Denominator: 5
Numerator: Total no. of unique patients from Denominator who are considered for calculating
Numerator: 5
Total % of successful patients QRDA-III files imported and calculated = [Numerator/ Denominator]* 100
Total % of successful patients QRDA-III files imported and calculated = 100%

Analysis and Key Findings
Clinics record CQM results and submit QRDA-III files to CMS the following year. The number of clinics exported to QRDA-III was below average due to the reporting period wasn’t on the due dates of CMS submission.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the reporting interval and care settings. The date range was extended due to low number of clinics performing QRDA3 export.

Relied Upon Software
None

RWT Measure #8. View, download, and transmit to 3rd party

Associated Criteria: 315(e)(1)
Patient Engagement: 170.315(e)(1) View, download, and transmit to 3rd party

Testing Methodology: Reporting/Logging

Measurement Description
Login to patient portal to view clinical summary, create and make a valid C-CDA available to download or transmit to 3rd party.

Care Settings:
Podiatry, Family, OB/GYN, Surgery, Endocrine, Fertility, Sleep Medicine, Oncology, Rheumatology, Pediatrics, Behavioral Health, Cardiology, Weight Loss, Pain Medicine,  Medical Spa, Dermatology, Neurology.

Testing Results:
Number of Clinics: 56
Reporting Interval: 07/01/2023 – 07/14/2023
Number Of Download or View or Transmit CCDA : 1023
Reporting Interval: 07/01/2023 – 07/14/2023
Denominator = Total no. patients who attempted to download or view or transmit CCDA – Data Entry/Authentication Errors .
Denominator= 1023 – 7=1016
Numerator = 1016
view or download or transmit is performed successfully.
Total Percentage = [(Numerator)\(Denominator)]*100
Total Percentage = 100%

Analysis and Key Findings
Some patients login and click view CCDA and download it or transmit to a 3rd party. However this is rarely used by patients. Some clinics ask their patients to login to the portal and download the CCDA.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the care settings.

Relied Upon Software
MyPortal.MD

RWT Measure #9. Transmission to public health agencies-immunization registry

Associated Criteria: 315(f)(1)
Public Health: 170.315(f)(1) Transmission to immunization registries

Transmission to immunization registries Measure 1:
Testing Methodology: Reporting/Logging

Measurement Description
Demonstration of the ability to add an immunization to a patient and generate a VXU message for an administered immunization and transmit it via HL7 2.5.1 to a public health agency successfully. Verify the immunization log of successful VXU message transmissions.

Care Settings:
Pediatrics, Family Medicine.

Testing Results:
Number of Clinics With Records: 2
Reporting Interval: 07/01/2023 – 07/14/2023
Number Of transmissions : 92
Denominator: Total no. of Vaccination information transmitted – Errors due to invalid data entry
Denominator: 94-2=92
Numerator: Total no. of Vaccination information transmitted successfully
Numerator:92
Total %:  [Numerator/Denominator] * 100
Total%: 100%

Vaccine Hx and Vaccine Forecast Measure: 2: Total percentage of successful requests executed to get
Vaccine history and Vaccine forecast.

Transmission to immunization registries Measure 2:
Testing Methodology: Manual

Measurement Description:
Total percentage of successful requests executed to get Vaccine history and Vaccine forecast.

Care Settings:
Test family practice with non-real patient data.

Testing Results:
Number of Clinics With Records: 1
Reporting Interval: 12/01/2023 – 12/01/2023
Number Of Files : 1
Denominator: : Total percentage of successful requests executed to get Vaccine history and Vaccine forecast.
Numerator: Total no of valid messages manually tested
Denominator:1
Numerator:1
Total %:  [Numerator/Denominator] * 100
Total%: 100%

Analysis and Key Findings
Few clinics are using the feature. If the end user uses the state inventory, lot# has to be accurate. No clinics are using the forcast feature

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the  care settings . We had to manual test measure 2 due to no usage reported by any client.

Relied Upon Software
None

RWT Measure #10. Transmission to Public Health Agencies

Associated Criteria: 315(f)(5)
Public Health: 170.315(f)(5) Transmission to public health agencies — electronic case reporting

Testing Methodology: Manual

Measurement Description
Demonstration of the ability to generate a transmission to public health agencies depending on a table of trigger codes per encounter.

Care Settings:
Test family practice with non-real patient data.

Testing Results:
Number of Clinics With Records: 1
Reporting Interval: 12/01/2023 – 12/01/2023
Number Of Files : 2

Analysis and Key Findings
We don’t have any clinics using this feature. A manual test was performed. We have a system that can query encounters for trigger codes. Once the encounter was flagged, an eICR CCDA was generated. Since we implemented FHIR, We are utilizing eCRNow for 2023.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the care settings. 

Relied Upon Software
None

RWT Measure #11. Application access – patient selection

Associated Criteria: 315(g)(7)
Application Programming Interfaces: 170.315(g)(7) Application access— patient selection

Testing Methodology: Reporting/Logging

Measurement Description
Demonstration of a patient’s ability to make a data request through API for one or more data elements from the Common Clinical data Set.

Care Settings:
Family Medicine

Testing Results:
Number of Clinics With Records: 1
Reporting Interval: 07/01/2023 – 07/14/2023
Number Of Queries : 28
Total no. of API requests received for the patients: 28 
Total no. of invalid API requests for the patients: 28
Total no. of successful API responses from Veracity: 100%

Analysis and Key Findings
Very few facilities are using this feature.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the care settings.

Relied Upon Software
None

RWT Measure #12. Application access – all data request

Associated Criteria: 315(g)(9)
Application Programming Interfaces: 170.315(g)(9) Application access— all data request 

Testing Methodology: Reporting/Logging

Measurement Description
Demonstration of a patient’s ability to make a data request through API for one or more data elements from the Common Clinical Data Set.

Care Settings:
Family Medicine

Testing Results:
Number of Clinics With Records: 2
Reporting Interval: 07/01/2023 – 07/14/2023
Number Of Queries : 617
Denominator = (Total no. of requests for All Patient Data – Total no. of Invalid API requests)
Denominator = 617
Numerator = Total no. of successful responses given to requests in Denominator
Numerator = 617
Total % of Successful Transmissions = (Numerator/Denominator) X 100
Total % of Successful Transmissions = 100%

Analysis and Key Findings
Very few facilities are using this feature. Only two were found. We are hoping for better utilization with g (10) roll out.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the care settings.

Relied Upon Software
None

RWT Measure #13. Direct Project

Associated Criteria: 315(h)(1)
Electronic Exchange: 170.315(h)(1) Direct Project

Testing Methodology: Reporting/Logging

Measurement Description
This use case is tracking various measurements associated with Direct messaging successfully
sent and received with the EHR Module and a 3rd party over the course of a given interval. We
are tracking and measuring both the number of Direct messages sent, the unique destinations
Direct messages were sent to, and how many Direct messages were received.

Care Settings:
Family Medicine, Pediatrics, Surgery,  OBGYN, Rheumatology, Geriatric, Infectious Disease, Pain Medicine, Nephrology, Behavioral Health

Testing Results:
Number of Clinics With Records: 18
Reporting Interval: 07/01/2023 – 07/14/2023
Number of Messages: 50
Denominator: Total no. of Outgoing and incoming Transmissions initiated from Veracity
Denominator: 50
Numerator: Total no. of Outgoing and incoming wrapped Transmissions was successful
Numerator: 50
Total % of Successful Outgoing and incoming Transmissions = [Numerator/Denominator] * 100
Total % of Successful Outgoing and incoming Transmissions = 100%

Analysis and Key Findings
Our clients do not regularly share data through C-CDA files so we have few records of Direct exchange C-CDAs. Some clinics use third-party stand-alone apps to send direct messages.

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the care settings.

Relied Upon Software
Phimail

RWT Measure #14. FHIR

Associated Criteria: 315(g)(10)
Standardized API for patient and population services: 170.315(g)(10)
Testing Methodology: Manual using inferno

Measurement Description
The data logs of API requests and responses are collected during the data logging schedule of 60 days. This data will prove that EHR has the real time working modules to receive and respond to API requests in FHIR format.

Care Settings:
Test family practice with non-real patient data.

Testing Results:
Number of Clinics With Records: 1
Reporting Interval: 12/01/2023 – 12/01/2023
Denominator: 1
Numerator: 1
Tests passed with inferno testing
Analysis and Key Findings
We don’t have any client that implemented FHIR API. 

Non-Conformities or Errors Discovered
During our testing, we did not discover any errors or criteria non-conformities. We did not make any changes to this measure from our original test plan except for the care settings and date range.

Relied Upon Software
none