In this article we will address the evidence gathering techniques for auditing under the CryptoCurrency Security Standard (CCSS).

Why is Collecting Evidence Important for the Audit?

When conducting an audit, the auditor must collect evidence as part of the auditing process. This enables the auditor to:

  1. Refer to the collected evidence when writing the audit report instead of just relying on memory which degrades with time.
  2. Provide proof that the conclusions reached by the auditor are based on fact presented at the time of audit.
  3. Reduce the amount of time and effort the auditor needs to reacquaint with the in-scope environment for the next audit.
  4. Allow for a more in-depth review of the in-scope environment. Reviewing in-scope systems, devices as well as interviewing persons of interest for the audit can be stressful for both the auditor and the assessed entity. Often in-scope reviews and interviews can be rushed or not completed sufficiently. Reviewing the collected evidence in the auditor’s own time can help identify issues that were not identified while conducting the interviews and reviews.

CCSS Auditor Guide

In regard to auditing an entity for CCSS compliance the CCSS committee has provided an “Auditors Guide” which provides some basic guidance for the auditing process.

In regard to what categories of evidence to collect the Auditor Guide provides the following guidance.

“1.2.2. Completeness and accuracy of Information Provided by the Entity (IPE)

Engagements are designed to be performed at least annually and cover the preceding 12 month period. Where a first time audit is performed the period covered may be 6 to 18 months.

The CCSSA is responsible for obtaining sufficient evidence over the completeness and accuracy of all information obtained in the performance of the CCSS audit.

This evidence and procedures performed should also be documented for a peer reviewer to be able to inspect and verify the accuracy and completeness of information.

The following are examples of information/evidence that should be retained for IPE purposes

  • Date report was generated
  • Data source (a specific system, application, or relevant database table)
  • System/tool generating IPE
  • Report owner
  • Screenshots of parameters, scripts, and/or queries used to run the report
  • Report validation supporting documents

CCSSA’s should also consider the nature of the report generated. For standard and canned system reports, little additional testing would be required. For custom reports of ad-hoc queries, the CCSSA should consider additional procedures such as inspecting the query parameters or database query to ensure data produced is accurate and complete.

These considerations should take into account input, processing, and output risks around the report.”

Also, the section below from the Auditor Guide provides further evidence requirements which we will discuss in this article.

“2.1 Minimum audit documentation required
The audit documentation should at a minimum detail the following.

  • Procedures performed to reach conclusion (ie. inspection, inquiry, reperformance, observation etc.);
  • Evidence of procedures performed over Information Produced by the entity (IPE) to demonstrate completeness and accuracy;
  • Rational and methodology used when applying sampling over a population of items;
  • Rational for conclusion reached on the CCSS level of compliance;”

 

Evidence Gathering Techniques

Techniques for evidence gathering for audit purposes are:

  1. Observation – the auditor observes a process being undertaken. For example, observing a key ceremony or a customer service representative attending to a customer. While observing, the auditor must take notes on what is observed in order to provide an accurate account in future and to provide proof that the observation did occur.
  2. Inspection – documentation, records, and data. Documentation can be in the form of written policies, standards, and procedures. Records are generally the outputs of activities such as a vulnerability scan or a query of user accounts that have been inactive for 90 days. Data can be device or system configurations such as firewall rulesets or a switch’s running configuration. Data can also include data residing in databases, files etc… and data in transit for example network data packets.
  3. Reperformance – an auditor performs a task that the assessed entity would perform. For example, the auditor configures and runs a report that the assessed entities finance team executes every month to ensure the written process actually matches what was physically undertaken. Another example is the auditor attempting to log into a system using default credentials provided by the device or systems vendor. NOTE: Reperformance as a technique for evidence collection for an information security management systems audit is seldom undertaken by an external auditor due to the nature of the systems involved. Information security controls have highly restrictive access, require advance training to manage and any actions undertaken on the control could impact any number of compliance or regulatory requirements the entity must adhere to. In our experience observing an authorised user of a control or in-scope system conducting a process under audit is sufficient and risk-free unlike reperformance.
  4. Interview – Interviewing is a technique that requires “soft skills”. A skilled auditor will be able to read several indicators presented by the interviewee including, if there is a desire to answer questions truthfully, the mood of the interviewee and how the interviewee presents information which could be influenced by personal experiences, culture, and beliefs. An auditor who does not consider the interviewees indicators can often result in poor evidence collection from the interview. The purpose of the interview is to gather information about applicable processes and controls then confirm that the information provided by the interviewee is correct based on the written policies, standards and procedures used by the assessed entity. Basically, what is often written is ignored and what is done is often by the easiest of approaches.

NOTE: If any screen shots or copies of evidence is captured during an observation, inspection or interview ensure that the filename provides key information as to the source of the evidence. For example, if reviewing a HSM password policy configuration attempt to name the file as HSM_1_Password_Policy.jpeg. You could even add the date and time of when the screen shot was taken.

Evidence Requirements Breakdown

The CCSS Auditor Guide defines the following requirements for evidence gathering which we will comment on inline.

“The CCSSA is responsible for obtaining sufficient evidence over the completeness and accuracy of all information obtained in the performance of the CCSS audit.”

Our interpretation of the above statement is that it is up to the auditor to decide the type of evidence gathering techniques to be utilized and the amount of evidence to gather to confirm that the assessed entity is compliant with a requirement (and to what CCSS Level). For example, will an interview be sufficient or is an inspection of the in-scope systems required as well as an interview?

In our experience auditing under PCI DSS and ISO27001, evidence gathering requires more than one technique to ensure that an assessed entity is compliant with a requirement. For example:

  1. The auditor inspects the assessed entities policies, standards and procedures to gain an understanding on what officially is required,
  2. The auditor then conducts interviews with the assessed entities personnel where their role is impacted by the documentation to ensure the interviewee is aware of the documentation and understands the documentations purpose,
  3. The auditor inspects all or a sample of the devices and systems in-scope to ensure that the documentation’s requirements has been implemented or, observe a procedure being undertaken.

You can see that this approach requires three evidence gathering techniques (inspection, observation, and interviews).

“This evidence and procedures performed should also be documented for a peer reviewer to be able to inspect and verify the accuracy and completeness of information.”

To ensure that the peer review process is undertaken with a consistent approach regardless of the reviewer, detailed guidance on how to conduct the peer review process should be provided. A peer review report template should also be provided to enforce standardization of reporting. At the time of publishing this article the CCSS committee has not yet provided any public information pertaining to how the peer review process will be managed to ensure consistency, and what tools and guidance will be provided to CCSSAs. However, the CCSS committee has provided the following guidance on examples of the types of evidence that need to be retained:

“The following are examples of information/evidence that should be retained for IPE purposes

  • Date report was generated
  • Data source (a specific system, application, or relevant database table)
  • System/tool generating IPE
  • Report owner
  • Screenshots of parameters, scripts, and/or queries used to run the report
  • Report validation supporting documents”

 

What should a CCSSA Record?

The above guidance defines some examples of what data should be captured during evidence gathering. To elaborate further we have provided our own thoughts on what should be captured for each evidence gathering technique below.

Observation

If the CCSSA is observing a process, then record the following:

  1. What process is being observed – for example “Annual Key Rotation Process”.
  2. Reference the policy, standards, and procedure documentation that the process is defined upon – for example “Key Management” policy and “Key Custodian Procedures”.
  3. Date and time and location of the observation.
  4. Who was undertaking the process – include full name and role of the person?
  5. The outcome of the process and any other pertinent events that are used as considerations by the CCSSA as to the compliance status.
  6. Any outstanding evidence that needs to be supplied and new actions that were identified out of the observation.

Inspection

Documentation Inspection

If the CCSSA is reviewing documentation such as policy, standards, procedures then record the following:

  1. File name of the document.
  2. Title of the document.
  3. Purpose of the document.
  4. Version of the document.
  5. Owner of the document.
  6. When the document was last reviewed and updated.
  7. Location of the document – if viewing online documentation.
  8. CCSSA findings from the review of the document. This might be used for reporting on remediation such as the report is missing “x” which is required for CCSS compliance.
  9. Any outstanding evidence that needs to be supplied and new actions that were identified out of the inspection.

Record Inspection

If the CCSSA is reviewing records such as reports then record the following:

  1. Name of record.
  2. Date and time of the records generation.
  3. Brief description of how the record is generated including what roles can generate the record.
  4. Purpose of the record.
  5. CCSSA findings from the review of the record.
  6. Any outstanding evidence that needs to be supplied and new actions that were identified out of the review.

Configuration & Data Inspection

If the CCSSA is inspecting configuration data and data at rest, then record the following:

  1. What is the device or system that the configuration data pertains to – for example “Firewall 1 NTP settings” or where is the data located – e.g. in what database server, in what database, in what table?
  2. Date and time of the configuration review and the person who provided access to the data – for example “Bob Smith – database administrator”.
  3. CCSSA findings from the review of the data.
  4. Any outstanding evidence that needs to be supplied and new actions that were identified out of the inspection.

Reperformance

As noted above, reperformance as an evidence gathering technique is not normally utilized in auditing information security management systems due to the inherent risks of allowing an external auditor to undertake the process. Reperformance can be substituted by observation, inspection, and interviews.

There may be an action that can be performed by the CCSSA where the risk of impacting the availability, confidentially or integrity of the system and/or information is negligible. In that instance record the following:

  1. What process is being undertaken by the CCSSA.
  2. Who authorized (in writing) that the CCSSA could undertake the process?
  3. Brief description of the process.
  4. The outputs of the process.
  5. CCSSA findings from the undertaking of the process.
  6. Any outstanding evidence that needs to be supplied and new actions that were identified out of the reperformance.

Interviews

For each interview conducted record the following:

  1. Name of interviewee – full name and role.
  2. Date and time and location of the interview.
  3. Duration of the interview.
  4. Topics covered within the interview.
  5. Any outstanding evidence that needs to be supplied and new actions that were identified out of the interview.

Ensure that notes are taken during the interview or if possible and with prior authorisation voice or video record the interview.

Applying all we Know About Evidence Gathering

To provide an example of how we would approach a CCSS audit, a requirement from the CCSS has been selected and the evidence gathering techniques that we believe apply to the requirement have been added to an example CCSSA audit worksheet.

Note: this example is based on the current guidance provided by the CCSS committee at the time of publishing this article.

Summary

In this article we covered the evidence gathering techniques we believe are best suited for CCSS, which is an information security management standard. We also provided an example audit worksheet defining the evidence gathering techniques we believe are applicable to the selected CCSS requirement.