compliance package

Subpackages

• compliance.scripts package
  • Submodules
  • compliance.scripts.compliance_cli module
    • ComplianceCLI
      • ComplianceCLI.name
    • run()
  • Module contents
• compliance.utils package
  • Subpackages
    • compliance.utils.services package
      • Submodules
      • compliance.utils.services.github module
      • compliance.utils.services.pagerduty module
      • Module contents
  • Submodules
  • compliance.utils.credentials module
    • Config
      • Config.get()
  • compliance.utils.data_parse module
    • deep_merge()
    • format_json()
    • get_sha256_hash()
    • parse_dot_key()
  • compliance.utils.exceptions module
    • DependencyFetcherNotFoundError
    • DependencyUnavailableError
    • EvidenceNotFoundError
    • HistoricalEvidenceNotFoundError
    • LockerPushError
    • StaleEvidenceError
    • UnverifiedEvidenceError
  • compliance.utils.http module
    • BaseSession
      • BaseSession.prepare_request()
  • compliance.utils.path module
    • get_module_path()
    • get_toplevel_dirpath()
    • load_evidences_modules()
    • substitute_config()
  • compliance.utils.test module
    • parse_test_id()
  • Module contents

Submodules

compliance.agent module

Compliance check automation module.

class compliance.agent.ComplianceAgent(name=None, use_agent_dir=True)

Bases: object

Compliance agent class.

AGENTS_DIR = 'agents'

PUBLIC_KEYS_EVIDENCE_PATH = 'raw/auditree/agent_public_keys.json'

classmethod from_config()

Load agent from configuration.

get_path(path)

Get the full evidence path.

Parameters:

path – The relative evidence path as a string.

Returns:

The full evidence path as a string.

hash_and_sign(data_bytes)

Hash and sign evidence using the agent private key.

Parameters:

data_bytes – The data to sign as bytes.

Returns:

A tuple containing the hexadecimal digest string and the base64 encoded signature string. Returns tuple (None, None) if the agent is not configured to sign evidence.

load_public_key_from_locker(locker)

Load agent public key from locker.

Parameters:

locker – A locker of type compliance.locker.Locker.

property name

Get agent name.

property private_key

Get agent private key.

property public_key

Get agent public key.

signable()

Determine if the agent can sign evidence.

verifiable()

Determine if the agent can verify evidence.

verify(data_bytes, signature_b64)

Verify evidence using the agent public key.

Parameters:

• data_bytes – The data to verify as bytes.

• signature_b64 – The base64 encoded signature string.

Returns:

True if data can be verified, else False.

compliance.check module

Compliance check automation module.

class compliance.check.ComplianceCheck(*args, **kwargs)

Bases: TestCase

Compliance check automation TestCase class.

add_evidence_metadata(evidence_path, evidence_dt=None, evidence_locker=None)

Add evidence metadata to the evidence_metadata property of each check.

Parameters:

• evidence_path – the evidence path.

• evidence_dt – the evidence date.

• evidence_locker – the locker the evidence was retrieved from. Use when historical evidence is found in a secondary locker.

add_failures(section_key, section_value)

Add failures for each check.

Param:

section_key: key for the failures.

Param:

section_value: all the failures for that key.

add_issue_if_diff(actual, expected, msg, as_warning=False)

Add failures/warnings if differences exist between actual and expected.

Parameters:

• actual – the “actual” set items.

• expected – the “expected” set items.

• msg – the section key used if differences are found.

• as_warning – warn if True, fail if False.

add_successes(section_key, section_value)

Add successes for each check.

Param:

section_key: key for the successes.

Param:

section_value: all the successes for that key.

add_warnings(section_key, section_value)

Add warnings for each check.

Param:

section_key: key for the warnings.

Param:

section_value: all the warnings for that key.

assertEquals(first, second, msg=None)

Fail if the two objects are unequal as determined by the ‘==’ operator.

property evidence_metadata

Metadata of all evidence used by each check as a property.

evidences_for_check(results)

Provide evidences used by all tests in the check.

Parameters:

results – contains all the test results.

property failures

Failures as a property for each check.

failures_count()

Count failures for a specific test.

failures_for_check(results)

Provide failures associated with all tests in the check.

Parameters:

results – contains all the test results.

failures_for_check_count(results)

Count failures for all tests in check.

Parameters:

results – contains all the test results.

property fixed_failure_count

Count of failures fixed by the fix routine.

get_historical_evidence(evidence_path, evidence_dt)

Retrieve historical evidence from the locker and track as metadata.

Parameters:

• evidence_path – the evidence path.

• evidence_dt – the evidence date.

id()

Reset the test id if it has been transplanted and return it.

property reports

List of the check reports.

run(result=None)

Override the unittest.TestCase.run method.

Decorates test_ methods by adding common functionality (e.g. failures count check).

property runbook_url

Runbook URL associated with the check.

If runbook links are enabled in the configuration, the value returned is based on the runbook base URL and the check’s class name. It is assumed that the runbook exists and is in HTML format.

property successes

Successes as a property for each check.

successes_count()

Count successes for a specific test.

successes_for_check(results)

Provide successes associated with all tests in the check.

Parameters:

results – contains all the test results.

successes_for_check_count(results)

Count successes for all tests in check.

Parameters:

results – contains all the test results.

property title

Check title, normally used in check reports.

total_issues_count(results)

Total number of issues as property for the check.

Parameters:

results – contains all the test results.

property warnings

Warnings as a property for each check.

warnings_count()

Count warnings for a specific test.

warnings_for_check(results)

Provide warnings associated with all tests in the check.

Parameters:

results – contains all the test results.

warnings_for_check_count(results)

Count failures for all tests in the check.

Parameters:

results – contains all the test results.

compliance.config module

Compliance configuration automation module.

class compliance.config.ComplianceConfig

Bases: object

The configuration for a compliance run.

Credentials and a cache of known evidences are included.

DEFAULTS = {'locker': {'dirname': 'compliance', 'repo_url': 'https://github.com/YOUR_ORG/YOUR_PROJECT'}, 'notify': {'gh_issues': {}, 'ghe_issues': {}, 'pagerduty': {}, 'slack': {}}, 'org': {'name': 'YOUR_ORG', 'settings': {}}, 'runbooks': {'base_url': 'https://example.com/runbooks', 'enabled': False}}

add_evidences(evidence_list)

Add a list of evidence objects to the evidence cache.

Parameters:

evidence_list – a list of evidence objects.

property creds

Credentials used for locker management and running fetchers.

property evidences

All evidence objects currently in the evidence cache.

get(config_path, default=None)

Provide the configuration value for the supplied config_path.

Returns the default if config_path cannot be retrieved.

Parameters:

config_path – dot notation path with the following format 'key[.subkey]. For instance, locker.dirname.

get_evidence(evidence_path)

Provide an evidence object from the evidence cache.

Parameters:

path – the path to the evidence within the Locker. For example, raw/source1/evidence.json

get_template_dir(test_obj=None)

Provide absolute path to the template directory for the test object.

The associated path will be the first directory found named templates in the test object absolute path traversed in reverse. If test_obj is None, then current directory '.' is assumed as initial path.

Parameters:

test_obj – a compliance.ComplianceTest object from where the template directory search will start from.

load(config_file=None)

Load configuration from a JSON file.

Parameters:

config_file – the path to the JSON config file. If None, the DEFAULT configuration is used.

property raw_config

Raw configuration settings as a dictionary.

compliance.config.get_config()

Provide the global configuration object.

compliance.controls module

Compliance check controls automation module.

class compliance.controls.ControlDescriptor(dirs=None)

Bases: object

Class abstraction for controls.json content.

Used when processing controls.json content.

property accred_checks

Provide all checks by accreditation (key) as a dictionary.

property as_dict

Provide control descriptor content as a modifiable dictionary.

get_accreditations(test_path)

Provide the accreditation list for a given test_path.

This works for original and simplified controls formats.

Original: {‘test_path’: {‘key’: {‘sub-key: [‘accred1’, ‘accred2’]}}} Simplified: {‘test_path’: [‘accred1’, ‘accred2’]}

Parameters:

test_path – the Python path to the test. For example: package.category.checks.module.TestClass

is_test_included(test_path, accreditations)

Provide boolean result of whether a check is part of accreditations.

Parameters:

• test_path – the Python path to the test. For instance: package.accr1.TestClass or package.accr2.test_function

• accreditations – list of accreditations names where test_path may be included.

property paths

All absolute paths to controls.json file(s).

compliance.evidence module

Compliance evidence management automation module.

class compliance.evidence.DerivedEvidence(name, category, ttl=86400, description='', **kwargs)

Bases: _BaseEvidence

The derived evidence class.

EVIDENCE_TYPE = 'derived'

class compliance.evidence.ExternalEvidence(name, category, ttl=31536000, description='', **kwargs)

Bases: _BaseEvidence

The external evidence class.

EVIDENCE_TYPE = 'external'

class compliance.evidence.LazyLoader(path, ev_class)

Bases: tuple

ev_class

Alias for field number 1

path

Alias for field number 0

class compliance.evidence.RawEvidence(*args, **kwargs)

Bases: _BaseEvidence

The raw evidence class.

Raw evidence is partition-able if the content is JSON and by providing a list of partition fields along with an optional partition root. The evidence partitioning can be defined in configuration JSON or directly upon construction of a RawEvidence object. The constructor route overrides the configuration settings, if both are provided.

EVIDENCE_TYPE = 'raw'

get_partition(key)

Provide a slice of content based on the supplied partition key.

Return a JSON document that is a slice of the original evidence content based on the list of key values provided. Key values are expected to match the key fields provided during evidence instantiation and are relative to the root partition provided during evidence instantiation.

Parameters:

key – A list of key values to partition by

Returns:

A JSON document filtered by the key values provided

property is_partitioned

Raw evidence partitioned status.

property partition_keys

All partition keys for this evidence object.

Return all key values based on evidence content and the key fields provided during evidence instantiation. Key values are relative to the root partition provided during evidence instantiation.

Returns:

A list of key values where the key values are lists as well.

class compliance.evidence.ReportEvidence(name, category, ttl=86400, description='', **kwargs)

Bases: _BaseEvidence

The report evidence class.

EVIDENCE_TYPE = 'reports'

class compliance.evidence.TmpEvidence(name, category, ttl=86400, description='', **kwargs)

Bases: _BaseEvidence

The temporary evidence class.

EVIDENCE_TYPE = 'tmp'

class compliance.evidence.derived_evidence(locker, sources, target)

Bases: object

Helper context manager for a typical fetch_ method implementation.

Use when retrieving derived evidence by a fetcher and the name of the evidence is dynamic. When the evidence name is static/known use the @store_derived_evidence decorator instead.

If TTL has expired on the derived evidence, the context manager yields a dictionary containing all of the source evidences as well as the derived evidence objects specified in the sources and target arguments.

If TTL has not expired on the derived evidence, the context manager yields None and expects the calling with block to exit gracefully immediately.

Inside the with block:

If sources are provided as a list then the source evidence is accessible through the source path as the key for the yielded dictionary of evidence.

If sources are provided as a dictionary where evidence short names are keys and evidence source paths are the values then the evidence is accessible through the evidence short name as the key for the yielded dictionary of evidence.

If a single evidence source path is provided as a string then the source evidence is accessible by using ‘source’ as the key for the yielded dictionary of evidence.

The derived evidence object is always accessible by using ‘derived’ as the key for the yielded dictionary of evidence.

The context manager expects the derived evidence content to be generated and set via set_content by the calling with block.

class compliance.evidence.evidences(obj, from_evidences)

Bases: object

Helper context manager for a typical test_ method implementation.

Use when processing evidence by a check and the name(s) of the evidence is/are dynamic. When the evidence names are static/known use the @with_*_evidences decorators instead.

Inside the with block:

If evidence_paths are provided as a list then the evidence is accessible through the evidence path as the key for the yielded dictionary of evidence.

If evidence_paths are provided as a dictionary where evidence short names are keys and evidence paths are the values then the evidence is accessible through the evidence short name as the key for the yielded dictionary of evidence.

If a single evidence path is provided as a string then the evidence object is yielded by this context manager.

compliance.evidence.get_evidence_base_classes()

Provide a list of all valid base evidence class objects.

compliance.evidence.get_evidence_by_path(path, locker=None)

Provide an evidence object specified by the given path.

The following strategy is used:

• Return evidence if it is present in the evidence cache populating content if necessary and possible.

• If evidence is not in the evidence cache but a locker is provided, then the evidence will be retrieved from the locker.

• Otherwise, an evidence object is built with the default parameters.

Parameters:

path – relative path to the evidence within the Locker. For example, raw/source1/evidence.json.

Returns:

the evidence object.

compliance.evidence.get_evidence_class(evidence_type)

Provide the appropriate evidence class based on the supplied type.

Returns the corresponding evidence class object based on the category provided. If no match None is returned.

Parameters:

evidence_type – the type of evidence class desired as a string. Valid values are tmp, reports, derived, raw, external.

Returns:

the appropriate evidence class.

compliance.evidence.get_evidence_dependency(path, locker, fetcher=None)

Provide evidence to fetchers that depend on that other evidence to fetch.

Use when a fetcher needs evidence fetched by another fetcher in order to complete its fetch process. The premise is that if the evidence is in the evidence cache then return that because it was placed there by another fetcher. If not then get the evidence directly from the locker without putting that evidence into the evidence cache. When an evidence dependency is not found the fetcher is queued up as a dependency rerun for subsequent processing.

Parameters:

• path – relative path to the evidence within the Locker. For example, raw/source1/evidence.json.

• locker – evidence Locker object.

• fetcher – optional Python notation path to fetcher method. If provided, this defines the fetcher that is added to the re-run list. Otherwise the execution stack is traversed to find the fetcher caller.

Returns:

the evidence object.

compliance.evidence.get_evidence_types()

Provide a list of all valid evidence types.

class compliance.evidence.raw_evidence(locker, evidence_path)

Bases: _EvidenceContextManager

Helper context manager for a typical fetch_ method implementation.

Use when retrieving raw evidence by a fetcher and the name of the evidence is dynamic. When the evidence name is static/known use the @store_raw_evidence decorator instead.

If TTL has expired, the context manager yields the evidence object specified by the evidence_path. The context manager expects the evidence content to be retrieved and set via set_content by the calling with block.

If TTL has not expired, the context manager yields None and expects the calling with block to exit gracefully immediately.

compliance.evidence.store_derived_evidence(evidences, target)

Decorate a typical fetcher_ method fetching derived evidence.

Use when retrieving derived evidence by a fetcher and the name of the evidence is static/known. When the evidence name is dynamic use the derived_evidence context manager instead.

The decorator expects that the decorated method returns the content to be stored in the locker. The storing of the evidence is also handled by this decorator.

Parameters:

• evidences – a list of relative paths to the evidences needed by the fetcher. They will be passed to the method call. For example, [raw/src/foo.json, derived/src/bar.json].

• target – relative path to the evidence to be stored in the Locker. For example, src/evidence.json.

compliance.evidence.store_raw_evidence(evidence_path)

Decorate a typical fetcher_ method fetching raw evidence.

Use when retrieving raw evidence by a fetcher and the name of the evidence is static/known. When the evidence name is dynamic use the raw_evidence context manager instead.

The decorator expects that the decorated method returns the content to be stored in the locker. The storing of the evidence is also handled by this decorator.

Parameters:

path – relative path to the evidence within the Locker. For example, source1/evidence.json.

compliance.evidence.store_tmp_evidence(evidence_path)

Decorate a typical fetcher_ method fetching temporary evidence.

Use when retrieving temporary evidence by a fetcher and the name of the evidence is static/known. When the evidence name is dynamic use the tmp_evidence context manager instead.

Parameters:

path – relative path to the evidence within the Locker. For example, source1/evidence.json.

class compliance.evidence.tmp_evidence(locker, evidence_path)

Bases: _EvidenceContextManager

Helper context manager for a typical fetch_ method implementation.

Use when retrieving temporary evidence by a fetcher and the name of the evidence is dynamic. When the evidence name is static/known use the @store_tmp_evidence decorator instead.

If the evidence is not present, the context manager yields the evidence object specified by the evidence_path. The context manager expects the evidence content to be retrieved and set via set_content by the calling with block.

If the evidence is present, the context manager yields None and expects the calling with block to exit gracefully immediately.

compliance.evidence.with_derived_evidences(*from_evidences)

Decorate a typical test_ check method processing derived evidences.

Use when processing derived evidence by a check and the name(s) of the evidence is/are static/known. When the evidence names are dynamic use the evidences context manager instead.

Parameters:

from_evidences – relative paths to evidences as strings or LazyLoader namedtuples that contain relative paths to evidences and evidence classes required by the check.

compliance.evidence.with_external_evidences(*from_evidences)

Decorate a typical test_ check method processing external evidences.

Use when processing external evidence by a check and the name(s) of the evidence is/are static/known. When the evidence names are dynamic use the evidences context manager instead.

Parameters:

from_evidences – relative paths to evidences as strings or LazyLoader namedtuples that contain relative paths to evidences and evidence classes required by the check.

compliance.evidence.with_raw_evidences(*from_evidences)

Decorate a typical test_ check method processing raw evidences.

Use when processing raw evidence by a check and the name(s) of the evidence is/are static/known. When the evidence names are dynamic use the evidences context manager instead.

Parameters:

from_evidences – relative paths to evidences as strings or LazyLoader namedtuples that contain relative paths to evidences and evidence classes required by the check.

compliance.evidence.with_tmp_evidences(*from_evidences)

Decorate a typical test_ check method processing temporary evidences.

Use when processing temporary evidence by a check and the name(s) of the evidence is/are static/known. When the evidence names are dynamic use the evidences context manager instead.

Parameters:

from_evidences – relative paths to evidences as strings or LazyLoader namedtuples that contain relative paths to evidences and evidence classes required by the check.

compliance.fetch module

Compliance fetcher automation module.

class compliance.fetch.ComplianceFetcher(*args, **kwargs)

Bases: TestCase

Compliance fetcher automation TestCase class.

fetchCloudantDoc(db_url, params=None)

Retrieve a Cloudant document.

Helper/Convenience method.

Parameters:

• db_url – the URL to the Cloudant doc to retrieve.

• params – optional parameters to include in the GET request.

Returns:

the Cloudant document content.

fetchLocalCommands(commands, cwd=None, env=None, show_exit_status=False, show_timestamp=False, timeout=20, user=None)

Retrieve the output from locally executed commands.

Helper/Convenience method.

Parameters:

• commands – A list of string commands to be executed.

• cwd – Override the current working directory. Defaults to None.

• env – Override environment variables. Defaults to None.

• show_exit_status – Show the exit status for each command. Defaults to False.

• show_timestamp – Show a timestamp denoting when each command was executed. Defaults to False.

• timeout – Terminate after timeout seconds. Defaults to 20.

• user – Execute commands as specified user. Defaults to current user.

Returns:

Command output as a string.

fetchURL(url, params=None, creds=None)

Retrieve remote content through an HTTP GET request.

Helper/Convenience method.

Parameters:

• url – the URL of the file.

• params – optional parameters to include in the GET request.

• creds – optional tuple with (user, password) to be used.

Returns:

the remote content.

classmethod session(url=None, creds=None, **headers)

Provide a requests session object with User-Agent header.

Parameters:

• url – optional base URL for the session requests to use. A url argument triggers a new session object to be created whereas no url argument will return the current session object if one exists.

• creds – optional authentication credentials.

• headers – optional kwargs to add to session headers.

Returns:

a requests Session object.

classmethod tearDownClass()

Perform clean up.

compliance.fetch.fetch(url, name)

Write content retrieved from provided url to a file.

Parameters:

• url – the URL to GET content from.

• name – the name of the file to write to in the TMPDIR.

Returns:

the path to the file.

compliance.fix module

Compliance fixer automation module.

class compliance.fix.Fixer(results, dry_run=True, out=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Bases: object

This class attempts to resolve check failures.

execute_fix(test_obj, fct, args=None)

Execute the fix routine for a given check test object.

This method gets called by fix_* methods in checks. It gets passed a function (method) that must be called in order to fix one specific issue. The method’s fix will either get executed, or if in dry-run mode, will print out the method’s docstring, injecting any arguments if needed. The check’s fixed_failure_count is incremented if the fix was successful.

Parameters:

• test_obj – instance of subclass of compliance.utils.check.ComplianceCheck on which to execute the fix

• fct – a callback function that will actually perform the fix. this function will be passed a reference to this fixer, along with a reference to a compliance.utils.credentials.Config object. this callback will also need to have a docstring, which is what will be displayed in dry-run mode. the docstring will get formatted with the arguments passed to the fix function.

• args – dictionary of named arguments to pass to the fix function fct.

fix()

Perform all fix routines found in executed checks.

Iterate through all compliance checks and looks for fix_* methods with corresponding test_* methods. These are executed, and results are recorded if fixes are made.

Note: Instead of individual fix_* methods, you can instead define a single fix_failures method which handles all fixes for the class.

compliance.locker module

Compliance locker management automation module.

class compliance.locker.Locker(name=None, repo_url=None, branch=None, creds=None, do_push=False, ttl_tolerance=0, gitconfig=None, local_path=None, use_extra_lockers=True)

Bases: object

The locker (context manager) class.

The evidence locker maintains a git repository of evidence and places new evidence into it. It can validate a piece of evidence against a time to live. The Locker is a context manager. On instantiation it will retrieve the configured git repository.

add_content_to_locker(content, folder='', filename=None)

Add non-evidence text content to the local locker.

Parameters:

• content – the content to add to the local locker.

• folder – the folder in the local locker to add the content to.

• filename – the name of the file in the local locker.

add_evidence(evidence, checks=None, evidence_used=None)

Add the evidence to the locker.

The locker will generate the file internally and add it to the local git repository.

Parameters:

• evidence – the evidence to add. Note that evidence content attribute must be populated.

• checks – A list of checks used to generate report content. Only applicable for check generated ReportEvidence.

• evidence_used – metadata for evidence used by a check. Only applicable for check generated ReportEvidence.

checkin(message=None)

Commit changed files to the local git repository.

Parameters:

message – the message to apply on the commit.

checkout()

Pull (clone) the remote repository to the local git repository.

property clone_depth

Provide the commit depth used when cloning the repository.

Returns:

the clone depth.

property clone_shallow_since_days

Provide the maximum commit age to fetch when cloning the repository.

Returns:

the shallow since days.

create_tombstone_metadata(candidate, metadata, reason)

Create tombstones for evidence being removed from the locker.

Parameters:

• candidate – either the name of the evidence file or an iterable of partition hashes.

• metadata – the evidence metadata dictionary.

• reason – the reason evidence is being removed.

Returns:

a list of tombstones.

delete_repo_locally()

Remove the local git repository.

get_abandoned_evidences(threshold=None)

Provide a set of evidence where the update threshold has passed.

Parameters:

threshold (int) – the time in seconds after TTL expires that evidence can remain un-updated before it is considered abandoned. The abandoned evidence threshold defaults to 30 days if none is provided.

Returns:

a set of abandoned evidence file relative paths.

get_content_from_locker(folder='', filename=None)

Read non-evidence text content from the local locker.

Parameters:

• folder – the folder in the local locker to get the content from.

• filename – the name of the file in the local locker.

get_dependency_reruns()

Provide dot notation path to fetcher methods to be re-executed.

Fetchers will only be considered for re-execution if they initially failed to execute due to a dependency error.

Returns:

a list of fetcher methods to re-execute.

get_empty_evidences()

Provide a list of evidence paths to empty evidence files.

Evidence content is considered empty based on an evidence object’s is_empty property. This information is stored in evidence metadata.

Returns:

a list of empty evidence file relative paths.

get_evidence(evidence_path, ignore_ttl=False, evidence_dt=None)

Provide the evidence from the locker.

The evidence may or may not exist in the locker. If the evidence is historical but not found in the immediate locker and a previous locker URL is provided in the configuration then an attempt to get the historical version of the evidence will be made from the previous locker.

Parameters:

• evidence_path – string path of the evidence within the locker. For example, raw/service1/output.json.

• ignore_ttl – boolean value to ignore evidence time to live. Defaults to False.

• evidence_dt – datetime of the evidence file version to retrieve. Defaults to None which translates to “now”.

Returns:

the populated evidence object.

get_evidence_metadata(evidence_path, evidence_dt=None)

Provide evidence metadata from package index file.

Parameters:

• evidence_path – the evidence relative path.

• evidence_dt – the upper bound evidence commit date.

Returns:

the metadata for the evidence specified.

get_file(filename)

Provide the path for a file in the locker.

The file may or may not exist in the locker.

Parameters:

filename – the name of a file in the locker.

Returns:

the path to the filename provided.

get_hash()

Provide the current master revision.

Returns:

the commit SHA or None.

get_index_file(evidence)

Provide the full path to the index file of the given evidence.

Parameters:

evidence – the evidence object to be used.

get_index_file_by_path(evidence_path)

Provide the full path to the supplied evidence’s index file.

Parameters:

evidence_path – the path to the evidence within the evidence locker.

Returns:

the full path to the evidence’s index file.

get_large_files(size=50000000)

Provide a dictionary of “large” evidence locker files.

A “large” file is one whose size is > the size argument provided.

Parameters:

size (int) – file size threshold.

Returns:

a dictionary of file paths and sizes of “large” files.

get_latest_commit(path, dt=None)

Provide the most recent commit for the file and date specified.

Parameters:

• path – the relative path to the file.

• dt – the upper bound commit date.

Returns:

the latest commit containing the file specified.

get_locker_repo(locker='evidence locker')

Pull (clone) the remote repository to the local git repository.

Parameters:

locker – the locker “name” used in logging

get_remote_location(filename, include_commit=True, sha=None, locker_url=None)

Provide the path for a file/commit SHA in the locker.

The file may or may not exist in the locker.

Parameters:

• filename – the name of a file in the locker.

• include_commit – if the latest commit SHA should be included.

• sha – use this commit SHA; requires include_commit to be False.

• locker_url – use this locker_url instead of self.repo_url.

Returns:

the remote repository path to the filename provided.

get_reports_metadata()

Provide all metadata related to reports as a dictionary.

Returns:

reports metadata.

property head_commit_hexsha

Get SHA of repository head commit.

Returns:

40 byte hexadecimal version of 20 byte binary SHA.

index(evidence, checks=None, evidence_used=None)

Add an evidence to the git index.

Parameters:

• evidence – the evidence object.

• checks – A list of checks used to generate report content. Only applicable for check generated ReportEvidence.

• evidence_used – metadata for evidence used by a check. Only applicable for check generated ReportEvidence.

init()

Initialize the local git repository.

If repo_url was provided to the locker object, the repository will be cloned. If not, a fresh new git repository will be created locally.

init_config()

Apply the git configuration.

load_content(evidence, ignore_ttl=False, evidence_dt=None)

Populate the content of the evidence from the locker.

Parameters:

• evidence – an evidence object.

• ignore_ttl – Boolean for TTL validation. Defaults to False.

• evidence_dt – The date of the evidence content to load.

Returns:

The evidence object with content.

lock = <unlocked _thread.lock object>

logger_init_msgs()

Log locker initialization information.

push()

Push the local git repository to the remote repository.

remove_partitions(evidence, hashes)

Remove partition files for the specified evidence hash keys.

Used when switching from partitioned evidence to unpartitioned evidence and when partitioned file content is no longer part of the overall evidence content.

Parameters:

• evidence – the evidence object.

• hashes – an iterable with evidence partition hashes.

reset_depenency_rerun()

Reset the dependency_rerun attribute to an empty list.

property touched_files

Provide paths to files that have been touched in the local locker.

Files are considered “touched” if they have been added/updated in the locker.

Returns:

a list of touched files.

validate(evidence, ignore_ttl=False)

Validate the evidence against the time to live.

Parameters:

• evidence – an Evidence object.

• ignore_ttl – ignore time to live if set to True.

Returns:

True/False time to live validation.

write_pkg_indexes()

Add package index files to the local git repository index.

Update the package index files with the new timestamp for the updated evidence.

compliance.locker.is_evidence_file(git_obj)

Confirm whether the supplied git object is an evidence file.

Parameters:

git_obj – A GitPython object

Returns:

True or False (Object is or isn’t an evidence file)

compliance.locker.is_index_file(obj)

Confirm whether the supplied object is, or points to, an index file.

Parameters:

obj – Either a GitPython object or a relative file path as a string

Returns:

True or False (Object is or isn’t an index file)

compliance.notify module

Compliance notification management automation module.

class compliance.notify.FDNotifier(results, controls, fd=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, push_error=False)

Bases: _BaseNotifier

File descriptor notifier class.

Notifications are written to the file descriptor specified. Defaults to STDOUT.

notify()

Write notifications into the file descriptor.

class compliance.notify.GHIssuesNotifier(results, controls, push_error=False)

Bases: _BaseMDNotifier

Github notifier class.

Notifications are sent to Github as repository issues. This notifier is configurable via compliance.config.ComplianceConfig.

notify()

Send notifications to Github as repository issues.

class compliance.notify.LockerNotifier(results, controls, locker, push_error=False)

Bases: _BaseMDNotifier

Evidence Locker notifier class.

Notifications are written to the evidence locker.

Parameters:

• results – dictionary generated by compliance.runners.CheckMode at the end of the execution.

• controls – dictionary of checks and the accreditations and controls that they belong to.

notify()

Write notifications into the evidence locker.

class compliance.notify.PagerDutyNotifier(results, controls, push_error=False)

Bases: _BaseNotifier

PagerDuty notifier class.

notify()

Send notifications as PagerDuty alerts.

class compliance.notify.SlackNotifier(results, controls, push_error=False)

Bases: _BaseNotifier

Slack notifier class.

Notifications are sent to Slack channel(s). This notifier is configurable via compliance.config.ComplianceConfig.

MESSAGE_COLORS = {'error': '#9932CC', 'fail': '#D00000', 'pass': '#00D000', 'warn': '#FFD300'}

notify()

Send notifications to Slack channel(s).

compliance.notify.get_notifiers()

Provide a dictionary of all notifier class objects.

This dictionary contains all valid notifier choices for the --notify option in the CLI as keys and their corresponding notifier classes as values.

NOTE: When adding/removing a notifier, update this dictionary accordingly.

compliance.report module

Compliance report build automation module.

class compliance.report.ReportBuilder(locker, results, controls)

Bases: object

This class builds all the required reports for the tests.

build()

Build the reports and store them in the locker.

generate_check_results(rpt_metadata)

Combine the check execution results with associated reports metadata.

This method combines check results with details about associated reports and evidences used, found in the report metadata. It returns a dictionary keyed by check class dot path.

Parameters:

rpt_metadata – Metadata from all report evidence index.json files

generate_toc(rpt_metadata)

Generate a check reports table of contents.

This method generates a TOC based on all report evidence metadata and appends that TOC to the bottom of an evidence locker’s README.md file.

Parameters:

rpt_metadata – Metadata from all report evidence index.json files

get_template_for(test_obj, evidence)

Provide the file path of the template associated to the given evidence.

Parameters:

• test_report_obj – the test object that needs to create report.

• evidence – the ReportEvidence object expected.

render_evidence_with_template(evidence, test_obj)

Render content based on a template.

If template system was selected, this method renders the content using the template for the given evidence.

Parameters:

• evidence – the evidence object to be rendered using its template.

• test_obj – test object which has all the tests for check.

compliance.runners module

Compliance automation flow management module.

class compliance.runners.CheckMode(opts, extra_opts)

Bases: _BaseRunner

The check process flow.

build_reports()

Generate reports based on check results.

fix_failures()

Fix failures if fixer methods are included in checks.

get_checks()

Provide the appropriate compliance framework check classes.

init_config()

Initialize the framework configuration for check execution.

run_checks()

Execute checks.

Returns:

Success (True) if no errors encountered

run_notifiers()

Execute all requested notifiers.

class compliance.runners.ComplianceBaseResult(stream, descriptions, verbosity)

Bases: TextTestResult

Base Compliance result class.

startTest(test)

Start test timer for each test.

stopTest(test)

Report on execution time at the end of each test.

class compliance.runners.ComplianceCheckResult(*args, **kwargs)

Bases: ComplianceBaseResult

Compliance check result class.

addError(test, err)

Add test errors to check results.

Parameters:

• test – a unittest.TestCase object.

• err – a tuple of the form returned by sys.exc_info()

addFailure(test, err)

Add test failures to check results.

Parameters:

• test – a unittest.TestCase object.

• err – a tuple of the form returned by sys.exc_info()

addSuccess(test)

Add test successes and warnings to check results.

Parameters:

test – a unittest.TestCase object.

record(test, status)

Populate the results as expected by downstream reports and notifiers.

Parameters:

• test – a unittest.TestCase object.

• status – a string status (pass, warn, fail, or error)

class compliance.runners.ComplianceTestWrapper(test)

Bases: object

TestCase wrapper class.

Wraps the TestCase test object as an attribute to ensure backwards compatibility with checks, reporting, and notifiers.

class compliance.runners.FetchMode(opts, extra_opts)

Bases: _BaseRunner

The fetcher process flow.

get_fetchers()

Provide all compliance framework fetcher classes.

run_fetchers(reruns=None)

Execute fetchers.

Parameters:

reruns – A list of fetchers in dot notation to rerun

Returns:

Success (True) if no errors other than dependency unavailable

Module contents

Compliance automation package.