Compliance reports¶

The freshdata.compliance subpackage turns a CleanReport into a regulatory audit artifact. Each generator maps the transformations freshdata applied — imputations, outlier handling, normalisations, deduplication, PII masking — onto a named control framework and emits a structured report you can attach to a data-governance workflow.

The generators are purely additive: they never mutate the input report or DataFrame. They read what freshdata already recorded and re-express it against a framework's controls.

Report generation, not certification

A compliance artifact supports a compliance workflow; it does not constitute a legal determination of compliance. Every report carries a verbatim caveat (GENERAL_CAVEAT) to that effect, and review by a qualified compliance professional is required.

Quickstart¶

import freshdata as fd
from freshdata.compliance import generate_compliance_report, ComplianceConfig

cleaned, report = fd.clean(df, return_report=True)

bundle = generate_compliance_report(
    report,
    frameworks=["21cfr_11", "hipaa_safe_harbor"],
    config=ComplianceConfig(operator_id="svc-1", retention_days=2555),
    dataframe=df,            # optional: recovers column roles + missing ratios
)

bundle.summary()
# {'21cfr_11': {'passed': True, 'warnings': [], 'errors': []},
#  'hipaa_safe_harbor': {'passed': True, 'warnings': [...], 'errors': []}}

bundle["hipaa_safe_harbor"].passed     # -> True / False
print(bundle.to_json())                # full audit artifact as JSON

generate_compliance_report also accepts an enterprise result directly: generate_compliance_report(enterprise_result, frameworks=[...]) (or report exposing clean_report + trust_after) folds in the embedded report and trust/mask data automatically.

Frameworks¶

Pass one or more of these keys in frameworks=:

Key	Framework	What it documents
`21cfr_11`	21 CFR §11.10(e)	A tamper-evident audit trail: one entry per action, each marked non-obscuring (a pre-image was retained or the change cannot hide prior data).
`gdpr_30`	GDPR Article 30 + Article 17	Record of processing activities plus erasure (right-to-be-forgotten) evidence.
`alcoa_plus`	ALCOA+	Data-integrity attributes (Attributable, Legible, Contemporaneous, Original, Accurate, +).
`sox_404`	SOX-404 transformation control	Internal-control evidence over data transformations, gated on the Data Trust Score.
`hipaa_safe_harbor`	HIPAA Safe Harbor	Coverage of the 18 Safe Harbor identifiers, flagging any that remain unmasked.

Unknown keys raise ValueError listing the valid keys.

Configuration¶

ComplianceConfig carries the caller-supplied context; every field has a sensible default, so ComplianceConfig() is valid. The most useful knobs:

Field	Default	Purpose
`operator_id`	`None` (`"system"`)	Operator recorded on each 21 CFR audit entry.
`retention_days`	`2555` (7 years)	Retention period stamped on audit entries.
`masked_columns`	`[]`	Columns known to be PII-masked (report-only path; merged with any enterprise mask report).
`trust_score`	`None`	0–100 trust score for the SOX / 21 CFR gates when no enterprise result is supplied.
`fail_on_hipaa_gap`	`False`	Raise `ComplianceGapError` instead of warning when Safe Harbor gaps remain.
`strict_cfr_normalization`	`False`	See below.
`controller_name` / `controller_contact` / `processing_purpose` / `legal_basis` / `data_subject_categories`	—	GDPR Article 30 record fields.

`strict_cfr_normalization`¶

Controls how the 21 CFR audit classifies normalising rewrites (whitespace trims, sentinel canonicalisation):

False (default) — these lossless normalisations are treated as non-obscuring: trimming " Ann " to "Ann" cannot hide prior information, so the gate stays green. Only genuine value rewrites with no retained pre-image (outlier capping, fuzzy clustering) fail the gate.
True — any rewrite that did not retain a pre-image, including normalisation, is treated as obscuring: the entry's original_value_class becomes "not_captured", non_obscuring_guarantee becomes False, a warning is emitted, and the gate fails. Use this when your SOP requires an independently recorded pre-image for every value change.

Richer evidence¶

Two optional, keyword-only arguments add evidence when available:

dataframe= — the source frame. Recovers per-column roles and missing ratios via freshdata.infer_roles, sharpening the HIPAA and ALCOA reports. Without it, the full column list comes from the report's input_columns (recorded by freshdata.clean). If neither is available, the HIPAA report sets coverage_verifiable to False, adds a warning, and does not pass.
enterprise_result= — an enterprise result supplying the 0–100 Data Trust Score, PII-masking events, and fuzzy-clustering lineage.

Output¶

generate_compliance_report returns a ComplianceBundle — a mapping of framework key to FrameworkReport:

On `ComplianceBundle`	Returns
`bundle.summary()`	`{key: {"passed", "warnings", "errors"}}` for a quick gate check.
`bundle[key]`	The `FrameworkReport` for one framework.
`key in bundle`, `len(bundle)`, `list(bundle)`	Membership, count, and the framework keys.
`bundle.to_dict()` / `bundle.to_json(indent=2)`	The full artifact as a dict / JSON string.
`bundle.to_frame()`	All reports flattened into a single `pandas.DataFrame`.

Each FrameworkReport exposes framework_key, framework_name, passed (bool), warnings, errors, and data (the framework-specific payload, e.g. the 21 CFR audit_entries or the HIPAA identifier coverage), plus its own to_dict(), to_json(), and to_frame().

Pseudonymisation keys¶

The privacy policy engine (freshdata.enterprise.apply_privacy_policy) ships HIPAA, FERPA, PCI and GDPR packs. Their pseudonymize rules (the GDPR pack's default action, the HIPAA date-of-birth rule and the FERPA grade rule) are keyed, as are the tokenize, surrogate and fpe strategies of MaskingRule in anonymize and clean_enterprise.

Pass a secret key for stable, joinable output, preferably from the environment:

from freshdata.enterprise import PrivacyPolicy, apply_privacy_policy, load_compliance_pack

policy = PrivacyPolicy(
    packs=(load_compliance_pack("gdpr"),),
    jurisdiction="EU",
    key_env="FRESHDATA_PSEUDONYM_KEY",
)
out, report = apply_privacy_policy(df, policy)

Without a key, each call uses a random key and emits EphemeralKeyWarning (importable from freshdata.enterprise), and report.metadata["ephemeral_key_rules"] lists the rules that used it. The output cannot be recomputed from FreshData's source, but it changes on every call, so it cannot be joined across runs. Policy tokenize without a key still raises ValueError, as do reversible=True tokenize / fpe masking rules.

Reproducing output from before 2.1.0¶

Before 2.1.0 these paths used constants from the public source when no key was set. Anyone with that output and a list of candidate values can recompute the pseudonyms, so treat it as reversible and re-pseudonymise it with a secret key. If you must reproduce the old output for a while (for example, to join against an existing table during a migration), pass the old constant as an explicit key. These keys are public: never use them for new data.

Before 2.1.0 (no key)	Explicit key that reproduces it
`MaskingRule(name=rule_name, strategy="tokenize")`	`key=hmac.new(b"freshdata-default-token-salt", rule_name.encode(), hashlib.sha256).hexdigest()[:32]`
`MaskingRule(strategy="surrogate")`	`key="freshdata-surrogate"`
`MaskingRule(strategy="fpe")`	`key="freshdata-surrogate"`, only when `pyffx` is not installed (with `pyffx`, a keyed `fpe` rule uses real FPE)
policy `pseudonymize`	`PrivacyPolicy(..., key="freshdata-surrogate")`, only when `pyffx` is not installed (a keyed `pseudonymize` uses FPE when `pyffx` is available)

Errors¶

ValueError — an unknown framework key was requested.
ComplianceGapError — HIPAA Safe Harbor gaps remain and config.fail_on_hipaa_gap=True. Otherwise gaps surface as warnings on the report and passed is False.

Caveat¶

Every report embeds GENERAL_CAVEAT (importable from freshdata.compliance) unless a framework defines its own verbatim caveat. It states plainly that the artifact documents freshdata's transformations and their mapping to the named control framework, but is not a certified compliance system and does not constitute a legal determination — professional review is required.