API Reference¶

Use pygarble.screening for secrets, PII and profanity, and pygarble.gibberish for gibberish detection. The original pygarble exports remain supported. This reference describes 0.11.0; see Upgrading when upgrading from 0.8.0.

Standalone screening¶

class pygarble.screening.Scanner(categories: Iterable[str] = ('secrets', 'pii', 'profanity'), *, backends: Iterable[str] = (), backend_options: Mapping[str, Mapping[str, Any]] | None = None, detectors: Iterable[ScreeningDetector] = (), builtin: bool = True, min_confidence: float = 0.5, kinds: Iterable[str] | None = None, exclude_kinds: Iterable[str] = (), locales: Iterable[str] = ('us', 'uk', 'in'), profanity_allowlist: Iterable[str] | None = None, profanity_tiers: Iterable[str] = ('strong', 'mild'), secrets_without_context: bool = False, max_input_length: int | None = None)[source]¶

Screen secrets, PII and profanity; no gibberish model is loaded.

Backends supplement built-in rules. Set builtin=False to use only backends/custom detectors. Construct once and reuse for throughput.

categories : Tuple[str, ...]¶

min_confidence : float¶

max_input_length : int | None¶

iter_scan(texts: Iterable[str]) → Iterator[ScanReport]¶

redact(text: str, *, mode: str = 'placeholder', placeholder: str = '[{KIND}]', mask_char: str = '*', categories: Iterable[str] | None = None) → Redaction¶

scan(text: str) → ScanReport¶

scan_batch(texts: Sequence[str]) → list[ScanReport]¶

pygarble.screening.scan(text: str, **kwargs: Any) → ScanReport[source]¶

pygarble.screening.redact(text: str, *, mode: str = 'placeholder', placeholder: str = '[{KIND}]', mask_char: str = '*', **kwargs: Any) → Redaction[source]¶

The default categories are secrets, pii and profanity. Native rules are enabled; optional backends run only when explicitly selected. There is no gibberish profile or threshold setting on this scanner. min_confidence controls whether findings flag the input and are redacted; findings below it remain in scan reports. Reuse a Scanner for repeated calls: the convenience functions construct an instance per call. See Standalone screening and optional backends for options, filtering and custom detectors.

Optional backends and extension contract¶

class pygarble.screening.backends.PhoneNumbersDetector(region: str | None = None)[source]¶

category = 'pii'¶

kinds = frozenset({'phone'})¶

detect(text: str) → tuple[Finding, ...][source]¶

class pygarble.screening.backends.StdnumDetector(formats: Iterable[str] = ('iban', 'in_.pan', 'in_.aadhaar', 'us.ssn', 'gb.nhs', 'br.cpf', 'de.idnr'))[source]¶

category = 'pii'¶

kinds = frozenset({'aadhaar', 'cpf_br', 'iban', 'idnr_de', 'nhs_number', 'pan', 'ssn_us'})¶

detect(text: str) → tuple[Finding, ...][source]¶

class pygarble.screening.backends.DetectSecretsDetector(plugins: Iterable[str] | None = None)[source]¶

category = 'secrets'¶

kinds = frozenset({'detect_secrets_secret'})¶

detect(text: str) → tuple[Finding, ...][source]¶

class pygarble.screening.backends.GitleaksDetector(executable: str = 'gitleaks', timeout: float = 10.0, config: str | None = None)[source]¶

Requires Gitleaks >=8.19,<9. One process is started per document.

category = 'secrets'¶

kinds = frozenset({'gitleaks_secret'})¶

detect(text: str) → tuple[Finding, ...][source]¶

class pygarble.screening.ScreeningDetector(*args, **kwargs)[source]¶

Custom detectors also declare the kinds they can emit.

kinds : FrozenSet[str]¶

exception pygarble.screening.BackendError[source]¶: A requested backend could not complete; the input is not clean.

Backend constructors require their optional package or executable. Importing these adapter classes does not load optional dependencies. Scan errors propagate; they do not become clean reports. A custom detector declares its category and kinds, and returns findings with original-text offsets.

Compatible combined scanner¶

class pygarble.Scanner(categories: Iterable[str] = ('secrets', 'pii', 'profanity', 'gibberish'), *, min_confidence: float = 0.5, kinds: Iterable[str] | None = None, exclude_kinds: Iterable[str] = (), locales: Iterable[str] = ('us', 'uk', 'in'), profile: str = 'english', threshold: float = 0.5, allowlist: Iterable[str] | None = None, profanity_allowlist: Iterable[str] | None = None, secrets_without_context: bool = False, max_input_length: int | None = None)[source]¶

Screen text for secrets, PII, profanity and gibberish.

min_confidence gates the rule categories; gibberish is gated by threshold.

categories : Tuple[str, ...]¶

min_confidence : float¶

max_input_length : int | None¶

iter_scan(texts: Iterable[str]) → Iterator[ScanReport]¶

redact(text: str, *, mode: str = 'placeholder', placeholder: str = '[{KIND}]', mask_char: str = '*', categories: Iterable[str] | None = None) → Redaction¶

scan(text: str) → ScanReport¶

scan_batch(texts: Sequence[str]) → list[ScanReport]¶

pygarble.scan(text: str, **kwargs: Any) → ScanReport[source]¶

pygarble.redact(text: str, *, mode: str = 'placeholder', placeholder: str = '[{KIND}]', mask_char: str = '*', categories: Iterable[str] | None = None, **kwargs: Any) → Redaction[source]¶

This scanner keeps all four categories, including gibberish, enabled by default. Its convenience functions reuse a cached scanner per configuration. It does not accept the standalone scanner’s backends or detectors settings. See Combined screening and redaction for the existing contract.

Findings¶

class pygarble.findings.Finding(category: str, kind: str, start: int, end: int, confidence: float, reason: str)[source]¶

category : str¶

kind : str¶

start : int¶

end : int¶

confidence : float¶

reason : str¶

to_dict() → dict[str, Any][source]¶

class pygarble.findings.ScanReport(findings: tuple[pygarble.findings.Finding, ...], flagged: bool, length: int)[source]¶

findings : Tuple[Finding, ...]¶

flagged : bool¶

length : int¶

by_category() → dict[str, tuple[Finding, ...]][source]¶

kinds() → tuple[str, ...][source]¶

to_dict() → dict[str, Any][source]¶

class pygarble.findings.Redaction(text: str, findings: tuple[pygarble.findings.Finding, ...], count: int)[source]¶

text : str¶

findings : Tuple[Finding, ...]¶

count : int¶

Native screening detectors¶

class pygarble.screening.SecretsDetector(kinds: Iterable[str] | None = None, exclude_kinds: Iterable[str] = (), without_context: bool = False)[source]¶

category = 'secrets'¶

detect(text: str) → tuple[Finding, ...][source]¶

class pygarble.screening.PIIDetector(kinds: Iterable[str] | None = None, exclude_kinds: Iterable[str] = (), locales: Iterable[str] = ('us', 'uk', 'in'))[source]¶

category = 'pii'¶

detect(text: str) → tuple[Finding, ...][source]¶

class pygarble.screening.ProfanityDetector(allowlist: Iterable[str] | None = None, tiers: Iterable[str] = ('strong', 'mild'), obfuscation: bool = True)[source]¶

category = 'profanity'¶

detect(text: str) → tuple[Finding, ...][source]¶

Gibberish detection¶

Language strategies target English. Meaningful non-English text may be flagged; these checks do not establish semantic meaning or identify languages. Scores are heuristic values, not calibrated probabilities.

class pygarble.gibberish.Strategy(*values)[source]¶

CONTROL_CHARACTERS = 'control_characters'¶

LOCAL_ANOMALY = 'local_anomaly'¶

PATTERN_MATCHING = 'pattern_matching'¶

ENTROPY_BASED = 'entropy_based'¶

VOWEL_RATIO = 'vowel_ratio'¶

KEYBOARD_PATTERN = 'keyboard_pattern'¶

MARKOV_CHAIN = 'markov_chain'¶

NGRAM_FREQUENCY = 'ngram_frequency'¶

WORD_LOOKUP = 'word_lookup'¶

SYMBOL_RATIO = 'symbol_ratio'¶

REPETITION = 'repetition'¶

HEX_STRING = 'hex_string'¶

MOJIBAKE = 'mojibake'¶

PRONOUNCEABILITY = 'pronounceability'¶

UNICODE_SCRIPT = 'unicode_script'¶

BIGRAM_PROBABILITY = 'bigram_probability'¶

LETTER_POSITION = 'letter_position'¶

CONSONANT_SEQUENCE = 'consonant_sequence'¶

VOWEL_PATTERN = 'vowel_pattern'¶

LETTER_FREQUENCY = 'letter_frequency'¶

RARE_TRIGRAM = 'rare_trigram'¶

FUNCTION_WORD_DENSITY = 'function_word_density'¶

AFFIX_DETECTION = 'affix_detection'¶

ZIPF_CONFORMITY = 'zipf_conformity'¶

WORD_COLLOCATION = 'word_collocation'¶

LOG_LIKELIHOOD_RATIO = 'log_likelihood_ratio'¶

WORD_ANOMALY = 'word_anomaly'¶

KEYBOARD_ADJACENCY = 'keyboard_adjacency'¶

CROSS_PARSING = 'cross_parsing'¶

PRIMED_COMPRESSION = 'primed_compression'¶

NGRAM_RANK = 'ngram_rank'¶

PERMUTATION_TEST = 'permutation_test'¶

GarbleDetector¶

class pygarble.gibberish.GarbleDetector(strategy: Strategy | str, threshold: float = 0.5, threads: int | None = None, *, allowlist: Iterable[str] | None = None, max_input_length: int | None = None, timeout_per_text: float | None = None, strategy_kwargs: Mapping[str, Any] | None = None, **kwargs: Any)[source]¶

Single-strategy detector.

timeout_per_text only bounds work submitted to the thread pool, which is used for batches of 10 or more when threads is 2 or more. Single strings and small batches run inline and cannot be interrupted.

applicable(text: str) → bool[source]¶

analyze(X: str | list[str]) → Analysis | list[Analysis][source]¶

predict(X: str | list[str]) → bool | list[bool][source]¶

predict_proba(X: str | list[str]) → float | list[float][source]¶: Return a heuristic score, not a calibrated probability.

score(X: str | list[str]) → float | list[float]¶: Return a heuristic score, not a calibrated probability.

EnsembleDetector¶

Combine several strategies under one voting rule.

voting=”fisher” reads each applicable member’s score against that strategy’s score distribution on a synthetic English null (word salads drawn by word frequency) to get a tail p-value from a coarse grid (1.0 down to 0.001), combines them with Fisher’s method, and reports 1 - p ** (ln 0.5 / ln fisher_alpha) so that the default threshold 0.5 flags a combined p-value at or below fisher_alpha. The result is a heuristic: members are correlated and the null is synthetic, so fisher_alpha is not a guaranteed false-positive rate. The tails describe default-constructed strategies on unmodified text; member options and allowlists shift scores but not the tails.

analyze(X: str | list[str]) → Analysis | list[Analysis][source]¶

predict(X: str | list[str]) → bool | list[bool][source]¶

predict_proba(X: str | list[str]) → float | list[float][source]¶

Heuristic aggregate.

Under voting=’majority’ the decision counts member votes, so Analysis.garbled can be True while Analysis.score is below threshold. Under voting=’fisher’ the score maps the combined null p-value; it is not the probability that the text is garbled.

score(X: str | list[str]) → float | list[float]¶

Heuristic aggregate.

Under voting='majority' the decision counts member votes, so Analysis.garbled can be True while Analysis.score is below threshold.

Both classes accept a string or list of strings. predict returns bools, score and predict_proba return floats, and analyze returns immutable analysis records. Batch inputs are validated before any member is evaluated.

Configuration¶

Both constructors accept these settings:

Common settings¶
Setting	Default	Meaning
`threshold`	`0.5`	Decision cutoff in [0, 1]; reaching the cutoff flags applicable evidence.
`allowlist`	`None`	Iterable of vocabulary words for shared English scoring, normalized for case and diacritics. A plain string is not a valid allowlist.
`threads`	`None`	Optional positive worker count for batches. Serial execution is the default.
`max_input_length`	`None`	Optional positive maximum length in Python string characters per input.
`timeout_per_text`	`None`	Optional finite positive timeout in seconds for threaded-result waits.
`strategy_kwargs`	`None`	Settings for one strategy in `GarbleDetector`; mapping from selected `Strategy` members to their settings in `EnsembleDetector`.

GarbleDetector accepts a Strategy enum member or its string value, such as "markov_chain"; an unknown name raises ValueError. EnsembleDetector accepts either a named profile or a nonempty list of strategies. Select one mechanism. weights correspond to strategy-list order and are required for voting="weighted". Weights must be finite, nonnegative, correctly sized, and not all zero. weights are only used with voting="weighted"; passing them with any other voting mode emits a FutureWarning and they are ignored. This will become an error in a future release.

from pygarble.gibberish import GarbleDetector, Strategy

detector = GarbleDetector(
    Strategy.CONTROL_CHARACTERS,
    threshold=0.5,
    strategy_kwargs={"max_combining_run": 8},
)
assert detector.predict("hello\x00world") is True

Return values¶

Scalar and batch results¶
Method	String input	List of strings
`predict`	`bool`	`List[bool]`
`score` / `predict_proba`	`float` in [0, 1]	`List[float]`
`analyze`	`Analysis`	`List[Analysis]`

Batch order is preserved, and an empty batch returns an empty list. Bytes, generators, tuples, and nonstring batch members are not accepted by these methods. GarbleDetector.applicable(text) accepts one string and reports whether its strategy supplies evidence; applicability does not mean the input is garbled.

Profiles and aggregation¶

EnsembleDetector() selects the english profile.

Named profiles¶
Profile	Intended use
`english`	Default. General English screening: Markov, likelihood ratio, word anomaly, mojibake, keyboard adjacency and control characters.
`english_extended`	Adds pattern matching, localized anomalies and repetition; more aggressive, with more potential false positives.
`legacy`	The former three-member set: Markov, likelihood ratio and word anomaly.
`corruption`	Mojibake and control artifacts, independent of English plausibility.
`spoofing`	Unicode script and confusable heuristic.
`llm_output`	Repetition, control characters, mojibake and local anomaly; a deterministic pre-check for degenerate model output that stays quiet on code and technical prose.
`english_fusion`	Opt-in. Word lookup, likelihood ratio and cross parsing combined with `fisher` voting; English plausibility only, so other languages written in Latin letters can be flagged and other scripts get no signal (they read as clean).

See Choosing strategies to choose checks and Detection Strategies for its current members. Profiles use union voting by default (english_fusion uses fisher); an explicit strategies list defaults to majority voting. An explicit voting argument overrides either default. Configure members independently through strategy_kwargs={Strategy.MARKOV_CHAIN: {"min_length": 4}}.

Only applicable members participate. any and all use maximum and minimum scores; average and weighted use applicable means. Majority decisions require strictly more than half the applicable members to cross the threshold, while the reported score is their mean. Thresholding that mean may therefore produce a different decision. Zero-weight members abstain from weighted decisions.

fisher reads each applicable member’s score against a table of that strategy’s scores on a synthetic English null (word salads drawn by word frequency) and takes the smallest tail probability, from 0.5 down to 0.001, whose null threshold the score strictly exceeds; a score that exceeds none gives 1. Fisher’s method combines these p-values, and the reported score is 1 - p ** (ln 0.5 / ln fisher_alpha), so with the default threshold of 0.5 the text is flagged when the combined p-value is at most fisher_alpha (keyword-only, default 0.001, strictly between 0 and 1). Passing fisher_alpha with another voting mode emits a FutureWarning and it is ignored; weights are ignored under fisher as under every mode except weighted. The value is a heuristic: members are correlated and the null is synthetic, so fisher_alpha is not a guaranteed false-positive rate. The null tables describe default-constructed strategies, so member settings and allowlists shift scores without shifting the tables. A strategy listed twice raises ValueError under fisher, since its evidence would count twice; other modes accept repeats as before. The tables load only when fisher voting is used.

from pygarble.gibberish import EnsembleDetector

fusion = EnsembleDetector(profile="english_fusion")
assert fusion.voting == "fisher"
assert fusion.predict("The committee will meet again next week.") is False

An empty input or a set with no applicable members yields False and insufficient_evidence. This does not certify meaningful English.

Limits and compatibility¶

max_input_length raises ValueError for oversized scalar or batch input. The legacy opt-in max_string_length still classifies long non-URL tokens as suspicious. There is no longer a universal implicit long-string decision.

threads must be a positive integer. timeout_per_text affects waits for threaded results, not scalar/serial execution or a hard wall-clock deadline. Python workers cannot be killed; executor shutdown may wait. Errors and timeouts propagate rather than producing clean fallback predictions.

Unknown legacy strategy options emit FutureWarning, attributed to the line in your code that constructed the detector; they will become errors in a future release. EnsembleDetector forwards its shared keyword arguments only to the member strategies that accept them, and warns once for any key that no selected member accepts. Unknown keys in strategy_kwargs warn once per member and are not passed on. Scores and decisions may change after the documented preprocessing and correctness fixes, including when the legacy strategy set is selected.

Calibration¶

pygarble.gibberish.calibrate(detector: Any, garbled: Iterable[str], clean: Iterable[str], *, objective: str = 'f1', max_false_positive_rate: float | None = None, thresholds: Iterable[float] | None = None) → CalibrationReport[source]¶

Score both samples once and sweep candidate thresholds.

objective=”f1” picks the highest F1; objective=”max_fpr” picks the highest recall whose false-positive rate stays within max_false_positive_rate, which is required for and only accepted with that objective. If no candidate satisfies the limit, the cut 1.0 is recommended and recommended.false_positive_rate shows the unmet constraint. Ties resolve to the highest threshold. Under voting=”majority” the ensemble decision counts member votes, so the recommended threshold is applied per member; the report still measures the aggregate score.

The recommended threshold is the midpoint of the gap between the chosen cut and the highest score below it, so it is not itself an observed score; points still lists every observed candidate. The midpoint applies only when the chosen cut is an observed score and candidates were not passed explicitly, which keeps the reported metrics exact; it never applies to the 1.0 fallback.

class pygarble.gibberish.CalibrationReport(recommended: pygarble.gibberish.calibration.ThresholdPoint, objective: str, max_false_positive_rate: float | None, garbled: int, clean: int, points: tuple[pygarble.gibberish.calibration.ThresholdPoint, ...])[source]¶

recommended : ThresholdPoint¶

objective : str¶

max_false_positive_rate : float | None¶

garbled : int¶

clean : int¶

points : Tuple[ThresholdPoint, ...]¶

class pygarble.gibberish.ThresholdPoint(threshold: float, precision: float, recall: float, f1: float, false_positive_rate: float)[source]¶

threshold : float¶

precision : float¶

recall : float¶

f1 : float¶

false_positive_rate : float¶

calibrate(detector, garbled, clean, *, objective="f1", max_false_positive_rate=None, thresholds=None) accepts any detector with a score method that takes a list of strings. garbled and clean must each contain at least one string. objective="max_fpr" requires max_false_positive_rate in [0, 1], and passing it with objective="f1" raises ValueError.

CalibrationReport is a frozen dataclass with recommended (a ThresholdPoint), objective, max_false_positive_rate, the garbled and clean sample counts, and points, a tuple of ThresholdPoint for every candidate in ascending order. ThresholdPoint holds threshold, precision, recall, f1 and false_positive_rate. See Calibrating the threshold for a walkthrough.

Result records¶

class pygarble.gibberish.Analysis(garbled: bool, score: float, status: str, signals: tuple[pygarble.gibberish.analysis.Signal, ...], profile: str, model_version: str = 'english-v2')[source]¶

garbled : bool¶

score : float¶

status : str¶

signals : Tuple[Signal, ...]¶

profile : str¶

model_version : str = 'english-v2'¶

property spans : tuple[Span, ...]¶

class pygarble.gibberish.Signal(strategy: str, score: float, applicable: bool, reason: str, spans: tuple[pygarble.gibberish.analysis.Span, ...] = ())[source]¶

strategy : str¶

score : float¶

applicable : bool¶

reason : str¶

spans : Tuple[Span, ...] = ()¶

class pygarble.gibberish.Span(start: int, end: int, reason: str)[source]¶

start : int¶

end : int¶

reason : str¶

Spans use offsets into the original Python string. Analysis can be converted with dataclasses.asdict and serialized as JSON. The allowlist applies to every strategy: allowlisted words are excluded from English scoring and blanked out of the text that raw-text checks scan. SYMBOL_RATIO reads the raw text, because allowlisted letters can only lower its score. Control characters and encoding damage outside allowlisted words are still reported.

Analysis.status is garbled when the decision is positive, clean when applicable evidence does not flag the input, or insufficient_evidence when no member participates. clean is a detector status, not a guarantee of meaning. Signal.strategy is the strategy’s string identifier. Signal.applicable indicates participation, and reasons describe heuristic evidence. Some strategies report no spans even when their score is positive.

Analysis.model_version identifies the inference contract and is distinct from pygarble.__version__. Record both, along with configuration, when persisting results. Determinism assumes fixed package, settings, and Python/Unicode tables.