BCCard/MoAI-Privacy-Filter

1. Overview

A Korean/English PII detection model for the finance domain, built by full fine-tuning openai/privacy-filter (1.4B MoE, 50M active) on synthetic finance-domain PII data. It tags 18 PII entity types (73 BIOES classes) at token level and is intended as the NER layer of a multi-layer PII-masking gateway in front of LLM services, as well as a PII detection component for offline privacy review and audit workflows.

On held-out validation it reaches strict span-F1 0.956 (ko) / 0.969 (en). On an independent, adversarially-hardened Golden Set it holds 0.944 (ko) / 0.907 (en) with masking coverage 0.996 (ko) / 0.998 (en) — i.e. ≥99.5% of gold PII characters are covered by predicted spans.

1.1. TL;DR

• Base model: openai/privacy-filter — 1.4B-parameter MoE (128 experts, 50M active), 8 layers, hidden 640, bidirectional banded attention (±128), o200k tokenizer
• Domain / Language: Finance (BC Card — cards, accounts, national IDs, customer service text) / Korean + English
• Task: Token classification (BIOES) → character-offset PII spans → masking
• Labels (18): PERSON, RRN, FRN, CARD_NUMBER, ACCOUNT_NUMBER, SECRET, USER_ID, EMAIL, PHONE, PASSPORT, DRIVER_LICENSE, GENERIC_ID, ADDRESS, ZIPCODE, DATE, CARD_EXPIRY, CVC, IPIN
• Method: Full fine-tuning (all parameters incl. experts & router) with a re-initialized 73-class head (rows copied from the base head by taxonomy mapping)
• Decoding: constrained BIOES Viterbi (not per-token argmax) + whitespace span refinement — the bundled viterbi_calibration.json exposes precision↔recall operating-point biases without retraining
• Format: BF16 (attention sinks kept FP32), single safetensors + tokenizer + label taxonomy + Viterbi calibration sidecar
• Sequence length: trained on sequences ≤ 768 tokens — chunk longer inputs
• Intended use:
  1. In-house PII masking gateway (detect → mask before text reaches an LLM)
  2. Offline privacy review and audit support (PII discovery in stored text, logs and documents)

1.2. Label Taxonomy (N=18)

The 18 labels re-map the upstream ai4privacy source labels to the granularity a Korean financial masking policy needs - merging fragments into single spans (GIVENNAME/SURNAME → PERSON, CITY/STREET/BUILDINGNUM → ADDRESS) and adding Korea-specific classes absent upstream (RRN, FRN, IPIN, CARD_EXPIRY, CVC, SECRET). data source records the row-source buckets in which each label occurs: ko means openpii-1.5m-ko, en means openpii-1.5m-en, and domain means locally synthesized rows.

label	description	data source
PERSON	full name (surname + given, single span)	ko, en, domain
RRN	resident registration number (Korea)	ko, domain
FRN	foreign registration number (Korea)	domain
CARD_NUMBER	credit/debit card PAN	ko, en, domain
ACCOUNT_NUMBER	bank account number	ko, domain
SECRET	auth secret (password / API key / token)	ko, domain
USER_ID	online member ID	ko, en, domain
EMAIL	email address	ko, en, domain
PHONE	phone number (mobile / landline)	ko, en, domain
PASSPORT	passport number	ko, en, domain
DRIVER_LICENSE	driver's license number	ko, en, domain
GENERIC_ID	generic identifier without a more specific taxonomy class	ko, en, domain
ADDRESS	address (city / street / building, single span)	ko, en, domain
ZIPCODE	postal code	ko, en, domain
DATE	date / time	ko, en, domain
CARD_EXPIRY	card expiry date	domain
CVC	card verification code	domain
IPIN	I-PIN number (Korea only)	domain

Each entity type has B-, I-, E- and S- boundary classes, plus the background class O. This yields 73 output classes.

1.3. Usage

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

model_id = "BCCard/MoAI-Privacy-Filter"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForTokenClassification.from_pretrained(model_id)
model.eval()

text = "고객 모아이님(000000-0000000)께서 010-0000-0000로 연락 요청하셨습니다."
enc = tokenizer(text, return_offsets_mapping=True, add_special_tokens=False, return_tensors="pt")
offsets = enc.pop("offset_mapping")[0].tolist()

with torch.no_grad():
    logits = model(**enc).logits.float()  # [B, T, 73]

# Decode logits[0] with constrained BIOES Viterbi and map token tags through offsets.
print(tuple(logits.shape))

Raw logits shape and decoded spans:

(1, 30, 73)
[
  {'start': 3, 'end': 6, 'label': 'PERSON'},
  {'start': 8, 'end': 22, 'label': 'RRN'},
  {'start': 26, 'end': 39, 'label': 'PHONE'}
]

The offsets use Python's half-open character interval [start, end). Masking is downstream policy logic. For example, the spans above can produce:

고객 [PERSON]님([RRN])께서 [PHONE]로 연락 요청하셨습니다.

For batches, enable right padding and pass only input_ids and attention_mask to the model. offset_mapping stays outside the model and is used only to map decoded token tags back to the original text. Convert logits to FP32 before constrained Viterbi decoding, as shown above.

Decoding note - this model, like its base, is trained with a supervised token-level BIOES classification objective and is intended to be decoded with constrained Viterbi over the BIOES transition grammar, not independent per-token argmax. Independent argmax can emit invalid BIOES sequences and is not the decoding path used for the reported metrics. The bundled viterbi_calibration.json follows the upstream operating-point schema. Its six transition biases allow users to adjust the precision-recall trade-off without retraining. All-zero biases mean no additive operating-point adjustment; BIOES transition constraints remain active.

1.4. Training Data

Dataset	Role	Size
(Public) BCCard/pii-masking-openpii-finance (v2)	Training / Validation	~58.5k train rows · ~14.5k validation rows
(Private) BCCard/pii-masking-openpii-finance-test (v2)	Golden Set (release evaluation; not used for training/tuning)	2,000 rows (ko 1,460 / en 540)

• Sources: curated Korean subset of ai4privacy/pii-masking-openpii-1.5m (label taxonomy remapped, name spans merged & naturalized) + finance-domain synthetic templates + ~30% English replay (catastrophic forgetting guard)
• Hard-example design baked into v2: surface-similar non-PII decoys (FP suppression), label-confusion pairs in one sentence (RRN↔FRN, DRIVER_LICENSE↔GENERIC_ID), weak-context true PII (FN suppression), long-span address boundary variants
• All values are synthetic; validity-pattern collisions with real identifiers are removed at generation time (e.g. card numbers are forced to fail Luhn)

1.5. Training Procedure

Item	Value
Method	Full fine-tuning (1.4B params — experts and router included)
Head	33-class base head → 73-class head, initialized by copying base rows via taxonomy mapping
Loss	Token-level cross-entropy
Batch	effective 16 (per-device × world × accum), fixed across hardware layouts
LR / scheduler	1e-4 / linear decay, warmup 3%
Optimizer	AdamW (fused), weight decay 0.0, max_grad_norm 1.0
Epochs	5 — best checkpoint by validation span micro-F1, decoded with the same constrained Viterbi as deployment
Precision	FP32 master weights + BF16 autocast; MoE router/experts explicitly kept FP32 during compute
Hardware	1× NVIDIA H100 (~5h)

2. Evaluation

2.1. Setup

• Golden Set: independently generated 2,000-row test set (ko 1,460 / en 540), adversarially hardened — weak-context PII, decoys, confusion pairs and boundary variants are deliberately over-represented, so scores here read lower than typical in-distribution synthetic benchmarks
• Protocol: strict exact-match span P/R/F1 (CoNLL-style; boundary and label must both match) + masking coverage (share of gold PII characters covered by predicted spans, label-agnostic — the leakage-oriented metric)
• Decoding: constrained Viterbi + whitespace refinement — identical to the deployment chain

2.2. Results

validation dataset

Metric	v2 model / v2 validation
micro F1	0.9599
macro F1	0.9603
ko strict micro F1	0.9562
ko macro F1	0.9568
en strict micro F1	0.9688
en macro F1	0.9631
masking coverage	0.9979
ko masking coverage	0.9987
en masking coverage	0.9965

Observed masking coverage is 99.8% overall and at least 99.6% in both the Korean and English validation slices. This leakage-oriented metric is reported as a diagnostic rather than a release gate.

• These values were measured post-hoc by running the exported final-bf16 artifact over all 14,543 v2 validation rows (ko 10,460 / en 4,083) through the deployment-equivalent chain: constrained Viterbi, actual tokenizer character offsets and whitespace refinement.
• Overall micro F1 and masking coverage pool all ko/en spans or characters before scoring. Overall macro F1 pools per-label TP/FP/FN across both languages and then averages the 18 label F1 values.
• Character coverage counts are ko 482,861 / 483,488 and en 264,979 / 265,898 gold PII characters.
• The training-time checkpoint-selection metrics remain ko micro F1 0.9820, en micro F1 0.9739 and global macro F1 0.9764 at epoch 5. They compare entity spans on token indices, so they are not interchangeable with the character-span values above and do not include masking coverage.

test dataset

Independently generated Golden Set — deliberately harder than validation: weak-context PII, surface-similar decoys, label-confusion pairs and long-span boundary variants are over-represented. Δ = vs. the post-hoc validation baseline above using the same final-bf16 artifact and character-span evaluation chain. The difference measures test hardening and distribution shift, not model regression.

Metric	v2 model / v2 test	Δ
micro F1	0.9336	-2.63%p
macro F1	0.9308	-2.94%p
ko strict micro F1	0.9441	-1.21%p
ko macro F1	0.9416	-1.53%p
en strict micro F1	0.9065	-6.24%p
en macro F1	0.9017	-6.14%p
masking coverage	0.9964	-0.16%p
ko masking coverage	0.9956	-0.31%p
en masking coverage	0.9984	+0.18%p

• Masking coverage stays ≥0.9956 on the adversarial set — only 0.44% (ko) / 0.16% (en) of gold PII characters are uncovered; most strict-F1 losses are boundary or label-name errors, not leaks
• English ADDRESS holds on hard boundary variants: strict recall 0.983 on long-span address forms (state suffixes, unit/floor tails) that are heavily represented in this set
• Weak-context person names are the main remaining leak channel: ko PERSON strict recall 0.875 with 82 full-span misses (see Limitations)
• Label-swap errors (e.g. en ACCOUNT_NUMBER predicted as GENERIC_ID/CARD_NUMBER) keep coverage 1.0 — the value is still masked; only the label name is wrong

2.3. Reading the numbers

Strict exact-match span-F1 on an adversarial test is a deliberately harsh score: a one-character boundary miss or a swapped label counts as a full error. For diagnosing character-level exposure, masking coverage is the direct diagnostic metric: 0.44% (ko) / 0.16% (en) of gold PII characters uncovered, concentrated in weak-context person names.

2.4. Limitations

• One layer of defense — inherits the base model's positioning: not an anonymization or compliance guarantee. Deploy behind a regex backstop for fully structured identifiers (RRN patterns, card numbers, phones) and combine with policy-level controls.
• Weak-context person names — Korean names without honorifics/particles or list-form values are the main miss channel (ko PERSON recall 0.875 on the adversarial set). Consider a recall-leaning Viterbi operating point in high-sensitivity deployments.
• Alphanumeric ID confusion — USER_ID/SECRET/GENERIC_ID/ACCOUNT_NUMBER share surface forms; without cue words the label may swap (masking still applies — coverage stays ~1.0).
• Synthetic-only training & evaluation — no real customer text was used or evaluated. Real-world robustness (typos, slang, OCR noise) is unvalidated; shadow-mode rollout is recommended before enforcement.
• Fixed label policy — the 18-label taxonomy is baked in at fine-tuning time; changing masking policy granularity requires re-fine-tuning (runtime keep/mask toggles must operate on these labels).
• Context window — banded attention limits each token's context to ±128 tokens; trained sequence regime is ≤768 tokens (chunk longer documents).

3. Future Work

• v3 data enhancements - weak-context person-name hard positives, more diverse cue words for alphanumeric IDs, and privacy-safe failure collection from shadow-mode operation
• Operating point - recall-leaning Viterbi transition biases tuned on validation or a dedicated calibration set without Golden Set feedback
• Serving - target-hardware latency, throughput and memory benchmarks for the separately published INT8 weight-only ONNX artifact

4. Meta Info

4.1. Citation

@misc{bccard2026moaiprivacyfilter,
  title        = {MoAI-Privacy-Filter: A Korean Finance-Domain PII Detection Model},
  author       = {BC Card AX Team},
  year         = {2026},
  howpublished = {https://huggingface.co/BCCard/MoAI-Privacy-Filter},
  note         = {Full fine-tune of openai/privacy-filter for Korean/English PII masking in the BC Card domain}
}

4.2. See Also

• Base model: openai/privacy-filter
• INT8 ONNX artifact: BCCard/MoAI-Privacy-Filter-INT8
• Training dataset: BCCard/pii-masking-openpii-finance
• Source data attribution: ai4privacy/pii-masking-openpii-1.5m (CC-BY-4.0)