Normalize input without corrupting legitimate characters

How does PHP normalize text input without corrupting names, accents, or punctuation?

The application accepts a defined character encoding and only standardizes properties whose meaning is unambiguous, such as CRLF in text fields or outer whitespace. The normalized value is validated field-specifically; HTML, SQL, mail, or shell escaping only occurs in the respective output context.

Context-late escaping

Field-specific rule

• Field-specific rule – Name, email, identifier, and free text have different allowed characters, lengths, and normalization steps.

• Preserved meaning – Accents, hyphens, apostrophes, and non-Latin characters are preserved if they are a legitimate part of the value.

• Context-late escaping – The stored business value is not pre-encoded in HTML but is handled safely according to the specific output.

Practical scenario: "Name manipulation"

A name field previously removed everything except the letters A through Z, resulting in incorrect values ​​for names like O’Connor and Łukasz. The new rule only trims outer spacing, accepts Unicode characters and appropriate punctuation; the subsequent HTML output is escaped separately.

Name manipulation

• Name manipulation An ASCII whitelist removes umlauts, diacritics, and apostrophes, and alters real-world personal and company names.

• Double encoding HTML escaping before saving is reapplied during subsequent output, displaying entities instead of the original characters.

• Security Illusion Character sanitization does not replace parameterized database queries or context-specific protection in HTML or headers.

Preserved meaning

1. Document the business meaning, expected encoding, legitimate characters, length, and multi-line format for each input field.

2. Normalize only unique margins, line endings, and, if applicable, Unicode format, and test changes in a traceable manner.

3. Validate the normalized value and only use it at each sink with its intended parameter or escaping mechanism.

What questions arise next?

Storing Passwords with Modern Hashes Instead of Custom Logic Answers the next practical question: How does PHP store and update password hashes without its own cryptographic logic?

Securely and User-Friendly Integrating File Uploads into the Request Process continues this line of thought with another question: How can file uploads be integrated securely and user-friendly into a request process?

If you want to practically implement "lossless normalization of input," you can refer to Robust Website Systems This document focuses on "form input and secure processing" and "field-based rules."

Conclusion: Lossless normalization of input

Normalization should create comparable representations, not simplify language. Field knowledge and late context escaping retain their importance, while genuine security mechanisms address the data loss points.

Sources and Further Information

The primary sources define the technical framework for "lossless normalization of input."

• Unicode Normalization Forms — Unicode Standard Annex #15Unicode UAX #15 defines the normalization forms NFC, NFD, NFKC, and NFKD, as well as their different compatibility properties.

• Normalizer::normalize — PHP ManualThe PHP manual documents programmatic Unicode normalization and the selectable normalization forms.

• Input Validation Cheat Sheet — OWASPOWASP explains syntactic and semantic validation, Unicode handling, and context-aware positive rules for input.