Best Practices: Backgrounds in Natural Language Code Policies

Examples of what to do and what to avoid when writing the "Background" for your Natural Language Code Policies.

Purpose

When you provide more clarity in your Natural Language Code Policy's background, your policy becomes more reliable and accurate.

What is a Background?

When writing your Natural Language Code Policy, you provide:

• Question: The question to ask against a pull request. This should be a yes or no question.
• Background: Optional background information about the question. This should provide context for your question.
• Guidance - Optional guidance your policy will provide a viewer when a finding is identified. This should be a clear set of instructions.

Applying the tips below to your background will help your policies to:

• Accurately scope which files are considered
• Reason about your code
• Decide when deeper tools (like codebase search or full-file loading) are necessary

1. Scope File Paths When Possible

✅ Do this:

• "Only apply this check to files in /src/controllers/**"
• "Limit this check to files under services/auth/ and middleware/"

❌ Avoid this:

• "Check the whole codebase for this" — (too vague, slow, and may introduce noise)
• "Anywhere this might happen" — (ambiguous and not helpful for targeting)

2. Target Specific Layers or Components

✅ Do this:

• "Authorization is enforced at the controller layer — be sure to check it"
• "Verify any additions to the service layer that might bypass our middleware"

❌ Avoid this:

• "Just make sure this is secure" — (doesn't guide where your policy should look or what to expect)
• "Check backend stuff" — (too generic)

3. Use File or Function Naming Cues

✅ Do this:

• "Look for function names starting with validate or containing auth"
• "Check for uses of unsafeEval() or similar patterns in JavaScript"

❌ Avoid this:

• "Make sure dangerous functions aren't used" — (without naming the functions or providing examples)
• "Find anything sketchy" — (subjective and lacks technical signal)

4. Indicate When Only PR Diffs Should Be Analyzed

✅ Do this:

• "Only analyze changes introduced in this PR"
• "Ignore unchanged files — only evaluate lines added or modified"

❌ Avoid this:

• "Look through the repo to see if anything is wrong" — (forces full analysis unnecessarily)
• "Verify the system works as intended" — (Your policy is not test automation!)

5. Mention When Full-File or Tool-Based Analysis Is Needed

✅ Do this:

• "If a helper is called, trace its definition from the imports"
• "Use search tools if the current file doesn't define the authorization logic"

❌ Avoid this:

• "You might need to look around" — (too vague to trigger action)
• "Do whatever you need to do" — (unhelpful for your policy to provide meaningful insights)

6. Explain Known Indirection Patterns

✅ Do this:

• "Database access may happen via helper functions in db/helpers.ts"
• "Authorization is often wrapped in ensureAuthorizedUser() — follow that chain"

❌ Avoid this:

• "Things are abstracted sometimes" — (does not explain how or where)
• "Might be hidden" — (useless without hints for tracing)

7. Define What "Secure" Means

✅ Do this:

• "Tokens must be validated using secureCompare()"
• "CORS must not allow * unless isTrustedOrigin(origin) returns true"

❌ Avoid this:

• "Just make sure it is secure" — (Every organization defines "secure" differently!)
• "CORS should be safe" — (What is does "safe" even mean?)

8. Call Out Recognized Enforcement Mechanisms

✅ Do this:

• "Authorization is enforced via the require_role decorator"
• "Secure file uploads must go through validateFileType()"

❌ Avoid this:

• "Look for auth stuff" — (doesn't clarify what the correct mechanism is)
• "Check file uploads for problems" — (What kind of problems?)

9. Avoid Vague or Abstract Language

✅ Do this:

• "Check that logging statements do not include variables like password, token, or apiKey"

❌ Avoid this:

• "Make sure secrets are not logged" — (What is a "secret"? What logging system?)
• "Scan for unsafe logging" — (What qualifies as unsafe?)

Recap

The tips above provide examples of things to do and things to avoid in the background of your Natural Language Code Policy so that your policy will work smarter for you!