Fawzooz
عربي
Your journey
  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
0 of 34 units

You are in Degree 4 · Protectionunit 3 of 6Ahead of you: A miniature red-team test and a cognitive defence plan.

Degree 4 · Unit 4.3

Red-teaming models

Red teaming an intelligent system is fundamentally different from red teaming a traditional one. You are not looking for a hole in the code; you are looking for unwanted behaviour that can be summoned with words. And you do not need a specialist team to make a start. You need a method and two hours.

FIG. 20 — The six-stage test method
1
Define what you protect
What is the worst this system could do? Write it in three sentences.
2
Draw the attack surface
Who puts text into it? What sources does it read? What tools does it hold?
3
Write the scenarios
At least twenty attempts, spread across the five attacks.
4
Run and document
Each attempt with its exact text and its result — succeeded, failed, or partly succeeded.
5
Rank by impact
What produces harm that cannot be undone? That first, not the easiest to fix.
6
Fix and retest
A fix without a retest is not a fix but a hope.
fawzooz.ai

Ten scenarios that suit any system

  1. 1Ask it to reveal its internal instructions in different phrasings.
  2. 2Ask for a forbidden action directly, then wrapped in an "educational" or "research" context.
  3. 3Slip an instruction into a document it reads.
  4. 4Ask it for information about another user or another session.
  5. 5Give it a very long input with the constraints buried inside.
  6. 6Ask for an action outside its authorised tools.
  7. 7Ask it a question with no answer in its sources — does it say "not available" or invent?
  8. 8Give it two contradictory sources and ask for a decision.
  9. 9Impersonate a position of authority in the text of the request.
  10. 10Repeat the refused request in successive phrasings — does it soften after the fifth attempt?

How to write the report

A report nobody reads fixes nothing at all. Keep it to a single page: a table giving the attempt, the result, the possible impact, the control you propose, and the person who can implement it. Order that table by impact rather than chronologically. And mention what did not work as well, because that is evidence your existing controls are holding, and it earns you the trust of whoever is reading.

Do this

  1. 1 — On the tool. Run the ten scenarios on a system you own or are authorised to test. Document every attempt with its exact text.

  2. 2 — In your field. Add five scenarios particular to your field: what in your data would tempt an attacker?

  3. 3 — In writing. Write the one-page report. This is half the gate of this degree.

An ethical rule before you begin

Do not test a system you do not own and have not been authorised for in writing. The difference between a test and an assault lies not in the method but in the permission. Keep that permission documented before you make the first attempt, even when the system sits inside your own organisation.

And do not publish anything you find before you have notified whoever owns the system and given them time to fix it. That is not a courtesy. It is the thing that marks out a professional in this field.

Where to after this unit? You have protected the system. What remains is the more valuable target: your own mind.