Understanding Security Validation

Security validation uses controlled tests and evidence to check whether security controls work as expected. It moves beyond asking whether a control is installed and asks whether the organisation can see, detect, investigate, and respond to relevant activity.

What you will learn

This page explains common security-validation approaches, where Breach and Attack Simulation (BAS) fits, how a safe validation exercise works, and where Caldera fits.

What is security validation?

Security validation compares an expected defensive result with evidence from an authorised test. A useful exercise checks more than whether a test action can run.

Repeatable testing

Run the same approved behaviour again after a defensive change to confirm whether the improvement works.

Visibility checks

Confirm that endpoint, network, identity, or cloud activity reaches the tools expected to record it.

Detection checks

Verify that useful alerts are created with enough information for an analyst to understand and investigate them.

Response checks

Test whether the expected manual or automatic response occurs and whether it causes unintended disruption.

Use only with clear authorisation

Attack simulations can interrupt services, change systems, or trigger automatic security actions. Define the approved systems, actions, timing, owners, stop conditions, and recovery plan before testing.

Common validation approaches

Assessment	Main question	Typical approach
Vulnerability scanning	Is software known to have a weakness?	Checks versions, settings, and known weaknesses
Penetration testing	Can an authorised tester find and use a path into the environment?	Human-led testing within an agreed scope
Red teaming	Can a team achieve a realistic objective while avoiding detection?	Goal-based exercise that tests people, processes, and technology
Adversary emulation	How would the environment respond to behaviour associated with a particular threat?	Recreates selected attacker methods in a controlled way
BAS	Do security controls repeatedly detect and respond to selected attack behaviours?	Automated or partly automated, repeatable security validation

These approaches overlap. A platform may support more than one, and the appropriate method depends on the question being asked.

Where BAS fits

Breach and Attack Simulation (BAS) runs repeatable, automated or partly automated test actions against approved systems. These actions imitate selected attacker behaviours, such as running a command, changing a system setting, or contacting a controlled network destination.

BAS is particularly useful for checking the same defensive behaviour regularly and retesting after a change. It does not replace human-led penetration testing, red teaming, or careful exercise design.

Security validation workflow

1. DefineDecide which defensive control, detection, response, or assumption needs to be tested.Ask one clear question
2. ScopeApprove the systems, accounts, network paths, actions, owners, and test window.Set firm boundaries
3. Check safetyUnderstand each action, expected effect, dependencies, stop condition, and recovery method.Control the risk
4. PrepareConfirm that security tools, system owners, and responsible teams are ready.Make results observable
5. RunExecute only the approved actions while monitoring systems and services.Watch for impact
6. CompareCheck expected visibility, alerts, investigation details, and response against actual results.Record evidence
7. ImproveFix useful gaps in data, detection, configuration, process, or response.Make a targeted change
8. RetestRepeat the controlled test to confirm that the improvement works.Verify the outcome

Try a defensive validation

Choose a test scenario, then record which defensive checkpoints succeeded. Running the simulated action alone does not prove that the defence worked.

1. Select a test

Command executionRun an approved test command Account behaviourUse a controlled test account Network connectionContact an approved test destination

2. Record successful checkpoints

Action completedThe test behaved as planned Activity recordedThe expected security data arrived Alert createdThe detection fired with useful context Response verifiedThe expected response worked safely

0/4 Defence not yet validated Select the checkpoints supported by evidence from the exercise.

Key takeaway

A successful test action is only the beginning. Validation should follow the complete chain from activity and visibility through detection, investigation, and response.

What should be measured?

• Whether the test action completed as expected
• Whether relevant activity records were collected
• Whether the intended detection fired
• Whether the alert contained enough context
• How long detection and investigation took
• Whether automatic or manual response worked
• Whether the test caused unexpected impact
• Whether improvements still work when the test is repeated

A failed detection can still be a useful result

BAS is designed to reveal gaps safely. Record why the test was missed—for example, missing data, incorrect rules, unsupported behaviour, or a process failure—then make and verify a targeted improvement.

Safe testing principles

• Use dedicated test systems where practical.
• Understand every action before running it.
• Begin with low-impact tests.
• Keep test accounts and information separate from real users and sensitive data.
• Coordinate with system owners and the security team.
• Monitor resource use and service health.
• Define an immediate stop method.
• Record actions, results, changes, and unexpected effects.
• Restore the environment and remove test material afterward.
• Never treat a tool's default content as automatically safe for every environment.

Caldera platform profile

Caldera is an open-source platform for automated adversary emulation and security assessment. Adversary emulation means reproducing selected attacker behaviours in a controlled and authorised exercise.

Area	Evaluation starting point
Best suited to	Teams that want repeatable, ATT&CK-aligned security exercises and are prepared to review and control test actions
Core model	Agents run approved abilities—individual test actions—as part of an operation
Planning	Operations can follow defined sequences or use automated planning to select available actions
Security use	Validate endpoint and network visibility, detections, investigation processes, and selected response controls
Operational planning	Isolation, permissions, test accounts, plugins, agent control, network access, logging, cleanup, and recovery
Important boundary	Caldera can support BAS and adversary-emulation work, but the platform does not make an exercise safe or useful without careful design and oversight

Open Caldera guide ›

Continue learning

• Explore Caldera to build and run an authorised adversary-emulation exercise.
• Understand Threat Frameworks to learn how attacker behaviours are organised and described.