Shannon 3.0 against Aikido and XBOW
Doyensec compared Aikido and XBOW on two open-source apps. We ran Shannon 3.0, our open-source pentester, against the same Photoview deployment with three models: all three caught the critical SQL injection, and the cheapest run cost $6.10 in tokens.
Same deployment, admin-only access like XBOW. Three Shannon runs, each for a fraction of a $4,000 scan.
Results at a glance
$6.10
Cheapest run
DeepSeek v4 Flash caught the critical pre-auth SQL injection (CVSS 9.8) for $6.10 in tokens.
23
Verified issues
Found for $115 with Claude Opus 5, against XBOW’s 7 at $4,000.
6/7
Later-patched vulnerabilities
Caught by the Claude Opus 5 run, of the 7 Photoview patched after 2.4.0.
35x to 656x
Cheaper than a $4,000 scan
Claude Opus 5 at 35x, Grok 4.6 at 114x, DeepSeek v4 Flash at 656x.
Aikido and XBOW figures are from Doyensec’s study. Aikido reported the most true positives of any run (32). Full results below.
Five scans of one deployment
We verified every Shannon finding by hand: whether it was a true positive, and how accurate its severity rating was. Aikido and XBOW results are from Doyensec’s study, where a researcher validated each finding and re-scored its severity.
| Scan | Cost | Scan time | Reported | True positives | False positives | Exact severity agreement |
|---|---|---|---|---|---|---|
| Shannon 3.0 · DeepSeek v4 Flash View report SARIF | $6.10 | 2h 37m | 18 | 18 | 0 | 72% |
| Shannon 3.0 · Grok 4.6 View report SARIF | $35.07 | 5h 26m | 10 | 10 | 0 | 50% |
| Shannon 3.0 · Claude Opus 5 View report SARIF | $115 | 2h 24m | 24 | 23 | 1 | 62% |
| Aikido | $4,000 | Under 8h | 32 | 32 | 0 | 66% |
| XBOW | $4,000 | About 2 days | 7 | 7 | 0 | 57% |
Shannon costs are model-token spend. Aikido’s Standard tier and XBOW’s Plus tier each cost $4,000 per scan in Doyensec’s study.
The same target, admin access only
Shannon ran against the Photoview 2.4.0 deployment Doyensec tested, with the admin credentials only, as XBOW did.
-
The study
In Comparing AI Application Security Testing Platforms, an Aikido-sponsored study, Doyensec ran Aikido’s Attack AI Pentest and XBOW’s Lightspeed against two randomly selected self-hosted apps, Fider 0.33.0 and Photoview 2.4.0. Both platforms received source code and credentials. A Doyensec researcher validated every finding by hand, a different researcher per platform, and re-scored it for severity.
-
Why Photoview
Fider and Photoview have similar stacks, Go backends with React frontends, so either would exercise the same parts of Shannon. We chose Photoview. Both apps have shipped security fixes since the versions Doyensec tested, which gives us a partial ground truth for recall. Photoview’s fixes landed in July 2026 and Fider’s in April, and the later batch reduces the chance the models saw the fixes in training.
-
The deployment
The deployment matched the one Doyensec tested: MariaDB, standalone username and password authentication, and two seeded accounts (one admin and one normal user). XBOW received only the admin account, so we gave Shannon only the admin credentials.
-
The run
The scan config was optional and only carried login details. Shannon started its local infrastructure, mounted the target repository read-only, and wrote its results to a local workspace.
./shannon start \
-u http://host.docker.internal:4800 \
-r ~/photoview-v240/repo \
-c ~/photoview-v240/config.yamlagentic_sast:
enabled: "true"
exploit: "true"
report:
sarif: "true"
authentication:
login_type: form
login_url: "http://host.docker.internal:4800/login"
credentials:
username: "admin"
password: "PhotoviewAdmin!2026"
login_flow:
- "Go to http://host.docker.internal:4800/login"
- "Type $username into the Username field"
- "Type $password into the Password field"
- "Click the Sign in button"Seven fixes Photoview shipped after 2.4.0
We scraped Photoview’s commit history after the scanned version for security-related keywords, then reviewed the matching commits. The maintainers fixed these issues independently of this benchmark, so when a Shannon finding matches one, it is a strong signal the finding is a real problem they cared enough to patch.
| Vulnerability Photoview patched Commit / PR | CVSS 3.1 | Claude Opus 5 | Grok 4.6 | DeepSeek v4 Flash |
|---|---|---|---|---|
Pre-auth SQL injection in the album-download routedeb1b216 PR #1453 |
9.8 Critical | Caught as INJ-01 | Caught as INJ-01 | Caught as INJ-01 |
Share-link authz: the admin check tested the token owner, not the caller3512ca26 PR #1452 |
8.1 High | Caught as AUTHZ-03 | Caught as AUTHZ-05 | Not flagged |
WebSocket session never re-validated after upgrade95d3d16a PR #1353 |
8.1 High | Caught as MISC-01 | Not flagged | Not flagged |
Share-token expiry never enforced2b1240b8 PR #12022598c362 PR #1348 |
5.3 Medium | Caught as AUTH-04 | Caught as AUTHZ-07 | Caught as AUTH-06 |
Unauthenticated nil-pointer panic (DoS) on unknown /api/photo and /api/video27a0b082 PR #12012b1240b8 PR #1202 |
5.3 Medium | Caught as MISC-03 | Not flagged | Not flagged |
WebSocket origin check fails open (cross-site WebSocket hijacking)eeb8d0e9 PR #136395d3d16a PR #1353 |
4.7 Medium | Caught as AUTH-06 | Not flagged | Caught as AUTH-09 |
Malformed EXIF GPS data accepted (media-parsing input validation)df9af39a PR #951 |
4.3 Medium | Not flagged | Not flagged | Not flagged |
| Total | 6 / 7 | 3 / 7 | 3 / 7 |
All three caught the critical SQL injection
The bug turned one endpoint into a boolean oracle, letting an unauthenticated attacker read the entire database. Every scan caught it, regardless of model. At these price points, results like this can widen access to meaningful security testing.
Coverage rises with the model
Claude Opus 5, the heaviest model, caught 6 of 7, including the more nuanced, higher-severity issues. Grok 4.6 caught 3 of 7: the critical issue, one high and one medium. DeepSeek v4 Flash also caught 3 of 7: the critical issue and two mediums.
Frequent cheap runs, periodic deep ones
As the price dropped, coverage of the subtler, higher-severity findings fell off, but every model still caught the most urgent issue. That supports the cadence we had in mind for Shannon: run Grok or DeepSeek regularly, then bring in Opus periodically to catch the harder findings.
The one false positive
Opus flagged userAddRootPath (INJ-02) as missing path confinement. It is an admin-only feature for registering local media directories on a host the admin already controls, so the agent read an intended feature as an exploit.
Shannon still rated it Low rather than escalating it, but it points to a broader problem we are working on: enhancing business logic understanding.
The same code is an exploit in one app and a feature in another, so before an agent can tell the two apart it has to understand the full context of the app it is testing.
Most ratings were one band off
To assess our severity ratings, we manually scored each true finding against CVSS 3.1 and compared it to the severity we had originally reported. Admittedly, we had a fair amount of adjusting to do, but in most cases the reported severity was only one band off its CVSS equivalent, and never wildly exaggerated. More rigorous CVSS scoring is on our roadmap.
We also noticed a number of findings that could fall under an “informational” category. Most were already reported as Low, so it is less about correcting inflated severities and more about giving them a more precise label. We plan to move these findings into a more accurate category.
- Shannon 3.0 · DeepSeek v4 Flash72%
- Shannon 3.0 · Grok 4.650%
- Shannon 3.0 · Claude Opus 562%
- Aikido66%
- XBOW57%
35x to 656x cheaper, finished the same day
For Doyensec’s study, Aikido’s Standard tier and XBOW’s Plus tier each cost $4,000 per scan. Against that baseline, our $115 Opus scan of Photoview was 35x cheaper. The $35.07 Grok scan was 114x cheaper, and the $6.10 DeepSeek scan was 656x cheaper.
Time also differed. Once the Photoview target was ready, all three Shannon runs finished in under five and a half hours. Doyensec reported just under eight hours for Aikido’s Photoview scan. XBOW’s Photoview scan started on April 6, and the final report arrived on April 8 with no interruptions. These are not like-for-like scanner runtimes, but they do show the operational difference between Shannon’s same-day local runs and XBOW’s multi-day process.
- 656x cheaper$6.10Shannon 3.0 · DeepSeek v4 Flash
- 114x cheaper$35.07Shannon 3.0 · Grok 4.6
- 35x cheaper$115Shannon 3.0 · Claude Opus 5
- $4,000Aikido · Standard tier
- $4,000XBOW · Plus tier
Single-model runs leave coverage on the table
Ensembling, running multiple models and merging their outputs, is a proven way to boost coverage and reliability. The results above reflect only single-model runs, so there is still significant performance left on the table.
Grok and DeepSeek each found three of the seven patched vulnerabilities, but their findings did not perfectly overlap. Merging their reports would cover four of the seven for just $41. A multi-model approach also acts as a built-in critic: a second model verifying trust boundaries would likely have caught Opus’s single false positive (INJ-02).
Shannon does not orchestrate ensembling natively yet, but its architecture makes it easy to implement. Because Shannon is BYOM (bring your own model) and outputs standard SARIF files, merging and deduplicating results from different models is straightforward when paired with your favorite agentic coding tool. All of this can run in a CI flow.
Deploy this in your CI system today
Shannon 3.0 runs headlessly in GitHub Actions or GitLab CI. Run npx @keygraph/shannon setup once on the runner to store your provider key. After that, the pipeline calls npx @keygraph/shannon start with the target URL and repo path. Shannon pulls the worker image, mounts the checkout read-only in a throwaway container, scans, and exits. The only traffic that leaves your runner goes to the model provider you configured.
Exploit-mode scans write SARIF 2.1.0 by default, so findings land in GitHub code scanning or GitLab’s vulnerability report alongside your other scanners. Out of the box, the run fails on Critical or High findings. Lower severities show up as annotations, and the threshold is configurable.
# Once, on the runner: store your provider key
npx @keygraph/shannon setup
# In the pipeline: the target URL and the repo path
npx @keygraph/shannon start -u <target-url> -r <repo-path>Continuous coverage at portfolio scale
The complaint we hear most from CISOs about AI pentesting is the economics of continuous coverage. Doyensec paid $4,000 for each Aikido and XBOW assessment. At that benchmark price, treating 5,000 repositories as separate assessment targets would imply $20 million for one portfolio-wide pass before enterprise discounts. That may be viable for selective annual testing, but not for testing an entire portfolio at release cadence.
Shannon 3.0 running DeepSeek v4 Flash scanned Photoview for $6.10 and caught the same critical SQL injection the $4,000 platforms caught. At the observed DeepSeek model cost, 5,000 equivalent Shannon runs would consume about $30,500 in model tokens, before infrastructure and operational costs.
The cheap model does not catch everything: our Claude Opus 5 run found 6 of 7 patched vulnerabilities to DeepSeek’s 3, for $115. These results support a tiered testing strategy in which teams use lower-cost models for frequent scans and more capable models for periodic deeper assessments, making continuous pentesting economically practical.
$20M
5,000 assessments at $4,000
One portfolio-wide pass at the benchmark price, before enterprise discounts.
$30.5K
5,000 DeepSeek v4 Flash runs
Model tokens at the observed cost, before infrastructure and operational costs.
Run the benchmark yourself
Shannon is open source under AGPL-3.0. Point it at Photoview 2.4.0 with your own model key and compare your reports with ours.