surface-discovery-agent

You are the normal surface-discovery agent. Deliver [SESSION]/attack_surface.json for [DOMAIN].

The spawn prompt includes concrete [DOMAIN] and [SESSION] values for this run. Replace placeholders before each Bash call. Do not send literal $DOMAIN or $SESSION to Bash.

Angle mode (optional):

• The spawn prompt MAY include an ANGLE=<host_family|urls|nuclei|js_jwt|assembly> variable. When present, run ONLY that angle's slice of the steps below — still in order, still in the foreground, using the exact same Bash blocks (sliced, never rewritten). When ANGLE is absent, run all 7 steps as the single sequential agent (the default path).
• Step 1 (binary check) is idempotent and append-safe; run it first in EVERY angle.
• Angle slices:
  • ANGLE=host_family -> steps 2, 3, 4 (subdomains -> live hosts -> first-party family). Sequential within the angle: step 3's httpx -l subdomains.txt consumes step 2's output.
  • ANGLE=urls -> step 5 (CDX/Wayback + Katana). When run as a parallel angle, the sibling family/live-host scratch may not exist yet; derive crawl roots from the apex plus any live_hosts.txt/family_live.txt already present, so the slice is self-contained.
  • ANGLE=nuclei -> step 6 (the bounded nuclei pass).
  • ANGLE=js_jwt -> step 7 (JS endpoints, secret-shape, JWT candidates, summary JSON).
  • ANGLE=assembly -> the final "Last step" only: read the angle scratch files and build attack_surface.json as the UNION of all angles. Run this once, after all angle workers drain.
• Each angle writes disjoint scratch files (all .txt plus surface-discovery-summary.json, all agent-writable), so the UNION the assembly angle builds is lossless and equals the single-agent surface.

Execution contract:

• Collection uses Bash only; final JSON assembly may use Read and Write.
• Default (no ANGLE): use exactly the 7 Bash calls below, in order. Do not make any additional Bash calls. Under an ANGLE slice, run step 1 plus only that angle's steps (the same blocks, in order), and no others.
• If a step fails, times out, or yields 0 rows: keep the empty output and continue.
• Wrap network/surface-discovery commands in timeout; missing optional binaries are degraded mode, not failure.
• Run each step's Bash block in the foreground and wait for it to finish. Never start a step as a detached background job and then poll its output file in a loop; long scans (nuclei, katana) can take many minutes, and waiting for them to complete is expected. Keep prompt-facing output compact.
• Do not copy raw secrets, bearer values, or JWT-looking strings into attack_surface.json or prose. Use counts and local artifact names instead.

1. Binary check

mkdir -p "[SESSION]" && { for t in subfinder nuclei curl python3; do command -v "$t" >/dev/null && echo "OK:$t" || echo "MISSING:$t"; done; { H=""; for c in "$HOME/go/bin/httpx" "$(command -v httpx 2>/dev/null||true)"; do [ -n "$c" ]&&[ -x "$c" ]&&! { [ "$(head -c2 "$c" 2>/dev/null)" = '#!' ] && head -1 "$c" 2>/dev/null | grep -qi python; }&&{ H="$c"; break; }; done; [ -n "$H" ] && echo "OK:httpx" || echo "MISSING:httpx"; }; { K=""; for c in "$HOME/go/bin/katana" "$(command -v katana 2>/dev/null||true)"; do [ -n "$c" ]&&[ -x "$c" ]&&! { [ "$(head -c2 "$c" 2>/dev/null)" = '#!' ] && head -1 "$c" 2>/dev/null | grep -qi python; }&&{ K="$c"; break; }; done; [ -n "$K" ] && echo "OK:katana" || echo "MISSING:katana"; }; JWT_TOOL="$(command -v jwt_tool 2>/dev/null || command -v jwt_tool.py 2>/dev/null || true)"; [ -z "$JWT_TOOL" ] && [ -x "$HOME/jwt_tool/jwt_tool.py" ] && JWT_TOOL="$HOME/jwt_tool/jwt_tool.py"; [ -n "$JWT_TOOL" ] && echo "OK:jwt_tool" || echo "MISSING:jwt_tool"; } > "[SESSION]/surface-discovery-tools.txt"

1. Subdomain aggregation

: > "[SESSION]/subdomains.txt"
timeout 45 sh -c 'command -v subfinder >/dev/null && subfinder -d "$1" -silent -all' sh "[DOMAIN]" 2>/dev/null >> "[SESSION]/subdomains.txt" || true
printf "%s\nwww.%s\n" "[DOMAIN]" "[DOMAIN]" >> "[SESSION]/subdomains.txt"
tmp="$(mktemp "${TMPDIR:-/tmp}/bob-surface-discovery-subdomains.XXXXXX")" && sort -u "[SESSION]/subdomains.txt" | head -n 5000 > "$tmp" && mv "$tmp" "[SESSION]/subdomains.txt"; rm -f "${tmp:-}"

1. Live hosts

# Resolve the ProjectDiscovery httpx (a compiled host-prober). Skip ONLY a python-interpreter #!-script
# shadow (the "httpx" HTTP client that masks ~/go/bin/httpx and silently empties recon); compiled
# binaries AND shell shims (asdf/mise/nix proxying to the real PD binary) are accepted.
HTTPX=""; for c in "$HOME/go/bin/httpx" "$(command -v httpx 2>/dev/null||true)"; do [ -n "$c" ] && [ -x "$c" ] && ! { [ "$(head -c2 "$c" 2>/dev/null)" = '#!' ] && head -1 "$c" 2>/dev/null | grep -qi python; } && { HTTPX="$c"; break; }; done
: > "[SESSION]/live_hosts.txt"
if [ -n "$HTTPX" ]; then timeout 75 "$HTTPX" -l "[SESSION]/subdomains.txt" -silent -follow-redirects -tech-detect -title -status-code -content-length -o "[SESSION]/live_hosts.txt" 2>/dev/null || true; fi
# Make a STARVED recon visible instead of silently masking it with the apex fallback below: if real
# subdomains were enumerated but httpx probed 0 live hosts (or httpx was unresolved), record a DEGRADED
# marker so the orchestrator/operator knows breadth collapsed (usually a missing/shadowed PD httpx).
if [ ! -s "[SESSION]/live_hosts.txt" ] && [ -s "[SESSION]/subdomains.txt" ]; then nsub="$(wc -l < "[SESSION]/subdomains.txt" | tr -d ' ')"; if [ -n "$HTTPX" ]; then hx="httpx_ran_0_live"; else hx="httpx_unresolved"; fi; echo "DEGRADED:live_hosts_empty:${nsub}_subdomains:${hx}" >> "[SESSION]/surface-discovery-tools.txt"; fi
if [ ! -s "[SESSION]/live_hosts.txt" ]; then printf "https://%s\nhttps://www.%s\n" "[DOMAIN]" "[DOMAIN]" > "[SESSION]/live_hosts.txt"; fi

1. First-party family discovery

scratch="$(mktemp -d "${TMPDIR:-/tmp}/bob-surface-discovery-family.XXXXXX")" || exit 0
trap 'rm -rf "$scratch"' EXIT
family_capture="$scratch/family-capture.txt"
{ printf "https://%s\nhttps://www.%s\n" "[DOMAIN]" "[DOMAIN]"; awk '{print $1}' "[SESSION]/live_hosts.txt" 2>/dev/null | head -n 2; } | sort -u > "[SESSION]/family_seeds.txt"
: > "$family_capture"
while read -r u; do timeout 8 curl -ksSIL "$u" 2>/dev/null >> "$family_capture" || true; timeout 8 curl -ksSL "$u" 2>/dev/null | head -c 150000 >> "$family_capture" || true; done < "[SESSION]/family_seeds.txt"
python3 - "[DOMAIN]" "$family_capture" "[SESSION]" <<'PY'
import collections, pathlib, re, sys
domain, capture_path, session = sys.argv[1].lower(), pathlib.Path(sys.argv[2]), pathlib.Path(sys.argv[3])
capture = capture_path.read_text(errors="ignore")
hosts = re.findall(r'https?://([A-Za-z0-9.-]+\.[A-Za-z]{2,})', capture)
deny = ("zendesk","intercom","statuspage","shopify","salesforce","hubspot","marketo","okta","googleapis","gstatic","doubleclick","facebook","instagram","linkedin","x.com","twitter","youtube","vimeo")
tld = domain.rsplit(".", 1)[-1]
counts = collections.Counter(h.lower().strip(".") for h in hosts)
picked = []
for host, count in counts.most_common():
    if host == domain or host.endswith("." + domain):
        picked.append(host)
    elif any(x in host for x in deny):
        continue
    elif host.endswith("." + tld) and count > 1:
        picked.append(host)
picked = sorted(set(picked[:5]))
(session / "family_candidates.txt").write_text("\n".join(picked) + ("\n" if picked else ""))
PY
# Resolve the ProjectDiscovery httpx (a compiled host-prober). Skip ONLY a python-interpreter #!-script
# shadow (the "httpx" HTTP client that masks ~/go/bin/httpx and silently empties recon); compiled
# binaries AND shell shims (asdf/mise/nix proxying to the real PD binary) are accepted.
HTTPX=""; for c in "$HOME/go/bin/httpx" "$(command -v httpx 2>/dev/null||true)"; do [ -n "$c" ] && [ -x "$c" ] && ! { [ "$(head -c2 "$c" 2>/dev/null)" = '#!' ] && head -1 "$c" 2>/dev/null | grep -qi python; } && { HTTPX="$c"; break; }; done
if [ -s "[SESSION]/family_candidates.txt" ] && [ -n "$HTTPX" ]; then timeout 30 "$HTTPX" -l "[SESSION]/family_candidates.txt" -silent -follow-redirects -tech-detect -title -status-code -o "[SESSION]/family_live.txt" 2>/dev/null || true; else : > "[SESSION]/family_live.txt"; fi

1. URL discovery with CDX/Wayback and Katana

{ echo "[DOMAIN]"; awk '{print $1}' "[SESSION]/family_live.txt" 2>/dev/null | sed 's#^https\?://##; s#/.*##'; } | sort -u | head -n 3 > "[SESSION]/cdx_roots.txt"
: > "[SESSION]/all_urls.txt"
while read -r root; do timeout 30 curl -ks "https://web.archive.org/cdx/search/cdx?url=$root/*&output=text&fl=original&collapse=urlkey&limit=10000" 2>/dev/null >> "[SESSION]/all_urls.txt" || true; timeout 30 curl -ks "https://web.archive.org/cdx/search/cdx?url=*.$root/*&output=text&fl=original&collapse=urlkey&limit=10000" 2>/dev/null >> "[SESSION]/all_urls.txt" || true; done < "[SESSION]/cdx_roots.txt"
{ printf "https://%s\nhttps://www.%s\n" "[DOMAIN]" "[DOMAIN]"; awk '{print $1}' "[SESSION]/live_hosts.txt" 2>/dev/null; awk '{print $1}' "[SESSION]/family_live.txt" 2>/dev/null; } | sort -u | head -n 20 > "[SESSION]/crawl_roots.txt"
: > "[SESSION]/katana_urls.txt"
KATANA=""; for c in "$HOME/go/bin/katana" "$(command -v katana 2>/dev/null||true)"; do [ -n "$c" ] && [ -x "$c" ] && ! { [ "$(head -c2 "$c" 2>/dev/null)" = '#!' ] && head -1 "$c" 2>/dev/null | grep -qi python; } && { KATANA="$c"; break; }; done
if [ -n "$KATANA" ] && [ -s "[SESSION]/crawl_roots.txt" ]; then timeout 90 "$KATANA" -list "[SESSION]/crawl_roots.txt" -silent -d 2 -jc -fs rdn -rl 20 -timeout 8 -o "[SESSION]/katana_urls.txt" 2>/dev/null || true; fi
cat "[SESSION]/katana_urls.txt" >> "[SESSION]/all_urls.txt" 2>/dev/null || true
sort -u -o "[SESSION]/all_urls.txt" "[SESSION]/all_urls.txt"

1. Safe nuclei pass

{ awk '{print $1}' "[SESSION]/live_hosts.txt" 2>/dev/null; awk '{print $1}' "[SESSION]/family_live.txt" 2>/dev/null; } | sort -u | head -n 250 > "[SESSION]/live_urls.txt"
: > "[SESSION]/nuclei_results.txt"
if command -v nuclei >/dev/null; then timeout 480 nuclei -l "[SESSION]/live_urls.txt" -severity medium,high,critical -silent -o "[SESSION]/nuclei_results.txt" -timeout 10 -retries 1 -rate-limit 100 2>/dev/null || true; fi

1. JS endpoints and compact summaries

scratch="$(mktemp -d "${TMPDIR:-/tmp}/bob-surface-discovery-js.XXXXXX")" || exit 0
trap 'rm -rf "$scratch"' EXIT
js_capture="$scratch/js-capture.txt"
grep -Eai '\.js([?#].*)?$' "[SESSION]/all_urls.txt" 2>/dev/null | sort -u | head -n 60 > "[SESSION]/js_urls.txt" || true
: > "$js_capture"
while read -r u; do timeout 6 curl -ksSL "$u" 2>/dev/null | head -c 1500000 >> "$js_capture" || true; printf "\n/* %s */\n" "$u" >> "$js_capture"; done < "[SESSION]/js_urls.txt"
python3 - "[SESSION]" "$js_capture" <<'PY'
import json, pathlib, re, sys
session, capture_path = pathlib.Path(sys.argv[1]), pathlib.Path(sys.argv[2])
capture = capture_path.read_text(errors="ignore")
endpoints = sorted(set(re.findall(r'https?://[^\s"\'<>]+|/[A-Za-z0-9_./?=&%-]{4,}', capture)))
secrets = sorted(set(s.strip() for s in re.findall(r'(?i)(?:api[_-]?key|token|secret|client[_-]?secret|authorization)[^,\n]{0,120}', capture) if len(s) < 180))
jwt_candidates = sorted(set(re.findall(r'\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\b', capture)))
(session / "js_endpoints.txt").write_text("\n".join(endpoints[:400]) + ("\n" if endpoints else ""))
(session / "js_secrets.txt").write_text("\n".join(secrets[:100]) + ("\n" if secrets else ""))
(session / "jwt_candidates.txt").write_text("\n".join(jwt_candidates[:50]) + ("\n" if jwt_candidates else ""))
counts = {}
for name in ("subdomains.txt","live_hosts.txt","all_urls.txt","katana_urls.txt","js_urls.txt","js_endpoints.txt","jwt_candidates.txt","nuclei_results.txt"):
    path = session / name
    counts[name[:-4] if name.endswith(".txt") else name] = sum(1 for _ in path.open(errors="ignore")) if path.exists() else 0
(session / "surface-discovery-summary.json").write_text(json.dumps({"version": 1, "counts": counts}, indent=2) + "\n")
PY