LiteLLM v1.107.0 Supply Chain Attack: PyPI Poisoning and AWS Key Leaks

What makes software supply chain attacks so dangerous is that malicious code enters your infrastructure signed and distributed through official channels. When the poisoned v1.107.0 release of LiteLLM reached PyPI and Docker Hub, thousands of servers pulled the package during routine CI/CD runs without triggering a single perimeter alarm.

By installing a legitimate library used to connect enterprise apps to OpenAI, Anthropic, and Amazon Bedrock, production systems handed over cloud IAM credentials and private API keys within seconds of startup.

In my experience auditing production breaches, cloud credential theft rapidly produces tens of thousands of dollars in fraudulent GPU compute bills, intellectual property exfiltration, and operational downtime.

At Webxtek Studio, where we engineer cloud platforms for cybersecurity & pen-testing firms and B2B enterprise SaaS platforms, this incident reinforces the core doctrine established in our CISA, NSA & FBI Emergency Playbook: PHP, Node.js & Port Hardening: blindly trusting upstream dependencies without runtime sandboxing destroys perimeter security in minutes.

Below I break down how the exploit weaponized Python startup hooks (sitecustomize.py and .pth) and provide the exact commands to purge the infection from your servers.

1. From the Factory Floor to the Python Interpreter: Anatomizing the LiteLLM v1.107.0 Poisoning

To mitigate the LiteLLM v1.107.0 supply chain attack, you must immediately destroy virtual environments and Docker images containing this build, audit site-packages directories for malicious .pth and sitecustomize.py files, revoke static AWS IAM keys, inspect sts:GetCallerIdentity calls in CloudTrail, and migrate GitHub Actions workflows to ephemeral authentication via OIDC.

To understand why automated code scanning missed this compromise, we must bridge the factory analogy to how the Python ecosystem and Docker distribute software. When your engineering team runs pip install litellm or pulls an image from Docker Hub, your server fetches a pre-built wheel package (.whl) generated automatically by the project’s continuous integration servers on GitHub. The intrusion in version v1.107.0 did not stem from an algorithmic flaw in AI logic; it was a textbook operational hygiene failure in secret management:

Execution Chain of the LiteLLM v1.107.0 Supply Chain Attack:
1. Initial Secret Leak: A high-privilege GitHub Personal Access Token (PAT) leaks into unmasked gateway debug logs.
2. CI/CD Pipeline Hijack: The threat actor captures the token and injects a malicious release step into GitHub Actions.
3. Artifact Poisoning: The official pipeline compiles and uploads litellm v1.107.0 to PyPI and the :latest tag on Docker Hub.
4. Stealth Persistence: The wheel drops distutils-sys.pth and sitecustomize.py directly into the site-packages directory.
5. Startup Execution: Any invocation of python3 evaluates the native site.py module, executing code inside the .pth file.
6. Secret Harvesting: The payload harvests ~/.aws/credentials, environment variables (.env), and API keys (OpenAI, Anthropic).
7. Validation & Exfiltration: Stolen secrets are dispatched to a C2 server and validated within 180 seconds via the AWS STS API.

Stealth Engineering: How .pth Hooks and sitecustomize.py Work

In my daily engineering work, when I explain this attack vector to junior developers or technical managers, the instinctive response is surprise: “But we never even imported LiteLLM in our application code, we only ran unit tests in the container!”. The technical elegance of this attack lies in how CPython initializes at the operating system layer.

The attackers never touched a single line of Python application code inside LiteLLM’s existing modules to avoid raising alerts in static application security testing (SAST) tools. Instead, they dropped two stealth artifacts into the root of the site-packages directory inside the .whl archive:

1. Arbitrary Execution via .pth Files (distutils-sys.pth): By historical design in Python, whenever python or python3 initializes on Linux, the C runtime boots the interpreter and automatically imports the native site.py module (unless the -S flag is explicitly passed). The site.py module scans all library directories for path configuration files (.pth). Python’s specification dictates an obscure rule: if a line within a .pth file begins with import (or import\t), Python does not treat it as a directory path; it executes the rest of that line as executable Python code via exec(). By placing import os, sys, base64; exec(base64.b64decode('...')) onto a single line of a .pth file, malware executes silently whenever any Python process launches on the machine.
2. Global Interception via sitecustomize.py: Immediately after evaluating .pth files, site.py attempts to execute import sitecustomize. By planting a sitecustomize.py script into site-packages, the threat actors ensured that even if a developer ran an innocuous command like python3 -m pip list, pytest, or django-admin, the infostealer executed in memory before the first line of the legitimate tool ran.

Targeted Harvesting and Automated AWS STS Probing

Once active in host or CI/CD runner memory, the Python payload operated with surgical discretion to avoid CPU spikes or network traffic alerts:

• Direct filesystem harvesting of ~/.aws/credentials, ~/.aws/config, ~/.git-credentials, ~/.ssh/id_rsa, ~/.ssh/id_ed25519, and .env files in project directories.
• Full inspection of os.environ in RAM, filtering for critical variables like AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, OPENAI_API_KEY, ANTHROPIC_API_KEY, DATABASE_URL, and REDIS_URL.
• Silent HTTP queries to the cloud instance metadata service (http://169.254.169.254/latest/meta-data/iam/security-credentials/) to harvest temporary IAM role credentials on unprotected EC2 instances or Kubernetes (EKS) worker nodes.

As verified in forensic disclosures by The Hacker News and researchers at Socket.dev, less than three minutes after exfiltrating data to command-and-control (C2) infrastructure, automated botnets fired sts:GetCallerIdentity calls against the AWS API to verify role ARNs and initiate unauthorized compute and Amazon Bedrock consumption.

2. The “Black October” Landscape: GhostAction, ChainDrop, and the eMudhra Collapse

Viewing the LiteLLM compromise in isolation leaves your organization exposed to the next attack. On the ground, what we are confronting across October 2026 (dubbed “Black October” across security research circles) is a coordinated offensive targeting the software engineering supply chain:

1. The GhostAction Campaign on GitHub Actions: Threat actors harvesting developer tokens no longer delete repositories or demand ransom notes. Instead, they inject disguised workflow files into .github/workflows/ under credible names like “Security Audit CI”. When triggered on official runners, the script dumps process memory (/proc/$PID/mem) to exfiltrate organization secrets (GITHUB_TOKEN, PyPI, npm, and AWS keys) injected into the virtual machine.
2. The ChainDrop (Shai-Hulud 2.0) Worm in npm: Concurrently with PyPI, popular AI and Web3 npm packages (including libraries associated with Tensorlake and SubQuery) were poisoned via preinstall scripts. ChainDrop introduced a dangerous technical leap: rather than relying on static C2 domains that law enforcement can seize in hours, the worm queries smart contract states on the Ethereum blockchain via public RPC endpoints to read live C2 exfiltration IP addresses dynamically.
3. eMudhra (emSign) Cryptographic Collapse and certifi Risk: Compounding the crisis, the eMudhra root Certificate Authority faced emergency distrust following the issuance of certificates generated with low-entropy private keys. Why does this impact your Python runtime? Because pip and the requests library do not rely on your host Linux CA store; they rely on a bundled cacert.pem file shipped by the Python certifi package. If your containers or virtual environments run outdated certifi builds that still trust eMudhra roots, an attacker on the local network can spoof TLS connections to pypi.org (Man-in-the-Middle) and deliver backdoored wheels during routine pip install commands without triggering terminal warnings.

3. Incident Response Protocol: Purging Python, Docker Caches, and AWS Credentials

If your infrastructure runs Python services, AI gateways, or CI/CD pipelines, execute this five-phase remediation sequence deployed across all Webxtek Studio environments.

Phase 1: Hardening the Pip TLS Layer and Enforcing Cryptographic Hashes

Before installing replacement packages, update the TLS trust store in your Python environments to expunge untrusted eMudhra roots and mandate cryptographic hash pinning:

# 1. Update operating system root certificate authorities (Debian / Ubuntu)
sudo apt-get update && sudo apt-get install --only-upgrade ca-certificates openssl -y

# 2. Upgrade pip and the certifi bundle across all active virtual environments
for venv_python in /opt/*/venv/bin/python /var/www/*/venv/bin/python; do
    if [ -x "$venv_python" ]; then
        echo "[HARDENING TLS] $venv_python"
        "$venv_python" -m pip install --upgrade pip certifi
    fi
done

# 3. If using the uv package manager in production
uv self update

# 4. Enforce strict SHA-256 hash verification and prohibit uncompiled source distributions
pip install --require-hashes --only-binary :all: -r requirements.txt

Phase 2: Forensic Scan for .pth Hooks and sitecustomize.py

Because a malicious .pth file executes the moment python3 starts, never use Python or pip commands to search for malware before inspecting the disk with native Linux C/POSIX tools (find and grep). Running pip show litellm on an infected host will trigger another round of credential harvesting!

# 1. Audit the disk using native POSIX utilities (WITHOUT launching the Python interpreter!)
# Search for LiteLLM 1.107.0 metadata directories
find / -path "*/site-packages/litellm-1.107.0.dist-info" 2>/dev/null

# 2. Hunt for all .pth files across the filesystem containing executable instructions
find / -name "*.pth" 2>/dev/null -exec grep -HnE "^(import[[:space:]]|.*exec\(|.*base64)" {} +

# 3. Locate any unauthorized sitecustomize.py or usercustomize.py scripts
find / \( -name "sitecustomize.py" -o -name "usercustomize.py" \) 2>/dev/null -exec ls -la {} +

# 4. If you must run Python for diagnostics on a suspicious host, ALWAYS pass -I and -S
# (-S disables automatic site.py import, blocking .pth and sitecustomize.py execution)
python3 -I -S -c "import sys; print('Interpreter started in isolated mode without site.py hooks')"

If you discover traces of 1.107.0 or modified .pth files, do not rely on pip uninstall: destroy the virtual environment directory completely (rm -rf), rebuild it from scratch, and pin LiteLLM to an audited release with verified hashes.

Phase 3: Purging Docker Layers and Enforcing SHA-256 Digest Immutability

In containerized environments, stopping a container does not clear cached image layers (overlay2) stored on the host filesystem. Execute a complete forensic purge:

# 1. Identify all containers (active or stopped) built from LiteLLM images
docker ps -a --filter "ancestor=ghcr.io/berriai/litellm" --format "table {{.ID}}\t{{.Image}}\t{{.Status}}"

# 2. Inspect whether cached images contain suspicious .pth files without running the container
for img in $(docker images --format "{{.Repository}}:{{.Tag}}" | grep litellm); do
    echo "=== Auditing image: $img ==="
    docker run --rm --network none --entrypoint /bin/sh "$img" -c \
      "find / -name '*.pth' 2>/dev/null -exec grep -H 'import ' {} +"
done

# 3. Destroy containers, local images, and Docker build caches immediately
docker rm -f $(docker ps -aq --filter "ancestor=ghcr.io/berriai/litellm") 2>/dev/null || true
docker rmi -f $(docker images -q "*litellm*") 2>/dev/null || true
docker builder prune --all --force
docker system prune --all --volumes --force

Inside your docker-compose.yml files or Kubernetes manifests, strictly prohibit mutable tags like :latest or :main. Pin all production images to their cryptographic digest (@sha256:...):

# File: docker-compose.production.yml
services:
  ai-gateway:
    # IMMUTABLE: Docker aborts startup if a single byte changes in the remote registry
    image: ghcr.io/berriai/litellm@sha256:d8b24ef096e22e847f551c9d81d22e032b4bf3c988df9e8ff5f9e830e0f3bbec
    read_only: true
    tmpfs:
      - /tmp:size=64M,noexec,nosuid,nodev
    security_opt:
      - no-new-privileges:true
    cap_drop:
      - ALL

Phase 4: AWS Containment, STS Session Invalidation, and CloudTrail Audit

If any machine or pipeline running LiteLLM v1.107.0 held AWS credentials in environment variables or configuration files, assume those keys are in attacker hands. Deactivating the AKIA... key is vital, but many administrators overlook a critical operational detail: if the attacker already exchanged the static key for temporary session credentials via sts:GetSessionToken or sts:AssumeRole, that temporary token (ASIA...) remains active even after the parent key is deleted!

Execute this protocol via the AWS CLI to terminate both static keys and active STS sessions:

# Define the compromised IAM principal
COMPROMISED_USER="svc-litellm-prod"

# 1. Immediately deactivate all static IAM Access Keys for the user
for key_id in $(aws iam list-access-keys --user-name "$COMPROMISED_USER" --query 'AccessKeyMetadata[*].AccessKeyId' --output text); do
    echo "[EMERGENCY] Deactivating IAM key: $key_id"
    aws iam update-access-key --user-name "$COMPROMISED_USER" --access-key-id "$key_id" --status Inactive
done

# 2. Attach an inline 'AWSRevokeOlderSessions' policy to immediately invalidate temporary STS tokens
CURRENT_UTC=$(date -u +"%Y-%m-%dT%H:%M:%SZ")

cat <<EOF > /tmp/revoke-sessions-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": [ "*" ],
      "Resource": [ "*" ],
      "Condition": {
        "DateLessThan": {
          "aws:TokenIssueTime": "${CURRENT_UTC}"
        }
      }
    }
  ]
}
EOF

aws iam put-user-policy \
  --user-name "$COMPROMISED_USER" \
  --policy-name "EmergencyRevokeOlderSTSSessions" \
  --policy-document file:///tmp/revoke-sessions-policy.json

# 3. Audit AWS CloudTrail over the past 72 hours for reconnaissance or anomalous API activity
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue="$COMPROMISED_USER" \
  --start-time "$(date -u -d '72 hours ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --query 'Events[*].[EventTime,EventName,SourceIPAddress,EventSource]' \
  --output table

Phase 5: Eliminating Static Keys via OpenID Connect (OIDC) Federation

The only permanent guarantee that malware in Python or npm packages cannot steal static AWS credentials is not having static credentials to steal. Delete AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from your GitHub repository secrets today and implement native OpenID Connect (OIDC) identity federation.

With OIDC, GitHub Actions signs a short-lived cryptographic JWT token for each specific workflow run, trading it directly with AWS STS for a temporary session that expires in 15 minutes, bound exclusively to your repository and production branch:

# File: .github/workflows/production-deploy.yml
name: Hardened Zero-Static-Secrets Deployment
on:
  push:
    branches:
      - main

# Restrict permissions under the principle of least privilege
permissions:
  id-token: write # Required for runner to request OIDC JWT signed by GitHub
  contents: read   # Read-only access to source code

jobs:
  deploy_infrastructure:
    runs-on: ubuntu-24.04
    timeout-minutes: 15
    steps:
      - name: 1. Immutable Checkout by Commit SHA (Immune to tag hijacking)
        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
        with:
          persist-credentials: false

      - name: 2. Ephemeral Authentication to AWS via OIDC (Zero Keys on Disk)
        uses: aws-actions/configure-aws-credentials@e3ddf4a3c70b8976a20c47d46e0824a7ffca9a83 # v4.0.2
        with:
          role-to-assume: arn:aws:iam::123456789012:role/WebxtekGitHubOIDCDeployRole
          role-session-name: GitHubActions-Prod-Deploy
          role-duration-seconds: 900 # Session mathematically expires in 15 minutes
          aws-region: eu-west-1
          audience: sts.amazonaws.com

      - name: 3. Install Python Dependencies with Strict SHA-256 Hash Verification
        run: |
          python3 -m pip install --upgrade pip certifi
          python3 -m pip install --require-hashes --only-binary :all: -r requirements.lock

      - name: 4. Validate Assumed Role Identity and Execute Deploy
        run: |
          aws sts get-caller-identity

Notice the persist-credentials: false setting in the checkout step: this prevents actions/checkout from writing the temporary GitHub token into .git/config on runner disk, neutralizing one of the primary targets of the GhostAction campaign.

4. Architecture Comparison: Vulnerable Pipeline vs. Hardened Supply Chain

The table below contrasts standard practices that enabled the LiteLLM breach with the defensive engineering standard we maintain across Webxtek Studio systems:

To automate container image verification and SBOM generation across your CI/CD pipelines, explore our guide on Container Supply Chain: Trivy, Syft SBOM & Cosign. And to continuously audit data flows and code dependencies before shipping to production, implement the architecture detailed in Security Audit Agents: Dual Hunter-Verifier Architecture.

5. Survival Checklist for Engineering and DevOps Leaders

Before closing your internal audit into this incident, ensure your engineering team verifies these five checkpoints:

1. Non-Python Filesystem Audits: Confirm with find and grep that no .pth files with executable import statements or unauthorized sitecustomize.py scripts exist across host filesystems or base images.
2. IMDS Restrictions on Container Nodes: On AWS EC2 or EKS clusters running AI gateways like LiteLLM, mandate IMDSv2 and set the hop limit (HttpPutResponseHopLimit) to 1, preventing containerized workloads from querying 169.254.169.254.
3. Comprehensive Secret Rotation: If version 1.107.0 touched your infrastructure, regenerate all OpenAI, Anthropic, database, and IAM credentials today, attaching an inline deny policy to invalidate older STS sessions.
4. Dependency Quarantine Windows: Never deploy new minor package releases into production pipelines the moment they publish on PyPI or npm; maintain an enforced 72-hour quarantine buffer except for verified critical security fixes.
5. Egress Firewall Filtering: Enforce strict outbound firewall rules so that your AI gateway host can only communicate with authorized model API endpoints (api.openai.com, api.anthropic.com, bedrock-runtime.*.amazonaws.com), blocking data exfiltration to unrecognized C2 servers.

Securing AI-powered software requires the same architectural rigor across the software supply chain as you apply to host operating system hardening. At Webxtek Studio, we help engineering teams eliminate blind spots across continuous integration and runtime delivery. Through our high-security managed hosting services, specialized technical consulting, and cloud infrastructure DevOps architecture, we ensure your infrastructure remains resilient, verifiable, and immune to supply chain poisoning.