AWS CodeArtifact Python Package Pipeline

This repository contains a Jenkins pipeline for building and publishing Python packages to AWS CodeArtifact.

Overview

This CI/CD pipeline automates the process of building, testing, securing, and publishing Python packages to a private AWS CodeArtifact repository. It ensures consistent builds, security compliance, and reliable package distribution in an enterprise environment.

Pipeline Architecture

Environment Configuration

• Leverages Jenkins Credentials for secure storage of AWS configuration
• Uses Docker containers (Python 3.11-slim) for consistent build environments
• Configures AWS CodeArtifact as primary package index with PyPI fallback

Pipeline Stages

1. Checkout

• Retrieves source code from Git repository
• Standard SCM checkout process

2. Authenticate & Configure

• Generates short-lived AWS CodeArtifact authentication tokens (12-hour expiration)
• Constructs secure repository URLs with embedded credentials
• Uses AWS IAM roles via Jenkins AWS plugin for authentication

3. Build

• Executes within isolated Docker container
• Installs build dependencies (setuptools, wheel, twine)
• Produces both source distribution (.tar.gz) and wheel (.whl) artifacts
• Configures PATH for pip user installations

4. Security Scan (Parallel Execution)

• Trivy Scanner: Performs filesystem vulnerability analysis
• Safety Check: Analyzes Python package dependencies
• Fails pipeline on HIGH/CRITICAL severity vulnerabilities
• Archives all security reports for compliance

5. Test

• Installs built package in clean environment
• Executes pytest test suite if present
• Generates JUnit XML reports for Jenkins integration
• Continues pipeline even if tests fail (for visibility)

6. Publish

• Uploads package artifacts to AWS CodeArtifact repository
• Uses temporary .pypirc configuration
• Publishes all artifacts from dist/ directory

Post-Build Actions

• Archives build artifacts for traceability
• Cleans workspace to prevent credential leakage
• Sends notifications on success/failure (when configured)

Technical Implementation

Security Features

• Temporary authentication tokens minimize credential exposure
• Non-root container execution
• Parallel security scanning for comprehensive coverage
• Workspace cleanup prevents sensitive data persistence

Docker Strategy

• Each stage runs in isolated container using docker.image().inside()
• Shared workspace volume for artifact passing between stages
• Non-root execution with HOME=/tmp configuration

Key Design Decisions

1. AWS CodeArtifact Integration

   • Private package hosting for security
   • AWS IAM integration for authentication
   • Automatic PyPI package caching
   • Enterprise compliance support

2. Containerized Builds

   • Eliminates Python version conflicts
   • Ensures reproducible builds
   • Provides clean build environment

3. Parallel Security Scanning

   • Reduces overall pipeline execution time
   • Multiple vulnerability detection methods
   • Comprehensive security coverage

Configuration

Required Jenkins Credentials

• AWS_REGION: AWS region for CodeArtifact
• AWS_ACCOUNT_ID: AWS account identifier
• CODEART_DOMAIN: CodeArtifact domain name
• CODEART_REPO: CodeArtifact repository name
• jenkins-codeartifact: AWS IAM credentials

Pipeline Options

disableConcurrentBuilds()      // Prevents parallel execution
buildDiscarder(logRotator(numToKeepStr: '50'))  // Retains 50 builds
timeout(time: 30, unit: 'MINUTES')  // 30-minute execution limit

Usage

Installing Published Packages

# Configure AWS CodeArtifact
aws codeartifact login --tool pip \
  --domain YOUR_DOMAIN \
  --repository YOUR_REPO \
  --domain-owner YOUR_ACCOUNT_ID

# Install package
pip install hello-codeartifact

Using the Package

from hello_pkg import greet

message = greet()
print(message)  # Output: Hello, CodeArtifact!

Monitoring and Reporting

• Build Artifacts: Stored in Jenkins for each build
• Test Results: JUnit XML reports integrated with Jenkins
• Security Reports: JSON reports from Trivy and Safety
• Build History: 50 builds retained for audit purposes

Future Enhancements

• Implement semantic versioning automation
• Add code coverage reporting
• Integrate SBOM (Software Bill of Materials) generation
• Implement multi-environment deployment strategies
• Add performance benchmarking

Troubleshooting

Common Issues

• Authentication Failures: Verify AWS credentials and IAM permissions
• Build Failures: Check Docker availability and Python syntax
• Security Scan Failures: Review vulnerability reports in archived artifacts

Debug Commands

# Verify CodeArtifact connection
aws codeartifact list-packages --domain DOMAIN --repository REPO

# Test package installation
pip install hello-codeartifact --index-url CODEARTIFACT_URL

Contributing

When modifying the pipeline:

1. Test changes in a development branch
2. Ensure security scans pass
3. Verify package publishes correctly
4. Update documentation as needed