n-get - Downloader

Features

Unleash the Power of Smart Download Management

⚡ Parallel Downloads

Download multiple files simultaneously with configurable concurrency limiting for optimal performance

📊 Smart Concurrency

Intelligent queue management with semaphore-based concurrency control and resource pooling

🔄 Enhanced Resume

Intelligent resumption with numbered selection, batch resume, and HTTP range requests

🎯 Custom Destination

Specify where to save downloaded files with flexible path resolution

🎨 Beautiful UI

Rich UTF-8 emojis, colored output, and animated progress bars for enhanced user experience

📊 Real-time Progress

Live progress tracking with download speed and ETA calculations

🔒 SSH/SFTP Support

Secure file downloads via SSH with key authentication and resume capability

⚡ Unix Pipes Support

Full stdin/stdout support for seamless integration with shell pipes and command chaining

🔍 Recursive Downloads

Website mirroring with depth control, pattern matching, and robots.txt support

📝 Smart File Naming

Automatic file deduplication with incremental naming (file.zip, file.zip.1, file.zip.2)

⚙️ Enterprise Configuration

YAML-based hierarchical configuration with validation, profiles, and environment-specific settings

🤖 AI-Native Design

Built specifically for AI agents with structured output, contextual tracking, and framework integration

📊 Structured Output

JSON, YAML, CSV output formats for seamless AI agent integration and data processing

🛡️ Enhanced Error Handling

Structured error codes with severity levels, categories, and actionable recovery suggestions

🔍 Capabilities Discovery

Machine-readable tool capabilities for AI frameworks to understand and integrate features

📝 Rich Metadata Collection

Content-type detection, checksums, performance metrics, and contextual information for analysis

🔗 Contextual Tracking

Session, request, and conversation IDs for multi-turn AI agent interactions and operation correlation

Enterprise Configuration Management

Powerful YAML-based configuration system with validation, profiles, and hot-reloading

🏗️ Hierarchical Configuration

Multi-layer configuration with precedence: CLI args > env vars > local.yaml > environment.yaml > default.yaml

✅ Schema Validation

Joi-powered validation ensures configuration integrity with type checking and constraint enforcement

⚡ Hot Reloading

Automatic configuration reloading in development mode when YAML files change

🎯 Performance Profiles

Pre-configured profiles for different scenarios: fast, secure, bulk, and careful downloading

🌍 Environment Support

Separate configurations for development, production, staging, and testing environments

📊 Metrics & Monitoring

Built-in configuration metrics, history tracking, and performance monitoring

Example Configuration Structure

# config/default.yaml - Base configuration
version: "2.0.0"
http:
  timeout: 30000
  maxRetries: 3
  maxConnections: 20
  userAgent: "N-Get-Enterprise/2.0"

downloads:
  maxConcurrent: 3
  enableResume: true
  progressReporting: true

security:
  maxFileSize: 10737418240  # 10GB
  allowedProtocols: ["https", "http", "sftp"]
  certificateValidation: true

profiles:
  fast:
    description: "Optimized for speed and high throughput"
    http:
      maxConnections: 50
      timeout: 15000
    downloads:
      maxConcurrent: 10
  
  secure:
    description: "Maximum security settings"
    security:
      allowedProtocols: ["https"]
      blockPrivateNetworks: true
      certificateValidation: true

# Environment variables: NGET_HTTP_TIMEOUT=45000
# CLI overrides: --config-downloads-maxconcurrent 5

AI Agent Integration

Purpose-built for AI agents with MCP server support and dynamic configuration

🔌 MCP Server Support

Compatible with Model Context Protocol (MCP) for seamless AI agent integration

⚙️ Dynamic Configuration

AI agents can dynamically switch profiles and adjust settings based on task requirements

📈 Learning Capabilities

Optional learning from successful downloads to optimize future configuration choices

🎛️ Agent-Controlled

No autonomous decisions - AI agents maintain full control over configuration and behavior

AI Agent Integration Examples

// AI Agent using n-get with ConfigManager
const { ConfigManager } = require('n-get/lib/config/ConfigManager');
const recursivePipe = require('n-get/lib/recursivePipe');

// Initialize configuration for AI agent
const configManager = new ConfigManager({
  environment: 'production',
  logger: myAILogger
});

// Get available profiles for AI decision making
const profiles = configManager.getAvailableProfiles();
console.log('Available profiles:', Object.keys(profiles));

// AI agent decides to use 'fast' profile for bulk download
await configManager.applyProfile('fast');

// Get AI-optimized configuration summary
const summary = configManager.getAIConfigSummary();
console.log('Current settings:', summary.keySettings);

// Download files with AI-configured settings
const results = await recursivePipe(urls, destination, {
  configManager,
  maxConcurrent: summary.keySettings.maxConcurrentDownloads
});

// Learn from successful outcome
configManager.learnFromOutcome({
  success: true,
  duration: 30000,
  throughput: 5242880,
  errors: {}
});

// Export data for AI model training
const trainingData = configManager.exportForAITraining();

MCP Server Integration

# Enable MCP server support
export NGET_AI_MCP_ENABLED=true
export NGET_AI_MCP_PORT=8080

# AI agent can now use configuration profiles
nget --config-ai-profile fast https://example.com/files.zip

# Environment-specific configuration for AI
nget --config-environment production --config-ai-profile secure

CrewAI / AutoGen / LangChain Integration

# Python AI agent using n-get via subprocess
import subprocess
import json

class NGetTool:
    def __init__(self):
        self.config_profiles = {
            'fast': 'Optimized for speed and high throughput',
            'secure': 'Maximum security settings',
            'bulk': 'Designed for large batch downloads',
            'careful': 'Conservative settings for unreliable connections'
        }
    
    def download_files(self, urls, profile='fast', destination='./downloads'):
        """Download files using n-get with specified profile"""
        cmd = [
            'nget',
            '--config-ai-profile', profile,
            '--config-environment', 'production',
            '-d', destination
        ] + urls
        
        result = subprocess.run(cmd, capture_output=True, text=True)
        return {
            'success': result.returncode == 0,
            'output': result.stdout,
            'error': result.stderr
        }
    
    def get_available_profiles(self):
        """Get available configuration profiles"""
        return self.config_profiles

# Usage in CrewAI agent
agent = Agent(
    role='Download Manager',
    goal='Efficiently download files based on requirements',
    tools=[NGetTool()],
    backstory='Expert in optimizing download performance'
)

Architecture & Patterns

Modern Node.js design patterns with separation of concerns and modular architecture

📁 Project Structure

n-get/ ├── index.js # CLI entry point & argument parsing ├── lib/ │ ├── recursivePipe.js # Core download logic using streams │ ├── uriManager.js # URL validation & protocol handling │ ├── resumeManager.js # Resume functionality & metadata │ ├── sftpManager.js # SSH/SFTP download support │ ├── ui.js # Enhanced UI with progress bars │ ├── chdir.js # Directory operations │ ├── recursiveDownloader.js # Website mirroring │ └── config/ # Enterprise configuration system │ └── ConfigManager.js # YAML config with validation ├── config/ # Configuration files │ ├── default.yaml # Base configuration │ ├── development.yaml # Development overrides │ ├── production.yaml # Production settings │ └── local.yaml # Local overrides (git-ignored) ├── docs/ # AI integration documentation │ └── AI_AGENT_INTEGRATION.md # AI agent usage guide ├── test/ # Comprehensive test suite ├── package.json # Dependencies & CLI configuration └── README.md # Documentation

🏗️ Design Patterns

Module Pattern: Each file exports specific functionality with clear interfaces
Stream Processing: Uses Node.js streams for memory-efficient file downloads
Promise-based API: Modern async/await patterns throughout
Separation of Concerns: CLI, validation, and download logic are decoupled

⚡ Core Components

CLI Layer: Command-line interface with argument parsing
URI Manager: URL validation and protocol normalization
Download Engine: Stream-based file downloading with progress tracking
Resume Manager: Intelligent resume with metadata and validation
SFTP Manager: SSH/SFTP secure download support
UI Manager: Enhanced terminal UI with progress bars

🔧 Technical Stack

Runtime: Node.js >= 18.0.0
HTTP Client: node-fetch (modern replacement for request)
SSH/SFTP: ssh2 and ssh2-sftp-client for secure downloads
Configuration: js-yaml for YAML parsing, Joi for validation
UI: cli-progress and colors for enhanced user experience
Streams: Native Node.js streams for memory efficiency
AI Integration: MCP compatible, supports CrewAI, AutoGen, LangChain

Installation

Get started with n-get in seconds

📦 Global Installation (Recommended)

npm install n-get -g

🔧 From Source

git clone https://github.com/bingeboy/n-get cd n-get npm install npm link # For global access

Usage Examples

Simple and intuitive command-line interface

                            # Download a single file
nget https://example.com/file.zip

# Download to specific directory
nget https://example.com/file.zip -d /path/to/destination

# Download multiple files
nget https://example.com/file1.zip https://example.com/file2.pdf -d ./downloads
                        

                            # Use configuration profiles for different scenarios
nget --config-ai-profile fast https://example.com/files.zip

# Environment-specific configuration
nget --config-environment production https://example.com/data.zip

# Override specific configuration values via CLI
nget --config-downloads-maxconcurrent 10 --config-http-timeout 60000 file.zip

# Use environment variables for configuration
export NGET_HTTP_MAXCONNECTIONS=50
export NGET_DOWNLOADS_MAXCONCURRENT=8
nget https://example.com/file.zip

# Profile-based downloads for different use cases
nget --config-ai-profile secure https://sensitive-site.com/file.zip  # Use secure profile
nget --config-ai-profile bulk file1.zip file2.zip file3.zip         # Use bulk profile
nget --config-ai-profile careful https://unreliable-server.com/file.zip  # Use careful profile

# Configuration precedence (highest to lowest):
# 1. CLI arguments (--config-*)
# 2. Environment variables (NGET_*)
# 3. local.yaml (git-ignored)
# 4. environment.yaml (development/production/staging)
# 5. default.yaml (base configuration)
                        

                            # AI Agent Integration Examples

# Basic AI agent usage with profiles
nget --config-ai-profile fast https://example.com/dataset.zip

# Enable learning mode for AI optimization (optional)
export NGET_AI_PROFILES_LEARNINGENABLED=true
nget --config-ai-profile fast https://example.com/files.zip

# MCP Server integration for AI agents
export NGET_AI_MCP_ENABLED=true
export NGET_AI_MCP_PORT=8080
nget --config-environment production https://example.com/data.zip

# Different profiles for different AI tasks:

# Fast profile - for high-throughput AI data collection
nget --config-ai-profile fast \
  https://api.example.com/dataset1.json \
  https://api.example.com/dataset2.json \
  https://api.example.com/dataset3.json

# Secure profile - for sensitive AI training data
nget --config-ai-profile secure \
  https://secure-data-source.com/training-data.zip

# Bulk profile - for large AI model downloads
nget --config-ai-profile bulk \
  https://huggingface.co/model1.bin \
  https://huggingface.co/model2.bin

# Careful profile - for unreliable AI data sources
nget --config-ai-profile careful \
  https://unstable-api.com/data.json

# AI agent can query available profiles
# (Available profiles: fast, secure, bulk, careful)
                        

                            # Download multiple files with default concurrency (3 concurrent)
nget https://example.com/file1.zip https://example.com/file2.pdf https://example.com/file3.jpg

# Configure concurrency for optimal performance
nget file1.zip file2.zip file3.zip file4.zip --max-concurrent 5

# Use sequential downloads (1 concurrent) for slow servers
nget https://slow-server.com/file1.zip https://slow-server.com/file2.zip --max-concurrent 1

# High performance download with resume support
nget resume all --max-concurrent 5

# Monitor concurrent downloads
nget --list-resume --max-concurrent 3

# Performance testing with different concurrency levels
nget file1.zip file2.zip file3.zip file4.zip file5.zip --max-concurrent 10
                        

                            # Auto protocol detection
nget example.com/file.zip

# Multiple files with mixed protocols
nget google.com/file.txt https://github.com/user/repo/archive/main.zip

# Custom destination with quiet mode
nget https://example.com/file.zip -d ./downloads -q

# List resumable downloads
nget --list-resume -d ./downloads
                        

                            # Read URLs from stdin
echo "https://example.com/file.zip" | nget -i -

# Download to stdout (pipe to other commands)
nget https://example.com/file.txt -o - | grep "pattern"

# Chain with other tools using pipes
echo "https://httpbin.org/json" | nget -i - -o - | jq '.url'

# Read multiple URLs from file via stdin
cat urls.txt | nget -i -

# Quiet mode with pipes (suppress progress output)
echo "https://example.com/data.json" | nget -i - -o - -q | process-data
                        

                            # Resume interrupted downloads (default behavior)
nget https://example.com/large-file.zip
# If interrupted, run again to resume:
nget https://example.com/large-file.zip

# List resumable downloads with numbers
nget --list-resume -d ./downloads

# Resume specific download by number
nget resume 1

# Resume all downloads from list
nget resume all

# Resume from specific directory
nget resume -d ./downloads

# Disable resume functionality
nget --no-resume https://example.com/file.zip
                        

                            # Basic SFTP download (auto-detects SSH keys)
nget sftp://user@server.com/path/to/file.zip

# Download multiple files including SFTP
nget https://example.com/file1.pdf sftp://server.com/file2.zip -d ./downloads

# Custom SSH key authentication
nget sftp://user@server.com/file.zip --ssh-key ~/.ssh/custom_key

# Encrypted private key with passphrase
nget sftp://user@server.com/file.zip --ssh-key ~/.ssh/encrypted_key --ssh-passphrase mypassword

# Password authentication (not recommended)
nget sftp://user@server.com/file.zip --ssh-password mypassword

# Resume interrupted SFTP downloads
nget sftp://user@server.com/large-file.zip
# If interrupted, run again to resume:
nget sftp://user@server.com/large-file.zip
                        

                            # Enable recursive downloading
nget -R https://example.com/gallery/ --level 3 -d ./gallery

# Website mirroring with pattern control
nget -R https://site.com --accept "*.pdf,*.zip" --reject "*.tmp"

# Restrict to current directory (no parent access)
nget -R https://docs.site.com --no-parent --level 2

# Custom User-Agent for crawling
nget -R https://site.com --user-agent "MyBot/1.0" --level 1

# Comprehensive recursive download
nget -R https://example.com/docs/ \
  --level 5 \
  --accept "*.pdf,*.doc,*.txt" \
  --reject "*.tmp,*.log" \
  --no-parent \
  -d ./downloaded-docs
                        

                            # All supported flags and options:

# Input/Output
-i, --input-file FILE    Read URLs from file (use '-' for stdin)
-o, --output-file FILE   Write to file (use '-' for stdout)
-d, --destination DIR    Download destination directory

# Behavior
-q, --quiet             Suppress progress output
-h, --help              Show help information
-r, --resume            Enable resume (default: true)
--no-resume             Disable resume functionality
-l, --list-resume       List resumable downloads
-c, --max-concurrent N  Maximum concurrent downloads (default: 3)

# Configuration Options (NEW!)
--config-environment ENV         Set configuration environment (development/production/staging)
--config-ai-profile PROFILE     Use AI configuration profile (fast/secure/bulk/careful)
--config-* VALUE                Override any configuration value (e.g., --config-http-timeout 60000)

# SSH/SFTP Options
--ssh-key PATH          Path to SSH private key file
--ssh-password PASS     SSH password (use with caution)
--ssh-passphrase PASS   Passphrase for encrypted SSH key

# Recursive Options
-R, --recursive         Enable recursive downloading
--level DEPTH           Maximum recursion depth (default: 5)
--no-parent            Don't ascend to parent directories
-A, --accept PATTERNS   Comma-separated accepted file patterns
-j, --reject PATTERNS   Comma-separated rejected file patterns
--user-agent STRING     Custom User-Agent for crawling

# Configuration Environment Variables (NEW!)
NGET_HTTP_TIMEOUT=30000           # Set HTTP timeout
NGET_DOWNLOADS_MAXCONCURRENT=5    # Set max concurrent downloads
NGET_AI_MCP_ENABLED=true          # Enable MCP server for AI agents
NGET_AI_PROFILES_LEARNINGENABLED=true  # Enable AI learning mode

# Examples with new configuration features:
nget https://example.com/file.zip -d ./downloads
nget --config-ai-profile fast https://example.com/files.zip
nget --config-environment production --config-downloads-maxconcurrent 10 file.zip
nget -i urls.txt -d ./downloads -q
nget https://example.com/data.json -o - -q
nget --list-resume -d ./downloads
nget resume 1
nget resume all
nget file1.zip file2.zip file3.zip --max-concurrent 5
nget --help
                        

Programmatic API

Use n-get as a module in your Node.js applications

const recursivePipe = require('n-get/lib/recursivePipe');
const ConfigManager = require('n-get/lib/config/ConfigManager');

// Basic usage without configuration
recursivePipe(['https://example.com/file.zip'], './downloads')
  .then(results => {
    console.log('Download results:', results);
  })
  .catch(error => {
    console.error('Download failed:', error);
  });

// Advanced usage with ConfigManager (Enterprise features)
const configManager = new ConfigManager({
  environment: 'production',
  enableHotReload: false
});

// Apply AI profile for optimized downloads
await configManager.applyProfile('fast');

// Get AI-optimized settings
const summary = configManager.getAIConfigSummary();
console.log('Using settings:', summary.keySettings);

// Download with enterprise configuration
const options = {
  configManager,
  maxConcurrent: summary.keySettings.maxConcurrentDownloads,
  enableResume: true,
  quietMode: false
};

recursivePipe([
  'https://example.com/file1.zip',
  'https://example.com/file2.pdf',
  'https://example.com/file3.jpg'
], './downloads', options)
  .then(results => {
    console.log(`Downloaded ${results.length} files`);
    results.forEach(result => {
      console.log(`${result.success ? '✅' : '❌'} ${result.url}`);
    });
    
    // Learn from successful outcome for AI optimization
    if (results.every(r => r.success)) {
      configManager.learnFromOutcome({
        success: true,
        duration: Date.now() - startTime,
        throughput: calculateThroughput(results),
        errors: {}
      });
    }
  });

// Configuration management examples
const profiles = configManager.getAvailableProfiles();
console.log('Available profiles:', Object.keys(profiles));

// Export configuration for AI training
const trainingData = configManager.exportForAITraining();
console.log('Training data collected:', trainingData.history.length, 'events');