# Market Trends Scraper

A powerful and flexible Python web scraper for collecting and analyzing pricing and product trends from e-commerce websites. This tool provides comprehensive market insights by extracting product data, tracking price changes, and generating detailed trend analyses.

## Features

- **Multi-Source Scraping**: Collect data from multiple e-commerce websites simultaneously
- **Flexible Configuration**: Easy-to-use YAML configuration for different sources and scraping parameters
- **Dual Scraping Methods**: Supports both requests/BeautifulSoup and Selenium for dynamic content
- **Data Analysis**: Built-in analysis of pricing trends, ratings, and product availability
- **Multiple Output Formats**: Save data in CSV, JSON, or Excel formats
- **Robust Error Handling**: Comprehensive error handling and retry mechanisms
- **Professional Logging**: Detailed logging with configurable levels and outputs
- **Extensible Architecture**: Modular design for easy customization and extension

## Installation

### Prerequisites

- Python 3.8 or higher
- Chrome browser (for Selenium functionality)
- ChromeDriver (compatible with your Chrome version)

### Setup

1. Clone the repository:
```bash
git clone https://github.com/iwasforcedtobehere/market-trends-scraper.git
cd market-trends-scraper
```

2. Create and activate a virtual environment:
```bash
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
```

3. Install dependencies:
```bash
pip install -r requirements.txt
```

4. Install ChromeDriver (if using Selenium):
```bash
# For Ubuntu/Debian:
sudo apt-get install chromium-chromedriver

# For macOS using Homebrew:
brew install chromedriver

# For Windows, download from: https://chromedriver.chromium.org/
```

## Configuration

The scraper uses a YAML configuration file to define scraping sources and parameters. A default configuration will be created automatically at `config/config.yaml` when you first run the scraper.

### Example Configuration

```yaml
scraper:
  delay_between_requests: 1.0
  timeout: 30
  max_retries: 3
  user_agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
  headless: true
  window_size: [1920, 1080]

sources:
  - name: "example_ecommerce"
    url: "https://example-ecommerce.com/search"
    type: "ecommerce"
    enabled: true
    use_selenium: false
    selectors:
      product: "div.product-item"
      name: "h2.product-title"
      price: "span.price"
      rating: "div.rating"
      availability: "div.stock-status"
    pagination:
      next_page: "a.next-page"
      max_pages: 10

output:
  format: "csv"
  include_timestamp: true
  filename: "market_trends_data"

database:
  url: "sqlite:///data/market_trends.db"
  echo: false

analysis:
  price_history_days: 30
  trend_threshold: 0.05
  generate_charts: true
```

### Configuration Options

#### Scraper Settings
- `delay_between_requests`: Delay between requests in seconds (default: 1.0)
- `timeout`: Request timeout in seconds (default: 30)
- `max_retries`: Maximum number of retry attempts for failed requests (default: 3)
- `user_agent`: User agent string for HTTP requests
- `headless`: Run browser in headless mode (default: true)
- `window_size`: Browser window size as [width, height] (default: [1920, 1080])

#### Source Configuration
- `name`: Unique identifier for the data source
- `url`: Base URL for scraping
- `type`: Type of website (e.g., "ecommerce")
- `enabled`: Whether to scrape this source (default: true)
- `use_selenium`: Use Selenium instead of requests (default: false)
- `selectors`: CSS selectors for extracting data
- `pagination`: Pagination settings

#### Output Settings
- `format`: Output format ("csv", "json", or "excel")
- `include_timestamp`: Include timestamp in output filename (default: true)
- `filename`: Base filename for output files

#### Analysis Settings
- `price_history_days`: Number of days to consider for price history (default: 30)
- `trend_threshold`: Minimum price change percentage to consider as a trend (default: 0.05)
- `generate_charts`: Generate trend charts (default: true)

## Usage

### Command Line Interface

Run the scraper with default settings:
```bash
python main.py
```

Specify a custom configuration file:
```bash
python main.py --config path/to/config.yaml
```

Specify output file:
```bash
python main.py --output path/to/output.csv
```

Run in verbose mode:
```bash
python main.py --verbose
```

Run browser in non-headless mode (for debugging):
```bash
python main.py --no-headless
```

### Python API

```python
from src.config_manager import ConfigManager
from src.scraper import MarketTrendsScraper
from src.logger import setup_logger

# Setup logging
setup_logger(verbose=True)

# Load configuration
config_manager = ConfigManager("config/config.yaml")
config = config_manager.load_config()

# Initialize scraper
with MarketTrendsScraper(config, headless=True) as scraper:
    # Scrape data
    data = scraper.scrape_market_trends()
    
    # Save data
    scraper.save_data(data, "output.csv")
    
    # Analyze trends
    analysis = scraper.analyze_trends(data)
    
    # Save analysis
    scraper.save_analysis(analysis, "analysis.json")
```

## Output

### Data Output

The scraper produces structured data with the following fields:

| Field | Description |
|-------|-------------|
| name | Product name |
| price | Product price (as float) |
| rating | Product rating (as float) |
| availability | Product availability status |
| url | Product URL |
| source | Data source name |
| scraped_at | Timestamp when data was scraped |

### Analysis Output

The trend analysis includes:

- **Summary Statistics**: Total products, source distribution
- **Price Analysis**: Average, min, max, median prices and distribution
- **Rating Analysis**: Average, min, max ratings and distribution
- **Availability Analysis**: Count of products by availability status
- **Price Trends by Source**: Comparative analysis across sources

Example analysis output:
```json
{
  "total_products": 150,
  "sources": {
    "example_ecommerce": 100,
    "another_store": 50
  },
  "price_analysis": {
    "average_price": 49.99,
    "min_price": 9.99,
    "max_price": 199.99,
    "median_price": 45.00,
    "price_distribution": {
      "count": 150,
      "mean": 49.99,
      "std": 35.25,
      "min": 9.99,
      "25%": 25.00,
      "50%": 45.00,
      "75%": 75.00,
      "max": 199.99
    }
  },
  "rating_analysis": {
    "average_rating": 4.2,
    "min_rating": 1.0,
    "max_rating": 5.0,
    "rating_distribution": {
      "5.0": 45,
      "4.0": 60,
      "3.0": 30,
      "2.0": 10,
      "1.0": 5
    }
  }
}
```

## Testing

Run all tests:
```bash
pytest
```

Run unit tests only:
```bash
pytest -m unit
```

Run integration tests only:
```bash
pytest -m integration
```

Run tests with coverage report:
```bash
pytest --cov=src --cov-report=html
```

## Project Structure

```
market-trends-scraper/
├── src/
│   ├── __init__.py
│   ├── config_manager.py    # Configuration management
│   ├── logger.py            # Logging utilities
│   └── scraper.py           # Main scraper implementation
├── tests/
│   ├── __init__.py
│   ├── test_config_manager.py
│   ├── test_logger.py
│   ├── test_scraper.py
│   └── test_integration.py
├── config/
│   └── config.yaml          # Configuration file
├── data/                    # Output data directory
├── main.py                  # Main entry point
├── requirements.txt         # Python dependencies
├── pytest.ini              # Test configuration
└── README.md               # This file
```

## Contributing

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

### Development Guidelines

- Follow PEP 8 style guidelines
- Write comprehensive tests for new features
- Update documentation as needed
- Ensure all tests pass before submitting

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Disclaimer

This tool is for educational and research purposes. Users are responsible for:
- Complying with websites' terms of service
- Respecting robots.txt files
- Using the tool ethically and responsibly
- Not overwhelming servers with too many requests

The authors are not responsible for any misuse of this tool.

## Support

If you encounter any issues or have questions:

1. Check the [Issues](https://github.com/iwasforcedtobehere/market-trends-scraper/issues) page
2. Create a new issue with detailed information
3. For general questions, use the [Discussions](https://github.com/iwasforcedtobehere/market-trends-scraper/discussions) tab

## Acknowledgments

- [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/) for HTML parsing
- [Selenium](https://www.selenium.dev/) for browser automation
- [Pandas](https://pandas.pydata.org/) for data analysis
- [Loguru](https://github.com/Delgan/loguru) for logging