COMP1314 Assessments | Skim’s Blog

type

status

date

slug

summary

Stage 1: Environment preparation and Git initialization

Step 1: Install the necessary software

Jq is very useful in processing JSON data (such as Bitcoin price) returned by API.

Git is used for version control.

Gnuplot is used for plotting graphs.

Step 2: Create a project folder

Step 3: Initialize Git

Step 4: Set the ignore file (Security) to create .gitignore to prevent password leakage.

Paste the following content, and then press Ctrl+O (Save), Enter, Ctrl+X (Exit):

Stage 2: Database Settings (MySQL)

Create a database that conforms to 3NF (the third paradigm)

Download through the terminal command before create database

Here, I have to use another way to use database

Solution (if it is aarch64)

Since I am on Linux (Ubuntu), I don't need XAMPP. The package manager that comes with Linux installs Apache, MySQL and PHP, which is more stable and native than XAMPP.

Running the following command directly to replace XAMPP :

Step 1: Update the software source

Step 2: Install Apache, MySQL (MariaDB) and PHP

For Windows (WSL):

Try:

Update ur system:

Install Apache

Test: Open in Windows web browser

Install MySQL

Set up the security configuration

Install PHP

Check version

Let Apache identify php

Create your PHP project directory in WSL

Apache default website path：

Test

Then go ur web:

Step 5: Launch XAMPP

Step 6: Write an SQL script

Step 7: Git Commit #1

Stage 3: Data Collection

Step 8: Create a configuration file and isolate the password.

Step 9: Write a scraping script (tracker.sh)

Step 10 : Testing:

Step 11: Git Commit #2

Stage 4: Automated Tasks (Crontab)

Step 12: Set scheduled task

Add new line at the end of the file, run the file each 6 hours automatically.

Step 13: Update the document #3

Stage 5 : Drawing Script (GNU Plot)

Step 14: Write drawing scripts (plot_graph.sh)

Step 15: Testing

Step 16: Git Commit #4

🫂

How to view the images in localhost

Use Python to Open a Temporary Web Page

As long as your virtual machine and your computer are on the same network

Find the Virtual Machine's IP Address:

Find an address similar to 192.168.x.x or 10.0.2.15.

Start the Temporary Server:

Download on the Real Device:

Go back to your Windows/Mac browser and enter:

For example, for mine is:

Close the Server:

Press Ctrl + C in the virtual machine terminal to stop.

Some graphs you need to know

This graph records what we really need

This graph shows my folder

This file records the time and price

This file allows me to track the price and draw the image every 12 h.

Git commands

ERD Graph:

Tables In MySQL:

SHOW DATABASES;

USE crypto_tracker;

Two tables:

SELECT * FROM currencies;

SELECT * FROM price_history;

💻

Transfer the file from My mac to multipass ubuntu

README.md

Crypto Price Tracker

A comprehensive cryptocurrency price tracking system that automatically collects, stores, analyzes, and visualizes price data from various sources.

Features

Automated Data Collection: Fetches cryptocurrency price data from APIs at regular intervals

Database Storage: Stores price history with MySQL database including current, high, and low prices

Data Cleaning: Removes duplicate entries and validates data integrity

Visualization: Generates 10 different charts and graphs for price analysis

Error Handling: Robust retry mechanisms and comprehensive error reporting

Scheduled Execution: Automated execution via cron jobs

Project Structure

Prerequisites

MySQL/MariaDB server

Required tools: curl, jq, gnuplot

Bash shell environment

Installation

Set up the database:

Create configuration file: Create config.env with your database and API settings:

Install required tools:

Usage

Manual Execution

Collect price data:

Clean the data:

Generate charts:

Automated Execution

Set up cron jobs for automated execution:

Add the following lines (adjust paths as needed):

This will:

Run the tracker every hour at minute 0

Generate charts every hour at minute 5

Data Analysis

The system provides multiple analysis views:

Hourly Statistics: Average prices per hour

Daily Statistics: Daily averages, maximums, minimums, and counts

Weekly Statistics: Weekly average prices

Recent Trends: Latest 30 data points

Price Distribution: Histogram analysis

Combined Trends: Multi-timeframe comparisons

Generated Charts

The system automatically generates various chart types:

Hourly Average Price Trend

Daily Average Price

Weekly Average Price

Combined Trend Analysis

Full Price History

Price Distribution Histogram

Daily Maximum Prices

Daily Minimum Prices

Records Per Day

Recent 30 Price Points

Error Handling

The system includes comprehensive error handling:

Retry Mechanism: Automatic retries for failed API calls (up to 3 attempts)

Data Validation: Validates timestamp and price formats

Backup Creation: Automatic backups before data cleaning

Logging: Detailed logs for troubleshooting

Configuration

Database Configuration

Edit config.env:

API Configuration

Set your preferred cryptocurrency API endpoint:

Troubleshooting

Common Issues

Permission Denied: Ensure scripts have execute permissions

Database Connection Error: Verify MySQL credentials and service status

Missing Dependencies: Install required tools

Empty Data Files: Check API connectivity and log files

Data Backup

The system automatically creates backups before data cleaning:

Backup files are stored in backup/ directory

Filename format: my_data_backup_YYYYMMDD_HHMMSS.txt

Manual backup: cp data/my_data.txt backup/my_data_backup_$(date +%Y%m%d_%H%M%S).txt

License

This project is part of COMP1314 coursework.