About The Project

HSCR is a custom set of python scripts that utilize the Hammerspace Toolkit (hstk). Hammerspace hscli expressions are used to query the underlying Hammerspace filesystems for user specific and system specific metrics. Initially the tool provides reporting on fileshare replication stats, and user capacity and file aging stats. The scripts are designed to run in the backround. The replication monitoring script runs every 5 minutes, and the two file capacity and file aging scripts run in the backround, but wake up only once per day at 3AM and 4AM. Each of these two capacity reporting scripts walk the filesystems and can be Hammerspace Anvil CPU intensive. In light of that it is recommended to only run these once per day during low periods of system utilization. The output files generated by these scripts are feed to Grafana via Prometheous and Windows Exporter.

(back to top)

Built With

• MS Windows
• Python
• Hammerspace Toolkit
• Windows Exporter
• Prometheous
• Grafana

(back to top)

Getting Started

Here are some basics on getting up and running.

Prerequisites

1. MS Windows
2. Python 3.11 +
3. Hammerspace Toolkit (hstk) installed: $ pip install hstk
4. Prometheous
5. Grafana
6. Hammerspace File Systems Mounted As SMB Shares

(back to top)

Installation

1. Download and install the following packages in their default locations (Windows Exporter, Prometheus, Grafana)
2. Create the following directories:

• C:\Reporting
• C:\Reporting\Scripts
• C:\Reporting\tmp

3. Download the three reporting python scripts and install them in the C:\Reporting\Scripts directory
4. In the C:\Reporting\tmp directory create an empty text file for each fileshare that you intend to monitor. (i.e Z.txt, Y.txt, W.txt). These files are used to calculate the latest metadata replication bandwidth for each monitored share. When additional shares are added to the montoring scheme a txt file will need to be created for each new share using the drive letter only.
5. Copy and modify the hsconfig.txt file to the C:\Reporting\Scripts directory. An example file is available on this git repo.

(back to top)

Starting

1. Right click C:\Reporting\Scripts\startup.ps1 and "Run with PowerShell"

(back to top)

Verify Installation

1. Look in the C:\Program Files\Windows Exporter\textfile_inputs directory and verify that the following .prom files have been written. The userstats.py script runs at the top of the hour and the userstats_aged.py script runs at the bottom of the hour. It will take at least one hour for all prom files to be present.

2. In your web browser go to 127.0.0.1:9182 (Windows Exporter GUI). Here you will see the list of custom metrics that are being generated by the python scripts.

(back to top)

3. In your web browser go to 127.0.0.1:9090 (Prometheus GUI). Here you can list the individual metrics and validate they are reporting correctly.

4. You can now go to your Grafana instance and start building out your custom dashboards for the User Capacity, File Aging, and Replication metrics.