Feat rework docs

[vibhatha]

This PR re-works the website based on the updated data.
And also adds minor improvements to data visualization and citation.

[gemini-code-assist]

Summary of Changes

Hello @vibhatha, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly re-works the website's data presentation and underlying data processing logic. The primary goal is to enhance how datasets are organized and discovered by introducing a robust categorization system. This not only refines the interactive data browsing experience but also adds crucial documentation for data citation and source transparency, making the platform more user-friendly and academically sound.

Highlights

• Enhanced Data Categorization: Implemented a new Python function to automatically categorize datasets (e.g., Tourism, Foreign Employment, Immigration, Budget, Foreign Affairs) based on their names, improving data organization and discoverability.
• Improved Interactive Data Browser: The interactive data browser now utilizes the new categorization system, allowing users to filter datasets by these defined categories instead of ministries, and includes updated statistics and search capabilities.
• New Documentation Pages: Added dedicated documentation pages for 'Cite This Dataset' (providing various academic citation formats) and 'Data Sources' (listing original source documents with archive links), enhancing the usability and credibility of the data.
• Website Content and UI Updates: The website's overview, data matrix, and missing datasets report have been updated to reflect the latest data and categorization. A new 'CopyableCode' React component was introduced to facilitate easy copying of code snippets, particularly for citations.
• Refined Data Ingestion Process: The data generation script now specifically targets the 'statistics' folder for data scanning and includes the new category information in the generated dataset index.

Ignored Files

• Ignored by pattern: .github/workflows/** (1)
  • .github/workflows/preview-docs.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request significantly improves the data browsing experience by introducing a categorization system, replacing the previous ministry-based filtering. The changes are well-implemented across the data generation script and the frontend components. I appreciate the addition of new documentation for data sources and citations, which enhances the project's usability for researchers. I have one suggestion to improve the maintainability of the new categorization logic in the Python script.

[gemini-code-assist]

For better maintainability and readability, you could refactor this function to be more data-driven. Instead of a series of if statements, consider using a dictionary to map categories to their keywords. This makes it easier to add or modify categories in the future without changing the logic of the function.

def categorize_dataset(dataset_name: str) -> str:
    """Categorize a dataset based on its name"""
    name_lower = dataset_name.lower()

    category_keywords = {
        'Tourism': [
            'tourist', 'tourism', 'accommodation', 'occupancy',
            'top 10 source', 'location vs revenue', 'tourist attraction'
        ],
        'Foreign Employment': [
            'slbfe', 'remittance', 'foreign exchange', 'local arrival',
            'local departure', 'legal division', 'complaints', 'raids'
        ],
        'Immigration': [
            'asylum', 'deportation', 'refugee', 'refused entry',
            'fake passport', 'fraudulent visa'
        ],
        'Budget': [
            'capital expenditure', 'recurrent expenditure', 'budget'
        ],
        'Foreign Affairs': [
            'ministry news', 'mission news', 'staff of mission',
            'staff of the ministry', 'special notice', 'news from other',
            'cadre management'
        ]
    }

    for category, keywords in category_keywords.items():
        if any(keyword in name_lower for keyword in keywords):
            return category

    return 'Other'