From 887880e73bd40706474e77eec79c7afad1ece3bc Mon Sep 17 00:00:00 2001 From: Francesco Boi Date: Mon, 11 Aug 2025 18:40:24 +0200 Subject: [PATCH 1/4] Francesco Boi homework --- API-breaking-change/readme.md | 52 +++++++++++++++++ weather-service/Dockerfile | 17 ++++++ weather-service/app/__init__.py | 11 ++++ weather-service/app/externals.py | 45 +++++++++++++++ weather-service/app/routes.py | 88 +++++++++++++++++++++++++++++ weather-service/app/structures.py | 40 +++++++++++++ weather-service/docker-compose.yaml | 15 +++++ weather-service/pytest.ini | 3 + weather-service/readme.md | 32 ++++++++++- weather-service/requirements.txt | 4 ++ weather-service/tests/conftest.py | 19 +++++++ weather-service/tests/test_app.py | 25 ++++++++ 12 files changed, 350 insertions(+), 1 deletion(-) create mode 100644 weather-service/Dockerfile create mode 100644 weather-service/app/__init__.py create mode 100644 weather-service/app/externals.py create mode 100644 weather-service/app/routes.py create mode 100644 weather-service/app/structures.py create mode 100644 weather-service/docker-compose.yaml create mode 100644 weather-service/pytest.ini create mode 100644 weather-service/requirements.txt create mode 100644 weather-service/tests/conftest.py create mode 100644 weather-service/tests/test_app.py diff --git a/API-breaking-change/readme.md b/API-breaking-change/readme.md index ce5e823..062a84c 100644 --- a/API-breaking-change/readme.md +++ b/API-breaking-change/readme.md @@ -23,20 +23,72 @@ Imagine you’re designing and maintaining an internal or public-facing **Weathe Provide examples of what would constitute a **breaking change** to this API response for the frontends that are using tihs endpoints. provide at least 3 example. +*A braking change is a change that modifies or removes the structure of the existing contract, which is what the client is expecting to find in the response. Given that we are dealing with a json file changes to the keys are breaking. So removing the outmost key "Weather" is one for example. Changing each key name is another breaking change (example changing from "temperature" to "temp" or even to "Temperature", changing the key "hour" to "hr", from "condition" to "cond" and so on).* + +*Another breaking change is if the api is supposed to return always a 24 element array and at somepoint a new version returns for example only data related to the morning.* + +*Also in the given json clients are expecting hours in 24h format. Changing that to am/pm format can be a breaking change because in the old contract clients might rely on reading just an integer, whereas now they might apply extra transformation*. + +1) removal of one or more existing keys is a breaking change: access to an element of the array changed from `result["Weather"][0]` to `result[0]` + +``` +[ + { "hour": 0, "temperature": "18", "condition": "Clear" }, + { "hour": 1, "temperature": "17", "condition": "Clear" }, + ... + { "hour": 23, "temperature": "16", "condition": "Cloudy" } +] +``` + +2) modifying one or more existing key: access to field has changed from e.g., `result["Weather"][0]["hour"]` to `result["Weather"][0]["hour"]` + +``` +{ + "Weather": [ + { "hr": 0, "temp": "18", "cond": "Clear" }, + { "hr": 1, "temp": "17", "cond": "Clear" }, + ... + { "hr": 23, "temp": "16", "cond": "Cloudy" } + ] +} +``` + +3) modifying the format of expected result + +``` +{ + "Weather": [ + { "hour": "0am", "temperature": "18", "condition": "Clear" }, + { "hour": "1am", "temperature": "17", "condition": "Clear" }, + ... + { "hour": "11pm", "temperature": "16", "condition": "Cloudy" } + ] +} +``` + ### 2 Coordinating Across Multiple Frontends You have **multiple frontend clients** some update imidiately and some take their update only every 1–2 months. **How would you handle an API schema change across all of them safely?** + +*The problem is mostly for breaking changes, non breaking changes, such as adding an extra field should not be that problematic.* + +*In most cases the solution is to inform the clients on time and give them enough time to adapt to the new contract, keeping during that period of time both old and new version. In case of removing a field, one can mark that field as deprecated in the contract. Another possible solutions is to use API versioning (for example specifying the verion in the url) although this implies a larger effort in maintenance. * + + ### 3 How to Catch Breaking Changes During Development Describe how a team can **detect breaking changes early**, in your experince. please elaburate. +*The first thing is to perform comparisons between the old and new schema and perform some analysis on what might be a breaking changes (as a conservative measure, all non additive changes can be breaking). In my experience having an extensive set of unit tests is on the fields specified by the contract is a good way. If a key, field, or format is changed in the response hopefully that will make the unit test fail. I found a TDD approach a good onae, since you start writing tests before source code development, so unbiased on any implementation but looking only at the contract. For example once it's decided a given field should be part of a response, than write the write a unit test for that.* + ### 4 Policy for Releasing Changes What internal **policy/process** was established to manage schema changes safely, in your previous team? +*The process implied treating the Schema file as source code and versioning it (Infrastructure as Code). Changes to it would got through the MR review process.* ## πŸ§ͺ Acceptance Criteria - Answer these four questions thoroughly – at least one paragraph each, maximum half a page. diff --git a/weather-service/Dockerfile b/weather-service/Dockerfile new file mode 100644 index 0000000..484ac61 --- /dev/null +++ b/weather-service/Dockerfile @@ -0,0 +1,17 @@ +FROM python:3.11 +WORKDIR /app + +COPY requirements.txt requirements.txt +RUN pip install -r requirements.txt + +# Set environment variables +#comment if not a package +ENV FLASK_APP=app +ENV FLASK_RUN_HOST=0.0.0.0 + +#COPY app/weather.py ./ +COPY app /app +EXPOSE 5000 +#CMD ["flask", "run", "--host=0.0.0.0"] +#CMD python weather.py #ok +CMD ["flask", "--app=routes", "run", "--host=0.0.0.0", "--port=5000"] \ No newline at end of file diff --git a/weather-service/app/__init__.py b/weather-service/app/__init__.py new file mode 100644 index 0000000..9f19a80 --- /dev/null +++ b/weather-service/app/__init__.py @@ -0,0 +1,11 @@ +""" +Initialisation module. It initialise Flask server and Redis client +""" +from flask import Flask +from redis import Redis + +# initialise Flask +app = Flask(__name__) +# initialise Redis +redis = Redis(host="redis", port=6379) +from app import routes \ No newline at end of file diff --git a/weather-service/app/externals.py b/weather-service/app/externals.py new file mode 100644 index 0000000..e92e7c0 --- /dev/null +++ b/weather-service/app/externals.py @@ -0,0 +1,45 @@ +""" +Module to simulate external calls. +""" +from datetime import datetime, timedelta +import random +from app.structures import APIWeatherData + +def get_weather_data(city: str): + """Function to simulate external API returning weather ata + + Args: + city (str): the city for which weather data should be returned + + Returns: + _type_: weather data + """ + # after 1hr the counter must be reset + MAX_TIME_DELTA = timedelta(hours=1) + date = datetime.now() + + if len(get_weather_data.calls) == 0: + pass # date will be added at the end + elif (date - get_weather_data.calls[0])>MAX_TIME_DELTA: + # remove old calls + i = 0 + for i, call in enumerate(get_weather_data.calls): + if date-call=100: + return {} + weather_type = {0:"Cloudy", 1: "Clear", 2: "Rainy", 3: "Foggy"} + result: APIWeatherData = dict() + result["city"] = city + result["result"] = list() + base_temp = random.randrange(-20,35) + temp = base_temp + for i in range(date.hour+1): + temp = temp + random.randrange(-3,3) + result["result"].append({ "hour": i, "temperature": str(temp)+"Β°C", + "condition": weather_type[random.randrange(0,len(weather_type))] }) + get_weather_data.calls.append(date) + return result + +get_weather_data.calls = list() diff --git a/weather-service/app/routes.py b/weather-service/app/routes.py new file mode 100644 index 0000000..42e82d7 --- /dev/null +++ b/weather-service/app/routes.py @@ -0,0 +1,88 @@ +""" +Module implementing endpoints and some utilities +""" +from datetime import datetime +import json +from typing import Optional +from flask import request, jsonify +from app import app, redis +from app.structures import HourlyData, RedisEntry, Date, APIWeatherData +from app.externals import get_weather_data + +def get_last_update(last_updated: Optional[Date]=None) -> bool: + """ + Function compares the passed datetime to the current + and return the old object if year, month, day and hour are the same, the new one if not + or if the input is none. + + Args: + last_updated (dict): datetime of the last updated entry in Redis + + Returns: + bool: true if the redis entry needs to be updated, i.e., the last entry + does not contain the new hour(s) + """ + current_date = datetime.now() + current_date: Date = Date(year=current_date.year, month=current_date.month, + day=current_date.day, hour=current_date.hour) + new_update_date = last_updated + if last_updated is None: + new_update_date = current_date + elif current_date.year>last_updated.year or current_date.month>last_updated.month or \ + current_date.day>last_updated.day or current_date.hour>last_updated.hour: + new_update_date = current_date + return new_update_date + + +@app.route('/weather') +def weather(): + """ + Decorator for "/weather" endpoing + Returns: + """ + try: + city: str = request.args.get('city').lower().capitalize() + except: + return jsonify({"result": []}) + redis_entry: RedisEntry = redis.get(city) + last_weather_data = {} + if not redis_entry: + # no entry means it's the first insertion for that city + dict_date: Date = Date.model_validate(get_last_update()) + new_entry: RedisEntry = RedisEntry.model_validate({"last_update": dict_date, "data": list()}) + # ASSUMPTION: a dictionary is returned directly + # ASSUMPTION: external api returns same data for past hours + #get data from external call + weather_data: APIWeatherData = APIWeatherData.model_validate(get_weather_data(city)) + for hour_weather in weather_data.result: + hour_weather_json = hour_weather.model_dump() + new_entry.data.append({int( hour_weather_json.pop("hour")) : hour_weather_json}) + redis.set(city, new_entry.model_dump_json()) + last_weather_data = weather_data + else: + redis_data: RedisEntry = RedisEntry.model_validate(json.loads(redis.get(city))) + last_update: Date = Date.model_validate(redis_data.last_update) + new_update_date: Date = Date.model_validate(get_last_update(last_update)) + if last_update == new_update_date: + # convert data from redis represention to API + weather_data: APIWeatherData = APIWeatherData.model_validate( + { + "city": city, + "result": [{"hour": list(element.keys())[0], **list(element.values())[0].model_dump()} + for element in redis_data.data + ] + } + ) + last_weather_data = weather_data + else: + # call external service + weather_data: APIWeatherData = APIWeatherData.model_validate(get_weather_data(city)) + last_hours_inserted: int = len(redis_data.data)-1 + for new_hourly_data in weather_data.result[last_hours_inserted+1:]: + hour_weather_json = new_hourly_data.model_dump() + hourly_data = {hour_weather_json.pop("hour") : hour_weather_json} + redis_data.data.append(hourly_data) + redis_data.last_update = new_update_date + redis.set(city, redis_data.model_dump_json()) + last_weather_data = weather_data + return jsonify(last_weather_data.model_dump()) diff --git a/weather-service/app/structures.py b/weather-service/app/structures.py new file mode 100644 index 0000000..8a7adea --- /dev/null +++ b/weather-service/app/structures.py @@ -0,0 +1,40 @@ +""" +Moule providing custom structures +""" +from typing import TypedDict, List, Dict +from pydantic import BaseModel + +class Date(BaseModel): + """ + Class to represent last update of the redis update with hour granularity + """ + year: int + month: int + day: int + hour: int + +class HourlyData(BaseModel): + """ + Structure representing a single hour weather data + """ + temperature: str + condition: str + +class RedisEntry(BaseModel): + """ + Class representing a + """ + last_update: Date + data: List[Dict[int, HourlyData]] + +class APIHourlyData(HourlyData): + """Structure representing Hourly data returned by APIs""" + hour: int + + +class APIWeatherData(BaseModel): + """ + Class representing a + """ + city: str + result: List[APIHourlyData] \ No newline at end of file diff --git a/weather-service/docker-compose.yaml b/weather-service/docker-compose.yaml new file mode 100644 index 0000000..5c70c27 --- /dev/null +++ b/weather-service/docker-compose.yaml @@ -0,0 +1,15 @@ +services: + web: + build: . + ports: + - "5000:5000" + volumes: + - ./app:/app + environment: + - FLASK_ENV=development + image: development:latest + redis: + image: "redis:alpine" + container_name: redis + ports: + - "6379:6379" \ No newline at end of file diff --git a/weather-service/pytest.ini b/weather-service/pytest.ini new file mode 100644 index 0000000..b8757d1 --- /dev/null +++ b/weather-service/pytest.ini @@ -0,0 +1,3 @@ +# pytest.ini +[pytest] +pythonpath = . diff --git a/weather-service/readme.md b/weather-service/readme.md index ba70ec8..faa1f63 100644 --- a/weather-service/readme.md +++ b/weather-service/readme.md @@ -1,6 +1,36 @@ +# HOW TO RUN +## Program execution +Run `sudo docker-compose up --build` from `dev-challenge/weather-service` +In your host open the webrowser and to the url: https://localhost.com/weather?city= +## Pytest +To run pytests go to `dev-challenge/weather-service` and run `pytest` or `pytest tests/` -# Resilient Weather Service +# PROJECT STRUCTURE +``` +weather-service +β”œβ”€β”€ app +β”‚ β”œβ”€β”€ __init__.py: initialises Redis and Flask +β”‚ β”œβ”€β”€ externals.py: simulates external calls +β”‚ β”œβ”€β”€ routes.py: app endpoints +β”‚ └── structures.py: file containing custom classes for type hinting +β”œβ”€β”€ tests +β”‚ β”œβ”€β”€ conftest.py: config file for mocking Redis, etc. +β”‚ └── test_app.py: unit test file +β”œβ”€β”€ docker-compose.yaml +β”œβ”€β”€ Dockerfile +β”œβ”€β”€ pytest.ini +β”œβ”€β”€ readme.md +└── requirements.txt +``` +# ASSUMPTIONS +* Assuming all data (24 entry for each hour in the day), the wather data of a single city is roughly 1.6Kbytes. Having 2500 cities, the total memory would roughly 4Mbytes which is fine for Redis +* The function which is simulating external call is assuming (just for simplicity) the same time zone in order to focus on app development +* The hour date is available at the beginning of each hour + +# REQUIREMENTS + +## Resilient Weather Service build a backend API that exposes weather data to a frontend. The frontend requests the today's weather for the city the user is in β€” there's a catch: the only way to get weather information is via an external weather API that is rate-limited. Your goal is to design a resilient backend that: diff --git a/weather-service/requirements.txt b/weather-service/requirements.txt new file mode 100644 index 0000000..88a6573 --- /dev/null +++ b/weather-service/requirements.txt @@ -0,0 +1,4 @@ +flask +redis +pytest +pydantic diff --git a/weather-service/tests/conftest.py b/weather-service/tests/conftest.py new file mode 100644 index 0000000..2aa0eff --- /dev/null +++ b/weather-service/tests/conftest.py @@ -0,0 +1,19 @@ +from unittest.mock import patch +import pytest + +@pytest.fixture +def mock_redis(): + # Define a fake Redis cache + redis_cache = {} + + class MockRedis: + def get(self, key): + return redis_cache.get(key) + + def set(self, key, value): + redis_cache[key] = value + + mock_instance = MockRedis() + # PATCH THE REDIS USED IN app.routes (not app.__init__) + with patch("app.routes.redis", mock_instance): + yield mock_instance diff --git a/weather-service/tests/test_app.py b/weather-service/tests/test_app.py new file mode 100644 index 0000000..25cfa43 --- /dev/null +++ b/weather-service/tests/test_app.py @@ -0,0 +1,25 @@ +import pytest +from unittest.mock import patch +from app import app +from app.externals import get_weather_data + +@pytest.fixture +def client(): + with app.test_client() as client: + yield client + +def test_weather(client, mock_redis): + get_weather_data.calls = list() + for _ in range(150): + response = client.get("/weather?city=Astana") + assert response.status_code == 200 + assert len(get_weather_data.calls) == 1 + +def test_weather_case_insensitive_city(client, mock_redis): + get_weather_data.calls = list() + response1 = client.get("/weather?city=Astana") + assert response1.status_code == 200 + assert len(get_weather_data.calls) == 1 + response2 = client.get("/weather?city=astana") + assert response2.status_code == 200 + assert len(get_weather_data.calls) == 1 \ No newline at end of file From da7df03dd88a68620e975be818e76fafe80cc7f6 Mon Sep 17 00:00:00 2001 From: Francesco Boi Date: Wed, 3 Sep 2025 12:09:41 +0200 Subject: [PATCH 2/4] - Adding kafka docker - moving Dockerfile and requirements.txt of app inside its folder - Creating a kafka producer - using only GET api - add tests for POST DELETE PUT methods - move tests inside app - update readme --- weather-service/{ => app}/Dockerfile | 10 ++-- weather-service/{ => app}/requirements.txt | 1 + weather-service/app/routes.py | 7 ++- weather-service/app/structures.py | 24 +------- weather-service/{ => app}/tests/conftest.py | 0 weather-service/app/tests/test_app.py | 57 +++++++++++++++++++ weather-service/docker-compose.yaml | 50 ++++++++++++++-- weather-service/readme.md | 3 +- .../structures/redis_structures.py | 25 ++++++++ weather-service/tests/test_app.py | 25 -------- weather-service/weatherSensor/Dockerfile | 15 +++++ .../weatherSensor/requirements.txt | 4 ++ .../weatherSensor/weather_sensor.py | 29 ++++++++++ 13 files changed, 189 insertions(+), 61 deletions(-) rename weather-service/{ => app}/Dockerfile (73%) rename weather-service/{ => app}/requirements.txt (70%) rename weather-service/{ => app}/tests/conftest.py (100%) create mode 100644 weather-service/app/tests/test_app.py create mode 100644 weather-service/structures/redis_structures.py delete mode 100644 weather-service/tests/test_app.py create mode 100644 weather-service/weatherSensor/Dockerfile create mode 100644 weather-service/weatherSensor/requirements.txt create mode 100644 weather-service/weatherSensor/weather_sensor.py diff --git a/weather-service/Dockerfile b/weather-service/app/Dockerfile similarity index 73% rename from weather-service/Dockerfile rename to weather-service/app/Dockerfile index 484ac61..e203049 100644 --- a/weather-service/Dockerfile +++ b/weather-service/app/Dockerfile @@ -1,16 +1,18 @@ FROM python:3.11 + +RUN pip install --upgrade pip WORKDIR /app -COPY requirements.txt requirements.txt -RUN pip install -r requirements.txt +#COPY app/weather.py ./ +COPY app /app +COPY ./structures/redis_structures.py /app +RUN pip install -r /app/requirements.txt # Set environment variables #comment if not a package ENV FLASK_APP=app ENV FLASK_RUN_HOST=0.0.0.0 -#COPY app/weather.py ./ -COPY app /app EXPOSE 5000 #CMD ["flask", "run", "--host=0.0.0.0"] #CMD python weather.py #ok diff --git a/weather-service/requirements.txt b/weather-service/app/requirements.txt similarity index 70% rename from weather-service/requirements.txt rename to weather-service/app/requirements.txt index 88a6573..0ce8190 100644 --- a/weather-service/requirements.txt +++ b/weather-service/app/requirements.txt @@ -2,3 +2,4 @@ flask redis pytest pydantic +kafka-python \ No newline at end of file diff --git a/weather-service/app/routes.py b/weather-service/app/routes.py index 42e82d7..ffe129a 100644 --- a/weather-service/app/routes.py +++ b/weather-service/app/routes.py @@ -6,7 +6,8 @@ from typing import Optional from flask import request, jsonify from app import app, redis -from app.structures import HourlyData, RedisEntry, Date, APIWeatherData +from app.structures import APIWeatherData +from app.redis_structures import Date, HourlyData, RedisEntry from app.externals import get_weather_data def get_last_update(last_updated: Optional[Date]=None) -> bool: @@ -34,7 +35,7 @@ def get_last_update(last_updated: Optional[Date]=None) -> bool: return new_update_date -@app.route('/weather') +@app.route('/weather', methods=['GET']) # By default it accepts only GET methods def weather(): """ Decorator for "/weather" endpoing @@ -56,7 +57,7 @@ def weather(): weather_data: APIWeatherData = APIWeatherData.model_validate(get_weather_data(city)) for hour_weather in weather_data.result: hour_weather_json = hour_weather.model_dump() - new_entry.data.append({int( hour_weather_json.pop("hour")) : hour_weather_json}) + new_entry.data.append({int( hour_weather_json.pop("hour")) : HourlyData.model_validate(hour_weather_json)}) redis.set(city, new_entry.model_dump_json()) last_weather_data = weather_data else: diff --git a/weather-service/app/structures.py b/weather-service/app/structures.py index 8a7adea..c2aba43 100644 --- a/weather-service/app/structures.py +++ b/weather-service/app/structures.py @@ -3,29 +3,7 @@ """ from typing import TypedDict, List, Dict from pydantic import BaseModel - -class Date(BaseModel): - """ - Class to represent last update of the redis update with hour granularity - """ - year: int - month: int - day: int - hour: int - -class HourlyData(BaseModel): - """ - Structure representing a single hour weather data - """ - temperature: str - condition: str - -class RedisEntry(BaseModel): - """ - Class representing a - """ - last_update: Date - data: List[Dict[int, HourlyData]] +from app.redis_structures import HourlyData class APIHourlyData(HourlyData): """Structure representing Hourly data returned by APIs""" diff --git a/weather-service/tests/conftest.py b/weather-service/app/tests/conftest.py similarity index 100% rename from weather-service/tests/conftest.py rename to weather-service/app/tests/conftest.py diff --git a/weather-service/app/tests/test_app.py b/weather-service/app/tests/test_app.py new file mode 100644 index 0000000..6267349 --- /dev/null +++ b/weather-service/app/tests/test_app.py @@ -0,0 +1,57 @@ +import pytest +from unittest.mock import patch +from app import app +from app.externals import get_weather_data + +@pytest.fixture +def client(): + """Function to run the app + + Yields: + _type_: web server instance + """ + with app.test_client() as client: + yield client + +def test_weather(client, mock_redis): + get_weather_data.calls = list() + for _ in range(150): + response = client.get("/weather?city=Astana") + assert response.status_code == 200 + assert len(get_weather_data.calls) == 1 + +def test_weather_case_insensitive_city(client, mock_redis): + get_weather_data.calls = list() + response1 = client.get("/weather?city=Astana") + assert response1.status_code == 200 + assert len(get_weather_data.calls) == 1 + response2 = client.get("/weather?city=astana") + assert response2.status_code == 200 + assert len(get_weather_data.calls) == 1 + +def test_post_method(client): + """Test that the http post request is not allowed + + Args: + client (_type_): web server + """ + response = client.post("/weather?city=Astana") + assert response.status_code==405 + +def test_delete_method(client): + """Test that the http delete request is not allowed + + Args: + client (_type_): web server + """ + response = client.delete("/weather?city=Astana") + assert response.status_code==405 + +def test_put_method(client): + """Test that the http put request is not allowed + + Args: + client (_type_): web server + """ + response = client.put("/weather?city=Astana") + assert response.status_code==405 \ No newline at end of file diff --git a/weather-service/docker-compose.yaml b/weather-service/docker-compose.yaml index 5c70c27..8e1676e 100644 --- a/weather-service/docker-compose.yaml +++ b/weather-service/docker-compose.yaml @@ -1,15 +1,55 @@ services: web: - build: . + build: + context: . + dockerfile: app/Dockerfile ports: - "5000:5000" - volumes: - - ./app:/app + #volumes: + # - ./app:/app avoid this to avoid ovrewriting the effect of copying environment: - FLASK_ENV=development - image: development:latest + image: development:latest redis: image: "redis:alpine" container_name: redis ports: - - "6379:6379" \ No newline at end of file + - "6379:6379" + kafka: + image: confluentinc/cp-kafka:latest + hostname: kafka + container_name: kafka + ports: + - "9092:9092" + - "9093:9093" + environment: + KAFKA_KRAFT_MODE: "true" # Enables KRaft mode. + KAFKA_PROCESS_ROLES: controller,broker # Kafka acts as both controller and broker. + KAFKA_NODE_ID: 1 # Unique ID for the Kafka instance. + KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka:9093" # Controller quorum. + KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093 + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT + KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT + KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER + KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 + KAFKA_LOG_DIRS: /var/lib/kafka/data # Log storage location. + KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true" # Enables automatic topic creation. + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 # Single replica for simplicity. + KAFKA_LOG_RETENTION_HOURS: 168 # Log retention period (7 days). + KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 # No rebalance delay. + CLUSTER_ID: "Mk3OEYBSD34fcwNTJENDM2Qk" # Unique Kafka cluster ID. + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - ./kafka/data:/var/lib/kafka/data # Maps logs to local storage. + CagliariProducer: + build: + context: ./weatherSensor + dockerfile: Dockerfile + image: weathersensor:latest + container_name: CagliariWeatherSensor + volumes: + - ./weatherSensor:/weatherSensorCagliari + environment: + - LOCATION=Cagliari + depends_on: + - kafka diff --git a/weather-service/readme.md b/weather-service/readme.md index faa1f63..39ae73a 100644 --- a/weather-service/readme.md +++ b/weather-service/readme.md @@ -4,7 +4,8 @@ Run `sudo docker-compose up --build` from `dev-challenge/weather-service` In your host open the webrowser and to the url: https://localhost.com/weather?city= ## Pytest -To run pytests go to `dev-challenge/weather-service` and run `pytest` or `pytest tests/` +Install pytest on your host or you virtualenvironment. +To run pytests go to `dev-challenge/weather-service` and run `pytest` or `pytest tests/`. # PROJECT STRUCTURE ``` diff --git a/weather-service/structures/redis_structures.py b/weather-service/structures/redis_structures.py new file mode 100644 index 0000000..1da0a9c --- /dev/null +++ b/weather-service/structures/redis_structures.py @@ -0,0 +1,25 @@ +from typing import List, Dict +from pydantic import BaseModel + +class Date(BaseModel): + """ + Class to represent last update of the redis update with hour granularity + """ + year: int + month: int + day: int + hour: int + +class HourlyData(BaseModel): + """ + Structure representing a single hour weather data + """ + temperature: str + condition: str + +class RedisEntry(BaseModel): + """ + Class representing a + """ + last_update: Date + data: List[Dict[int, HourlyData]] \ No newline at end of file diff --git a/weather-service/tests/test_app.py b/weather-service/tests/test_app.py deleted file mode 100644 index 25cfa43..0000000 --- a/weather-service/tests/test_app.py +++ /dev/null @@ -1,25 +0,0 @@ -import pytest -from unittest.mock import patch -from app import app -from app.externals import get_weather_data - -@pytest.fixture -def client(): - with app.test_client() as client: - yield client - -def test_weather(client, mock_redis): - get_weather_data.calls = list() - for _ in range(150): - response = client.get("/weather?city=Astana") - assert response.status_code == 200 - assert len(get_weather_data.calls) == 1 - -def test_weather_case_insensitive_city(client, mock_redis): - get_weather_data.calls = list() - response1 = client.get("/weather?city=Astana") - assert response1.status_code == 200 - assert len(get_weather_data.calls) == 1 - response2 = client.get("/weather?city=astana") - assert response2.status_code == 200 - assert len(get_weather_data.calls) == 1 \ No newline at end of file diff --git a/weather-service/weatherSensor/Dockerfile b/weather-service/weatherSensor/Dockerfile new file mode 100644 index 0000000..b097147 --- /dev/null +++ b/weather-service/weatherSensor/Dockerfile @@ -0,0 +1,15 @@ +FROM python:3.11 + +RUN pip install --upgrade pip +WORKDIR /weatherSensor + +#COPY requirements.txt . +COPY . /weatherSensor +RUN pip install -r /weatherSensor/requirements.txt + +# Set environment variables +#comment if not a package +#COPY app/weatherSe.py ./ +#CMD ["flask", "run", "--host=0.0.0.0"] +#CMD python weather.py #ok +CMD ["python", "weather_sensor.py"] \ No newline at end of file diff --git a/weather-service/weatherSensor/requirements.txt b/weather-service/weatherSensor/requirements.txt new file mode 100644 index 0000000..c26186b --- /dev/null +++ b/weather-service/weatherSensor/requirements.txt @@ -0,0 +1,4 @@ +pytest +pydantic +kafka-python +confluent-kafka diff --git a/weather-service/weatherSensor/weather_sensor.py b/weather-service/weatherSensor/weather_sensor.py new file mode 100644 index 0000000..2ce4d38 --- /dev/null +++ b/weather-service/weatherSensor/weather_sensor.py @@ -0,0 +1,29 @@ +import os +import time, datetime +from confluent_kafka import Producer + + +LOCATION_KEY="LOCATION" +# Optional delivery callback +def delivery_report(err, msg): + if err is not None: + print(f"❌ Message delivery failed: {err}") + else: + print(f"βœ… Message delivered to {msg.topic()} [{msg.partition()}]") + + +if __name__=="__main__": + time.sleep(30) + LOCATION = os.getenv(LOCATION_KEY) + KAFKA_HOST = "kafka:9092" # Or the address you want + producer = Producer({"bootstrap.servers": KAFKA_HOST}) + topic = "testTopic" + while True: + MESSAGE = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')[:-3] + ENCODED_MESSAGE = MESSAGE.encode("utf-8") + try: + producer.produce(topic=topic, key=LOCATION, value=ENCODED_MESSAGE, on_delivery=delivery_report) + producer.flush() + except Exception as ex: + print("Exception happened :",ex) + time.sleep(3) From e39f08514f1629f8a69667afdeb4f1f0161e265d Mon Sep 17 00:00:00 2001 From: Francesco Boi Date: Thu, 4 Sep 2025 18:27:42 +0200 Subject: [PATCH 3/4] Adding a Kafka consumer --- .gitignore | 2 + weather-service/docker-compose.yaml | 19 ++++++-- weather-service/transformer/Dockerfile | 15 ++++++ weather-service/transformer/requirements.txt | 4 ++ weather-service/transformer/transformer.py | 47 +++++++++++++++++++ weather-service/weatherSensor/Dockerfile | 4 +- .../weatherSensor/weather_sensor.py | 10 ++-- 7 files changed, 91 insertions(+), 10 deletions(-) create mode 100644 .gitignore create mode 100644 weather-service/transformer/Dockerfile create mode 100644 weather-service/transformer/requirements.txt create mode 100644 weather-service/transformer/transformer.py diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..b80b289 --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +weather-service/kafka/data/* +*.pyc diff --git a/weather-service/docker-compose.yaml b/weather-service/docker-compose.yaml index 8e1676e..7610a21 100644 --- a/weather-service/docker-compose.yaml +++ b/weather-service/docker-compose.yaml @@ -43,13 +43,24 @@ services: - ./kafka/data:/var/lib/kafka/data # Maps logs to local storage. CagliariProducer: build: - context: ./weatherSensor - dockerfile: Dockerfile + context: . + dockerfile: weatherSensor/Dockerfile image: weathersensor:latest container_name: CagliariWeatherSensor - volumes: - - ./weatherSensor:/weatherSensorCagliari + # volumes: + # - ./weatherSensor:/weatherSensorCagliari #avoid overwriting already copied files + environment: + - LOCATION=Cagliari + depends_on: + - kafka + Transformer: + build: + context: ./ + dockerfile: transformer/Dockerfile + image: transformer:latest + container_name: Transformer environment: - LOCATION=Cagliari depends_on: - kafka + - CagliariProducer diff --git a/weather-service/transformer/Dockerfile b/weather-service/transformer/Dockerfile new file mode 100644 index 0000000..47780fe --- /dev/null +++ b/weather-service/transformer/Dockerfile @@ -0,0 +1,15 @@ +FROM python:3.11 + +RUN pip install --upgrade pip +WORKDIR /transformer + +#COPY requirements.txt . +COPY transformer /transformer +RUN pip install -r /transformer/requirements.txt + +# Set environment variables +#comment if not a package +#COPY app/weatherSe.py ./ +#CMD ["flask", "run", "--host=0.0.0.0"] +#CMD python weather.py #ok +CMD ["python", "-u", "transformer.py"] \ No newline at end of file diff --git a/weather-service/transformer/requirements.txt b/weather-service/transformer/requirements.txt new file mode 100644 index 0000000..c26186b --- /dev/null +++ b/weather-service/transformer/requirements.txt @@ -0,0 +1,4 @@ +pytest +pydantic +kafka-python +confluent-kafka diff --git a/weather-service/transformer/transformer.py b/weather-service/transformer/transformer.py new file mode 100644 index 0000000..d52cff8 --- /dev/null +++ b/weather-service/transformer/transformer.py @@ -0,0 +1,47 @@ +import os +import time +from confluent_kafka import Consumer, KafkaError +import json + +LOCATION_KEY="LOCATION" +# Optional delivery callback +def delivery_report(err, msg): + if err is not None: + print(f"❌ Message delivery failed: {err}") + else: + print(f"βœ… Message delivered to {msg.topic()} [{msg.partition()}]") + + +if __name__=="__main__": + time.sleep(30) + LOCATION = os.getenv(LOCATION_KEY) + KAFKA_HOST = "kafka:9092" # Or the address you want + consumer_group = LOCATION+"Consumer" + consumer = Consumer({"bootstrap.servers": KAFKA_HOST, "group.id": consumer_group, "auto.offset.reset": "latest"}) + topic = topic = "weather_" + LOCATION + consumer.subscribe([topic,]) + try: + while True: + msg = consumer.poll(timeout=1.0) # Poll for message (timeout in seconds) + if msg is None: + continue # No message this time + if msg.error(): + print(f"❌ Error: {msg.error()}") + continue + + # Message received + print(f"βœ… Received message: {msg.value().decode('utf-8')}") + + # If it's JSON, decode it + try: + data = json.loads(msg.value()) + print(f"πŸ” Parsed JSON: {data}") + except json.JSONDecodeError: + print("⚠️ Not a valid JSON message") + + except KeyboardInterrupt: + print("πŸ‘‹ Stopping consumer...") + + finally: + # Always close the consumer on exit + consumer.close() diff --git a/weather-service/weatherSensor/Dockerfile b/weather-service/weatherSensor/Dockerfile index b097147..1feccb4 100644 --- a/weather-service/weatherSensor/Dockerfile +++ b/weather-service/weatherSensor/Dockerfile @@ -4,7 +4,7 @@ RUN pip install --upgrade pip WORKDIR /weatherSensor #COPY requirements.txt . -COPY . /weatherSensor +COPY weatherSensor /weatherSensor RUN pip install -r /weatherSensor/requirements.txt # Set environment variables @@ -12,4 +12,4 @@ RUN pip install -r /weatherSensor/requirements.txt #COPY app/weatherSe.py ./ #CMD ["flask", "run", "--host=0.0.0.0"] #CMD python weather.py #ok -CMD ["python", "weather_sensor.py"] \ No newline at end of file +CMD ["python", "-u", "weather_sensor.py"] \ No newline at end of file diff --git a/weather-service/weatherSensor/weather_sensor.py b/weather-service/weatherSensor/weather_sensor.py index 2ce4d38..f557b7b 100644 --- a/weather-service/weatherSensor/weather_sensor.py +++ b/weather-service/weatherSensor/weather_sensor.py @@ -1,7 +1,8 @@ import os import time, datetime from confluent_kafka import Producer - +from random import randint +import json LOCATION_KEY="LOCATION" # Optional delivery callback @@ -17,12 +18,13 @@ def delivery_report(err, msg): LOCATION = os.getenv(LOCATION_KEY) KAFKA_HOST = "kafka:9092" # Or the address you want producer = Producer({"bootstrap.servers": KAFKA_HOST}) - topic = "testTopic" + topic = "weather_"+LOCATION + print(topic) while True: MESSAGE = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')[:-3] - ENCODED_MESSAGE = MESSAGE.encode("utf-8") + MESSAGE = {"temperature_celsius": randint(20, 35), "weather": "Cloudy", "Humidity": "50%", "Alert": "No", "time": MESSAGE} try: - producer.produce(topic=topic, key=LOCATION, value=ENCODED_MESSAGE, on_delivery=delivery_report) + producer.produce(topic=topic, key=LOCATION, value=json.dumps(MESSAGE).encode('utf-8'), on_delivery=delivery_report) producer.flush() except Exception as ex: print("Exception happened :",ex) From 6f0d8cd2019dc83e0a213baaa4e0d089f325cf39 Mon Sep 17 00:00:00 2001 From: Francesco Boi Date: Thu, 4 Sep 2025 18:41:21 +0200 Subject: [PATCH 4/4] Formatting folder --- .gitignore | 2 +- API-breaking-change/readme.md | 100 ------------------ README.md | 50 +++++---- {weather-service/app => app}/Dockerfile | 0 {weather-service/app => app}/__init__.py | 0 {weather-service/app => app}/externals.py | 0 {weather-service/app => app}/requirements.txt | 0 {weather-service/app => app}/routes.py | 0 {weather-service/app => app}/structures.py | 0 .../app => app}/tests/conftest.py | 0 .../app => app}/tests/test_app.py | 0 ...docker-compose.yaml => docker-compose.yaml | 0 weather-service/pytest.ini => pytest.ini | 0 .../redis_structures.py | 0 .../transformer => transformer}/Dockerfile | 0 .../requirements.txt | 0 .../transformer.py | 0 weather-service/readme.md | 92 ---------------- .../Dockerfile | 0 .../requirements.txt | 0 .../weather_sensor.py | 0 21 files changed, 32 insertions(+), 212 deletions(-) delete mode 100644 API-breaking-change/readme.md rename {weather-service/app => app}/Dockerfile (100%) rename {weather-service/app => app}/__init__.py (100%) rename {weather-service/app => app}/externals.py (100%) rename {weather-service/app => app}/requirements.txt (100%) rename {weather-service/app => app}/routes.py (100%) rename {weather-service/app => app}/structures.py (100%) rename {weather-service/app => app}/tests/conftest.py (100%) rename {weather-service/app => app}/tests/test_app.py (100%) rename weather-service/docker-compose.yaml => docker-compose.yaml (100%) rename weather-service/pytest.ini => pytest.ini (100%) rename {weather-service/structures => structures}/redis_structures.py (100%) rename {weather-service/transformer => transformer}/Dockerfile (100%) rename {weather-service/transformer => transformer}/requirements.txt (100%) rename {weather-service/transformer => transformer}/transformer.py (100%) delete mode 100644 weather-service/readme.md rename {weather-service/weatherSensor => weatherSensor}/Dockerfile (100%) rename {weather-service/weatherSensor => weatherSensor}/requirements.txt (100%) rename {weather-service/weatherSensor => weatherSensor}/weather_sensor.py (100%) diff --git a/.gitignore b/.gitignore index b80b289..f3cf373 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,2 @@ -weather-service/kafka/data/* +**/kafka/data/ *.pyc diff --git a/API-breaking-change/readme.md b/API-breaking-change/readme.md deleted file mode 100644 index 062a84c..0000000 --- a/API-breaking-change/readme.md +++ /dev/null @@ -1,100 +0,0 @@ - - -# Handling API Breaking Changes - - -Imagine you’re designing and maintaining an internal or public-facing **Weather API**. A basic version of the response looks like: - -``` -{ - "Weather": [ - { "hour": 0, "temperature": "18", "condition": "Clear" }, - { "hour": 1, "temperature": "17", "condition": "Clear" }, - ... - { "hour": 23, "temperature": "16", "condition": "Cloudy" } - ] -} -``` - - Assumming this is the first published contract is consumed by multiple frontend apps already we need to introduced a change. please answer to these questions: - - -### 1 What Is a Breaking Change? - -Provide examples of what would constitute a **breaking change** to this API response for the frontends that are using tihs endpoints. provide at least 3 example. - -*A braking change is a change that modifies or removes the structure of the existing contract, which is what the client is expecting to find in the response. Given that we are dealing with a json file changes to the keys are breaking. So removing the outmost key "Weather" is one for example. Changing each key name is another breaking change (example changing from "temperature" to "temp" or even to "Temperature", changing the key "hour" to "hr", from "condition" to "cond" and so on).* - -*Another breaking change is if the api is supposed to return always a 24 element array and at somepoint a new version returns for example only data related to the morning.* - -*Also in the given json clients are expecting hours in 24h format. Changing that to am/pm format can be a breaking change because in the old contract clients might rely on reading just an integer, whereas now they might apply extra transformation*. - -1) removal of one or more existing keys is a breaking change: access to an element of the array changed from `result["Weather"][0]` to `result[0]` - -``` -[ - { "hour": 0, "temperature": "18", "condition": "Clear" }, - { "hour": 1, "temperature": "17", "condition": "Clear" }, - ... - { "hour": 23, "temperature": "16", "condition": "Cloudy" } -] -``` - -2) modifying one or more existing key: access to field has changed from e.g., `result["Weather"][0]["hour"]` to `result["Weather"][0]["hour"]` - -``` -{ - "Weather": [ - { "hr": 0, "temp": "18", "cond": "Clear" }, - { "hr": 1, "temp": "17", "cond": "Clear" }, - ... - { "hr": 23, "temp": "16", "cond": "Cloudy" } - ] -} -``` - -3) modifying the format of expected result - -``` -{ - "Weather": [ - { "hour": "0am", "temperature": "18", "condition": "Clear" }, - { "hour": "1am", "temperature": "17", "condition": "Clear" }, - ... - { "hour": "11pm", "temperature": "16", "condition": "Cloudy" } - ] -} -``` - -### 2 Coordinating Across Multiple Frontends - -You have **multiple frontend clients** some update imidiately and some take their update only every 1–2 months. -**How would you handle an API schema change across all of them safely?** - - -*The problem is mostly for breaking changes, non breaking changes, such as adding an extra field should not be that problematic.* - -*In most cases the solution is to inform the clients on time and give them enough time to adapt to the new contract, keeping during that period of time both old and new version. In case of removing a field, one can mark that field as deprecated in the contract. Another possible solutions is to use API versioning (for example specifying the verion in the url) although this implies a larger effort in maintenance. * - - -### 3 How to Catch Breaking Changes During Development - -Describe how a team can **detect breaking changes early**, in your experince. please elaburate. - -*The first thing is to perform comparisons between the old and new schema and perform some analysis on what might be a breaking changes (as a conservative measure, all non additive changes can be breaking). In my experience having an extensive set of unit tests is on the fields specified by the contract is a good way. If a key, field, or format is changed in the response hopefully that will make the unit test fail. I found a TDD approach a good onae, since you start writing tests before source code development, so unbiased on any implementation but looking only at the contract. For example once it's decided a given field should be part of a response, than write the write a unit test for that.* - - -### 4 Policy for Releasing Changes - -What internal **policy/process** was established to manage schema changes safely, in your previous team? - -*The process implied treating the Schema file as source code and versioning it (Infrastructure as Code). Changes to it would got through the MR review process.* - -## πŸ§ͺ Acceptance Criteria -- Answer these four questions thoroughly – at least one paragraph each, maximum half a page. -- Provide practical examples from your own experience. Don’t just rely on ChatGPT’s first suggestion β€” dig deeper! - - - - - diff --git a/README.md b/README.md index 52707bc..7868675 100644 --- a/README.md +++ b/README.md @@ -1,25 +1,37 @@ +# HOW TO RUN +## Program execution +Run `sudo docker-compose up --build`. +In your host open the webrowser and to the url: https://localhost.com/weather?city= +The project comprises a kafka producer that produces weather data, a kafka consumer which uses those data, a kafka broker, a web app in Flask which responds to http get requests, nd a Redis DB for readily access weather data. -# NATIX Dev Challenge +Next step: +- use elastic search +- add SQL DB (SQLAlchemy) +- do some data transformations on the consumer and publish on redis to be used for the web-app +- add MongoDB +- Create alert Red and manage those from a Consumer +- add a second producer on a different city -Welcome to the official **NATIX Dev Challenge** repository! πŸš€ -The challenges are designed to evaluate your technical skills through facing with a practical real-world problems. +## Pytest +Install pytest on your host or you virtualenvironment. +To run pytests go to `dev-challenge/weather-service` and run `pytest` or `pytest tests/`. -## 🧠 Challenges -```bash -. -β”œβ”€β”€ API-breaking-change/ -β”œβ”€β”€ weather-service/ -└── README.md # You are here +# PROJECT STRUCTURE +``` +./ +β”œβ”€β”€ app +β”‚ β”œβ”€β”€ __init__.py: initialises Redis and Flask +β”‚ β”œβ”€β”€ externals.py: simulates external calls +β”‚ β”œβ”€β”€ routes.py: app endpoints +β”‚ └── structures.py: file containing custom classes for type hinting +β”œβ”€β”€ tests +β”‚ β”œβ”€β”€ conftest.py: config file for mocking Redis, etc. +β”‚ └── test_app.py: unit test file +β”œβ”€β”€ docker-compose.yaml +β”œβ”€β”€ Dockerfile +β”œβ”€β”€ pytest.ini +β”œβ”€β”€ readme.md +└── requirements.txt ``` - -## πŸ“¬ Questions? - -If anything is unclear or missing, feel free to reach out via the channel you received the challenge on. - - -Good luck, and have fun! -β€” *NATIX* - - diff --git a/weather-service/app/Dockerfile b/app/Dockerfile similarity index 100% rename from weather-service/app/Dockerfile rename to app/Dockerfile diff --git a/weather-service/app/__init__.py b/app/__init__.py similarity index 100% rename from weather-service/app/__init__.py rename to app/__init__.py diff --git a/weather-service/app/externals.py b/app/externals.py similarity index 100% rename from weather-service/app/externals.py rename to app/externals.py diff --git a/weather-service/app/requirements.txt b/app/requirements.txt similarity index 100% rename from weather-service/app/requirements.txt rename to app/requirements.txt diff --git a/weather-service/app/routes.py b/app/routes.py similarity index 100% rename from weather-service/app/routes.py rename to app/routes.py diff --git a/weather-service/app/structures.py b/app/structures.py similarity index 100% rename from weather-service/app/structures.py rename to app/structures.py diff --git a/weather-service/app/tests/conftest.py b/app/tests/conftest.py similarity index 100% rename from weather-service/app/tests/conftest.py rename to app/tests/conftest.py diff --git a/weather-service/app/tests/test_app.py b/app/tests/test_app.py similarity index 100% rename from weather-service/app/tests/test_app.py rename to app/tests/test_app.py diff --git a/weather-service/docker-compose.yaml b/docker-compose.yaml similarity index 100% rename from weather-service/docker-compose.yaml rename to docker-compose.yaml diff --git a/weather-service/pytest.ini b/pytest.ini similarity index 100% rename from weather-service/pytest.ini rename to pytest.ini diff --git a/weather-service/structures/redis_structures.py b/structures/redis_structures.py similarity index 100% rename from weather-service/structures/redis_structures.py rename to structures/redis_structures.py diff --git a/weather-service/transformer/Dockerfile b/transformer/Dockerfile similarity index 100% rename from weather-service/transformer/Dockerfile rename to transformer/Dockerfile diff --git a/weather-service/transformer/requirements.txt b/transformer/requirements.txt similarity index 100% rename from weather-service/transformer/requirements.txt rename to transformer/requirements.txt diff --git a/weather-service/transformer/transformer.py b/transformer/transformer.py similarity index 100% rename from weather-service/transformer/transformer.py rename to transformer/transformer.py diff --git a/weather-service/readme.md b/weather-service/readme.md deleted file mode 100644 index 39ae73a..0000000 --- a/weather-service/readme.md +++ /dev/null @@ -1,92 +0,0 @@ -# HOW TO RUN -## Program execution -Run `sudo docker-compose up --build` from `dev-challenge/weather-service` -In your host open the webrowser and to the url: https://localhost.com/weather?city= - -## Pytest -Install pytest on your host or you virtualenvironment. -To run pytests go to `dev-challenge/weather-service` and run `pytest` or `pytest tests/`. - -# PROJECT STRUCTURE -``` -weather-service -β”œβ”€β”€ app -β”‚ β”œβ”€β”€ __init__.py: initialises Redis and Flask -β”‚ β”œβ”€β”€ externals.py: simulates external calls -β”‚ β”œβ”€β”€ routes.py: app endpoints -β”‚ └── structures.py: file containing custom classes for type hinting -β”œβ”€β”€ tests -β”‚ β”œβ”€β”€ conftest.py: config file for mocking Redis, etc. -β”‚ └── test_app.py: unit test file -β”œβ”€β”€ docker-compose.yaml -β”œβ”€β”€ Dockerfile -β”œβ”€β”€ pytest.ini -β”œβ”€β”€ readme.md -└── requirements.txt -``` -# ASSUMPTIONS -* Assuming all data (24 entry for each hour in the day), the wather data of a single city is roughly 1.6Kbytes. Having 2500 cities, the total memory would roughly 4Mbytes which is fine for Redis -* The function which is simulating external call is assuming (just for simplicity) the same time zone in order to focus on app development -* The hour date is available at the beginning of each hour - -# REQUIREMENTS - -## Resilient Weather Service -build a backend API that exposes weather data to a frontend. The frontend requests the today's weather for the city the user is in β€” there's a catch: the only way to get weather information is via an external weather API that is rate-limited. - -Your goal is to design a resilient backend that: -- Moderates (minimizes) calls to the external weather API -- Handles API failures gracefully - - -## Context and Scope -1. The external weather API is limited to 100 requests per hour. -2. The external weather API returns detailed weather data for a given city on the current day everytime it's called. As shown in the example -3. You must support approx. 100,000 daily active users across approx. 2,500 different cities across the globe. Users use the service at any time throughout the day. -4. User authentication and external API authentication are out of scope of this task. Simply assume that the API you develop will be open to any call and the external weather API will reply to requests coming from our cluster according to the limit mentioned in 1. - - -Example of response for a passed city to the external weather API - The result is the weather for today - -``` -{ - "result": [ - { "hour": 0, "temperature": "18Β°C", "condition": "Clear" }, - { "hour": 1, "temperature": "17Β°C", "condition": "Clear" }, - ... - { "hour": 23, "temperature": "16Β°C", "condition": "Cloudy" } - ] -} -``` - - - -## Endpoint Base Info - -``` GET /weather?city=CityName ``` - - -Response: -``` -{ - "weather": [ - { "hour": 0, "temperature": "18", "condition": "Clear" }, - { "hour": 1, "temperature": "17", "condition": "Clear" }, - ... - { "hour": 23, "temperature": "16", "condition": "Cloudy" } - ], - … -} -``` - - - -## πŸ§ͺ Acceptance Criteria -- You may use any programming language. Even pseudocode or structured texts (e.g. workflow-style logic in written fromat) is acceptable β€” what matters is the clarity and quality of your technical design and solution. -- You may mock any libraries or databases you need. The focus is not on third-party integerations. -- Write down any assumptions β€” either as comments in the code or as side notes in a document. -- Clearly describe the input and output of each major function/step in your solution. This helps us understand your reasoning behind your technical design. -- Improve the response object: the example provided is minimal. Based on your experience, design a response that communicates effectively with the frontend/UI. - - - diff --git a/weather-service/weatherSensor/Dockerfile b/weatherSensor/Dockerfile similarity index 100% rename from weather-service/weatherSensor/Dockerfile rename to weatherSensor/Dockerfile diff --git a/weather-service/weatherSensor/requirements.txt b/weatherSensor/requirements.txt similarity index 100% rename from weather-service/weatherSensor/requirements.txt rename to weatherSensor/requirements.txt diff --git a/weather-service/weatherSensor/weather_sensor.py b/weatherSensor/weather_sensor.py similarity index 100% rename from weather-service/weatherSensor/weather_sensor.py rename to weatherSensor/weather_sensor.py