Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
8a9abb6
feat(data-source): Implementar la arquitectura de software completa
Anyerrr Apr 5, 2025
d0cb919
Merge pull request #18 from nlp4se/feature/DS1-integrate-first-data-s…
Anyerrr Apr 5, 2025
c028e4b
feat(tests): Verificar la persistència de les dades
Anyerrr Apr 8, 2025
3bed8ec
Merge pull request #19 from nlp4se/feature/PV1-tests
Anyerrr Apr 9, 2025
c91053d
feat(software-arch): Implementar la arquitectura de software completa
Anyerrr Apr 9, 2025
5cf8c8a
Merge pull request #20 from nlp4se/feature/software-architecture
Anyerrr Apr 9, 2025
9365ff9
feat(all-tests): Repassar tots els units tests i el test d'integració…
Anyerrr Apr 9, 2025
0350bfb
feat(all-tests): Repassar tots els units tests i el test d'integració…
Anyerrr Apr 9, 2025
e2a4085
Merge pull request #21 from nlp4se/feature/PV1-all-tests
Anyerrr Apr 9, 2025
9dcaec2
Refactor: move discover_sources into SourceRegistrationService as sta…
Anyerrr Apr 10, 2025
9c004a5
Merge pull request #22 from nlp4se/refactor/discovery-in-service
Anyerrr Apr 10, 2025
37398b7
Refactor: Borrar el directori definitions de sources.
Anyerrr Apr 10, 2025
91e88a3
Merge pull request #23
Anyerrr Apr 10, 2025
16b60fa
Hotfix: Arreglar el servei de metric, també crear el test
Anyerrr Apr 10, 2025
38e4366
Merge pull request #24
Anyerrr Apr 10, 2025
84bba57
feat(news): Dissenyar i implementar el connector a l’API corresponent…
Anyerrr Apr 16, 2025
08b3db0
feat(news): Guardar les dades obtingudes a la base de dades
Anyerrr Apr 16, 2025
ac156f5
feat(news): Creació de la commanda poll_metrics per a recullir-les di…
Anyerrr Apr 16, 2025
6cd95c5
Merge pull request #25 from nlp4se/feature/DS2-integrate-news
Anyerrr Apr 16, 2025
ad8e555
feat(reddit): Dissenyar i implementar el connector a l’API correspone…
Anyerrr Apr 18, 2025
f8f83dd
Merge pull request #26 from nlp4se/feature/DS2-integrate-reddit
Anyerrr Apr 18, 2025
0b8c5de
feat(test): Escriure tests per al connector de Reddit
Anyerrr Apr 18, 2025
2182349
Merge pull request #27 from nlp4se/feature/PV2-tests-reddit
Anyerrr Apr 18, 2025
a9049d5
feat(google-play): Crear el connector de Google Play dins l’app sources
Anyerrr Apr 18, 2025
d997c52
feat(test-google-play): Escriure tests per als connectors de Google Play
Anyerrr Apr 18, 2025
783b1d7
feat(google-play): Implementar la funció fetch per obtenir dades d’un…
Anyerrr Apr 18, 2025
f2b6518
feat(google-play): Implementar la funció per a obtenir les reviews de…
Anyerrr Apr 25, 2025
60814c0
Merge pull request #28 from nlp4se/feature/DS3-integrate-google-play
Anyerrr Apr 25, 2025
d755ec6
hotfix(api-test): Canvi de nom de DAILY_REDDIT_MENTIONS a DAILY_SOCIA…
Anyerrr Apr 25, 2025
12f908b
Merge pull request #29 from nlp4se/hotfix/test-fail
Anyerrr Apr 25, 2025
3af6819
feature(api-rest): Integrar DRF Spectacular al projecte (llibreries i…
Anyerrr Apr 25, 2025
b113a7b
feature(api-rest): Crear les views i urls de l’app app i documentar-ho
Anyerrr Apr 25, 2025
e81fc7b
feature(api-rest): Crear les views i urls de l’app app i documentar-ho
Anyerrr Apr 27, 2025
7de3abd
feature(api-rest): Crear les views i urls de l’app source i documenta…
Anyerrr Apr 28, 2025
3c19469
Remove tracked __pycache__ files
Apr 29, 2025
7d8dbd9
Added tag Apps
Apr 29, 2025
789ad02
feat(api-rest): Crear les views i urls de l’app review i documentar-h…
Apr 29, 2025
731e706
Remove tracked __pycache__ files
Anyerrr Apr 29, 2025
bcf6520
Merge remote-tracking branch 'origin/feature/DS5-api-rest' into featu…
Anyerrr Apr 29, 2025
2188ea5
feature(api-rest): Crear les views i urls de l’app review i documenta…
Anyerrr Apr 29, 2025
ec0656b
feat(api-rest): post de poll_reviews manual
Anyerrr Apr 30, 2025
2b3371c
feat(api-rest): Import no usats
Anyerrr Apr 30, 2025
88adedc
feat(api-rest): Crear les views i urls de l’app metrics i documentar-ho
Anyerrr Apr 30, 2025
1173d55
feature(api-rest): Correció d'errors
Anyerrr May 2, 2025
68a9073
Merge pull request #30 from nlp4se/feature/DS5-api-rest
Anyerrr May 4, 2025
a174b39
feature(metrics): Implementar les mètriques restants al sistema (mode…
Anyerrr May 4, 2025
68aaf04
feature(metrics): Implementar funció per a la recollida periòdica de …
Anyerrr May 4, 2025
d1094cd
feat(celery): Implementar funció per a la recollida periòdica de mètr…
Anyerrr May 5, 2025
3004fc6
feat(celery): Configuració correcte de docker compose
Anyerrr May 5, 2025
4508663
refact(readme): canvi en el readme i creada la license
Anyerrr May 5, 2025
a816820
feat(celery): Acabar de polir els processos
Anyerrr May 5, 2025
aedff28
feat(poll_reviews): poll_reviews manual acabat
Anyerrr May 5, 2025
b9ca752
feature(poll_metrics): Implementar funció per a la recollida periòdic…
Anyerrr May 5, 2025
dc98d41
feature(poll_metrics): Implementar funció per a la recollida periòdic…
Anyerrr May 5, 2025
b5063df
Merge pull request #31 from nlp4se/feature/DS7-finish-metrics
Anyerrr May 6, 2025
5d3e53a
feature(re-miner): Configurar la connexió als microserveis de RE-Mine…
Anyerrr May 11, 2025
bce5fb2
Merge pull request #32 from nlp4se/feature/DS4-re-miner
Anyerrr May 11, 2025
0e02e5c
hotfix(tests): Arreglar tests d'integritat i unitaris.
Anyerrr May 11, 2025
4acfe16
Merge pull request #33 from nlp4se/hotfix/tests
Anyerrr May 11, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .flake8
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[flake8]
max-line-length = 88
max-line-length = 100
extend-ignore = E203, W503
exclude = .venv,venv,migrations,__pycache__,.git
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2025 [Anyer Moreno Alcaraz]

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
109 changes: 92 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,31 +1,106 @@
# MetaAppCollector - Backend

MetaAppCollector és una eina per agregar metadades d'aplicacions mòbils des de diverses fonts externes, com botigues d'apps, xarxes socials i motors de cerca.
**MetaAppCollector** és una eina *open source* per monitoritzar aplicacions mòbils agregant informació pública de diverses fonts com botigues d'apps i xarxes socials.

Aquest projecte és el backend del sistema, desenvolupat amb **Django**.
> 🧪 Desenvolupada amb Django, Celery, Redis i Docker. Pensada per a desenvolupadors, investigadors i equips de màrqueting que volen entendre millor la visibilitat de les seves apps.

## 🔧 Requisits
![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)

- Python 3.12
- Django 5.1.7
- Git
---

## 🚀 Instal·lació
## 🔧 Funcionalitats principals

- 🔍 Recuperació de metadades i reviews d'aplicacions mòbils (Google Play, etc.)
- 📊 Agregació de mètriques d'interacció i visibilitat
- ⏱️ Polling periòdic configurable per recollida automàtica
- 🧠 Arquitectura extensible amb adaptadors dinàmics
- 🐇 Execució asíncrona de tasques amb Celery + Redis

---

## 🚀 Posada en marxa amb Docker

Assegura’t de tenir instal·lats:

- [Docker](https://www.docker.com/)
- [Docker Compose](https://docs.docker.com/compose/)

### 1. Clona el projecte

```bash
git clone https://github.com/el-teu-usuari/metaappcollector-backend.git
cd metaappcollector-backend
```

### 2. Configura variables d'entorn

Crea un fitxer .env a l'arrel amb el següent contingut mínim:

```bash
DJANGO_SECRET_KEY=your-secret-key
DJANGO_DEBUG=True
```

### 3. Executa els serveis

```bash
git clone git@github.com:Anyerrr/MetaAppCollector-Backend.git
cd MetaAppCollector-Backend
python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt
docker compose up --build
```
Això iniciarà:

## 🛠 ️ Execució
- El servidor Django a http://localhost:8000
- El worker de Celery
- El Beat de Celery (scheduler)
- Redis com a broker

## 🧪 Prova Celery
Obre una shell dins del contenidor web:
```bash
docker compose exec web python manage.py shell
```
I prova una tasca:
```bash
from polling.tasks import prueba_celery
prueba_celery.delay()
```
Pots veure els resultats al log del worker:
```bash
docker compose logs -f celery
```

## 🧱 Estructura del projecte
```bash
python manage.py runserver
├── polling/ # Lògica de polling i tasques Celery
├── metric/ # Agregació de mètriques
├── review/ # Gestió de reviews
├── source/ # Fonts de dades integrades
├── app/ # Aplicacions mòbils registrades
├── config/ # Configuració general de Django i Celery
├── docker-compose.yml # Definició dels serveis
```

## 🧪 Proves
## 📚 Comandes útils
```bash
python manage.py test
```
docker compose exec web python manage.py migrate # Aplicar migracions
docker compose exec web python manage.py createsuperuser
docker compose exec web python manage.py shell # Obrir shell
```

## 🤝 Contribució
Les contribucions són benvingudes! Si vols col·laborar:

1. Fes un fork del repositori
2. Crea una branca nova (git checkout -b feature/nou-adaptador)
3. Fes els teus canvis i commiteja (git commit -am 'Afegit adaptador per X')
4. Puja la branca (git push origin feature/nou-adaptador)
5. Fes una pull request 🙌

Abans d’enviar, assegura’t de seguir l’estil de codi del projecte i escriure tests quan sigui necessari.


## ✉️ Contacte
Per dubtes tècnics o col·laboració, contacta amb el desenvolupador principal:
📧 a.moreno@estudiantat.upc.edu

## 📄 Llicència
Aquest projecte està llicenciat sota la MIT License – consulta el fitxer LICENSE per a més informació.
19 changes: 19 additions & 0 deletions app/migrations/0002_app_code.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Generated by Django 5.1.7 on 2025-04-05 14:17

from django.db import migrations, models


class Migration(migrations.Migration):

dependencies = [
("app", "0001_initial"),
]

operations = [
migrations.AddField(
model_name="app",
name="code",
field=models.CharField(default="default_code", max_length=100, unique=True),
preserve_default=False,
),
]
18 changes: 18 additions & 0 deletions app/migrations/0003_alter_app_developer.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Generated by Django 5.1.7 on 2025-04-27 12:31

from django.db import migrations, models


class Migration(migrations.Migration):

dependencies = [
("app", "0002_app_code"),
]

operations = [
migrations.AlterField(
model_name="app",
name="developer",
field=models.CharField(max_length=100, null=True),
),
]
3 changes: 2 additions & 1 deletion app/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,12 @@


class App(models.Model):
code = models.CharField(max_length=100, unique=True)
name = models.CharField(max_length=200)
description = models.TextField()
appstore_id = models.CharField(max_length=100, null=True, unique=True)
playstore_id = models.CharField(max_length=100, null=True, unique=True)
developer = models.CharField(max_length=100, null=True, unique=True)
developer = models.CharField(max_length=100, null=True)
available_on_ios = models.BooleanField(default=False)
available_on_android = models.BooleanField(default=False)
pegi_rating = models.CharField(max_length=10, null=True) # PEGI rating
Expand Down
26 changes: 26 additions & 0 deletions app/repositories.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
from .models import App


class AppRepository:
@staticmethod
def get_all():
return App.objects.all()

@staticmethod
def get_by_id(app_id):
return App.objects.get(id=app_id)

@staticmethod
def create(data):
return App.objects.create(**data)

@staticmethod
def update(instance, data):
for attr, value in data.items():
setattr(instance, attr, value)
instance.save()
return instance

@staticmethod
def delete(instance):
instance.delete()
28 changes: 28 additions & 0 deletions app/serializers.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
from rest_framework import serializers

from .models import App


class AppSerializer(serializers.ModelSerializer):
class Meta:
model = App
fields = [
"id",
"code",
"name",
"description",
"appstore_id",
"playstore_id",
"developer",
"available_on_ios",
"available_on_android",
"pegi_rating",
"release_date",
"min_ios_version",
]


class AppCreateSerializer(serializers.ModelSerializer):
class Meta:
model = App
fields = ["code", "name", "description", "appstore_id", "playstore_id"]
93 changes: 93 additions & 0 deletions app/services.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
from datetime import datetime

from django.core.exceptions import ObjectDoesNotExist
from rest_framework.exceptions import NotFound

from .repositories import AppRepository


class AppService:
def __init__(self, itunes_adapter=None, google_play_adapter=None):
self.repo = AppRepository()
self._itunes_adapter = itunes_adapter
self._google_play_adapter = google_play_adapter

@property
def itunes_adapter(self):
if self._itunes_adapter is None:
from source.adapters.itunes import ItunesSearchAPIAdapter

self._itunes_adapter = ItunesSearchAPIAdapter()
return self._itunes_adapter

@property
def google_play_adapter(self):
if self._google_play_adapter is None:
from source.adapters.google_play_scraper import GooglePlayScraperAdapter

self._google_play_adapter = GooglePlayScraperAdapter()
return self._google_play_adapter

def list_apps(self):
return self.repo.get_all()

def get_app(self, app_id):
try:
return self.repo.get_by_id(app_id)
except ObjectDoesNotExist:
raise NotFound(f"The app with ID '{app_id}' is not registered.")

def create_app(self, validated_data):
self._fetch_appstore_data(validated_data)
self._fetch_playstore_data(validated_data)
return self.repo.create(validated_data)

def _fetch_appstore_data(self, validated_data):
if not validated_data.get("appstore_id"):
return

result = self.itunes_adapter.lookup_app(validated_data["appstore_id"])
if not result:
return

validated_data["available_on_ios"] = True
validated_data["developer"] = result.get("artistName")
validated_data["pegi_rating"] = result.get("contentAdvisoryRating")
validated_data["min_ios_version"] = result.get("minimumOsVersion")

released_str = result.get("releaseDate")
if released_str:
try:
if "T" in released_str:
release_date = datetime.fromisoformat(released_str.replace("Z", "")).date()
validated_data["release_date"] = release_date
except Exception as e:
print(f"Error parsejant release_date (AppStore): {released_str} ({e})")

def _fetch_playstore_data(self, validated_data):
if not validated_data.get("playstore_id"):
return

result = self.google_play_adapter.lookup_app(validated_data["playstore_id"])
if not result:
return

validated_data["available_on_android"] = True
validated_data["developer"] = validated_data.get("developer") or result.get("developer")
validated_data["pegi_rating"] = validated_data.get("pegi_rating") or result.get(
"contentRating"
)

released_str = result.get("released")
if released_str and not validated_data.get("release_date"):
try:
release_date = datetime.strptime(released_str, "%b %d, %Y").date()
validated_data["release_date"] = release_date
except ValueError:
print(f"Format de data invàlid (PlayStore): {released_str}")

def update_app(self, instance, validated_data):
return self.repo.update(instance, validated_data)

def delete_app(self, instance):
return self.repo.delete(instance)
18 changes: 18 additions & 0 deletions app/tests.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
import pytest

from app.models import App


@pytest.mark.django_db
def test_create_source():
a = App.objects.create(
code="test_app",
name="Test App",
description="Descripction of the test app",
appstore_id="123456",
)
assert a.pk is not None
assert a.code == "test_app"
assert a.name == "Test App"
assert a.description == "Descripction of the test app"
assert a.appstore_id == "123456"
11 changes: 11 additions & 0 deletions app/urls.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
from django.urls import include, path
from rest_framework.routers import DefaultRouter

from .views import AppViewSet

router = DefaultRouter()
router.register(r"apps", AppViewSet, basename="app")

urlpatterns = [
path("", include(router.urls)),
]
Loading