BOND model returning extremely low performance - version compatibility issue?

[fra533]

Environment

• OS: Windows
• Python version: 3.11.0
• VS Code version: 1.101.0

Package Versions

• torch: 2.2.2
• torch-geometric: 2.5.3
• scikit-learn: 1.4.0
• numpy: 1.23.4
• pandas: 1.5.1
• tqdm: 4.66.1

Hello, I'm trying to run the BOND model following the documentation, but after executing python demo.py --post_match True --mode train and subsequently python demo.py --post_match True --mode valid, I keep getting extremely low F1 scores, below 0.07.

Can you help me understand what I'm doing wrong? I've tested both with pre-built embeddings and by creating them from scratch.

The script I used for evaluation is SNDeval.py from the WhoIsWho pipeline.

[tsiotas]

Hi, I'm having the same problem. I ran:
python demo.py --post_match True --mode train (F1 = 0.87 on train_author.json)
python demo.py --post_match True --mode valid (F1 = 0.07 on sna_valid_example.json)
using python -m whoiswho.evaluation.SNDeval
I definitely expected a lower F1 for the validation dataset than the train dataset, but is it normal to have such a low score in the evaluation of the valid dataset or should I fine-tune some hyperparameters on the valid process, such as --post_match False --db_eps --db_min ?

[xiaolubuhuizhuzhou]

Could you please provide samples of the validation-set outputs if possible? This would help us verify the model's correctness.

Additionally, could you share the model's loss performance in validation-set? It is expected to be under 10.0 for each author graph.