The number of IR nodes is configurable

[End-rey]

Closes #73.

[End-rey]

@cthulhu-rider Now it shows that it is running, although there are a lot of errors in the ir nodes. Can you check the configs and see if I'm doing everything right?

[End-rey]

@cthulhu-rider Everything works. However, after the launch, the following errors appear in the ir node:

debug    network/server.go:1216  p2p notary request      {"error": "already exists in mempool", "hash": "22c5d8035255adf13142eec3e1f1f3256fa8d6874d1806738af44f0601939b09", "main": "dec54cf2feecb63c96ee8ce7f1c372b13bf626e15a0f1acb8dc0991e2811e42d"}

error  governance/process_update.go:82  can't update inner ring list with new alphabet keys  {"error": "test invocation failed: unsupported parameter noderoles.Role"}

error  governance/process_update.go:91  can't update list of notary nodes in FS chain  {"error": "test invocation failed: unsupported parameter noderoles.Role"}

error  governance/process_update.go:100  can't update list of alphabet nodes in neofs contract  {"error": "could not invoke alphabetUpdate: chain/client: contract execution finished with state FAULT; exception: at instruction 3666 (THROW): unhandled exception: \"this method must be invoked by alphabet\""}

error  governance/process_update.go:100  can't update list of alphabet nodes in neofs contract  {"error": "could not invoke alphabetUpdate: chain/client: contract execution finished with state FAULT; exception: at instruction 312 (SYSCALL): System.Contract.Call failed: called contract 6a715fe77786ad16a5f543e45c7ba242f196d01b not found: key not found"}

and in storage node:

debug      calculator/calls.go:231 all daughters trust iterator's init failure     {"error": "no data in 1 epoch for daughters"}
debug   calculator/calls.go:62  consumers trust iterator's init failure {"epoch": 1, "iteration": 1, "error": "no data for 0 iteration in 1 epoch for consumers's trusts"}
debug   calculator/calls.go:120 daughter trust iterator's init failure  {"error": "no data in 2 epoch for daughter: o5H2uGUa7cwBvb1GSx62SUL8Mkpp4PbgB1oUstPgnfo8"}
debug   calculator/calls.go:62  consumers trust iterator's init failure {"epoch": 1, "iteration": 2, "error": "no data for 1 iteration in 1 epoch for consumers's trusts"}
debug   calculator/calls.go:62  consumers trust iterator's init failure {"epoch": 2, "iteration": 2, "error": "no data for 1 iteration in 2 epoch for consumers's trusts"}
debug   calculator/calls.go:62  consumers trust iterator's init failure {"epoch": 1, "iteration": 3, "error": "no data for 2 iteration in 1 epoch for consumers's trusts"}
debug   calculator/calls.go:62  consumers trust iterator's init failure {"epoch": 2, "iteration": 3, "error": "no data for 2 iteration in 2 epoch for consumers's trusts"}
debug   calculator/calls.go:120 daughter trust iterator's init failure  {"error": "no data in 3 epoch for daughter: 2Bsv2nitVMDi9S77fuYNqDeYtRHcxAmp1J6Q7Hx79Wwem"}
debug   calculator/calls.go:62  consumers trust iterator's init failure {"epoch": 3, "iteration": 2, "error": "no data for 1 iteration in 3 epoch for consumers's trusts"}
debug   calculator/calls.go:62  consumers trust iterator's init failure {"epoch": 3, "iteration": 3, "error": "no data for 2 iteration in 3 epoch for consumers's trusts"}
debug   calculator/calls.go:120 daughter trust iterator's init failure  {"error": "no data in 4 epoch for daughter: o5H2uGUa7cwBvb1GSx62SUL8Mkpp4PbgB1oUstPgnfo8"}

Which of these errors are not expected?

[cthulhu-rider]

@End-rey

debug network/server.go:1216 p2p notary request {"error": "already exists in mempool", "hash": "22c5d8035255adf13142eec3e1f1f3256fa8d6874d1806738af44f0601939b09", "main": "dec54cf2feecb63c96ee8ce7f1c372b13bf626e15a0f1acb8dc0991e2811e42d"}

and SN debug messages are np for now

"error": "test invocation failed: unsupported parameter noderoles.Role"
"error": "test invocation failed: unsupported parameter noderoles.Role"

IR bug, pls report

can't update list of alphabet nodes in neofs contract {"error": "could not invoke alphabetUpdate: chain/client: contract execution finished with state FAULT; exception: at instruction 3666 (THROW): unhandled exception: "this method must be invoked by alphabet""}

either an IR bug or setup problem. Report too just in case

can't update list of alphabet nodes in neofs contract {"error": "could not invoke alphabetUpdate: chain/client: contract execution finished with state FAULT; exception: at instruction 312 (SYSCALL): System.Contract.Call failed: called contract 6a715fe77786ad16a5f543e45c7ba242f196d01b not found: key not found"}

most likely setup problem. Check that NeoFS contract is successfully deployed in the main chain and IR uses correct hash

can't update inner ring list with new alphabet keys
can't update list of notary nodes in FS chain
can't update list of alphabet nodes in neofs contract

IR should not attempt to do all these in the DevEnv

Alphabet

u shouldnt merge 2 setups in 1 config here, the list must equal fschain.committee list foreach setup

Notary

forgot about this. Add P2PNotary list the same as NeoFSAlphabet

[End-rey]

All the errors above, except for SN debug and already exists in mempool, are gone.

[cthulhu-rider]

@End-rey dont forget to report bugs https://github.com/nspcc-dev/neofs-node

[cthulhu-rider]

@End-rey pls upd README with new run abilities

UPD: did not see u've updated docs/ir.md. Its enough

[cthulhu-rider]

my trial's summary

Fetch and run right away

make up/basic ☑️
neofs-cli container create ☑️

$ docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED              STATUS                        PORTS     NAMES
c7d39a3c2bba   nspccdev/neofs-storage:0.44.1   "neofs-node --config…"   About a minute ago   Up About a minute (healthy)             s04
ce82d7a1cd85   nspccdev/neofs-storage:0.44.1   "neofs-node --config…"   About a minute ago   Up About a minute (healthy)             s03
410cea04da4b   nspccdev/neofs-storage:0.44.1   "neofs-node --config…"   About a minute ago   Up About a minute (healthy)             s02
087d12987812   nspccdev/neofs-storage:0.44.1   "neofs-node --config…"   About a minute ago   Up About a minute (healthy)             s01
890db6f0c6f7   nspccdev/neofs-ir:0.44.1        "neofs-ir --config /…"   2 minutes ago        Up 2 minutes (healthy)                  ir01
1d8f909711ea   nspccdev/neo-go:0.107.1         "/usr/bin/privnet-en…"   2 minutes ago        Up 2 minutes (healthy)                  main_chain
0c8cf4bc8dd8   debian:10                       "/bin/sleep infinity"    2 minutes ago        Up 2 minutes                            bastion

$ docker volume ls
DRIVER    VOLUME NAME
local     ir1_neofs_chain_ir01
local     ir_neofs_chain_ir01
local     storage_storage_s01
local     storage_storage_s02
local     storage_storage_s03
local     storage_storage_s04

Switch to 4

3:

$ make up/basic
Invalid IR number 3; supported numbers: (1 4 7)
make: *** [Makefile:75: check_nodes] Error 1

☑️

-3:

$ make up/basic
Usage: grep [OPTION]... PATTERNS [FILE]...
Try 'grep --help' for more information.
Invalid IR number -3; supported numbers: (1 4 7)

☑️

4:

$ make up/basic
ERROR: for ir01  Cannot create container for service ir01: Conflict. The container name "/ir01" is already in use by container "890db6f0c6f78873a12c51e6e7083d3c34d80e4bbff6d0f4cdfdc7597858bf43". You have to remove (or rename) that container to be able to reuse that name.
Creating ir03 ... done
Creating ir04 ... done
Creating ir02 ... done

ERROR: for ir01  Cannot create container for service ir01: Conflict. The container name "/ir01" is already in use by container "890db6f0c6f78873a12c51e6e7083d3c34d80e4bbff6d0f4cdfdc7597858bf43". You have to remove (or rename) that container to be able to reuse that name.

$ docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED          STATUS                     PORTS     NAMES
b81add0ef95f   nspccdev/neofs-ir:0.44.1        "neofs-ir --config /…"   9 minutes ago    Up 9 minutes (unhealthy)             ir03
5f9d1e43ee4c   nspccdev/neofs-ir:0.44.1        "neofs-ir --config /…"   9 minutes ago    Up 9 minutes (unhealthy)             ir02
59dc765ed1bc   nspccdev/neofs-ir:0.44.1        "neofs-ir --config /…"   9 minutes ago    Up 9 minutes (unhealthy)             ir04
c7d39a3c2bba   nspccdev/neofs-storage:0.44.1   "neofs-node --config…"   13 minutes ago   Up 13 minutes (healthy)              s04
ce82d7a1cd85   nspccdev/neofs-storage:0.44.1   "neofs-node --config…"   13 minutes ago   Up 13 minutes (healthy)              s03
410cea04da4b   nspccdev/neofs-storage:0.44.1   "neofs-node --config…"   13 minutes ago   Up 13 minutes (healthy)              s02
087d12987812   nspccdev/neofs-storage:0.44.1   "neofs-node --config…"   13 minutes ago   Up 13 minutes (healthy)              s01
890db6f0c6f7   nspccdev/neofs-ir:0.44.1        "neofs-ir --config /…"   14 minutes ago   Up 14 minutes (healthy)              ir01
0c8cf4bc8dd8   debian:10                       "/bin/sleep infinity"    14 minutes ago   Up 14 minutes                        bastion

$ docker volume ls
DRIVER    VOLUME NAME
local     aio_cache
local     aio_data
local     ir1_neofs_chain_ir01
local     ir4_neofs_chain_ir01
local     ir4_neofs_chain_ir02
local     ir4_neofs_chain_ir03
local     ir4_neofs_chain_ir04
local     ir_neofs_chain_ir01
local     storage_storage_s01
local     storage_storage_s02
local     storage_storage_s03
local     storage_storage_s04

3 nodes unhealthy, 1 is healthy. Container can be created. 3 nodes stuck in with info blockchain/blockchain.go:534 waiting for synchronization with the blockchain network.... I think this is caused by namespace conflict, intersection leads to this behavior.

devenv keep goin in 1X run mode, this is unexpected to me. On the other side, error was printed. Id prefer purely independent states. Some kind of 🛑 to me

$ make down
error while removing network: network basenet_internet id e101c36b4b028202b1f9cfe0d934391281bf12b5b6afa4b9cd7228230f158940 has active endpoints

$ docker ps
CONTAINER ID   IMAGE                      COMMAND                  CREATED          STATUS                    PORTS     NAMES
890db6f0c6f7   nspccdev/neofs-ir:0.44.1   "neofs-ir --config /…"   23 minutes ago   Up 23 minutes (healthy)             ir01

Clean prevoius runs

$ docker stop ir01
$ docker rm 890db6f0c6f78873a12c51e6e7083d3c34d80e4bbff6d0f4cdfdc7597858bf43
890db6f0c6f78873a12c51e6e7083d3c34d80e4bbff6d0f4cdfdc7597858bf43
$ make up/basic
ERROR: for ir01  Cannot create container for service ir01: Conflict. The container name "/ir01" is already in use by container "890db6f0c6f78873a12c51e6e7083d3c34d80e4bbff6d0f4cdfdc7597858bf43". You have to remove (or rename) that container to be able to reuse that name.

$ make down
$ make clean
ir4_neofs_chain_ir01
ir4_neofs_chain_ir02
ir4_neofs_chain_ir03
ir4_neofs_chain_ir04
storage_storage_s01
storage_storage_s02
storage_storage_s03
storage_storage_s04
The target 'clean' completed successfully
$ docker rm 70a324511ecb52b3c912016a4840c49178282d50745fcbc2c0ae85e6166d9c95
70a324511ecb52b3c912016a4840c49178282d50745fcbc2c0ae85e6166d9c95
$ make up/basic
The target 'up/basic' completed successfully

and container successfully created. down, up/basic - state is kept, container still available
☑️

Switch to 7

same cleaning. Works ☑️

Result

• some errors occur, they are clear and i did not need to ask what todo. First fix i tried worked
• volumes are independent, this is very good
• dry runs work

so, i like how it behaves and what abilities it provides

enhancements i can suggest:

1. is to make non-overlapping containers
2. make up/basic 1|4|7, as a replacement of .env file modification, would be a bit more convenient to me

[roman-khimov]

I think this boils down to ability to run independent devenvs of various kinds and requirement to make down before switch to some different env. Both are fine for me at this stage and can be solved separately.

[carpawell]

I looks like services/irX better be placed inside services/ir.