initial support for automated testing with NTFC #17728
Conversation
szafonimateusz-mi
requested review from
btashton,
liuguo09,
lupyuen,
michallenc,
simbit18,
xiaoxiang781216 and
yamt
as code owners
github-actions
bot
added
Area: Tooling
Area: CI
Board: simulator
Size: M
szafonimateusz-mi
force-pushed
the
ntfc_for_sim
branch
from
9ffdacc to
25b7506
Compare
|
I have updated the description according to the CI results. Linux sim-01 took 1h 13m 56s, before this change it took 58m 25s, so tests take 15 minutes more. For comparison, the entire CI workflow took 3 hours. Summary from pytest before this PR: Summary from pytest for this PR: |
szafonimateusz-mi
force-pushed
the
ntfc_for_sim
branch
2 times, most recently
from
d3eddaa to
37e6d83
Compare
raiden00pl
force-pushed
the
ntfc_for_sim
branch
from
37e6d83 to
a6d029e
Compare
szafonimateusz-mi
force-pushed
the
ntfc_for_sim
branch
from
a6d029e to
5373c99
Compare
Add support for NTFC testing framework. At the moment NTFC is downloaded from a private repo, and installed with pip and venv. In the future NTFC repositories will be migrated to Apache organization. Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>
export currently build configuration and path to artifacts so these can be used by NTFC run script Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>
szafonimateusz-mi
force-pushed
the
ntfc_for_sim
branch
from
5373c99 to
50e19e1
Compare
migrate sim/citest to NTFC test cases Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>
Executable files called "run" don't pass checkpath verification, so we change the name to "run.sh" Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>
szafonimateusz-mi
force-pushed
the
ntfc_for_sim
branch
from
50e19e1 to
ebf548e
Compare
acassis
left a comment
acassis
left a comment
There was a problem hiding this comment.
@szafonimateusz-mi please include basic Documentation about this new test framework
|
What do you want documented? The NTFC documentation is in the NTFC repository https://github.com/szafonimateusz-mi/nuttx-ntfc/ |
github-actions
bot
added
the
Area: Documentation
szafonimateusz-mi
force-pushed
the
ntfc_for_sim
branch
from
f82a560 to
2b8bc21
Compare
Sorry, I missed that Running CI Tests with |
szafonimateusz-mi
force-pushed
the
ntfc_for_sim
branch
from
2b8bc21 to
f82fcd5
Compare
|
@acassis No need to sorry, I updated this file after your comment :P I've also created a new top section in the docs: "Testing." Ultimately, CI's operation can be better described there. |
| The framework and official test cases are available at: | ||
|
|
||
| - https://github.com/szafonimateusz-mi/nuttx-ntfc | ||
| - https://github.com/szafonimateusz-mi/nuttx-testing |
There was a problem hiding this comment.
I hope the plan is to eventually move the framework and official test cases under the apache/nuttx repositories?
There was a problem hiding this comment.
why moving test case to nuttx repo?
|
Hi all, as I said in my review comment, I hope the plan is to eventually move the framework and official test cases under the apache/nuttx repositories. |
|
Question: Could this repository Apache NuttX Testing be reused to contain both repositories NuttX Test Framework and NTFC test cases ? |
|
@simbit18 we need 2 separate repositories. NTFC is not dependent on NTFC test cases in any way, the user can use their own test cases. In addition, NTFC is organized as a Python package and has different requirements regarding maintainence and CI. @hartmannathan yes, it will be moved to Apache, we just need one more repository for nuttx-ntfc. Do we need INFRA to create a new repository or can we do it ourselves? |
raiden00pl
force-pushed
the
ntfc_for_sim
branch
from
f82fcd5 to
4720f92
Compare
update citests.rst and move page from Guides to Testing Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>
@raiden00pl We can create repositories using INFRA's Self-Service tools, but I think that will create empty repositories. I think what we need in this case is to clone the repos; correct me if this isn't the case. I don't see a way to create a repo as a clone. We can ask INFRA to do it by opening a Jira ticket and telling them the URLs of the repos we want cloned... Thoughts? |
|
@hartmannathan We can clone the repo with INFRA, but we can also do it differently: create an empty repository and add all changes in one pull request. Both options are OK for me, the question is which one is faster :) |
|
Here's a PR for nuttx-testing adding test cases to repo: apache/nuttx-testing#102 |
6 participants
Summary
First step to solve #17717
This PR adds support for NTFC in CI and migrates
sim/citestto use NTFC instead oftools/ci/testruntool.The rest of
citestconfigs uses the old testing method.635cb70 must be also ported to nuttx-apps.
EDIT: PR for nuttx-apps is here apache/nuttx-apps#3291 [DONE]
Impact
NTFC performs more tests so
sim-01will take longer but will provide better code coverage.EDIT: With this change
Linux sim-01took 1h 13m 56s, before this change it took 58m 25s.Testing
Tested in a private repository, let's see if it works with upstream.
Here is an example run test for
sim/citeston my fork with all tests passed:https://github.com/szafonimateusz-mi/nuttx/actions/runs/20580444513/job/59106747416#step:11:154
I will update this section when upstream CI is finished.
EDIT: successful run is here:
https://github.com/apache/nuttx/actions/runs/20582851657/job/59113926720?pr=17728#step:11:247
Test start:
Test end:
Linux sim-01 took 1h 13m 56s, before this change it took 58m 25s, so tests for this target take 15 minutes more (not bad):
Summary from pytest before this PR:
Summary from pytest for this PR: