Mac support with M1 processor (darwin, arm64)

[victorpaleologue]

When using the VSCode PDDL extension, I am given the opportunity to download and configure a pre-built VAL tool. The matching binary is not found for my platform: Mac OS on an M1 processor (darwin, arm64).
I reported the issue there jan-dolejsi/vscode-pddl#127, however I feel like this issue is that the VAL release for that platform is missing.

[victorpaleologue]

Can I help? I am at ease with CMake and GitHub Actions.
I would have tried out updating the Azure pipeline, but I cannot find the relevant YAML file.
If someone can point at where to start, I could do something.

[DerekLong101]

Ah, that is possible. We checked compilation on a Mac, but I don't think any of us uses one as a matter of course, so we don't tend to be good at keeping up with them. So, yes, we could probably benefit from some help. The Azure pipeline is inside the company firewalls, so I don't think it will be easy to give access. But we might be able to give you the scripts we use so that you can do the surgery and send them back for us. Let me check.

…

________________________________ From: Victor Paléologue ***@***.***> Sent: 16 June 2022 16:43 To: KCL-Planning/VAL ***@***.***> Cc: Subscribed ***@***.***> Subject: [KCL-Planning/VAL] Mac support with M1 processor (darwin, arm64) (Issue #56) When using the VSCode PDDL extension, I am given the opportunity to download and configure a pre-built VAL tool. The matching binary is not found for my platform: Mac OS on an M1 processor (darwin, arm64). I reported the issue there jan-dolejsi/vscode-pddl#127<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjan-dolejsi%2Fvscode-pddl%2Fissues%2F127&data=05%7C01%7Cderek.long%40kcl.ac.uk%7C7ea983be626f4c13649b08da4faef7c7%7C8370cf1416f34c16b83c724071654356%7C0%7C0%7C637909910149592278%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jgnEUBHZ3ceeXDlPm%2F7AtJRIhGItMbTs69MoPyXc9bI%3D&reserved=0>, however I feel like this issue is that the VAL release for that platform is missing. — Reply to this email directly, view it on GitHub<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FKCL-Planning%2FVAL%2Fissues%2F56&data=05%7C01%7Cderek.long%40kcl.ac.uk%7C7ea983be626f4c13649b08da4faef7c7%7C8370cf1416f34c16b83c724071654356%7C0%7C0%7C637909910149602273%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XgflaIeWBrHm4cIVS8M%2F2%2F3qxIEeOq1gcZqWodzkhaU%3D&reserved=0>, or unsubscribe<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAB2KOVHONTC2767QWAVSRU3VPNDSHANCNFSM5Y7HUIBA&data=05%7C01%7Cderek.long%40kcl.ac.uk%7C7ea983be626f4c13649b08da4faef7c7%7C8370cf1416f34c16b83c724071654356%7C0%7C0%7C637909910149607271%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=08UumKY4b6O0FCGfIout9f5ARdyWkHLMkFeo%2BMi8ST0%3D&reserved=0>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[victorpaleologue]

FYI it compiles well on Mac, all we need is a release.
I'm still available for help.

[DerekLong101]

Thanks, Victor. I will definitely come back to you on this - I am currently submerged with a few other jobs. This is on the stack!

…

________________________________ From: Victor Paléologue ***@***.***> Sent: 13 July 2022 11:38 To: KCL-Planning/VAL ***@***.***> Cc: Long, Derek ***@***.***>; Comment ***@***.***> Subject: Re: [KCL-Planning/VAL] Mac support with M1 processor (darwin, arm64) (Issue #56) FYI it compiles well on Mac, all we need is a release. I'm still available for help. — Reply to this email directly, view it on GitHub<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FKCL-Planning%2FVAL%2Fissues%2F56%23issuecomment-1183058955&data=05%7C01%7Cderek.long%40kcl.ac.uk%7C3a14725ad7954a4d814408da64bbcfc9%7C8370cf1416f34c16b83c724071654356%7C0%7C0%7C637933055045490886%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5dzTWgPMD1b7qXwD2fadZpA4rE7Lnokf4jRZPRCGSF0%3D&reserved=0>, or unsubscribe<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAB2KOVAKPYKBFPEAJ4WTJ53VT2MB5ANCNFSM5Y7HUIBA&data=05%7C01%7Cderek.long%40kcl.ac.uk%7C3a14725ad7954a4d814408da64bbcfc9%7C8370cf1416f34c16b83c724071654356%7C0%7C0%7C637933055045490886%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OWsutMOVkVT8dx3fneUborOAMxAOLJScgcv46Dqy%2BqU%3D&reserved=0>. You are receiving this because you commented.Message ID: ***@***.***>

[jan-dolejsi]

Can I help? I am at ease with CMake and GitHub Actions. I would have tried out updating the Azure pipeline, but I cannot find the relevant YAML file. If someone can point at where to start, I could do something.

Yes, you can definitely help. Modifying the azure pipeline to harvest and publish the binary artifacts would be the simplest change at the end. The hard part is to change the build scripts, so they produce the arm binaries. You can see the azure-pipeline.yml in the root of the repo. It is simply calling ./scripts/build_linux_windows.sh and ./scripts/build_macos.sh on ubuntu and MacOS respectively. The arm64 script should be added either to the build_linux_windows.sh, if the compilation for arm64 can be done on linux, or as a separate build_arm64.sh script, which we would add to the pipeline to run on hosted arm64 pool (not sure it exists).

The pipeline then runs over there: https://dev.azure.com/schlumberger/ai-planning-validation/_build?definitionId=2

[DerekLong101]

Thanks, Jan!

…

________________________________ From: Jan Dolejší ***@***.***> Sent: 13 July 2022 17:02 To: KCL-Planning/VAL ***@***.***> Cc: Long, Derek ***@***.***>; Comment ***@***.***> Subject: Re: [KCL-Planning/VAL] Mac support with M1 processor (darwin, arm64) (Issue #56) Can I help? I am at ease with CMake and GitHub Actions. I would have tried out updating the Azure pipeline, but I cannot find the relevant YAML file. If someone can point at where to start, I could do something. Yes, you can definitely help. Modifying the azure pipeline to harvest and publish the binary artifacts would be the simplest change at the end. The hard part is to change the build scripts, so they produce the arm binaries. You can see the azure-pipeline.yml in the root of the repo. It is simply calling ./scripts/build_linux_windows.sh and ./scripts/build_macos.sh on ubuntu and MacOS respectively. The arm64 script should be added either to the build_linux_windows.sh, if the compilation for arm64 can be done on linux, or as a separate build_arm64.sh script, which we would add to the pipeline to run on hosted arm64 pool (not sure it exists). The pipeline then runs over there: https://dev.azure.com/schlumberger/ai-planning-validation/_build?definitionId=2<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdev.azure.com%2Fschlumberger%2Fai-planning-validation%2F_build%3FdefinitionId%3D2&data=05%7C01%7Cderek.long%40kcl.ac.uk%7Cb91c76e4a86c423b7a5408da64e90ce1%7C8370cf1416f34c16b83c724071654356%7C0%7C0%7C637933249343815555%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=EeK7bPRALOqRtwxoYED0wbldkqc0QCnZsZ8uBTHQomA%3D&reserved=0> — Reply to this email directly, view it on GitHub<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FKCL-Planning%2FVAL%2Fissues%2F56%23issuecomment-1183405462&data=05%7C01%7Cderek.long%40kcl.ac.uk%7Cb91c76e4a86c423b7a5408da64e90ce1%7C8370cf1416f34c16b83c724071654356%7C0%7C0%7C637933249343815555%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=WcWaeJ5pj3ZCGYvpIiI3bC0kS%2BcdgSARkl4DH6gy7Zc%3D&reserved=0>, or unsubscribe<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAB2KOVESXWZKX6EZ7656LODVT3SAJANCNFSM5Y7HUIBA&data=05%7C01%7Cderek.long%40kcl.ac.uk%7Cb91c76e4a86c423b7a5408da64e90ce1%7C8370cf1416f34c16b83c724071654356%7C0%7C0%7C637933249343815555%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tHOYi%2BYioJG%2FG1P5DknvqGY4jVa9iWVfk9LxcxiPB7c%3D&reserved=0>. You are receiving this because you commented.Message ID: ***@***.***>

[victorpaleologue]

You can see the azure-pipeline.yml in the root of the repo.

Sorry but I cannot find that file. Could you provide me a link to it?

[jan-dolejsi]

Oh, my bad. I did not realize the pipeline file was only in my fork. Looking closer, it was work I started and did not finish. It only includes the build job for linux and windows (targeting linux build machine). But it would eventually include the MacOS job too. The build pipeline that we currently run on Azure DevOps is using the infrastructure that predates yaml.

Nevertheless, what is needed for the M1 processor binaries is a build script first. Adding it to the pipeline would just mean calling the new script from using an additional task in the pipeline. Like what you can see on the line 17.

[victorpaleologue]

Alright, I've put myself on it, but... Microsoft-hosted Azure agents for Mac OS do not seem to be running on M1 processors:

Agents that run macOS images are provisioned on Mac pros with a 3 core CPU, 14 GB of RAM, and 14 GB of SSD disk space.

Knowing that M1 processors have many more CPU cores, this hardware is certainly x86_64. GitHub runners have the same hardware specs, btw, and they are explicit on the architecture being x86_64.

I checked scripts/build_macos.sh and it works fine already. But I won't be able to get your CI pipeline run it. Sorry, I genuinely hoped to help =/

However I can think of Conan and ConanCenter, which are dedicated to distribute portable native binaries to every platform. They do produce MacOS arm64 releases, and VAL could be distributed there too. Another big advantage of Conan is that if the binary is missing, it has access to all the recipes to rebuild it for the current host.
But that does not really fit the workflow of the PDDL plug-in for VSCode.

[jan-dolejsi]

@victorpaleologue are you saying that the script scripts/build_macos.sh produces binaries that you can run on the new shiny Mac?
Or only if you run the build locally, while the macos binaries that come out of the pipeline are not compatible?

Yes the pipeline can be fixed eventually. Looks like Microsoft is working on the sup6 for M1 build machines.

I am not against an alternative build and download location. Ad long as the platform specific binaries can be downloaded from a fixed url. Ideally the version should ge part of the url too.

[victorpaleologue]

@victorpaleologue are you saying that the script scripts/build_macos.sh produces binaries that you can run on the new shiny Mac? Or only if you run the build locally, while the macos binaries that come out of the pipeline are not compatible?

Yes, running the script on a new shiny Mac builds a binary compatible with new shiny Macs. But the ones that come out of the pipeline, built on older-but-perhaps-still-shiny Macs, are not compatible.

Yes the pipeline can be fixed eventually. Looks like Microsoft is working on the sup6 for M1 build machines.

I am not against an alternative build and download location. Ad long as the platform specific binaries can be downloaded from a fixed url. Ideally the version should ge part of the url too.

Ok. I could publish my release someplace that won't move, but building manually next versions would be annoying. IMO it has to be automated, and rely on a machine dedicated for that. So I'd rather wait for Microsoft's support...

[DerekLong101]

I think the shine has come off the Macs.

…

________________________________ From: Victor Paléologue ***@***.***> Sent: 03 August 2022 09:22 To: KCL-Planning/VAL ***@***.***> Cc: Long, Derek ***@***.***>; Comment ***@***.***> Subject: Re: [KCL-Planning/VAL] Mac support with M1 processor (darwin, arm64) (Issue #56) @victorpaleologue<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fvictorpaleologue&data=05%7C01%7Cderek.long%40kcl.ac.uk%7Cbe1d2bc8bf8b4862ba8f08da75295b05%7C8370cf1416f34c16b83c724071654356%7C0%7C0%7C637951117724245016%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=f4MPe36nui%2BSX3sb3dS5Bkxr%2BoTMYU7W%2FNc8WCZDQ%2Bw%3D&reserved=0> are you saying that the script scripts/build_macos.sh produces binaries that you can run on the new shiny Mac? Or only if you run the build locally, while the macos binaries that come out of the pipeline are not compatible? Yes, running the script on a new shiny Mac builds a binary compatible with new shiny Macs. But the ones that come out of the pipeline, built on older-but-perhaps-still-shiny Macs, are not compatible. Yes the pipeline can be fixed eventually. Looks like Microsoft is working on the sup6 for M1 build machines. I am not against an alternative build and download location. Ad long as the platform specific binaries can be downloaded from a fixed url. Ideally the version should ge part of the url too. Ok. I could publish my release someplace that won't move, but building manually next versions would be annoying. IMO it has to be automated, and rely on a machine dedicated for that. So I'd rather wait for Microsoft's support... — Reply to this email directly, view it on GitHub<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FKCL-Planning%2FVAL%2Fissues%2F56%23issuecomment-1203639560&data=05%7C01%7Cderek.long%40kcl.ac.uk%7Cbe1d2bc8bf8b4862ba8f08da75295b05%7C8370cf1416f34c16b83c724071654356%7C0%7C0%7C637951117724245016%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vqgOOxr3dAV8EGfbtDmxr7rDjW%2Beiokyk%2BD6ePQ7Ezw%3D&reserved=0>, or unsubscribe<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAB2KOVGOQ7SG2YTEJCGOUL3VXIT5TANCNFSM5Y7HUIBA&data=05%7C01%7Cderek.long%40kcl.ac.uk%7Cbe1d2bc8bf8b4862ba8f08da75295b05%7C8370cf1416f34c16b83c724071654356%7C0%7C0%7C637951117724245016%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VnDxh81IyN9hZbzYa2S7JyNaYkpLYD6ZFOKUP8MVG%2B8%3D&reserved=0>. You are receiving this because you commented.Message ID: ***@***.***>

[DavidKaub]

Any updates on this so far?

[victorpaleologue]

Azure support for arm64-powered MacOS has not changed.
I did not take time to produce my own release. I cannot tell when I can do this.
In the meantime you will have to rebuild it yourself, it should build fine.

[victorpaleologue]

Would you be interested by having it available through Conan?