Syncing APIM fails

Hi Eng Swee,
I tried to sync the API Manager with Git, but there was one API which could not be synced:

2025-05-09T06:16:53.1591548Z �[90m09 May 25 06:16 UTC�[0m �[32mINF�[0m �[1m📢 Begin processing for APIProxy API_BUSINESS_PARTNER�[0m
2025-05-09T06:16:53.1592859Z �[90m09 May 25 06:16 UTC�[0m �[32mINF�[0m �[1mDownloading APIProxy API_BUSINESS_PARTNER�[0m
2025-05-09T06:17:08.2017785Z �[90m09 May 25 06:17 UTC�[0m �[32mINF�[0m �[1mUnzipping contents to /tmp/apim/download/API_BUSINESS_PARTNER�[0m
2025-05-09T06:17:08.7471493Z �[90m09 May 25 06:17 UTC�[0m �[31mFTL�[0m �[1m*fs.PathError open /tmp/apim/download/API_BUSINESS_PARTNER/APIProxies/API_BUSINESS_PARTNER/APIResource/A_CustSlsAreaAddrDepdntTaxInfo(Customer='{Customer}',SalesOrganization='{SalesOrganization}',DistributionChannel='{DistributionChannel}',Division='{Division}',AddressID='{AddressID}',DepartureCountry='{DepartureCountry}',CustomerTaxCategory='{CustomerTaxCategory}').xml: file name too long
2025-05-09T06:17:08.7475986Z /home/runner/work/flashpipe/flashpipe/internal/file/file.go:161 (0x88995f)
2025-05-09T06:17:08.7478078Z /home/runner/work/flashpipe/flashpipe/internal/file/file.go:131 (0x8895fa)
2025-05-09T06:17:08.7479763Z /home/runner/work/flashpipe/flashpipe/internal/api/apiproxy.go:87 (0x88ade5)
2025-05-09T06:17:08.7481040Z /home/runner/work/flashpipe/flashpipe/internal/sync/syncer.go:84 (0xa20765)
2025-05-09T06:17:08.7482201Z /home/runner/work/flashpipe/flashpipe/internal/cmd/apim.go:99 (0xa260e9)
2025-05-09T06:17:08.7483539Z /home/runner/work/flashpipe/flashpipe/internal/cmd/apim.go:54 (0xa2cf11)
2025-05-09T06:17:08.7485393Z /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985 (0x5ef6aa)
2025-05-09T06:17:08.7488126Z /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 (0x5eff7f)
2025-05-09T06:17:08.7488966Z /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041 (0xa29810)
2025-05-09T06:17:08.7491187Z /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041 (0xa29806)
2025-05-09T06:17:08.7494956Z /home/runner/work/flashpipe/flashpipe/cmd/flashpipe/flashpipe.go:11 (0xa2d28f)
2025-05-09T06:17:08.7498837Z /opt/hostedtoolcache/go/1.23.4/x64/src/internal/runtime/atomic/types.go:194 (0x43ae0b)
2025-05-09T06:17:08.7503797Z /opt/hostedtoolcache/go/1.23.4/x64/src/runtime/asm_amd64.s:1700 (0x4767e1)
2025-05-09T06:17:08.7505517Z �[0m
2025-05-09T06:17:08.7532218Z ##[error]Process completed with exit code 1.

It is the OData Service Business Partner (A2X) ( https://api.sap.com/api/API_BUSINESS_PARTNER/overview), which seems to be too big for flashpipe.

[engswee]

Hi @stmac314

From the error (file name too long), it looks like one of the files in the proxy archive has a length of 269 which exceeds the limit in Linux file system.
A_CustSlsAreaAddrDepdntTaxInfo(Customer='{Customer}',SalesOrganization='{SalesOrganization}',DistributionChannel='{DistributionChannel}',Division='{Division}',AddressID='{AddressID}',DepartureCountry='{DepartureCountry}',CustomerTaxCategory='{CustomerTaxCategory}').xml

Since this is an SAP standard API, maybe you may not be able to make any changes to it. I can't think of any quick solution to resolve this issue at the moment. For the time being, if you want to continue syncing other APIs, you can add this to the exclude list.

[ChBrockmann]

Hi @engswee

I am working together with @stmac314 and we discussed a potential fix/workaround for this issue. I tried implementing it, and it seems to work.

Short summary of what I did:

• Added a flag shorten-too-long-filenames which defaults to false
• If this flag is set and the filename exceeds 255 characters the filename gets significantly shortened with a 8 character - SHA256 value.
• That has the advantage of being deterministic to still functions with VCS

The previous output of A_CustSlsAreaAddrDepdntTaxInfo(Customer='{Customer}',SalesOrganization='{SalesOrganization}',DistributionChannel='{DistributionChannel}',Division='{Division}',AddressID='{AddressID}',DepartureCountry='{DepartureCountry}',CustomerTaxCategory='{CustomerTaxCategory}').xml
would be shortened to
A_CustSlsAreaAddrDepdntTaxInfo_HASH-62336ec9_.xml

I haven't created a PR for this, since I personally think I am not 100% there yet since I am new to Go and we are just interested in a small part of functionality from FlashPipe and I don't fully understand all features of FlashPipe.

Maybe you can take a look at my changes and provide me some feedback. If you are satisfied with it I would like to contribute this fix to FlashPipe.

[engswee]

Hi @ChBrockmann

Thanks for the update. It's amazing to hear that you are learning Go (and I do believe you will enjoy it), and are interested in contributing to FlashPipe ❤️

Please allow me sometime to look through your changes. I'll get back to you on that.

In the meantime, I wanted to check if you managed to test if this workaround causes any issue on the reverse direction, i.e. would the shortened file name work when syncing the content from Git back to the tenant? In the past, I've noticed that the upload API is quite sensitive and could fail if some of the content does not meet some of its internal validations.

[ChBrockmann]

Thanks @engswee for your kind words 😊

My plan was to store the original filename and the new "hash-name" in a file. Sort of like a key-value dictionary. That process from the VCS back to SAP needs to be adjusted for sure.
This part of FlashPipe isn't our usecase for now. I don't think, that it would work out of the box right now.

I will take a look at it sometime. For now I was more focused on the "would it even work"-part

[engswee]

Hi @ChBrockmann

I tried the workaround manually using Postman with the following two files renamed based on your approach.

When the API is called, it returns the following error.

So definitely you will need to revert the filename back to the original. Having a key-value dictionary may work, but you need to think about where to store it, as well as how to execute the renaming because you can't do it directly the filesystem.