Packaging and deploying using external configuration files

[cwardcode]

Hi,

I'm very excited to have come across this project, and want to incorporate it into some of my own serverless-based microservices. I just did a quick test and noticed that running serverless package --stage local does not allow custom variables/parameters to be read correctly from my serverless.local.yml configuration file. It reports that the variables are missing. If I supply the parameters using the --param flag, it will pick them up, but i would prefer to not pass each one in individually each time I want to package or deploy my service.

Here's an example of the output from running serverless package --stage local

serverless package --stage local
Environment: darwin, node 18.20.3, framework 3.49.0
Docs:        github.com/oss-serverless/serverless

Error:
Cannot resolve serverless.yml: Variables resolution errored with:
  - Cannot resolve variable at "custom.TestParam": The param "MyCustomParam" cannot be resolved from stage params. If you are using Serverless Framework Compose, make sure to run commands via Compose so that all parameters can be resolved

And an example for specifying the param in the command

serverless package --stage local --param="MyCustomParam=test"

Packaging online-library-oss for stage local (eu-west-1)
Compiling with Typescript...
Using local tsconfig.json - tsconfig.json
Typescript compiled.

✔ Service packaged (57s)

I have uploaded a minimal serverless project that reproduces this behavior here.

In serverless.yml, I have defined, under custom:

custom:
  region: ${opt:region, 'us-east-1'}

  Configuration: ${opt:Configuration, 'serverless.${self:provider.stage}.yml'}
  TestParam: ${param:MyCustomParam, file(${self:custom.Configuration}):MyCustomParam}

Under the provider definition, I have stage defined as: stage: ${opt:stage, 'local'}.

The "other" serverless seems to support this just fine, so I was wondering if there may be something that I'm doing wrong, or if the syntax may have changed in oss-serverless?

Thank you!

[mnapoli]

Hi! This project should work as a drop-in replacement for serverless v3. I'm surprised there's a difference (though the params variable had to be adjusted a little bit to drop support for Serverless Dashboard, so maybe a bug slipped in)

Could you clarify:

• what you are describing works correctly with the official serverless CLI, but it is broken with this project
• it works correctly with serverless v3 specifically (not v4)

?

[cwardcode]

Hi @mnapoli,

If you run this command ( serverless package --stage local ) with the official serverless (v3) CLI, it works completely fine, running it with oss-serverless results in the error. The same applies to serverless deploy

[rageycomma]

Might be useful to include full .serverless output betwee n

Hi @mnapoli,

If you run this command ( serverless package --stage local ) with the official serverless (v3) CLI, it works completely fine, running it with oss-serverless results in the error. The same applies to serverless deploy

Which version of the v3 CLI? Are you also able to give a serverless.log from serverless and one from osls?

[cwardcode]

@rageycomma
Sure thing, here's the output of serverless --version and also running the package command with --verbose --debug="*" for the most complete amount of logs possible.

Chris-MacBook-Pro.local: [09:19 PM] cward's in ~/Documents/GitHub/osls-external-configuration ->serverless --version
osls version: 3.49.0
Chris-MacBook-Pro.local: [09:19 PM] cward's in ~/Documents/GitHub/osls-external-configuration ->serverless package --stage local --verbose --debug "*"
process: resolve CLI input (no service schema)
process: resolve CLI input (service schema)
process: resolve eventual service configuration
process: service configuration found at /Users/cward/Documents/GitHub/osls-external-configuration/serverless.yml
process: service configuration file successfully parsed
process: resolve variables meta
process: resolve variables in core properties
process: resolve variables in all properties
process: resolve CLI input (AWS service schema)
process: construct Serverless instance
process: initialize Serverless instance
lifecycle:command:register: config
lifecycle:command:register: config:credentials
lifecycle:command:register: create
lifecycle:command:register: install
lifecycle:command:register: package
lifecycle:command:register: package:function
lifecycle:command:register: deploy
lifecycle:command:register: deploy:function
lifecycle:command:register: deploy:list
lifecycle:command:register: deploy:list:functions
lifecycle:command:register: invoke
lifecycle:command:register: invoke:local
lifecycle:command:register: info
lifecycle:command:register: logs
lifecycle:command:register: metrics
lifecycle:command:register: print
lifecycle:command:register: remove
lifecycle:command:register: rollback
lifecycle:command:register: rollback:function
lifecycle:command:register: plugin
lifecycle:command:register: plugin
lifecycle:command:register: plugin:list
lifecycle:command:register: plugin
lifecycle:command:register: plugin:search
lifecycle:command:register: config
lifecycle:command:register: config:credentials
lifecycle:command:register: aws
lifecycle:command:register: aws:common
lifecycle:command:register: aws:common:validate
lifecycle:command:register: aws:common:cleanupTempDir
lifecycle:command:register: aws:common:moveArtifactsToPackage
lifecycle:command:register: aws:common:moveArtifactsToTemp
lifecycle:command:register: aws
lifecycle:command:register: aws:package
lifecycle:command:register: aws:package:finalize
lifecycle:command:register: aws
lifecycle:command:register: aws:deploy
lifecycle:command:register: aws:deploy:deploy
lifecycle:command:register: aws:deploy:finalize
lifecycle:command:register: aws
lifecycle:command:register: aws:info
lifecycle:command:register: upgrade
lifecycle:command:register: uninstall
lifecycle:command:register: invoke
lifecycle:command:register: invoke:local
lifecycle:command:register: offline
lifecycle:command:register: offline:functionsUpdated
lifecycle:command:register: offline:start
process: resolve CLI input (+ plugins schema)
process: resolve variables
process: handle error
process: finalize { error:
   { ServerlessError: Cannot resolve serverless.yml: Variables resolution errored with:
     - Cannot resolve variable at "custom.TestParam": The param "MyCustomParam" cannot be resolved from stage params. If you are using Serverless Framework Compose, make sure to run commands via Compose so that all parameters can be resolved
       at module.exports (/usr/local/lib/node_modules/osls/lib/configuration/variables/eventually-report-resolution-errors.js:23:9)
       at /usr/local/lib/node_modules/osls/scripts/serverless.js:599:13
       at async /usr/local/lib/node_modules/osls/scripts/serverless.js:511:7
     [stack]:
      'ServerlessError: Cannot resolve serverless.yml: Variables resolution errored with:\n  - Cannot resolve variable at "custom.TestParam": The param "MyCustomParam" cannot be resolved from stage params. If you are using Serverless Framework Compose, make sure to run commands via Compose so that all parameters can be resolved\n    at module.exports (/usr/local/lib/node_modules/osls/lib/configuration/variables/eventually-report-resolution-errors.js:23:9)\n    at /usr/local/lib/node_modules/osls/scripts/serverless.js:599:13\n    at async /usr/local/lib/node_modules/osls/scripts/serverless.js:511:7',
     [message]:
      'Cannot resolve serverless.yml: Variables resolution errored with:\n  - Cannot resolve variable at "custom.TestParam": The param "MyCustomParam" cannot be resolved from stage params. If you are using Serverless Framework Compose, make sure to run commands via Compose so that all parameters can be resolved',
     code: 'VARIABLES_RESOLUTION_ERROR',
     decoratedMessage: undefined,
     [name]: 'ServerlessError' },
  shouldBeSync: undefined }
Environment: darwin, node 18.20.3, framework 3.49.0
Docs:        github.com/oss-serverless/serverless

Error:
Cannot resolve serverless.yml: Variables resolution errored with:
  - Cannot resolve variable at "custom.TestParam": The param "MyCustomParam" cannot be resolved from stage params. If you are using Serverless Framework Compose, make sure to run commands via Compose so that all parameters can be resolved

Running with the official serverless CLI, here's the same commands output

Chris-MacBook-Pro.local: [09:23 PM] cward's in ~/Documents/GitHub/osls-external-configuration ->serverless --version
Running "serverless" from node_modules
Framework Core: 3.38.0 (local) 3.40.0 (global)
Plugin: 7.2.0
SDK: 4.5.1

Chris-MacBook-Pro.local: [09:24 PM] cward's in ~/Documents/GitHub/osls-external-configuration ->serverless package --stage local --verbose --debug "*"
Running "serverless" from node_modules
config: Retrieve local config, auth data undefined
config: Retrieve global config, auth data undefined
process: resolve CLI input (no service schema)
process: resolve CLI input (service schema)
process: resolve eventual service configuration
process: service configuration found at /Users/cward/Documents/GitHub/osls-external-configuration/serverless.yml
process: service configuration file successfully parsed
process: resolve variables meta
process: resolve variables in core properties
process: resolve variables in all properties
process: resolve CLI input (AWS service schema)
process: construct Serverless instance
process: initialize Serverless instance
lifecycle:command:register: config
lifecycle:command:register: config:credentials
lifecycle:command:register: create
lifecycle:command:register: install
lifecycle:command:register: package
lifecycle:command:register: package:function
lifecycle:command:register: deploy
lifecycle:command:register: deploy:function
lifecycle:command:register: deploy:list
lifecycle:command:register: deploy:list:functions
lifecycle:command:register: invoke
lifecycle:command:register: invoke:local
lifecycle:command:register: info
lifecycle:command:register: logs
lifecycle:command:register: metrics
lifecycle:command:register: print
lifecycle:command:register: remove
lifecycle:command:register: rollback
lifecycle:command:register: rollback:function
lifecycle:command:register: slstats
lifecycle:command:register: plugin
lifecycle:command:register: plugin
lifecycle:command:register: plugin:list
lifecycle:command:register: plugin
lifecycle:command:register: plugin:search
lifecycle:command:register: config
lifecycle:command:register: config:credentials
lifecycle:command:register: aws
lifecycle:command:register: aws:common
lifecycle:command:register: aws:common:validate
lifecycle:command:register: aws:common:cleanupTempDir
lifecycle:command:register: aws:common:moveArtifactsToPackage
lifecycle:command:register: aws:common:moveArtifactsToTemp
lifecycle:command:register: aws
lifecycle:command:register: aws:package
lifecycle:command:register: aws:package:finalize
lifecycle:command:register: aws
lifecycle:command:register: aws:deploy
lifecycle:command:register: aws:deploy:deploy
lifecycle:command:register: aws:deploy:finalize
lifecycle:command:register: aws
lifecycle:command:register: aws:info
lifecycle:command:register: upgrade
lifecycle:command:register: uninstall
lifecycle:command:register: invoke
lifecycle:command:register: invoke:local
lifecycle:command:register: offline
lifecycle:command:register: offline:functionsUpdated
lifecycle:command:register: offline:start
lifecycle:command:register: login
lifecycle:command:register: logout
lifecycle:command:register: generate-event
lifecycle:command:register: test
lifecycle:command:register: dashboard
lifecycle:command:register: output
lifecycle:command:register: output:get
lifecycle:command:register: output:list
lifecycle:command:register: param
lifecycle:command:register: param:get
lifecycle:command:register: param:list
process: resolve CLI input (+ plugins schema)
process: resolve variables
config: Retrieve local config, auth data undefined
config: Retrieve global config, auth data undefined
process: run Serverless instance

Packaging online-library-oss for stage local (us-east-1)

lifecycle:command:invoke: Invoke package
lifecycle:command:invoke:hook: [1] < before:package:cleanup
Packaging
lifecycle:command:invoke:hook: [1] > before:package:cleanup
lifecycle:command:invoke:hook: [2] < package:cleanup
lifecycle:command:invoke: Invoke aws:common:validate
lifecycle:command:invoke:hook: [4]   < aws:common:validate:validate
lifecycle:command:invoke:hook: [4]   > aws:common:validate:validate
lifecycle:command:invoke: Invoke aws:common:cleanupTempDir
lifecycle:command:invoke:hook: [7]   < aws:common:cleanupTempDir:cleanup
lifecycle:command:invoke:hook: [7]   > aws:common:cleanupTempDir:cleanup
lifecycle:command:invoke:hook: [2] > package:cleanup
lifecycle:command:invoke:hook: [11] < package:initialize
lifecycle:command:invoke:hook: [11] > package:initialize
lifecycle:command:invoke:hook: [11] < package:initialize
lifecycle:command:invoke:hook: [11] > package:initialize
lifecycle:command:invoke:hook: [14] < package:setupProviderConfiguration
lifecycle:command:invoke:hook: [14] > package:setupProviderConfiguration
lifecycle:command:invoke:hook: [16] < before:package:createDeploymentArtifacts
Compiling with Typescript...
Using local tsconfig.json - tsconfig.json
Typescript compiled.
lifecycle:command:invoke:hook: [16] > before:package:createDeploymentArtifacts
lifecycle:command:invoke:hook: [17] < package:createDeploymentArtifacts
Excluding development dependencies for function "main"
lifecycle:command:invoke:hook: [17] > package:createDeploymentArtifacts
lifecycle:command:invoke:hook: [18] < after:package:createDeploymentArtifacts
lifecycle:command:invoke:hook: [18] > after:package:createDeploymentArtifacts
lifecycle:command:invoke:hook: [19] < before:package:compileLayers
lifecycle:command:invoke:hook: [19] > before:package:compileLayers
lifecycle:command:invoke:hook: [20] < package:compileLayers
lifecycle:command:invoke:hook: [20] > package:compileLayers
lifecycle:command:invoke:hook: [22] < before:package:compileFunctions
lifecycle:command:invoke:hook: [22] > before:package:compileFunctions
lifecycle:command:invoke:hook: [22] < before:package:compileFunctions
lifecycle:command:invoke:hook: [22] > before:package:compileFunctions
lifecycle:command:invoke:hook: [23] < package:compileFunctions
lifecycle:command:invoke:hook: [23] > package:compileFunctions
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [26] < package:compileEvents
lifecycle:command:invoke:hook: [26] > package:compileEvents
lifecycle:command:invoke:hook: [29] < package:finalize
lifecycle:command:invoke: Invoke aws:package:finalize
lifecycle:command:invoke:hook: [31]   < aws:package:finalize:addExportNameForOutputs
lifecycle:command:invoke:hook: [31]   > aws:package:finalize:addExportNameForOutputs
lifecycle:command:invoke:hook: [34]   < aws:package:finalize:mergeCustomProviderResources
lifecycle:command:invoke:hook: [34]   > aws:package:finalize:mergeCustomProviderResources
lifecycle:command:invoke:hook: [37]   < aws:package:finalize:stripNullPropsFromTemplateResources
lifecycle:command:invoke:hook: [37]   > aws:package:finalize:stripNullPropsFromTemplateResources
lifecycle:command:invoke:hook: [40]   < aws:package:finalize:saveServiceState
lifecycle:command:invoke: Invoke aws:common:moveArtifactsToPackage
lifecycle:command:invoke:hook: [42]     < aws:common:moveArtifactsToPackage:move
lifecycle:command:invoke:hook: [42]     > aws:common:moveArtifactsToPackage:move
lifecycle:command:invoke:hook: [40]   > aws:package:finalize:saveServiceState
lifecycle:command:invoke:hook: [29] > package:finalize
lifecycle:command:invoke:hook: [45] < after:package:finalize
lifecycle:command:invoke:hook: [45] > after:package:finalize

✔ Service packaged (42s)
process: finalize { error: undefined, shouldBeSync: undefined, telemetryData: { outcome: 'success' }, shouldSendTelemetry: false }

1 deprecation found: run 'serverless doctor' for more details
config: Retrieve local config, auth data undefined
config: Retrieve global config, auth data undefined
config: Retrieve local config, auth data undefined
config: Retrieve global config, auth data undefined
config: Retrieve local config, auth data undefined
config: Retrieve global config, auth data undefined
config: Retrieve local config, auth data undefined
config: Retrieve global config, auth data undefined
telemetry: payload { ciName: null,
  isTtyTerminal: true,
  isDockerInstalled: undefined,
  dockerVersion: undefined,
  cliName: 'serverless',
  command: 'package',
  commandOptionNames: [ 'stage', 'verbose', 'debug', 'region', [length]: 4 ],
  dashboard: { userId: null, orgUid: undefined },
  firstLocalInstallationTimestamp: 1699491596,
  frameworkLocalUserId: '4c3caea0-7e9b-11ee-94e6-938d38d206e8',
  installationType: 'local:fallback',
  isAutoUpdateEnabled: false,
  isUsingCompose: false,
  notificationsMode: 'on',
  timestamp: 1743211543059,
  timezone: 'America/New_York',
  triggeredDeprecations: [ 'PACKAGE_PATTERNS', [length]: 1 ],
  versions: { serverless: '3.38.0', '@serverless/dashboard-plugin': '7.2.0' },
  commandDurationMs: 44718,
  hasLocalCredentials: true,
  npmDependencies:
   [ '@codegenie/serverless-express',
     '@nestjs/common',
     '@nestjs/core',
     '@nestjs/passport',
     '@nestjs/platform-express',
     '@types/uuid',
     'reflect-metadata',
     'rimraf',
     'rxjs',
     'supertest',
     'typescript-collections',
     '@nestjs/testing',
     '@types/aws-lambda',
     '@types/express',
     '@types/jest',
     '@types/node',
     'acorn',
     'jest',
     'nodemon',
     'prettier',
     'serverless-offline',
     'serverless-plugin-optimize',
     'serverless-plugin-typescript',
     'serverless-prune-plugin',
     'serverless-stage-manager',
     'ts-jest',
     'ts-loader',
     'ts-node',
     'tsconfig-paths',
     'typescript',
     [length]: 30 ],
  config:
   { configValidationMode: 'warn',
     provider: { name: 'aws', runtime: 'nodejs18.x', stage: 'local', region: 'us-east-1' },
     variableSources: [ 'opt', 'param', 'file', 'self', [length]: 4 ],
     plugins: [ 'serverless-plugin-typescript', 'serverless-plugin-optimize', 'serverless-offline', [length]: 3 ],
     functions: [ { url: false, runtime: 'nodejs18.x', events: [ [Object], [length]: 1 ] }, [length]: 1 ],
     resources: { general: [ [length]: 0 ] },
     paramsCount: 0 },
  isConfigValid: true,
  commandUsage: {} }

[cwardcode]

Hi @mnapoli, I was wondering if there has been any investigation into this issue? I could look into it, but i'm not familiar with the serverless codebase, so wouldn't exactly know where to begin to debug this.

[mnapoli]

@cwardcode sorry no I haven't had time to investigate this.