Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions pages/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -362,7 +362,7 @@ performance from actual user sessions in near real time.

Environments are workspaces where one or multiple teams can work. They are the door between our users in data.all and AWS, that is
why we say that we "link" environments because we link each environment to **ONE** AWS account, in **ONE** specific region.
Under each environment we create other data.all resources, such as datasets, pipelines and notebooks.
Under each environment we create other data.all resources, such as datasets and notebooks.

For the deployment of CloudFormation stacks we call upon a CDK trust policy between the Deployment account and the Environment account.
As for the SDK calls, from the deployment account we assume a certain IAM role in the environment accounts, the pivotRole.
Expand Down Expand Up @@ -396,7 +396,7 @@ manually update the pivot role template. IAM policies cannot be scoped down to n

### CDK bootstrap <a name="cdk"></a>
We need to bootstrap the environment account to provision resources the AWS CDK needs to perform the deployment of
environments, datasets, pipelines and other data.all resources.
environments, datasets, and other data.all resources.

Run the following command with AWS credentials of the environment account:
```bash
Expand Down Expand Up @@ -578,8 +578,9 @@ and underlying access to the data in AWS.

### Pipelines

A Pipeline has a pipeline-Team with UI permissions on the Pipeline
and underlying access to the data in AWS.
> [!WARNING]
> As of v2.7.0 - data.all Pipelines is no longer a supported resource due to the restriction of the CodeCommit service to only existing AWS Customers. To learn more about CodeCommit's availability, please refer to the documentation [here](https://aws.amazon.com/blogs/devops/how-to-migrate-your-aws-codecommit-repository-to-another-git-provider/).


### Dashboards

Expand Down
19 changes: 5 additions & 14 deletions pages/code.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,21 +171,12 @@ The data.all `base.api` package contains the `gql` sub-package to support GraphQ

#### cdkproxy
This package contains the code associated with the deployment of CDK stacks that correspond to data.all resources.
`cdkproxy` is a package that exposes a REST API to run registered cloudformation stacks using AWS CDK. It is deployed as a docker container running on AWS ECS.
`cdkproxy` is a package that runs registered cloudformation stacks using AWS CDK. It is bundled as a docker image and run as a AWS ECS task which is triggered on infastrcutre as code (IaC) operations on data.all (e.g. CRUD of data.all resources).

When a data.all resource is created, the API sends an HTTP request
to the docker service and the code runs the appropriate stack using `cdk` cli.
When an API request is made to create a data.all resource, such as a new dataset, the data.all backend sends a new message to an SQS Queue to asynchronously be read off the queue and start a new cdkproxy ECS task.
The code uses a `cdk` cli wrapper to register infrastructure and manage cdk commands, and runs the appropriate stack using `cdk` cli to deploy the IaC of the respective data.all resource.

These stacks are deployed with the `cdk` cli wrapper
The API itself consists of 4 actions/paths:

- GET / : checks if the server is running
- POST /stack/{stackid} : creates or updates the stack
- DELETE /stack/{stackid} : deletes the stack
- GET /stack/{stackid] : returns stack status

The webserver is running on docker, using Python's [FASTAPI](https://fastapi.tiangolo.com/)
web framework and running using [uvicorn](https://www.uvicorn.org/) ASGI server.
For local data.all deployments, a webserver runs on docker using Python's [FASTAPI](https://fastapi.tiangolo.com/) web framework and [uvicorn](https://www.uvicorn.org/) ASGI server. Subsequnetly, data.all sends POST API Requests to the `cdkproxy` web server to start the data.all infrastructure task.

### core/ <a name="core"></a>
Core contains those functionalities that are indispensable to run data.all. Customization of the core should be limited
Expand Down Expand Up @@ -855,7 +846,7 @@ when `Datasets` or `Dashboards` modules are disabled.
"active": true
},
"datapipelines": {
"active": true
"active": false
},
"datasets": {
"active": false
Expand Down
Loading