og-aws
open-guides ā¢ Updated Sep 20, 2024
Beanstalk
- Elastic Beanstalk consists of a single ZIP file or WAR file (you can include multiple WAR files inside your ZIP file).
- It must not exceed 512 MB and must not include a parent folder or top-level directory (subdirectories are fine).
You can launch a cluster of Multi-container instances in a single-instance or autoscaling Elastic Beanstalk environment using the Elastic Beanstalk console. The single container and Multi-container Docker platforms for Elastic Beanstalk support the use of Docker images stored in a public or private online image repository. You can specify images by name in the
Dockerrun.aws.json
file and save it in the root of your source directory.You can add AWS Elastic Beanstalk configuration files (with the
.ebextensions
extension) to your web application's source code to configure your environment and customize the AWS resources it contains. Configuration files are YAML- or JSON-formatted documents with a .config
file extension that you place in a folder named .ebextensions
and deploy in your application source bundle.For example, you could include a configuration file for setting the load balancer type in
.ebextensions/load-balancer.config
.ECS
EC2 Auto Scaling
- One of its features is a setting that allows Auto Scaling to automatically replace an instance if it becomes unhealthy.
- If you are already using a load balancer, you can use the same health checks for both the load balancer and Auto Scaling.
- You can also send health check signals using the Auto Scaling API, either directly from your instances or from an external monitoring service.
- Auto Scaling can be used as a launch template for your EC2 instances, allowing you to adjust the number of running instances according to your needs.
Ā
Task placement constraint
distinctInstance
Place each task on a different container instance. This task placement constraint can be specified when either running a task or creating a new service.
memberOf
Place tasks on container instances that satisfy an expression. For more information about the expression syntax for constraints, seeĀ Cluster Query Language.
Serverless Application Model (SAM)
A serverless application is a combination of Lambda functions, event sources, and other resources that work together to perform tasks.
AWS SAM templates are an extension of AWS CloudFormation templates, with some additional components that make them easier to work with. To create a Lambda function using an AWS SAM template, the developer can use the
AWS::Serverless::Function
resource type. This resource type can be used to create a Lambda function, an IAM execution role, and event source mappings that trigger the function.Ā
Deploy
The Developer will need to package and then deploy the template. To do this the source code must be available in the same directory or referenced using the ācodeuriā parameter. Then, the Developer can use the āaws cloudformation packageā or āsam packageā commands to prepare the local artifacts (local paths) that your AWS CloudFormation template references.
The command uploads local artifacts, such as source code for an AWS Lambda function or a Swagger file for an AWS API Gateway REST API, to an S3 bucket. The command returns a copy of your template, replacing references to local artifacts with the S3 location where the command uploaded the artifacts.
Once that is complete the template can be deployed using the āaws cloudformation deployā or āsam deployā commands. Therefore, the developer has two options to prepare and then deploy this package:
1. RunĀ
aws cloudformation package
Ā and then aws cloudformation deploy2. RunĀ
sam package
Ā and thenĀ sam deploy
AWS X-Ray
AWS X-Ray is a distributed tracing system that collects data from your applications to enable analysis of the flow of requests across microservices and provides insights into application performance and behavior. X-Ray makes it simple to identify and debug issues related to latency, errors, and faults, and helps you to detect and troubleshoot performance bottlenecks in your applications. It also provides visibility into the performance of your services, including latency, request rates, and error rates.
API gateWay
API Gateway is a fully managed service that makes it easy to create, publish, maintain, monitor, and secure APIs at any scale. It acts as a "front door" for applications to access data, business logic, or functionality from back-end services. It handles tasks such as traffic management, authorization and access control, monitoring, and API version management, and can handle up to hundreds of thousands of concurrent API calls. You can use it to create an API that acts as a "front door" for applications to access data, business logic, or functionality from back-end services, such as workloads running on Amazon EC2, code running on AWS Lambda, or any web application.
HTTP: This type of integration enables an API to expose HTTP endpoints in the backend. It is also known as the HTTP custom integration. You must configure both the integration request and the integration response, as well as set up the necessary data mappings from the method request to the integration request, and from the integration response to the method response.
HTTP_PROXY: The HTTP proxy integration allows clients to access the backend HTTP endpoints with a simplified integration setup on a single API method. You do not need to configure the integration request or the integration response. API Gateway passes the incoming request from the client to the HTTP endpoint and passes the outgoing response from the HTTP endpoint back to the client.
Ā
Lambda
The
/tmp
directory provides 512 MB of storage space that can be used by a function.A Lambda alias is like a pointer to a specific Lambda function version. Users can access the function version using the alias ARN.
Lambda as a plugin system for other AWS services
Lambda is best suited for small, infrequently changing code snippets. We view Lambda functions as part of the infrastructure, rather than part of the application. One of our favorite uses for Lambda is to use it as a plugin system for other AWS services. Here are some examples:
S3
- S3 does not provide an API to resize an image after it is uploaded to a bucket, but Lambda can be used to add this capability.
Application Load Balancer (ALB)
- ALB's API allows for a fixed response for a given route, but it cannot respond with an image. Lambda can be used to enable this.
CloudFront
- CloudFront cannot rewrite a request URL based on request cookies (which is useful for A/B testing), but Lambda can be used to make CloudFront do this with a few lines of code.
CloudWatch
- CloudWatch does not support regex-based alerting on application logs, but this feature can be added with a few lines of Lambda code.
Kinesis
- Kinesis does not provide an API to filter records and write them to DynamoDB, but this can be easily accomplished with Lambda.
CloudFormation
- CloudFormation's native modeling language has many limitations and, for example, it cannot create and validate a new TLS certificate from the AWS Certificate Manager. Lambda can be used to extend the CloudFormation language to add almost any capability.
And so onāyou get the idea. Lambda is a great way to extend existing AWS features.
Limitations
- Cold start
- Limit on code bundle
- Stateless ( Cloud use other services such as DynamoDB and ElasticCache to preserve the states)
Serverless functions or Lambda functions
CloudFormation has two resources for Lambda functions:
AWS::Lambda::Function
and AWS::Serverless::Function
. The latter is extended by SAM and provides additional benefits, such as automatically creating IAM policies and event triggers. If you need more direct control over IAM and trigger invocation, you can use the lower-level resource. However, you will then need to create the security policies yourself.Ā
Ā
An example of triggering an alarm when the number of messages in the Dead Letter Queue (DLQ) exceeds a threshold.
Ā
CDK stack compatible changesĀ
Ā
Ā
Ā
IAM
Ā
AWS APIs are gateways to control the access via the API which is subject to IAM. (IAM protects AWS APIs only).
Ā
Ā
Principal
- IAM users (long lived)
- password-based Console access
- access-key-based (Access Key ID - Secret Access Key pairs)
- IAM roles
- process-based
- configure S3 to replicate a bucketās contents into another bucket
- a Lambda function needs to access other resources
Assume roles can give IAM users temporary credentials
- An IAM policy attached to the role that defines who can assume it.
- Configure a Lambda function to use an IAM Role
Ā
Ā
How to get the temporary rule?
Ā
Ā
Resource
A resource is an AWS entity that serves as theĀ target of an operation.
AnĀ Amazon Resource Name (ARN) is a global identifier for resources inside AWS.
- TheĀ
partition
Ā is usuallyĀaws
, the exceptions are regions in China and the US GovCloud.
- TheĀ
service
Ā is the AWS product, such asĀs3
Ā orĀiam
.
- TheĀ
region
Ā specifies which region the resource is located if any. There are global resources, such as IAM users, that do not have a region.
- TheĀ
account-id
Ā is the 12-digits account number.
- And finally, theĀ
resource-id
Ā is the local identifier of the resource. It can specify sub-resources too, such as the name of objects inside an S3 bucket.
Ā
Ā
Service control policies (SCPs)
As an administrator of the master account of an organization, you can use Service Control Policies (SCPs) to specify the maximum permissions for users and roles in member accounts. SCPs allow you to restrict access to AWS services, resources, and individual API actions. You can also define conditions for when to restrict access to these services, resources, and API actions.
Ā
Ā
Best practices
Best practices
- Use groups to assign permissions to users
- Create standalone policies instead of using inline policies
CloudTrail logging
CloudTrail is a security service that captures what is happening inside the account and across multiple accounts.
- CloudTrail stores logs for 90 days, but itās a best practice to configure it to send them to an S3 bucket for permanent storage. We can configure aĀ trail to do that.
- It can take up to 15 minutes for an event to appear in CloudTrail.
Logs contains:
- event name
- event source
- etc
Ā
IAM Policies
policy behaves is determined byĀ whatĀ it is attached to
- Identity-based policies: Attached to an IAM user, group, or role
- Resource-based policies: Attached to AWS resources
- Service-control policies: Attached to accounts
- Session policy: Attached to assume role sessions
- Permissions boundary: Attached to identities as a boundary
Identity-based policies
Types:
- Customer managed policy
- AWS managed policies
- AdministratorAccess - gives all permissions an identity can have
- PowerUserAccess - except for permissions and account management
- service-based policies - allow access to a single service inside an account
- AmazonS3FullAccess
- AmazonS3ReadOnlyAccess
Resource-based policies
Ā
allow user1 to access the S3 object
Ā
Service control policy (SCP)
purposes:
- restrict which regions an account can use
- a development account does not need to buy reserved EC2 instances
- restrict access, meaning an operation is denied if it does not explicitly allow it.
Ā
Session policy
The identity that uses a role can use the full permission set of that role, but with a session policy, it can choose to use less privilege, restraining itself.
Ā
JSON files
Ā
Each statement can contain
- anĀ Effect (Allow or Deny)
- anĀ Action
s3:PutObject
- aĀ Principal (
"NotPrincipal"
or"Principal"
) "AWS":Ā "<iam>:user/user2"
specific an IAM user"AWS":Ā "<iam>:role/role2"
specific roles"AWS":Ā "*"
anonymous user, which is used to give public access"Service":Ā "lambda.amazonaws.com"
give access to anĀ AWS service
- aĀ Resource (
"NotResource"
or"Resource"
) "arn:aws:s3:::<bucket>"
"arn:aws:s3:::<bucket>/*"
"arn:aws:iam::<accountId>:user/admin/*"
- aĀ Condition
StringLike
StringEquals
Ā
- IfĀ theĀ
access
Ā tag isĀprojectA
,Ā thenĀ the policyĀ matches.
- IfĀ theĀ
access
Ā tag is something else, such asĀprojectB
, *thenthe policyĀ does not match.
- If there is noĀ
access
Ā tag,Ā thenĀ the policyĀ does not match.
Ā
- IfĀ theĀ
access
Ā tag isĀprojectA
,Ā thenĀ the policyĀ matches.
- IfĀ theĀ
access
Ā tag is something else, such asĀprojectB
,Ā thenĀ the policyĀ does not match.
- IfĀ there is noĀ
access
Ā tag,Ā thenĀ the policyĀ matches.
Ā
Example:
Create the Correct S3 Restricted Policies and Roles
Perform the following tasks:
- Create theĀ
S3RestrictedPolicy
Ā IAM policy. Ensure only the appconfig buckets are accessible. - Select the S3 service and all S3 actions
- Select all resources except bucket
- Add the appconfig bucket names to the policy
- Create theĀ
S3RestrictedRole
Ā IAM role. - Set the trusted entity to another AWS account
- Add your account ID
- For permissions, select the S3RestrictedPolicy
- Attach theĀ
S3RestrictedPolicy
Ā to theĀdev1
Ā user.
Ā
Configure IAM So the dev3 User Can Assume the Role
Perform the following tasks:
- Create theĀ
AssumeS3Policy
Ā IAM policy. - Select the STS service
- Select AssumeRole under the write options
- Add the S3RestrictedRole
- Attach theĀ
AssumeS3Policy
Ā to theĀdev3
Ā user.
- Assume theĀ
S3RestrictedRole
Ā as theĀdev3
Ā user. - Log in as the dev3 user
- Switch roles to the S3RestrictedRole
- Verify access in S3
Evaluation flow
VPC (virtual private cloud)
VPCs and subnets
AĀ subnetĀ is a range of IP addresses in your VPC. You can connect a subnet to the internet, other VPCs, and your own data centers, and route traffic to and from your subnets using route tables.
Ā
aĀ default VPC in each Region
aĀ default subnet in each Availability Zone in the Region
Ā
Ā
Access AWS services
Public service endpoints
protocol
://
service_code
.
region_code
.amazonaws.com
AWS PrivateLink
Regional DNS
ndpoint_id
.
service_id
.
region
.vpce.amazonaws.com
zonal DNS
endpoint_id
-
az_name
.
service_id
.
region
.vpce.amazonaws.com
Ā
Amazon EC2 | Amazon Fargate |
You can explicitly provision EC2 instances | The control plan asks for resources and Fargate automatically provisions |
Youāre responsible for upgrading, patching, care of EC2 pool | Fargate provisions compute as needed |
You must handle cluster optimization | Fargate handles customer optimizations |
More granular control over infrastructure | Limited control, as infrastructure is automated |
Ā
Ā
Amazon ECS | Amazon EKS |
Managed, highly available, highly scalable container platform | Managed, highly available, highly scalable container platform |
AWS-specific platform that supports Docker Containers | Compatible with upstream Kubernetes so itās easy to lift and shift from other Kubernetes deployments |
Considered simpler and easier to use | Considered more feature-rich and complex with a steep learning curve |
Leverages AWS services like Route 53, ALB, and CloudWatch | A hosted Kubernetes platform that handles many things internally |
āTasksā are instances of containers that are run on underlying compute but more of less isolated | āPodsā are containers collocated with one another and can have shared access to each other |
Limited extensibility | Extensible via a wide variety of third-party and community add-ons. |
Ā
EBS volumes must be the same available zone as the EC2 instances
Ā
AWS CLI
Potential timeout errors
Using the AWS CLI to list all the files in an S3 bucket containing thousands of files can cause your API call to exceed the maximum allowed time. This can result in a "timed out" error. To prevent this, you can use the
-page-size
option to request a smaller number of items from each call to the AWS service.Ā
RDS is for OLTP work loads not suitable for OLAP
Online Transaction Processing (OLTP)
processes data from transactions in real-time, e.g., customer orders, banking transactions, payments, and booking systems.
OLTP is all about data processing, and computing large numbers of small transactions in real-time.
Online Analytics Processing (OLAP)
Processes complex queries to analyze historical data, e.g., analyzing net profit figures from the past 3 years, and sales forecasting.
OLAP is all about data analysis using large amounts of data, and complex queries that take a long time to complete.
Ā
RDS Multi-AZ (Standby to handle the failure)
Different AZ
RDS read replica (increase or scale read performance)
Same AZ, cross-AZ, cross-region
Ā
Backup and Snapshots
Database Snapshot
Manual, ad-hoc, and user-initiated. It provides a snapshot of the storage volume attached to the DB instance.
Ā
Automated backup (store in S3)
Enabled by default. It creates daily backups or snapshots that run during a backup window that you define. transaction logs are used to replay transactions.
Recover database to any point in the time within a āretention periodā
Recover: need to the snapshot and transaction logs.
retention periods of 1 - 35 days.
Ā
Encrypt unencrypted RDS DB instance
Unencrypted DB ā Snapshot ā Encrypted Snapshot ā Encrypted DB
Ā
Ā
ElastiCache
Memcached
basic object caching
scales horizontally, but no persistence, Multi-AZ or failover
Redis
sophisticated solution
Persistence, Multi-AZ or failover
Support sorting and ranking data
complex data types like lists and hashes
Ā
Compare
CloudWatch
CloudWatch stores data about a metric as a series of data points, each with an associated time stamp. You can also publish an aggregated set of data points, known as a statistic set.
Ā
Metrics can be either:
- Standard resolution, with data having a one-minute granularity
- High resolution, with data at a granularity of one second
- CloudWatch agent - define your own metrics.
- CloudWatch Logs - monitor operating system and application logs
DynamoDB
Amazon DynamoDB supports identity-based policies only
On-demand:
The capacity you get with DynamoDB is not completely on-demand. Behind the scenes, DynamoDB sets a limit on the number of reads and writes per second, and these limits can change depending on your usage. This process is not visible to you, so if you want to make sure you have enough capacity for large fluctuations in usage, you may want to consider using provisioned capacity for added assurance.
Provisioned capacity:
- Requests are evenly distributed throughout the day.
- There is no capacity headroom, meaning that if there were more than one million requests in a day, you would be throttled. Therefore, you must monitor your utilization and proactively provision the necessary capacity.
Primary key
The Partition key is a unique attribute (e.g. user ID) whose value is input to an internal hash function which determines the partition or physical location on which the data is stored. If the Partition key is used as the Primary key, then no two items can have the same Partition key.
A Composite key is a combination of the Partition key and a Sort key. For example, if a user is posting to a forum, the Partition key would be the user ID and the Sort key would be the timestamp of the post. Two items may have the same Partition key, but they must have a different Sort key. All items with the same Partition key are stored together, then sorted according to the Sort key value, allowing multiple items with the same partition key to be stored.
Indexes
Local indexes
- Theyāre immediately consistent
- Donāt allows a table to keep growing indefinitely goes away.
- All the records that share the same partition key need to fit in 10 GB
Ā
Global indexes
- Donāt constrain your table size
- Eventually consistent
- Much less worrisome than that of local indexes
- The provisioned throughput of a global index happens to be insufficient to keep up with updates on the main table, then that queue can get full. When that happens, disaster strikes: All write operations on the main table start failing.
Ā
RCUs/WCUs
RCUs and WCUs are units of throughput for read and write operations on DynamoDB. RCUs (read capacity units) measure the throughput for reads, and WCUs (write capacity units) measure the throughput for writes. Provisioning more RCUs and WCUs gives you more read and write throughput.
For example, a strongly consistent read of an 8 KB item requires two RCUs (8/4), an eventually consistent read requires one RCU (8/8), and a transactional read requires four RCUs(8/2) .
For example, a standard write request of a 1 KB item requires one WCU, a standard write request of a 3 KB item requires three WCUs, and a transactional write request of a 3 KB item requires six WCUs.
Ā
S3
Benefit
- Highly durable
- Very easy to use
- Infinite bandwidth
- Infinite storage space
- Zero capacity management
- Set replication for a S3 bucket
Ā
Limitations of S3
- You cannot append to objects.
- If you have rapidly changing data (e.g. log files), you must buffer updates and periodically flush them to S3 as new objects. This reduces data durability, as the buffered data is typically stored in a single place without replication.
- A solution to this issue is to buffer data in a durable queue, such as SQS or Kinesis Streams.
ELB
Ā
Elastic Load Balancer
- Application Load Balancer
- HTTP and HTTPS
- at Layer 7
- handle multiple domains (via host-based routing)
- proper reverse proxies that sit between the internet and your application
- Network Load Balancer
- TCP and high performance
- at layer 4
- most expensive option
- TCP passthrough (Only support in the NLB rather than ALB)
- load balancers
- Classic Load Balancer (Legacy)
- HTTP/HTTPS and TCP
- Gateway Load Balancer
- Third party virtual appliance (AWS marketplace, virtual firewalls)
- X-Forwarded-For
Error 504 Gateway timeout
- couldnāt build connection to server
- application has issues
Support for TLS/HTTPS
Both ALBs and NLBs support TLS/HTTPS, and they integrate very well with the AWS Certificate Manager. This lets you set up TLS certificates and forget about them. The certificates get renewed automatically and deployed to your load balancers without any downtime. And all the certificates are free.
To have end-to-end TLS from the caller to your application, you will also have to enable TLS on your application. Otherwise, the traffic from the load balancer to your application will travel unencrypted on part of the network. Unfortunately, certificates from the Certificate Manager cannot be exported, so you canāt use them for your application. Instead, common practice is to create a self-signed certificate on your host and use that for your application. The load balancers do not validate the serverās certificate (neither the name, nor the expiry), so in this case, a self-signed certificate works fine.
Ā
Ā
CloudFormation
Deploy a Lambda function using CloudFormation:
- Zip the function codes and upload them to S3, then add a reference to it in an AWS:Lambda:Function resource in the template.
- Create an AWS:Lambda:Function resource and write the Lambda function codes inline.
Kinesis
Similar to SQS, it is a highly-durable linked list. Much cheaper than SQS.
Kinesis vs SQS
- SQS can only have one consumer, while Kinesis can have many.
- Once an SQS message gets consumed, it gets deleted from the queue. But Kinesis records get added to a list in a stable order, and any number of consumers can read a copy of the stream by keeping a cursor over this never-ending list.
- Multiple consumers donāt affect each other, and if one falls behind, it doesnāt slow down the other consumers.
- Whenever consumers read data out of Kinesis, they will always get their records in the same order.
Ā
On the other hand, Kinesis is not as user-friendly as SQS. With SQS, you can enqueue an unlimited number of messages without worrying about capacity or throttling limits. However, a Kinesis stream is composed of shards, and it is your responsibility to determine the number of shards needed, monitor shard utilization, add shards, and route records to shards in a way that ensures an equal amount of traffic. Unfortunately, this is a significant operational burden.
Ā
Ā
KMS
The
GenerateDataKey
API is used with KMS services to generate a unique symmetric data key. This operation returns two copies of the data key: one plaintext and one encrypted under a customer master key (CMK) specified by you. The plaintext key can be used to encrypt data outside of AWS KMS, and the encrypted data key can be stored with the encrypted data.STS
The AWS STS decode-authorization-message API decodes extra information about the authorization status of a request from an encoded message returned in response to an AWS request. The output is then decoded into a more human-readable format that can be viewed in a JSON editor.
Cognito
Amazon Cognito identity pools (federated identities) enable you to create unique identities for your users and federate them with identity providers. An identity pool allows you to obtain temporary AWS credentials with limited privileges to access other AWS services.
Amazon Cognito allows you to store end user data in datasets containing key-value pairs. This data is associated with an Amazon Cognito identity, so it can be accessed across logins and devices. To sync this data between the Amazon Cognito service and an end user's devices, invoke the
synchronize
method. Each dataset can have a maximum size of 1 MB, and you can associate up to 20 datasets with an identity.The Amazon Cognito Sync client creates a local cache for the identity data. Your app interacts with this local cache when it reads and writes keys. This ensures that all changes made on the device are immediately available on the device, even when offline. When the
synchronize
method is called, changes from the service are pulled to the device, and any local changes are pushed to the service. At this point, the changes are available to other devices for synchronization.Ā
CloudFront/Global Accelerator
Origin Protocol Policy
CloudFront connects to your origin using HTTP, HTTPS, or the same protocol as the viewer.
CloudFront improves performance for both cacheable content (such as images and videos) and dynamic content (such as API acceleration and dynamic site delivery).
Global Accelerator improves performance for a wide range of applications over TCP or UDP by proxying packets at the edge to applications running in one or more AWS Regions. Global Accelerator is a good fit for non-HTTP use cases, such as gaming (UDP), loT (MQTT), or Voice over IP, as well as for HTTP use cases that specifically require static IP addresses or deterministic, fast regional failover.
Both services integrate with AWS Shield for DDoS protection.
Viewer Protocol Policy
- HTTP and HTTPS
- Redirect HTTP to HTTPS
- HTTPS Only
Ā
Appsync
This service enable us to use graphQL api and do sub-pub mechanise.
- Use
graphQL
backed by kinds of data sources like DynamoDB and Lambda.
- Use with
Cognito
to do user authorization
Ā