AWS CLI

 AWS CLI Pagination

  • You can control the number of items included in the output when you run a CLI command.
  • By default, the AWS CLI uses a page size of 1000.
  • i.e. If you run “AWS S3 api list–objects my-bucket” on a bucket which contains 2500 objects, the CLI actually makes 3 API calls to S3 but displays the entire output in one go.
AWS CLI Pagination- Errors
  • If you see errors when running list commands on a large number of resources, the default page size of 1000 might be too high.
  • You are most likely to see a “timed out” error, because the API call has exceeded the maximum allowed time to fetch the required results.
Fixing AWS CLI Pagination- Errors
  • To fix this, use the  - -page-size option to have the CLI request a smaller number of items from each call.
  • The CLI still retrieves the full list, but performs a large number of API calls in the background, and retrieves a smaller number of items with each call.
  • The commands are as follows
    • aws s3 api list-objects - - bucket my-bucket - - pagesize 100.
    • - - max-items option to return fewer items in the CLI output.
    • aws s3 api list-objects - - bucket my-bucket - - max-items 100

Comments

Post a Comment

Popular posts from this blog

Effect : Deny vs No Action

No action allows us to provide granular level control over resources operations. For example, if we use effect, deny For all actions in IAM An, then we use effect allow over some actions in IAM then deniable supersede allow. The way to overcome above issue is to use NoAction on all IAM operations And action on Allowed IAM operations. Example { "Version" : "2012-10-17" , "Statement" : [ { "Effect" : "Allow" , "NotAction" : [ "iam:*" , "organizations:*" , "account:*" ], "Resource" : "*" }, { "Effect" : "Allow" , "Action" : [ "iam:CreateServiceLinkedRole" , "iam:DeleteServiceLinkedRole" , "iam:ListRoles" , "organizations:DescribeOrganization" , "account:ListRegions" ,
Read more

AWS Summaries

Database Summary  RDS(OLTP) SQL My SQL Post Gre SQL Oracle Aurora Maria DB Dynamo DB(No SQL) Redshift OLAP Elastic cache Memcached Redis RDS runs on virtual machines We cannot log into this operating systems however Patching of our RDS Operating system and DB is Amazons responsibility. RDS is not server-less. Aurora surverless is seven less which is an exception. Dynamo DB is server-less. There are two types of backups for RDS Automated backups Database snapshots Read replicas Can be multi availability zones. Use to increase performance Must have backups turned on. Can be in different regions. Can be MySQL,PostgreSQL,Maria DB, Oracle, Aurora, SQL Server is not supported. Can be promoted to master, this will break read replica. Multi availability zone Used for DRG only and not performance. You can force a fail over from one availability zone to another by re-booting the RDS instance. Encryption at Rest Is supported for My SQL, Oracle, SQL server, Postgre SQL, Maria DB and Aurora. Encryp
Read more

Infrastructure Setup using Cloud Formation Templates

Services Required to create CICD Code Build Build code and deploy image in jfrog Code Pipeline Has (Continuous Integration Continuous Deployment)CICD flow Cloud Formation To deploy and create infrastructure we use yml files. S3 Store our data packages and yml files which are used by CFT. ECS To deploy our application on fargate servers We can also create our manual EC2 instances from pipeline. Secrets Manager Manages all the secrets. ELB/ALB Elastic/Application Load Balancer To map context paths of different services GitHub Stores our code Jfrog Used to store all the docker images Dockerfile Used to create a customised docker image Scanning for Vulnerabilities free code Sonarcube Checkmarx Twistlock Cloudwatch Used to store logs We create log group for each of instances Lambda Functions Used for serverless technologies. Creating a new Environment Create the YML images of the different resources required in the environment. The CFT templates are stored in S3 or can be uploaded from syst
Read more