AWS EBS & EFS

  • An EBS(Elastic Block Store) volume is a network drive, we can attach to our instances while they run.
  • It allows our instances to persist data, even after termination.
  • They can be attached/mounted to one instance at a time. However, some EBS have multi attach feature.
  • It is bound to a specific availability zone.
  • They are a sort of network USB stick.
  • It is a network drive
    • It uses network to communicate with the instance, so there might be a bit of latency.
    • It can be detached from an EC2, instance and Attached to another one Quickly.
  • It is locked to an availability zone.
    • An EBS volume in us-east-la Cannot be attached to us-east-lb.
    • To move a volume across, We first need to snapshot it.
  • We have to provision capacity in Advance(Size in GB, and IOPS).
    • We get billed for all the provisioned capacity.
    • We can increase the capacity of drive overtime.
  • EBS – delete on termination attribute
    • We set this attribute during EBS creation.
    • Controls the EBS behaviour When our EC2 instance Is terminated.
    • By default, it is set for root volume and unset for any EBS volume attached to instance.
    • By default, the root EBS volume is deleted(attribute is enabled).
    • By default, any other attached, EBS volume is not deleted(Attribute is disabled)
    • We can control by enabling/disabling delete on termination for EBS volumes.
    • We can disable delete on termination for root volume thus When instance is terminated, volume will not be deleted.
  • In the storage tab, of our instance, we can see the volumes attached to our instance.
  • By default, root volume is attached to our instance.
  • In the volume’s interface of AWS, we can create a new volume with following properties
    • Volume type
    • Size
    • IOPS
    • Availability zone(Make sure it is in same as of instance).
  • Once volume is created, from “Actions” attach volume to your instance.
  • Now in storage, we will see two devices attached to your instance.
  • We can use EBS volume through Linux as Configured in the following link.
  • Two non-root volumes attached to an Instance have to be in different availability zone.
    • A volume attached to an instance has to be in the same availability zone as of instance.
    • So we can only attach one root volume and one non-root EBS volume to an instance.
  • EBS Snapshots
    • An EBS snapshot Is a backup of our EBS volume at a point in time.
    • We do not need to detach volume to do snapshot, but it is recommended.
    • We can copy EBS snapshots across availability zones or regions.
      • We can move copy of our instance, volume, restore it and attach it to an instance in a different availability zone or Region.
    • EBS snapshots features
      • EBS snapshot archive
        • Move a snapshot to an “archive tier”That is 75% cheaper.
        • Takes Within 24 to 72 hours For restoring the archive.
      • Recycle bin for EBS snapshots
        • Setup rules to retain deleted Snapshots so you can recover them after an accidental deletion.
        • Specify retention (From one day to 1 year)
      • Fast snapshot restore(FSR)
        • Forceful initialisation of snapshot to have no see on the first use.
    • We can create a snapshot from actions on the EBS dashboard with following properties
      • Description
    • On the left hand side menu on EBS dashboard, click snapshots to list all your snapshot.
    • We can copy a snapshot into another region in Destination
      • This helps in disaster recovery strategy to ensure data is backed up in AWS in another region.
    • We can recreate a volume for the snapshot. From action menu.
    • In snapshots, we also have a recycle bin
      • Used to protect our EBS snapshots and Amazon machine images(AMI’s) From accidental deletion.
      • We create retention rule in recycle bin with following properties
        • Name
        • Resource type(EBS snapshots, Amazon Machine Image(AMI))
        • Retention period
        • Rule lock settings
        • Tags
      • If we delete our snapshots, it will show in recycle bin and we can recover them from there.
  • AMI
    • AMI stands for Amazon Machine Image.
    • AMI are customization of an EC2 instance.
    • You add your own software, configuration, operating system, monitoring.
      • Faster boot/configuration time because all your software is prepackaged.
    • AMI are built for a specific region(And can be copied across regions).
    • We can launch EC2 instances from
      • A public AMI Which are AWS provided.
        • For example, Amazon AWS linux2 AMI.
      • We can create our own AMI Which we can make and maintain them ourselves.
      • An AWS marketplace AMI, which someone else has made and sells in marketplace.
    • Creating AMI from EC2 instance
      • Start an EC2 instance and customise it.
      • Stop the instance(For data integrity)
      • Build an AMI, this will also create EBS snapshots.
        • Right, click on instance, and click on “image and template”, and then create image.
        • Add the following properties
          • Image name
          • Volume type
          • Size of volume
          • Delete on termination
      • In left hand side, after creation, we click on AMI under images and we will see our AMI listed.
        • Wait for the status to be available before use.
      • Launch instances from own/other AMI’s.
  • EC2 Instance store
    • EBS volumes, are network device with good but limited performance.
    • If we need a high-performance hardware disk, we should use EC2 instance store.
    • These disk are a part of physical servers, which run EC2 instances.
    • A special type of EC2, instance leverages EC2 store, which is the hard drive attached to a server.
      • It provides better input/output performance.
      • It provides good throughput.
    • EC2 instance store Loses its storage if they are stopped. These are thus called as ephemeral storage.
    • These are good for buffer/cache/Scratch data/temporary content.
      • For long-term storage, EBS is a good use case.
    • Risk of data loss If Hardware fails
      • Back ups and Replication  are users responsibility.
  • EBS volume types
    • EBS volumes come in six types
      • GP 2/GP3(General Purpose SSD)
        • General purpose SSD volume that Balances price and performance For a wide variety of workloads.
      • IO1/IO2 Block Express(SSD)
        • Highest performance, SSD volume for Mission critical low latency or high throughput Workloads
      • ST1(HDD)
        • Low cost, HDD volume designed for frequently accessed, Throughput intensive workload.
      • SC1(HDD)
        • Lowest cost, HDD volume designed for less frequently accessed Workloads.
    • EBS volumes are characterised in size/throughput/IOPS(I/O operations per second)
    • Only GP2/GP3 And IO1/IO2 Block express can be used as boot volumes.
    • EBS volume types, use cases
      • General purpose SSD
        • Cost effective storage, low latency
        • System boot volumes, virtual desktops , Development and test environments
        • 1GB – 16 TB
          • GP3
            • Base line of 3000 IOPS and throughput of 125MB per second.
            • Can increase IOPS up to 16,000 and throughput upto 1000 MB/s independently.
          • GP2
            • Small GP2 volumes can burst IOPS up to 3000.
            • Size of the volumes and IOPS are linked, Maximum IOPS is 16,000.
            • 3 IOPS per GB, means at 5334 GB we are at max IOPS.
      • Provisioned IOPS(PIOPS) SSD
        • Used for critical business applications with sustained IOPS performance.
          • Applications that need More than 16,000 IOPS.
        • Great for databases, workloads(Sensitive storage, performance, and consistency)
          • IO1(4GB – 16 TB)
            • Maximum PIOPS: 64,000 for Nitro EC2 instances and 32,000 for other.
            • Can increase PIOPS independently, from storage size.
          • IO2(Block express 4GB – 64TB)
            • Sub milliseconds Latency
            • Max PIOPS 256,000 With an IOPS GIB ratio of 1000:1
        • Support EBS Multi Attach
      • Hard disk Drives(HDD)
        • Cannot be a boot volume.
        • 125GB to 16 TB
        • Throughput optimised HDD(st1)
          • Big data, Data warehousing, Log processing
          • Max throughput 500 MiB/s-max IOPS 500
        • Cold HDD(sc1)
          • For data that is infrequently accessed.
          • Scenarios where lowest cost is important.
          • Maximum through put is 250 MB/s-max IOPS 250.
    • https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volume-types.html
  • EBS multi attach(io1/io2 family)
    • Attach the same EBS volume to multiple EC2 instances in the same availability zone.
    • This feature is only available in one availability zone. This feature is not available between availability zones.
    • Each instance has full read and write permissions To the high performance volume.
    • Use case
      • Achieve higher application availability in clustered Linux applications, example Tera data.
      • Application must manage concurred write operations.
    • It can be attached to up to 16 EC2 instances At a time.
    • We must use a file system, that’s cluster aware(Not XFS,EXT4 etc)
  • It can be attached to one instance, only except in case of multi attach IO1/IO2.
  • EBS volumes are locked at the availability zone(AZ) Level.
    • GP2: IO increases if the Disk size increases.
    • GP3/IO1: Can increase IO independently
  • To migrate an EBS volume across availability zone
    • Take a snapshot
    • Restore snapshot To another availability zone.
  • Amazon EFS( Elastic File System)
    • Elastic file system is a managed. NFS(Network file system) That can be mounted on many EC2.
    • EFS has higher price point than EBS.
    • EFS works with E2 instances in multi Availability zone.
    • It is highly available, scalable, expensive that is may have cost three times of GP2 and is pay per use.
    • EFS can leverage storage tiers For cost savings.
    • Using the security group of EFS, multiple EC2 instances can connect at the same time.
    • Use cases
      • Content management, Web Serving, Data sharing, WordPress.
    • Uses NFS version 4.1 protocol.
    • Uses security group to control access to EFS.
    • Compatible with Linux based AMI(Not Windows).
    • Encryption at rest Using KMS.
    • POSIX File system(Standard file system of Linux) That has a standard file API.
    • File system Scales Automatically, pay per use No capacity planning.
    • EFS performance and storage classes
      • EFS scale
        • Thousand’s of concurrent NFS clients, 10 GB per second throughput.
        • Grow to Petabyte Scale network, File systems automatically.
      • Performance mode(set at EFS creation time)
        • General purpose(Default): It is latency sensitive, use cases are web server, CMS et cetera.
        • Maximum I/O: Higher latency, Throughput, highly parallel(bigdata, Media processing)
        • Throughput mode:
          • Bursting-One TB Is equals to 50 MB per second + burst of up to 100 MB/s
          • Provisioned - Set your throughput regardless of storage, size, example 1GB/s For one TB Storage.
          • Elastic-Automatically scales Throughput Up or down based on your work loads.
          • Up to 3GB/s for reads and 1GB/s For writes.
          • Used for unpredictable workloads.
      • EFS-Storage classes
        • Storage Tiers(Life cycle management feature-Move file after N days)
          • Standard
            • For frequently access files.
          • In frequent access(EFS-1A)
            • Cost to retrieve files, Lower price to store.
          • Archive
            • Rarely accessed data(Few times each year)
            • 50% cheaper
        • Implement life cycle, Policies to move files between storage tiers.
        • Availability and durability
          • Standard
            • Multi availability zone, great for production
          • One zone
            • One availability zone, great for development, backup enabled by default, compatible with IA(EFS one zone - IA)
        • Over 90% in Cost saving if we use right storage classes.
    • Create a file system with following properties
      • Name
      • Virtual private cloud
      • Next, we have some customisable properties, which are optional
        • File system settings
          • File system type
            • Regional
              • Good for production environments
            • One zone
              • Choose one Availability Zone
              • Good for development environments
              • If availability zone becomes unavailable, then our EFS won’t be available.
          • Enable automatic backups
          • Life cycle management(Duration)
            • Duration to transition to in frequent access(1A)
            • Duration to transition to archive.
            • Duration to transition into standard.
          • Encryption
          • Performance settings
            • Throughput mode
              • Enhanced
                • Elastic(Recommended)
                  • Use this mode for workload with unpredictable input/output.
                  • With elastic mode, your throughput scale automatically, and you only pay for what you use.
                • Provisioned
                  • Use this mode, if you can estimate your workloads throughput requirements.
                  • With provision mode, you configure your file systems throughput and Pay for throughput provisioned.
              • Bursting
                • Provides through put that skills with the amount of storage for work loads with basic performance requirements.
            • Performance mode.
              • General purpose(Recommended)
                • Ideal for a variety of diverse workload, including high-performance and latency sensitive applications.
              • Max I/O
                • Designed for highly parallelised workloads That can tolerate higher latencies.
                • Good for big-data settings
        • Network Access Settings
          • VPC
          • Mount targets
            • Multi availability zone only comes when we have selected regional type of file system.
            • SubnetId 
            • IP address
            • Specify a security group
        • File system policy
          • A policy for the file system, which defines its access and permissions by default, for example
            • Prevent root access by default
            • Enforce, read only access by default
            • Prevent Anonymous access
            • Enforce in transit, encryption for all clients
            • Any other custom policies.
        • Review and create
          • Review settings and click on, create to create EFS.
    • Best setting for file system is enhanced throughput Mode with elastic and General purpose performance mode.
    • Once created, the initial size will show about 6KiB Since we only pay for the storage, we use.
    • Create an instance, and while creating add EFS file system, make sure you have selected subnet before adding The same.
      • Select your shared file system and its mount point.
      • We can automatically create and attach security groups
      • We can automatically mount shared file system by attaching required user data script.
      • Select number of instances and launch it.
    • Create another instance, in another availability zone and attach file system to it.
    • In EFS dashboard. Now in network tab, we see each availability zone is mapped to multiple security groups.
    • Also, in EC2 Instance in security groups in inbound rules, it will allow NFS on port 2049 with source security group of other EC2 instance.
    • Connect to EC2 instance, console and Check the mount directory of EFS file system.
      • We can use it now for creating files.
    • Create a file from first instance and access it from the second instance.
    • For cleanup terminate all the instances and delete EFS file system and delete security groups.
  • Instance Store
    • It is physically attached to the EC2 Instance.
    • If we loose EC2 instance We loose Storage as well.
  • Cleanup
    • Delete file systems created.
    • Terminate all EC2 instances
    • Delete volumes
    • Delete snapshots
    • Delete security groups.

Comments

Post a Comment

Popular posts from this blog

Infrastructure Setup using Cloud Formation Templates

Services Required to create CICD Code Build Build code and deploy image in jfrog Code Pipeline Has (Continuous Integration Continuous Deployment)CICD flow Cloud Formation To deploy and create infrastructure we use yml files. S3 Store our data packages and yml files which are used by CFT. ECS To deploy our application on fargate servers We can also create our manual EC2 instances from pipeline. Secrets Manager Manages all the secrets. ELB/ALB Elastic/Application Load Balancer To map context paths of different services GitHub Stores our code Jfrog Used to store all the docker images Dockerfile Used to create a customised docker image Scanning for Vulnerabilities free code Sonarcube Checkmarx Twistlock Cloudwatch Used to store logs We create log group for each of instances Lambda Functions Used for serverless technologies. Creating a new Environment Create the YML images of the different resources required in the environment. The CFT templates are stored in S3 or can be uploaded from syst...
Read more

VPC,Cloud Front,EC2,RDS in AWS

 VPC allows you to provision a logically isolated section of AWS where we can launch AWS resources in a virtual network. Cloud Front is a Fast Highly Secure and Programmable Content Delivery Network EC2 provides compute resources  RDS is Amazon's Relationship Database System.
Read more

High Availability Architecture

Following points can be considered while creating a highly available architecture depending on scenarios  A spread Placement group Supports a maximum of seven running Instances per availability zone.for example, In A region with three Availability zones, you can run total of 21 instances in the group.If you try to start an eighth Instance in the same availability zone and in the same spread placement group, the Instance will not launch.If you need to have more than seven instances in an availability zone,Then the Recommendation is to use multiple spread Placement Groups. Using multiple spread Placement Groups does not provide guarantee about the spread of instance between groups, but it does ensure the spread for each group, thus limiting impact from certain classes of failures. Durability refers to the ongoing existence of the object or resource. Note that it does not mean you can access it, only that it continues to exist. Groups, Scaling options, configuration templates are comp...
Read more