Code Cheatsheets
  • Code Cheatsheets
  • Chrome Shortcuts
  • General Debugging
  • Google Advanced
  • GraphQL
  • General Mac Shortcuts
  • markdown
  • Regular Expression
  • Tech to Try
  • Typora Shortcuts
  • All
    • Elm
    • Headless CMS
    • Java
    • Swift
    • Solana
    • Solidity
    • APIS
      • Chatwoot
      • Daily
      • DuckDuckGo
      • Facebook Login
      • Google APIS
      • Intercom
      • Mux
      • OpenAI
      • Reddit API
      • Segment
      • Sendgrid
      • Sentry (Debug In Prod)
      • Spotify
      • Stripe
      • Transloadit
      • Zendesk
    • Bash
      • IO in Bash
      • Kills
      • Makefile
      • awk
      • Bash_Profile
      • Cron
      • Curl
      • Diff
      • Emacs
      • Find
      • general
      • gGit
      • Grep
      • man
      • Editing File Metadata
      • Sed
      • Secure File Transfer Protocol
      • SSH
      • Vi
      • zip
      • ZSH
      • installed
        • ASDF
        • Bit
        • ffmpeg
        • Generact
        • Graphite
        • ngrok
        • Node Version Manager
        • Spell
        • Tmux
    • C++
      • ptrs
    • CS_Fundamentals
      • Authentication
      • Comparisons
      • Semaphore
      • Content Security Policy
      • Data Structures
      • Editors Comparison
      • Email
      • Encryption
      • Errors
      • Feature Flags
      • HTTP
      • IDs
      • Language Types
      • Microservices
      • Optimization
      • Password Hashing
      • Real Time Messaging Protocol
      • Real-Time
      • Sorting
      • StoringSecrets
      • Testing
      • WebRTC
      • Webhook
    • README
      • Advanced (Postgres?)
      • Database Desigm
      • Firebase
      • Hosts
      • Code Abstractions
      • General Mongo
      • Mongo Aggregation
      • Opensearch
      • Postgres
      • Setup
      • Redis
      • Tips
      • SQL
        • Aggregate
        • Foreign Key Constraints
        • JOINS
        • Optimizations
        • SQL Lite
    • Dev_Apps
      • Featured Shortcuts
      • Iterm2
      • Matlab/Octave
      • Notion
      • Retool
      • Slack
      • Tableplus
      • vimium
      • Featured Shortcuts
      • Warp
      • Windows Terminal
    • Docker
      • Build and Run
      • Docker-compose
      • Dockerfile
      • Multistage builds
      • Swarm
    • Elixir
      • OTP
      • Config
      • DateTime
      • Erlang
      • Errors
      • File
      • Functions
      • Math
      • Modules
      • Regex
      • Setup
      • Time
      • URI
      • Phoenix
        • Channels
        • Handling Controller Errors
        • Controllers
        • Deployment
        • Default File Structure
        • LiveView
        • Plugs
        • Presence
        • Router
        • Session
        • Setup
        • Testing
        • Phoenix.Token
        • Views
        • Workers
        • Ecto
          • ChangeSets and validations
          • Generators
          • Insert/Update
          • Many to Many
          • Migration
          • Query With Repo
          • Relations
          • Schema
        • packages
          • CorsPlug
          • ExRated
          • Ueberauth
      • Collections
        • Lists(linked)
        • Map
        • Set
        • String
        • Structs
        • Tuples
      • Packages
        • Genstage
        • Geolix
        • json
        • Logger
        • Nanoid
        • Oban
        • Redix
        • Sentry
        • Stripe
        • Swoosh
        • Timex
        • http
          • HTTP
          • HTTPoison
          • Tesla
      • Processes / Concurrency
        • Agent
        • Cron
        • GenServer (Generic Server)
        • Registry
        • Supervisors
        • Tasks
    • README
      • IO
      • JSON
      • assert
      • basics
      • control
      • dep
      • errors
      • files
      • func
      • GoRoutines
      • interfaces
      • math
      • request
      • server
      • structs
      • testing
      • time
      • README
        • arrays
        • maps
        • strings
    • HTML Basics
      • Forms
      • Fun
      • SEO
      • Favicon
      • Images
      • Optimizations
      • Scripts
      • overview
        • Handlesbars
        • jade&pug
        • mustache
    • PHP
      • AMP
      • Hack
      • requests
      • setup
    • Python
      • 2.7v3.5
      • advanced
      • Argparse
      • Async Options
      • Classes
      • Secrets
      • environments
      • Importing
      • JSON
      • Os
      • random
      • re
      • testing
      • Datetime
      • Django
        • Databases
        • Models
        • Router
        • Serializers
        • Setup
        • Views
        • packages
          • Secrets
      • General Iterables
        • Dictionaries
        • fancyDicts
        • Heap
        • List
        • Set
        • Strings
      • Flask
        • Blueprint
        • cli
        • Config
        • Deploying Options
        • flash
        • LoginManager
        • packages
        • Requests/Routing
        • RESPONSES
        • templating
        • packages
          • Cors
          • flask_mongoengine
          • Socket IO
        • Postgres in Flask
          • Flask-Migrate
          • Flask-SQLAlchemy
      • packages
        • AIOHTTP
        • Celery
        • cython
        • Environment Variables
        • Excel
        • Google
        • Jupyter Notebook
        • Langchain
        • matplotlib
        • Numpy
        • Pandas
        • PassLib
        • PIL
        • Python Reddit API Wrapper(PRAW)
        • Prisma*
        • Pydub
        • Pymongo
        • Pyppetter
        • Python Requests
        • SQL Alchemy
        • aws
          • AWS Boto3
          • Dynamo
    • README
      • class
      • collections
      • exceptions
      • functions
      • futures
      • guice
      • loops
      • objects
      • options
      • packages
      • pattern&conditional
      • performance&style
      • sbt(running)
      • Strings
      • subtyping
      • traits
  • CSS
    • Obscure
    • Animations
    • Background
    • border
    • colors
    • cursors
    • display
    • Fonts
    • gradients
    • Lists
    • Media Queries
    • Object-fit
    • Overflow
    • pseudoElements
    • Selectors Master Race
    • Transforms
    • Transitions
    • extras
      • Scss
    • Placing Common Tasks
      • HTML Content Box
      • columns
      • Flexbox
      • Grid
      • Position
      • tables
    • Tailwind
      • Animation
      • general
      • Media Queries
      • position
      • Setup
      • Size and Spacing
      • Text
      • Typography
  • Deployment Solutions
    • DNS
    • Digital Ocean
    • Cloudflare
    • Heroku
    • loadtesting
    • Nginx
    • Process Management
    • Redis
    • Socket Alternatives
    • Speed Optimizations
    • {Surge
    • Task Queues
    • Terraform
    • Vercel
    • AWS
      • Certificate Manager
      • Amplify
      • \Cloudfront
      • Cloudwatch
      • Codebuild
      • Codedeploy
      • Code Pipeline
      • Docker in AWS
      • EB
      • ELB (Elastic Load Balancing)
      • IAM roles
      • AWS Lambda (Serverless)
      • Simple Email Service
      • Simple Queue Service
      • Virtual Private Cloud (VPC)
      • Serverless
      • Zappa
      • EC2
        • Scripts
        • Types of EC2
        • EC2(Elastic Compute)
      • Storage
        • Aurora
        • Autoscaling
        • CustomEC2
        • DynamoDB
        • RDS(Relational DB Service)
        • S3(Simple Storage Service)
    • Cloud GPUS
      • Beam
      • Paperspace
      • Replicate
    • Google Cloud Platform
      • Compute
      • Network
    • Github
    • Render
  • Javascript
    • Async and Await
    • Closure
    • Dates
    • Debugging
    • Errors
    • ES6 Specific Features
    • Functions
    • Importing
    • Promises
    • regex
    • Making a request
    • Timeout
    • All
      • Bun
      • Angular
        • HTML
        • HTTP
        • bash
        • input
        • observable_Subject
        • router
        • typescript
      • Chrome Plugins
        • Browser Action
        • Config
        • Content Scripts
        • Context Menu
        • Deploying
        • Desktop Capture
        • Events
        • To Get Identity
        • Manifest
        • Messages
        • Notifications
        • Popup
        • Service Workers
        • Setup
        • Shortcuts / Commands
        • Chrome Storage
        • Tabs
        • Match Patterns
      • Electron
        • Deep Links
        • Deploying
        • Desktop Captuer
        • Dialog
        • Images
        • Menu
        • Messages
        • Differences Between Platforms
        • notarizing
        • Notifications
        • Renderer Process
        • screen
        • Storage Options
        • Tray
        • Windows
        • Electron Packages
          • Electron builder
          • Electron-is-Dev
          • Electron-Log
          • electron-store
          • Electron-Forge
          • Sentry
      • Node
        • Routing
        • ejs
        • Env
        • Errors in Express
        • Files*
        • Forms
        • Logging
        • Middleware
        • Req
        • require
        • Response
        • Setup for Nodejs/Express
        • db
          • Mongoose
          • IoRedis
          • Knex
          • MongoDB
          • Prisma
        • Packages
          • Commander
          • Config
          • Crypto
          • Dotenv
          • Grant
          • Helmet
          • Http-errors
          • Essential Middleware
          • Passport
          • puppeteer
          • Expressjs/session
    • bundlers
      • Parcel
      • Vite
      • Webpack
    • Collections
      • JSON
      • Array
      • Blob (Binary Large Object)
      • Classes
      • Map
      • Objects
      • Set
      • Strings
    • Packages
      • AI
      • AWS SDK
      • Babel
      • Bcrypt
      • Cheerio
      • Concurrently
      • Csv
      • Cypress
      • Eslint
      • Jest
      • Lodash
      • Matter
      • Monaco Editor
      • Use to compile SCSS
      • Prettier
      • Query-string
      • Socket.io
      • Supabase
      • Trpc
      • Zod
    • TypeScript
      • 3rd Party Types
      • Classes
      • Enum
      • Generics
      • Interfaces
      • Migration
      • Narrowing Types
      • Types
      • Utility Types
      • packages
        • TRPC
    • JS In the Browser
      • CORS
      • DOM Control
      • Shadow DOM
      • Audio
      • Canvas
      • Clipboard
      • Cookies?
      • Events
      • Iframe
      • Intersection Observers
      • Keyboard Input
      • Media
      • Navigator
      • Scroll
      • Serving an index.html
      • Storage
      • Web Workers
      • packages
        • Tabulator
        • Bowser
        • Boxicons
        • d3
        • mapbox
        • MouseTrap Key Bindings
        • Videojs
  • ML
    • Embedding
    • Github
    • Vector Store
    • Deploy
      • Beam Cloud
      • Paperspace
    • NLP
      • Pre-processing text
      • 2vectorizing
      • 3featureengineering
      • 4ml
      • 4ml2
      • Overview
      • nltk
    • lowlevel
      • ML BuzzWords Guide
      • Regularization
      • DataSets
      • basics
        • Datasets
        • GPU Optimization
        • Neural Nets
  • README
    • Images, Fonts, and Files
    • Children
    • Context
    • Deployment
    • useEffect
    • Errors in React
    • Events
    • Forms
    • Hooks
    • JSX
    • Lists
    • Optimization
    • Rendering
    • Server Components
    • Setting up React
    • State and Lifecycle
    • Style
    • Suspense
    • Next
      • Image
      • Route Handlers
      • App Router
      • Assets
      • Next Config
      • CSS
      • Deploying
      • Layouts
      • Adding Markdown
      • Metadata
      • App Router
      • Server Actions
      • Setup
      • PreAppRouter
        • SSR
        • Api Routes
        • Head
        • Layouts
        • Pages
      • packages
        • ContentLayer
        • MDX Enchanced
        • T3 Env
    • How to fetch Data
      • React Query
    • packages
      • SWR
      • Classnames
      • Color
      • DatePicker
      • Draggable
      • Emotion
      • React-Feather
      • Fluxible
      • General
      • Google Login
      • Intersection Observer
      • React Lazy Load
      • Script
      • Proptypes
      • React Router v6
      • Recharts
      • React-Select
      • ShadCn
      • Slate
      • React-Table
      • Tanstack Router
      • Telephone Input
      • Three-Fiber
      • React-Toastify
      • React-Tooltip
      • React Video JS
      • Waypoint
      • React-window
      • XState
    • How to manage state in React
      • Jotai
      • Mobx
      • Zustand
      • Redux
        • Immer
        • Middleware
        • Redux
        • Redux
        • Redux Toolkit
        • selectors
        • Setup
        • Thunk
    • All
      • Gatsby
        • Cli
        • Data Integrations
        • Link
        • Pages
        • Plugins
        • Setup
        • Style
        • Transformer
        • packages
          • MDX
          • TailwindCss
      • Prehooks
        • Class Components
        • Ref
        • State and Lifecycle
      • React Native
        • ListViews
        • SafeArea
        • Built-in Components
        • Networking
        • Gestures
        • Navigation
        • Notifications
        • Orientation
        • Style
        • Dimensions
        • Expo
          • Development Build
          • Env
          • Setup
        • packages
          • Reanimated
          • React Native View Shot
  • Rust
    • Error
    • Importing
    • Ownership
Powered by GitBook
On this page
  • Creation
  • CLI
  • Publically Accessible
  • Static Website Hosting
  • Advanced
  1. Deployment Solutions
  2. AWS
  3. Storage

S3(Simple Storage Service)

PreviousRDS(Relational DB Service)NextCloud GPUS

Last updated 8 months ago

Object Storage

Basically nice http filesystem as a service

Buckets that can be accessed over https with uri like https://s3.us-east-2.amazonaws.com/jsfuentes-test/JorgeFuentes_Nov2018.pdf

Can configure access, default private

Uses:

  • store files, images, videos, etc

  • could be used as a ghetto key value store or ghetto GitHub

  • can serve as a static web site hosting

Creation

Make new bucket give unique name, create bucket, super ez,

CLI

aws s3 sync . s3://mybucket

Publically Accessible

Make public, can just follow link to object ez ezezezezezezz

Permissions->ACL to give public access

Permissions->CORS configuration: Might need CORS, unsure...

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 3000
    }
]

Can create an endpoint to create signed urls to access specific resources

Static Website Hosting

  1. Set to publically accessible in the bucket policy

  • policy type → S3

  • Effect → Allow

  • Principal → *

  • Actions → GetObject

  • Amazon Resource Number (ARN) → Copy “arn” from permissions tab add /* or path

    {
      "Id": "Policy1567137118824",
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "Stmt1567137117737",
          "Action": [
            "s3:GetObject"
          ],
          "Effect": "Allow",
          "Resource": "arn:aws:s3:::ecstatic-iframe-plugin/*",
          "Principal": "*"
        }
      ]
    }
  1. Go to Properties tab of bucket and turn on server web hosting set both index and error to index.html

  • You need index.html in EACH subfolder and path requests will route to that index.html

  • You will get a http url

  1. To setup https, you use Cloudfront look at Cloudfront.md

  • ? You might be able to just use cloudfront and skip step 2?

Extra: Look at codebuild.md to get a built react app or github updates into the S3, use the AWS cli to get the artifact in the root

Advanced

MultiPart Upload

Can upload a file part by part and even in parallel, then it will be constructed when you signal completion

import AWS from "aws-sdk";
import bytes from "bytes";
const debug = require("debug")("app:Folder:AWSUploader");

const REGION = "us-west-2";
const POOL_ID = "us-west-2:banabnbanabnbanbfnabnabnabnabna";
const BUCKET_NAME = "sigma-direct";

export default class AWSUploader {
  constructor(filename, type) {
    this.bucketName = BUCKET_NAME; //audio file store
    this.etag = []; // etag is used to save the parts of the single upload file
    this.partNumber = 0; // multipart requires incremetal so that they can merge all parts by ascending order
    this.filename = filename; //unique filename
    this.type = type;
    this.uploadId = ""; // upload id is required in multipart
    this.uploadPromises = [];
    this.curBlob = null;

    AWS.config.region = REGION;
    AWS.config.credentials = new AWS.CognitoIdentityCredentials({
      IdentityPoolId: POOL_ID
    });
    this.s3 = new AWS.S3();
    //make start request now, but don't block
    this.initalizedP = this.startMultiUpload();
  }

  upload(blob) {
    const prevUploads = [...this.uploadPromises]; //needs to be copy of array in prev state

    const f = async () => {
      await this.initalizedP; //make sure start request happened, I assume multiple blobs should never be waiting here
      await Promise.all(prevUploads); //ensure all prevUploads are done
      if (this.curBlob === null) {
        this.curBlob = blob;
      } else {
        this.curBlob = new Blob([this.curBlob, blob], { type: this.type });
      }

      debug("Currently", bytes(this.curBlob.size), "sends at 5mb");
      if (this.curBlob.size > bytes("5mb")) {
        const cb = this.curBlob;
        this.curBlob = null;
        await this.continueMultiUpload(cb);
      }
    };

    const uploadP = f();
    this.uploadPromises.push(uploadP);
    return uploadP;
  }

  /*
      Initiates a multipart upload and returns an upload ID.
      Upload id is used to upload the other parts of the stream
  */
  startMultiUpload() {
    debug("STARTING MULTIUPLOAD");
    const startParams = {
      Bucket: this.bucketName,
      Key: this.filename,
      ContentType: this.type,
      ACL: "private"
    };

    return new Promise(async (resolve, reject) => {
      this.s3.createMultipartUpload(startParams, (err, data) => {
        if (err) {
          reject(err);
        } else {
          debug("Created", data);
          this.uploadId = data.UploadId;
          resolve(data);
        }
      });
    });
  }

  /*
      Uploads a part in a multipart upload.
      The following code uploads part of a multipart upload. 
      it specifies a file name for the part data. The Upload ID is same that is returned by the initiate multipart upload. 
  */
  continueMultiUpload(blob) {
    this.partNumber += 1;
    const curPartNumber = this.partNumber;
    const params = {
      Body: blob,
      Bucket: this.bucketName,
      Key: this.filename,
      PartNumber: curPartNumber,
      UploadId: this.uploadId
    };
    debug("Continuing upload with", params);
    return new Promise((resolve, reject) => {
      this.s3.uploadPart(params, (err, data) => {
        if (err) {
          reject(err);
        } // an error occurred
        else {
          /*
              Once the part of data is uploaded we get an Entity tag for the uploaded object(ETag).
              which is used later when we complete our multipart upload.
          */
          debug("Uploaded part", curPartNumber);
          this.etag.push({
            ETag: data.ETag,
            PartNumber: curPartNumber
          });
          resolve(data);
        }
      });
    });
  }

  // Completes a multipart upload by assembling previously uploaded parts.
  async completeMultiUpload() {
    //wait for all current uploads, then check if any blobs left over
    await Promise.all(this.uploadPromises);
    debug("Leftover blobs", this.curBlob);
    if (this.curBlob) {
      await this.continueMultiUpload(this.curBlob);
    }

    debug("Finalizing parts", this.etag);
    this.etag = this.etag.sort((a, b) => a.PartNumber - b.PartNumber);
    const params = {
      Bucket: this.bucketName, // required
      Key: this.filename, // required
      UploadId: this.uploadId, // required
      MultipartUpload: {
        Parts: this.etag
      }
    };

    return new Promise((resolve, reject) => {
      this.s3.completeMultipartUpload(params, (err, data) => {
        if (err) {
          reject(err);
        } else {
          resolve(data);
        }
      });
    });
  }

  async abortMultiUpload() {
    //wait for current uploads just to be sure
    await Promise.all(this.uploadPromises);
    //s3 error if you try to abort an invalid uploadID or an uploadID with no parts uploaded yet
    if (!this.uploadId || this.partNumber === 0) {
      debug("Nothing to abort");
      return;
    }

    debug("Aborting");
    const params = {
      Bucket: this.bucketName, // required
      Key: this.filename, // required
      UploadId: this.uploadId // required
    };

    return new Promise((resolve, reject) => {
      this.s3.completeMultipartUpload(params, (err, data) => {
        if (err) {
          debug("Err had", err);
          reject(err);
        } else {
          debug("Aborted");
          resolve(data);
        }
      });
    });
  }
}

You are charged for incomplete multipart uploads, but you can add a lifecycle policy detailed

Nice coursera Overview
here