Task Queues

Elixir has Broadway

Python has Celery to process jobs, but you still need a message broker/connector (differences here) and here

Task Queues

Messege-broker-comparison-table

  • Kafka

    • Seems like most complex with messages pulled in batch

    • offsets saved in brokers? for easy restarting

    • Allows rate of 100k+/sec and focus on throughput

    • Apache Opensource

  • Rabbitmq

    • seems paid

    • can subscribe to topics

    • allows blocking/socket connections to request tasks

    • VMWare opensource

  • SQS

    • Simplest to spin up in AWS with free tier offering 1M free messages

    • Does long polling

    • Offers high throughput Standard "at least-once-delievery" which requeues a message if it was consumed from the queue and a visibility timeout passes

    • Can also do up to 3k message/sec when order important with SQS FIFO "exactly once delivery"

    • Base rate of 20k+/sec with more expensive options to request

    • Messages must be less than 256KiB in size, but can turn it into S3 object

  • SNS

    • AWS similar but broadcasts messages instead of being consumed by single consumer

Previous{SurgeNextTerraform

Last updated