Serverless

Data movement into, through and out of Redshift are a key part of your data solution. There are many full featured solutions that work with Redshift each with a meaningful cost. Also many of these solutions require that your data moves through their appliction and in the world of Big Data moving data more places than is absolutely required can be a performance limiter and cost amplifier. Many want a low-cost, simple soltution that moves their data efficiently and utilizes the AWS offerings to perform at the lowest possible cost.

At Weiner Advanced Development LLC we advocate for ELT (Extract, Load, Transform) patterns where Redshift performs the needed data transformations at near zero cost and the orchestration of these processes are performed by low cost serverless solutions. AWS Step Functions provide the workflow framework for the orchestration with AWS Lambda providing the "action" steps. These tools are extremely cheap at the execution rates needed to support data movement. Redshift is a very powerful tool for transforming data and can perform nearly all data manipulations done by standard ETL tools.

This is not the best pattern for large or complex ETL workflows but for basic ETL workflows, a serverless ELT pattern is often more flexible and far cheaper. Contact us at bill-wad0@gmail.com to learn more.