Deduplication is a specialised compression technique to remove repeating copies of duplicate data. It is used to improve storage utilisation and reduce the number of bytes to be sent via a network. Deduplication needs to be performed before ingesting data into a Data Warehouse, so that the data is clean…
ABC of Data Engineering
Why Data Engineering? Let’s look at the following problems and see what’s common to them For a supermarket find the products that have the lowest shelf life (amount of time they spend on the shelf before being purchased) For a national telecom provider, find all cell tower’s call drop ratio…
Build a Highly Available Distributed Cron
Crons are widely used to set up scheduled jobs and automate certain parts of a system. Setting up Cron for a simple application is very straightforward. All you need to do is configure it in your server using `crontab` to run at scheduled intervals and trigger the job. But what…
Microservice Architecture – Beyond HTTP
This talk explores why HTTP is not the best fit for microservice architectures and how it results in the need for complex cloud infrastructure components. This presents a different alternative using Redis pubsub and some other synchronisation facilities of redis. Its a demo of the simplicity and extensibility of the solution…
Resiliency in microservices
Microservices One of the important aspects while building a system involving lot of micro-services is the ability to heal or contain failure. Resilience. How do you ensure resiliency, avoiding cascading failures in microservices? Let’s take an example. Client calls Service A Service A depends on Service B to satisfy the request Service B…
Terraform, null_resources & Azure ASM API
Recently, I was trying to bring up virtual machines in Microsoft Azure but ran into this interesting & annoying problem of not being able to upload SSH keys via the terraform DSL. There is a provision to provide a ssh_key_thumbprint but sadly no way to upload what you would call a KeyPair in…
Terraform RemoteState Server
Terraform is a pretty nifty tool to layout complex infrastructures across cloud providers. It is an expressway to overcome the otherwise mundane and tedious task of going through insane amount of API documentations. The output of terraform a run is a JSON which carries an awesome lot of information that…
Singletons in Golang
I was recently writing an application in Golang which required some Database interaction. The db library I was using had inbuilt Pooling so I didn’t have to bother about connection recycling and reusing, as long as I could initialise a DbPool and continue to call DbPool.new(). Having a module level…