Move twilio recordings to azure and save on your twilio monthly bill

Twilio is an amazing cloud communication service which excels in a lot of things, however their recording storage is quite expensive. They charge you by minute stored per month, which means that if you have a recording of 10 mins, you will be charged for those 10 mins each month as long as you keep the recording on twilio's servers. Each account gets 10,000 free minutes per month and after that they charge you $0.0005 per minute per month (details can be seen in https://www.twilio.com/help/faq/voice/how-much-does-it-cost-to-record-a-call). Although this seems pretty low, the minutes start accumulating on your account, because after 3 months, you will be paying, for all the minutes recorded during the first month, plus the ones from the second and the third months too. If you have a high traffic callcenter, you can easily reach millions on minutes in a few months.

I checked multiple recordings, and in average, 1 minute equals 1MB in the recording file. And the minute is their unit, so no matter if your recording is 90 seconds long they charge you 2 minutes for that recording, although they only have to save 1.5MB in their datacenters.

Azure pricing is per GB, and at the time of this writing, its $0.048 per GB using geographically redundancy (6 copies of each file are stored).

Lets compare the costs of storing recordings in twilio versus storing them in azure in a real life scenario which is actually the reason why I did this in the first place. Currently, I'm the lead developer on a call center system that increased the calls that handles 3500 calls per day, with an average duration of 10 mins per call. We noticed that we were paying a lot of money for recording storage and this is the analysis that we did:

Twilio: (3500 [calls per day] * 10 [avg mins per call] * 30 [days in a month] - 10000 [free storage mins]) * $0.0005 [storage cost per min] = $520 per month

Azure: 3500 [calls per day] * 10 [avg mins per call] * 30 [days in a month] / 1024 [MB per GB] * $0.048 [storage cost per GB] = $49 per month

Note: The azure pricing calculation does not include the bandwidth cost of uploading the recordings, but for this same scenario its around $10 extra per month only one time.

So, just by moving the calls to azure we were able to save $471 per each month's recordings storage. And this is accumulable, because if you have been running the call center for 3 months, at the end of the 3rd month, you needed to store the recordings of the first month for the 3 months, plus the recordings of the second month for 2 months, plus the recording of the third month for 1 month. So after 3 months we were able to save $471 * (3 + 2 + 1) = $2826 saved in 3 months

Conclusion, lets the communications provider (twilio) for the communications parts of our solution, and lets use a cloud storage provider for storing the recordings (azure).

I have a github repository with the code needed to move your recordings to azure and start saving on your twilio's monthly bill right away: https://github.com/juanrodriguezcen/twilio-recordings-in-azure

This repository contains 2 projects:

A) MoveRecordingsToAzure: This is a console program that moves recordings from your twilio account to your azure account, including metadata of the recording like the duration and when it was created. This program has a configuration parameter that lets you tell the program to keep the recordings of the last XX days in twilio. If you want to move every recording to azure as soon as its generated, you can set it to 0. In order to move the recordings automatically, you would need to set it up as a windows scheduled task.

B) TwilioRecordings: Is an Asp.net MVC website with just 1 available url (/recordings/{sid}.wav). This url allows you to stream files from twilio and azure transparently. If you have the recording's SID in your db, you can use this url and you dont have to worry if your recording was already moved to azure or if its still in twilio. This website checks if the audio is still in twilio and lets you stream from there (delegating the streaming details to twilio), and if it is not it grabs it from the corresponding azure blob and handles the streaming details by its own (allowing file seeking).

Alias for SQL Server named instance

In this entry I'll describe a simple solution for a common problem that I saw while working on development teams with multiple coders.

Every time you install SQL Server, you might give the instance that you are installing a new name, or not and SQL Server will use ".". The problem is that when you start working with other coders on the same project, their instances names probably wont match yours and this generate problems in the project's connection strings.  This happened to me multiple times and every time I did a pull from our git repository I had fix the connection string.

Named instances aliases are a great solutions for this problem! Basically the only thing that you need to do is to define an alias with your coworkers and set it up. If you work with different groups of people in different projects, you can adds new aliases pointing to your SQL Server instance, and you can forget about fixing connection strings.

Lets see how to do this:

1) Open "Sql Server Configuration  Manager"

2) Open the network configuration and enable the TCP/IP protocol if you have it disabled

Note: I have 2 sql server instances, Im doing this for my "SQLSERVER2008R2" instance

3) Open the TCP/IP protocol properties and enable the "Listen All" properties so accept connetion on any ip that your computer has.

4) Grab the port where your instance will be accepting the TCP/IP connections. For this, right to the bottom you need to set "0" as the value for "TCP Dynamic Ports" and restart your sql instance. Once you do this, you will get a port number assigned and it will be reserved for this instance in the future. Copy the port number as you'll need it for the next step

Here I already have the port assigned

5) Now do a right click on the "Aliases" section and select "New Alias". You will need to specify, the name for the alias (this is what you will use in your connection strings, we used "SQLDev" with my team), the port number that you grabbed from step 4, the protocol (TCP/IP) and the server (which is the machine name, you can use ".", and the named instance if your instance is named if not it works with ".").

6) Use your Alias! 

This is how to use it with SQL Server Management Studio

You can use this alias in any connection string. Enjoy!

Automatically download subtitles from subdivx (Python Script)

Im a big fan of tv shows (the big bang theory, modern family, walking dead, etc). Since some time ago I automated the process of downloading the new episodes of my favorite shows using an bittorent client and an rss feed from a bittorent tracker, but as my primary language is spanish, I still had to manually download the subtitles. After seeing that the process of downloading subtitles from the site that I use was pretty much repetitive, I decided to make a python script to do it automatically for me. The site that I use for subtitle is www.subdivx.com, which is really great because you usually can find multiple versions of each subtitle (for different video files, from different authors, etc). Although the subtitles of this site are awesome, the big problem is that they dont have an API to get the subtitles from them, so in the script I had to scrape their site.

Basically the python script checks all the files in a configurable folder path, looking for video files that doesnt have a subtitle yet (an .srt file with the same name), and then scrapes subdivx.com searching for the subtitle, and if it finds it, it downloads it and renames it to match the video file name.

The usage is pretty straight forward, at the top you will find a couple of paths that you need to replace with your own, and there is also a variable called "default_subgroups" which specifies which are the subtitles authors that I like, in this case I selected 'argenteam', 'thesubfactory', 'substeam', because their subtitles always have excelent quality.

After you set up those variables, you just need to run it.

Here is the github link from where you can download the script: Link

This script has worked really good for me, I always download 720p mkv files, I didnt try it with other type of files, but it shouldnt be too complicated to adapt it. Hope it works fine for you too!

Important Note: There is a  file called "requirements.txt" that has all the required python modules. In order to install them you just need to run: pip install -r requirements.txt

Dropbear SSH passwordless authentication (public key authentication).

If you want to access a remote linux server over the internet using ssh and you are concerned about security, using public key authentication and disabling the password authentication its a good option.

My scenario is that I have a Raspberry PI running a raspbmc as a media-player/home-server at my home. As I wanted to access it by ssh through the internet while keeping my personal data secure, I decided to use this option, which its actually pretty easy to use.

Step 1: Generate a public/private key pair 

You will use these keys to authenticate with the server, the server will have your public key, and whenever you want to connect to it through ssh, you use your private key.

If you already have a private/key pair that you use for other server, you can skip this part (you dont need a pair of keys for each server you use).

In linux (it can be on your computer if you are using linux, or just do it on the server through ssh), run the following command:

ssh-keygen

You will be asked for:

1) A location for the key you can use any filename. This utility will create 2 files with the name that you enter, one without any extension which its your private key, and one with extension .pub which its your public key.
2) A passphrase. Its very important to pick a safe passwords, because this passphrase its what protects the key in case anyone gets your private key file. This is something awesome, because anybody that wants to login to your servers need 2 things, the private key file PLUS the passphrase. Please dont pick 123456
3) It will ask you to confirm the passphrase

Now you have the 2 files that are your private and public key.

Step 2: Set up your public key in the server

Login to your server with your user (or maybe just root if you only use root).

You need to create a new file in "~/.ssh/authorized_keys" and copy in it the content of your public key  (its the filename that you picked with the extension .pub).

This tells the ssh server which are the public keys that are authorized to login as the current user.

Step 3: Test that you can login using your private key

Logout from the server, and try log in using your private key. This depends on the ssh client that you are using, so Im not including an explanation on how to do it on each client, but a quick google search for the client that you use will do the trick. 

This is very important because on the next step you are going to disable the password authentication, so if the public key authentication method is not working before we disable the password authentication you wont be able to login to the server again.

Step 4: Disable password authentication

Dont disable until you double checked that your public key authentication is working. 

This instructions work for dropbear ssh server. 

You need to edit the file "/etc/xinetd.d/ssh"

Change the line:

server_args = -i

With

server_args = -i -s

Thats all, your server now only accepts ssh connections authenticated using public key authentication.

Homemade Dynamic DNS Solution

In a lot of countries (Uruguay is one of them), ISPs offer domestic internet services with dynamic ips which change every some hours. In the case of Uruguay, our ISP changes our ip every 12 hours.

This makes makes things difficult for those of us who want to host some service in a computer on our house. For example, lets say that you have a computer with a webserver in your house and you want to access it outside your LAN, over the internet. In order to access the computer at your home you need your public ip (which is the ip that your ISP provides you), but this ip is constantly changing.

The solution is to have a domain pointing to your current ip, and a service that updates your domain dns entries every time your ISP changes your ip.

There are multiple commercial services out there, probably one of the most common one is "dyndns". The problem with these solutions is that they are quite expensive (specially for what they offer) and they dont work 100% of the time.

So, why dont go with the DIY option? This is the path that I took, coding a really basic python script that uses Amazon's Route 53 dns service. What amazon provides its a dns service at a low cost ($0.5 per domain/month) and it gives you an API that allows you to programmatically update the dns entries of your domain.

The basic idea is that every certain amount of time (lets say 5 mins), I do a request to a public webservice that returns my current ip in json format and if the ip changed, it updates my domain's dns entry at Amazon.

This is the python script:

from boto.route53.connection import Route53Connection
import urllib2
import json

# your amazon keys
key = "XXXXXXXXXXXXXXXXXXXXXXX"
access = "XXXXXXXXXXXXXXXXXXXXXXXX"

#Get your current ip 
content = urllib2.urlopen("http://ifconfig.me/all/json").read()
decoded = json.loads(content)
external_ip = decoded['ip_addr']

#Get the current dns entry value from amazon
route53 = Route53Connection(key, access)
zone = route53.get_zone("your-domain.com")
current_dns_ip = zone.get_a("subdomain.your-domain.com.").resource_records[0]

#If your current ip is different than the value that amazon dns service has, update it
if current_dns_ip != external_ip:
        zone.update_a("subdomain.your-domain.com.", external_ip)

(The script uses "boto" which is amazon's official library for python)

Im using it in a Raspberry PI at my home, so I can access to it through ssh over the internet. I set it up as cron to run every 5 mins.

In your crontab file you just need to add (use "crontab -e" for editing your crontab rules):

*/5 * * * * /usr/bin/python /home/pi/dynamicDNS/updateDNS.py

Thats all, with this you will have your "homemade dynamic dns service".

Arduino - How to improve the realiability of sensors readings

Some days ago, as part of an obstacle avoiding robot project, I was trying to identify nearby objects using an ultrasonic distance sensor in order to turn right or left to avoid them.

In theory, this was a pretty straight forward task... but when I coded the described behavior, the robot started turning when it shouldnt. I looked at the code and everything looked fine, the only possible reason was that the robot thought that there was an object when the was no object.

I connected the arduino to the computer and started printing the distance sensor readings to the Arduino's IDE serial monitor. I was using 20cms as the distance where the robot should stop and turn in order avoid an object. So, I placed an object at 40cms and started watching the readings through the serial monitor, and I noticed that I was getting weird values. For example, I was getting:

39 40 41 3 39 4 42 42 3000 2800 40 41 38 2 40

I was aware that sensors are not accurate things because they are affected by multiple variables (quality of its construction, electrical noise, enviromental changes, etc), I was expecting values near the real value, for example if the real value was 40cm, I was expecting values between 35 and 45 maybe... but I never expected outliers that far from the real value.

Probably the better thing to do is to buy a sensor with the quality needed for the project, and also do everything you can to reduce the electrical noise, however no matter what you do, you wont get exact values on every reading. So, what can you do to improve the realiability of your sensors so they dont mess your program's logic?

I found that getting multiple samples and processing them sustantially improves the accuracy of the sensors readings.

So, every time that we want to measure the distance to an object, we are going to get multiple sample and get only one value from them.

Your first thought might be "lets average the numbers and we'll probably get near to the real value". Well, this is a BAD IDEA.

Lets say that for each measure we use 7 sensor readings and we get:

42 40 3000 41 3 38 43

The average is: 42 + 40 + 3000 + 41 + 3 + 38 + 43 = 3207 / 7 = 458cm (aprox)

A better way of processing the 7 samples is to use the median method. This is a very simple method, where you first sort all the values and then you use the value in the middle position, so half of the values are lower and half of the values are higher.

Lets apply the median method to this set of 7 samples:

First we sort it, getting : 3 38 30 41 42 43 3000

The result of the median method is 41, which is really close to the real value (which was 40).

I applied this method to the obstacle avoiding robot and I had really good results. Its important to notice that even with data filtering, you wont get the exact value, so you need to use "safe thresholds" in your program's logic. For example Im using 20 cms as the value where the robot should stop and turn in order to avoid an object. If there is an object at 19cms and the sensor returns a distance of 21cm, there is no problem, on the next iteration of the loop the robot will be closer to the object and I'll get a distance lower 20cms, so it will stop and turn before hitting the object.

There is an existing library for Arduino that allows us to easily get the median value of a set of samples. You can get it from here.

This is an example of how to use it:

#include 
#include 

//Define the pins for the ultrasonic sensor's echo and trigger pins 
#define DISTANCE_ECHO 4
#define DISTANCE_TRIGGER 7
#define MAX_ULTRASONIC_DISTANCE 200

NewPing sonar(DISTANCE_TRIGGER, DISTANCE_ECHO, MAX_ULTRASONIC_DISTANCE);

void setup()
{
    //Initialize serial communication to send the distance value  to the serial monitor
    Serial.begin(9600);
}

void loop()
{
  int distance_cm;
  distance_cm = getDistanceFromUltrasonicSensor();
  
  Serial.print(distance_cm);
  Serial.print("cm");
  Serial.println();
}

int getDistanceFromUltrasonicSensor(){
  //Create an instance of the RunningMedian class with the number of samples to use
  RunningMedian samples = RunningMedian(7);
  
  //Get the 7 samples
  for(int i= 0; i<5; i++){
      delay(50);
      int pingVal = sonar.ping();
      
      int distance = pingVal / US_ROUNDTRIP_CM;
      if(distance == 0){
        distance = 3000;
      }
     samples.add(distance);
   }   
   //Return the median value
   return samples.getMedian();  
}

Here I applied the median method for an ultrasonic distance sensor, but this applies to any other sensor, for example, temperature sensors, compass sensor, etc.

Arduino - Controlling DC motors with L298N motor controller

If you want to control a motor with the Arduino board, you'll need to power the motor with an external power source because the Arduino only provides up to 40 mA on its outputs, which is not enough to move DC motors. You will also need some circuit that allows you to move the motor forward and backwards (H-Bridge).

The L298N motor controller solves these 2 things for us, it allows us to power the motor from an external power source, while also allowing us to control the motor speed and directions with low current signals (which the arduino can handle).

 

Off topic: I bought mine from DealExtreme, which if you dont know I definitely recommend visiting. Its an online store that sells an enormous variety of things, at low prices and with free international shipping.

Lets see the input and outputs that this controller has:

Battery inputs	These are 2 of the blue connectors that you can see in the image above. This requires a battery which will power the 2 motors. The battery’s voltage should be compatible with the motors that you are going to use.  Check the specs of your controller to see the max allowed voltage .
Motor A outputs	These are 2 of the blue connectors that you can see in the image above. This will power the motor A
Motor B outputs	These are 2 of the blue connectors that you can see in the image above. This will power the motor B.
EN A	This signal should be connected to an Arduino’s PWM output pin. It will determine the speed of the motor A, a value of 0 means stopped and a value of 255 is max speed.
EN B	This signal should be connected to an Arduino’s PWM output pin. It will determine the speed of the motor B, a value of 0 means stopped and a value of 255 is max speed.
IN1 and IN2	These are 2 digital signals that will determine Motor’s A behavior (going forward, going backwards)
IN3 and IN4	These are 2 digital signals that will determine Motor’s B behavior (going forward, going backwards)

The following values will allow you to set the motors directions from your arduino.

	IN1	IN2		IN3	IN4
Motor A Forward	HIGH	LOW	Motor B Forward	HIGH	LOW
Motor A Backwards	LOW	HIGH	Motor B Backwards	LOW	HIGH

In the next example I'll show how to control 2 independant DC motor.

//Define the speed for the motors, Im just going to use 200 (almost top speed)
#define SPEED 200

//Define the signal pins for motor A
#define ENA 3
#define MAIN1  8
#define MAIN2  9

//Define the signal pins for motor B
#define ENB 5
#define MBIN1 10
#define MBIN2 11

void setup()
{
    pinMode(MAIN1, OUTPUT);  
    pinMode(MAIN2, OUTPUT);
    pinMode(MBIN1, OUTPUT);  
    pinMode(MBIN2, OUTPUT);
}

void loop(){
  forwardMotorA(SPEED);
  delay(2000); //let it running for 2 seconds
  backwardsMotorA(SPEED);
  delay(2000); 
  stopMotorA();

 
  forwardMotorB(SPEED);
  delay(2000); //let it running for 2 seconds
  backwardsMotorB(SPEED);
  delay(2000); 
  stopMotorB();
} 

void forwardMotorA(int speedVal){
  digitalWrite(MAIN1,HIGH);  
  digitalWrite(MAIN2,LOW); 

  analogWrite(ENA, speedVal);
}

void backwardsMotorA(int speedVal){
  digitalWrite(MAIN1,HIGH);  
  digitalWrite(MAIN2,LOW); 

  analogWrite(ENA, speedVal);
}

void stopMotorA(){
  analogWrite(ENA, 0);
}

void forwardMotorB(int speedVal){
  digitalWrite(MBIN1,HIGH);  
  digitalWrite(MBIN2,LOW); 

  analogWrite(ENB, speedVal);
}

void backwardsMotorB(int speedVal){
  digitalWrite(MBIN1,HIGH);  
  digitalWrite(MBIN2,LOW); 

  analogWrite(ENB, speedVal);
}

void stopMotorB(){
  analogWrite(ENB, 0);
}