Automatically download subtitles from subdivx (Python Script)

Im a big fan of tv shows (the big bang theory, modern family, walking dead, etc). Since some time ago I automated the process of downloading the new episodes of my favorite shows using an bittorent client and an rss feed from a bittorent tracker, but as my primary language is spanish, I still had to manually download the subtitles. After seeing that the process of downloading subtitles from the site that I use was pretty much repetitive, I decided to make a python script to do it automatically for me. The site that I use for subtitle is www.subdivx.com, which is really great because you usually can find multiple versions of each subtitle (for different video files, from different authors, etc). Although the subtitles of this site are awesome, the big problem is that they dont have an API to get the subtitles from them, so in the script I had to scrape their site.

Basically the python script checks all the files in a configurable folder path, looking for video files that doesnt have a subtitle yet (an .srt file with the same name), and then scrapes subdivx.com searching for the subtitle, and if it finds it, it downloads it and renames it to match the video file name.

The usage is pretty straight forward, at the top you will find a couple of paths that you need to replace with your own, and there is also a variable called "default_subgroups" which specifies which are the subtitles authors that I like, in this case I selected 'argenteam', 'thesubfactory', 'substeam', because their subtitles always have excelent quality.

After you set up those variables, you just need to run it.

Here is the github link from where you can download the script: Link

This script has worked really good for me, I always download 720p mkv files, I didnt try it with other type of files, but it shouldnt be too complicated to adapt it. Hope it works fine for you too!

Important Note: There is a  file called "requirements.txt" that has all the required python modules. In order to install them you just need to run: pip install -r requirements.txt

Dropbear SSH passwordless authentication (public key authentication).

If you want to access a remote linux server over the internet using ssh and you are concerned about security, using public key authentication and disabling the password authentication its a good option.

My scenario is that I have a Raspberry PI running a raspbmc as a media-player/home-server at my home. As I wanted to access it by ssh through the internet while keeping my personal data secure, I decided to use this option, which its actually pretty easy to use.

Step 1: Generate a public/private key pair 

You will use these keys to authenticate with the server, the server will have your public key, and whenever you want to connect to it through ssh, you use your private key.

If you already have a private/key pair that you use for other server, you can skip this part (you dont need a pair of keys for each server you use).

In linux (it can be on your computer if you are using linux, or just do it on the server through ssh), run the following command:

ssh-keygen

You will be asked for:

1) A location for the key you can use any filename. This utility will create 2 files with the name that you enter, one without any extension which its your private key, and one with extension .pub which its your public key.
2) A passphrase. Its very important to pick a safe passwords, because this passphrase its what protects the key in case anyone gets your private key file. This is something awesome, because anybody that wants to login to your servers need 2 things, the private key file PLUS the passphrase. Please dont pick 123456
3) It will ask you to confirm the passphrase

Now you have the 2 files that are your private and public key.

Step 2: Set up your public key in the server

Login to your server with your user (or maybe just root if you only use root).

You need to create a new file in "~/.ssh/authorized_keys" and copy in it the content of your public key  (its the filename that you picked with the extension .pub).

This tells the ssh server which are the public keys that are authorized to login as the current user.

Step 3: Test that you can login using your private key

Logout from the server, and try log in using your private key. This depends on the ssh client that you are using, so Im not including an explanation on how to do it on each client, but a quick google search for the client that you use will do the trick. 

This is very important because on the next step you are going to disable the password authentication, so if the public key authentication method is not working before we disable the password authentication you wont be able to login to the server again.

Step 4: Disable password authentication

Dont disable until you double checked that your public key authentication is working. 

This instructions work for dropbear ssh server. 

You need to edit the file "/etc/xinetd.d/ssh"

Change the line:

server_args = -i

With

server_args = -i -s

Thats all, your server now only accepts ssh connections authenticated using public key authentication.

Homemade Dynamic DNS Solution

In a lot of countries (Uruguay is one of them), ISPs offer domestic internet services with dynamic ips which change every some hours. In the case of Uruguay, our ISP changes our ip every 12 hours.

This makes makes things difficult for those of us who want to host some service in a computer on our house. For example, lets say that you have a computer with a webserver in your house and you want to access it outside your LAN, over the internet. In order to access the computer at your home you need your public ip (which is the ip that your ISP provides you), but this ip is constantly changing.

The solution is to have a domain pointing to your current ip, and a service that updates your domain dns entries every time your ISP changes your ip.

There are multiple commercial services out there, probably one of the most common one is "dyndns". The problem with these solutions is that they are quite expensive (specially for what they offer) and they dont work 100% of the time.

So, why dont go with the DIY option? This is the path that I took, coding a really basic python script that uses Amazon's Route 53 dns service. What amazon provides its a dns service at a low cost ($0.5 per domain/month) and it gives you an API that allows you to programmatically update the dns entries of your domain.

The basic idea is that every certain amount of time (lets say 5 mins), I do a request to a public webservice that returns my current ip in json format and if the ip changed, it updates my domain's dns entry at Amazon.

This is the python script:

from boto.route53.connection import Route53Connection
import urllib2
import json

# your amazon keys
key = "XXXXXXXXXXXXXXXXXXXXXXX"
access = "XXXXXXXXXXXXXXXXXXXXXXXX"

#Get your current ip 
content = urllib2.urlopen("http://ifconfig.me/all/json").read()
decoded = json.loads(content)
external_ip = decoded['ip_addr']

#Get the current dns entry value from amazon
route53 = Route53Connection(key, access)
zone = route53.get_zone("your-domain.com")
current_dns_ip = zone.get_a("subdomain.your-domain.com.").resource_records[0]

#If your current ip is different than the value that amazon dns service has, update it
if current_dns_ip != external_ip:
        zone.update_a("subdomain.your-domain.com.", external_ip)

(The script uses "boto" which is amazon's official library for python)

Im using it in a Raspberry PI at my home, so I can access to it through ssh over the internet. I set it up as cron to run every 5 mins.

In your crontab file you just need to add (use "crontab -e" for editing your crontab rules):

*/5 * * * * /usr/bin/python /home/pi/dynamicDNS/updateDNS.py

Thats all, with this you will have your "homemade dynamic dns service".

Arduino - How to improve the realiability of sensors readings

Some days ago, as part of an obstacle avoiding robot project, I was trying to identify nearby objects using an ultrasonic distance sensor in order to turn right or left to avoid them.

In theory, this was a pretty straight forward task... but when I coded the described behavior, the robot started turning when it shouldnt. I looked at the code and everything looked fine, the only possible reason was that the robot thought that there was an object when the was no object.

I connected the arduino to the computer and started printing the distance sensor readings to the Arduino's IDE serial monitor. I was using 20cms as the distance where the robot should stop and turn in order avoid an object. So, I placed an object at 40cms and started watching the readings through the serial monitor, and I noticed that I was getting weird values. For example, I was getting:

39 40 41 3 39 4 42 42 3000 2800 40 41 38 2 40

I was aware that sensors are not accurate things because they are affected by multiple variables (quality of its construction, electrical noise, enviromental changes, etc), I was expecting values near the real value, for example if the real value was 40cm, I was expecting values between 35 and 45 maybe... but I never expected outliers that far from the real value.

Probably the better thing to do is to buy a sensor with the quality needed for the project, and also do everything you can to reduce the electrical noise, however no matter what you do, you wont get exact values on every reading. So, what can you do to improve the realiability of your sensors so they dont mess your program's logic?

I found that getting multiple samples and processing them sustantially improves the accuracy of the sensors readings.

So, every time that we want to measure the distance to an object, we are going to get multiple sample and get only one value from them.

Your first thought might be "lets average the numbers and we'll probably get near to the real value". Well, this is a BAD IDEA.

Lets say that for each measure we use 7 sensor readings and we get:

42 40 3000 41 3 38 43

The average is: 42 + 40 + 3000 + 41 + 3 + 38 + 43 = 3207 / 7 = 458cm (aprox)

A better way of processing the 7 samples is to use the median method. This is a very simple method, where you first sort all the values and then you use the value in the middle position, so half of the values are lower and half of the values are higher.

Lets apply the median method to this set of 7 samples:

First we sort it, getting : 3 38 30 41 42 43 3000

The result of the median method is 41, which is really close to the real value (which was 40).

I applied this method to the obstacle avoiding robot and I had really good results. Its important to notice that even with data filtering, you wont get the exact value, so you need to use "safe thresholds" in your program's logic. For example Im using 20 cms as the value where the robot should stop and turn in order to avoid an object. If there is an object at 19cms and the sensor returns a distance of 21cm, there is no problem, on the next iteration of the loop the robot will be closer to the object and I'll get a distance lower 20cms, so it will stop and turn before hitting the object.

There is an existing library for Arduino that allows us to easily get the median value of a set of samples. You can get it from here.

This is an example of how to use it:

#include 
#include 

//Define the pins for the ultrasonic sensor's echo and trigger pins 
#define DISTANCE_ECHO 4
#define DISTANCE_TRIGGER 7
#define MAX_ULTRASONIC_DISTANCE 200

NewPing sonar(DISTANCE_TRIGGER, DISTANCE_ECHO, MAX_ULTRASONIC_DISTANCE);

void setup()
{
    //Initialize serial communication to send the distance value  to the serial monitor
    Serial.begin(9600);
}

void loop()
{
  int distance_cm;
  distance_cm = getDistanceFromUltrasonicSensor();
  
  Serial.print(distance_cm);
  Serial.print("cm");
  Serial.println();
}

int getDistanceFromUltrasonicSensor(){
  //Create an instance of the RunningMedian class with the number of samples to use
  RunningMedian samples = RunningMedian(7);
  
  //Get the 7 samples
  for(int i= 0; i<5; i++){
      delay(50);
      int pingVal = sonar.ping();
      
      int distance = pingVal / US_ROUNDTRIP_CM;
      if(distance == 0){
        distance = 3000;
      }
     samples.add(distance);
   }   
   //Return the median value
   return samples.getMedian();  
}

Here I applied the median method for an ultrasonic distance sensor, but this applies to any other sensor, for example, temperature sensors, compass sensor, etc.

Arduino - Controlling DC motors with L298N motor controller

If you want to control a motor with the Arduino board, you'll need to power the motor with an external power source because the Arduino only provides up to 40 mA on its outputs, which is not enough to move DC motors. You will also need some circuit that allows you to move the motor forward and backwards (H-Bridge).

The L298N motor controller solves these 2 things for us, it allows us to power the motor from an external power source, while also allowing us to control the motor speed and directions with low current signals (which the arduino can handle).

 

Off topic: I bought mine from DealExtreme, which if you dont know I definitely recommend visiting. Its an online store that sells an enormous variety of things, at low prices and with free international shipping.

Lets see the input and outputs that this controller has:

Battery inputs	These are 2 of the blue connectors that you can see in the image above. This requires a battery which will power the 2 motors. The battery’s voltage should be compatible with the motors that you are going to use.  Check the specs of your controller to see the max allowed voltage .
Motor A outputs	These are 2 of the blue connectors that you can see in the image above. This will power the motor A
Motor B outputs	These are 2 of the blue connectors that you can see in the image above. This will power the motor B.
EN A	This signal should be connected to an Arduino’s PWM output pin. It will determine the speed of the motor A, a value of 0 means stopped and a value of 255 is max speed.
EN B	This signal should be connected to an Arduino’s PWM output pin. It will determine the speed of the motor B, a value of 0 means stopped and a value of 255 is max speed.
IN1 and IN2	These are 2 digital signals that will determine Motor’s A behavior (going forward, going backwards)
IN3 and IN4	These are 2 digital signals that will determine Motor’s B behavior (going forward, going backwards)

The following values will allow you to set the motors directions from your arduino.

	IN1	IN2		IN3	IN4
Motor A Forward	HIGH	LOW	Motor B Forward	HIGH	LOW
Motor A Backwards	LOW	HIGH	Motor B Backwards	LOW	HIGH

In the next example I'll show how to control 2 independant DC motor.

//Define the speed for the motors, Im just going to use 200 (almost top speed)
#define SPEED 200

//Define the signal pins for motor A
#define ENA 3
#define MAIN1  8
#define MAIN2  9

//Define the signal pins for motor B
#define ENB 5
#define MBIN1 10
#define MBIN2 11

void setup()
{
    pinMode(MAIN1, OUTPUT);  
    pinMode(MAIN2, OUTPUT);
    pinMode(MBIN1, OUTPUT);  
    pinMode(MBIN2, OUTPUT);
}

void loop(){
  forwardMotorA(SPEED);
  delay(2000); //let it running for 2 seconds
  backwardsMotorA(SPEED);
  delay(2000); 
  stopMotorA();

 
  forwardMotorB(SPEED);
  delay(2000); //let it running for 2 seconds
  backwardsMotorB(SPEED);
  delay(2000); 
  stopMotorB();
} 

void forwardMotorA(int speedVal){
  digitalWrite(MAIN1,HIGH);  
  digitalWrite(MAIN2,LOW); 

  analogWrite(ENA, speedVal);
}

void backwardsMotorA(int speedVal){
  digitalWrite(MAIN1,HIGH);  
  digitalWrite(MAIN2,LOW); 

  analogWrite(ENA, speedVal);
}

void stopMotorA(){
  analogWrite(ENA, 0);
}

void forwardMotorB(int speedVal){
  digitalWrite(MBIN1,HIGH);  
  digitalWrite(MBIN2,LOW); 

  analogWrite(ENB, speedVal);
}

void backwardsMotorB(int speedVal){
  digitalWrite(MBIN1,HIGH);  
  digitalWrite(MBIN2,LOW); 

  analogWrite(ENB, speedVal);
}

void stopMotorB(){
  analogWrite(ENB, 0);
}

Raspberry PI and Arduino - Great combination!

Raspberry PI

For those of you who dont know the Raspberry PI yet, it is a VERY cheap and VERY tiny computer. It costs around $35 - $40 and its of the size of a credit card. Of course its not a core i7 with 16GB of RAM, but has a quite decent ARM 700MHz CPU with 512MB of RAM, which allows you to do a lot of things. You can see the Raspberry PI's offical site to see all the crazy ideas that people come up with the Raspberry PI.

The Raspberry PI has some interesting benefits:

• For the cost its pretty powerful (of course its useful for some applications, its not for everything, you wont be able to play GTA V on it, ha)
• It runs linux
• It works without a fan (yeah! no noise!)
• Low power consumption (700-1000mA for Model B, which is the model with an ethernet port)
• Its really small

The Raspberry PI has some limitations though and during its usage I found some issues. The one that struggled me most, is that the USB ports cannot power most of the devices, so you need a powered hub for them, and the Raspberry PI is not compatible with every USB hub... so its a bit complicated (you can get a list of compatible devices from here). For example some webcams, wifi dongles and external hdd need a powered USB hub.

Although the Raspberry PI its quite powerful, it has a major drawback for robotics or any electronic projects... it has no built-in analog inputs/outputs.

Arduino

Arduino is very popular electronics prototyping platform that makes everything easier for those of us that are amateurs on the electronics field. There are multiple Arduino boards, but I'll talk about the "Arduino Leonardo" which is the one that I bought. The Arduino Leonardo is a cheap board (you can get it for around $20) that includes a bunch of digital and analogs pins for input/output and a serial port which can be used to communicate the Arduino with a computer (this is also used to load the programs)

Arduino has its own IDE for coding and you code in an "arduino programming language" which is pretty similar to to C. One of the greatest things, is that it has an awesome community, so its really easy to find solutions to your problems and there are a lot of libraries to interact with sensors  and actuactors (ultrasonic for distance, infrared, servos, etc).

The arduino is awesome but it also has its limitations... it has a 16Mhz microprocessor and 32KB of memory. So, forget about doing heavy processing in the arduino.

Raspberry PI + Arduino = SUCCESS!

As you may have noticed, the Raspberry PI and the Arduino complement each other pretty well. The Raspberry PI is pretty powerful but it lacks of analog inputs/outputs which the arduino has. So, I think that the combination of both is a great success.

For my projects Im planning to communicate both boards through the serial interface (USB). The Arduino will just be an interface between the Raspberry PI and the hardware, exposing the sensors readings and getting commands for the actuator through a simple custom reusable serial protocol that Im going to code. And in the Raspberry PI I'll create a program for each project in the language that I want (for example python) and I can use all the power of the Raspberry PI to do the computation.