Fabric is a Python library and command-line tool for automating tasks of application deployment or system administration via SSH. It provides tools for executing local and remote shell commands and for transferring files through SSH and SFTP, respectively. With these tools, it is possible to write application deployment or system administration scripts, which allows to perform these tasks by the execution of a single command.

In order to work with Fabric, you should have Python and the Fabric Library installed on your local computer. Within this article, we consider using a Debian-based distribution on the local computer (such as Ubuntu, Linux Mint and others).

On the local computer, you may not need to install Python as it is shipped by default on most of the GNU/Linux distributions. To install the Fabric library, you may use pip. On Debian-based distributions, pip can be installed with apt-get through the python-pip package:

$ sudo apt-get install python-pip

After installing it, you may update it to the latest version using pip itself:

$ sudo pip install pip --upgrade

After that, you may use pip to install Fabric:

$ sudo pip install fabric

To work with Fabric, you must have SSH installed and properly configured with the necessary user’s permissions on the remote servers you want to work on. In the examples, we will consider a Debian System with IP address 192.168.250.150 and a user named “administrator” with sudo powers, which is required only for performing actions that require superuser rights.

One way to use Fabric is to create a file called fabfile.py containing one or more functions that represent the tasks we want to execute, for example:

File: fabfile.py

# -*- coding: utf-8 -*-

from fabric.api import *

env.hosts = ['192.168.250.150']
env.user  = 'administrator'

def remote_info():
    run('uname -a')

def local_info():
    local('uname -a')

In this example, we have defined two tasks called remote_info and local_info, which are used to retrieve local and remote systems information through the command uname -a. Also, we have defined the host user and address we would like to use to connect to the remote server using a special dictionary called env.

Having this defined, it is possible to execute one of the tasks using the shell command fab. For example, to execute the task local_info, from within the directory where fabfile.py is located, you may call:

fab local_info

which gives:

[192.168.250.150] Executing task 'local_info'
[localhost] local: uname -a
Linux renato-laptop 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Similarly, you could execute the task called remote_info, calling:

$ fab remote_info

In this case, Fabric will ask for the password of the user “administrator”, as it is connecting to the server via SSH:

[192.168.250.150] Executing task 'remote_info'
[192.168.250.150] run: uname -a
[192.168.250.150] Login password for 'administrator':
[192.168.250.150] out: Linux debian-vm 2.6.32-5-686 #1 SMP Sun May 6 04:01:19 UTC 2012 i686 GNU/Linux
[192.168.250.150] out:

Done.
Disconnecting from 192.168.250.150... done.

There are lots of parameters that can be used with the fab command. To obtain a list with a brief description of them, you can run fab --help. For example, using fab -l, it is possible to check the Fabric tasks available on the fabfile.py file. In this example, it outputs

Available commands:

    local_info
    remote_info

As in the previous example, on the file fabfile.py, the function run() may be used to run a shell command on a remote server and the function local() may be used to run a shell command on the local computer. Besides these, there are some other possible functions to use on fabfile.py:

  • sudo('shell command'): to run a shell command on the remote server using sudo;
  • put('local path', 'remote path'): to send a file from a local path on the local computer to the remote path on the remote server;
  • get('remote path', 'local path'): to get a file from a remote path on the remote server to the local path on the local computer.

Also, it is possible to set many other details about the remote connection with the dictionary env. To see a full list of env vars that can be set, visit http://docs.fabfile.org/en/1.6/usage/env.html#full-list-of-env-vars. Among the possible settings, its worth spend some time commenting on some of them:

  • user: defines which user will be used to connect to the remote server;
  • hosts: a Python list with the addresses of the hosts that Fabric will connect to perform the tasks. There may be more than one host, e.g.,
env.hosts = ['192.168.250.150','192.168.250.151']
  • host_string: with this setting, it is possible to configure a user and a host at once, e.g.
env.host_string = "administrator@192.168.250.150"

As it could be noticed from the previous example, Fabric will ask for the user’s password to connect to the remote server. However, for automated tasks, it is interesting to be able to make Fabric run the tasks without prompting for any user input. To avoid the need of typing the user’s password, it is possible to use the env.password setting, which permits to specify the password to be used by Fabric, e.g.

env.password = 'mysupersecureadministratorpassword'

If the server uses SSH keys instead of passwords to authenticate users (actually, this is a good practice concerning the server’s security), it is possible to use the setting env.key_filename to specify the SSH key to be used. Considering that the public key ~/.ssh/id_rsa.pub is installed on the remote server, you just need to add the following line to fabfile.py:

env.key_filename = '~/.ssh/id_rsa'

It is also a good security practice to forbid root from logging in remotely on the servers and allow the necessary users to execute superuser tasks using the sudo command. On a Debian system, to allow the “administrator” user to perform superuser tasks using sudo, first you have to install the package sudo, using

# apt-get install sudo

and then, add the “administrator” user to the group “sudo”, which can be done with the command

# adduser administrator sudo

Having this done, you could use the sudo() function on Fabric scripts to run commands with sudo powers. For example, to create a mydir directory within /home, you may use:

File: fabfile.py

# -*- coding: utf-8 -*-

from fabric.api import *

env.hosts = ['192.168.250.150']
env.user  = 'administrator'
env.key_filename = '~/.ssh/id_rsa'

def create_dir():
    sudo('mkdir /home/mydir')

and call

$ fab create_dir

which will ask for the password of the user “administrator” to perform the sudo tasks:

[192.168.250.150] Executing task 'create_dir'
[192.168.250.150] sudo: mkdir /home/mydir
[192.168.250.150] out:
[192.168.250.150] out: We trust you have received the usual lecture from the local System
[192.168.250.150] out: Administrator. It usually boils down to these three things:
[192.168.250.150] out:
[192.168.250.150] out:     #1) Respect the privacy of others.
[192.168.250.150] out:     #2) Think before you type.
[192.168.250.150] out:     #3) With great power comes great responsibility.
[192.168.250.150] out:
[192.168.250.150] out: sudo password:

[192.168.250.150] out:

Done.
Disconnecting from 192.168.250.150... done.

When using SSH keys to log in to the server, you can use the env.password setting to specify the sudo password, to avoid having to type it when you call the Fabric script. In the previous example, by adding,

env.password = 'mysupersecureadministratorpassword'

would be enough to make the script run without the need of user intervention.

However, some SSH keys are created using a passphrase, required to log in to the server. Fabric treat these passphrases and passwords similarly, which can sometimes cause confusion. To illustrate Fabric’s behavior, consider the user named “administrator” is able to log in to a remote server only by using his/her key named ~/.ssh/id_rsa2.pub, created using a passphrase, and the following Fabric file:

File: fabfile.py

# -*- coding: utf-8 -*-

from fabric.api import *

env.hosts = ['192.168.250.150']
env.user  = 'administrator'
env.key_filename = '~/.ssh/id_rsa2'

def remote_info():
    run('uname -a')

def create_dir():
    sudo('mkdir /home/mydir')

In this case, calling

fab remote_info

makes Fabric ask for a “Login password”. However, as you shall notice, this “Login password” refers to the necessary passphrase to log in using the SSH key:

[192.168.250.150] Executing task 'remote_info'
[192.168.250.150] run: uname -a
[192.168.250.150] Login password for 'administrator':
[192.168.250.150] out: Linux debian-vm 2.6.32-5-686 #1 SMP Sun May 6 04:01:19 UTC 2012 i686 GNU/Linux
[192.168.250.150] out:

Done.
Disconnecting from 192.168.250.150... done.

In this case, if you specify the env.password setting, it will be used as the SSH passphrase and, when running the create_dir script, Fabric will ask for the password of the user “administrator”. To avoid typing any of these passwords, you may define env.password as the SSH passphrase and, within the function that uses sudo(), redefine it as the user’s password:

File: fabfile.py

# -*- coding: utf-8 -*-

from fabric.api import *

env.hosts = ['192.168.250.150']
env.user  = 'administrator'
env.key_filename = '~/.ssh/id_rsa2'
env.password = 'sshpassphrase'

def remote_info():
    run('uname -a')

def create_dir():
    env.password = 'mysupersecureadministratorpassword'
    sudo('mkdir /home/mydir')

Alternatively, you could specify the authentication settings from within the task function:

File: fabfile.py

# -*- coding: utf-8 -*-

from fabric.api import *

env.hosts = ['192.168.250.150']

def create_dir():
    env.user  = 'administrator'
    env.key_filename = '~/.ssh/id_rsa2'
    env.password = 'sshpassphrase'
    run(':')
    env.password = 'mysupersecureadministratorpassword'
    sudo('mkdir /home/mydir')

On this example, the command ‘:’ does not do anything. It only serves as a trick to enable setting env.password twice: first for the SSH passphrase, required for login and then to the user’s password, required for performing sudo tasks.

If necessary, it is possible to use Python’s with statement (http://www.python.org/dev/peps/pep-0343/), to specify the env settings. A compatible create_dir() task using the with statement, would be:

File: fabfile.py

# -*- coding: utf-8 -*-

from fabric.api import *

env.hosts = ['192.168.250.150']

def create_dir():
    with settings(user  = 'administrator',
                  key_filename = '~/.ssh/id_rsa2',
                  password = 'sshpassphrase'):
        run(':')
        env.password = 'mysupersecureadministratorpassword'
        sudo('mkdir /home/mydir')

The fab command is useful to performing system administration and application deployment tasks from a shell console. However, sometimes you may want to execute tasks from within your Python scripts. To do this, you may simply call the Fabric functions from your Python code. To build a script that runs a specific task automatically, such as create_dir() shown above, you may do something like:

File: mypythonscript.py

#! /usr/bin/env python
# -*- coding: utf-8 -*-

from fabric.api import *

def create_dir():
    with settings(host_string  = 'administrator@192.168.250.150',
                  key_filename = '~/.ssh/id_rsa2',
                  password = 'sshpassphrase'):
        run(':')
        env.password = 'mysupersecureadministratorpassword'
        sudo('mkdir /home/mydir')

if __name__ == '__main__':
    create_dir()

As we have seen, with Fabric, it is possible to automate the execution of tasks that can be done by executing shell commands locally, and remotely, using SSH. It is also possible to use Fabric’s features on other Python scripts, and perform dynamic tasks, enabling the developer to automate virtually anything that can be automated. The main goal of this article was to show Fabric’s basic features and try to show a solution to different scenarios of remote connections, regarding different types of authentication. From this point, you may customize your Fabric tasks to your needs using basically the functions local(), run() and sudo() to run shell commands and put() and get() to transfer files.

To conclude, we show a more practical example of a Python script that uses Fabric to deploy a very basic HTML application on a server. The script creates a tarball from the local HTML files at ~/website, sends it to the server, expands the tarball and moves the files to the proper directory (/var/www/website) and restarts the server. Hope this article helped you learning a bit about Fabric to automate some tasks!

#! /usr/bin/env python
# -*- coding: utf-8 -*-

from fabric.api import *

def deploy_html():
    with settings(host_string  = 'administrator@192.168.250.150',
                  key_filename = '~/.ssh/id_rsa2',
                  password = 'sshpassphrase'):
        run(':')
        env.password = 'mysupersecureadministratorpassword'
        local('cd ~; tar -czvf website.tar.gz ./website/*')
        put('~/website.tar.gz', '~')
        run('tar -xzvf ~/website.tar.gz')
        sudo('mv /home/administrator/website /var/www')
        sudo('chown -R www-data:www-data /var/www/website')
        sudo('/etc/init.d/apache2 restart')
        local('rm ~/website.tar.gz')

if __name__ == '__main__':
    deploy_html()

Ethernet Shield enables us to connect Arduino to an ethernet network and build projects which send and receive information via local network and internet. The device, along with its control library, allows the design of network applications easily. There is no need to worry about the complicated signaling schemes in ethernet networks since all control and connection are made by library routines. After the connection is established, we may just read or write data through it, similarly to a conventional serial connection.

In the following, we show an example of a project of a web page that allows you to turn on or turn off a digital port of the Arduino via the local network or the internet. In a practical application, the project could be used to control a lamp, serving as a good “Hello World” application for home automation with Arduino. In this example, the idea is to focus on the design of the software required for the application so that we will consider the connection of led to the Arduino port and show one of the most expensive ways to light a led.

In this project, we used the Ethernet Shield based on the Wiznet W5100 chip.This chip provides all the TCP/IP stack transparently, turning the project very simple since there is no need to worry about the signaling schemes and controls for TCP/IP. It is worth to notice that there are other ethernet shields, simpler and cheaper, using chips that require to control the TCP/IP signaling via software implementation in the Arduino sketch.

The hardware configuration for this project is very simple: just connect the Ethernet Shield to the Arduino and connect a led in series with a 1 KOhm resistor between pin 6 of the Shield and ground (GND).

The idea is to access the IP address associated with the Arduino using a browser and get a page with a link to turn on the led and another to turn it off. For this, we used the following code:

#include <SPI.h>
#include <Ethernet.h>
 
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 192, 168, 1, 235 };
    
EthernetServer server(80);
 
String readString;
int Pin = 6;
 
void setup(){
 
  pinMode(Pin, OUTPUT);
  Ethernet.begin(mac, ip);
  server.begin();
}
 
void loop(){
  EthernetClient client = server.available();
  if (client) {
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
 
        if (readString.length() < 100) {
          readString += c;              
        }

        if (c == '\n') {
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println();
          
          client.println("<HTML>");
          client.println("<BODY>");
          client.println("<H1>Control LED</H1>");
          client.println("<hr />");
          client.println("<br />");
          
          client.println("<a href=\"/?ledon\"\">Turn on led</a>");
          client.println("<a href=\"/?ledoff\"\">Turn off led</a><br />");    
          
          client.println("</BODY>");
          client.println("</HTML>");
          
          delay(1);
          client.stop();
          
          if(readString.indexOf("?ledon") > 0)
          {
            digitalWrite(Pin, HIGH);
          }
          else {
            if(readString.indexOf("?ledoff") > 0)
            {
              digitalWrite(Pin, LOW);
            }
          }
          readString="";     
        }
      }
    }
  }
}

To control the Ethernet Shield, the Ethernet library is used . The communication between Arduino and the Shield is made using the SPI protocol, hence the need to import the SPI library as well. Remember that communication via SPI is done via pins 10, 11, 12 and 13 on the Arduino Uno, which cannot be used in the design.

The configuration of the server running on port 80 is made by lines

EthernetServer server(80);
Ethernet.begin(mac, ip);
server.begin();

where the MAC and IP addresses are defined by the vectors mac and ip created in lines 4 and 5.

After the initial setup, the program waits for a client connection and, when it detects it, it reads the characters sent to the server regarding the GET request made by the client though the browser.

The characters are received one by one and are stored sequentially in the ReadString variable. A conditional is used to limit the number of characters received at 100. At each character received, a test is made to check if it is an end of line (\n, an “ENTER”). In this case, a successful response is sent to the request of the client (HTTP/1.1 200 OK ), containing an HTML page with links to the turn on and turn off the led.

The links to turn on and turn off the led use the query strings ?ledon and ?ledoff so that the server can identify the command from the client. After responding to the request, a test is made to check if the address requested by the client contains one of these query strings, by checking the variable ReadString for the presence of the sequences ?ledon or ?ledoff. From this information, the led is turned on or off, and finally, the variable ReadString is restarted.

This way, it is possible to control the led from an access point to the LAN. It is also possible to control it via internet, configuring a port forward of the port 80 in the router to the IP associated with the Arduino and accessing the page using the public IP provided by the internet provider. However, it is worth mentioning that this setting does not take into account the any security issues, allowing anyone with the public IP to control the led via internet.

Listening to the Linux Action Show podcast, I heard about a Python script to backup e-mails from IMAP accounts that downloads messages and attachments and offers everything on a nice local HTML page.

The script is called NoPriv.py and can be found on this page and on this Github repository. Given an IMAP e-mail account and a list of folders to be copied, it creates the HTML files structure to access the messages as shown in this demo page. The backup can be made incrementally, transfering only the new messages each time the script is run.

To install it, just clone the Github repository and ajust the configurations on the nopriv.py file:

###########################
# Do not edit above here  #
###########################

IMAPSERVER = "imap.gmail.com"
IMAPLOGIN = "janeway@gmail.com"
IMAPPASSWORD = "Voyager1"
IMAPFOLDER = ["[Gmail]/Sent Mail", "INBOX", "[Gmail]/Starred", "Captains_Log",
              "Important"]
ssl = True
incremental_backup = True

###########################
# Do not edit below here  #
###########################

The list IMAPFOLDER must contain the names of the folders that are going to be copied. It is worth noticing that when using Gmail server, the default folder names must be preceded by the [Gmail] prefix whereas the names of the folders created by the user must not have this prefix.

To start the copy, just run:

python nopriv.py

When finished, just open the index.html file and navigate through the messages and attachments. The file can also be opened on command line navigators such as Liks2.

The right to read

“The right to read” is an article by Richard Stallman which was published formerly on an issue of Communications of the ACM and later in the book “Free Software, Free Society: Selected Essays of Richard M. Stallman“. In the article, he discusses about the possible changes regarding the reader’s freedom due to adoption of e-books instead of traditional books. Good reading!

The right to read

philosophical-gnu-sm[This article appeared in the February 1997 issue of Communications of the ACM (Volume 40, Number 2)].

From The Road To Tycho, a collection of articles about the antecedents of the Lunarian Revolution, published in Luna City in 2096.

For Dan Halbert, the road to Tycho began in college—when Lissa Lenz asked to borrow his computer. Hers had broken down, and unless she could borrow another, she would fail her midterm project. There was no one she dared ask, except Dan.

This put Dan in a dilemma. He had to help her—but if he lent her his computer, she might read his books. Aside from the fact that you could go to prison for many years for letting someone else read your books, the very idea shocked him at first. Like everyone, he had been taught since elementary school that sharing books was nasty and wrong—something that only pirates would do.

And there wasn’t much chance that the SPA—the Software Protection Authority—would fail to catch him. In his software class, Dan had learned that each book had a copyright monitor that reported when and where it was read, and by whom, to Central Licensing. (They used this information to catch reading pirates, but also to sell personal interest profiles to retailers.) The next time his computer was networked, Central Licensing would find out. He, as computer owner, would receive the harshest punishment—for not taking pains to prevent the crime.

Of course, Lissa did not necessarily intend to read his books. She might want the computer only to write her midterm. But Dan knew she came from a middle-class family and could hardly afford the tuition, let alone her reading fees. Reading his books might be the only way she could graduate. He understood this situation; he himself had had to borrow to pay for all the research papers he read. (Ten percent of those fees went to the researchers who wrote the papers; since Dan aimed for an academic career, he could hope that his own research papers, if frequently referenced, would bring in enough to repay this loan.)

Later on, Dan would learn there was a time when anyone could go to the library and read journal articles, and even books, without having to pay. There were independent scholars who read thousands of pages without government library grants. But in the 1990s, both commercial and nonprofit journal publishers had begun charging fees for access. By 2047, libraries offering free public access to scholarly literature were a dim memory.

There were ways, of course, to get around the SPA and Central Licensing. They were themselves illegal. Dan had had a classmate in software, Frank Martucci, who had obtained an illicit debugging tool, and used it to skip over the copyright monitor code when reading books. But he had told too many friends about it, and one of them turned him in to the SPA for a reward (students deep in debt were easily tempted into betrayal). In 2047, Frank was in prison, not for pirate reading, but for possessing a debugger.

Dan would later learn that there was a time when anyone could have debugging tools. There were even free debugging tools available on CD or downloadable over the net. But ordinary users started using them to bypass copyright monitors, and eventually a judge ruled that this had become their principal use in actual practice. This meant they were illegal; the debuggers’ developers were sent to prison.

Programmers still needed debugging tools, of course, but debugger vendors in 2047 distributed numbered copies only, and only to officially licensed and bonded programmers. The debugger Dan used in software class was kept behind a special firewall so that it could be used only for class exercises.

It was also possible to bypass the copyright monitors by installing a modified system kernel. Dan would eventually find out about the free kernels, even entire free operating systems, that had existed around the turn of the century. But not only were they illegal, like debuggers—you could not install one if you had one, without knowing your computer’s root password. And neither the FBI nor Microsoft Support would tell you that.

Dan concluded that he couldn’t simply lend Lissa his computer. But he couldn’t refuse to help her, because he loved her. Every chance to speak with her filled him with delight. And that she chose him to ask for help, that could mean she loved him too.

Dan resolved the dilemma by doing something even more unthinkable—he lent her the computer, and told her his password. This way, if Lissa read his books, Central Licensing would think he was reading them. It was still a crime, but the SPA would not automatically find out about it. They would only find out if Lissa reported him.

Of course, if the school ever found out that he had given Lissa his own password, it would be curtains for both of them as students, regardless of what she had used it for. School policy was that any interference with their means of monitoring students’ computer use was grounds for disciplinary action. It didn’t matter whether you did anything harmful—the offense was making it hard for the administrators to check on you. They assumed this meant you were doing something else forbidden, and they did not need to know what it was.

Students were not usually expelled for this—not directly. Instead they were banned from the school computer systems, and would inevitably fail all their classes.

Later, Dan would learn that this kind of university policy started only in the 1980s, when university students in large numbers began using computers. Previously, universities maintained a different approach to student discipline; they punished activities that were harmful, not those that merely raised suspicion.

Lissa did not report Dan to the SPA. His decision to help her led to their marriage, and also led them to question what they had been taught about piracy as children. The couple began reading about the history of copyright, about the Soviet Union and its restrictions on copying, and even the original United States Constitution. They moved to Luna, where they found others who had likewise gravitated away from the long arm of the SPA. When the Tycho Uprising began in 2062, the universal right to read soon became one of its central aims.

Author’s Note

[This note has been updated several times since the first publication of the story.]

The right to read is a battle being fought today. Although it may take 50 years for our present way of life to fade into obscurity, most of the specific laws and practices described above have already been proposed; many have been enacted into law in the US and elsewhere. In the US, the 1998 Digital Millennium Copyright Act (DMCA) established the legal basis to restrict the reading and lending of computerized books (and other works as well). The European Union imposed similar restrictions in a 2001 copyright directive. In France, under the DADVSI law adopted in 2006, mere possession of a copy of DeCSS, the free program to decrypt video on a DVD, is a crime.

In 2001, Disney-funded Senator Hollings proposed a bill called the SSSCA that would require every new computer to have mandatory copy-restriction facilities that the user cannot bypass. Following the Clipper chip and similar US government key-escrow proposals, this shows a long-term trend: computer systems are increasingly set up to give absentees with clout control over the people actually using the computer system. The SSSCA was later renamed to the unpronounceable CBDTPA, which was glossed as the “Consume But Don’t Try Programming Act”.

The Republicans took control of the US senate shortly thereafter. They are less tied to Hollywood than the Democrats, so they did not press these proposals. Now that the Democrats are back in control, the danger is once again higher.

In 2001 the US began attempting to use the proposed Free Trade Area of the Americas (FTAA) treaty to impose the same rules on all the countries in the Western Hemisphere. The FTAA is one of the so-called free trade treaties, which are actually designed to give business increased power over democratic governments; imposing laws like the DMCA is typical of this spirit. The FTAA was effectively killed by Lula, President of Brazil, who rejected the DMCA requirement and others.

Since then, the US has imposed similar requirements on countries such as Australia and Mexico through bilateral “free trade” agreements, and on countries such as Costa Rica through another treaty, CAFTA. Ecuador’s President Correa refused to sign a “free trade” agreement with the US, but I’ve heard Ecuador had adopted something like the DMCA in 2003.

One of the ideas in the story was not proposed in reality until 2002. This is the idea that the FBI and Microsoft will keep the root passwords for your personal computers, and not let you have them.

The proponents of this scheme have given it names such as “trusted computing” and “Palladium”. We call it “treacherous computing” because the effect is to make your computer obey companies even to the extent of disobeying and defying you. This was implemented in 2007 as part of Windows Vista; we expect Apple to do something similar. In this scheme, it is the manufacturer that keeps the secret code, but the FBI would have little trouble getting it.

What Microsoft keeps is not exactly a password in the traditional sense; no person ever types it on a terminal. Rather, it is a signature and encryption key that corresponds to a second key stored in your computer. This enables Microsoft, and potentially any web sites that cooperate with Microsoft, the ultimate control over what the user can do on his own computer.

Vista also gives Microsoft additional powers; for instance, Microsoft can forcibly install upgrades, and it can order all machines running Vista to refuse to run a certain device driver. The main purpose of Vista’s many restrictions is to impose DRM (Digital Restrictions Management) that users can’t overcome. The threat of DRM is why we have established the Defective by Design campaign.

When this story was first written, the SPA was threatening small Internet service providers, demanding they permit the SPA to monitor all users. Most ISPs surrendered when threatened, because they cannot afford to fight back in court. One ISP, Community ConneXion in Oakland, California, refused the demand and was actually sued. The SPA later dropped the suit, but obtained the DMCA, which gave them the power they sought.

The SPA, which actually stands for Software Publishers Association, has been replaced in its police-like role by the Business Software Alliance. The BSA is not, today, an official police force; unofficially, it acts like one. Using methods reminiscent of the erstwhile Soviet Union, it invites people to inform on their coworkers and friends. A BSA terror campaign in Argentina in 2001 made slightly veiled threats that people sharing software would be raped.

The university security policies described above are not imaginary. For example, a computer at one Chicago-area university displayed this message upon login:

This system is for the use of authorized users only. Individuals using this computer system without authority or in the excess of their authority are subject to having all their activities on this system monitored and recorded by system personnel. In the course of monitoring individuals improperly using this system or in the course of system maintenance, the activities of authorized user may also be monitored. Anyone using this system expressly consents to such monitoring and is advised that if such monitoring reveals possible evidence of illegal activity or violation of University regulations system personnel may provide the evidence of such monitoring to University authorities and/or law enforcement officials.

This is an interesting approach to the Fourth Amendment: pressure most everyone to agree, in advance, to waive their rights under it.

More courses on the web

It’s been a while since the emergence of university courses on the web. Currently, almost all major universities maintain projects regarding on line courses such as Open Course Ware provided by MIT and Open Yale Courses, by Yale University.

In these projects, at first, only written material such as student’s notes and exercises were provided. However, with the arise of video distribution through the web, it didn’t take long before cameras were used to record the classes and make the videos available on the web.

This initiative seems to be very nice and seems like it is working as more projects like these appear every month. Among them, it is worth mentioning Academic Earth, which serves as an index to other courses of many universities, Udemy, which besides indexing university courses, also houses some free courses and Khan Academy, which indexes a series of short videos with lessons in many areas such as math, physics, and finance.

I don’t believe that e-learning can replace the traditional university courses very soon but I think the idea is very interesting and must be improved and explored more and more. In this context, Stanford University brought an innovation in the last months of the year. It launched three courses that go beyond the simple provision of multimedia material, with the proposal of classes and exercises with monitoring by teachers according to predetermined schedule. In these courses, of introduction to artificial intelligence, introduction to databases and, machine learning, the students participate handing in exercises and doing tests and can obtain a certificate at the end of the course. The participation is not worth credits for the university but the result for the student ends up being much better in this model of course. The idea seems to be working since the university recently announced that intends to increase the number of courses available in 2012, including game theory, natural language processing, SaaS software engineering, probabilistic graphical models, computer science, human-computer interaction, technology entrepreneushio, cryptography, information theory, anatomy and, design and analysis of algorithms.

Hello World with Arduino

In 2005, I came across a project called Arduino, which drew my attention since it was a project of open source hardware. At the same year, I tried to assemble one but I ended leaving the project aside due to the college activities. Today, I regret I haven’t dedicated more some time to this project in the past, given the way it grew.

For those who don’t know, Arduino is a project composed by hardware and software for the development of applications which interact with the real world using sensors and actuators. It was conceived to permit the creation of projects by people who don’t have much knowledge on hardware or electronics such as hobbyists, artists and designers.

There are a lot of Arduino versions available, from the most common, called Arduino Uno to special ones like the LilyPad Arduino which was meant to be sewn on clothes. Besides that, one of the most interesting features of the project is the possibility to install shields, which are expansion boards that easily enables to use Arduino on applications such as ethernet, bluetooth, motor control, etc.

The applications depend only on the creativity of the developer :-). Check out some examples:

It took a while but I began to learn a little about the platform. Today, it is very easy to obtain an Arduino. On the project site, there is a list of stores that sell the “original” Arduino but since it is an open hardware platform, there are other manufacturers that sell arduino-like boards, that can be found on eBay or on Deal Extreme.

Like I mentioned, Arduino also includes a software project, including a development environment which eases the process of programming the microcontroller. In this environment, you can type your program and upload it to the Arduino board through a USB cable very easily.

I haven’t developed anything interesting on Arduino yet but, to have an idea of a program, here is an example of a program that blinks a LED connected to port 13, also known as the “Hello World” program on Arduino:

int ledPin = 13;                 // LED connected to pin #13

void setup()
{
  pinMode(ledPin, OUTPUT);      // configures the digital pin as output
}

void loop()
{
  digitalWrite(ledPin, HIGH);   // turns on the led
  delay(1000);                  // waits 1 second
  digitalWrite(ledPin, LOW);    // turns off the led
  delay(1000);                  // waits one second
}

The programming language is based on the Wiring Language, which is based on Processing and has a syntax similar to C. Basically, the program consists of two functions: setup and loop. The setup of the project is done on the setup function, in this case, the configuration of the LED pin as an output. The loop function, houses the code that will be executed indefinitely by the microcontroller. In this case, the LED will be turned on followed by a pause of one second and, after that, it will be turned off followed by a pause of one second.

For a first contact with Arduino, I recommend the book “Getting started with Arduino“, by Massimo Banzi, which is one of the creators of the project. In this book, basic functions to interact with sensors, actuators and the serial port are described. Besides that, it’s worth looking for a Hackerspace to visit on your town, where there usually are lots of interesting stuff using Arduino.

Lately, I’ve been studying a little about Python and the more I learn the more I become interested about this language. It was the first time I had contact with a language that uses dynamic typing and since the first time I used it, I had the impression it could be a good replace for Matlab. I like Matlab, but the fact of being proprietary and not being a general-purpose language makes Python a better solution for me.

For scientific applications, Python counts on three fundamental libraries:

  • NumPy, which is the fundamental library for scientific applications in Python. It offers special types for matrices and vectors representation and also other tools for linear algebra and random numbers;
  • SciPy, which depends on NumPy provides tools for working on many areas such as numerical integration, optimization and signal processing. Scipy libraries can be thought as replacements for Matlab toolboxes;
  • Matplotlib, which permits to plot graphs for visualizing data.

Some NumPy functions have similar syntax of its Matlab counterpart, which makes things easier for those who use Matlab and want to migrate to Python. For example, to create a vector with 1000 elements ranging from 0 to 1, linearly spaced, you should do:

import numpy as np
x = np.linspace(0, 1, 1000)

In this example, x will be an object with type array representing the vector with 1000 elements. Although the similarities with Matlab, there are also some differences. To create a vector with the elements [1, 2, 3], for example, you should do:

import numpy as np
x = np.array([1, 2, 3])

To reference the vector items, brackets must be used and the initial index is zero, i. e., it should be used x[0] to reference the first item of the vector x.

In this example, besides the syntax, there is another significant difference from Matlab: unlike Matlab, which represents every matrix or vector with at least two dimensions, Python allows to create one-dimension vectors. Thus, on the previous example, a call like x[0,0] would return an exception opposing to the Matlab behavior, which allows the calls x(1) or x(1,1) for the first element of the vector [1, 2, 3]. However, Python also permits to create vectors with two dimensions. The code to create a vector similar to the one from the previous example, with two dimensions is:

import numpy as np
x = np.array([[1, 2, 3]])

It is worth to notice that two brackets must be used: the external one related to the matrix (in this case with 1 line and 3 columns) and the internal one related to the first line of the matrix. In this case, x[0] references the first line of the matrix, i.e, the one-dimension vector [1, 2, 3]. To reference the first item of this vector, due to the two-dimension representation, we must use x[0,0]

Another significant difference is the matrix multiplication syntax. In Python, to multiply matrices created as arrays, we use the function dot. For example:

import numpy as np
# 2x2 Matrix A = [1 2
#                 3 4]
A = np.array([[1, 2], [3, 4])
# 2x2 Matrix B = [5 6
#                 7 8]
B = np.array([[5, 6], [7, 8])
# C = A*B
C = dot(A,B)

Besides that, to perform a element-wise multiplication such as done with the .* operator in Matlab, we must use a single asterisk in Python.

There are also several other differences to Matlab. On the page at http://www.scipy.org/NumPy_for_Matlab_Users, the most important are described. Furthermore, this document is also an excellent reference to learn the basic differences between Matlab and Python.

Regarding matplotlib, I haven’t done much yet but it seems that the syntax is very similar to the one used in Matlab. To plot a period of a sine wave, for example, we must run:

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0, 2*np.pi, 1000)
y = np.sin(x)
plt.plot(x,y)
plt.show()

Last, I cannot forget to mention iPython. It is an enhanced Python shell with many improvements such as auto-completion and access to the history of commands. Besides that, on the last version there are a lot of new features, as shown on SciPy 2011.

Another interesting Emacs mode is Org-Mode. It permits to make annotations and manage to-do lists and projects using text files with some special marks.

The marks as interpreted by Emacs, so that it separates the various parts of the text, which facilitates obtaining information. It is possible to organize an agenda with appointments set up on specific dates and also tasks without due dates. With every agenda item, it is possible to associate tags, which can be used to filter search results. These features make Org-Mode a good resource for those who use GTD. For those who don’t know GTD, it is worth to take a look. It was proposed by David Allen and is described in the book “Getting things done”. A brief description of the method can be seen on this presentation which was given at Google and on this reference card.

Besides the agenda, it is possible to create lists and texts, which can be exported to HTML or Latex, directly from Emacs.

One of the nicest features of Org-Mode is the capability to create tables. By inserting delimiting characters, it is possible to define a table and, as characters are inserted, the cells of table are re-sized appropriately. Moreover, it is possible to make calculations with the data on the table as on a spreadsheet, using Emacs Calc. Although it works with a simple text file, Emacs Calc is able to do more than simple calculations, as shown on the following video.

As the whole system is stored on a text file, it is possible to edit and share the information easily. Besides that, for those who have a device with Apple iOS or Android, there is an app called MobileOrg that can be used to consult Org-Mode files. It is prepared to access an agenda file on a Dropbox account, which facilitates the integration with the PC and another devices.

To know a bit more about Org-Mode, I recommend this episode of the FLOSS Weekly Podcast, with the interview of Carsten Dominik, who proposed the tool.

Some years ago, one of the problems I had when I was migrating my desktop to GNU/Linux was finding a development environment for LaTeX. For those who don’t know LaTeX, it’s worth checking out. It is a “document preparation system” widely used for producing mathematical and scientific texts due to its high typographic quality.

By the time I was using MS Windows, I used TeXnicCenter to edit LaTeX projects. Despite its limitations, it did its job well done. On GNU/Linux, I initially tried the Kile editor. Although I consider it an excellent editor, I missed a basic feature: spell correction on the fly (those red marks that are shown under the words when they are spelled incorrectly). Perhaps I had this problem because I was trying to use Kile with GNOME as it was originally developed to be used with KDE.

Because of this issue, I started looking for other options to edit LaTeX files and I found out a very nice solution with Emacs and AUCTEX. At first, I was avoiding using Emacs because I’ve heard it was an editor that was difficult to use. I must admit that it is necessary some training to get used to the way of working with Emacs but in the case of editing LaTeX files, its worth the effort.

AUCTEX is an extension package to ease the process of editing LaTeX files on Emacs. With AUCTEX, many processes are automatized. To insert a block that defines an environment, such as equation, for instance, you only need to type “C-c e” followed by the command equation. After that, some parameters will be requested, such as the label to be used with the equation, and finally the environment will be created. The bibliographic citations and cross-references can be easily inserted in the text with the commands “C-c [” and “C-c (” respectively. One of the most impressive features, which I haven’t seen in any other editor, is the capability to preview mathematical equations, tables and figures inside the editor. This feature can speed up the process of setting the sizes of these elements as it eliminates the need of compiling the whole project to check the result.

Besides these features, AUCTEX has many other modes. Using it with some other programs makes it possible to have a very practical development environment for LaTeX documents. The following video shows some features of Emacs with AUCTEX. Although it is a little old and in italian, it is worth watching to have an idea of the possibilities.

Hello World!

Welcome to my personal blog! I’ve started this because I needed to save some information for further reference and I think a blog is an good alternative to do so. The information is about a variety of subjects: softwares I’ve tested, computer simulations, texts, links and service or product impressions I consider worth. Moreover, I will use this space to practice my writing which needs some improvement 🙂

Currently, I’m working towards the Ph. D. degree in electrical engineering and I intend to publish some of the things I’m studying. Besides that, I’m starting a project on software development and I’d like to publish the experience I’m having with analysis, data collection and the development itself.

Since 2005, when I got in touch with the free software culture, I’ve been studying and using many free software solutions. During the last years, I’ve been migrating my desktop environment and today, almost all the software I use is free software. I’d like to write about some of these softwares, including the experiences I’m having with Emacs and some of its “modes” as my development environment and Python for application development and computer simulations.

I hope I find people interested on these subjects so that we can discuss and learn new stuff. Feel free to contact me and participate on the comments 🙂

[]’s

Renato Candido.