Free Open Source and Linux in Vietnam

The lsusb command in linux is very useful to find out the unique ids of your USB hardware devices in your computer.

For examle, right now if I do:

yen@yen-laptop:~$ lsusb

Bus 005 Device 003: ID 5986:0100 Bison Acer OrbiCam

Bus 005 Device 001: ID 0000:0000

Bus 004 Device 001: ID 0000:0000

Bus 003 Device 001: ID 0000:0000

Bus 002 Device 003: ID 046d:c01b Logitech, Inc. MX310 Optical Mouse

Bus 002 Device 001: ID 0000:0000

Bus 001 Device 001: ID 0000:0000

It is able to find my external USB Mouse and my integrated USB Camera.

Those 8 letter-digits that you can see are really useful to find help in the Internet to make your device work on your Linux distribution. For example if I search in google: 5986:0100 , I will get around 700 results of people talking about my exactly same model of Webcam and how to make it work in linux (in case it doesn’t work out of the box).

Then I wondered how is the program able to find the description of the device, example: “Bison Acer OrbiCam”, I guessed he was using some kind of internal database and I made a few more research. I realized that everything is stored in a file named usb.ids (Note: Freebsd structure is different).

That file can be found at:

/var/lib/misc/usb.ids which is maintained by SVN here.

and it’s maintained by the people of the linux-usb project.

To update your usb.ids files to the most recent one, you can use the command “sudo update-usbids” which basically will download it from their site.

So far so good, but around December last year, the database was still a bit poor (only around 4.000 USB devices) and it coudn’t find many of my stuff, so I decided to help them.

Profiting that I love python and python is perfect for text treatment(most easy and powerful language I’ve ever seen) and Web Scraping.

First thing I had to do is to find out there some sources to take USB device descriptions from USB Ids, after googling a bit I got important sources:

- The FreeBSD separated usb ids project. (479 new USB devices)

- Official USB.org list (278 new USB vendor ids)

-Some personal project site. (156 new usb ids)

-And the big pie…. the DriverAgent.com website(web scrapping from python script) (5470 new ids).

I have attached the python script that I created to extract all the new Ids.

[usb.py] and [driveragent.py]

Finally I only had to compare both files and do a patch to send them to the usdb.id maintainers.

I feel proud to think that I contributed to double the amount of the ids for that project. Let’s see if some day I dare to do the same for the lscpi command and database, anybody does? .

We could say I’m currently in holidays because I’m not officially working in any company, but I spend much part of my free time these days working in my delayed opensource projects. The first one I had to dedicate more time is SubDownloader, which is a Subtitle Automatic Downloader.

The project started few years ago because I need a way to not waste my time searching trought subtitles sites in order to find subtitles for my movies/series. I contacted Brano, the admin of a opensubtitles.org and together we created the API to connect my software with his site.

Why it saves so much time? Well, let’s say you have a folder full with 30 movies, that need subtitles, but you don’t want to go and search 1 by 1 all the titles of the movies in order to find subtitles.

Well, in SubDownloader, you just have to click on the folder that contains that movies and it will show up automatically the subtitles found in few seconds. Then you select the language that you want and voilà, they are downloaded easily.

But what assures me that the subtitle is good for my videos ? Well, due that the database is full of hashes of the videos, you can be sure that the user that upload that subtitle, was having exactly the same videofile that you have now, so that means that it fits almost perfect.

SubDownloader is , of course, Free Open Source Software, FOSS, it was started using WxPython but then I started to like QT so I moved to PyQT, the forge of the project is on Launchpad, so we use bazar, launchpad bugs and translations which makes our work more easy.

The number of downloads passed 1 million 6month ago, so we calculate will be around 1.5 millions by now, it’s a pitty that Launchpad doesn’t seem to care to support the Downloads Counter in their forge, so we cannot see statistics of our current downloads, but we observer that in peek time, there are around 1000 users using connected to the software server at the same time, which is awesome so far.

Plans for the future:

- Make contact with more subtitles sites to advertise the application.

- Make it torrent compatible so the subtitles can be shared by users instead of the current server-client link.

I also have to say that this software won the 1st University Open Source Contest of Spain in the category of Multimedia in 2007.

Also I need to say that free software can never be illegal and this one isn’t. We don’t own any server hosting subtitles, SubDownloader is just the client to the API of opensubtitles.org .
Our program is mainly based to be used with legal copies of movies, copyleft movies, public domain movies, so if any MPAA-like or SGAE(Spain) try to sue us, that only will give us a lot of publicity in the medias, so go ahead!!!

With the donations so far, we are going to hire some developers to work in Babiloo, our second open source project that I will talk soon here.

Now we are back in Vietnam, visiting my wife’s parents for a bit before coming back to Spain to start again. 2 years in Asia (Vietnam and Singapore) has been great but it’s good changing environments some times.

I’m left Singapore for personal reasons that make me return to Spain, but I’d like to say the good things and bad things of that mini-country:

Advantages:

- Salary (for those who are just graduated, Singapore gives around a 160% more than Spain).

- Technology and prices: Singapore is probably the cheapest place in all easy to buy technology, they have their famous SimLim Square where you can even bargain for the price of an GPS, PSP, laptop, etc.

-Internet: It’s not the fastest, but it’s very cheap, I rented a 3G USB modem of 512Kbps with unlimited bandwith for 12eur a month only.

- Organization (also see in disadvantages) : The whole system is organized until the smallest detail, constructions everywhere, reparations, consistent bureaucracy (most of the procedures can be done online).

- Education system and lack of racism: Singapore is a pot of cultures: Chinese, Malaysians, Indians, Indonesians, Filipinos, Europeans, etc. The education system is really good and strong, that’s probably the thing they are more proud about. Children in school learn to have friends from many different cultures and races, and after few generations, they live together peacefully. A important thing for Europeans to learn.

-Low criminality: That is true, you never feel scared walking at night time in Singapore, the police most of the time are bored in the offices. Government, thought, as spread around the country some banners that says: “Low criminality doesn’t mean no criminality”, and they are right, because they have got few cases very popular.

- Foodcourts: Eating outside in Singapore it can be very cheap (2 eur) thanks to many foodcourts spread around the country, near every office building you can find one, and in every shopping mall. The choices are quite good, from Japanese food to Chinese, Indian,Vietnamese, European, etc

Disadvantages:

- Big Brother feeling: Due that Singapore is so small and the efficacy of the burocracy is high, you cannot avoid to hold a feeling of being watched all the time. Singaporeans behave sometimes like robots, never doing naughty things and following the laws and rules strictly. As a example, when you are going to cross the street, most of Singaporeans will patiently wait until the green light appears, even no car has passed for the last 2min

-High beer price: Alcohol is almost “forbidden” due to its hight taxes, I guess to discourage their citizen to act not properly and maintain low the criminality rate, but at the same time makes the environment boring if you cannot afford going bars few times a month. Of course an European expat will tell you that Singapore is very fun, but then ask him how much money he spend in restaurants and drinks and you’ll how expensive is have to have “fun” with friends.

- Workaholics: is crazy how hard Singaporeans work for their own private companies, even knowing that their extra efforts just goes to bosses’ pockets. Having to have dome so many extra hours like the rest of my colleagues and living in such a closed country gave me a weird feeling of claustrophobia.

- Death penalty and no freedom of speech: Is not as bad as Vietnam but it’s very shame to have this in such a “developed” country, what is more shocking for me (as European) is that citizens don’t seem to care about it as long as their economy goes well and companies success. No Demonstrations are allowed. Also for example drug dealers face the death penalty in few weeks after being caught, even your plane was only doing scale in Singapore from neighbor countries.

-Senglish: For a non native English speaker like me sometimes is really hard to understand Singaporeans accent, funny called “Senglish”. It’s basically English with an strong Chinese accent, I forgot to say that Chinese race is majority in Singapore, around 60% of population.

I arrived last 21th October to Singapore. I’ve found a job in a Singapore Linux Company named Adeptiva Linux.
I miss really a lot Vietnam and Vietnamese people. Singapore it’s a too easy city and not as much exciting as Vietnam, anyway we need to move forward and some day I’ll come back for working.

Anyway my wife is Singapore, currently studying tourism here in Singapore so we’ll come back at least 2 times per year to Vietnam to visit friend’s and family.

The good thing about working in a Linux company is that everything is Open Source, I don’t need, like usually, to explain any boss about how or why to migrate to Open Source. I’m happily working in a Ubuntu Gutsy Dual Core and 23” screen with 4GB of Ram . The phones work with Asterisk, the mail with Zimbra, etc.

My mission here is Web Programmer oriented to new WebApplication frameworks like Groovy and Grails and QooxDoo Widgets, once I’ve got used to it I feel really comfortable and it’s amazing to see the results that these languages-frameworks can provide you in few moments.

I’ll try to get sometime in order to keep working in my 2 main Open Source community projects. Subdownloader and Babiloo, they are not dead, they will never be because they’re open

A Vietnamese governement delegation represented by Mr. Nguyen Trung Quynh, Mrs. Nguyen Thuy Nhi y Mr. Luu Van Khang has arrived to Badajoz, city of the province of Extremadura, Spain, the most advanced city involving Free Software in my country. In the photo we can see how Julio Fuster, town councillor of Merida (Extremadura’s capital) is gonna be their tour leader these days :-). I’m sure he will show them the beauties of that Region.

The interest of VietNam in this province is to see the initiatives of the Information Society and Free Software started in Extremadura.

Extremadura was one of the poorest region of Spain involving technology, but when in 2002 they took officially the decision of migrating everything to Free Software , the environment changed completely. Now all the eyes pay attention to this province and make very jealous the others(which still have price-agreements with Microsoft). Regions in Malaysia, UE, and now VietNam try to follow their steps. How to make a poor region become one of the leaders in Information Society and Software development, quickly answer: Free Software Open Source.

Extremadura created an linux distribution called gnuLinEx installed in all the governement computers, schools, etc. With the money saved in licenses, it has been possible to have more computers per student (2 per each) than any region in Europe.

During two days they will be discussing about strategies to be followed, agreements, etc. We’ll keep an eye on them. I hope VietNam governement will reinforce their bet for the OpenSource and decide to migrate all their minds to OpenSource. Let’s remind them than there are a lot of local Linux Distributions to be taken (Ubuntu-vi, Vubuntu, etc) available in Vietnamese language.

Totally 20.000 computers will migrate to OpenOffice suite instead of starting to pay the Microsoft licenses which in VietNam represents a lot of money saved.

Their reasons are mainly economical, but also they will enjoy some extra features:

A fantastic localization/translation of OpenOffice to Vietnamese and the SpellChecker in Vietnamese.

Original new here. http://english.vietnamnet.vn/tech/2007/09/739409/

In order to create a very good rich and extensible VN-EN dictionary (and other languages in the future for sure) I made a research of the current open source environment.

Dictionary formats(rich text, XML metalanguage):

We pretend our dictionaries to be extensible by the community, kind of wiki style, in order to do that, the dictionaries must be well formatted so we can recognize units like NOUN,VERB, DEFINITIONS,EXAMPLES, etc

XDXF (XML Dictionary Exchange Format) is one of the best and well structured languages. This format is good to make the computer understand what is each element and that way transform them to specific colors, sizes, etc. I hate to see the dictionaries in boring plain-text all the time.

The second advantage is the extensibility of the dictionaries by the communities,let’s see this example:

Let’s say that somebody wants to add the meaning for the “play” when used as a verb, so instead of messing the content adding new lines, change the colour, etc , the dictionary software should have the necessary tools(buttons) to allow the user to add new XML structures to the content.

StarDict format The structure is not XML, all byte-coded, it can contain Images,Audio,XDXF,HTML,Wiki links, I don’t really like this because it’s being created from StarDict and almost all the dictionaries are just plain-text not profiting this features.

Here there is a list of other formats.

Engine dictionary (compression,memory usage):

DICT It’s widely used because the StarDict files are compressed using this engine. Info,Index and Content are separated in different files. StarDict project has optimized the engine creating some extra files (cache, collation,etc)

Sdictionary. Info, Index and Content is compressed in the same file. There are tools to create files using HTML content.

Both engines allow a minimum usage of memory because only the index is loaded in memory and only the requested content is being taken by the compressed file.

Open Source Software dictionaries containers:

Our decision: If nobody changes their mind, I think we’ll go for the XDXF/Sdictionary/Jalingo combination to create the VN-EN project.

Last Sunday at Stella Cafe we had the collaboration of Thai Le Quoc and Nguyễn Tú Trinh , both JAVA experts and free software enthusiasts.
We talked about which strategie should we follow in order to create a very good content for a VN-EN dictionary, the main points were:

-Where to get the content.

-Which dictionary format(rich text, metatags) and Engine(compression, memory usage) should we follow.

-Which opensource dictionary container software should we use: (Jalingo JAVA, StarDict C++, QStardict C/QT )

I will publish a extended comparative of all that in my next post.

Here some photo of the meeting.

My spanish friend Luis Gomez produced and directed an excelent documentary about the homeless children in HCMC.

Street children and, in particular, the kids of one shelter in Ho Chi Minh City,
Vietnam, are the protagonists of this documentary; no one is more
aware of life in the streets of this city than they. So at this time
we are not the teachers, but the observers of their own reflections
and experiences.

His work is copyleft licensed for sure. Here the links:

http://www.archive.org/details/RunningAwayToBeHappy (AVI file good quality with English Subtitles)
http://www.archive.org/details/HuyendoParaSerFeliz (AVI file good quality with Spanish Subtitles)
http://video.google.com/videoplay?docid=-818178417051943991&hl=en (Online stream with English Subtitles)

Now Luis is traveling around Vietnam with his Minsk showing us his photos and his adventures. Finally he’ll try to come back to Spain BY LAND from Vietnam. This guy never stop surprising me all the time.

Leave him a comment on this blog.

After a big rain in HCMC

ME: left side , LUIS : right side