Yes, this post is indeed about a Quick and Dirty tool to “Convert a Wammu backup file to a .csv” ready for gMail import. I dicked around with the wammu.backup file to get it to plain text for reference and then figured Id better write some quick perl to get it to a .csv so I could import that into my gMail account. Since my Android phone automagically synchronizes with my gMail contacts (gotta love it)
I started writing the perl after I dicked around a bit with the file. The file is by default DOS formatted, illegible and apparently encoded in utf-16. Since I first wanted it as a reference (to grep, cat or less in a term window) so I did the following. You ll have to do this too (do feel free to submit the Perl code in the comments to skip this part ;))
- Open the file using vi.
- Save file (yes, save BEFORE editing, this converts the binary to a regular file)
:w ConvertMe.txt
- If the conversion using vi doesnt work for you, you can also use flip (Thanks to AK for the tip, check his comment for more info!)
flip -ub ConvertMe.txt
- If the conversion using vi doesnt work for you, you can also use flip (Thanks to AK for the tip, check his comment for more info!)
- Set unix encoding
:set ff=unix
- Reset the encoding
:set fileencoding=default
- Save! (use the :w)
Next part is the Perl script. (The code is after the jump..)
The script reads a file called ConvertMe.txt – You can change this in the script, but really, why 🙂
- run convert.pl , this will output the .csv on the screen
- run convert.pl > converted.csv to get the google ready csv..
Importing into gMail is really simple. Login and click “Contacts” (Left sidebar), click “Import” (on top of the contacts list to the right) and import it using the menu. I imported slightly over 260 contacts and decided it was a bad idea seconds later since I now have a lot of work to sort those out. Thats a pain in the arse. And Im not sure why, but the find duplicates doesnt find nearly half of them. Apparently it doesnt compare the phone nrs. Probably something to do with false positives.. Cool option though.. I DID import them into a separate category and that does help! Beats doing them by hand..
What if this doesnt work?
There are three options for output..
You ll find the following in the code. Its not that long, do a search 😉
# Print de codes?
my $codes = 1;
to print all the codes in the file you re having parsed. This reads the file and spews out all the different tags in there in an easy list.
# Print de raw data
my $raw = 1;
Print the raw data. Prints all the tags for every contact, so you dan easilly check if all looks right
# Print de geparste code
my $parse = 1;
This is enabled by default. This option parses all your data into the .csv format.
If the last fails, its possible you need to change or add to the tags described in the .csv file. This defenition starts at line 66, the # Parse data line. First check what tags your file provides by setting the $codes option and then fill these in on the right lines in the script. Dont remove the commas or the .csv will fail to parse correctly. Just add the fields in front, analogous to the current settings. Dont worry about the tags that wont be filled, they will be empty. Dont worry, it ll work 😉
The Source.. Moved to http://www.gertschepens.be/wammu2cvs-code
Edit: People have been experiencing problems with the code and after a long and hard look I discovered the error to be the blog having stripped the <MYFILE> from the code, thinking it to be faulty html and thus creating an endless loop.. This explains a lot 😉 Kudos to AK for helping me sort it all out! And sorry for the mess..
Edit2: Code moved to http://www.gertschepens.be/wammu2cvs because of formatting problems in this post.
blog.cone 
your perl script doesn’t work for me (Debian squeeze)
running in debug mode (-d):
$ ./convert.pl
Loading DB routines from perl5db.pl version 1.32
Editor support available.
Enter h or `h h’ for help, or `man perldebug’ for more help.
main::(./convert.pl:4): my $id;
DB q
LikeLike
main::(./convert.pl:4): my $id;
DB l
4==> my $id;
5: my $type;
6: my %data;
7: my %dataTypes;
8
9 # Print de codes?
10: my $codes;
11 # Print de raw data
12: my $raw;
13 # Print de geparste code
DB s
main::(./convert.pl:5): my $type;
DB p
DB n
main::(./convert.pl:6): my %data;
DB
LikeLike
Hi Wewek,
I dont really see a problem with the output you pasted here..
Why are you running it trough the debugger? Have you tried just running it? Do you have an actual error message for me to look at?
I hope we can solve your problem!
LikeLike
Doesn’t work for me either. Tried debug mode but same results. Top shows perl script consuming resources but nothing happens. I am running ubuntu 9.04
perl -d convert.pl
Loading DB routines from perl5db.pl version 1.3
Editor support available.
Enter h or `h h’ for help, or `man perldebug’ for more help.
main::(convert.pl:4): my $id;
DB
LikeLike
What command do you use? Does it generate a file? Does the code produce any errors?
The debugging output is normal, the debugger is waiting for you to tell it to go to the next line or print the value of a variable. If the script runs into errors, it spits them when they happen. Before the script even runs. If there was a grave error, you wouldnt even get that far in the debugger.
You can however try replacing the -X in the first line with -w; that will enable more warnings. It will probably spit too much data, but there might be something in there
The first part of the code isnt supposed to generate much output by the way; it reads the ‘ConvertMe.txt’ file and parses the data in there; afterwards, it prints the data it just parsed.
Id propose you send me your data file, but considering the content, you ll probably rather not 🙂
LikeLike
Thanks Gert for quick reply.
When i run the script normally using ./convert.pl (do i have to use convert.pl > converted.csv instead?) and it doesn’t show any output on screen and top show script running with 90 to 100% cpu usage on core 0 on core2duo. No file gets generated in directory, i assume it should be converted.csv (as from your instructions), is the .csv file generated at the end or ought to be created at beginning? I ‘ve never done coding in perl so i know less about the script except that it is parsing output to comma delimited form.
Also, how can i check encoding in vi whether it is set to unix?
LikeLike
It says so on the bottom, you should at any rate convert the file, just set them as described; cant hurt at any rate.. If you didnt do this, that might be the reason the script keeps on going.
The script first silently parses your data into memory and afterwards prints any data you request (the cvs by default). The script doesnt _write_ a file, it only prints the data to screen. The “> file.csv” part actually puts the output in the file. So you d probably best test the script without that part. (so you ll see the output on the screen)
Also, the larger the file, the more time it will take; how big is the file you re trying to have parsed?
LikeLike
Thanks Gert.
I realised later i missed redirecting stdout to file converted.csv. I will try it again when i get home.
Please could you tell me how long it normally takes parsing the input to stdout for a normal file size of lets say 50K?
thanks again for speedy response
LikeLike
If you open the file in vim and it says
> “ConvertMe.txt” [converted][dos]
you probably forgot to do the first part (encoding etc), if you run the file without doing that you will not get any parsed output (table headers at most)
a 200K file takes a few seconds
Id be very interested in recreating the bug you are describing..
LikeLike
lol, I just actually recreated the error 🙂 I copied the code from here and ran it; the code loops trough the file using while (<MYFILE>) but the blog stripped the <MYFILE>, considering it to be HTML..
Update in a sec..
Sorry for the crap
LikeLike
Below is the sample data I am trying to parse using your script. I am using latest wammu (0.32.1) from ubuntu ppa repo to backup contacts.
ConvertMe.txt
This file format was designed for Gammu and is compatible with Gammu+
; See for more info
[Backup]
IMEI = “XXXXXXXXXXX”
Phone = “Sony Ericsson C902 XXXXXXXX_HUTCHISON_AO”
Creator = “Wammu 0.32.1”
DateTime = 20100327
Format = 1.04
[PhonePBK001]
Location = 001
Entry00Type = LastName
Entry00Text = “And L”
Entry01Type = FirstName
Entry01Text = “A”
Entry02Type = FormalName
Entry02Text = “A And L”
Entry03Type = NumberMobile
Entry03Text = “01240000000”
Entry04Type = LUID
Entry04Text = “0002000888FB”
[PhonePBK003]
Location = 003
Entry00Type = FirstName
Entry00Text = “AA”
Entry01Type = FormalName
Entry01Text = “AA”
Entry02Type = NumberMobile
Entry02Text = “0870550000”
Entry03Type = LUID
Entry03Text = “00020000018F”
[PhonePBK006]
Location = 006
Entry00Type = FirstName
Entry00Text = “Abbey”
Entry01Type = FormalName
Entry01Text = “Abbey”
Entry02Type = NumberWork
Entry02Text = “0190888883”
Entry03Type = NumberGeneral
Entry03Text = “05511430470”
Entry04Type = NumberMobile
Entry04Text = “08007317878”
Entry05Type = LUID
Entry05Text = “00020000067A”
[SIMPBK087]
Location = 092
Entry00Type = NumberGeneral
Entry00Text = “07777187010”
Entry01Type = Name
Entry01Text = “joe”
I tried encoding again it seems when doing :set ff=unix, pressing return doesn’t set encoding, as :set ff=unix still shows at the bottom asking for more input even after pressing return.
LikeLike
Changed the encoding using flip program and still same issue. Script keeps on going for 90K file
LikeLike
are you using the new and updated code WITH the correct while statement? Because that is the cause of the infinite loop..
The data parses to output, but the newlines between the data are screwing the output (how did those get there?)
By the way, the vim not clearing that line .. same here, but it works, so.. 😉
LikeLike
Nice one mate. Looks like brackets sorted it out for the script. It would be nice to add it in ubuntu tutorials section on ubuntuforums for more people to take benefit of this wammu csv export script.
Thanks for the script and sorting it out. It only took milliseconds to convert the file this time.
🙂
LikeLike
More info for people trying this script, in case they get stuck follow below steps –
vim didn’t change encoding for my binary file produced by wammu 0.32.1, therefore i used program called flip and installed it from ubuntu repos.
sudo apt-get install flip
usage – to convert DOS to unix encoding, run below:
flip -ub ConvertMe.txt
Important: where -b is to include encode even binary files which in my case is true.
Extra step to make sure encoding is set, then I opened file in vim:
vim ConvertMe.txt and followed Gert’s steps to set encoding as below:
# Set unix encoding
:set ff=unix
# Reset the encoding
:set fileencoding=default
Rest followed the guide exactly. My special thanks to Gert for sharing this excellent script.
@[Gert] My ConvertMe file was in binary format, flip program can be good addition to your tutorial.
regards,
Ash
LikeLike
Gert to your question:
“The data parses to output, but the newlines between the data are screwing the output (how did those get there?)”
Reply: I think when i copy pasted the sample text, html on this blog parsed it induce blank lines between each line I pasted. That explains why the output seems all wonky.
LikeLike
Im glad to have been of service!
And thanks for seeing this trough, without you I would’ve never realised the mistake..
Ill edit the flip hint in tonight 🙂
LikeLike
Only if this blog supported
/tags, would have never been the problem. But i guess one learns alot more with every try.Nice job Gert. cheers
LikeLike
^^ sorry
tags i meant not “/”
LikeLike
#!/bin/bash
echo "This is test for code tags working in this blog"
LikeLike
Oh, so the [ code ] tags work now? exciting 🙂 I recently updated the blog and got a new skin.. the old skin didnt handle the code tags too well.. The post has the html equivalent, but apparently that gets cleaned out.. Ill have to look into that some time soon
I edited contribution into the original post, tnx for the help 🙂
LikeLike
Update:
I have just imported my contacts to gmail contacts and looks like all data imported without a hitch. Gert I am thinking to work on a python-gtk taking ideas from your perl script to implement a gui wammu-csv export tool, as wammu developers seems to be excluding export tool with every release of wammu ( and I don’t know why?)
LikeLike
Thanks Gert, i appreciate your comments on contribution.
tags are not working but i guess html code tags are working “” “
LikeLike
^^^ beginning with
LikeLike
sorry it is not letting me display the tags as it parses the text between arrows
LikeLike
Great idea, I d love to see that 🙂
No clue though as to why Wammu doesnt implement this by default.. Though I can understand that their first priority would be device support. It always dazzles me though how difficult it is at all levels to really control your own cellphone related data.. (Kudos to Google there for their Data Liberation Front and Android GMail sync)
Edit: Im looking into setting up a personal pastebin (with code highlighting) to ease the code pasting on here.. (and Im not using the hosted ones because I want to be able to backup those in my default mysql backups..)
LikeLike
why not just implement drupal CMS framework to handle blogs instead?
http://www.thesitewizard.com/gettingstarted/how-to-setup-drupal.shtml
LikeLike
Oh, Ive got some druppal running too; but I prefer WordPress for blogging.
WordPress is the king of cms usability for all Im concerned. Much because its a blog and nothing more, so in a different ballpark than Drupal. Drupal is an amazing product too, but in backend usability it doesnt even come close to the Ajaxed up heaven that is WordPress 🙂
LikeLike
Interesting, I always got fascinated by web technologies. Although, my stuff is mainly desktop application oriented.. i would still love to look into wordpress someday.
In terms of web framework, ASP.NET was buggiest framework i’ve ever come across so far.
LikeLike
BTW, what Android phone you use? I have got hero recently and thinking to put xda rom to use 2.1
LikeLike
HTC Magic with Cyanogenmod (from XDA) – I hear villainmod is a decent 2.1 Hero image.. Havent tried though.. I AM considering flashing my wifes Hero, but.. I need to run such a rom myself first 😉
CMSes & all are indeed interesting en fun stuffs 🙂
And ASP.Net is horrible, I know.. I wrote a bit in it and .. blurgh. And it requires Windows web servers :s
LikeLike
There are ways to run asp.net under linux apache servers using Mod_mono but i wont count on that, my testing resulted asp.net pages rendered characterset pretty weird on cross-platform. i gave up soon on asp.net as it seemd look alike of J2EE in first place but a bad attempt in implementation.
I am currently waiting for HTC hero 2.1 rom release from my provider, which they say is end of April, if not then i definitely take a plunge and stuff hero with custom rom. Thanks for pointing the rom, i’ll into villainmod. Android made things endless…..Thanks to google for that as well
LikeLike
my blog SUCKS when displaying code. So, I put it up on http://www.gertschepens.be/wammu2cvs
LikeLike
Amazing! Your script saved me hours of work… Worked like a charm!
LikeLike
Well, then the time it took putting it together was well spent 🙂 Glad it worked for you & tnx for making the effort to leave a comment!
LikeLike
Thanks for the tutorial!
I used the program ‘recode’ to convert the utf-16 file:
recode utf-16..utf-8 bla.backup
(converts in place)
And modified your code to skip the none Phone contact records from the backup file, and read the filename from argument:
The modified version of the script can be found here: http://dblaci.hu/wammu2csv.pl
LikeLike
Glad to have been of help 🙂
Thanks for the update; Great modifications!
Would you mind I put your updates online on the code page? (with due credit of course ;))
LikeLike
Hi!
thanks a lot for the script! worked like a charm after encoding is properly set. Maybe you could add some kind of output to inform people they haven’t set it correct.
Keep up the good work!
LikeLike
Thanks for the feedback; Im always glad to hear when people were able to use some of my code!
The encoding step can be passed over by using the modifications DBLaci posted. I will add his work to the code; that should help future users. Thanks for the tip though!
Tnx 🙂
LikeLike
Thanx a lot! Very helpful script, you’ve saved a lot of my time and patience )
LikeLike
happy to have helped
LikeLike
I’m teacher and I want to send sms of exam results to my student. I write exam results to calc in Libre office. In a line there is two column one is phone number, other is exam results. Can I send all the results in one time?
LikeLike
Hi Onur! Im afraid that what you’re asking is utterly beyond what my script is written for and what it can do! You would obviously be able to do what you’re talking about but you’ll need the supporting technology first. Do you have any way of sending texts from a machine that you can run code on? Any hardware or software that can actually send texts?
Depending on what tech you have, this should be possible; probably without too much of a hassle 🙂
LikeLike
Yes I have 3g usb modem and bluetooth connection with my mobile phone. Also I use wammu on ubuntu and vodafone software program in windows
LikeLike
then you should be able to write that; though I cant really help you there.
The script this post is about talks about transferring your phonebook from wammu to google.
LikeLike
I don’t know when this happen, I was looking for a sync wammu-android to restore my backup so I endeed here and you give me the hint that gmail do import cvs, I check there and find out gmail also accept vCard format that nowadays wammu support. And that’s all you need, to export in vCard format from wammu.
LikeLike
Thats AMAZING news! How did the vCard import work out for you; all went ok?
So for the folks still looking to do this; No need messing with convertion etc; just export to vCard and import in gMail 🙂
LikeLike
Everything went better than expected! 🙂 … just perfect without trash.
LikeLike
Hi!
I also tried to use your guide to convert my gammu-phonebook to csv, however did not succeed.
I assume there has been a change in the gammu format.
I created a backup file of the gammu export for comparison.
After ALL the vim/flip steps (both ways tried, also tried 3rd way with streamline editor sed:
the unmodified backup file and the modified files were bytewise identical.
I checked and found: gammu-export never contained \r\n, which are changed by flip.
Gert, thanks to your intro, I finally had an idea that worked around all the (not-working) vim-stuff:
I could then run convert.pl
LikeLike
Hi Nils,
glad you figured it out! One of the previous commenters also used iconv or a similar tool to convert and posted an updated blob of code. I’ll see about making that update more prominently visible in here..
thanks for commenting 🙂
LikeLike