Convert a Wammu backup file to a .csv

Yes, this post is indeed about a Quick and Dirty tool to “Convert a Wammu backup file to a .csv” ready for gMail import. I dicked around with the wammu.backup file to get it to plain text for reference and then figured Id better write some quick perl to get it to a .csv so I could import that into my gMail account. Since my Android phone automagically synchronizes with my gMail contacts (gotta love it)

I started writing the perl after I dicked around a bit with the file. The file is by default DOS formatted, illegible and apparently encoded in utf-16. Since I first wanted it as a reference (to grep, cat or less in a term window) so I did the following. You ll have to do this too (do feel free to submit the Perl code in the comments to skip this part ;))

  • Open the file using vi.
  • Save file (yes, save BEFORE editing, this converts the binary to a regular file)

    :w ConvertMe.txt

    • If the conversion using vi doesnt work for you, you can also use flip (Thanks to AK for the tip, check his comment for more info!)

      flip -ub ConvertMe.txt

  • Set unix encoding

    :set ff=unix

  • Reset the encoding

    :set fileencoding=default

  • Save! (use the :w)

Next part is the Perl script. (The code is after the jump..)
The script reads a file called ConvertMe.txt – You can change this in the script, but really, why πŸ™‚

  • run convert.pl , this will output the .csv on the screen
  • run convert.pl > converted.csv to get the google ready csv..

Importing into gMail is really simple. Login and click “Contacts” (Left sidebar), click “Import” (on top of the contacts list to the right) and import it using the menu. I imported slightly over 260 contacts and decided it was a bad idea seconds later since I now have a lot of work to sort those out. Thats a pain in the arse. And Im not sure why, but the find duplicates doesnt find nearly half of them. Apparently it doesnt compare the phone nrs. Probably something to do with false positives.. Cool option though.. I DID import them into a separate category and that does help! Beats doing them by hand..

What if this doesnt work?
There are three options for output..

You ll find the following in the code. Its not that long, do a search πŸ˜‰

# Print de codes?
my $codes = 1;

to print all the codes in the file you re having parsed. This reads the file and spews out all the different tags in there in an easy list.

# Print de raw data
my $raw = 1;

Print the raw data. Prints all the tags for every contact, so you dan easilly check if all looks right

# Print de geparste code
my $parse = 1;

This is enabled by default. This option parses all your data into the .csv format.

If the last fails, its possible you need to change or add to the tags described in the .csv file. This defenition starts at line 66, the # Parse data line. First check what tags your file provides by setting the $codes option and then fill these in on the right lines in the script. Dont remove the commas or the .csv will fail to parse correctly. Just add the fields in front, analogous to the current settings. Dont worry about the tags that wont be filled, they will be empty. Dont worry, it ll work πŸ˜‰

The Source.. Moved to http://www.gertschepens.be/wammu2cvs-code

Edit: People have been experiencing problems with the code and after a long and hard look I discovered the error to be the blog having stripped the <MYFILE> from the code, thinking it to be faulty html and thus creating an endless loop.. This explains a lot πŸ˜‰ Kudos to AK for helping me sort it all out! And sorry for the mess..

Edit2: Code moved to http://www.gertschepens.be/wammu2cvs because of formatting problems in this post.

Published by Gert

Person-at-large.

51 thoughts on “Convert a Wammu backup file to a .csv

  1. your perl script doesn’t work for me (Debian squeeze)
    running in debug mode (-d):

    $ ./convert.pl

    Loading DB routines from perl5db.pl version 1.32
    Editor support available.

    Enter h or `h h’ for help, or `man perldebug’ for more help.

    main::(./convert.pl:4): my $id;
    DB q

    Like

  2. main::(./convert.pl:4): my $id;
    DB l
    4==> my $id;
    5: my $type;
    6: my %data;
    7: my %dataTypes;
    8
    9 # Print de codes?
    10: my $codes;
    11 # Print de raw data
    12: my $raw;
    13 # Print de geparste code
    DB s
    main::(./convert.pl:5): my $type;
    DB p

    DB n
    main::(./convert.pl:6): my %data;
    DB

    Like

    1. Hi Wewek,
      I dont really see a problem with the output you pasted here..
      Why are you running it trough the debugger? Have you tried just running it? Do you have an actual error message for me to look at?
      I hope we can solve your problem!

      Like

  3. Doesn’t work for me either. Tried debug mode but same results. Top shows perl script consuming resources but nothing happens. I am running ubuntu 9.04

    perl -d convert.pl

    Loading DB routines from perl5db.pl version 1.3
    Editor support available.

    Enter h or `h h’ for help, or `man perldebug’ for more help.

    main::(convert.pl:4): my $id;
    DB

    Like

    1. What command do you use? Does it generate a file? Does the code produce any errors?

      The debugging output is normal, the debugger is waiting for you to tell it to go to the next line or print the value of a variable. If the script runs into errors, it spits them when they happen. Before the script even runs. If there was a grave error, you wouldnt even get that far in the debugger.

      You can however try replacing the -X in the first line with -w; that will enable more warnings. It will probably spit too much data, but there might be something in there

      The first part of the code isnt supposed to generate much output by the way; it reads the ‘ConvertMe.txt’ file and parses the data in there; afterwards, it prints the data it just parsed.

      Id propose you send me your data file, but considering the content, you ll probably rather not πŸ™‚

      Like

  4. Thanks Gert for quick reply.

    When i run the script normally using ./convert.pl (do i have to use convert.pl > converted.csv instead?) and it doesn’t show any output on screen and top show script running with 90 to 100% cpu usage on core 0 on core2duo. No file gets generated in directory, i assume it should be converted.csv (as from your instructions), is the .csv file generated at the end or ought to be created at beginning? I ‘ve never done coding in perl so i know less about the script except that it is parsing output to comma delimited form.

    Also, how can i check encoding in vi whether it is set to unix?

    Like

    1. It says so on the bottom, you should at any rate convert the file, just set them as described; cant hurt at any rate.. If you didnt do this, that might be the reason the script keeps on going.

      The script first silently parses your data into memory and afterwards prints any data you request (the cvs by default). The script doesnt _write_ a file, it only prints the data to screen. The “> file.csv” part actually puts the output in the file. So you d probably best test the script without that part. (so you ll see the output on the screen)

      Also, the larger the file, the more time it will take; how big is the file you re trying to have parsed?

      Like

  5. Thanks Gert.

    I realised later i missed redirecting stdout to file converted.csv. I will try it again when i get home.

    Please could you tell me how long it normally takes parsing the input to stdout for a normal file size of lets say 50K?

    thanks again for speedy response

    Like

    1. If you open the file in vim and it says
      > “ConvertMe.txt” [converted][dos]
      you probably forgot to do the first part (encoding etc), if you run the file without doing that you will not get any parsed output (table headers at most)

      a 200K file takes a few seconds

      Id be very interested in recreating the bug you are describing..

      Like

  6. lol, I just actually recreated the error πŸ™‚ I copied the code from here and ran it; the code loops trough the file using while (<MYFILE>) but the blog stripped the <MYFILE>, considering it to be HTML..

    Update in a sec..

    Sorry for the crap :/

    Like

  7. Below is the sample data I am trying to parse using your script. I am using latest wammu (0.32.1) from ubuntu ppa repo to backup contacts.

    ConvertMe.txt

    This file format was designed for Gammu and is compatible with Gammu+

    ; See for more info

    [Backup]

    IMEI = “XXXXXXXXXXX”

    Phone = “Sony Ericsson C902 XXXXXXXX_HUTCHISON_AO”

    Creator = “Wammu 0.32.1”

    DateTime = 20100327
    Format = 1.04

    [PhonePBK001]

    Location = 001

    Entry00Type = LastName

    Entry00Text = “And L”

    Entry01Type = FirstName

    Entry01Text = “A”

    Entry02Type = FormalName

    Entry02Text = “A And L”

    Entry03Type = NumberMobile

    Entry03Text = “01240000000”

    Entry04Type = LUID

    Entry04Text = “0002000888FB”

    [PhonePBK003]

    Location = 003

    Entry00Type = FirstName

    Entry00Text = “AA”

    Entry01Type = FormalName

    Entry01Text = “AA”

    Entry02Type = NumberMobile

    Entry02Text = “0870550000”

    Entry03Type = LUID

    Entry03Text = “00020000018F”

    [PhonePBK006]

    Location = 006

    Entry00Type = FirstName

    Entry00Text = “Abbey”

    Entry01Type = FormalName

    Entry01Text = “Abbey”

    Entry02Type = NumberWork

    Entry02Text = “0190888883”

    Entry03Type = NumberGeneral

    Entry03Text = “05511430470”

    Entry04Type = NumberMobile

    Entry04Text = “08007317878”

    Entry05Type = LUID

    Entry05Text = “00020000067A”

    [SIMPBK087]

    Location = 092

    Entry00Type = NumberGeneral

    Entry00Text = “07777187010”

    Entry01Type = Name

    Entry01Text = “joe”

    I tried encoding again it seems when doing :set ff=unix, pressing return doesn’t set encoding, as :set ff=unix still shows at the bottom asking for more input even after pressing return.

    Like

  8. are you using the new and updated code WITH the correct while statement? Because that is the cause of the infinite loop..
    The data parses to output, but the newlines between the data are screwing the output (how did those get there?)

    By the way, the vim not clearing that line .. same here, but it works, so.. πŸ˜‰

    Like

  9. Nice one mate. Looks like brackets sorted it out for the script. It would be nice to add it in ubuntu tutorials section on ubuntuforums for more people to take benefit of this wammu csv export script.

    Thanks for the script and sorting it out. It only took milliseconds to convert the file this time.

    πŸ™‚

    Like

  10. More info for people trying this script, in case they get stuck follow below steps –

    vim didn’t change encoding for my binary file produced by wammu 0.32.1, therefore i used program called flip and installed it from ubuntu repos.
    sudo apt-get install flip

    usage – to convert DOS to unix encoding, run below:
    flip -ub ConvertMe.txt

    Important: where -b is to include encode even binary files which in my case is true.

    Extra step to make sure encoding is set, then I opened file in vim:
    vim ConvertMe.txt and followed Gert’s steps to set encoding as below:
    # Set unix encoding
    :set ff=unix

    # Reset the encoding
    :set fileencoding=default

    Rest followed the guide exactly. My special thanks to Gert for sharing this excellent script.

    @[Gert] My ConvertMe file was in binary format, flip program can be good addition to your tutorial.

    regards,
    Ash

    Like

  11. Gert to your question:
    “The data parses to output, but the newlines between the data are screwing the output (how did those get there?)”

    Reply: I think when i copy pasted the sample text, html on this blog parsed it induce blank lines between each line I pasted. That explains why the output seems all wonky.

    Like

    1. Im glad to have been of service!
      And thanks for seeing this trough, without you I would’ve never realised the mistake..

      Ill edit the flip hint in tonight πŸ™‚

      Like

  12. Only if this blog supported / tags, would have never been the problem. But i guess one learns alot more with every try.

    Nice job Gert. cheers

    Like

  13. Oh, so the [ code ] tags work now? exciting πŸ™‚ I recently updated the blog and got a new skin.. the old skin didnt handle the code tags too well.. The post has the html equivalent, but apparently that gets cleaned out.. Ill have to look into that some time soon

    I edited contribution into the original post, tnx for the help πŸ™‚

    Like

  14. Update:
    I have just imported my contacts to gmail contacts and looks like all data imported without a hitch. Gert I am thinking to work on a python-gtk taking ideas from your perl script to implement a gui wammu-csv export tool, as wammu developers seems to be excluding export tool with every release of wammu ( and I don’t know why?)

    Like

  15. Thanks Gert, i appreciate your comments on contribution.

     

    tags are not working but i guess html code tags are working “” “

    Like

  16. Great idea, I d love to see that πŸ™‚

    No clue though as to why Wammu doesnt implement this by default.. Though I can understand that their first priority would be device support. It always dazzles me though how difficult it is at all levels to really control your own cellphone related data.. (Kudos to Google there for their Data Liberation Front and Android GMail sync)

    Edit: Im looking into setting up a personal pastebin (with code highlighting) to ease the code pasting on here.. (and Im not using the hosted ones because I want to be able to backup those in my default mysql backups..)

    Like

    1. Oh, Ive got some druppal running too; but I prefer WordPress for blogging.

      WordPress is the king of cms usability for all Im concerned. Much because its a blog and nothing more, so in a different ballpark than Drupal. Drupal is an amazing product too, but in backend usability it doesnt even come close to the Ajaxed up heaven that is WordPress πŸ™‚

      Like

  17. Interesting, I always got fascinated by web technologies. Although, my stuff is mainly desktop application oriented.. i would still love to look into wordpress someday.

    In terms of web framework, ASP.NET was buggiest framework i’ve ever come across so far.

    Like

  18. HTC Magic with Cyanogenmod (from XDA) – I hear villainmod is a decent 2.1 Hero image.. Havent tried though.. I AM considering flashing my wifes Hero, but.. I need to run such a rom myself first πŸ˜‰

    CMSes & all are indeed interesting en fun stuffs πŸ™‚
    And ASP.Net is horrible, I know.. I wrote a bit in it and .. blurgh. And it requires Windows web servers :s

    Like

  19. There are ways to run asp.net under linux apache servers using Mod_mono but i wont count on that, my testing resulted asp.net pages rendered characterset pretty weird on cross-platform. i gave up soon on asp.net as it seemd look alike of J2EE in first place but a bad attempt in implementation.

    I am currently waiting for HTC hero 2.1 rom release from my provider, which they say is end of April, if not then i definitely take a plunge and stuff hero with custom rom. Thanks for pointing the rom, i’ll into villainmod. Android made things endless…..Thanks to google for that as well

    Like

  20. Well, then the time it took putting it together was well spent πŸ™‚ Glad it worked for you & tnx for making the effort to leave a comment!

    Like

  21. Thanks for the tutorial!

    I used the program ‘recode’ to convert the utf-16 file:

    recode utf-16..utf-8 bla.backup
    (converts in place)

    And modified your code to skip the none Phone contact records from the backup file, and read the filename from argument:

    The modified version of the script can be found here: http://dblaci.hu/wammu2csv.pl

    Like

    1. Glad to have been of help πŸ™‚

      Thanks for the update; Great modifications!
      Would you mind I put your updates online on the code page? (with due credit of course ;))

      Like

  22. Thanks for the feedback; Im always glad to hear when people were able to use some of my code!

    The encoding step can be passed over by using the modifications DBLaci posted. I will add his work to the code; that should help future users. Thanks for the tip though!

    Tnx πŸ™‚

    Like

  23. I’m teacher and I want to send sms of exam results to my student. I write exam results to calc in Libre office. In a line there is two column one is phone number, other is exam results. Can I send all the results in one time?

    Like

    1. Hi Onur! Im afraid that what you’re asking is utterly beyond what my script is written for and what it can do! You would obviously be able to do what you’re talking about but you’ll need the supporting technology first. Do you have any way of sending texts from a machine that you can run code on? Any hardware or software that can actually send texts?
      Depending on what tech you have, this should be possible; probably without too much of a hassle πŸ™‚

      Like

  24. Yes I have 3g usb modem and bluetooth connection with my mobile phone. Also I use wammu on ubuntu and vodafone software program in windows

    Like

  25. then you should be able to write that; though I cant really help you there.
    The script this post is about talks about transferring your phonebook from wammu to google.

    Like

  26. I don’t know when this happen, I was looking for a sync wammu-android to restore my backup so I endeed here and you give me the hint that gmail do import cvs, I check there and find out gmail also accept vCard format that nowadays wammu support. And that’s all you need, to export in vCard format from wammu.

    Like

    1. Thats AMAZING news! How did the vCard import work out for you; all went ok?
      So for the folks still looking to do this; No need messing with convertion etc; just export to vCard and import in gMail πŸ™‚

      Like

  27. Hi!
    I also tried to use your guide to convert my gammu-phonebook to csv, however did not succeed.
    I assume there has been a change in the gammu format.
    I created a backup file of the gammu export for comparison.
    After ALL the vim/flip steps (both ways tried, also tried 3rd way with streamline editor sed:

    sed -e 's/\r\n/\n/g' handy-telbuch.backup &gt; ConvertMe.txt

    the unmodified backup file and the modified files were bytewise identical.

    I checked and found: gammu-export never contained \r\n, which are changed by flip.

    Gert, thanks to your intro, I finally had an idea that worked around all the (not-working) vim-stuff:

    iconv -f UTF-16 -t UTF-8 gammu.backup -o ConvertMe.txt

    I could then run convert.pl

    Like

    1. Hi Nils,

      glad you figured it out! One of the previous commenters also used iconv or a similar tool to convert and posted an updated blob of code. I’ll see about making that update more prominently visible in here..

      thanks for commenting πŸ™‚

      Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: