The Great Power of Subversion
(or other source version control systems).
I’m always surprised by people reluctant to use version control software, being CVS, SVN, Git. I’ve heard very weird explanation, such as “I never need to go in the past, I’m always looking further”, or “not needed here, I’m just making copies” (ending in .old, .bak, .bad, .keep, .v1, .v2, .vX, .orig,…) or even “I’m not using these kind of tools because it’s not working”…
Yet sometimes things are getting bad. A simple and small change and everything suddenly gets stuck, nothing is working anymore. And this is highly frustrating particularly when designing embedded software, because there are so many layers where the problem can come from that it’s very hard to debug.
Today I faced a weird problem. I wanted to interface an air quality sensor to one of my Jaluino Bee board. That was supposed to be trivial, that’s just a matter of reading an analog value. Changed the code, compiled, programmed, and… erratic behavior ! When alive, Bee is supposed to connect to a gateway using Xbee, then ask for the current time so it can setup internal RTC. But it was just keeping telling the gateway it was alive, never asking for current time… Really, changes in code was very minimal, I did this several times without issues. But today, things went bad. Was it coming from the board, Xbee module, XBee library, the program code itself, some other library, compiler ?
SVN to the rescue.
I had a previously committed HEX file of an old version of the program. Best practices for source control software often say “don’t put compiled program under revision control, since it can be re-generated”. But since there are many changes that can occur, and mostly because these can be both in hardware of software worlds, putting a HEX on SVN helps tracking steps, as miletones (“this HEX file is know to be working, if it doesn’t, that’s a hardware issue”). For the same reasons, I put HEX files from tested sample under Subversion, so people can test theirs Bee boards, for sure, without worrying about all the software stack.
I programmed the chip and successfully make Bee alive again. Nothing wrong with the hardware. But where was it coming from ? The HEX file has been committed on June 24th 2012. I had a quick look at Jaluino SVN repository, but found no relevant changes. Maybe this was coming from Jallib repository itself ? There were many commits in a year, quite impossible to check everything. But on June 24th, I also released xbee_api.jal library to jallib, as the same I committed the HEX file. Revision 3066 on jallib. So, I replaced current jallib repository by the one at that time (back in the past):
$ svn co http://jallib.googlecode.com/svn/trunk@3066 jallib_svn
Compile, programmed and… it worked ! A recursive diff between jallib@3066 and jallib@HEAD showed many differing files, but when limiting the some used in my program, I’ve been able to isolate the “guilty” one, 18f27j53.jal device file. Putting this single file into the latest repository showed it still worked. This device files had several revision since 3066:
$ svn log 18f27j53.jal | grep ^r
r3246 | robhamerling | 2013-04-28 21:00:20 +0200 (Sun, 28 Apr 2013) | 16 lines r3232 | robhamerling | 2013-03-20 14:36:48 +0100 (Wed, 20 Mar 2013) | 8 lines r3228 | robhamerling | 2013-03-04 10:15:54 +0100 (Mon, 04 Mar 2013) | 5 lines r3167 | robhamerling | 2013-01-25 14:31:12 +0100 (Fri, 25 Jan 2013) | 23 lines r3110 | robhamerling | 2012-11-10 20:52:49 +0100 (Sat, 10 Nov 2012) | 17 lines r3090 | robhamerling | 2012-09-08 21:30:55 +0200 (Sat, 08 Sep 2012) | 1 line r3071 | robhamerling | 2012-07-08 14:30:42 +0200 (Sun, 08 Jul 2012) | 3 lines r3068 | robhamerling | 2012-07-01 20:08:28 +0200 (Sun, 01 Jul 2012) | 8 lines
Proceeding by testing old revisions, I identified the very last was the guilty one. It works @rev 3232, not after. So what are the differences:
$ svn diff -r3232:HEAD 3rdparty/jallib_svn/include/device/18f27j53.jal
[snip] @@ -48,8 +48,8 @@ pragma target bank 0x0100 pragma stack 31 pragma code 131064 -pragma data 0x1-0xEAF -pragma shared 0x0-0x0 +pragma data 0x0-0xEAF +pragma shared 0x0-0x5F
It shows different pragma shared memory configuration, and some removed pragma inline. I’m suspecting this “shared memory” thing to be the guilty one. Changing the code on the latest revision by setting old shared memory settings, the program is working again.
Time to ask some more information to Rob, on jallib !
This took ten minutes to identify what changed, some 30 minutes more to investigate some more and find the code actually involves in the issue. Thanks to the Great Power of Subversion. Or other source control software.