Fixing deficiencies in the Arduino IDE

2015-03-15 18:43 by Ian

In this poast (yes: "poast"), I will show you how to include libraries within libraries in an Arduino-esque IDE and not break functionality elsewhere.

The problem:

I should start out by saying: I love the idea of Arduino. It brought many new people into microcontrollers and C++ that would have otherwise been discouraged by the initial complexity of setting up a toolchain and buying (or building) expensive development boards. So it is good in my mind, and any deficiencies one encounters are simply evidence that you have outgrown the Arduino cocoon. Ultimately, you will have to leave the Arduino world, but to the ends of making that transition easier, we're going to take a simple first-step: Fixing Arduino's treatment of gcc with respect to #include.

By doing this, you will be able to write more sophisticated software without having to delve into the minutia of bootloaders and toolchains. And when you finally do leave the Arduino world, your libraries will still be relevant and buildable without the training wheels.

Analysis of problem:

Case-study: StringBuilder
For this discussion, I will be using the StringBuilder class from my github as the most-basal library. StringBuilder is a boilerplate dependency for nearly all of my libraries that deal with dynamic buffers.

To write other libraries against StringBuilder, the new library (which is outside of your main sketch) needs to #include header files that point to it.

The Arduino IDE would have you put StringBuilder.* in this location... 
libraries/StringBuilder/StringBuilder.h
libraries/StringBuilder/StringBuilder.cpp
and this is perfectly fine. The problem is that in order to USE that library, you would be expected to... 
#include <StringBuilder.h>
...when the best-practice would be to... 
#include <StringBuilder/StringBuilder.h>
...because you might want to have some other class in that same package that isn't named the same way. For instance... 
#include <DataStructures/PriorityQueue.h>

How does Arduino get away with omitting path information?
When you click verify, the Arduino IDE parses out your include statements in your sketch (not those in your included libraries), and then adds the include file's path to the gcc command. Any libraries that are outside of your main sketch will not be parsed by the IDE, and the required paths will not be added to the gcc command.

To see this for yourself, hold the SHIFT key while you click "Verify" (in MPIDE), or add "build.verbose=true" to your preferences file (Arduino), and you will see the full command used to build your sketch. In MPIDE, i see this (it is basically the same for Arduino)....
pic32-g++ -O2 -c -mno-smart-io -w -fno-exceptions -ffunction-sections -fdata-sections -g3 -mdebugger -Wcast-align -fno-short-double -fframe-base-loclist -mprocessor=32MZ2048ECG100 -DF_CPU=200000000UL -DARDUINO=23 -D_BOARD_WIFIRE_  -DMPIDEVER=16777998 -DMPIDE=23   -I/home/ian/mplab-sketches/IanTest  -I/home/ian/mpide-0023-linux64-20140821/hardware/pic32/cores/pic32  -I/home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/Wire  -I/home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/ManuvrOS  -I/home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/Adafruit_GFX  -I/home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/RGBmatrixPanel  -I/home/ian/mpide-0023-linux64-20140821/hardware/pic32/variants/WiFire  -I/home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/Wire  -I/home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/ManuvrOS  -I/home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/Adafruit_GFX  -I/home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/RGBmatrixPanel  /tmp/build4472982496961362497.tmp/IanTest.cpp -o /tmp/build4472982496961362497.tmp/IanTest.cpp.o

The thing we are looking for is the -I flags. Those are instructions to the compiler about where to look for include files. You can add many -I flags, and they will be resolved in left-to-right order. Knowing this, let's look at that compile command, formatted in a manner that makes it obvious what is happening.... 
pic32-g++ 
-I/home/ian/mplab-sketches/IanTest  
-I/home/ian/mpide-0023-linux64-20140821/hardware/pic32/cores/pic32  
-I/home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/Wire  
-I/home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/ManuvrOS  
-I/home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/Adafruit_GFX  
-I/home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/RGBmatrixPanel  
-I/home/ian/mpide-0023-linux64-20140821/hardware/pic32/variants/WiFire  
-I/home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/Wire  
-I/home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/ManuvrOS  
-I/home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/Adafruit_GFX  
-I/home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/RGBmatrixPanel

So in my sketch when I.... 
#include <Adafruit_GFX.h>

...gcc will look for these files in this order... 
Not found:    /home/ian/mplab-sketches/IanTest/Adafruit_GFX.h
Not found:    /home/ian/mpide-0023-linux64-20140821/hardware/pic32/cores/pic32/Adafruit_GFX.h
Not found:    /home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/Wire/Adafruit_GFX.h
Not found:    /home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/ManuvrOS/Adafruit_GFX.h
Found:        /home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/Adafruit_GFX/Adafruit_GFX.h

Notice.... Nowhere is there a path to /home/ian/mpide-0023-linux64-20140821/hardware/pic32/libraries. Only subdirectories of it for each library I included from my main sketch.

A well-coded header file would not rely on outside code to include dependencies for its sake, nor the path interpolation (as done by the Arduino environment). So when your header file needs StringBuilder, it will... 
#include <StringBuilder/StringBuilder.h>
Which in Arduino land means this file search... 
Not found:    /home/ian/mplab-sketches/IanTest/StringBuilder/StringBuilder.h
Not found:    /home/ian/mpide-0023-linux64-20140821/hardware/pic32/cores/pic32/StringBuilder/StringBuilder.h
Not found:    /home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/Wire/StringBuilder/StringBuilder.h
Not found:    /home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/ManuvrOS/StringBuilder/StringBuilder.h
Not found:    /home/ian/mpide-0023-linux64-20140821/./hardware/pic32/libraries/Adafruit_GFX/StringBuilder/StringBuilder.h
...

...until failure with a complaint about non-existent included files..

The substance of the fix is the same in all Arduino-esque environments: unconditionally add that root include path to the gcc arguments as the first -I flag. This will allow properly-coded header files to find their own dependencies without having to manually include each and every dependency in your main sketch. It also lets us avoid writing preprocessor case-offs to work around the brain damaged behaviour. Better yet, because we haven't disrupted the original behavior, we can still use libraries that were built to conform to the brain damage.

The instructions will differ for the Arduino IDE versus some of its derivatives. I will cover both Arduino, and MPIDE (Microchip's chipKIT line).

Fixing the brain damage (Arduino):

I am using a Teensy3.1 in the Arduino IDE. So my instructions will reflect this. But the instructions will be the same for any target the Arduino IDE supports. In my case, the relevant file is located here: 
/home/ian/arduino/hardware/teensy/boards.txt

And in that file I'm looking for the "build.option" lines for my target... 
teensy31.build.option1=-mthumb
teensy31.build.option2=-nostdlib
teensy31.build.option3=-D__MK20DX256__
teensy31.build.option4=-DTEENSYDUINO=120

Now I just add another line... 
teensy31.build.option5=-I/home/ian/arduino/hardware/teensy/cores/teensy3/

Make the change appropriate to your home directory, save the file, and restart Arduino.

Fixing the brain damage (MPIDE):

It appears that MPIDE is designed to accommodate a more heterogeneous collection of architectures than Arduino. So the platform config reflects this. For what it's worth, I am working with the Digilent WiFire board. But these instructions ought to be valid for any of the PIC32 boards supported under MPIDE. In my case, the file I am concerned with is: 
/home/ian/mpide-0023-linux64-20140821/hardware/pic32/platforms.txt

And in that file I find these lines.... 
pic32.compiler.c.flags=-O2::-c::-mno-smart-io::-w::-ffunction-sections::-fdata-sections::-g3::-mdebugger::-Wcast-align::-fno-short-double::-fframe-base-loclist

pic32.compiler.cpp.flags=-O2::-c::-mno-smart-io::-w::-fno-exceptions::-ffunction-sections::-fdata-sections::-g3::-mdebugger::-Wcast-align::-fno-short-double::-fframe-base-loclist

...which I change to this... 
pic32.compiler.c.flags=-I/home/ian/mpide-0023-linux64-20140821/hardware/pic32/libraries/::-O2::-c::-mno-smart-io::-w::-ffunction-sections::-fdata-sections::-g3::-mdebugger::-Wcast-align::-fno-short-double::-fframe-base-loclist

pic32.compiler.cpp.flags=-I/home/ian/mpide-0023-linux64-20140821/hardware/pic32/libraries/::-O2::-c::-mno-smart-io::-w::-fno-exceptions::-ffunction-sections::-fdata-sections::-g3::-mdebugger::-Wcast-align::-fno-short-double::-fframe-base-loclist

Make the change appropriate to your home directory, save the file, and restart MPIDE.

Testing the fix:

My quick test is to undo my change and restart the IDE. Verify that you can no longer include libraries within libraries (Arduino default).

Once you have verified this, open any sketches that you feel the need to validate with the new include flags, and build each one of them; taking note of their sizes. I copy-pasta this output into a text file... 
Binary sketch size: 82,152 bytes (of a 262,144 byte maximum)
Estimated memory use: 5,260 bytes (of a 65,536 byte maximum)

If you want an extra level of care, you could also hash the resulting HEX file... 
ian@iAN-MAiN ~ $ md5sum /tmp/build2670874902671277751.tmp/ViamSonus.cpp.hex 
61bab0ec529b7f0956d08275701c34d2  /tmp/build2670874902671277751.tmp/ViamSonus.cpp.hex

Once you've done that for all the sketches you care about, re-instate the change to the gcc flags, and do the same thing. You are trying to verify that...

If check A fails, please send me an email and discuss it with me, so I can fix this poast if there is something incorrect.

If checks B or C fail, the likely cause is that you have a copy of the library code in your sketch, and it differs from what is in your platform library directory. You will need to reconcile the difference and remove the copy in your sketch, OR change your sketch to treat the library as part of the sketch itself.

If neither of those options sound palatable, you can also do the thing that you are almost certainly ready for, and abandon the Arduino environment entirely, and begin writing Makefiles. I will cover that in a later poast.

Previous:
Next: