Message Primitive Developement Technique : Hitting more mangoes in one shot

         Telecom software modules provides primitive to user modules. Usually these primitives are implemented as messages. Framework may exist to delivery these messages between software modules. Message Queue, Socket  other mechanism can be used to send these primitive messages. It is always good to provide encode and decode logic which converts the message to intermediate format which can be sent across the process using socket, message queue etc.  This also should take care of endian so that each software modules can run on the any processor etc. It is also possible testing framework may run on one endian and system under testing module may run on another endian. In this situation also encode/decode logic helps. It is always good to develop wireshark dissector of those messages which helps debugging. 

           It is always good to automate development of these messages, encode/decode logic and wireshark dissector development. We can define simple format for defining message structure may be in csv format. A script can take this csv format as input and can generate header file for the message and implementation for encode/decode. Another script can take same csv format file as input and can generate source code for wireshark dissector which can be integrated with wireshark.

          Following path takes to the location of sample script which  does above job(todo).

Memory Leak Test in Real Time Embedded System

Memory related problems are common in software development for embedded system. During module testing, unit testing it is always essential to make sure that there is no memory related error. There are open source free tools available. Val grind is one of them. There are paid tools too.

Most of the memory related problems are associated with dynamic memory allocation. Following are  list of some of the common memory related problems.

1. Memory Leak. This is the case when memory is not freed once its usage is finished.
2. Invalid Write. This is the case writing to memory location without proper memory allocation. May be writing to memory location after freed up.
3. Invalid Read. This is the case reading memory location without proper memory initialization. May be reading memory after it freed up.

In context of embedded system/real time system following are the challenges of finding out memory related problems.

1. Real time system has constraints of time so enabling memory leak testing may be difficult which has extra processing overhead.

2. In embedded system most of the time pool of memory is allocated during boot up and dynamically allocated from the pool rather from heap. This is required to achieve predictability behaviour which is very much essential in embedded application. When we use memory pool library instead of standard library of memory allocation, memory debugging tool like valgrind can’t find any of the memory related error mentioned before.


Following section gives some of the technique to address above mentioned challenges.

First challenge can be addressed by having host based testing environment. It is worth to spend time to build adapters required to simulate target behaviour (need not be 100%) and perform memory leak testing on host PC using memory debugging tool such as valgrind. We can compromise on real time constraints and can execute functionality on host and find memory related error if not every error but most of the error.

Second challenge can be addressed by having macros for memory allocation/de-allocation API and by having condition compilation to enable system call rather memory pool allocation Api, whenever memory debugging is required.


void* alloc_mem (struct mem_pool*  pool, int size)
{
          void* mem = NULL;
#ifdef MEMORY_TEST_ENABLED
          mem = malloc (size);
#else
          mem =  alloc_from_pool (pool, size);
#endif
          return (mem);

}

Binary Logs in Real Time Embedded System

Logging is  one of the important technique used by the software developer to debug their code. C/C++ developer usually define macros which can be used to log debugging text. Following snippet gives  sample.

LOG(<category>,<level> <format_string>[,variable number of arguments);

Usually log provides mechanism to enable disable during compile time and run time. Log level can be defined as ERROR, WARN, INFO, DEBUG etc. Log level can be enabled disabled at run time. It is always good to provide file name and line number where log entry present in the code with log message. This text log can be displayed on console, or written into file or send it over network to logging server.

Aim of this article to discuss logging in the context of hard real time application where there will be time constraints to finish the job. Definitely it is impractical to display these text format logs on the screen.  Better approach is to write these text logs to file or send it over socket. Since it is hard real time application usage of  blocking socket or file write is a problematic.

Now we are going to discuss binary logging, a technique of logging without compromising the any of the benefit we see in the text format logging what we discussed at the beginning of this article.

We can write a script which scans the source code and assigns numerical identifier to each of the logs. At the same time it can prepare meta data which contains mapping between format string and assigned identifier. It is always good to prefix some new literals to logging macro and suffix by information about number of argument, once script assigns identifier so that anybody can distinguish between converted and newly added logs.

Let us assume developer adds logs as follows.
#include "logging.h"
int main ()
{
             int i = 10;
             set_log_level(MAIN, DEBUG);
             LOG(MAIN, INFO, "Entering main\n");
             LOG(MAIN, DEBUG, "Value of i = %d\n", i);
             printf ("Hello world\n");
             LOG (MAIN, INFO, "Exiting main\n");
 
}

after running the script the above code will be modified/instrumented by the script as follows.

#include "binary_logging.h"
int main ()
{
             int i = 10;
             set_log_level(MAIN, DEBUG);
             BIN_LOG_0(0,MAIN, INFO, "Entering main\n");
             BIN_LOG_1(1,MAIN, DEBUG, "Value of i = %d\n", i);
             printf ("Hello world\n");
             BIN_LOG_0 (2,MAIN, INFO, "Exiting main\n");
 }

Meta data can look as follows for above log.
#num_of_arg:category:level:format string
0:MAIN:INFO:Entering main\n
1:MAIN:DEBUG:Value of i = %d\n
0:MAIN:INFO:Exiting main\n


We can use circular buffer to store logging information such as log id and argument values, which will be read by separate thread and converts binary logs to user readable text logs.
It is always important to read logs quickly such that buffer full condition does not reach otherwise we might loose some of the logs. It is very necessary to avoid mutex instead we can use automatic set operation where writer thread/process  updates writer index without race condition.Another important aspect is supporting multi thread. We can allocate circular buffer for each thread flow and pointer to circular buffer can be stored in thread specific context using posix api. So each log function firs gets associated circular buffer reference from thread specific area and puts the log entry in the circular buffer. It is always good to reserve segment at the begining of circular buffer to store information such as reader_index, writer_index, log missed counter etc which helps debugging if there is any problem in the log.


Following link provides source code of binary log implementation. Script is written in perl.Here circular buffer is implemented over shared memory and log reader and log writer implemented as separate process.

<proof of concept code will be provided soon>