Antelope(Database Management System) - Contiki

From Contiki
Jump to: navigation, search

Back to Contiki Tutorials

Introduction

Antelope is a database management system which enables a sensor to act as a database server. It is a relational database management system(provides a set of relational database operations) which enables dynamic creation of databases and complex data querying.

Antelope is the first DBMS for resource-constrained sensor nodes, provides energy-efficient querying, high-level data modelling, while being index independent. MaxHeap indexing enables fast insertions and queries over large data sets while having a low energy cost. Antelope hides the low-level I/O details of flash storage, which have thus far often been exposed to application developers. Moreover, Antelope uses LogicVM, a novel virtual machine architecture that analyses and executes propositional logic expressed in a byte-code format. By compiling queries and executing them in the virtual machine, the performance of relational selection queries increases by an order of magnitude compared to repeated parsing.

You will learn

In this tutorial, we briefly study how antelope is implemented in Contiki operating system by going through some of the basic and important code blocks. We will also study how to run database client and server in cooja and get some hands on experience on some of the basic query operations of the antelope.

Antelope architecture

Components

Antelope consists of eight components as shown in the below figure.

Antelope arch.jpg


  • Query processor  : It parses AQL(query language for Antelope) queries.
  • Privacy Control  : Checks whether the query is allowed or not.
  • Logic VM  : It executes the queries.
  • Database kernel  : It holds the database logic and coordinates query execution.
  • Index abstraction : It holds the indexing logic.
  • Indexer process  : Builds indexes from existing data.
  • Storage abstraction : Contains all storage logic.
  • Result transformer : It presents the result of query in a way that makes it easy to use by programs.



Terminology

Antelope uses the standard terminology from the relational database management system. Below are some of the important terms:

  • Tuple: It is a set of attribute values.
  • Attribute: Different properties. E.g SensorID, Time.
  • Domain: Each attribute has got a domain that specifies the data type of the attribute values.
  • Relation: It is a collection of tuples that have the same set of attributes.
  • Primary Key: One of the attribues, which uniquely identifies the tuple in a relation.
  • Cardinality: The number of tuples in a relation is called the cardinality of the relation.

To put the above terminology in easy terms, informally we can say :

  • A relation is like a table, attribute is like a column and a tuple is like a row. Below is an example.
jpg

Source Code Directories

Navigate to the examples/antelope/netdb folder from the contiki home directory.

cd /examples/antelope/netdb

There are two important files in this(netdb) subfolder.

  • netdb-server.c
  • netdb-client.c

There is one more file netdb.csc which we are going to use for our simulation on cooja.

The implementation of Antelope database in present in the directory:

~/contiki-2.7/apps/antelope

Code building blocks

In this section, source code for the example given in the ~/contiki-2.7/examples/antelope/netdb/ is explored.

netdb-client.c

This client program provides a serial line interface with the help of which we can enter the queries. The shell process of the client program process these serial messages(queries) and sends the queries to the server program(running on another mote). Once it receives the reply to the query, it prints out a confirmation message in the shell.

Let's walk through the source file.

#ifndef SERVER_ID
#define SERVER_ID 4
#endif

SERVER_ID is used to send queries to server. Here, it is set to 4 because netdb-server.c file is uploaded onto the mote with id 4 in the netdb.csc(cooja simulation file given in the examples/antelope/netdb folder).

static unsigned server_id = SERVER_ID;
static struct mesh_conn mesh;

Last line in the above source code snippet creates a mesh_conn structure to store the details about the connection between the client and server. Antelope uses the Rime mesh routing protocol to exchange the messages between the client and the server. Below is another important data structure "mesh_callbacks" which contains call back functions which we are implemented in netdb_client program file.


struct mesh_callbacks {
  /** Called when a packet is received. */
  void (* recv)(struct mesh_conn *c, const rimeaddr_t *from, uint8_t hops);
  /** Called when a packet, sent with mesh_send(), is actually transmitted. */
  void (* sent)(struct mesh_conn *c);
  /** Called when a packet, sent with mesh_send(), times out and is dropped. */
  void (* timedout)(struct mesh_conn *c);
};
PROCESS(netdb_process, "NetDB");
PROCESS(shell_process, "Shell Process");
AUTOSTART_PROCESSES(&netdb_process);

The first two lines in the above source code creates two processes "netdb_process"(which is the database process) and "shell_process". "shell_process" is the process which provides serial interface at the client to input the queries. AUTOSTART_PROCESSES(&netdb_process) starts the netdb_process, means thread associated with this process will start running by the scheduler. Below is the code snippet of the thread for this process.


PROCESS_THREAD(netdb_process, ev, data)
{
  PROCESS_EXITHANDLER(mesh_close(&mesh));
  PROCESS_BEGIN();

  mesh_open(&mesh, NETDB_CHANNEL, &callbacks);
  process_start(&shell_process, NULL);

  PROCESS_END();
}
static const struct mesh_callbacks callbacks = {received, sent, timedout};

This will create a callbacks structure which contains the function pointers to the callback functions which are used upon a packet arrival.

A protothread is created for netdb_process. An exit handler is declared for the netdb_process. PROCESS_BEGIN() and PROCESS_END() are required inside a proto thread declaration. To know more about the Contiki Processes and Protothreads, please follow this link.

"mesh_open" opens a new connection and here the created mesh connection operates on NETDB_CHANNEL(70) and the third argument "callbacks" contains the function pointers to functions which will be called when a packet arrives on the channel. process_start(&shell_process, NULL) starts the shell process to provide the serial line interface(This is an example of starting one process from another process, instead of using AUTOSTART_PROCESSES).


Callback handlers for the netdb_client process mesh connection:

static const struct mesh_callbacks callbacks = {received, sent, timedout};
  • Callback function - sent
static void
sent(struct mesh_conn *c)
{
}

This function gets called when we try send a packet with mesh_send() and it is actually transmitted.


  • Callback function - timedout
static void
timedout(struct mesh_conn *c)
{
  printf("Failed to send packet: time out\n");
}

This function gets called when we try to send a packet with mesh_send() and the packet is dropped because of timeout. Here we are just printing "Failed to send the packet: timeout".


  • Callback function - received
static void
received(struct mesh_conn *c, const rimeaddr_t *from, uint8_t hops)
{
  char *data;
  unsigned len;
  static char reply[MAX_QUERY_SIZE + 1];

  data = (char *)packetbuf_dataptr();
  len = packetbuf_datalen();

  if(len > MAX_QUERY_SIZE) {
    printf("Too long query: %d bytes\n", len);
    return;
  }

  memcpy(reply, data, len);
  reply[len] = '\0';

  printf("%lu Reply received from %d.%d (%d hops): %s",
         clock_time(), from->u8[0], from->u8[1], (int)hops, reply);
}

This function is called when a packet is received. In the function handler "received" in netdb_client process, "c" denotes the mesh connection for the channel, "from" is the address of the server from which the packet is received and "hops" gives us the hop count between the client and server. Inside this function, one error condition is checked (len > MAX_QUERY_SIZE) and prints out an error for too long query if the condition evaluates to true. If there is nothing wrong with the query reply, reply is printed to the serial line interface.


Let's look at the source code for the "shell_process".

PROCESS_THREAD(shell_process, ev, data)
{
  rimeaddr_t addr;

  PROCESS_BEGIN();

  printf("NetDB client\n");

  for(;;) {
    PROCESS_WAIT_EVENT_UNTIL(ev == serial_line_event_message && data != NULL);
    if(strncmp(data, "server ", 7) == 0) {
      server_id = atoi((char *)data + 7);
    } else {
      printf("%lu Transmitting query \"%s\" to node %u\n", clock_time(), (char *)data, server_id);
      packetbuf_copyfrom(data, strlen(data));
      addr.u8[0] = server_id;
      addr.u8[1] = 0;
      packetbuf_set_attr(PACKETBUF_ATTR_PACKET_TYPE,
                         PACKETBUF_ATTR_PACKET_TYPE_STREAM);
      mesh_send(&mesh, &addr);
    }
  }

  PROCESS_END();
}

A proto thread is created for the process shell_start. Inside an infinite loop, we wait for an event to occur. There is a macro in the infinite 'for loop' "PROCESS_WAIT_EVENT_UNTIL(ev == serial_line_event_message && data != NULL);" which makes the thread to wait for serial line event message to occur. Serial line event message here defines user entering some information(data should not be NULL) from the serial line interface. If there is a server in the data(user entered data from the shell serial line interface), server id is copied from the data and "server_id" variable is set to this value, else the data is considered as a query message and the addr is set to the server's address and mesh_send function is called to send the packet to the server using mesh routing protocol.


netdb-server.c

Most of the source code for the netdb_server file is same as that of the above explained netdb_client source file like the mesh_conn data structure, callback functions and PROCESS and PROTO_THREADS. Below are some of the important code blocks which we will go into details and try to understand.

PROCESS(netdb_process, "NetDB");
AUTOSTART_PROCESSES(&netdb_process);

PROCESS_THREAD(netdb_process, ev, data)
{
  PROCESS_EXITHANDLER(mesh_close(&mesh));
  PROCESS_BEGIN();

  mesh_open(&mesh, NETDB_CHANNEL, &callbacks);
  process_start(&query_process, NULL);

  PROCESS_END();
}

Creation of NetDB process and the protothread for this process is shown in the above code snippet. Inside the protothread of netdb_process we start another process "query_process". Callback functions "sent" and "timedout" for the server program file are implemented exactly like client's callback functions. Only "received" callback function differs in the implementation. Below is the implementation of the "received" call back function at the server side.


static void
received(struct mesh_conn *c, const rimeaddr_t *from, uint8_t hops)
{
  char *data;
  unsigned len;
  static char query[MAX_BUFFER_SIZE + 1];

  data = (char *)packetbuf_dataptr();
  len = packetbuf_datalen();

  if(len > MAX_BUFFER_SIZE) {
    buffer_db_data("Too long query: %d bytes\n", len);
    return;
  }

  memcpy(query, data, len);
  query[len] = '\0';

  printf("Query received from %d.%d: %s (%d hops)\n",
         from->u8[0], from->u8[1], query, (int)hops);
  rimeaddr_copy(&reply_addr, from);

  process_post(&query_process, serial_line_event_message, query);
}

This function gets called when a packet is received on the channel. Here, sanity check(like length of the incoming query crossing the maximum buffer size) on incoming query takes place and "query_process" is called using "process_post" method to process the query and event type is passed as "serial_line_event_message".


PROCESS(query_process, "Query process");
PROCESS_THREAD(query_process, ev, data)
{
   PROCESS_BEGIN();
   ..
   ...
   ....
   PROCESS_END();
}

The above code block creates a process named "query_process" and a protothread for the process "query_process". Let's walk through the code of query_process.

db_init();
db_set_output_function(buffer_db_data);
db_query(NULL, "REMOVE RELATION samples;");
db_query(NULL, "CREATE RELATION samples;");
db_query(NULL, "CREATE ATTRIBUTE time DOMAIN INT IN samples;");
db_query(NULL, "CREATE ATTRIBUTE hum DOMAIN INT IN samples;");
db_query(NULL, "CREATE INDEX samples.time TYPE INLINE;");

The above source code initializes the database. "db_init" function sets up the memory needed for relation, attributes, indexes and everything needed for the antelope database to operate.

"db_query" executes the query passed as argument by parsing it and calling aql_execute(remember AQL is the query language for Antelope database).

#if PREPARE_DB
printf("Preparing the DB with %d tuples...\n", CARDINALITY);
errors = 0;
for(i = 1; i <= CARDINALITY; i++) {
PROCESS_PAUSE();
result = db_query(NULL, "INSERT (%u, %u) INTO samples;",
i, (unsigned)random_rand());
if(DB_ERROR(result)) {
errors++;
}
}
printf("Done. Insertion errors: %d\n", errors);
printf("Ready to process queries\n");
#else
etimer_set(&sampling_timer, SAMPLING_INTERVAL * CLOCK_SECOND);
#endif

The above source code prepares the database with CARDINALITY(1000) number of tuples based on the flag PREPARE_DB. It inserts CARDINALITY number of random integer samples into the database. If everything goes well, you should see the printf statement "Ready to process Queries" in your mote output like shown in the cooja simulation section below.

for(;;) {
    PROCESS_WAIT_EVENT();

    if(ev == serial_line_event_message && data != NULL) {
      printf("START %s\n", (char *)data);
      result = db_query(&handle, data);
      if(DB_ERROR(result)) {
        buffer_db_data("Query error: %s\n", db_get_result_message(result));
        stop_handler(NULL);
        db_free(&handle);
        continue;
      }

      if(!db_processing(&handle)) {
        buffer_db_data("OK\n");
        send_buffered_data();
        stop_handler(NULL);
        continue;
      }

      packetbuf_set_attr(PACKETBUF_ATTR_PACKET_TYPE,
                         PACKETBUF_ATTR_PACKET_TYPE_STREAM);

      db_print_header(&handle);

      matching = 0;
      processed = 0;

      while(db_processing(&handle)) {
        PROCESS_PAUSE();

        if(matching == RESPONSE_LIMIT) {
            buffer_db_data("Response suppressed at %u tuples: limit reached\n",
                           RESPONSE_LIMIT);
            stop_handler(NULL);
            db_free(&handle);
            break;
        }

        result = db_process(&handle);
        if(result == DB_GOT_ROW) {
          /* The processed tuple matched the condition in the query. */
          matching++;
          processed++;
          db_print_tuple(&handle);
        } else if(result == DB_OK) {
          /* A tuple was processed, but did not match the condition. */
          processed++;
          continue;
        } else {
          if(result == DB_FINISHED) {
            /* The processing has finished. Wait for a new command. */
            buffer_db_data("[%ld tuples returned; %ld tuples processed]\n",
                           (long)matching, (long)processed);
            buffer_db_data("OK\n");
          } else if(DB_ERROR(result)) {
            buffer_db_data("Processing error: %s\n",
                           db_get_result_message(result));
          }
          stop_handler(NULL);
          db_free(&handle);
        }
      }
      send_buffered_data();
    }
#if !PREPARE_DB
    if(etimer_expired(&sampling_timer)) {
      take_sample();
      etimer_reset(&sampling_timer);
    }
#endif
}

The above source code runs an infinite for loop which waits for an event to be posted to the process "query_process", data contains the query to be processed and db_query is called to execute the query and error conditions are processed after executing the query. This is the meat of the "query_process" process.

Cooja Simulation-Antelope

In this section, we will go over how to simulate Antelope database behavior in Cooja using sky motes. Navigate to the examples/antelope/netdb folder from the contiki home directory.

cd /examples/antelope/netdb

In this folder, there exists a simulation file "netdb.csc" which we are going to use to simulate antelope in cooja simulator.

Navigate to /tools/cooja/ from the contiki home directory.

cd /tools/cooja/

Type the following command.

sudo ant run

This command will bring up the cooja simulator (Follow this tutorial to learn more about cooja). From the cooja simulator, click File->open simulation -> Browse. Navigate to /examples/antelope/netdb and select "netdb.csc" file and click on open. Your screen should look something like below.

  • Network Window.

Below is the network window for the simulation. In the below network, Mote 51 is the antelope netdb client and Mote 4 is the server.

Antelope network.png

  • Mote output.

Below is the mote output window. A filter "ID:4$" is being employed at the output window so that only mote 4 information is displayed in the mote output window(selective filtering).

Antelope ID4.png

  • Mote Interface Viewer(Sky 51).

Below is the interface window for the antelope database client(Mote 51). We use this window to send queries to server and print the query replies. Antelope interface.png

Click on "start" in the simulation control window. Below are the screen shots for the Mote interface viewer and Mote output for Mote ID:4.

Ready.png
NetdbClient.png

Now, let's try to create a relation(table), attributes(Columns), inserting a tuple into the relation.

  • Creating a relation

We can create a relation (like a table which holds the tuples) with the help of following command.

create relation myrelation1;


Now, create some attributes in this relation.

create attribute id domain int in myrelation1;

The above command creates an attribute named "id" which is of type integer(int) in the relation myrelation1. Similarly we can create other attributes such as "name" of type string etc.

Below is the log of some of the queries that are sent to the server.

Rime started with address 51.0
MAC 33:00:00:00:00:00:00:00 Contiki 2.7 started. Node id is set to 51.
CSMA nullrdc, channel check rate 128 Hz, radio channel 16
Starting 'NetDB'
NetDB client
> create relation sample;
11525 Transmitting query "create relation sample;" to node 4
11674 Reply received from 4.0 (2 hops): OK
> create attribute id domain int in sample;
18093 Transmitting query "create attribute id domain int in sample;" to node 4
18099 Reply received from 4.0 (2 hops): OK
> insert (4) into sample;
25141 Transmitting query "insert (4) into sample;" to node 4
25146 Reply received from 4.0 (2 hops): OK
> insert (23) into sample;
29965 Transmitting query "insert (23) into sample;" to node 4
29970 Reply received from 4.0 (2 hops): OK
> insert (31) into sample;
33323 Transmitting query "insert (31) into sample;" to node 4
33328 Reply received from 4.0 (2 hops): OK
> select max(id) from sample;
41179 Transmitting query "select max(id) from sample;" to node 4
41189 Reply received from 4.0 (2 hops): [relation = db-result, attributes = (id)]
Row 1:31
41191 Reply received from 4.0 (2 hops): [1 tuples returned; 4 tuples processed]
OK

Below is the log file at the server node (Mote ID:4) which shows the processing of the queries we sent from the Mote:51.

00:00.607    ID:4    Rime started with address 4.0
00:00.616    ID:4    MAC 04:00:00:00:00:00:00:00 Contiki 2.7 started. Node id is set to 4.
00:00.623    ID:4    CSMA nullrdc, channel check rate 128 Hz, radio channel 16
00:00.624    ID:4    Starting 'NetDB'
00:00.626    ID:4    NetDB host
00:00.701    ID:4    Preparing the DB with 1000 tuples...
00:08.273    ID:4    Done. Insertion errors: 0
00:08.274    ID:4    Ready to process queries
01:31.565    ID:4    Query received from 51.0: create relation sample; (2 hops)
01:31.567    ID:4    START create relation sample;
01:31.619    ID:4    END
02:21.804    ID:4    Query received from 51.0: create attribute id domain int in sample; (2 hops)
02:21.808    ID:4    START create attribute id domain int in sample;
02:21.817    ID:4    END
03:16.861    ID:4    Query received from 51.0: insert (4) into sample; (2 hops)
03:16.863    ID:4    START insert (4) into sample;
03:16.870    ID:4    END
03:54.549    ID:4    Query received from 51.0: insert (23) into sample; (2 hops)
03:54.551    ID:4    START insert (23) into sample;
03:54.558    ID:4    END
04:20.783    ID:4    Query received from 51.0: insert (31) into sample; (2 hops)
04:20.786    ID:4    START insert (31) into sample;
04:20.792    ID:4    END
05:22.159    ID:4    Query received from 51.0: select max(id) from sample; (2 hops)
05:22.162    ID:4    START select max(id) from sample;
05:22.204    ID:4    END

Below are the snapshots for Antelope client and server logs in cooja.

Client Query.png Server Query.png

Similarly, we could try some other examples like below:

CREATE RELATION sensor;

CREATE ATTRIBUTE id DOMAIN INT IN sensor;
CREATE ATTRIBUTE name DOMAIN STRING(20) IN sensor;
CREATE ATTRIBUTE position DOMAIN LONG IN sensor;

CREATE INDEX sensor.id TYPE INLINE;
CREATE INDEX sensor.position TYPE MAXHEAP;


References

1 - A database in every sensor network, Nicolas Tsiftes ,Adam Dunkels Swedish Institute of Computer Science, Kista, Sweden.

2 - Contiki Operating System Wiki page.


Back to Contiki Tutorials

Edited by Gopi Krishna.