Contiki Coffee File System

From Contiki
Jump to: navigation, search

Back to Contiki Tutorials

Introduction

The tutorial covers the main features of Coffee File System available in ContikiOS 2.7.
Contiki provides a set of file systems for using various kinds of storage devices in resource-constrained systems. All of these file systems implement a subset of the Contiki File System (CFS) interface, and two of them provide the full functionality: CFS-POSIX and Coffee[1]. Coffee is used with device equipped with flash memories or EEPROM. Contiki takes care of underlying hardware implementation. Coffee file system will provide a API which is similar to normal C file operations: Open a file, read and write to it and then close it.
Due to the restrained resource and affected by the environment, the data transmission will get lost sometimes, which then need to be retransmitted, applying file system in the wireless sensor node will help to make the system become more reliable by store the data in the nodes locally. Since we can store the data in the node locally then we can send multiple data via one transmission, which leads to less number of transmission, then reduce the power consumption.

You Will Learn

Through this tutorial you will understand how Coffee File System works and will also learn how to program with the Coffee File System API.

Source Code

The source code for Contiki Coffee File System can be found at,

~/contiki/core/cfs/cfs-coffee.h
~/contiki/core/cfs/cfs-coffee.c

Coffee File System API

Memory Organization

/*
 * The protected memory consists of structures that should not be 
 * overwritten during system checkpointing because they may be used by 
 * the checkpointing implementation. These structures need not be 
 * protected if checkpointing is not used.
 */
static struct protected_mem_t {
  struct file coffee_files[COFFEE_MAX_OPEN_FILES];
  struct file_desc coffee_fd_set[COFFEE_FD_SET_SIZE];
  coffee_page_t next_free;
  char gc_wait;
} protected_mem;
static struct file * const coffee_files = protected_mem.coffee_files;
static struct file_desc * const coffee_fd_set = protected_mem.coffee_fd_set;
static coffee_page_t * const next_free = &protected_mem.next_free;
static char * const gc_wait = &protected_mem.gc_wait;

Coffee-structure.png[1]
Coffee File System organize the file_desc as an array and mark it as a member in the protected_mem_t.

get_available_fd

static int
get_available_fd(void)
{
  int i;

  for(i = 0; i < COFFEE_FD_SET_SIZE; i++) {
    if(coffee_fd_set[i].flags == COFFEE_FD_FREE) {
      return i;
    }
  }
  return -1;
}

Contiki is a Linux like system, everything is a file, and the opened file is represented by the file descriptor. The get_available_fd will return the smallest available fd by check the file_desc array in an ascending order, the first one with flag == 0(COFFEE_FD_FREE) is what we want. If all the flags are set, then return -1.

find_file

static struct file *
find_file(const char *name)
{
  int i;
  struct file_header hdr;
  coffee_page_t page;
  
  /* First check if the file metadata is cached. */
  for(i = 0; i < COFFEE_MAX_OPEN_FILES; i++) {
    if(FILE_FREE(&coffee_files[i])) {
      continue;
    }

    read_header(&hdr, coffee_files[i].page);
    if(HDR_ACTIVE(hdr) && !HDR_LOG(hdr) && strcmp(name, hdr.name) == 0) {
      return &coffee_files[i];
    }
  }
  
  /* Scan the flash memory sequentially otherwise. */
  for(page = 0; page < COFFEE_PAGE_COUNT; page = next_file(page, &hdr)) {
    read_header(&hdr, page);
    if(HDR_ACTIVE(hdr) && !HDR_LOG(hdr) && strcmp(name, hdr.name) == 0) {
      return load_file(page, &hdr);
    }
  }

  return NULL;
}

If the file correspond to the name is still resides in coffee_files[COFFEE_MAX_OPEN_FILES] and the physical file is still valid then it return the pointer to that file, otherwise scan the FLSAH, and cache the file into memory.

The CFS Programming Interface

cfs_open

cf_open is used to open a file. If it successfully open the file then it returns the file descriptor(fd), otherwise it returns -1.
cfs_open source code:

int
cfs_open(const char *name, int flags)
{
  int fd;
  struct file_desc *fdp;

  fd = get_available_fd(); //find the smallest available fd, see section 4.2
  if(fd < 0) {
    PRINTF("Coffee: Failed to allocate a new file descriptor!\n");
    return -1;
  }

  fdp = &coffee_fd_set[fd];
  fdp->flags = 0; //set the fd to FREE

  fdp->file = find_file(name); //find the file corresponding to name(not exist, In flash but not cached, cached)

  /*** if there isn't any corresponding file, then try to create new file ***/
  if(fdp->file == NULL) {
    if((flags & (CFS_READ | CFS_WRITE)) == CFS_READ) {
      return -1;
    }
    fdp->file = reserve(name, page_count(COFFEE_DYN_SIZE), 1, 0);
    if(fdp->file == NULL) {
      return -1;
    }
    fdp->file->end = 0; // Since it's a new created file, the end will be set to 0
  }
   /*** find the file,seek for the end of the file***/ 
   else if(fdp->file->end == UNKNOWN_OFFSET) {
    fdp->file->end = file_end(fdp->file->page);
  }

  fdp->flags |= flags;
  fdp->offset = flags & CFS_APPEND ? fdp->file->end : 0; //if the flag is set to APPEND, then the offset is set to the end of the file, otherwise set to 0
  fdp->file->references++; //reference count will increment

  return fd;
}

Flowchart.png

cfs_close

void cfs_close(int fd)
{
  if (FD_VALID(fd))
  {
    coffee_fd_set[fd].flags = COFFEE_FD_FREE;
    coffee_fd_set[fd].file->references--;
    coffee_fd_set[fd].file = NULL;
  }
}

#define FD_VALID(fd) ((fd)>= 0 && (fd)<COFFEE_FD_SET_SIZE && coffee_fd_set[(fd)].flags!=COFFEE_FD_FREE)

When an open file is no longer needed, the application should close it by using cfs_close(). By closing a file, the file system can deallocate its internal resources held for the file, and possibly commit any cached data to permanent storage.[1]
It first make sure fd is valid, then set the flag in this fd_desc to COFFEE_FD_FREE, and then decrease the reference count of that file. Finally set the file_desc point to NULL.

cfs_read

cfs_read() fills buf with at most len bytes, starting from the current position in the file that is stored in the file descriptor. It returns the amount of bytes read, or -1 if an error occurs.[1]

cfs_write

cfs_write() writes len bytes from the memory buffer buf into the file, starting from the current position in the file descriptor. The file must have been opened with the CFS_WRITE flag. It returns the amount of bytes written, or -1 if an error occurred.[1]

cfs_seek

cfs_seek() moves the current file position to the position determined by the combination of the offset and the whence. CFS_SEEK_SET tells cfs_seek() to compute the offset from the beginning of the file, i.e., as an absolute offset. CFS_SEEK_CUR specifies that the offset should be compute relative to the current position of the file position. Similarly, CFS_SEEK_END computes the offset in relation to the end of the file, and can be used to move beyond the end if the file system implementation allows it. Negative offset values are accepted by both CFS_SEEK_CUR and CFS_SEEK_END, if the program wishes to move the file position backwards from the base that is indicated by the whence parameter. cfs_seek() returns the new absolute file position upon success, or -1 if the file pointer could not be moved to the requested position.[1]

cfs_remove

cfs_remove remove the file correspond to the name. It first find the file by find_file(name). After that invoke remove_by_page to delete the file.

Coding With Coffee API

Modify the code provided by Contiki.
Open contiki-2.7/examples/sky/example-coffee.c, then rewrite the file_test function.
The Idea is based on Zane D. Purvis's Coffee filesystem example.

/*
 * Copyright (c) 2011, Swedish Institute of Computer Science.
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 * 1. Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 * 3. Neither the name of the Institute nor the names of its contributors
 *    may be used to endorse or promote products derived from this software
 *    without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE INSTITUTE AND CONTRIBUTORS ``AS IS'' AND
 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
 * ARE DISCLAIMED.  IN NO EVENT SHALL THE INSTITUTE OR CONTRIBUTORS BE LIABLE
 * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
 * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
 * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
 * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
 * SUCH DAMAGE.
 *
 * This file is part of the Contiki operating system.
 */

#include <stdio.h>
#include "contiki.h"
#include "cfs/cfs.h"
#include "cfs/cfs-coffee.h"
#include <string.h>

PROCESS(example_coffee_process, "Coffee example");
AUTOSTART_PROCESSES(&example_coffee_process);
#define FILENAME "test"
/* Formatting is needed if the storage device is in an unknown state; 
   e.g., when using Coffee on the storage device for the first time. */
#ifndef NEED_FORMATTING
#define NEED_FORMATTING 0
#endif

static int
file_test(const char *filename, char *msg)
{
	int fd;
	int r;
	char message[32];
	char buf[64];
	strncpy(message, "First Message", sizeof(message) - 1);
	message[sizeof(message) - 1] = '\0';
	strcpy(buf,message);
	/*First message is to test if the write will succeed*/
	printf("Write Test: Will write \"%s\" to file \"%s\"\n",buf,FILENAME);

  /* Obtain a file descriptor for the file, capable of handling both 
     reads and writes. */
	fd = cfs_open(FILENAME, CFS_WRITE | CFS_APPEND | CFS_READ);
	if(fd < 0) {
		printf("failed to open %s\n", FILENAME);
		return 0;
	}
  /*Write message to Filesystem*/
	r = cfs_write(fd, message, sizeof(message));
	if(r != sizeof(message)) {
		printf("failed to write %d bytes to %s\n",
				(int)sizeof(message), FILENAME);
		cfs_close(fd);
		return 0;
	}
	cfs_close(fd);
	printf("Write Test: Successfully wrote \"%s\" to \"%s\" wrote %d bytes\n ",message,FILENAME,r);

	strcpy(buf,"fail");
	fd = cfs_open(FILENAME, CFS_READ);
	if(fd < 0) {
		printf("failed to open %s\n", FILENAME);
		return 0;
	}
	r = cfs_read(fd, buf, sizeof(message));
	if(r != sizeof(message)) {
		printf("failed to write %d bytes to %s\n",(int)sizeof(message), FILENAME);
		cfs_close(fd);
		return 0;
	}
      /* compare with the original message to see if the message was read 
      correctly, if it reads fail then it will print fail*/
	printf("Read Test: Read \"%s\" from \"%s\"\n",buf, FILENAME);
	cfs_close(fd);
     /*Append test */
	strcpy(message,"Append Something");
	fd = cfs_open(FILENAME, CFS_WRITE | CFS_APPEND | CFS_READ);
	if(fd < 0) {
		printf("failed to open %s\n", FILENAME);
		return 0;
	}
	r = cfs_write(fd, message, sizeof(message));
	cfs_close(fd);
	printf("Append Test: Successfully \"%s\" to \"%s\" \n ",message,FILENAME);
	strcpy(buf,"fail");
	fd = cfs_open(FILENAME, CFS_READ);
	if(fd < 0) {
		printf("failed to open %s\n", FILENAME);
		return 0;
	}
	cfs_read(fd,buf,sizeof(message));
	printf("Read First Part \"%s\"\n",buf);
       /*seek test*/
	if(cfs_seek(fd, sizeof(message), CFS_SEEK_SET) == -1) {
		printf("seek failed\n");
		cfs_close(fd);
		return 0;
	}
  //cfs_seek(fd, sizeof(message), CFS_SEEK_SET);
    /*if the seek fails then the second message will not 
    be the same as the last message write to the file*/
	cfs_read(fd,buf,sizeof(message));
	printf("Read Second Part: \"%s\"\n",buf);
	cfs_close(fd);
  /* Release the internal resources held by Coffee for
     the file descriptor. */
	cfs_remove(FILENAME);
	fd = cfs_open(FILENAME, CFS_READ);
	if(fd != -1) {
		printf("ERROR: could read from memory\n");
		return 0;
	}
	printf("Successfully removed file\n");
	return 1;
}


PROCESS_THREAD(example_coffee_process, ev, data)
{
  PROCESS_BEGIN();

#if NEED_FORMATTING
  cfs_coffee_format();
#endif

  /* Ensure that we will be working with a new file. */
  cfs_remove(FILENAME);

  if(file_test(FILENAME, "The first test") == 0) {
    printf("file test 1 failed\n");
  }
  printf("test succeed\n");

  PROCESS_END();
}

Simulation Result

Result.png

We can see that Coffee File System works well.

General Issues

1. Use sizeof() instead strlen() in the strncpy() function.
2. Remember to include string.h, otherwise when you compile it will generate warnings.

Reference

[1] - Zane D. Purvis, "Coffee filesystem"
[2] - Zane D. Purvis, "Coffee filesystem example"


Back to Contiki Tutorials

Edited by: Zhikun Liu