Tutorial

...however this example is somewhat too simplistic because, normally, a Vstr string contains multiple nodes of informtion that are treated internally as a single entity.

So first we'll clean up the above example to move all the header inclusion into one file and create some helper functions to simplify the actually "hello world" code. Here's a quick overview of the functions moved into the header file...

• io_put: This function will move data from a Vstr string to a file descriptor (doesn't work well with non-blocking mode file descriptors).
• hw_init: This initializes the library and creates a Vstr string objects.
• hw_exit: This does cleanup on the resources aquired in hw_init.

This example is available here (2.07KB, Last modified: Mon Sep 22 15:07:00 2003).

Now using the above header file, we can re-write the initial example in a much more readable form...

/* hello world - Using a single piece of data (no data copying) */

#include "ex_hello_world.h" /* helper functions */

int main(void)
{
  Vstr_base *s1 = hw_init();

  vstr_add_cstr_buf(s1, s1->len, "Hello World\n");

  if (s1->conf->malloc_bad)
    errno = ENOMEM, err(EXIT_FAILURE, "Add string data");

  while (io_put(s1, STDOUT_FILENO) != IO_NONE) {}

  exit (hw_exit(s1));
}

We can now alter the above to still print "Hello World" on a line, but have the data come from multiple sources which is then stored internally in mutliple nodes (remember there is no copying of data still). However can treat this all as a single string from our point of view. Although this example is obviously contrived, this is much more representative of what a networked application looks like...

/* hello world - multiple sources of data (still no data copying) */

#include "ex_hello_world.h" /* helper functions */

int main(void)
{
  Vstr_base *s1 = hw_init();

  vstr_add_cstr_ptr(s1, s1->len, "Hello");
  vstr_add_cstr_ptr(s1, s1->len, " ");
  vstr_add_cstr_ptr(s1, s1->len, "World\n");

  /* we are checking whether any of the above three functions failed here */
  if (s1->conf->malloc_bad)
    errno = ENOMEM, err(EXIT_FAILURE, "Add string data");

  /* loop until all data is output... */
  while (io_put(s1, STDOUT_FILENO) != IO_NONE) {}

  exit (hw_exit(s1));
}

This is the final Hello World example, this is the first one that actually copies some of the data for the string. It also shows how you can add data at any point in the string, and substitute data within the string. Note that when giving the position to add data to the string you give the position before the position you wish the start of the data to be at, and when giving the position/length for a section (or sub-string) the position given is included within the section.

For people familiar with C++ this works out to be the same way that C++ std::string encodes positions for adding data (Ie. insert()), but not for getting sections (C++ often explains it as having a 0 index'd string and data is added before the point given, Vstr does uses a 1 poition for the first byte as that means that appending data is always done at the current length ... and a position of 0 can be used as invalid for searching etc.).

/* hello world - multiple pieces of data, includes substitution and
 *               inserting data into the middle of a string */

#include "ex_hello_world.h" /* helper functions */

int main(void)
{
  Vstr_base *s1 = hw_init();
  Vstr_ref *ref = NULL;

  vstr_add_cstr_ptr(s1, s1->len, "Hello");

  vstr_add_rep_chr(s1, s1->len, 'W', 5); /* add "WWWWWW" */

  if (s1->conf->malloc_bad)
    errno = ENOMEM, err(EXIT_FAILURE, "Add string data");

  /* substitute an 'o' for a 'W' */
  if (!vstr_sub_rep_chr(s1, strlen("HelloWW"), 1, 'o', 1))
    errno = ENOMEM, err(EXIT_FAILURE, "Substitute string data");

  /* substitute "WWW" for a "rld\n" -- */
  if (!vstr_sub_cstr_buf(s1, strlen("HelloWoW"), strlen("WWW"), "rld\n"))
    errno = ENOMEM, err(EXIT_FAILURE, "Substitute string data");

  if (!(ref = vstr_ref_make_ptr((char *)"XYZ ", vstr_ref_cb_free_ref)))
    errno = ENOMEM, err(EXIT_FAILURE, "Create data reference");
  /* now ref->ptr is "XYZ " */

  /* add space after "Hello", by skipping "XYZ" in reference */
  vstr_add_ref(s1, strlen("Hello"), ref, strlen("XYZ"), strlen(" "));

  vstr_ref_del(ref); /* delete our reference to the Vstr_ref */

  if (s1->conf->malloc_bad)
    errno = ENOMEM, err(EXIT_FAILURE, "Add string data");

  while (io_put(s1, STDOUT_FILENO) != IO_NONE) {}

  exit (hw_exit(s1));
}

This is a full header file needed to do simple non-blocking IO operations, it also puts into functions the common init and exit sections. This will be used by all of the following examples. The Simple GETOPT implementation isn't used for a while, so you can ignore that for now. A quick overview of the changes from the hello_world.h header file are...

• io_put: This can now work with non-blocking file descriptors.
• ex_init: (renamed hw_init) This now alters the Node size to be more efficient for IO, can create another Vstr string object, will create some data storage space for use by either Vstr string object and sets standard output to be non-blocking (allowing us to block on input and output at the same time).
• ex_exit: (renamed hw_init) This can now cleanup two Vstr string objects.

And a quick review of the additions...

• io_block: This is how applications block, now that all IO should be in non-blocking mode. This API only allows a single input and single output source to be blocked on, which is fine for the following examples (but networked applications will often wish to block on many sources).
• io_put_all: This can thought of as a blocking io_put(), even when dealing with non-blocking IO.
• io_get: This is a non-blocking read wrapper, in the same way that io_put is a non-blocking write wrapper.
• io_limit: This is calls io_block if the return values from both io_get and io_put are IO_BLOCK or either one is in the IO_BLOCK state and must be dealt with now.
• io_fd_set_o_nonblock: This is a simple function to set the O_NONBLOCK flag on a file descriptor.
• io_open: This is a simple function to open a file, and set the O_NONBLOCK flag on it (note that although real files tend not to honour the O_NONBLOCK flag, file system objects might not be files).

This example is available here (12.14KB, Last modified: Tue May 17 20:04:00 2005).

This is the unix cat command, implemented with the help of the functions in the above ex_utils.h header file. This uses the same Vstr string for both input and output and uses non-blocking IO, both of which are done for efficiency.

If you have seen something like the "simple" cat in LAD this version looks much bigger. However one of the main reasons for this is that the LAD version has many bugs. The main problem is the lack of checking on the IO calls, this is most easily demonstrated by running it like so...

perl -e 'use Fcntl; fcntl(STDIN, F_SETFL, O_NONBLOCK); exec(@ARGV);' ./cat

...will cause the LAD version to exit immediatley due to an EAGAIN being returned from read. This problem also affects the LAD version being used in a blocking pipe due to the fact that write() isn't required to write() all it's data.

The LAD version also doesn't open() any files, which is significant functionality. So after fixing those bugs we get something that is much closer to the Vstr version and it still suffers from performance problems due to the need to block on input and output separately. It is possible to create a version using read, write and poll that would perform the same as the Vstr version ... however even the simplest method would have to implement it's own ring buffer which is very prone to error and would almost certainly make it bigger than ex_cat.c and ex_utils.h combined.

/* This is a _simple_ cat program.
 * Reads from stdin if no args are given.
 *
 * This shows how to use the Vstr library at it's simpelest,
 * for easy and fast IO. Note however that all needed error detection is
 * included.
 *
 * This file is more commented than normal code, so as to make it easy to follow
 * while knowing almost nothing about Vstr or Linux IO programming.
 */
#include "ex_utils.h"

#define CONF_USE_MMAP_DEF FALSE

/*  Keep reading on the file descriptor until there is no more data (ERR_EOF)
 * abort if there is an error reading or writing */
static void ex_cat_read_fd_write_stdout(Vstr_base *s1, int fd)
{
  while (TRUE)
  {
    int io_w_state = IO_OK;
    int io_r_state = io_get(s1, fd);

    if (io_r_state == IO_EOF)
      break;
    
    io_w_state = io_put(s1, STDOUT_FILENO);

    io_limit(io_r_state, fd, io_w_state, STDOUT_FILENO, s1);    
  }
}

static void ex_cat_limit(Vstr_base *s1)
{
  while ((s1->len >= EX_MAX_W_DATA_INCORE) || (s1->len >= EX_MAX_R_DATA_INCORE))
  {
    if (io_put(s1, STDOUT_FILENO) == IO_BLOCK)
      io_block(-1, STDOUT_FILENO);
  }
}

/* This is "cat", using non-blocking IO and Vstr for buffer space */
int main(int argc, char *argv[])
{
  Vstr_base *s1 = ex_init(NULL); /* init the library etc. */
  int count = 1; /* skip the program name */
  int use_mmap = CONF_USE_MMAP_DEF;  
  
  /* parse command line arguments... */
  while (count < argc)
  { /* quick hack getopt_long */
    if (!strcmp("--", argv[count]))
    {
      ++count;
      break;
    }
    else if (!strcmp("--mmap", argv[count])) /* toggle use of mmap */
      use_mmap = !use_mmap;
    else if (!strcmp("--version", argv[count]))
    { /* print version and exit */
      vstr_add_fmt(s1, 0, "%s", "\
jcat 1.0.0\n\
Written by James Antill\n\
\n\
Uses Vstr string library.\n\
");
      goto out;
    }
    else if (!strcmp("--help", argv[count]))
    { /* print version and exit */
      vstr_add_fmt(s1, 0, "%s", "\
Usage: jcat [FILENAME]...\n\
   or: jcat OPTION\n\
Output filenames.\n\
\n\
      --help     Display this help and exit\n\
      --version  Output version information and exit\n\
      --mmap     Toggle use of mmap() to load input files\n\
      --         Treat rest of cmd line as input filenames\n\
\n\
Report bugs to James Antill <james@and.org>.\n\
");
      goto out;
    }
    else
      break;
    ++count;
  }
  
  /* if no arguments are given just do stdin to stdout */
  if (count >= argc)
  {
    io_fd_set_o_nonblock(STDIN_FILENO);
    ex_cat_read_fd_write_stdout(s1, STDIN_FILENO);
  }
  
  /* loop through all arguments, open the file specified
   * and do the read/write loop */
  while (count < argc)
  {
    unsigned int ern = 0;

    if (use_mmap)
      vstr_sc_mmap_file(s1, s1->len, argv[count], 0, 0, &ern);

    if (!use_mmap ||
        (ern == VSTR_TYPE_SC_MMAP_FILE_ERR_FSTAT_ERRNO) ||
        (ern == VSTR_TYPE_SC_MMAP_FILE_ERR_MMAP_ERRNO) ||
        (ern == VSTR_TYPE_SC_MMAP_FILE_ERR_TOO_LARGE))
    { /* if mmap didn't work ... do a read/alter/write loop */
      int fd = io_open(argv[count]);

      ex_cat_read_fd_write_stdout(s1, fd);

      if (close(fd) == -1)
        warn("close(%s)", argv[count]);
    }
    else if (ern && (ern != VSTR_TYPE_SC_MMAP_FILE_ERR_CLOSE_ERRNO))
      err(EXIT_FAILURE, "add");
    else /* mmap worked */
      ex_cat_limit(s1);

    ++count;
  }

  /* output all remaining data */
 out:
  io_put_all(s1, STDOUT_FILENO);

  exit (ex_exit(s1, NULL));
}

This is somewhat like the "nl" unix command, this is implemented in much the same way as the cat command. However the data has to have something added to the start of each line before it can be output, so we now have two string objects: One for input and one for output. Note that as the data is "moved" from the input to the output string object, it isn't copied instead a reference is created and shared between the two strings.

This example is available here (6.07KB, Last modified: Fri Oct 15 16:21:07 2004).

This is somewhat like the "hexdump" unix command, it also uses the same simple IO model used in the cat and nl commands. However the data is now output twice once as hex values, and a second time as characters (converting unprintable characters into '.' characters). So again we have two string objects: One for input and one for output. To get two copies of the data, we initially export the data to a buffer and then convert that to hex via vstr_add_fmt() (the printf like function). We then convert the data that is still in the string object so it is printable and move it from the input to the output string objcet. Note that as the data is "moved" from the input to the output string object, it is always copied even if it was a reference on input (Ie. mmap()ed).

One other thing that is new in the hexdump command is the use of the VSTR_FLAGXX() macro fucntion, this is a convienience feature for when you need to specify multiple flags at once.

This example is available here (5.36KB, Last modified: Mon Mar 14 23:46:21 2005).

Custom formatters are an exteremly useful feature of the Vstr string library, allowing you to safely implement one or more methods of printing a pointer to an arbitrary object. The most simple uses are to use the builtin custom formatters. This examples shows how you enable the IPv4 and Vstr custom formatters and then use how you can use them.

This example also includes vstr_sc_basename() which acts in a similar way to the POSIX basename function.

This example is available here (2.56KB, Last modified: Mon Feb 2 08:45:42 2004).

So, printing ipv4 address is nice ... but the big benifit, with custom formatters, come with using them on types that you have to deal with a lot. This usually means types that you've defined yourself, or are defined in a library you are using. So this example will show you how to create your own custom formatters. I'll use the GNU multiple precision arithmetic library which is a well used library for creating arbitrary precision numbers (Ie. numbers that can represent any value). One of the annoying features of using this library is that it is non-trivial to easily turn these numbers into strings and/or output them as you would with a normal int/long/size_t/intmax_t/etc.

The GMP library has a set of printf like functions, and while you can create a limited length string, newly allocated string or create output to a file using these functions they fail the "Easy" test for a number of reasons...

• It is not possible to use the grouping format specifier with the GNU MP variables. This means that when running under Linux/glibc, while...

  int d;
  char *ret = NULL;
  /* ... */
  gmp_asprintf(&ret, "%s is an num %'d.\n", "int", d);
  

  ...creates a C style string for the number in a readable format for the locale, the GNU MP variable equvilent...

  mpz_t z;
  char *ret = NULL;
  /* ... */
  gmp_asprintf(&ret, "%s is an bignum %'Zd.\n", "mpz_t", z);
  

  ...doesn't do anything other than create the number.

• Any static printf like function format specifier checking has to be disabled for these functions due to the way the hard coded custom formaters are implemented. This means that although if you do...

  int d;
  char *ret = NULL;
  /* ... */
  asprintf(&ret, "%.*s is an num %d.\n", "int", d);
  

  ...gcc will tell you there is an error, when you do...

  mpz_t z;
  char *ret = NULL;
  /* ... */
  gmp_asprintf(&ret, "%.*s is an bignum %Zd.\n", "mpz_t", z);
  

  ...the code will happily compile, and then almost certainly crash when you run it.

• You also lose all of the great benifits of using vstr_add_fmt(), the biggest problem here is that the gmp printf like functions pass all non gmp types directly to the underlying snprintf/etc. call. Although the extra speed/memory benifits of Vstr are nice too :).

• Even if all of these problems were fixed in some future version, this still only solves the problem for GMP types. So if you have one or more other custom types that you need to format, you'd need yet another function.

The custom formatter for mpz_t is about 25 lines of code in this example, it could be a little less if you removed some features (Ie. supporting positive or negative values) or always got libgmp to allocate storage and then free it (or if some libgmp APIs were defined in a more user friendly manner).

However the actual complexity is pretty small, and this not only fully implements everything that gmp_printf() can do for that variable (safely), but also implements grouping.

This example is available here (7.13KB, Last modified: Fri Mar 5 17:46:26 2004).

The above implements the equivalent of "%d", however with a couple of 1 line changes "%x" and "%o" can be done (and they can be signed, if desired). For instance, here is just the custom formatter callback code to implement all three...

This example is available here (9.88KB, Last modified: Tue Jul 26 04:48:22 2005).

This is a mapping from std. C functions operating on C-style strings to Vstr string functions...

This is a mapping from C++ std::string functions to Vstr string functions...