April 1998

Manipulating memory with TMemoryStream

by Kent Reisdorph

In the March issue of C++Builder Developer's Journal, the article "File I/O" explained the TFileStream class and how you can use this class to read and write to files on disk. Sometimes, however, you need to manipulate a chunk of data in memory rather than on disk. The good news is that TFileStream has a cousin called TMemoryStream designed for exactly this purpose. In this article, we'll show you how to use TMemoryStream to treat memory the same way you treated a file in our March article. We'll demonstrate how to use the same functions to read and write to memory as you would when performing file I/O.

OK, but why?

It may not immediately be apparent why you might want to deal with memory as you would a file. There are several good reasons, but the most compelling is speed of execution. Any time you manipulate data, execution speed is a concern. Let's say, for instance, that you want to save data to a file at a time-critical point in your program. You might not be able to afford the overhead associated with writing to a file on disk during a real-time process. Instead, you could write the data to memory, then copy the data from memory to disk after the real-time process is complete.

Here's another example: Sometimes you need to read a file from disk, manipulate the file in some way (encryption/decryption, for instance), and then write the file out again. Writing the file to memory and making the modifications there is much faster than reading the file, modifying data, then writing the file every time changes are necessary.

Finally, if you've ever performed operations on string data and then copied those strings to a TMemo, you know how slow this process can be. As an alternative, you can load the strings into memory and work with them there, copying the result to TMemo only when the data has been fully altered. We'll show you an example in just a bit, but for now let's take a closer look at what TMemoryStream offers.

TMemoryStream basics

TMemoryStream provides several properties and methods for working with data in memory. Table A lists the primary ones.

Table A: Important TMemoryStream properties and methods
PropertyDescription
Position The current value of the stream position indicator.
Position is a read/write property.
Size The number of bytes of data currently in the stream.

Method CopyFrom()

Copies a specified number of bytes from a stream to this stream.
LoadFromFile() Fills the memory stream with the contents of the specified file.
LoadFromStream() Fills the memory stream from another stream (either file or memory).
Read() Reads a specified number of bytes from the stream to the specified memory location (such as an array).
Write() Writes a specified number of bytes from a memory location to the stream.
SaveToFile() Saves the contents of the memory stream to a disk file.
SaveToStream() Saves the contents of the memory stream to another stream.
Seek() Moves the file-position indicator by the specified amount from the start of the file, from the end of the file, or from the current position.
SetSize() Allocates the specified amount of memory for the stream.

We covered the general concept of streams last month in "File I/O." The TMemoryStream properties and methods work identically to the TFileStream properties and methods discussed in that article. Rather than cover that ground again, let's take a look at some of the methods particular to TMemoryStream.

The Loadxxx and Savexxx methods simply make it easy to load or save a stream to and from either a file on disk or another memory stream. For example, you could load a file on disk, manipulate the data in memory, and save it back to disk, as follows:

TMemoryStream* stream = new TMemoryStream;
stream->LoadFromFile("myfile.dat");
// do some stuff to the stream
stream->SaveToFile("myfile.dat");
delete stream;
The SetSize() method allows you to set the amount of memory allocated for the stream. Setting the size isn't a requirement, because TMemoryStream will automatically allocate and deallocate memory as needed. If, however, you know the minimum initial size you'll need, you can set the size to that value, saving quite a few clock cycles if your operation is time-critical. For example, we ran tests on writing a 324Kb memory stream one byte at a time. Memory for streams is allocated in 8Kb blocks. Therefore, writing 324Kb of data to a stream results in 40 memory reallocations. When we set the stream size to 324Kb prior to writing the data, the average time of the operation decreased from 950 milliseconds to 570 milliseconds by letting VCL handle the memory automatically. That's a significant savings.

An example

Probably the best thing we can do at this point is to provide an example. Let's say you want to obfuscate a text file. You can do so simply by altering the ASCII value of each character slightly, in effect scrambling the file to make it unreadable. Later, you can descramble the file to view it. Or, better yet, you can descramble the file and send the text directly to a Memo component. The whole operation might look as follows:
TMemoryStream* stream = new TMemoryStream;
stream->LoadFromFile(OpenDialog1->FileName);
stream->Position = 0;
for (unsigned int i=0;i<stream->Size;i++) {
  char c;
  stream->Read(&c, 1);
  c += 32;
  stream->Position--;
  stream->Write(&c, 1);
}
// file is scrambled in memory 
// later... descramble and display
stream->Position = 0;
for (unsigned int i=0;i<stream->Size;i++) {
  char c;
  stream->Read(&c, 1);
  c -= 32;
  stream->Position--;
  stream->Write(&c, 1);
}
stream->Position = 0;
Memo1->Lines->LoadFromStream(stream);
Most of this code is pretty straightforward. Notice this line, though:
stream->Position--;
After you read the character at the current stream position, the stream-position counter advances by one. You need to back up one character before you write the scrambled character to the stream. Notice also that once you've descrambled the stream, you can easily load it into the Memo component using the LoadFromStream() method. This method, which is actually a member of TStringList, is very fast and powerful. You can use it in combination with memory streams anywhere you use string lists (and many VCL components use TStringList to store their data).

Listings A and B contain a program that scrambles a text file, descrambles it, and then displays the file in a Memo component. The elapsed time of the operation is displayed in a Label component. This program illustrates the power of memory streams and shows the speed of working with data via memory streams compared to working directly with the Memo's text.

Listing A: STREAMU.H

#ifndef StreamUH
#define StreamUH
//------------------------------------
	---------#include <vcl\Classes.hpp>
#include <vcl\Controls.hpp>
#include <vcl\StdCtrls.hpp>
#include <vcl\Forms.hpp>
#include <vcl\Dialogs.hpp>
//-------------------------------
	--------------class TForm1 : public TForm
{
__published:  // IDE-managed Components
  TMemo *Memo1;
  TLabel *Label1;
  TButton *Button3;
  TLabel *Label2;
  TOpenDialog *OpenDialog1;
  TGroupBox *GroupBox1;
  TButton *Button1;
  TButton *Button2;
  void __fastcall Button1Click(TObject *Sender);
  void __fastcall Button2Click(TObject *Sender);
  void __fastcall Button3Click(TObject *Sender);
  void __fastcall FormCreate(TObject *Sender);
private:  // User declarations
public:   // User declarations
  __fastcall TForm1(TComponent* Owner);
};
//-------------------------------
	--------------extern TForm1 *Form1;
//---------------------------------------------#endif
Listing B: STREAMU.CPP
#include <vcl\vcl.h>
#pragma hdrstop

#include "StreamU.h"
//-----------------------------
	----------------#pragma resource "*.dfm"
TForm1 *Form1;
//--------------------------------------
	-------__fastcall TForm1::TForm1(TComponent* Owner)
  : TForm(Owner)
{
}
//---------------------------------------------
void __fastcall 
TForm1::Button1Click(TObject *Sender)
{
  int start = GetTickCount();
  Memo1->Lines->
    LoadFromFile(OpenDialog1->FileName);
  char* buff = Memo1->Lines->GetText();
  Screen->Cursor = crHourGlass;
  for (unsigned int i=0;i<strlen(buff);i++) {
    char c = buff[i];
    c += 32;
    buff[i] = c;
  }
  for (unsigned int i=0;i<strlen(buff);i++) {
    char c = buff[i];
    c -= 32;
    buff[i] = c;
  }
  Memo1->Lines->Text = buff;
  Screen->Cursor = crDefault;
  char buff2[20];
  sprintf(buff2, "%.02f seconds",
    (GetTickCount() - start)/1000.0);
  Label2->Caption = buff2;
}
//---------------------------------------------
void __fastcall TForm1::Button2Click(TObject *Sender)
{
  TMemoryStream* stream = new TMemoryStream;
  stream->LoadFromFile(OpenDialog1->FileName);
  Screen->Cursor = crHourGlass;
  stream->Position = 0;
  for (unsigned int i=0;i<stream->Size;i++) {
    char c;
    stream->Read(&c, 1);
    c += 32;
    stream->Position--;
    stream->Write(&c, 1);
  }
  stream->Position = 0;
  for (unsigned int i=0;i<stream->Size;i++) {
    char c;
    stream->Read(&c, 1);
    c -= 32;
    stream->Position--;
    stream->Write(&c, 1);
  }
  stream->Position = 0;
  Memo1->Lines->LoadFromStream(stream);
  Screen->Cursor = crDefault;
  delete stream;
  char buff[20];
  sprintf(buff, "%.02f seconds", 
    (GetTickCount() - start)/1000.0);
  Label2->Caption = buff;
}
//---------------------------------------------
void __fastcall 
TForm1::Button3Click(TObject *Sender)
{
  if (OpenDialog1->Execute()) {
    Button1->Enabled = true;
    Button2->Enabled = true;
    Memo1->Text = 
      "File Selected: " + OpenDialog1->FileName;
  }
}
//---------------------------------------------
void __fastcall 
TForm1::FormCreate(TObject *Sender)
{
  Button1->Enabled = false;
  Button2->Enabled = false;
}
Run the program and try both methods. Be sure you try the program with some larger text files (more than 50Kb) in order to see the difference that memory streams can make. You'll find that the memory stream method is as much as 100 times faster than working directly with the Memo text.

Conclusion

Memory streams are very powerful. Once you learn how to use TMemoryStream, particularly in combination with TFileStream and TStringList, new programming vistas will open before your eyes.