Author Topic: Spaghety file format and AsyncIO with small files (Read 1204 times)

Jose · « **on:** March 10, 2006, 07:59:51 PM »

I have this multiple linked list or whatever (tree like but not with those rules, just multiple pointers in each node to multiple nodes with multiple pointers...) I'm saving to HD. I decided to save each list to a unique file that is kept open till all nodes on the tree are last updated and saved. Most node are structures whose fields aren't all saved so I have to take off some fields of each one of them before saving. The pointers in each node to others are saved as offsets to where the other node was saved. This results in a great mess specially since there's no prediction possible to when each node will be updated and saved. The list also has no limits on size.

Questions:
1- In this situation, would using AsyncIO still be worth it considering each node's size on average is bellow 0,5K ?
2- Do you guys think it would be worth the hassle putting some code there to only save nodes to HD in bigger clusters of various nodes ? This would take probably some time and delay the whole thing. Besides if power goes down and there's not a UPS being used the file won't be as up to date.

3- I can't remember some parts of the code I coded myself some moths ago and I'm having to read and understand it all again:-o I am just getting mad or it's normall ? :-D

Thomas · « **Reply #1 on:** March 10, 2006, 09:29:10 PM »

1. using asyncio.library only brings a benefit if you read a big file sequentially. In your case it does not even work because as I understood you randomly seek forth and back in the file.

However, you probably could still improve speed somewhat if you did asynchronous I/O yourself by sending packets to the file handler instead of using the Read() and Write() functions. So you start a write operation and continue creating other nodes and when you have to write the next time, you first wait for the first operation to finish before you start the next one.

2. That's something you have to decide on your own. If you think your program needs optimisation regarding speed and if you think that chunking nodes together would improve speed, then you should consider doing it.

But if you think that it is too much work or too complicated and that speed is ok now, then let it as it is.

3. It's normal. /* That's why every programming language has the ability to put comments into the code. */ :-D

Bye,
Thomas

Jose · « **Reply #2 on:** March 10, 2006, 09:47:23 PM »

Thanks Thomas. I don't know if I'll have the patience to learn to work with Dos packets but I'll take a look in the DevCD.

P.S.
My code was documented...

Thomas · « **Reply #3 on:** March 11, 2006, 02:30:22 PM »

It is not as diffiult as it seems. Here is an exmaple which copies a file with asynchronous I/O:

Code: [Select]


#include <proto/exec.h>
#include <proto/dos.h>


struct MsgPort *StartRead (BPTR file,APTR buffer,LONG size,struct MsgPort *port)

{
struct FileHandle *fh = BADDR(file);
struct DosPacket *packet;

if (fh && fh->fh_Type)
 {
 if (packet = AllocDosObject (DOS_STDPKT,TAG_END))
  {
  packet->dp_Port = port;

  packet->dp_Type = ACTION_READ;
  packet->dp_Arg1 = fh->fh_Arg1;
  packet->dp_Arg2 = (LONG)buffer;
  packet->dp_Arg3 = (LONG)size;

  PutMsg (fh->fh_Type,packet->dp_Link);
  return (port);
  }
 }

return (NULL);
}



struct MsgPort *StartWrite (BPTR file,APTR buffer,LONG size,struct MsgPort *port)

{
struct FileHandle *fh = BADDR(file);
struct DosPacket *packet;

if (fh && fh->fh_Type)
 {
 if (packet = AllocDosObject (DOS_STDPKT,TAG_END))
  {
  packet->dp_Port = port;

  packet->dp_Type = ACTION_WRITE;
  packet->dp_Arg1 = fh->fh_Arg1;
  packet->dp_Arg2 = (LONG)buffer;
  packet->dp_Arg3 = (LONG)size;

  PutMsg (fh->fh_Type,packet->dp_Link);
  return (port);
  }
 }

return (NULL);
}



LONG WaitDosIO (struct MsgPort *port)

{
struct Message *msg;
struct DosPacket *packet;
long rc;

WaitPort (port);
msg = GetMsg (port);
packet = (struct DosPacket *) msg->mn_Node.ln_Name;
rc = packet->dp_Res1;
FreeDosObject (DOS_STDPKT,packet);

return (rc);
}



#define BUFFSIZE 1024

int main (void)

{
struct RDArgs *rdargs;
struct {
 char *in_file;
 char *out_file;
 } args;
BPTR in_file;
BPTR out_file;
struct MsgPort *in_port;
struct MsgPort *out_port;
UBYTE buffer1[BUFFSIZE];
UBYTE buffer2[BUFFSIZE];
UBYTE *buffer;
long bytes_read;

if (rdargs = ReadArgs (&quot;INFILE/A,OUTFILE/A&quot;,(APTR)&args,NULL))
 {
 if (in_port = CreateMsgPort())
  {
  if (out_port = CreateMsgPort())
   {
   if (in_file = Open (args.in_file,MODE_OLDFILE))
    {
    if (out_file = Open (args.out_file,MODE_NEWFILE))
     {
     buffer = buffer1;
     bytes_read = Read (in_file,buffer,BUFFSIZE);
     while (bytes_read > 0)
      {
      StartWrite (out_file,buffer,bytes_read,out_port);
      buffer = (buffer == buffer1 ? buffer2 : buffer1);
      StartRead (in_file,buffer,BUFFSIZE,in_port);
      WaitDosIO (out_port);
      bytes_read = WaitDosIO (in_port);
      }

     Close (out_file);
     }
    else
     PrintFault (IoErr(),args.out_file);

    Close (in_file);
    }
   else
    PrintFault (IoErr(),args.in_file);

   DeleteMsgPort (out_port);
   }
  DeleteMsgPort (in_port);
  }

 FreeArgs (rdargs);
 }
else
 PrintFault (IoErr(),NULL);

return (0);
}

You can use the functions StartRead, StartWrite and WaitDosIO in you program.

Please note that between StartRead and WaitDosIO and between StartWrite and WaitDosIO you must not use the file handle in any other function and you must not use the MsgPort for any other purpose than to Wait() for I/O completion.

If you use Wait() to wait for I/O completion you still have to call WaitDosIO in order to clean up and to receive the result. This works similar to exec.library's SendIO and WaitIO. However, there is no AbortIO for DOS packets.

Bye,
Thomas

Author Topic: Spaghety file format and AsyncIO with small files (Read 1204 times)

Jose

Spaghety file format and AsyncIO with small files

Thomas

Re: Spaghety file format and AsyncIO with small files

Jose

Re: Spaghety file format and AsyncIO with small files

Thomas

Re: Spaghety file format and AsyncIO with small files