Fast File Traversal in C#

If you have tried to use the .NET method for traversing files and directories you have probably encountered problems. Problems include access denied exception and slow execution. File access seems to be implemented by .NET and NOT by the operating system. The code supplied in this blog was successfully run on Windows 10, 64 bit. The code was developed using Visual studio 2022 using .NET Core 5.0.

Why are the .NET methods troublesome?

In my opinion the .NET framework API contains some great code. The problem is that some API methods are somewhat lacking. The idea that traversing a file structure may throw exceptions seems a little bizarre.

Most of the methods for traversing the file structure use IEnumerable. Ultimately, using IEnumerable is a good idea. Problems arise however, due to the fact that an IEnumerable implementation, upon receiving an exception, tends to close the underlying stream. This is certainly the case for methods that obtain a list of directories or files.

I think the file system API, especially for traversal, is very poor. I should be able to view all directories and files. Attempting to open/modify said directories or files is a different matter (maybe this is where .NET file security should kick in?).

Top

Under the hood

Some of the .NET implementations appear to be wrappers over the Win32 API. The Win32 API, is old-school, C functions, of which there are many. Calling Win32 functions from a C# app incurrs overhead due to the marshalling of types. In my experiece, the overhead is not significant, but, may well be in tight loops. (So, just something to be aware of.)

Top

The code

using System;
using System.Collections.Generic;
using System.IO;
using System.Runtime.InteropServices;

public static partial class Functional
{
  #region Constants
  public static readonly IntPtr InvalidHandle = new IntPtr(-1);
  #endregion // Constants

  #region Data
  [StructLayout(LayoutKind.Sequential)]
  struct FILE_TIME
  {
    public uint Low;
    public uint High;
  }

  [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
  struct WIN32_FIND_DATA
  {
    public int Attributes;
    public FILE_TIME Created;
    public FILE_TIME LastAccessed;
    public FILE_TIME LastWrite;
    public int SizeLow;
    public int SizeHigh;
    public int Reserved;
    public int Reserved2;

    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 260)]
    public string Name;

    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 14)]
    public string AlternateName;

    public uint Type;
    public uint CreatorType;
    public uint FinderFlag;

    public bool IsValidDirectory()
    {
      int len = Name.Length;
      return len > 2 ||
        (len == 1) && (Name[0] != '.') ||
        (len == 2) && (Name[0] != '.') && (Name[1] != '.');
    }
  }
  #endregion // Data

  #region Imports
  [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
  private static extern IntPtr FindFirstFile(
  string fileName,
  [Out] out WIN32_FIND_DATA data);

  [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
  private static extern bool FindNextFile(
    IntPtr hndFindFile,
    [Out] out WIN32_FIND_DATA lpFindFileData);

  [DllImport("kernel32.dll")]
  private static extern bool FindClose(IntPtr handle);
  #endregion // Imports

  public static void FindFiles(string root)
  {
    WIN32_FIND_DATA fd = new WIN32_FIND_DATA();
    Stack<string> dirs = new Stack<string>();

    dirs.Push(root);

    while (dirs.Count > 0)
    {
      string curDir = dirs.Pop();
      string search = Path.Combine(curDir, "*.*");
      IntPtr handle = FindFirstFile(search, out fd);
      if (handle != InvalidHandle)
      {
        bool valid = true;
        while (valid)
        {
          bool isDir = 0 != (fd.Attributes & (int)FileAttributes.Directory);
          if (isDir && fd.IsValidDirectory())
            dirs.Push(Path.Combine(curDir, fd.Name));

          valid = FindNextFile(handle, out fd);
        }
        FindClose(handle);
      }
    }
  }
}

Top

Summary

First, note that the code does not yield/return useful results. The code simply serves as a starting point. It is fast, and can be modified accordingly to suit project needs.

Memory allocations (mainly strings as to be expected) are quite high. I tried using string interning, StringBuilder to build paths and so on. Some techniques yielded slightly less memory consumption, nothing so drastic as to make me think "yeah I should post that".

Top

C# Saving State

In this blog I show how one can save program state easily in XML format. Existing .NET code allows this but is somewhat hard to use. I have used the method I will show in many projects.

Overview

For this to work, you need to ensure that all data you wish to persist is can be converted to XML. This is usually as simple as placing the XmlSerializable attribute on a class. Not all classes can be serialized without additional effort. Generally best to stick to non-generic types to ensure reliability.

For this example I use a single state class. I also create a # define that allows me to either save or load data accordingly. The code will make sense, hopefully.

The code

I define a simple address class. Note, the class is a simple data class and contains no methods. I do include an overridden ToString method, useful when debugging.

[Serializable]
public class Address
{
  public int Number;
  public string Street;
  public string PostCode;

  public override string ToString()
  {
    return $"{Number}, {Street}. {PostCode}";
  }
}

Top

Next I declare a customer class. Customers are everywhere so the example seems apt.

[Serializable]
public class Customer
{
  public string FirstName;
  public string LastName;
  public Address Address = new Address();

  public override string ToString()
  {
    return $"{LastName}, {FirstName}, {Address}";
  }
}

Top

Next I define the state class. This would contain state required to run your software.

[Serializable]
public class State
{
  public Customer[] Customers;

  public State()
  {
    this.Customers = new Customer[0]; 
  }

  public State XmlSave()
  {
    XmlSerializer s = new XmlSerializer(typeof(State));
    using (var sw = new StreamWriter("state.xml"))
    {
      s.Serialize(sw, this);
      return this;
    }
  }

  public static State XmlLoad()
  {
    XmlSerializer xs = new XmlSerializer(typeof(State));
    using (var sr = new StreamReader("state.xml"))
    {
      return (State)xs.Deserialize(sr);
    }
  }
}

Top

The State class, as you might expect, maintains all state required for my program to run. The XmlSave function is a member function, so that one can call state.XmlSave(). That is, you need a state instance to save.

The XmlLoad function is static. This is simply because when your program runs you will not have a state instance. So, you will typically want to load and create a state instance. Bootstrap/startup code is simplified as one can simply code "var state = State.XmlLoad();"

Top

The main code, for testing follows...

class Program
{
  static State CreateData()
  {
    return new State()
    {
      Customers= new Customer[]
      {
        new Customer() {
          FirstName = "Karl",
          LastName = "Page",
          Address = new Address() {
            Number=80,
            PostCode="KKJ486",
            Street="Arcacia Avenue" }
        },

        new Customer() {
          FirstName = "Cecilia",
          LastName = "Page",
          Address = new Address() {
            Number=80,
            PostCode="KKJ486",
            Street="Arcacia Avenue" }
        },

        new Customer() {
          FirstName = "Fred",
          LastName = "Bloggs",
          Address = new Address() {
            Number=40,
            PostCode="LLJ86K",
            Street="Nowhere Avenue" }
        }
      }
    };
  }

  static void Main(string[] args)
  {
#if SAVE
    State s = CreateData();
    s.XmlSave();
#else
    State s = State.Load();
#endif
  }
}

Top

Testing

To test run the program as is using F5 to debug.

A file, state.xml should appear under the debug folder. You may have to dig, depends upon project settings.

Make sure you add the line #define SAVE before any usings, at the top of the progam file. This should write state to the the state.XML file. Remove #define SAVE to invoke loading.

Top

My XML appears as follows...

<?xml version="1.0" encoding="UTF-8"?>
<State xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Customers>
 <Customer>
  <FirstName>Karl</FirstName>
  <LastName>Page</LastName>
  <Address>
   <Number>80</Number>
   <Street>Arcacia Avenue</Street>
   <PostCode>KKJ486</PostCode>
  </Address>
 </Customer>

 <Customer>
  <FirstName>Cecilia</FirstName>
  <LastName>Page</LastName>
  <Address>
   <Number>80</Number>
   <Street>Arcacia Avenue</Street>
    lt;PostCode>KKJ486</PostCode>
  </Address>
</Customer>

 lt;Customer>
 <FirstName>Fred</FirstName>
 <LastName>Bloggs</LastName>
 <Address>
  lt;Number>40</Number>
  <Street>Nowhere Avenue</Street>
  <PostCode>LLJ86K</PostCode>
  </Address>
 </Customer>

</Customers>
</State>