Showing posts with label Software Engineering. Show all posts
Showing posts with label Software Engineering. Show all posts

Saturday, 4 February 2023

Commands

A command encapsulates a unit of work. Typically, a command might insert/update a database record, send an email or just update a model. There are numerous ways one can model a command. In this post I describe a simple command mechanism that is suitable for standalone applications. I will also discuss the pros and cons of the approach.


Top

In this post...

Top

Creating The Project

The project is a Console Application, using the .NET Framework (Ver 4.8). The project is named, CommandApp.

No external libraries are required for this project.

Top

Extension/Helper methods

To help visualise program flow I add an extension class.

using System;
namespace CommandApp
{
  public static class Extensions
  {
    //The Trace extension allows one to simply output a value to the console.
    //The method is easily modified to send output elsewhere..
    public static void Trace<T>(this T src) =>
      Console.WriteLine(src);
  }
}
Top

Simple Commands

The simple command encapsulates command values. The command is also responsible for its own behaviour, in this case, executing command code. The command is executed synchronously, that is, on the thread that created the command. The following diagram illustrates the concept.

The client is simply code that creates and executes the command. The code for the simple command is as follows...

Top
using System;

namespace CommandApp.SimpleCommands
{
  public abstract class Command
  {
    protected Command() { }

    public bool IsSuccess => null == Exception;

    public Exception Exception { get; private set; }

    public void Execute()
    {
      try
      {
        OnExecute();
      }
      catch (Exception e)
      {
        this.Exception = e;
      }
    }

    public abstract void OnExecute();
  }
}
Top

Code Walkthrough

  • The class is declared as abstract, concrete classes override/implement onExecute to implement the command.
  • Client(s) create a new command instance then call the Execute method which catches any exceptions and places into to the Exception property.
  • Client(s) can query the command to check for success or failure an act appropriately.

Simple command Pros and Cons

Pros

  • Simple to use.
  • No infrastructure required.
  • Command code, that is values and behaviour all in a single class.
  • Works when using the Command base class that refers to a concraete instance. Can thank Polymorphism for this. For example, Command myCmd = new MyCommand()
Top

Cons

  • Command is blocking - main thread blocks whilst waiting for command execution.
  • Including cross-cutting concerns is difficult unless such concerns are included in the Execute parameters. However, this will complicate calling the Execute method.
  • Client(s) can create and execute a given command. Not necessarily bad, but it might be best to separate command creation from command execution. This would allow the software to "inject" code before and after executing code.
Top

Sample Simple Command

The following code illustrates using the simple class. The example Sleeps for 1000ms to simulate a time consuming command.

using System.Threading;

namespace CommandApp.SimpleCommands
{
  public class AddCustomerCommand : Command
  {
    private readonly string _firstName;
    private readonly string _lastName;

    public int Id { get; private set; }

    public AddCustomerCommand(string firstName, string lastName)
    {
      _firstName = firstName;
      _lastName = lastName;
    }

    public override void OnExecute()
    {
      "---------------------------------------------------".Trace();
      $"AddCustomerCommand({_firstName}, {_lastName})".Trace();
      "About to sleep for 1000ms to simulate a time consuming command".Trace();
      Thread.Sleep(1000);
      "Setting Id to 1 to simulate adding a new database record and returning a new Id".Trace();
      Id = 1;
      "completed".Trace();
    }
  }
}
Top

Testing The simple Command

The following code was used to test the simple command...

namespace CommandApp.SimpleCommands
{
  public static class Test
  {
    public static void Run()
    {
      AddCustomerCommand cmd = new AddCustomerCommand("Fred", "Bloggs");
      cmd.Execute();
    }
  }
}

If all goes well, you should see the following text on the command prompt...

Top

Sunday, 15 January 2023

Working-With-Byte-Arrays

In this post I will show how to read/write different data type values to and from a byte buffer. This should prove useful when working with block data or pages. In this scenario, pages describes database pages or similar block data mechanisms.


Top

In this post...

Top

How To Handle Typed Values

The aim is to read/write different typed values from a byte array. In C++ one can achieve this by using a union. In C# we achieve this by using a struct/class using explicit layout. Explicit layout is the key here as it ensures types are aligned correctly.

The basic idea is to have a number of byte values offset from zero to maximum bytes required. Then add basic types, such as int 16, int 32, int 64, float, etc all at offset zero. Then one can set a float value and read of the four byte values. The same can be done for most basic types, set the value and the read the corresponding byte values. To read typed values, simply populate the byte values accordingly then read the basic type info (int 16, int 32, int float, etc).

For those with a COM background this approach is similar to the VARIANT type.

Top

The Union Value Type


using System;
using System.Runtime.InteropServices;
namespace Blog
{
  [StructLayout(LayoutKind.Explicit)]
  public struct Value
  {
    [FieldOffset(0)] public byte B0;
    [FieldOffset(1)] public byte B1;
    [FieldOffset(2)] public byte B2;
    [FieldOffset(3)] public byte B3;
    [FieldOffset(4)] public byte B4;
    [FieldOffset(5)] public byte B5;
    [FieldOffset(6)] public byte B6;
    [FieldOffset(7)] public byte B7;

    [FieldOffset(0)] public float Float;
    [FieldOffset(0)] public double Double;
    [FieldOffset(0)] public Int16 Int16;
    [FieldOffset(0)] public Int32 Int32;
    [FieldOffset(0)] public Int64 Int64;
  }
}

The above struct declaration shows eight bytes, B0-B7 for data transfer. If using the decimal type this will need to be expanded to include 16 bytes! As can be seen, B0-B7 is byte aligned and offset accordingly.

Note also, that basic types float, double, Int16, etc all start at offset zero. So, if one sets the Float field, the corresponding 4 bytes can be read from B0-B3. To reverse the action, set fields B0-B3, then read the float value.

It is imperative that one knows how many bytes are required for a typed value. The above structure should cover most data types bar decimal. DateTime values can also be stored by first converting to long and using the Int64 field or bytes B0-B7 to recreate.

Top

Testing The Union Value Type

The above can be tested using the following code...

static void RawTest()
{
  Value value = new Value();
  byte[] data = new byte[128];

  int cursor = 0;
  
  // *********************************************************************************************
  // Writing values to the data byte array.
  // *********************************************************************************************
  Int16 v1 = Int16.MaxValue - 1067;
  Int32 v2 = Int32.MaxValue - 10067;
  float v3 = float.MaxValue - 1.07896f;

  // Write an Int16 value to the data byte array.
  // First set the Value's Int16 field, then read two Value bytes.
  value.Int16 = v1;
  data[cursor++] = value.B0;
  data[cursor++] = value.B1;

  // Write an Int32 value to the data byte array.
  // First set the Value's Int32 field, then read four Value bytes.
  value.Int32 = v2;
  data[cursor++] = value.B0;
  data[cursor++] = value.B1;
  data[cursor++] = value.B2;
  data[cursor++] = value.B3;

  // Write a float value to the data byte array.
  // First set the Value's Float field, then read four Value bytes.
  value.Float = v3;
  data[cursor++] = value.B0;
  data[cursor++] = value.B1;
  data[cursor++] = value.B2;
  data[cursor++] = value.B3;

  
  // *********************************************************************************************
  // Reading values from the data byte array.
  // *********************************************************************************************
  cursor = 0;
  
  // Read an Int16 from the data byte array.
  // Set Value's first two byte fields, then read the Value's Int16 field.
  value.B0 = data[cursor++];
  value.B1 = data[cursor++];
  var v1Result = value.Int16;

  // Read an Int32 from the data byte array.
  // Set Value's first four byte fields, then read the Value's Int32 field.
  value.B0 = data[cursor++];
  value.B1 = data[cursor++];
  value.B2 = data[cursor++];
  value.B3 = data[cursor++];
  var v2Result = value.Int32;

  // Read a float from the data byte array.
  // Set Value's first four byte fields, then read the Value's Float field.
  value.B0 = data[cursor++];
  value.B1 = data[cursor++];
  value.B2 = data[cursor++];
  value.B3 = data[cursor++];
  var v3Result = value.Float;

  Console.WriteLine(v1);
  Console.WriteLine(v1Result);
  Console.WriteLine(v2);
  Console.WriteLine(v2Result);
  Console.WriteLine(v3);
  Console.WriteLine(v3Result);
}
Top

Writing Values To A Byte Array

Figure 2 illustrates a byte array filled with values from the previous test code. As the diagram illustrates, data values are byte aligned accordingly. That is, a 16 bit-integer requires 2 bytes, a float or 32-bit integer requires 4 bytes. This approach works well for data blocks comprising of byte arrays. Simply create a byte array, write values and save to disk. Conversely, load a byte array block from disk into memory, then proceed to read actual values.

Top

Improving The Value Interface

The current method of reading/writing values is somewhat verbose. One needs to track the byte array offset (cursor) and the offset to add following a read or write. In addition, explicitly reading and writing to the Value's byte fields (B0...BN) is tiresome and error-prone. The following should help alleviate these problems...

  • Specify the byte array and initial offset.
  • Specify a cursor that is relative to the specified initial offset.
  • Update the cursor accordingly following a read or write operation.
  • Allow the cursor's position to be set manually, the cursor will always be relative to the specified offset.

The following class, ByteBuffer, implements the above features.

using System;

namespace Blog
{
  /// <summary>
  /// Allows values to read/written to/from a byte array.
  /// Specify a byte buffer and initial offset in the constructor.
  /// Data will be read/written at this offset.
  /// The class uses a cursor to indicate current read/write position.
  /// The cursor is always offset by the offset specified in the constructor.
  /// </summary>
  public class ByteBuffer
  {
    private Value _value = new Value();
    private int _internalCursor;
    private int _offset;
    private readonly byte[] _buffer;

    public int Cursor => _internalCursor - _offset;

    public ByteBuffer(byte[] buffer, int offset)
    {
      _buffer = buffer;
      _offset = offset;
      _internalCursor = offset;
    }

    public ByteBuffer SetCursor(int position)
    {
      _internalCursor = _offset + position;
      return this;
    }

    public ByteBuffer Int16(Int16 value)
    {
      _value.Int16 = value;
      _buffer[_internalCursor++] = _value.B0;
      _buffer[_internalCursor++] = _value.B1;
      return this;
    }

    public ByteBuffer Int16(out Int16 result)
    {
      _value.B0 = _buffer[_internalCursor++];
      _value.B1 = _buffer[_internalCursor++];
      result = _value.Int16;
      return this;
    }

    public ByteBuffer Int32(Int32 value)
    {
      _value.Int32 = value;
      _buffer[_internalCursor++] = _value.B0;
      _buffer[_internalCursor++] = _value.B1;
      _buffer[_internalCursor++] = _value.B2;
      _buffer[_internalCursor++] = _value.B3;
      return this;
    }

    public ByteBuffer Int32(out Int32 result)
    {
      _value.B0 = _buffer[_internalCursor++];
      _value.B1 = _buffer[_internalCursor++];
      _value.B2 = _buffer[_internalCursor++];
      _value.B3 = _buffer[_internalCursor++];

      result = _value.Int32;
      return this;
    }

    public ByteBuffer Float(float value)
    {
      _value.Float = value;
      _buffer[_internalCursor++] = _value.B0;
      _buffer[_internalCursor++] = _value.B1;
      _buffer[_internalCursor++] = _value.B2;
      _buffer[_internalCursor++] = _value.B3;
      return this;
    }

    public ByteBuffer Float(out float result)
    {
      _value.B0 = _buffer[_internalCursor++];
      _value.B1 = _buffer[_internalCursor++];
      _value.B2 = _buffer[_internalCursor++];
      _value.B3 = _buffer[_internalCursor++];
      result = _value.Float;
      return this;
    }
  }
}
Top

Using The ByteBuffer Class

Using the ByteBuffer class is fairly straightforward. Simply call the constructor with a byte array and offset. No data can be read or written before the offset.

A sample test/driver program now follows...

static void Main(string[] args)
{
  byte[] data = new byte[128];
  ByteBuffer buffer = new ByteBuffer(data, 5);

  Int16 v1 = Int16.MaxValue - 1067;
  Int32 v2 = Int32.MaxValue - 10067;
  float v3 = float.MaxValue - 1.07896f;

  int bytesWritten = buffer
    .Int16(v1)
    .Int32(v2)
    .Float(v3)
    .Cursor;
  Console.WriteLine($"{bytesWritten} bytes written to buffer.");

  int bytesRead = buffer
    .SetCursor(0)
    .Int16(out var v1Read)
    .Int32(out var v2Read)
    .Float(out var v3Read)
    .Cursor;

  Console.WriteLine($"{bytesRead} bytes read from buffer.");
  Console.WriteLine($"v1Write:{v1} - v1Read:{v1Read}");
  Console.WriteLine($"v2Write:{v2} - v2Read:{v2Read}");
  Console.WriteLine($"v3Write:{v3} - v3Read:{v3Read}");
}

The above code wites the following to the console.

Notice how the test program uses an offset of 5 when constructing the ByteBuffer. The correct bytes read/wrriten of ten is still returned. The first five bytes in this example will be zeroed.

Top

Saturday, 26 March 2022

Pixel Graphics Winforms

In an older post I illustrated how one can achieve decent pixel-based graphics. Following further research, I built upon the original idea. In this post, I achieve render times of about 11 milliseconds. For 60 FPS one must render each frame in about 16.7 milliseconds.

Top

Background

In the early days of Windows, an application could fully control the graphics display. Early games took advantage of this and essentially bypassed the operating system. Modern Windows due to its multitasking nature, no longer allows an application to "own" any resources outright. Resources are shared, including graphical data. Personally, I think the operating system should have a "Game Mode" setting, where only critical processes, threads, etc are run. I mean, does anyone want to run multiple apps whilst running a game?

Modern games use the graphics card(GPU) to output graphics. Data is sent to the GPU, leaving the CPU to perform CPU-bounds tasks (updating player position, score and so on). This is great if your app has access to DirectX, Direct 3D, OpenGL. However, if your app is unable to access such technology, and you are bound by simple Winforms technology...Read on!

Top

Winforms Graphics Basics

Each Window is assigned a graphics context. This usually arises in a Window's OnPaint event. Paint messages are classed as low priority messages, and as such are posted rather than sent. A sent message yields an immediate return, posted messages will yield a result in the future. The above is based upon ealier versions of Windows, though I doubt much has changed. Regardless, the OnPaint and supplied Graphics context is what is of interest here.

Top

Memory

C# as a rule manages memory. One can allocate memory (not directly) by creating new classes, structures etc. The memory required for such items will be released when the class/struct instance is no longer required. The CLR runtime can also relocate memory as and when required. When dealing with bitmaps, the latter is not ideal. Relocating memory for large bitmaps is expensive in terms of CPU time.

It is possible to allocate memory outside of the CLR. Doing so ensures that the CLR will never try to reclaim the memory. Put another way, you will be responsible for memory management. Almost in C++ land!

Manual memory management is a must for fast pixel graphics.

Top

Image Processing Pipeline

The basic pipeline is as follows...

  • Create an array to hold image information
  • Create a bitmap that uses memory from 1st step
  • Update array to reflect image information (Bitmap points to array memory so is automatically updated)
  • Blit bitmap to main form or relevant window
  • Repeat until app ends
Top

Multithreading/async programming

>My own research proved that adding threads or using asynchronous programming did not yield better results.

A Windows message loop generally doesn't play well with other threads. Typically, additional threads, at some point must communicate results with the main UI thread. This requires thread synchronisation techniques, which tend to be quite expensive. Performing most operations on the same thread seems to give best results.

That said, updating non UI elements may be worthwhile performing on separate thread(s). For example, whilst the UI thread is blitting current content, another thread might update values for the next scene.

Top

Achieving Optimal Paint Results

As mentioned, Windows posts a WM_PAINT message to a window, when an application window needs re-painting. One can invoke the WM_PAINT, and hence cause window repainting by issuing an Invalidate command. This may be followed by an Update command to perform immediate repainting.

I found that the best way to do this is to override the Application.OnIdle event. In the handler, one can just call invalidate. This will post the WM_PAINT message to the window. Once painting has occurred, the idle event will be raised again. This essentially causes a forever painting loop whilst still being responsive to other events.

The Code

The project is a .NET Windows Application targeting .NET Framework 4.8. No 3rd party libraries are used.


  using System;
  using System.Drawing;
  using System.Drawing.Imaging;
  using System.Runtime.InteropServices;
  
  namespace FastGraphics
  {
    /// <summary>
    /// Uses pinned memory and direct array access to manipulate bitmap bits.
    /// Pinned memory is not owned by the CLR, so, it is important to free when done.
    /// </summary>
    public sealed class FastBitmap : IDisposable
    {
      private GCHandle _pinnedMemoryHandle;
      private IntPtr _pinnedMemoryAddress;    
  
      public int Width { get; private set; }
      public int Height { get; private set; }
      public uint[] Pixels { get; private set; }
      public Bitmap Image { get; private set; }
  
      public FastBitmap(int width, int height)
      {
        // Pixels is an array that may be used to modify pixel data.
        // Typical formula is ((y*width)+x) = Alpha | Colour;
        // Uses 4 bytes per pixel (Alpha, Red, Green and Blue channels, each 8 bits)
        Pixels = new uint[width * height];
  
        // Inform OS that pixel allocated array is pinned and not subject to normal garbage collection.
        _pinnedMemoryHandle = GCHandle.Alloc(Pixels, GCHandleType.Pinned);
  
        // Obtain the pinned memory address, required to create bitmap for use by .NET app.
        _pinnedMemoryAddress = _pinnedMemoryHandle.AddrOfPinnedObject();
  
        // Set pixel format and calculate scan line stride.
        PixelFormat format = PixelFormat.Format32bppArgb;
        int bitsPerPixel = ((int)format & 65280) >> 8;
        int bytesPerPixel = (bitsPerPixel + 7) / 8;
        int stride = 4 * ((width * bytesPerPixel + 3) / 4);
        Image = new Bitmap(width, height, stride, PixelFormat.Format32bppArgb, _pinnedMemoryAddress);
        Width = width;
        Height = height;      
      }
  
      public void Draw(Graphics target) =>
        target.DrawImage(Image, 0, 0);
  
      public void Dispose()
      {
        Image.Dispose();
  
        if (_pinnedMemoryHandle.IsAllocated)
          _pinnedMemoryHandle.Free();
  
        _pinnedMemoryAddress = IntPtr.Zero;
        Pixels = Array.Empty<uint>();
        Width = 0;
        Height = 0;
      }
    }
  }
  

    using System;
    using System.Windows.Forms;
    
    namespace FastGraphics
    {
      internal static class Program
      {
        [STAThread]
        static void Main()
        {
          Application.EnableVisualStyles();
          Application.SetCompatibleTextRenderingDefault(false);
    
          try
          {
            Form1 form = new Form1();
            Application.Idle += (s, e) => form.OnIdle();
            Application.Run(form);
          }
          catch (Exception ex)
          {
            MessageBox.Show(ex.Message);
          }
        }
      }
    }

      using System;
      using System.Drawing.Drawing2D;
      using System.Windows.Forms;
      
      namespace FastGraphics
      {
        public partial class Form1 : Form
        {
          private FastBitmap _fastBitmap;
          private float _offset = 0;
      
          private uint _renderTimeInMilliseconds;
      
          public Form1()
          {
            InitializeComponent();
          }
      
          /// <summary>
          /// Update on-screen text that displays render time in miliiseconds.
          /// </summary>
          public void UpdateRenderTimeText()
          {
            label1.Text = _renderTimeInMilliseconds.ToString();
          }
      
          /// <summary>
          /// Dispose of existing fast bitmap (if exists)
          /// Create new fast bitmap based upon client rectangle.
          /// </summary>
          private void CreateFastBitmap()
          {
            _fastBitmap?.Dispose();
            _fastBitmap = new FastBitmap(ClientRectangle.Width, ClientRectangle.Height);
          }
      
          public void OnIdle()
          {
            Invalidate();
            UpdateRenderTimeText();
          }
      
          // Resizing form changes drawing area.
          // Resize fast bitmap accordingly.
          protected override void OnResize(EventArgs e)
          {
            CreateFastBitmap();
          }
      
          /// <summary>
          /// Override OnPaintBackground to do nothing.
          /// The OnPaint will perform all drawing.
          /// </summary>
          /// <param name="e"></param>
          protected override void OnPaintBackground(PaintEventArgs e)
          {
          }
      
          /// <summary>
          /// Magic happens in OnPaint.
          /// 1. Render raw pixel data (Fast bitmap)
          /// 2. Due to the nature of fast bitmap, its image will reflect pixel data.
          /// 3. Draw fast bitmap image to this form's window.
          /// 4. Time all of this so we can determine time taken to render a frame.
          /// Optional - set interpolation mode.
          /// </summary>
          /// <param name="e"></param>
          protected override void OnPaint(PaintEventArgs e)
          {
            DateTime startTime = DateTime.UtcNow;
            e.Graphics.InterpolationMode = InterpolationMode.NearestNeighbor;
            RenderRawPixels();
            _fastBitmap.Draw(e.Graphics);
            this.BackgroundImage = _fastBitmap.Image;
            DateTime endTime = DateTime.UtcNow;
            _renderTimeInMilliseconds = (uint)(endTime - startTime).Milliseconds;
          }
      
          /// <summary>
          /// RenderRawPixels is called for each onPaint.    
          /// </summary>
          private void RenderRawPixels()
          {
            int rowAddress = 0;
            int width = _fastBitmap.Width;
            int height = _fastBitmap.Height;
            uint[] pixels = _fastBitmap.Pixels;
      
            for (int y = 0; y < height; y++)
            {
              float yF = (float)y;
              for (int x = 0; x < width; x++)
              {
                float xF = (float)x;
      
                float f =
                  ((xF * xF * xF * yF) / 16384) +
                  ((xF * yF * yF * yF) / 16384) +
                  ((xF * yF * _offset) / 16384);
                f *= _offset;
                f /= 65536;
                f /= 8192;
      
                uint alpha = 4278190080;
                // Alpha component tends to double rendering time
                // Setting to 0xFF000000 will ensure the above renders solid colours.
                // Uncomment the code below to introduce alpha component.
                //uint alpha = (uint)(
                //  (x * y) -
                //  (y * _offset)) /
                //  1024;
                //alpha <<= 24;
                //alpha &= 0xff000000;
      
      
                pixels[rowAddress + x] = alpha | (uint)(f * 16777215);
              }
              rowAddress += width;
            }
            BackgroundImage = _fastBitmap.Image;
            _offset += 0.005;
          }
      
          /// <summary>
          /// upon closing this form, we must dispose the fast bitmap.
          /// </summary>
          /// <param name="e"></param>
          protected override void OnClosed(EventArgs e)
          {
            _fastBitmap.Dispose();
            base.OnClosed(e);
          }
        }
      }

        namespace FastGraphics
        {
          partial class Form1
          {
            /// <summary>
            /// Required designer variable.
            /// </summary>
            private System.ComponentModel.IContainer components = null;
        
            /// <summary>
            /// Clean up any resources being used.
            /// </summary>
            /// <param name="disposing">true if managed resources should be disposed; otherwise, false.</param>
            protected override void Dispose(bool disposing)
            {
              if (disposing && (components != null))
              {
                components.Dispose();
              }
              base.Dispose(disposing);
            }
        
            #region Windows Form Designer generated code
            /// <summary>
            /// Required method for Designer support - do not modify
            /// the contents of this method with the code editor.
            /// </summary>
            private void InitializeComponent()
            {
              this.label1 = new System.Windows.Forms.Label();
              this.SuspendLayout();
              // 
              // label1
              // 
              this.label1.BackColor = System.Drawing.Color.Black;
              this.label1.ForeColor = System.Drawing.Color.White;
              this.label1.Location = new System.Drawing.Point(-2, -1);
              this.label1.Name = "label1";
              this.label1.Size = new System.Drawing.Size(43, 18);
              this.label1.TabIndex = 0;
              this.label1.Text = "Time";
              this.label1.TextAlign = System.Drawing.ContentAlignment.MiddleCenter;
              // 
              // Form1
              // 
              this.AutoScaleDimensions = new System.Drawing.SizeF(6, 13);
              this.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font;
              this.ClientSize = new System.Drawing.Size(800, 450);
              this.Controls.Add(this.label1);
              this.DoubleBuffered = true;
              this.Name = "Form1";
              this.Text = "Form1";
              this.WindowState = System.Windows.Forms.FormWindowState.Maximized;
              this.ResumeLayout(false);
        
            }
        
            #endregion
            public System.Windows.Forms.Label label1;
          }
        

Friday, 21 May 2021

Tips To Improve Software Development


1. Write A Problem Statement

Software is written to solve problems. If you don't fully understand the problem, you are unlikely to solve it correctly. Also, as time progresses you may forget the original problem statement or deviate from solving the original problem. Write (or type) the problem statement and keep it handy. I usually write an abstract text file and store it within the project. This ensures I can visit it frequently to keep the problem statement fresh in my mind.

Top

2. Break Problem Into Smaller Manageable Tasks

This is almost like a to do list. For example, assume you wish to create an app that populates a database with invoice details and posts to a financial system. One way to break this down might be as follows.

  1. Obtain invoice source data
  2. Validate source data
  3. Select/filter relevant data
  4. Populate invoice database
  5. Post invoice details to financial system

Notice that each task is brief, but completing each task will result in a solution. Before attempting a task, break into even smaller tasks until you reach a point where you feel ready to start writing some code. Using the above example, I might break the last task into the following smaller tasks.

  1. Initiate financial application session
  2. Select pending invoices from database
  3. For each pending invoice perform the following
    1. Log invoice details
    2. Validation (e.g. check supplier is valid, that net, VAT and gross tally and so on)
    3. Attempt to post to financial system
    4. Update database to reflect invoice posted or failed to post
    5. Log invoice post details
  4. Tidy up - close any database connections, logoff financial system

Using this approach you can enter tasks and sub tasks into a to do list or spreadsheet and mark as completed as you progress.

Top

3. Limit Use of Third-Party Libraries

Introducing additional dependencies into software also introduces additional failure points. The additional dependency may contain unknown bugs which will ultimately become your bugs.

Typically, when using third-party libraries I will code an API wrapper. I then use my wrapper rather than calling third-party code directly. This has a number of benefits.

  • If I spot a potential bug I can output trace information or place breakpoints for debugging within my wrapper function.
  • I can additional code within my wrapper function to fix bugs.
  • I can wrap multiple third-party library function calls into a single wrapper function.
Top

4. Use Immutable Data

One of the biggest problems with software is complexity. As expected, complexity increases as the software grows in size. State, particularly managing state is probably the biggest contributing factor to complexity.

There are two ways of managing state.

  • Mutable State

    Used in classic object-oriented programming, data is modified in-place. That is the original data is modified.

  • Immutable State

    Used in most (all?) functional programming languages. New data is created by transforming the original data.

As an example, consider a simple Person class that contains first and last names. The following C# snippet illustrates mutable data.

class Person
{
  public string FirstName { get; set; }
  public string LastName { get; set; }
  
  public Person(string firstName, string lastName)
  {
    this.FirstName = firstName;
    this.LastName = lastName;
  }
}

static class Test
{
  public static void Run()
  {
    // Create a new person Fred Bloggs
    Person p1 = new Person("Fred", "Bloggs");
    
    p1.LastName = "Smith";
    // Fred Bloggs has changed to Fred Smith, the original data is lost
  }
}

The following C# snippet illustrates immutable data.

class Person
{
  // Note, set has been removed
  public string FirstName { get; }
  public string LastName { get; }
  
  public Person(string firstName, string lastName)
  {
    this.FirstName = firstName;
    this.LastName = lastName;
  }
}

static class Test
{
  public static void Run()
  {
    // Create a new person Fred Bloggs
    Person p1 = new Person("Fred", "Bloggs");
    
    Person p1New = new Person(p1.FirstName, "Smith");
    // Fred Bloggs remains unchanged, p1New contains new name, "Fred Smith"
  }
}

Using immutable data has the following benefits

  • Thread safety

    Multiple threads can read data without using synchronisation. Immutable data is read only so data integrity is assured. If data must be modified (i.e. mutable) then thread synchronisation is required to ensure data integrity.

  • Good key elements

    Certain data structures (dictionaries, hash sets, most tree structures, and so on) require keys to access values. Once inserted, the key should never change. Changing the original key will result in the original value being unobtainable.

  • Easier to reason about code

    Functions tend to be free from side-effects as input parameters cannot be modified. Class invariant is established once (during construction) and remains unchanged throughout the duration of the program.

  • Easier to code

    Immutable data is contructed once and cannot change during the lifetime of the program. Functions (and associated logic, validation, etc) to modify existing data are not required.

  • Improved caching

    Caching is used when data creation is relatively expensive. Examples include reading configuration data, reading from a database and so on. In these scenarios you would expect the same data each time you ask the cache for a particular item.

Top

5. Avoid Primitive Obsession

Primitive obsession is where language primitives are used to describe a data type that is more complex than the primitive describing it. Most data types requires constraints. Examples include...

  1. A person's age is typically between 0 and 100. (Could use upper value of say 150 to be sure)
  2. An email address, as a minimum contains a @ character and a domain suffix (.COM, .NET, etc)
  3. A Bank Account Number only contains digits
  4. A unique database table row identifier is typically an integer (e.g. customer id)
  5. An IPv4 address contains four, three digit integers.

Referring to the above examples. Using primitives,
1,3 and 4 might use integers, the remainder would likely use strings.

Consider a customer data type that may be persisted to a database consists of a unique id, first and last name and an email address. Using primitive data types we might code this as follows...


class Customer
{
  public int Id { get; }
  public string FirstName { get; }
  public string LastName { get; }
  public string Email { get; }
  
  public Customer(int id, string firstName, string lastName, string email)
  {
    this.Id = id;
    this.FirstName = firstName;
    this.LastName = lastName;
    this.Email = email;
  }
}

There are several problems with this code.

  • It is possible to inadvertently mix email with first or last name as they are all strings.
  • There is no size constraint on the name fields whereas there will be on the database side.
  • The email address could be supplied an invalid value.
  • Any of the string fields could be supplied with null or zero-length values.
  • The id field is a signed integer whereas unique identifiers tend to be positive integers.
  • There is no distinction between a customer id and other unique identifiers that we will likely require.
string email = "fredbloggs@hotmail.com"; string email2 = "test.exe"

The first email address might well be valid. The second is most definitely not a valid email address. However, using a string to represent an email address does not tell us if the email address is well-formed. The email data type is more complex than the string describing it.

A better way to model an email address might be as follows.

public class EmailAddress
{
  public string Value { get; }
  
  public static EmailAddress Parse(string value)
  {
    string input = string.IsNullOrEmpty(value) ? "" : value.ToLower();
    // validate value, throw exception if supplied value is not a valid email address
    return new EmailAddress(value);
  }
  
  private EmailAddress(string value)
  {
    this.Value = value;
  }
}

The above code implements a simple email address type. In reality more would be needed, such as testing for equality, hash code generation and so on. Also, I use a static function (Parse, in this example) to validate input before creating the type. The constructor is made private to ensure that the email type can only be constructed using the static creation functions (parse in this case). The following code is what I might typically use. This example implements the age type

public class Age
{
  // Exception specific to the Age class. Aids in debugging and logging.
  public class InvalidValueException : Exception
  {
    public InvalidValueException(string message) : base(message) { }
  }

  // Value will always contain a valid age due to smart construction and private constructor
  public byte Value { get; }

  // Smart construction validates age range.
  // Returns a valid age or throws an Age.InvalidValueException
  public static Age Create(int value)
  {
    if (value < 0 || value > 120)
      throw new InvalidValueException($"Age should be a value between 0 and 120");
    return new Age((byte)value);
  }

  // Smart construction validates parses age from a string.
  // Returns either a valid age or throws an Age.InvalidValueException
  public static Age Parse(string value)
  {
    int result = 0;

    if (!int.TryParse(value, out result))
      throw new InvalidValueException($"Unable to convert string, {value} to age");
    return Create(result);
  }

  private Age(byte value)
  {
    this.Value = value;
  }
}  
Top

6. Never Use Global Variables

Did I say never? Generally (read almost always) speaking, global variables are bad, really bad, like, don't ever use them. The reason is simple. Unprotected global variables can be modified, anywhere, within the software. This makes tracking changes impossible. Of course data has to start somewhere, which, generally means that you might have one global variable. This single global variable typically contains the global state as used by the app.

A better idea is to have a bootstrap function that creates initial state and pass that on to the main app. If this is not possible then you should wrap the initial state and supply get and set accessors. This will at least ensure that you can add trace, breakpoints to anything that modifies the global state.

If you follow one of the strongest rules, always ensure functions receive dependencies as parameters, using global variables isn't such a big deal. What is a big deal, and what will come back to bite you as a huge problem is using global variables without passing as parameter(s).

Consider a simple example (in C#).

class Globals
{
  public static int ThisIsGlobal { get; set; } 
}

class Test
{
  public static void Func1()
  {
    Globals.ThisIsGlobal = 1;
  }
  
  public static void Func2()
  {
    Globals.ThisIsGlobal = 2;
  }
}

A somewhat contrived example, I agree. That said, I hope it proves a point. Throughout the duration of the software, Func1 or Func2 may be called. Both are bad functions in my opinion as firstly, they do not accept Globals as an input parameter. Secondly, both functions produce side-effects. The both modify the global state.

I probably need to, and will write more about this. I will say this, if you have global variables, have only one (it can be a structure a class, etc). Wrap the global variable so that potentially, if needs be you can log acess/modifications to said global variables. Better still, have a bootstrap file that creates initial state conditions and that state is passed to the main controller. The main controller will be a UI element in a UI-driven app, or an entry function in a non-UI app.

Top

7. Release First Draft ASAP

Releasing the first draft allows all stakeholders to determine if the software is on the right track. If it is great, move on to the next feature. If not, this is a great chance to redefine the original problem statement.

In the early days of software development, it was commonplace to write pages and pages of text that described the overall software purpose. Not only does this not work, my own experience shows that this approach also doesn't work. The reason why this doesn't work is that software requirements are not only complex but they also evolve as both stakeholders and engineers realise the system.

The initial release, prototype, if you will, is great for all involved to give an opinion. Again, in my experience, this is where the real requirements come about. So many times I have been given a set of requirements, delivered a basic solution, only to find that the final deliverable is not what the customer wanted. What a customer thinks they want and want they actually want, in my experience, are completely different.

I guess this is why we call this software and not hardware. Hardware is set in stone, cannot be changed. Software on the other hand...

Top

8. Release Often

Software design used to involve many drawing, static drawing, runtime analysis diagrams and so on. Basically design as much up front as possible, then, go away and code. The problem, gathering the requirements could take months at best. Knowing a little about human nature and a lot about customers. Customers never really know what they want, they have ideas as to what they want. This is not a bad thing it just means that pages and pages of requirements will never work.

So what is the answer. Regular deliverables. This allows stakeholders/customers to see the software in situ and see the software grow. Potential problems can be weeded out. You can never get this level of clarity in a written document. Sometimes, people just need to see!

Releasing software on a regular basis is also a good idea. You/team might spend a week developing a new feature (the outline only I hope). You then present this to the stakeholders. If they like, great, you can some question, get some depth, go away an implement a fill solution. If they hate it, you may have wasted 5 days development time. Not a big deal in the scheme of things. Even a failure is likely to bring out features. Sometimes when people see what they don't like, they have like a eureka moment and realise what they would like.

Top

9. Always Supply Function Dependencies

For maintainable software this a huge rule for me. Always supplying dependencies to function achieves the following

  • The function will (should) produce the same output given the same inputs
  • A (pure) function can only produce an output based upon its input

As an example, assume a function, SaveCustomer, that simply takes a customer object. What does save a customer really mean. Save to a database, save to a CSV file, XML file and so on.

In software some concepts are truly open ended. The concept of saving customer details is an example. Let's add code to see why.


class Customer
{
  public CustomerId Id { get; }
  public string FirstName { get; }
  public string LastName { get; }
}

Assume that the above code contains a single constructor where one specifies the id, first and last name. Now assume that we have a class, database that contains a function saves customer details. The class might look as follows


class Database
{
  public static void Save(Customer customer)
  {
    // code to save customer to a database
  }
}

You may even have a top entry class. I tend to do this and usually call the class App. My app class usually acts as a kind of gateway, even an API to allow other parts of my software to call important code. As an example I might have something like the following.


static class App
{
  private static readonly Database _db = new Database("connection string");
  
  public static void Save(Customer customer)
  {
    using (var trans = _db.Open())
    {     
      _db.Save(customer);
    }
  }
}

Of course, in the above code, the database is slightly different to the one I originally presented. My point was that at the top-level, not supplying dependencies is OK. In the above example, I don't supply the concrete database class. I specify this as a class static.

In summary, non-top level functions should always explicitly state inputs. Top-level functions can ignore this rule and create dependencies within the function body. This should only ever occur at top-level functions.

Top

Refactor Regularly

As a project progresses, you will most likely start to think that the original code leaves much to be desired. This has happened to me often (even though the code works and works well). This is not the end of the world, you simply need to refactor your code. Refactoring is similar to writing code from scratch using new knowledge. The real difference is that you will be replacing existing code with new code, possibly using new lessons learned--------------------------------------.

Following a release, I think it always worthwhile refactoring the existing code.

Use Higher-Order Functions

Use Interfaces For Varying

Treat Exceptions And Anticipated Errors Differently

Favour Composition Over Inheritance

Friday, 14 May 2021

Functional Programming

What is functional programming?

Functional programming places functions first. All aspects of the resultant software is built upon functions. In contrast, Object-Oriented development places objects first. In this case, objects encapsulate state and potentially, behaviour. In functional programming, data and behaviour are typically kept separate. In my experience, the two techniques can be summarised as follows.

  • Object-Oriented

    Well-suited for user interface development as widgets (e.g. buttons, text input, etc) can maintain state such as text, event handlers for when say a button is clicked or text is editied.

    Data types such as stacks, queues, dictionaries, etc. Managing state and exposing behaviour in single place makes sense.

  • Functional

    Better suited where partitioning state across different objects is non-sensical or becomes problematic.

    Data types are complex, for example invoice processing, game state and so on. More complex data types are better manipulated using functions rather than "attaching" functions to the data type (Functions attached to a data type are typically known as methods in Object-Oriented development).

Top

What is a function?

A function accepts one or more inputs and generates a single output. It is also possible for a function to accept zero inputs and just return a value. The main take should be that a function can only create an output based upon the input values supplied. This is important! What this means is that a function cannot use outside state (global state) to generate an output. A function that only uses supplied inputs to generate an output is known as a Pure Function

Top

How do I code a good function?

To create a solid, sound function that can be reasoned about ceratin rules must be followed.

  • Use pure functions

    As mentioned, a pure function generates an output based purely upon its inputs. Using global state is forbidden. Pure functions go a step further, executing code that produces any side-effects is forbidden. This includes writing to the console, writing to log files, updating a database. In short, a pure function should always produce the same output given the supplied inputs. No side-effects, no using global variables. In pure functional programming even throwing exceptions is no go. Makes sense as throwing an exception is a side-effect which goes against the notion of a pure function.

    An exception is really just a goto on steriods. Exceptions can make following program flow and debugging difficult. Well-written software shouldn't need debugging. Sure, debugging a new function whilst still in development is likely a must. Once the function has been tested you should be able to trust it and move on.

    There are ways to implement state changes whilst still adhering to the rules. I shall go into details later,

  • Use immutable data

    Mutable data is data than can change in place. I remember my C/C++ days when dealing with strings. It was common to modify existing strings. This is more performant than creating new string based upon old strings.

    Times have changed, as have data structures, memory and so on. Immutable data is now the way forward. This essentially means once you have created some data, an object, record whatever your language permits, it never changes. If you need different values, you create new data, possibly based upon the original data.

    Take some simple data, the data represents X and Y coordinates in 2D space. Using C#, the code to represent the data is as follows.

    
    public struct Point
    {
      // get and set are C# constructs that allow one to set or get a data value.
      // get allows one to read a data value.
      // set allows one to update (write to) a data value.
      public int X { get; set; }
      public int Y { get; set; }
    
      public Point(int x, int y)
      {
        this.X = x;
        this.Y = y;
      }
    
      public void Offset(int xOffset, int yOffset)
      {
        this.X += xOffset;
        this.Y += yOffset;
      }
    }
    

    Now assume the following test class

    
    public static class TestPoint
    {
      public static void Test1()
      {
        Point pt = new Point(20, 40);
        pt.X = 1000;
        pt.Offset(0,20);
        // pt now contains the values x=1000 and y=60.
      }
    }
    
    Top

    In the above code, pt was created to have an X value of 20 and a Y value of 40. The second statement assignes 1000 to X. This directly modifies the pt variable. This is known as in-place modification. The ability to modify data content in-place makes the data structure mutable. The code also contains an Offset method. This method adds x and y offsets to the existing data.

  • How do I refactor mutable code to immutable code?

    In this case, the transition from mutable to immutable code is easy. Make all data fields readonly and supply a constructor to initialise the data fields. The following code example illustrates the idea.

    
    public struct Point
    {
      // A field that contains only a get construct is deemed to be readonly.
      // The field value may only be assigned via the constructor.
      public int X { get; }
      public int Y { get; }
    
      public Point(int x, int y)
      {
        this.X = x;
        this.Y = y;
      }
    
      public Point WithOffset(int xOffset, int yOffset)
      {
        return new Point(X + xOffset, Y + yOffset);
      }
    }
    
    public static class TestPoint
    {
      public static void Test1()
      {
        // Create a pt variable as per previous example.
        Point pt = new Point(20, 40);
        
        // Unable to modify the pt variable as fields are readonly.
        // We need to create a new data, i.e. a new instance of Point.
        // We create a new point by adding 1000 to the current X and zero to the Y value.
        Point ptNew = pt.WithOffset(1000, 0);
      }
    }
    
    

    Notice how the new code doesn't quite behave in the same way as the previous code. In the previous code we directly set the X value to a 1000. To achieve the same result in the new, immutable data we need to add a new function, let's call it WithX.

    public struct Point
    {
      // A field that contains only a get construct is readonly.
      // The field value may only be assigned via the constructor.
      public int X { get; }
      public int Y { get; }
    
      public Point(int x, int y)
      {
        this.X = x;
        this.Y = y;
      }
    
      public Point WithOffset(int xOffset, int yOffset)
      {
        return new Point(X + xOffset, Y + yOffset);
      }
      
      public Point WithX(int x)
      {
        return new Point(x, Y);
      }
    }
    
    
Top