ecoist

Weave Assesment

               ....$....
            ..$%00000000$$.
           .$0000%$$$$0000$.
          .%000$.     .$000$
          $000$.    .  $000$.                                   ..$..                 ......
          $00%$  .$$%$$0000.                                    $000$                 $%%%$
          .$$$$. .$$0000%$.                                     .$$$.             .$$$$$$$$$$$.
          ..$$$$$.   ...                                                         .%00000000000.
       .$$%0000000$.         ..$$$$$$..      ..$$$$$$$.       .$$$$.  .$$$$$$$.  ...$$$$$.....
     .$000000$$$$0$.       .$%00000000$.   .$0000000000$.     $000$ .$%00000000$.  .$$$$.
   .$0000$$..    ..       .$000$$...$0$$ .$0000$$...$000$    .%000. .000$...$%$$.  $%$$$.
  .$000%.                $000%$         .$000$.     .%00#.   $000$  $000$          %000.`
  $000%.               .$000$          .%000.     .$%$$$$. .$000$   .%0000$.     .$000.    $%$$.
  $0000.              .$0000.         $%000$      .000%$$$$%000%.  .$0000%%$.   .$000$    $000$.
  .$000%.          ..$000000.       .$00000$.     .$%0000000000$  .$000$$$$$$..$00000.  .$000$.
   .$0000$$......$$%0000$000$......$%00%%000$....$$$$$$$$$$$000...%00%$  .$$$%000000$...$000$.
    .$$00000000000000%$..$%0000%%0000%$..$0000%0000%$.     $000%%000$..$$$00000$.$000%%000$$
       ..$$$$$$$$$$..     ..$$$$$$$$..    .$$$$$$$..        .$$$$$$$$00000$$$$$.  .$$$$$$.
                                                                .$0000$$...$$$%$
                                                                $000$.   .$%00$.
                                                                $000$..$$%000$.
                                                                .$00000000%$.
                                                                  ..$$$$...
Let's make the world a better place

Requirements

• make (tested with GNU Make 4.4, Built for x86_64-apple-darwin21.6.0)
• cat
• go 1.18
• go testify library

Assessment

Disclaimer, this is my first Go program ever written Although the assignment was clear, my main focus was having fun!
I wanted to explore and personally find out if Go could become my future weapon of choice.
I hope this comes across when exploring the code.

Solution

To create a solution the following assumptions we're made:

• The data consits of regular meterings every 15 min
• The metrings have ascending values every 15 min
• A metering point never starts with an invalid reading
• An invalid reading is always followed by a valid reading
• After splitting the data file it only contains chunks with valid start and endlines
• The data set is large and more or less rectangular, the amount of readings is a multitide of the amount of clients
• The input csv file doesn't contain a header

Dealing with invalid readings

To calculate cost we creata a sum of readings
For handling invalid reading we store the last valid reading.
If we have an invalid reading, we store it until we get a valid reading.
We then calculate the average cost over the period and continue.
Because of a reading being invalid depends on the previous reading, what makes a reading invalid is a bit fuzzy. We only have a rule for usage, but how do we determine if a reading is invalid if the previous is invalid?

Dealing with large data sets

Because of the characteristics of the data it should be possible to consume different parts concurrently. Whenever we receive a csv file, we split it in segments of about 100MB. We then process these segmenst concurrently. The file can be split in the middle of the readings of a specific client. To mitigate this each process can receive one or more segements. If the next segment starts with client of the previous segment, the process keeps on processing readings until the client changes.

Testing the solution.

• Run unit tests, make test
• Create a benchmark file, and run the benchmark test make run, Select Generate benchmark file (will take some time), make bench
• Run the app and select a file to process