How not to benchmark!

I came across a recent LinkedIn post about the let statement in LINQ and it's performance implication. And in typically influencer fashion it out right claimed that using let in LINQ is a bad idea and should be avoided. But is it a bad idea?

let

let is a keyword in LINQ that allows you to store the result of an expression in a variable and use it in the subsequent expressions. It's a very useful feature that can make your code more readable and maintainable. Here an example:

var query = from c in customers
            let totalOrders = c.Orders.Count()
            where totalOrders > 10
            select new { c.Name, totalOrders };

That is especially helpful if you need the variable in multiple places in your query.

The LinkedIn post

The title already starts with: "Avoid Using the "let" Keyword in LINQ Queries".

Now that is a bold claim. The interesting bit that peeked my interest was the runtime of only a few nanoseconds. That is a very small number and I was curious how they measured that. So I decided to investigate. Initially I suspected a very very small sample set, but no, the sample set was 10000 elements. Here the code:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

public class LinqQueryPerformanceBenchmark
{
    private List<Person> _peopleList;

    public LinqQueryPerformanceBenchmark()
    {
        _peopleList = new List<Person>();
        for (int index = 0; index < 10000; index++)
        {
            _peopleList.Add(new Person
            {
                FirstName = index % 2 == 0 ? "A" : "C",
                LastName = index % 3 == 0 ? "B" : "D"
            });
        }
    }

    [Benchmark]
    public void FilterPeopleWithoutLet()
    {
        var filteredPeople = from person in _peopleList
            where person.LastName.Contains("B")
                  && person.FirstName.Equals("A")
            select person;
    }

    [Benchmark]
    public void FilterPeopleUsingLet()
    {
        var filteredPeople = from person in _peopleList
            let isLastNameHaris = person.LastName.Contains("B")
            let isFirstNameAdmir = person.FirstName.Equals("A")
            where isLastNameHaris && isFirstNameAdmir
            select person;
    }

    public class Person
    {
        public string FirstName { get; set; }
        public string LastName { get; set; }
    }
}

public class BenchmarkProgram
{
    public static void Main(string[] args)
    {
        var benchmarkSummary = BenchmarkRunner.Run<LinqQueryPerformanceBenchmark>();
    }
}

That benchmark is just not benchmarking what the post claims. It's not measuring the performance of the enumeration at all! It's just measuring the time it takes to create the query. Why? For starters: There is no return value. That is often times not good, because the JIT^Just-In-Time compiler can optimize the code in a way that it distorts the results. But more importantly the queries are not materialized. Both functions FilterPeopleWithoutLet and FilterPeopleUsingLet are returning IEnumerable<Person>. But IEnumerable is a forward collection, so if you don't call something like ToList, ToArray, Sum or any other method that forces the enumeration, the query is not executed at all. So the benchmark is just measuring the time it takes to create the query, not the time it takes to execute it.

If you run the corrected benchmark, you will get a different picture:

[Benchmark]
public List<Person> FilterPeopleWithoutLet()
{
    return (from person in _peopleList
        where person.LastName.Contains("Haris")
              && person.FirstName.Equals("Admir")
        select person).ToList();
}

[Benchmark]
public List<Person> FilterPeopleUsingLet()
{
    return (from person in _peopleList
        let isLastNameHaris = person.LastName.Contains("Haris")
        let isFirstNameAdmir = person.FirstName.Equals("Admir")
        where isLastNameHaris && isFirstNameAdmir
        select person).ToList();
}

Results:

| Method                 | Mean      | Error    | StdDev   |
|----------------------- |----------:|---------:|---------:|
| FilterPeopleWithoutLet |  69.91 us | 0.942 us | 0.835 us |
| FilterPeopleUsingLet   | 204.65 us | 1.604 us | 1.422 us |

So now the numbers are making a bit more sense - and yes we can see that using let will have a performance impact. But is it significant? That depends on your use case. Are you dealing regularly with 10k elements? Are you doing this operation in a tight loop? If not, then the performance impact is negligible. And the readability and maintainability of your code is more important.

And if performance would be important, you probably wouldn't use LINQ in the first place. You would use a for loop and do the filtering manually (combined with SearchValues or some Spanification to maximize performance gains and minimize allocations). But that would make your code less readable and maintainable. So it's always a trade-off.

Conclusion

Always be careful when you see benchmarks. They can be misleading. And always make sure that the benchmark is actually measuring what it claims to measure and is correct for your use-case! And that isn't only correct for LinkedIn - that is also correct for this very own blog you are reading right now. I also can make mistakes and give you misleading information. And the probability of that happening is high 😉