Learning Kubernetes internals by configuring a cluster

In the last couple of weeks I’ve been learning Kubernetes. During that process, I wanted to understand which components are part of a Kubernetes cluster and how they work together. While searching for resources, I came across Joshua Sheppard’s Kubernetes by Component series. His goal was to “provide a deeper sense of what each component in a Kubernetes cluster is doing by building up a cluster one component at a time“. Sounded exactly like what I needed!

Joshua’s approach was to extend/update a similar series written in 2015 by Kamal Marhubi with his own learning. While following along his posts, I realized that the best approach was to do the same he had done: configure a cluster one component at a time and document the process. I used the most recent Kubernetes binaries and automated each step of the process using Vagrant VMs. The result can be found in my K8s Bootstrap repo.

Hope this helps!

IAsyncEnumerable – Migrating to .NET Core 3.0

C# 8 and .NET Core 3.0 introduce async streams, which “are asynchronous variants of enumerables, where getting the next element may involve an asynchronous operation“. This feature is built around the IAsyncEnumerable/IAsyncEnumerator interfaces and the new await foreach statement. A lot has already been written about the feature since its preview releases, so I’d like to focus in a different thing: migrating from a previous version of IAsyncEnumerable.

Prior to C# 8, there was already a version of IAsyncEnumerable/IAsyncEnumerator defined in the System.Interactive.Async package (version <= 3.2.0), which is part of the Reactive Extensions for .NET. The package also included LINQ-like extension methods for async enumerables and helpers to create them. You could be using the package and its facilities, for instance, when using gRPC streaming (version < 2).

If you want to migrate a project as described above to .NET Core 3.0 (or .NET Standard 2.1) you’ll get a conflict between the old (System.Interactive.Async) and new (.NET Standard 2.1) types. If the project at hands is a deployable (e.g. ASP.NET Core application) you can simply update the System.Interactive.Async package to a version >= 4.0.0, which no longer includes its versions of IAsyncEnumerable/IAsyncEnumerator. It’s worth noting, however, that the old and new interfaces are a bit different, so some tweaks are needed (more on this later).

Multi-targeting on shared packages for an easy migration

Another possible scenario is that you have a shared .NET Standard 2.0 library that depends on IAsyncEnumerable/IAsyncEnumerator from System.Interactive.Async and is used in multiple clients which you may not want (or be able) to upgrade to .NET Core 3.0 immediately. This was the case in my current project, where we had a shared library of gRPC utilities being used in different .NET Core 2.2 micro-services. We wanted to start migrating the services to .NET Core 3.0 and the types conflict arose. However, we didn’t want to migrate ALL the services now but we still wanted to keep updating the shared libraries

Along with the recent releases of .NET Core there is the Microsoft.Bcl.AsyncInterfaces package, which includes the IAsyncEnumerable/IAsyncEnumerator interfaces as a standalone release. The solution we found was multi-targeting both .NET Standard 2.0 and 2.1 in the shared library, referencing this package only for .NET Standard 2.0. Here’s the csproj of such a shared library:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>netstandard2.0;netstandard2.1</TargetFrameworks>
  </PropertyGroup>
  <ItemGroup Condition="'$(TargetFramework)' == 'netstandard2.0'">
    <PackageReference Include="Microsoft.Bcl.AsyncInterfaces" Version="1.0.0" />
  </ItemGroup>
</Project>

This allows consuming the library from projects on both .NET Standard versions and allows you to start using await foreach in the shared code. This is actually what the System.Interactive.Async packages does internally (version > 4.0.0).

Code changes

Besides picking types from the correct assembly, this migration also means doing some code changes, as the old and new interfaces are a bit different.

// Old (System.Interactive.Async)
public interface IAsyncEnumerable<out T>
{
    IAsyncEnumerator<T> GetEnumerator();
}
public interface IAsyncEnumerator<out T> : IDisposable
{
    T Current { get; }
    Task<bool> MoveNext(CancellationToken cancellationToken);
}

// New (NET Standard 2.1 / Microsoft.Bcl.AsyncInterfaces)
public interface IAsyncEnumerable<out T>
{
    IAsyncEnumerator<T> GetAsyncEnumerator(CancellationToken cancellationToken = default);
}
public interface IAsyncEnumerator<out T> : IAsyncDisposable
{
    T Current { get; }
    ValueTask<bool> MoveNextAsync();
}

The main difference is on how cancellation is handled: in the old version, each call to MoveNext would get a cancellation token, while on the new version the token is supplied on GetAsyncEnumerator. This means that the cancellation is captured when an enumeration starts, and the async enumerator is responsible for honoring it on the different calls to MoveNextAsync.

Cancellation tokens that were supplied to MoveNext should now be supplied to GetAsyncEnumerator. If you’re consuming async enumerables with await foreach, by default no cancellation token is supplied. In order to do that, the WithCancellation method can be used:

await foreach(var i in someAsyncEnumerable.WithCancellation(token))

More on cancellation

The previous example illustrates cancellation from the consumer perspective. What about usage of cancellation tokens when implementing async streams with the async foreach statement? Lets say we have the following method:

public async IAsyncEnumerable<int> GetSomeAsyncEnumerable()
{
    await foreach (var i in GetOtherAsyncEnumerable())
    {
        await Task.Delay(i);
        yield return i * i;
    }
}

The first impulse would be to add a cancellation token as a parameter of the method and use it on the call to Delay. However, this would mean that the cancellation token is bundled with the async enumerable itself and that it is used on different enumerations of the enumerable! That would be incorrect.

Keep in mind that the compiler will generate an implementation of the async enumerable which has a GetAsyncEnumerator(CancellationToken cancellationToken) method. What we actually need is a way to access the token that will be supplied by the consumers of our enumerable. Meet the [EnumeratorCancellation] attribute! The previous method can be re-written as:

public static async IAsyncEnumerable<int> GetSomeAsyncEnumerable([EnumeratorCancellation] CancellationToken token = default)
{
    await foreach (var i in GetOtherAsyncEnumerable())
    {
        await Task.Delay(i, token);
        yield return i * i;
    }
}

You can think of this attribute as something that turns the token parameter into a placeholder for the cancellation tokens supplied to GetAsyncEnumerator by the consumers of the enumerable. In addition, the compiler does a bit more magic:

  • If no cancellation token is supplied to GetSomeAsyncEnumerable, than token represents the cancellation token supplied to each enumeration.
  • If a cancellation token is supplied to GetSomeAsyncEnumerable, than token is a new instance created by the compiler which combines the initial parameter and the token supplied to each enumeration. This means that the enumeration is cancelled when either of those two is cancelled.

Summary

In this post I discussed a possible migration path – and code changes – for projects that want to upgrade to .NET Core 3.0 / .NET Standard 2.1 and were using the IAsyncEnumerable/IAsyncEnumerator interfaces defined in the System.Interactive.Async package. The easiest way to migrate is to use multi-target on shared libraries and depend on the new Microsoft.Bcl.AsyncInterfaces package.

I also discussed some of the required code changes as well as the new [EnumeratorCancellation] which can be used to easily support cancellation on async enumerables using await foreach.

Hope this helps!

TS intersection types, JS prototypes and NgRx selectors

Lately I’ve been working in an Angular project using NgRx for state management and I came across a scenario that I found to be common: exposing combined state from the store.

Lets say we have an application that manages products, providing at least two different views of those products: a ProductSummary list and a ProductDetails page. In addition, the application should “lock” a product for modification while a “modify” operation is ongoing. In this scenario, the state on the NgRx store could be represented as follows:

export interface State {
  products: ProductSummary[];
  selectedProduct: ProductDetails | null;
  lockedProductIds: string[];
}

Given this state, I’d like to disable some action buttons both on the list and on the details page while a product is locked (due to some pending action). Before diving into the solution I implemented, lets think of same possibilities:

  1. Define separate selectors for the product summary/details and the “is locked” flag. This works alright for the product details component, which would likely have two @Input properties, based on two different observables. This approach is used on the official NgRx sample application and has the advantage of only triggering state changes on the needed part of the state. However, if you think on the product list, it becomes messy, as we’d need to expose a third selector for the “set of locked products” and build the list’s UI based on two arrays (products and locked products).
  2. Define new “view model” types – ProductSummaryWithState and ProductDetailsWithState – that extend the existing product types and expose them via selectors, one for the list and one for the selected product.This is more cohesive but might mean more memory usage due to a copy from ProductSummary to ProductSummaryWithState, unless we use some tricks (more on this in a bit). Also, if the same concept is needed for other parts of the application, we’ll need to create more types such as SomeOtherEntityWithState.
  3. Define new view model types that wrap the product and the locked state, and define selectors that build up those models on the fly. This is good in terms of cohesion and memory usage, but the component templates look uglier, as we’d need to write item.product.nnn everywhere.

The solution I ended up implementing takes a bit from all the previous options and tries to combine them into a better approach. First I defined a new interface to represent the generic “control state”:

export interface ControlState {
  locked: boolean;
}

Then, instead of defining explicit new types to combine the control state with the products, I used Typescript’s intersection types. This means that when selecting state from the store I want to get Observables like the ones illustrated below.

products$: Observable<ProductSummary & ControlState> = this.store.pipe(select(getProducts));

selectedProduct$: Observable<ProductDetails & ControlState> = this.store.pipe(select(getSelectedProduct));

With this approach the templates can use both product.name and product.locked on the same object and that the TS compiler is fine with passing such an object in places that get either a ProductSummary or a ControlState.

To expose the data in this form we need to write selectors that combine a product and the control state (the same problem as described earlier on alternative 2). A first approach to this is using TS’s spread operator.

export const getSelectedProduct = createSelector(
  (state: State) => state.selectedProduct,
  (state: State) => state.lockedProductIds,
  (selectedProduct, lockedProductIds) => {
    if (selectedProduct === null) {
      return null;
    }
    const o = {
      ...selectedProduct,
      locked: !!lockedProductIds.find(id => id === selectedProduct.id)
    };
    return <ProductDetails & ControlState>o;
  }
);

While this works, it means that we actually copy the product object. This would be worst for the list page. A better option is to rely on Javascript prototypes!

(selectedProduct, lockedProductIds) => {
    if (selectedProduct === null) {
      return null;
    }
    const o = <ControlState>Object.create(selectedProduct);
    o.locked = !!lockedProductIds.find(id => id === selectedProduct.id);
    return <ProductDetails & ControlState>o;
  }

We create a new object for the control state that uses the product as its prototype. This means that when accessing the type intersection object, all the product properties are actually delegated to the prototype (the actual product object), but this is fine because the interface is still correct.

With this approach we avoid explicitly defining new types for “view models” and unnecessary memory usage when projecting state from the store. Hope this helps!

JSON.NET converter for type hierarchies

By default, JSON.NET supports serialization of type hierarchies – or, more generically, runtime types different from the ones declared in the respective properties – through the usage of TypeNameHandling. The serializer can add a $type property whose value is the fully qualified name of the serialized type. While this works, it can be a problem for interop and code maintainability.  There are ways to customize the type name that gets serialize but you still get stuck with the $type and global resolution of types from the name that is serialized (possible collisions if one isn’t careful).

I wrote the following converter that handles a class hierarchy by adding a discriminator property with configurable name and values.


class SubTypesConverter<T> : JsonConverter<T> where T: new()
{
[ThreadStatic]
private static bool isWriting;
private readonly string discriminatorName;
private readonly Dictionary<string, Func<T>> factories;
private readonly Dictionary<Type, string> discriminators;
public override bool CanRead => true;
public override bool CanWrite => !isWriting;
public SubTypesConverter(string discriminatorName)
{
this.discriminatorName = discriminatorName;
this.factories = new Dictionary<string, Func<T>>();
this.discriminators = new Dictionary<Type, string>();
var types = typeof(T).Assembly
.GetTypes()
.Where(t => typeof(T).IsAssignableFrom(t) && !t.IsAbstract);
foreach (var t in types)
{
var discriminator = this.GetDiscriminator(t);
this.factories.Add(discriminator, CreateFactory(t));
this.discriminators.Add(t, discriminator);
}
}
public override T ReadJson(JsonReader reader, Type objectType, T existingValue, bool hasExistingValue, JsonSerializer serializer)
{
if (hasExistingValue)
{
throw new NotSupportedException($"{nameof(SubTypesConverter<T>)} does not allow reading into an existing instance");
}
var jsonObject = JObject.Load(reader);
var discriminator = jsonObject[this.discriminatorName].Value<string>();
var value = this.factories[discriminator]();
serializer.Populate(jsonObject.CreateReader(), value);
return value;
}
public override void WriteJson(JsonWriter writer, T value, JsonSerializer serializer)
{
try
{
isWriting = true;
var jsonObject = JObject.FromObject(value, serializer);
jsonObject.AddFirst(new JProperty(this.discriminatorName, this.discriminators[value.GetType()]));
jsonObject.WriteTo(writer);
}
finally
{
isWriting = false;
}
}
protected virtual string GetDiscriminator(Type type)
{
return type.Name;
}
private static Func<T> CreateFactory(Type t)
{
var newExp = Expression.New(t.GetConstructor(Type.EmptyTypes));
return Expression.Lambda<Func<T>>(newExp).Compile();
}
}

You just need to create an instance passing the base class of your hierarchy as the generic argument and add it to JsonSerializerSettings.Converters. By default the converter uses the type name as the value of the discriminator, but you can change this by subclassing and overriding the GetDiscriminator method. Also, it assumes that all the types in the hierarchy starting in are in the same assembly as T.

Hope this helps!

ASP.NET Core on Azure Service Fabric: accessing Key Vault with Managed Identities (and the errors on the token endpoint!)

Managed Identities are an Azure feature that provides automatically managed identities to different Azure services (e.g. a VM) that can be used for authentication in other services (e.g. Key Vault). This avoids storing/managing credentials in the source code.

At the time of writing, VM Scale Sets support for Managed Identities is in preview. Since Service Fabric clusters are usually deployed on top of VM scale sets, it means that Managed Identities can be used with Service Fabric (tested with reliable services).

A common use-case is to have an ASP.NET Core service using Key Vault as a source of configuration values. If the cluster (actually the underlying VMs) are using a Managed Identity, you can authenticate into Key Vault using that identity. The easiest way to setup this scenario is using the Microsoft.Extensions.Configuration.AzureKeyVault package, which provides a AddAzureKeyVault extension method for IConfiguration. There are many articles on how to do this integration and configure the service identities and access policies.

I was recently doing this setup and I started getting the following exception when loading the configuration from Key Vault:

AzureServiceTokenProviderException
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://lo
gin.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47. Exception Message: Tried the following 3 methods to get an access
token, but none of them worked.

The error actually happens when the AzureServiceTokenProvider is trying to obtain a token to access the Vault using the service’s Managed Identity.

Lesson #1: pass “RunAs=App;” in the connectionString parameter  of AzureServiceTokenProvider. This way it will not try different modes to obtain a token, and the exception is a bit better.

The error now is something like: “cannot establish a connection because the target machine actively refused ir”.

Lesson #2: install/update the latest version of Microsoft.Azure.Services.AppAuthentication. The aforementioned package with the configuration extensions doesn’t depend on the latest version of this one. It happens that there’s an hard-coded URL of the endpoint that is used to get the Vault access token… and that endpoint changed since there was an “old way” of getting those tokens that is being deprecated.

After updating I got everything working. The versions I used for the packages are:

  • Microsoft.Extensions.Configuration.AzureKeyVault: 2.1.1
  • Microsoft.Azure.Services.AppAuthentication: 1.0.3 (version 1.0.1 won’t work with Managed Identities on Service Fabric).

All the “get the token” behavior is hidden in the AppAuthentication package and since it was an indirect dependency it was hard to spot where the problem was. Hope this helps!

Referencing system assemblies in Roslyn compilations

With the recent stable release of the Roslyn packages (.NET Compiler Platform) I finally decided to give it a try. I’ve found it pretty simple to compile some code in runtime, load the resulting assembly and execute it. Overall it’s pretty impressive, given the amount of work that’s being done.

There are loads of resources on Roslyn online, namely Josh Varty’s Learn Roslyn Now series and Rolsyn’s documentation on GitHub. On this post, however, I’ll focus one aspect that wasn’t obvious to me in the first place: referencing shared framework assemblies.

When writing code it’s very likely that one ends up using types defined in reference assemblies such as System.Runtime and System.Runtime.Extensions, specially with .NET Standard  and .NET Core’s pay-for play model. These assemblies are just facades and the used types will eventually be forwarded to implementation assemblies in runtime, depending on the target framework.

Going back to Roslyn, the idea is that we’ll be compiling/generating code in runtime, so the question arises: which assemblies should I reference during the compilation? The only satisfying explanation I’ve found is a bit lost in a GItHub issue, hence this post.

There are two main approaches we can use during compilation: 1) use runtime (implementation) assemblies; and 2) use reference (contract) assemblies.

The remainder of this post is based on the following sample program that uses Roslyn. Note that the code to compile (code variable) uses DateTime (defined in reference assembly System.Runtime) and Math (defined in reference assembly System.Runtime.Extensions).

class Program
{
    static void Main(string[] args)
    {
        var code = @"
        using System;
        public class Test
        {
            public double MagicNumber => Math.Round(42.42 * DateTime.Now.Second);
        }
        ";

        var syntaxTree = CSharpSyntaxTree.ParseText(code);

        List references = // TODO define references here!
        var options = new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary, optimizationLevel: OptimizationLevel.Release, allowUnsafe: false);

        var compilation = CSharpCompilation.Create(Guid.NewGuid().ToString("N"), new[] { syntaxTree }, references, options);

        using (var ms = new MemoryStream())
        {
           var compilationResult = compilation.Emit(ms);
           if (compilationResult.Success)
           {
                ms.Position = 0;
                var assembly = AssemblyLoadContext.Default.LoadFromStream(ms);
                var instance = Activator.CreateInstance(assembly.DefinedTypes.First().UnderlyingSystemType);
               // Do something with "instance"
            }
            else
            {
                foreach (var error in compilationResult.Diagnostics)
                {
                   Console.WriteLine(error.ToString());
                }
            }
        }
     }
}

Option 1 – Runtime assemblies

Since we’re invoking Roslyn, we’re already in the context of an executing application targeting a specific framework. This framework includes the runtime (implementation) assemblies that we need, so the first option for compilation is referencing those assemblies. But how can we know their location in a flexible way?

The .NET Core runtime defines a so called “trusted platform assemblies” list, accessible via the AppContext class:

var trustedAssembliesPaths = ((string)AppContext.GetData("TRUSTED_PLATFORM_ASSEMBLIES")).Split(Path.PathSeparator);

This list contains the locations of all the assemblies loaded from trusted locations, namely the ones on shared framework installations and NuGet caches. Example:

"C:\Program Files\dotnet\shared\Microsoft.NETCore.App\1.1.1\System.Linq.dll"
"C:\Users\luis\.nuget\packages\system.xml.xpath\4.3.0\lib\netstandard1.3\System.Xml.XPath.dll"

The aforementioned list contains a lot of assemblies that we probably don’t need (and don’t want to make available!) in our code, so it’s probably a good idea to filter the list. After that, creating a MetadataReference is straightforward. The full code to obtain the references that my code needs is:

var trustedAssembliesPaths = ((string)AppContext.GetData("TRUSTED_PLATFORM_ASSEMBLIES")).Split(Path.PathSeparator);
var neededAssemblies = new[]
{
    "System.Runtime",
    "mscorlib",
};
List references = trustedAssembliesPaths
    .Where(p => neededAssemblies.Contains(Path.GetFileNameWithoutExtension(p)))
    .Select(p => MetadataReference.CreateFromFile(p))
    .ToList();

Pros:

  • All the needed assemblies are made available by the framework. No need to include additional resources on the application package.

Cons:

  • Need to know the actual runtime (implementation) assemblies. Note that my code depends on System.Runtime.Extensions facade, but I actually add a reference to mscorlib.
  • If an implementation assembly is updated (e.g. shared framework updated) the code might break (it’s unlikely, but possible). Even though .NET APIs should be backward compatible, “compiler overload resolution might prefer an API added in the new version instead of the one that it used to pick before

Option 2 – Reference assemblies

Instead of relying on runtime assemblies, we can compile the code against reference assemblies. The NuGet packages for System.Runtime and alike include both implementation and reference/contract assemblies (you can see this on your local NuGet cache).

System.Runtime/4.3.0/
                   \_ lib/net462/System.Runtime.dll
                   \_ ref/netstandard1.5/System.Runtime.dll

As these assemblies are in a well know location, we can easily create MetadataReferences:

var nugetCache = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".nuget/packages");
List references = return new List<MetadataReference>
{
  MetadataReference.CreateFromFile(nugetCache + @"System.Runtime\4.3.0\ref\netstandard1.5\System.Runtime.dll"),
  MetadataReference.CreateFromFile(nugetCache + @"System.Runtime.Extensions\4.3.0\ref\netstandard1.5\System.Runtime.Extensions.dll"),
}
.ToList();

Note that in the sample above I’m using the local NuGet cache. In a real-world scenario you’d include the reference assemblies for the chosen .NET Standard version in the application package (e.g. as resources).

When the assembly generated using Roslyn is loaded, its assembly references are resolved into the corresponding runtime assemblies.

Pros:

  • Assemblies are loaded from a well-known location.
  • Since the reference assemblies won’t change (unless you explicitly update them) there’s a guarantee that the code won’t break on future versions of the .NET Core runtime.

Cons:

  • Additional items on the application package.

Conclusion

Despite both approaches being valid and the first one being used by the C# Scripting API, I tend to prefer the second as it is more stable and less obscure. Hope this helps!