Pair programming with AI - an experience report (using Pulumi, F#, and AWS)

This is a brief report on having a programming session with some AI assistance, using a tool combination I was not entirely familiar with. It was a useful learning experience, trying to address the shortcomings in the AI solution, and combining it with some good old-fashioned documentation search and actual example code review.

The setting

Recently I had a task to set up Amazon Managed Streaming for Kafka (Amazon MSK) for evaluation. The setup of the MSK cluster used Terraform. Amazon MSK is perhaps not the smoothest experience, and creating or changing an MSK cluster took between 30-90 minutes.

While waiting for another cluster update, I needed to set up an EC2 instance in the target VPC to connect to the MSK cluster, using Kafka system tools that are part of the regular Kafka distribution.

I decided to add a bit of joyful experimentation to the work, and choose to set up the infrastructure with Pulumi and F3, with a bit of AI help. While I have used Pulumi on multiple occasions, I have not really used it with F#, and I have not used F# in the past 4-5 years. I recently decided to pick it up again, so I thought it would be a fun small experiment to use that with Pulumi.

First stab at the problem

I decided to first try to get something by using the Pulumi AI web page. This should be trained specifically for Pulumi related tasks, so should hopefully give a decent result. Unfortunately, there was no specific section for F#, so I played around a bit with Python and C# instead.

I described that I wanted an EC2 instance to be provisioned in an existing VPC, running Amazon Linux 2023. The instance should have the latest Kafka installation version supported by MSK installed with the purpose to connecting to an existing MSK cluster. I also wanted to connect to the instance via CloudShell or Session Manager. VPC id and subnet id should be obtained from configuration, as well as the bootstrap cluster URL for connection.

Unfortunately that was too vague or confusing for the Pulumi AI (ChatGPT 4o), and it did multiple mistakes. Fair enough, it corrected errors I pointed out to it, but then also introduced new errors. After correcting these errors, it reintroduced old errors again, and so it continued for a while.

At this point, it became a bit more annoying than useful. I get that there is likely limited context for the free version of the AI used here, and writing a more AI-friendly specification of the task would perhaps given a bit better result.

But that did not look particularly joyful to explore further, and it does not solve the actual problem at hand. So I decided to test another AI tool.

Second stab at the problem - Hello Claude

Next, I decided to give it a go with claude.ai instead. It is one of my preferred AI tools, and I have had good results before for other tools and languages.

It did start out better with Claude (free version). It generated code and project files that at first glance looked like they might work. Claude had missed that the SSM agent and AWS CLI already comes pre-installed on Amazon Linux 2023, and it also initially got confused about which Kafka version to install. But it corrected these after my remarks and it did actually not mess up the rest, which was an improvement.

However, when actually trying to build and run the program, there were multiple compilation errors across the code. I fed these to Claude, and it provided updates to address these. These led to other compilation errors, which I gave Claude as well. New corrections, and more compilation errors. Old errors started to come back, so it seemed we got stuck in some kind of loop of error corrections.

The errors had pretty much all of them to do with type conversions and handling what is known as input values, and output values in Pulumi.

When you define properties on a resource in Pulumi, you specify inputs. Resources that are provisioned may have outputs which are values that you can get from a resource. The output values of these resources is not known immediately when the program compiles and runs, it is not available until the resource has been provisioned.

Pulumi has specific types that deal with inputs and outputs, and this caused trouble for Claude in combination with F# type system. Other languages may have implicit conversions between for example a list of strings and an input list of strings for example, but this is not the case for F#, where you have to be explicit.

For example, this code that obtains an AMI gave multiple compilations errors:

    let ami = 
        Ec2.GetAmi.Invoke(Ec2.GetAmiInvokeArgs(
            MostRecent = true,
            Owners = ["amazon"],
            Filters = [
                Ec2.Inputs.GetAmiFilterInputArgs(
                    Name = "name",
                    Values = ["al2023-ami-*-x86_64"]
                );
                Ec2.Inputs.GetAmiFilterInputArgs(
                    Name = "architecture", 
                    Values = ["x86_64"]
                )
            ]
        ))

The input fields Owners, Filters, and Values all expect InputList<T> for some type T. Claude could not fix that part by just looking at the compilation errors. Instead, I started a separate Clause session and just focused on a single expression and the corresponding error. In that case, Claude was more useful in analyzing the problem and suggestion different options - it did also help that the F# compilation errors were informative.

I and Claude ended up adding helper functions for these cases, for example:

let inputList (items: 'a seq) =
    let list = InputList<'a>()
    items |> Seq.iter list.Add
    list

This converts any sequence of a type 'a to an InputList of the same type. Since F# has good type inference, there is no need to explicitly state that type in the code. Thus, you can then write something like

Owners = inputList ["amazon"]

and the compiler would be somewhat happy. So we wrote a couple of helper functions for a few cases like this, and updated the code.

Briefly I thought, this should be something that Pulumi should provide for F# users, but then moved on.

Running the code

We got to a point where the code would compile with dotnet build. But when trying to deploy it with pulumi up it still failed when it tried to run the code.

It turned out that in Pulumi.yaml file, Claude had specified the runtime to use as

runtime:
  name: dotnet
  options:
    binary: dotnet

which is not right if you just want to use the default setup with dotnet and not a custom binary. Instead, it should be just

runtime: dotnet

After making that change the pulumi up command worked just fine!

Make it nice

There is a phrase “make it work, make it right, make it fast” attributed to Kent Beck, which states in what order you should address your code design work.

So if the code now actually works, can we make it a bit nicer? The code of the program looked a bit like this for the main function:

module Program

open Pulumi
open Pulumi.Aws
open Pulumi.Aws.Ec2
open Pulumi.Aws.Iam
open Pulumi.Aws.Ec2.Inputs
open System.Threading.Tasks

[<EntryPoint>]
let main args =
    Deployment.RunAsync(fun () ->
        // Configuration from Pulumi config
        let config = Config()
.
. // many lines of code here
.
) |> Async.AwaitTask |> Async.RunSynchronously

This part with referencing System.Threading.Tasks, the call to Deployment.RunAsync with the anonymous function, whose output were sent to Async.AwaitTask and Async.RunSynchonously, those did not feel right. I could not put my finger on it exactly, but it felt a bit ugly.

So I decided to create a new F# project with the pulumi new command, and picked one of the templates available for F#. The code I looked at looked nicer, and I realized I had missed a few things about Pulumi and F#:

module Program

open Pulumi.FSharp
open Pulumi.Aws.S3

let infra () =

  // Create an AWS resource (S3 Bucket)
  let bucket = Bucket "my-bucket"

  // Export the name of the bucket
  dict [("bucketName", bucket.Id :> obj)]

[<EntryPoint>]
let main _ =
  Deployment.run infra

The main program approach here looked much cleaner! This must be due to Pulumi.FSharp there in the code. It was not the easiest to find the details for in the Pulumi docs, but found it at: https://www.pulumi.com/docs/reference/pkg/dotnet/Pulumi.FSharp/Pulumi.FSharp.html

When I read the documentation there, I also saw the Ops module. In that module I found pretty much all the helper functions I wrote, and then some!

So actually Pulumi had implemented these helpers, but I had not noticed or remembered those, because Claude did not bring that up. Claude likely had more information for C# code with Pulumi and had worked from that point of view with the F# code.

I had not spotted that myself, if it had not been for my feeling that the code felt a bit ugly.

I did some changes to the code to use Pulumi.FSharp instead, and now was at least a bit nicer to read.

Final thoughts and code

I found the experience to get this infrastructure deployment in place pretty fun and a good learning experience. In this case, Claude helped me to safe a good amount of time that it was worth the effort to do the experiment. Had I figured out everything from scratch myself it would have taken longer, and it helped make the final version which was implemented differently. Also, I really enjoyed using F# with Pulumi, and the learning is useful for later!

There were a couple of other things that Claude did not get quite right at first, and some changes I just made myself without Claude. For example, using dnf instead of yum for the installation commands, and installing nc (which I used for connectivity testing).

Below is the code I ended up with after this “pair programming session” with Claude, which was not the final version used though. But that is for another time.

module Program

open Pulumi
open Pulumi.FSharp
open Pulumi.Aws
        
let infra () =
    // Configuration from Pulumi config
    let config = Config()
    let vpcId = config.Require "vpcId" 
    let subnetId = config.Require "subnetId"
    let bootstrapServers = config.Get "bootstrapServers"
    let awsConfig = Config "aws" 
    let region = awsConfig.Require "region" 
    
    // Create IAM role for EC2 instance to use Session Manager
    let ec2Role = 
        Iam.Role("kafka-ec2-role",
            Iam.RoleArgs(
                AssumeRolePolicy = """{
                    "Version": "2012-10-17",
                    "Statement": [
                        {
                            "Effect": "Allow",
                            "Principal": {
                                "Service": "ec2.amazonaws.com"
                            },
                            "Action": "sts:AssumeRole"
                        }
                    ]
                }""",
                Description = "IAM role for EC2 instance with Kafka tools"
            ))
    
    // Attach AWS managed policy for Session Manager
    let ssmPolicyAttachment = 
        Iam.RolePolicyAttachment("ssm-policy-attachment",
            Iam.RolePolicyAttachmentArgs(
                Role = ec2Role.Name,
                PolicyArn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
            ))
    
    // Attach policy for MSK access (optional - adjust permissions as needed)
    let mskPolicy = 
        Iam.RolePolicy("msk-access-policy",
            Iam.RolePolicyArgs(
                Role = ec2Role.Id,
                Policy = """{
                    "Version": "2012-10-17",
                    "Statement": [
                        {
                            "Effect": "Allow",
                            "Action": [
                                "kafka:*",
                                "kafka-cluster:*"
                            ],
                            "Resource": "*"
                        }
                    ]
                }"""
            ))
    
    // Create instance profile
    let instanceProfile = 
        Iam.InstanceProfile("kafka-instance-profile",
            Iam.InstanceProfileArgs(
                Role = ec2Role.Name
            ))
    
    // Create security group
    let securityGroup = 
        Ec2.SecurityGroup("kafka-ec2-sg",
            Ec2.SecurityGroupArgs(
                Description = "Security group for Kafka EC2 instance",
                VpcId = vpcId,
                Egress = inputList [
                    input(Ec2.Inputs.SecurityGroupEgressArgs(
                        FromPort = 0,
                        ToPort = 0,
                        Protocol = "-1",
                        CidrBlocks = inputList [input "0.0.0.0/0"],
                        Description = "All outbound traffic"
                    ))
                ],
                Ingress = inputList [],
                Tags = inputMap [
                    ("Name", input "kafka-ec2-security-group")
                ]
            ))
    
    // User data script to install Java and Kafka
    let userData = 
        let bootstrapEnvVar = 
            match bootstrapServers with
            | null | "" -> ""
            | servers -> $"export BOOTSTRAP_SERVERS=\"{servers}\"\necho 'export BOOTSTRAP_SERVERS=\"{servers}\"' >> /home/ec2-user/.bashrc\n"
        
        $"""#!/bin/bash
dnf update -y

# Install Java 17 (required for Kafka), and netcat for connectivity testing
dnf install -y java-17-amazon-corretto-headless nc

# Download and install Kafka 4.0.0
cd /opt
curl -O https://downloads.apache.org/kafka/4.0.0/kafka_2.13-4.0.0.tgz
tar -xzf kafka_2.13-4.0.0.tgz
mv kafka_2.13-4.0.0 kafka
chown -R ec2-user:ec2-user kafka
rm kafka_2.13-4.0.0.tgz

# Add Kafka bin to PATH for ec2-user
echo 'export PATH=/opt/kafka/bin:$PATH' >> /home/ec2-user/.bashrc
# Set memory for Kafka tools
echo 'export KAFKA_HEAP_OPTS="-Xmx1G -Xms512m"' >> /home/ec2-user/.bashrc

{bootstrapEnvVar}
# Create a directory for Kafka logs
mkdir -p /home/ec2-user/kafka-logs
chown ec2-user:ec2-user /home/ec2-user/kafka-logs

# Create a simple script to help connect to MSK
cat > /home/ec2-user/kafka-msk-helper.sh << 'EOF'
#!/bin/bash
echo "Kafka tools are installed in /opt/kafka/bin"

# Use BOOTSTRAP_SERVERS if set, otherwise show placeholder
SERVERS="${{BOOTSTRAP_SERVERS:-YOUR_MSK_BOOTSTRAP_SERVERS}}"
echo "Bootstrap servers: $SERVERS"
echo ""
echo "Common MSK commands:"
echo ""
echo "1. List topics:"
echo "   kafka-topics.sh --bootstrap-server $SERVERS --list"
echo ""
echo "2. Create a topic:"
echo "   kafka-topics.sh --bootstrap-server $SERVERS --create --topic test-topic --partitions 1 --replication-factor 3"
echo ""
echo "3. Produce messages:"
echo "   kafka-console-producer.sh --bootstrap-server $SERVERS --topic test-topic"
echo ""
echo "4. Consume messages:"
echo "   kafka-console-consumer.sh --bootstrap-server $SERVERS --topic test-topic --from-beginning"
echo ""

if [ -z "$BOOTSTRAP_SERVERS" ]; then
echo "Note: BOOTSTRAP_SERVERS environment variable not set."
echo "Example format: b-1.mycluster.abc123.c2.kafka.eu-north-1.amazonaws.com:9092,b-2.mycluster.abc123.c2.kafka.eu-north-1.amazonaws.com:9092"
fi
EOF

chmod +x /home/ec2-user/kafka-msk-helper.sh
chown ec2-user:ec2-user /home/ec2-user/kafka-msk-helper.sh

# Create a sample client properties file for SSL/SASL if needed
cat > /home/ec2-user/client.properties << 'EOF'
# Uncomment and configure these properties based on your MSK cluster security settings

# For SSL encryption (if cluster uses TLS)
# security.protocol=SSL

# For SASL/SCRAM authentication (if enabled)
# security.protocol=SASL_SSL
# sasl.mechanism=SCRAM-SHA-512
# sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="your-username" password="your-password";

# For IAM authentication (if using IAM access control)
# security.protocol=SASL_SSL
# sasl.mechanism=AWS_MSK_IAM
# sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
# sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
EOF

chown ec2-user:ec2-user /home/ec2-user/client.properties

echo "Setup complete! Run /home/ec2-user/kafka-msk-helper.sh for usage instructions."
"""
    
    // Get the latest Amazon Linux 2023 AMI
    let ami = 
        Ec2.GetAmi.Invoke(Ec2.GetAmiInvokeArgs(
            MostRecent = true,
            Owners = inputList [input "amazon"],
            Filters = inputList [
                input(Ec2.Inputs.GetAmiFilterInputArgs(
                    Name = "name",
                    Values = inputList [input "al2023-ami-*-x86_64"]
                ));
                input(Ec2.Inputs.GetAmiFilterInputArgs(
                    Name = "architecture", 
                    Values = inputList [input "x86_64"]
                ))
            ]
        ))
    
    // Create EC2 instance
    let instance = 
        Ec2.Instance("kafka-ec2-instance",
            Ec2.InstanceArgs(
                Ami = ami.Apply<string>(fun a -> a.Id),
                InstanceType = "t3.medium",
                SubnetId = subnetId,
                VpcSecurityGroupIds = inputList [io securityGroup.Id],
                IamInstanceProfile = instanceProfile.Name,
                UserData = userData,
                Tags = inputMap [
                    "Name", input "Kafka-MSK-Client"
                    "Purpose", input "MSK-Access"
                ]
            ))
    
    // Export useful information
    dict [
        "instanceId", instance.Id :> obj
        "privateIp", instance.PrivateIp :> obj
        "securityGroupId", securityGroup.Id :> obj
        "connectCommand", 
            instance.Id.Apply(fun id -> 
                sprintf "aws ssm start-session --target %s --region %s" id region) :> obj
    ]

         
[<EntryPoint>]
let main _ =
    Deployment.run infra