Luciano's blog

I’ve tried a few different AI coding harnesses recently, and so far I quite like Pi. It feels less opinionated than some of the alternatives, as you’ll read many people say, which makes it easier to adapt to the way I like to work rather than forcing me into a particular workflow.

I’ve spent a couple of days testing Oh My Pi, an AI agent harness built on top of Pi. I found out about it because someone recommended it at work, so I decided to give it a go.

One cool feature they have is Advisor.

The idea is simple: while your main agent is writing code, the advisor agent independently reviews what it’s doing. The feedback is then fed back into the main agent, which can apply the suggestions before continuing.

It’s a bit like having an always-on code reviewer sitting next to your AI agent.

Another thing I like is that you can configure the model the advisor uses. It doesn’t have to use the same model as the main agent, and it probably shouldn’t.

You can actually configure many models for different things: which one to use for something fast, or another for things that require more reasoning? Or what about image processing? And of course, the advisor. Check this simple setup:

providers: 
  webSearch: auto
setupVersion: 1
modelRoles: 
  default: deepseek/deepseek-v4-flash:xhigh
  advisor: deepseek/deepseek-v4-pro:xhigh
  tiny: deepseek/deepseek-v4-flash:xhigh
  vision: openai/chatgpt-image-latest
  slow: openai-codex/gpt-5.5:xhigh
defaultThinkingLevel: auto

I’m experimenting now, building a pipeline to develop a feature from Description to Pull Request. See phases below.

Pipeline Phases

Phase 1 — PRD & Milestone (prd-writer)

The prd-writer agent analyzes the requirements, explores existing code patterns, and produces a structured PRD covering the problem statement, scope, technical requirements, user stories, acceptance criteria, and success metrics. The PRD is saved as a GitHub milestone description.

Tools: read, bash, github_milestone_create

Phase 2 — Code Scout (scout)

The scout agent does fast reconnaissance of the codebase and returns compressed, structured findings: exact file paths with line ranges, key types and interfaces, architecture notes, and a recommended starting point. This context flows into the work splitter so issues are grounded in the real code.

Tools: read, grep, find, ls, bash

Phase 3 — Work Split (work-splitter)

The work-splitter agent receives the PRD and scout findings, then creates granular GitHub issues under the milestone. Each issue is:

• Independently deployable — can ship on its own
• Small — completable in one focused session
• Explicit — clear acceptance criteria and technical notes
• Dependency-aware — blocked issues get depends-on:#N labels

The splitter also outputs a parallelization plan: which issues can run concurrently and which must wait.

Tools: read, bash, github_issue_create, github_issue_list

Phase 4 — Parallel Workers & Testers (worker + tester)

This phase runs in batches. The agent checks feature_status to find unblocked issues, then for each one:

1. Isolate: git_worktree_create spins up a workspace at /tmp/omp-worktrees/issue-N/ on branch feature/issue-N
2. Implement: the worker agent modifies code in the worktree
3. Commit: git_commit_and_push stages, commits, and pushes the branch
4. Test: the tester agent writes unit and integration tests, runs them, and commits again
5. PR: github_pr_create opens a pull request linked to the issue with Closes #N

Multiple unblocked issues run in parallel via the subagent tool’s tasks array. As PRs merge, previously blocked issues become unblocked and the next batch starts.

Worker tools: all default
Tester tools: read, write, edit, bash, github_pr_create, git_commit_and_push

Dependency Tracking

Issues declare dependencies at creation time via dependsOn: [#42]. This adds a depends-on:#42 GitHub label. The feature_status tool checks whether all dependency issues are closed — blocked issues show as 🚫 and agents skip them until their dependencies merge.

Project Structure

omp-feature-factory/
├── .pi/
│   ├── extensions/
│   │   ├── feature-pipeline/      ← GitHub + worktree tools, /build-feature command
│   │   │   ├── index.ts
│   │   │   ├── github-tools.ts
│   │   │   └── worktree-tools.ts
│   │   └── subagent/              ← agent spawning (symlinked from pi)
│   ├── agents/                    ← agent role definitions
│   │   ├── prd-writer.md
│   │   ├── scout.md
│   │   ├── work-splitter.md
│   │   ├── worker.md
│   │   └── tester.md
│   └── prompts/                   ← workflow presets
│       ├── implement.md
│       ├── scout-and-plan.md
│       └── implement-and-review.md
├── .gitignore
└── README.md

It’s a work in progress, and I’ll share more here after some more testing. Oh, I forgot to mention, the code is in my GitHub, it’s called omp-feature-factory.

I’ve added a new command to create and post family tree threads on Bluesky.

It works in a similar way to the timeline command that I already had. To create a timeline, I can run:

cronolm story timeline ROLE_ID

For example, if the ID belongs to Prime Minister of the United Kingdom, the command gets that role from the neo4j database and finds all the related nodes connected with a POSITION_HOLDER edge.

MATCH (n:Entity { id: $personEntityId })-[r]-(y)
WHERE type(r) IN $relationshipTypes
AND (r.start_date IS NOT NULL OR r.end_date IS NOT NULL OR r.point_date IS NOT NULL)
AND r.statement_id IS NOT NULL
RETURN
    n.id AS PersonId,
    coalesce(nullif(n.name, 'Unknown'), n.name_en, n.name_es) AS PersonName,
    n.image_url AS PersonImage,
    n.date_of_birth AS DateOfBirth,
    n.date_of_death AS DateOfDeath,
    type(r) AS RelationshipType,
    r.name AS RelationshipName,
    r.start_date AS StartDate,
    r.end_date AS EndDate,
    r.point_date AS PointDate,
    y.id AS RelatedEntityId,
    labels(y) AS RelatedEntityLabels,
    coalesce(nullif(y.name, 'Unknown'), y.name_en, y.name_es) AS RelatedEntityName,
    y.description_en AS RelatedEntityDescriptionEn,
    y.description_es AS RelatedEntityDescriptionEs,
    y.image_url AS RelatedEntityImage
ORDER BY coalesce(
    date(left(replace(toString(r.start_date), '""', ''), 10)),
    date(left(replace(toString(r.end_date), '""', ''), 10)),
    date(left(replace(replace(replace(replace(r.point_date, '""', ''), '+', ''), '-00-', '-01-'), '-00T', '-01T'), 10))

Once it has all the information, it builds the thread and posts each part slowly to avoid hitting Bluesky’s rate limits.

You can see an example here: Roman emperor timeline on Bluesky.

The new family tree command looks like this:

cronolm story family PERSON_ID

This one gets the person from neo4j and finds their direct family relationships: parents, partners, siblings, and children. It then builds the family tree thread and posts it on Bluesky.

MATCH (ego:Human {id: $id})
/* ---------------- Parents ---------------- */
CALL {
    WITH ego
    CALL {
    WITH ego
    OPTIONAL MATCH (ego)-[r:FATHER|MOTHER]->(p:Human)
    WHERE r.statement_id IS NOT NULL
    RETURN p AS person,
            { relType: type(r), relProps: properties(r), direction: 'ego->parent' } AS relDetail
    UNION
    WITH ego
    OPTIONAL MATCH (p:Human)-[r:CHILD]->(ego)
    WHERE r.statement_id IS NOT NULL
    RETURN p AS person,
            { relType: type(r), relProps: properties(r), direction: 'parent->ego (via CHILD)' } AS relDetail
    }
    WITH person, relDetail WHERE person IS NOT NULL
    WITH person, collect(DISTINCT relDetail) AS relations
    RETURN collect({ person: person, relations: relations }) AS parents
}

/* ---------------- Children ---------------- */
CALL {
    WITH ego
    CALL {
    WITH ego
    OPTIONAL MATCH (ego)-[r:CHILD]->(c:Human)
    WHERE r.statement_id IS NOT NULL
    RETURN c AS person,
            { relType: type(r), relProps: properties(r), direction: 'ego->child' } AS relDetail
    UNION
    WITH ego
    OPTIONAL MATCH (c:Human)-[r:FATHER|MOTHER]->(ego)
    WHERE r.statement_id IS NOT NULL
    RETURN c AS person,
            { relType: type(r), relProps: properties(r), direction: 'child->ego (via FATHER|MOTHER)' } AS relDetail
    }
    WITH person, relDetail WHERE person IS NOT NULL
    WITH person, collect(DISTINCT relDetail) AS relations
    RETURN collect({ person: person, relations: relations }) AS children
}

/* ---------------- Partners ---------------- */
CALL {
    WITH ego
    OPTIONAL MATCH (ego)-[r:SPOUSE|UNMARRIED_PARTNER|PARTNER]-(p:Human)
    WHERE r.statement_id IS NOT NULL
    WITH ego, p AS person, r
    WHERE person IS NOT NULL
    WITH ego, person,
        collect(DISTINCT {
            relType: type(r),
            relProps: properties(r),
            direction: CASE WHEN startNode(r) = ego THEN 'ego->partner' ELSE 'partner->ego' END
        }) AS relations
    RETURN collect({ person: person, relations: relations }) AS partners
}

/* ---------------- Siblings ---------------- */
CALL {
    WITH ego
    OPTIONAL MATCH (ego)-[:FATHER|MOTHER]->(p:Human)
    WITH ego, collect(DISTINCT p) AS egoParents

    CALL {
    WITH ego, egoParents
    OPTIONAL MATCH (ego)-[r:SIBLING]-(sib:Human)
    WHERE r.statement_id IS NOT NULL
    RETURN sib AS person,
            {
                kind: 'direct',
                relType: type(r),
                relProps: properties(r),
                direction: CASE WHEN startNode(r) = ego THEN 'ego->sib' ELSE 'sib->ego' END
            } AS relDetail
    UNION
    WITH ego, egoParents
    OPTIONAL MATCH (ego)-[rE:FATHER|MOTHER]->(par:Human)<-[rS:FATHER|MOTHER]-(sib:Human)
    WHERE rE.statement_id IS NOT NULL AND rS.statement_id IS NOT NULL
    RETURN sib AS person,
            {
                kind: 'viaParent',
                parent: par,
                egoRelType: type(rE),
                egoRelProps: properties(rE),
                sibRelType: type(rS),
                sibRelProps: properties(rS)
            } AS relDetail
    }

    WITH person, relDetail, egoParents
    WHERE person IS NOT NULL AND person <> ego AND NOT person IN egoParents
    WITH person, collect(DISTINCT relDetail) AS relations
    RETURN collect({ person: person, relations: relations }) AS siblings
}

RETURN ego, parents, children, partners, siblings;

You can see an example here: George IV’s family tree thread on Bluesky.

So now creating one of these threads is just a matter of finding the person’s Wikidata ID and running the command.

The next step might be turning this into a Bluesky bot. Someone could mention the Cronologia account with something like:

@cronologia.bsky.social family tree person_name

The bot could search for the person and reply with the generated family tree thread. I haven’t built that part yet, but it feels like the obvious next step.

Fixing duplicate edges was a long process, but it is now mostly done. All messages have been processed, and the code now uses only edges with statement_id. For the final part, I need to remove all edges without statement_id. This should reduce the total number of edges by roughly half, and we are currently at a little over one billion.

I’m not in a hurry to remove these edges because all applications are already ignoring duplicates. I’m trying to find a way to avoid blocking the database during cleanup, since I still want to run import jobs and other queries. I’ll probably write a worker that keeps running and removes edges in batches.

I think the next step toward proper synchronization with wikidata is removing nodes that were deleted from wikidata. I’ll keep you updated!

Fixing the duplicate edges is taking longer than I expected. The dump file processing finished within the estimated time, but the edge worker(s) are taking a long time to process the new edges. There are currently a bit more than 800k messages, at around 500 edges per message. I’ve changed the number of workers from 5 to 1, as I could see many locks slowing down the process.

When running this query:

SHOW TRANSACTIONS
YIELD transactionId, elapsedTime, status, waitTime, activeLockCount, currentQuery
RETURN
  transactionId,
  elapsedTime,
  elapsedTime.seconds AS elapsedSeconds,
  status,
  waitTime,
  waitTime.seconds AS waitSeconds,
  activeLockCount
ORDER BY elapsedSeconds DESC;

I see transactions holding locks that other transactions are waiting for.

I don’t see blocked transactions now, but this is going to take a long time to finish. I’ll need to think about how I can improve this process. I wasn’t counting on different messages trying to modify the same nodes.

Once this is done, I’ll start deleting all edges without statement_id. That would reduce the number of edges by half. We’re currently at 600M edges, so that would be a big improvement.

Back to the Wikidata dump file processing: the edge de-duplication script ran and got rid of a bit more than 1 million edges. But while navigating some timelines, I noticed I still had plenty of duplicates. I kept seeing the same person in the same role in the same year (like Luis X of France as #7 and #8).

Checking that node and its edges in neo4j, I found I had 2 POSITION_HELD edges with slightly different dates: same year and month, but different start and end days. Looking at the same node in Wikidata, I could only see one statement.

Quick Wikidata Data Structure

flowchart TB
	A[Item: Luis X of France]
	B[Statement: P39 = King of France]
	Q1[Qualifier property: P580 start time]
	Q1V[Qualifier value: 1314-11-29]
	Q2[Qualifier property: P582 end time]
	Q2V[Qualifier value: 1316-06-05]
	G[Reference]
	H[statement_id used as unique key]

	A -->|has statement| B
	B -->|has qualifier| Q1
	Q1 -->|value| Q1V
	B -->|has qualifier| Q2
	Q2 -->|value| Q2V
	B -->|supported by| G
	B -->|mapped to| H

Basically, my edge de-duplication strategy was pretty bad. I was only looking at property type + start date. That allows, for example, the same person to have 2 terms as president (same POSITION_HELD, different start dates), which is valid. But if someone corrected a date in Wikidata before a second dump import, I’d end up with 2 POSITION_HELD edges for the same person and position, just with different dates, like in the Luis X of France example above.

So I came up with a plan: use the Wikidata statement ID as the de-duplication key. That way I’ll always update the same edge for the same statement.

This requires a phased process:

Part 1

A temporary first step to stop even more duplication in my timeline and family tree websites: change the services to ignore any edge where statement_id is null. That way I can start creating brand-new edges with statement_id and be sure they won’t be duplicated, while I work on the next phase.

Part 2

Modify scripts and workers to save statement_id on edges, and use that for de-duplication instead of start date.

Part 3

Deploy Part 1 and Part 2 changes and process the massive Wikidata dump file again.

Part 4

Once the new dump file is processed, modify all services again to show only edges with a statement_id property. Once I’m happy with the results, proceed to Part 5.

Part 5

Remove all edges without a statement_id property, since those were created with the old de-duplication strategy and are likely duplicated.

I’m currently on Part 3, processed up to line 9,672,263. Still plenty of lines to go to get to 120M+. I’ll share more updates as I go!

The app processing the Wikidata dump file finished today! It took less than a week, even with pauses in the middle to correct some issues. I still need to run some cleanup (duplicated edges or relationships) and wait for all messages to be processed.

Latest numbers:

• 120,266,937 lines processed
• 86,371,957 nodes
• 259,702,039 edges
• 14,525 messages behind in the nodes topic
• 3,913,389 messages behind in the edges topic
• I’ve used 7% of the new disk (around 127 GB)

I will now work on a dedup script to remove duplicated relationships, and then wait for all messages to be processed before sharing more updates.

Okay, not much to update. I had a couple of problems with the script: I noticed I was creating duplicate relationships (I was comparing the wrong thing in the merge). So I had to stop the import script and resume it after making changes, but that means I now have to run some cleanup scripts to remove duplicates. It is annoying because the database is big now, so I need to find a good way to do it.

Some updated numbers:

• Currently on line 47553093 of the wikidata dump file
• 53.5 million nodes
• 218 million relationships
• 7k messages behind in the nodes topic
• 600k messages behind in the relationships topic
• I’ve used 5% of the new disk (around 83.5 GB)

Forgot to mention: I keep receiving alerts from Netdata about CPU usage on the box running the script. It seems that streaming BZ2 line by line is quite CPU-intensive. I might need to run this from a different box as well, as it is currently running on the same box as Neo4j, which is not ideal.

The disk upgrade didn’t go as expected (as usual). I used dd to clone the disk, but when I swapped the disks, the new one wasn’t bootable. For some reason, the BIOS recognized the new disk randomly, so every other boot it would see the disk and sometimes it wouldn’t. After some reading, this might be because this is such an old PC/BIOS that it doesn’t support it. I might try to update the BIOS at some point, but for now, I swapped the disks back and plugged the new one as an external drive.

Next step was to change neo4j to use the new disk for data storage, in my case neo4j is running on a docker container, so I just had to change the volume mapping to point to the new disk. After that I just started the workers again and they are now writing to the new disk.

I’ll post another update in a couple of days to see how the new disk is performing.

I’ll keep talking about the Wikidata dump and my setup. Overnight, we went from 16 million nodes and 56 million relationships (See yesterday’s post) to 38.5 million nodes and 84.5 million relationships. I’m watching disk space closely; I’ve used 25%, so I have 144 GB left.

Wikidata has an estimated 110 million Q entities. It’ll take around 4-5 days to process nodes. There are way more relationships, and the queue is already behind by 1.2 million messages with 5 workers. Entity upsert is quicker; the worker is lagging by 3k messages with 1 worker. Each message in our topic contains a batch of entities/relationships, so the actual number of entities/relationships is higher than the message count. I’m reusing the architecture I had set up for the import while using the API, so currently it goes something like this:

flowchart TD
    consoleApp("Console app")
    kafkaEntity("Kafka Topic: entities to save")
    kafkaRelationship("Kafka Topic: relationships to save")
    relationshipWorkerOne("Worker 1: relationship upsert")
    relationshipWorkerTwo("Worker 2: relationship upsert")
    relationshipWorkerThree("Worker 3: relationship upsert")
    relationshipWorkerFour("Worker 4: relationship upsert")
    relationshipWorkerFive("Worker 5: relationship upsert")
    entityWorkerOne("Worker 1: entity upsert")
    neo4jDatabase("Neo4j Database")
    
    consoleApp --> kafkaEntity
    consoleApp --> kafkaRelationship
    kafkaEntity --> entityWorkerOne
    kafkaRelationship --> relationshipWorkerOne
    kafkaRelationship --> relationshipWorkerTwo
    kafkaRelationship --> relationshipWorkerThree
    kafkaRelationship --> relationshipWorkerFour
    kafkaRelationship --> relationshipWorkerFive
    entityWorkerOne --> neo4jDatabase
    relationshipWorkerOne --> neo4jDatabase
    relationshipWorkerTwo --> neo4jDatabase
    relationshipWorkerThree --> neo4jDatabase
    relationshipWorkerFour --> neo4jDatabase
    relationshipWorkerFive --> neo4jDatabase

I ended up buying a 2TB SSD to replace the current one. I don’t want to rebuild that machine, so I’m going to follow the advice of you know who and use a tool called dd to clone the current disk to the new one, and then swap the disks. I’ll have to stop the workers while I do this, but the console app can keep running and sending messages to our topics.

Hopefully, my next update will be that the disk upgrade went well and we are back to processing messages. I also need to start thinking about how to optimize the import process if I plan to do this every week (that’s how often Wikidata releases new dump files).

Wikidata dumps contain all the info you need. They are huge files compressed in bz2 format. They recommend using JSON dumps, but they also have XML dumps. I’ve downloaded the latest JSON one, it’s 94.2GB, and the good thing is that each line is a JSON object, so it’s easy to stream and process.

I’ve written a script to process it, and it’s currently running. It has processed 4 million lines. Lots of things are ignored in my logic, but it has already added more than a million nodes to my neo4j graph DB, currently sitting at 16 million nodes and 56 million relationships.

Now that I’m using the dump file instead of API calls, this is going very fast, which made me realize my setup might not be able to handle all this data. Neo4j is currently running on an old HP ProDesk Mini with 16GB of RAM and a 240GB SSD (that’s the only thing that machine is doing), and it’s already down to 166GB left. I think I’ll be ordering a 1TB SSD soon to replace the current one.

This graph DB is the backbone of a couple of pet projects, links at the bottom of this post. Bear in mind Wikidata is crowdsourced and not always accurate. I jump in now and then on Wikidata and add/edit some info.

There is the Timelines Website, currently with just a handful of timelines, and the Family tree website, with some recommended collections, but it also allows you to search for any person (as long as they are in the graph) and see their family tree. Some people have a summary and video generated with AI based on the node relationships with other people, places, and events.

So, I’ve been crawling Wikidata entities and relationships to build a local graph database, but I’ve been doing this through the Wikidata API, which is rate-limited per IP address. Turns out I’ve been an idiot and didn’t know that Wikidata provides dumps of all the data that can be easily downloaded.

I’m checking the format now, but I think it’ll have everything I need. I’ll post something later once I’ve learned a bit more about it.

Here’s more info: Wikidata Database Download

Ok, I ditched the old Angular CSR website for this new static site generated website. It’s so easy now with AI that it took me maybe a couple of hours to get it all live with GitHub Copilot. The prompt was something like:

let’s convert this Angular CSR into an SSG website. We should read the posts from a posts folder in the repo instead of our Firestore database. We should build the pages on every merge into main and deploy to GitHub Pages instead of Firebase. And let’s replace Google Analytics with Umami.is

From there it asked me a couple of questions about what to use and we were done.

There were a couple of follow-ups with style fixes and a problem with the base URL (when using a GitHub Pages URL or a custom domain), but it pretty much worked out of the box. Check the README file generated below.

Stack

• Astro — static site generator
• GitHub Actions — builds and deploys on every push to main
• GitHub Pages — hosting, served via custom domain
• Umami — privacy-focused analytics

Structure

site/                  # Astro project
├── src/
│   ├── content/
│   │   ├── posts/     # Published blog posts (.md)
│   │   └── drafts/    # Unpublished drafts (not built)
│   ├── layouts/       # BaseLayout and PostLayout
│   └── pages/         # index, /post/[slug], 404
└── public/            # Static assets (images, CSS, favicon)

Writing a post

1. Create site/src/content/posts/<slug>.md with the required frontmatter:

   ---
   title: Your Post Title
   publishedOn: YYYY-MM-DD
   ---

2. Push to main — GitHub Actions builds and deploys automatically.

To draft a post without publishing it, place the file in src/content/drafts/ instead.

Local development

cd site
npm install
npm run dev      # http://localhost:4321
npm run build    # production build → dist/
npm run preview  # preview production build

When developing a service on Azure Service Fabric you can work on your local cluster, it works quite well. You can decide between 1 or 5 node local cluster (check how to prepare your development environment from Microsoft docs). But if you are part of a team, you probably want everyone on the team to access the cluster. In our team, we have deployed a cluster in Azure for this, we called it dev environment, but then you have the QA team that needs to test, but this environment is too volatile for that, one minute is fine and the next is gone after we deployed something that broke the application. We needed an staging environment but we didn’t want to deploy another cluster (you need to pay for it and all that).

Deploying same application type multiple times on the same cluster

Follow these 2 steps to deploy the same application type multiple times on the same cluster, to act like multiple environments.

Parametrise any service URLs

If you have multiple services talking to each other, you need to parametrise any service URLs into your environment parameters file, in our dev.xml application parameters we’ll have something like this

<Application 
    xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    Name="fabric:/MyApp.dev" 
    xmlns="http://schemas.microsoft.com/2011/01/fabric">
  <Parameters>
     <Parameter Name="MyApiServiceOneUri" Value="fabric:/MyApp.dev/ServiceOne" />
     <Parameter Name="MytActorUri" Value="fabric:/MyApp.dev/ActorOne" />
  </Parameters>
</Application>

and below is our staging.xml parameters file

<Application 
    xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    Name="fabric:/MyApp.staging" 
    xmlns="http://schemas.microsoft.com/2011/01/fabric">
  <Parameters>
     <Parameter Name="MyApiServiceOneUri" Value="fabric:/MyApp.staging/ServiceOne" />
     <Parameter Name="MytActorUri" Value="fabric:/MyApp.staging/ActorOne" />
  </Parameters>
</Application>

The important thing to notice from both parameters files above is the name of your application, we added a suffix with the environment name, in this case .dev and .staging, and used the right URIs for same app services and actors.

Use different ports

If you are specifying ports explicitly in your service endpoints, then you need to be sure to use different ports for your different environments. We could add another parameter in our ‘dev.xml’ and ‘staging.xml’ parameters files.

<Application 
    xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    Name="fabric:/MyApp.staging" 
    xmlns="http://schemas.microsoft.com/2011/01/fabric">
  <Parameters>
     <Parameter Name="MyApiServiceOneUri" Value="fabric:/MyApp.staging/ServiceOne" />
     <Parameter Name="MyApiServiceOnePort" Value="9051" />
     <Parameter Name="MytActorUri" Value="fabric:/MyApp.staging/ActorOne" />
  </Parameters>
</Application>

When picking static ports remember this:

“By design static ports should not overlap with application port range specified in the ClusterManifest. If you specify a static port, assign it outside of application port range, otherwise it will result in port conflicts” (Specify resources in a service manifest).

To use the port from your parameters file in your service, you can add a ResourceOverride section in the ApplicationManifest as indicated in the Service Fabric endpoint override docs. Locate the ServiceManifestImport of the service you want to override the endpoint port and add the ResourceOverride section.

<ResourceOverrides>
   <Endpoints>
      <Endpoint Name="ServiceEndpoint" Port="[MyApiServiceOnePort]" />
   </Endpoints>
</ResourceOverrides>

Also, remember to declare the parameters at the top of the ApplicationManifest file.

<Parameters>
    <Parameter Name="MyApiServiceOnePort" DefaultValue="9999" />
</Parameters>

And that’s it, ready to deploy 🚀

Seems to be pretty common, check below what worked for me, found it in this WSL GitHub issue comment

From cmd execute

netsh winsock reset
netsh int ip reset all
netsh winhttp reset proxy
ipconfig /flushdns

From WSL execute

sudo bash -c 'echo "nameserver 8.8.8.8" > /etc/resolv.conf'
sudo bash -c 'echo "nameserver 8.8.4.4" >> /etc/resolv.conf'

It’s been a while since we deployed our first project to a Service Fabric cluster at work.

I’d like to share what we’ve done and the bits we’ve learned in the process.

The first thing…

Deploying the cluster

We had to create a cluster and we had to be able to recreate exactly the same cluster quickly and with minimum effort, using Azure Portal interface was not an option, an ARM template was the way to go.

We based our cluster template on one of Ivan Gavryliuk’s templates, I recommend his Pluralsight course Using Azure Service Fabric in Production. Ivan’s powershell script will create the resource group (if needed), key vault, import certificate and then apply the ARM template.

You can find more template examples on Azure Samples repository on GitHub.

Your template will change based on your needs, like operating system, VM size, load balancer, durability tier, et cetera. I’ll describe the changes we’ve done in the template and powershell scripts.

Load balancer

In our case we use two Azure load balancers, a public load balancer for cluster management and a private one with rules to expose the services we need to access from our private virtual network, we use an nginx load balancer as a reverse proxy for the services that need to be accessed from internet.

Check the load balancers configuration on this gists.

Docker images cleanup

After a couple of months of pushing upgrades to a container based service in our staging cluster we ran into disk space problems, if you are going to run containers in your cluster and don’t want to login to each node to prune your docker images you will need a couple of settings.

• PruneContainerImages if set to true will:

Remove unused application container images from nodes. When an ApplicationType is unregistered from the Service Fabric cluster, the container images that were used by this application will be removed on nodes where it was downloaded by Service Fabric. The pruning runs every hour, so it may take up to one hour (plus time to prune the image) for images to be removed from the cluster. Service Fabric will never download or remove images not related to an application. Unrelated images that were downloaded manually or otherwise must be removed explicitly.

• ContainerImagesToSkip will prevent the deletion of the listed images

More info about this settings can be found in the Service Fabric hosting settings docs and the Service Fabric containers getting started guide. This is how the settings look like on the template.

{
    "fabricSettings": [
        {
            "name": "Hosting",
            "parameters": [
                {
                    "name": "PruneContainerImages",
                    "value": "True"
                },
                {
                    "name": "ContainerImagesToSkip",
                    "value": "microsoft/windowsservercore|microsoft/nanoserver"
                }
            ]
        }
    ]
}

Service instance count

Another problem we ran into was changing the instance count when upgrading an application, suppose you deployed your application for the first time on your cluster with InstanceCount="2" and later you realise you actually need one instance of your service running on every node, so you change it to InstanceCount="-1" and deploy just to receive something like:

Start-ServiceFabricApplicationUpgrade : Default service descriptions can not be modified as part of upgrade.
To allow it, set EnableDefaultServicesUpgrade to true.

And that’s all you need to do actually, just add it to the cluster settings on your template as below.

{
    "fabricSettings": [
        {
            "name": "ClusterManager",
            "parameters": [
                {
                    "name": "EnableDefaultServicesUpgrade",
                    "value": "true"
                }
            ]
        }
    ]
}

You can read a bit more about the behaviour of changing default services during application upgrades in the Service Fabric application upgrade docs.

Security

There is a good article from Microsoft describing all the cluster security scenarios

In our case, we use:

• Wildcard certificate for server identity and SSL encryption of http communication
• Self signed client certificates for users and Azure DevOps Pipelines.

The powershell script to prepare for cluster deployment looks really similar to Ivan’s script. Instead of uploading a self signed certificate to the Vault we upload our wildcard certificate and create 3 self signed certificates for client access:

• Read only access
• Admin access
• Another admin access for DevOps Pipelines

. "$PSScriptRoot\..\Common.ps1"

# The declare come variables
$ResourceGroupName = "everything-sfcluster"
$Location = "North Europe"
$KeyVaultName = "cluster-vault"

# Check that you're logged in to Azure 
# before running anything at all, the call will
# exit the script if you're not
CheckLoggedIn

# Ensure resource group we are deploying to exists.
EnsureResourceGroup $ResourceGroupName $Location

# Ensure that the Key Vault resource exists.
$keyVault = EnsureKeyVault $KeyVaultName $ResourceGroupName $Location

# Upload buyagift wildcard certificate
$cert = UploadCertificate $KeyVaultName "certName" $PSScriptRoot "certPassword"

# Create three self signed certificates and return thumb to use on cluster template
$readOnlyThumb, $adminThumb, $devOpsThumb = EnsureSelfSignedClientCertificates $PSScriptRoot

To use our wildcard certificate we also had to change the service fabric explore url and create a DNS A record pointing to the public IP DNS name associated to our public load balancer, our A record looks like this service-fabric-explorer.our-domain.com => xxxxxxx.northeurope.cloudapp.azure.com

Check below the relevant properties of the service fabric resource on the ARM Template

"properties": {
                "certificateCommonNames": {
                    "commonNames": [
                    {
                        "certificateCommonName": "*.our-domain.com", # remember to add * if wildcard certificate
                        "certificateIssuerThumbprint": ""
                    }
                    ],
                    "x509StoreName": "My"
                },
                "clientCertificateThumbprints": [
                    {
                        "isAdmin": false,
                        "certificateThumbprint": "[parameters('readOnlyThumb')]" # cert thumb from previous powershell script
                    },
                    {
                        "isAdmin": true,
                        "certificateThumbprint": "[parameters('adminThumb')]" # cert thumb from previous powershell script
                    },
                    {
                        "isAdmin": false,
                        "certificateThumbprint": "[parameters('devOpsThumb')]" # cert thumb from previous powershell script
                    }
                ],
                "managementEndpoint": "[concat('https://service-fabric-explorer.our-domain.com:',variables('fabricHttpGatewayPort'))]",
            }

And the following on the Virtual Machine Scale Set resource

"osProfile": {
                        "adminPassword": "[parameters('adminPassword')]",
                        "adminUsername": "[parameters('adminUsername')]",
                        "computernamePrefix": "[parameters('vmNodeType0Name')]",
                        "secrets": [
                            {
                                "sourceVault": {
                                    "id": "[parameters('sourceVaultValue')]"
                                },
                                "vaultCertificates": [
                                    {
                                        "certificateStore": "My",
                                        "certificateUrl": "[parameters('certificateUrlValue')]" # secretId from cert uploaded to vault on previous powershell script.
                                    }
                                ]
                            }
                        ]
                    },

One important thing to remember is to replace your server certificate before it expires, if the certificate expires you’ll lose connection to the cluster (Service Fabric Explorer will stop working and you would not be able to deploy anything). It happened to the cluster of another team in the company and you’ll get a “Upgrade service unreachable” message on Azure Portal, the list of nodes and applications will be empty.

The message had a link to a Cluster not reachable document on github describing possible causes and mitigations. In our case it was the certificate.

If already happened then head on to fix expired cluster certificate steps document by Microsoft.

Follow up

I think it is enough for now, will continue on a follow up post soon.

I can’t remember where I got this one, but here it is:

[^\x00-\x7f]

Just be sure to search using regular expressions.

Turns out what I call permalink is actually called Slug, permalink is the full URL and slug is “the part of a URL that identifies a page in human-readable keywords” you can read more about on Wikipedia’s Clean URL / Slug article and Wikipedia’s Permalink article.

Below is the function I used to Slugify blog titles, got it from Matt Hagemann, this is the link to the gist.

function slugify(string) {
  const a = 'àáâäæãåāăąçćčđďèéêëēėęěğǵḧîïíīįìłḿñńǹňôöòóœøōõṕŕřßśšşșťțûüùúūǘůűųẃẍÿýžźż·/_,:;'
  const b = 'aaaaaaaaaacccddeeeeeeeegghiiiiiilmnnnnooooooooprrsssssttuuuuuuuuuwxyyzzz------'
  const p = new RegExp(a.split('').join('|'), 'g')

  return string.toString().toLowerCase()
    .replace(/\s+/g, '-') // Replace spaces with -
    .replace(p, c => b.charAt(a.indexOf(c))) // Replace special characters
    .replace(/&/g, '-and-') // Replace & with 'and'
    .replace(/[^\w\-]+/g, '') // Remove all non-word characters
    .replace(/\-\-+/g, '-') // Replace multiple - with single -
    .replace(/^-+/, '') // Trim - from start of text
    .replace(/-+$/, '') // Trim - from end of text
}

First buggy version of the blog is out!