Rob's Blog

AI Engineering Fluency: tracking your AI coding habits

2026-05-15T00:00:00+00:00

I’ve been building an extension that gives you insights into your AI usage when coding — how many tokens you consume, which models you prefer, how you structure your prompts, and what your environmental footprint looks like. It’s called AI Engineering Fluency and I recorded a walkthrough video showing what it does.

What it does

After installing the extension, it reads local log files from your AI editors — VS Code, Copilot CLI, Claude Code, Gemini CLI, Cursor, Crush, Mistral, and more. Everything stays on disk. Nothing leaves your machine unless you opt in to cloud sync.

The extension crawls those session logs and figures out what you’ve been doing with AI on that machine. It then groups the data to give you an overview right in the status bar: today’s token usage and a 30-day rolling total.

Token tracking and usage charts

Clicking the status bar icon opens a detailed panel showing:

Total tokens used today vs. last 30 days
Number of distinct working sessions
Interactions per session and tokens per session
Breakdown by editor (VS Code, Copilot CLI, Claude Code, etc.)
Breakdown by model (Claude Opus, Claude Sonnet, Mistral, GPT, etc.)
Estimated cost over time based on token usage

On the day I recorded the walkthrough, I had already hit 23 distinct working sessions and gone through over 120 million tokens across different tools. You can view charts per day, per week, or per month. I noticed my own usage has been following a hockey stick pattern since November — it just keeps going up. The charts also let you slice by model, editor, or even repository.

The dashboard also breaks activity down by mode. In my case, a lot of sessions happen in CLI tools and agent mode, but I still use ask mode for direct questions. That distinction matters because not every AI interaction is the same kind of work.

Multi-model usage insights

One of the more interesting findings: 18% of my sessions in the last 30 days used more than one model. The maximum was five different models in a single session. I apparently switch between heavier models for planning and lighter ones for implementation — something I wasn’t fully aware of until I saw the data.

Environmental impact

Since we have all this token data, the extension also estimates your environmental footprint. CO2 consumption, water usage, and equivalent metrics like “how far could you drive a car” or “how many smartphone charges is that.” These numbers are estimates based on public data for model/provider energy and water usage, so I treat them as directional rather than exact. The eye-opener for me was the estimated water consumption and the number of trees needed to offset my CO2 usage. There’s compute in a data center somewhere being consumed every time you send a prompt — this makes that visible.

The Fluency Score

The real payoff is what we call the Fluency Score. It maps your usage patterns against six aspects of AI engineering:

Prompt Engineering — Do you use multi-turn sessions? Do you mix ask mode with agent mode?
Context Engineering — Are you explicitly adding context to conversations?
Agentic — Do you use agent mode and autonomous features?
Tool Usage — Are you using built-in tools and MCP servers?
Customization — Have you configured model selection and repo-level settings?
Workflow Integration — Do you use AI regularly across different modes?

Each aspect has four stages (Skeptic, Explorer, Collaborator, Strategist). The score highlights where you’re strong and where you have room to grow — with links to documentation and short videos so you can learn about features you might not know exist.

For example, my prompt engineering is at stage four because I do multi-turn sessions with an average of 4.8 exchanges. But my context engineering could use work — I could be more explicit about adding files and symbols to conversations. The extension also tracks which files were pulled into conversations and whether I added that context myself or the editor did it for me, which makes the context engineering score feel less abstract.

Supported editors

The extension supports a wide range of tools:

VS Code (Stable, Insiders, Exploration) + GitHub Copilot
VSCodium, Cursor, Windsurf, Trae, Kiro
JetBrains IDEs + GitHub Copilot
GitHub Copilot CLI
Claude Code (Anthropic)
Gemini CLI (Google)
Mistral Vibe
OpenCode, Crush, Continue
Visual Studio 2022+

And has viewers for:

Visual Studio Code (and thus Codium based editors like Cursor, Windsurf, Trae, Kiro)
Visual Studio 2022+
JetBrains IDEs

And even a CLI version for quick stats in the terminal. If you use any of these tools, you can get insights into your AI usage right away.

Getting started

Install it from the VS Code Marketplace or the JetBrains Marketplace. There’s also a CLI version available via npm:

npx @rajbos/ai-engineering-fluency stats

It’s free, open source, and all your data stays local. The repo is at github.com/rajbos/ai-engineering-fluency — contributions and feature requests are welcome through the repo as well.

The AI subsidy era is ending: time to talk business value

2026-05-15T00:00:00+00:00

GitHub Copilot moves to usage-based billing on June 1. We have already seen customers that will see their AI spend on Copilot go to 2x or 3x (median), with some even going 8x! And this is happening across all model hosters and vendors: Anthropic has been pushing people onto their Max tiers ($200/month) and metered Claude Code usage. Gemini, Cursor, Windsurf, and the rest are doing the same math and arriving at the same answer. The era where AI was heavily subsidised by vendors trying to grab market share is ending, and now we get to pay the actual price of hosting all those GPUs.

Some companies are about to wake up and smell the roses.

Some reference stories of the last days:

Sander Trijssenaar wrote a solid breakdown of the new GitHub Copilot App combined with token billing and what that means for spend.
Hidde de Smet did the per-token math across every major provider.
State of Brand called the whole subsidy model a ticking time bomb.
AJ Enns put together a token economy playbook for keeping spend under control once the meter is running.
I see the same thing happening in the Copilot Discord.

Read all four references if you haven’t — they cover the numbers better than I will here.

What I want to talk about is what comes after the panic.

The trough we’re sliding into

If you look at Gartner Hype Cycle, this is the bit right after “Peak of Inflated Expectations” — the slide into the Trough of Disillusionment. We’re sliding into it right now. Here is last years Hype Cycle for reference, althoug this is on AI overall:

The negative sentiment on pricing is everywhere on LinkedIn, Mastodon, Reddit, and every dev community in between: individual users hopping between editors and CLIs trying to find the next thing that runs AI for free or close to it, and companies scrambling to get costs under control by limiting AI use instead of enabling their engineers.

That second part is the wrong path. Restricting AI to control the bill saves a few thousand euros a month and leaves a multiple of that in business value on the table. I see it in the numbers all the time.

What the numbers actually look like

At Xebia we’ve built a Copilot billing dashboard on top of the standard GitHub data, with extra logic that flags things like the Business → Enterprise upgrade math during the current promo period. The pattern that keeps showing up: a small group of heavy users carries most of the cost, and a much larger group is barely touching their allotment.

One example from a real customer: four users were responsible for roughly 50% of the company-wide bill. The rest of the org has AI Credits sitting unused every month. That’s not a cost problem — that’s a massive opportunity problem. Those underused seats are engineers who could be shipping faster, fixing tech debt, or building MVPs to test new ideas, and instead they’re using maybe a third of what they’re already paying for. Some companies approach giving out licenses the wrong way. I have shared my views on this at conferences left and right:

2024 - GitHub Universe: Lessons learned from enabling thousands of developers on GitHub Copilot

The AI Credits pooling model under the new billing for GitHub Copilot helps with this on paper — heavy users draw from the shared pool, light users effectively subsidise them. But it also masks the real story: a lot of seats are paid for and not used. That’s the gap I want teams to close, with data-driven insights and proactive management.

My own usage hockey stick

I’ve been tracking my own AI usage in detail for a while with this VS Code Extension I built. Here’s what it what my hockey stick growth looks like:

Near zero through October 2025. A small bump in November and December. Then it climbs through Q1 2026 and goes vertical in March and April. April hit ~530 sessions and ~1.4B tokens. May is projected around 1.8B tokens. And to be clear: I’m not running /fleet, Squad, or any other multi-agent army. This is one person, working interactively most of the time.

I vividly remember end of March where I hit my first 100 Million tokens in 30 days. I thought “oh, that’s a lot!”. Then I hit 1 Billion tokens just 3 weeks later. Let that sync in: from 100M to 1B in three weeks.

What changed? Two things. First, agent mode genuinely became my default for non-trivial work after December. Second, I got early access to the GitHub Copilot App that went into technical preview in May 2026, and it is by design a more agentic experience which of course consumes more tokens: parallel sessions, longer autonomous runs, Agent Merge handling CI failures, merge conflicts, and security review comments, all on its own. I wrote about that release the day it dropped here.

So yes, my AI bill will go up significantly. The question I want to talk about is whether that’s worth it and how I look at it.

A concrete case: $57 in tokens for a JetBrains extension

Here’s a session from my own custom dashboard that shows costs per session:

One session. A couple of hours of my time. $57.05 in AI spend across Claude Opus 4.7, Sonnet 4.6, Haiku 4.5, and GPT-5 mini. Output: a working JetBrains extension — rajbos/ai-engineering-fluency — that visualises in-depth information about your AI usage: sessions, tokens, chat turns, models used, and even an AI fluency score. Same kind of insights I’d been building for VS Code and Visual Studio, now ported to the JetBrains ecosystem.

If I had to build that from scratch, learning the JetBrains plugin SDK along the way, I’d estimate two weeks of work. Maybe more. Instead it cost me a couple of hours and $57 in tokens.

Was it worth it? An engineer’s hours for two weeks versus a few hours plus $57 — that’s not even a close call. The business case writes itself.

That’s the conversation we need to be having about every meaningful AI session: not “how much did the tokens cost” in isolation, but “what did we get for it, and what would the alternative have cost”. And of course, just raw cost and “did you ship anything” is not the right way to look at it either. The real question is “what was the business value of the session overall, and was that worth it”. Tracking these sessions to something sensible or tangible is hard! When I start research in session one, and then hop over to another tool, tagging those sessions together is unrealistic. But we can get close enough to have a meaningful conversation about the value of the work done in those sessions, and that’s what we should be doing. Also the other side is true: shipping more crap and bugs into production faster is also not a good business case. The value conversation has to be about the overall impact of the session in the long term, not just the singular output or the cost.

If a beefy session shaves off 2ms in a request on a system that does millions a day, that’s a huge business win. If it produces a feature that customers love and drives more revenue, that’s a huge business win. If it refactors a piece of code that was causing 10% of the bugs in production, that’s a huge business win. If it just produces a few more lines of code that don’t actually move the needle, then maybe it’s not worth it.

If a session teaches a new developer on the team how to do something they didn’t know before, that’s a huge business win. If it helps an experienced developer get unstuck on a problem that was blocking them for days, that’s a huge business win. If it just produces a few more lines of code that don’t actually move the needle, then maybe it’s not worth it.

Foundation first, then speed

Here’s where it ties back to something I’ve been saying for over a year. Back in April 2025 I wrote GitHub Copilot — Change the Narrative, arguing that the productivity framing for AI was the wrong one. The real value is that AI gives engineers time to put a sturdy DevOps foundation in place: automated pipelines, testing you actually trust, monitoring, the more-eyes principle, everything-as-code.

That argument is more relevant now than it was then. With usage-based billing, every team has to justify what their AI spend produces. And all of a sudden we will have a (justified) conversation again about the AI cost for that team. The teams that can answer “we used the time to get our pipelines, tests, and monitoring in shape, and now we ship features in days instead of weeks with confidence they won’t break production” are going to look very different from the teams that just shipped more features faster on top of a wobbly foundation.

Enabling the teams to get the most out of their AI usage is the job of engineering leadership. They need to give the team the time to absorb the new tools, tailor the AI setup to their stck and way of working, and then some. So many businesses make the mistake of handing out licenses and hope that they get 10x engineers. That’s not how any of this works!

Instead, spend the early time with the team on the foundation of their SDCLC setup. Get your confidence up to the point where you can actually go faster. Then go even faster.

Once that foundation is solid, tackling the technical debt backlog stops being a luxury item. AI sessions are very good at the kind of bounded, well-tested refactor work that usually sits at the bottom of the backlog forever. The cost of a refactor session is small. The cost of carrying that debt for another year, in slower delivery and more bugs, is much larger.

Then the real prize: speed of opportunity

Once foundation and tech debt are in a decent state, the next conversation is the one that actually matters to the business: cost of opportunity.

A team that’s fluent with AI in their SDLC, on top of a solid DevOps foundation, can turn on a dime. New feature idea on Monday, MVP in the user’s hands by Friday. That speed of response is the business value. Not “engineers type faster”. Not “we shipped 30% more story points”. The ability to react to a market signal or a customer request in days instead of months is what AI in the SDLC actually unlocks, and it’s what should be in the business case.

So the conversation with the business becomes: here’s the change, here’s the AI spend (training engineers, tokens, compute) it took to get it out, is the value worth that cost? My honest expectation is that the answer is an immediate yes most of the time. For the cases where it isn’t obvious, spin up an AI session to do the research and build a quick MVP — that’s trivial these days, and it’s usually enough to make a real business case from.

Look at the current spent to avoid surprises, then shift the conversation to business value

Here are few concrete things, in roughly the order I’d tackle them:

Get visibility into your usage before June 1. The GitHub preview bill is the bare minimum. If you want more, the Xebia Copilot billing dashboard is one option (we even share opportunities to upgrade Business licenses to Enterprise licenses based on the data), or look at the token economy playbook as it is another good starting point.
Look at who is and isn’t using their allotment. The underusers are not a saving — they’re an opportunity. Find out why they’re not using it (no training, no use case, fear of “doing it wrong”) and fix that. Talk to the engineers and see what holds them back. I’ve seen teams with editors out of 2023, teams on hosted or locked down editors that did not allow them to install new tools and extension, or teams that kept on saying “the AI does not get our stack”. All of those are solvable problems, and the solution is not “just don’t use AI then”. It’s “let’s fix the problem so you can use AI and be more effective”. Call us if you need help in this field!
Run the upgrade math during the promo period. For some users, moving from Business to Enterprise pays for itself thanks to the bigger included allotment, and the surplus credits help cover the heavy users without overage.
Stop measuring AI by “engineers shipping faster”. Start measuring it by what you spent the freed time on: foundation work, tech debt, MVPs that became real features, deep research that lead to improvements. I am a fan of the book Frictionless from Abi Noda and Nicole Forsgren, which has a great framework for measuring the value of work in terms of “friction removed”, as well as a way to explain this in business value. I think that is a much better way to look at the value of AI sessions than “how many lines of code did we ship”.
Have the business value conversation per change, not per seat. Tie AI spend to the feature it produced, then ask the business if it was worth it.

If you want to talk about any of this, reach out to me on LinkedIn.

The subsidy era ending isn’t a bad thing. It forces the conversation we should have been having from day one: what is this actually worth?

GitHub Copilot App is now in Technical Preview

2026-05-14T00:00:00+00:00

Today GitHub released the GitHub Copilot App into technical preview. I’ve been using it as my daily driver for a while now, so I started a video series to share what I’ve learned. This post covers the intro to the app and a first look at using it for repository maintenance.

The app is a standalone desktop experience — available on Windows, macOS, and Linux — built specifically for agent-driven development. The big idea is that all the context you’re working with already lives in GitHub, so the app starts there. Instead of context-switching between your terminal, IDE, and browser, you do all of that in one place.

A few things that stand out from the changelog announcement:

Parallel sessions — each session gets its own git work tree, branch, and task state, so you can run multiple agents against the same repo without them stepping on each other
Three session modes: Interactive (you collaborate step by step), Plan (agent proposes, you approve), or Autopilot (fully autonomous)
A GitHub inbox showing open issues and PRs across connected repos, so you can start a session directly from real work that’s waiting
Agent Merge — the agent watches CI checks, downloads failure logs, resolves merge conflicts, and pushes fixes automatically before merging (more on this below)
Model choice per session, including reasoning effort tuning

Out of this, Agent Merge is my favorite feature! It will fix merge conflicts, CI failures, Security Alerts, and other common issues that pop up during the PR lifecycle, without you having to lift a finger. Saves a lot of manual work and context switching, and it just works. I have a video below where I show it in action.

To get access right now:

Copilot Pro and Pro+: join the waitlist
Copilot Business and Enterprise: available this week once your org admin has enabled preview features and the Copilot CLI in policy settings

First look: introducing the app

In the first video I walk through what the app actually looks like and how a typical session flows — picking an issue from the inbox, watching the agent plan and implement it, then Agent Merge handling the PR without any babysitting from me.

The app uses git work trees under the hood. Each session lives in its own work tree, so three concurrent sessions against the same repo just work — no branch conflicts, no waiting.

The commit signing moment in the video is a real one: the agent ran into the GPG passphrase prompt, flagged it, and waited for me to type it in before continuing. Small thing, but it shows that the interactive mode is genuinely interactive rather than just autopilot with a pause button.

Using it for repo maintenance

Once you get past the intro, the maintenance use case is where this really starts to pay off. In the second video I take a repo with a backlog of open issues and a pile of Dependabot PRs, and use the app to triage and action them without writing a single line of code myself.

Starting from a natural language prompt like “triage open PRs and issues from a maintainer perspective” kicks off the repo status agent, which pulls in Dependabot vulnerability alerts alongside the regular issues and PRs. It then surfaces what’s safe to merge, what needs a look, and what’s been sitting untouched. From there you can tell it to kick off sub-agents for each issue in parallel — separate work trees, separate PRs, all running concurrently while you go deal with something else.

A tip that’s easy to miss: Ctrl+K / Cmd+K opens a command palette that lets you search all your recent sessions, jump to specific ones, and see their current status. Handy once you have more than a handful of sessions in flight.

To give a sense of boosts this app can give an engineer: you can find my contributions on a page (created with the GitHub Copilot App ofc): devex-metrics.github.io/devex-metrics where you can find 368 PR’s MERGED in the last 30 days. Just on OSS contributions (mostly my own projects).

In the image below you can find an overview of 2 days of sessions I have worked on with the App (and yes, I still work in the CLI and VS Code as well, although way less, might be only 5% outside of the App).

Amazing to see what a person can do and achieve these days!

I’ll keep adding videos to the series as I go deeper. Drop a comment on either video if there’s something specific you want me to cover or ping me on LinkedIn

Shooting yourself in the foot with AI

2026-05-08T00:00:00+00:00

Today I had a fun one, where I finally dove into something that was bothering me for days: since a couple of days I had been getting questions from my different GitHub Copilot sessions to close some GitHub nofications, and only from a very specific type of notification: deployment statuses.

I noticed I got these accross different projects, so I was tempted to blame the use of a new tool that recently got updated for this. I checked it low and wide on configuration, user system prompts, etc., even checked my copilot-instructions files, but nothing to be found. Already logged an error with the tool creator, thinking it was something they added without telling me.

Figuring out where this was coming from

Of course I dropped into a Copilot CLI session to figure this one out. Somewhere, in one of my config files, either on the repo or global level, this could have been added somewhere? Perhals a tool config, VS Code extension, or something else?

After checking a couple of these locations, Copilot found the issue:

So this was created in one of my sessions by myself (with Copilot), where had interpreted a command to become: write a prompt to disk to get notifications, as an extension for the Copilot CLI! I have been working on my own Agent that would look at my GitHub notifications and clean them up for me, so this must have flowed over into a tool (extension) somewhere :smile:

Cleaned it up and of course the behaviour is now gone! Happy days. So just an example of how easy it is to shoot yourself in the foot with AI. I take some pride in looking at the changes and what it does, and having a sense of what is happening in my sessions, but this one slipped through the cracks. I guess that’s what happens when you have a lot of different tools and sessions running at the same time, and you start to delegate more and more to AI assistants. It’s a good reminder to always check what your AI is doing, and to be careful with the commands you give it!

Side note: Copilot CLI extensions!

This whole thing now lead me to look at Copilot CLI extensions, which I had heard about before. There is some explanaion on the GitHub docs on installing plugins, but not really how that extension ecosystem is loaded. I found a great blogpost that goes into the details of how to create your own extensions, and how they are loaded by the Copilot CLI, which is really interesting. You can find it here: GitHub Copilot CLI Extensions: The Most Powerful Feature Nobody’s Talking About.

Have fun discovering that!

My Open Source Projects

2026-05-06T00:00:00+00:00

Over the past few years I’ve been building a collection of open source tools focused on GitHub, DevOps automation, and more recently AI tooling. Some have hundreds of dependents, others are still in the planning phase. I wanted to share an overview of these projects in one place, both to highlight the work I’ve been doing and to hopefully inspire others to contribute or build their own tools in this space. Here’s a quick rundown of each project, with links to the repos and a brief description of what they do. Feedback / ideas / and submissions are always welcome!

DevEx Metrics

Repo: devex-metrics/devex-metrics

DevEx Metrics is a developer experience reporting and dashboarding tool for GitHub organisations and users. It collects a broad set of metrics — open, merged and closed pull requests, lines of code changed, comments and commits per PR, estimated GitHub Actions minutes consumed, unique committers and reviewers over the last 90 days, and dependent repository counts — and turns them into a Markdown report and an HTML dashboard you can host on GitHub Pages.

You can run it locally or schedule it as a GitHub Actions workflow for daily snapshots. Both Personal Access Token and GitHub App authentication are supported, and results are JSON-cached so you’re not hammering the API on every run.

The live dashboard is available with my own OSS data at devex-metrics.github.io/devex-metrics.

AI Engineering Fluency

Repo: rajbos/ai-engineering-fluency

Previously known as the “GitHub Copilot Token Tracker”, AI Engineering Fluency helps you track your AI token usage and fluency score across a lof of different AI coding tool out there: VS Code + GitHub Copilot, JetBrains IDEs, Visual Studio, Claude Code, Gemini CLI, Cursor, Continue, and more. All data is read from local session logs — nothing leaves your machine unless you explicitly opt in.

The extension shows near real-time token usage in the status bar alongside a fluency score dashboard. There’s also a JetBrains plugin, a Visual Studio extension, and a CLI (npx @rajbos/ai-engineering-fluency stats) for headless environments. Teams who want to aggregate data across engineers can spin up the self-hosted sharing server — a Docker + SQLite container that authenticates via GitHub sessions, with no Azure account required.

Alternative GitHub Actions Marketplace

Repo: devops-actions/alternative-github-actions-marketplace

The official GitHub Actions Marketplace is functional but limited — no meaningful filtering, no dependents count, no easy way to evaluate community adoption. This project is my attempt at a better version.

It’s a React frontend backed by an Azure Functions API and Azure Table Storage, indexing 20,000+ GitHub Actions with search by name or owner, filtering by action type (Node/JavaScript, Docker, Composite), and showing the dependents count for each action so you can actually judge real-world adoption. Stack is Azure Static Web Apps (free tier) for the frontend, Azure Functions on a consumption plan for the API, and a companion npm package (@devops-actions/actions-marketplace-client) to push metadata into the store.

devops-actions Organisation

Org: github.com/devops-actions

The devops-actions organisation groups a set of GitHub Actions I’ve built for common CI/CD automation tasks. All actions follow supply-chain security best practices and are tracked with OpenSSF Scorecards.

Action	Dependents	What it does
action-get-tag	679 ⭐	Read the current Git tag during a workflow run
actionlint	319 ⭐	Lint your GitHub Actions workflow files with actionlint
issue-comment-tag	196	Create or update a tagged comment on an issue or PR
variable-substitution	73	Substitute environment variables into config files
json-to-file	58	Write JSON content to a file in the workspace
load-used-actions	18	Scan a repo or org for all GitHub Actions in use
load-available-actions	17	List all public actions available in an org or user account
azure-appservice-settings	6	Push settings to an Azure App Service from a workflow
load-runner-info	5	Retrieve runner details (labels, OS, status) for an org
github-copilot-pr-analysis	2	Run GitHub Copilot PR analysis as a workflow step
load-dependents-count	1	Count how many repos depend on a given action or package
secure-action-inputs	0	Validate and sanitise inputs passed to an action

The two most widely adopted are action-get-tag (679 dependents) — a simple utility for reading the current Git tag inside a workflow — and actionlint (319 dependents), which wraps the actionlint tool to lint your workflow YAML files in CI.

GHAzDoWidget

Repo: rajbos/GHAzDo-widget

GHAzDoWidget is an Azure DevOps extension that shows GitHub Advanced Security alert counts directly in your Azure DevOps dashboards. If your source code lives in GitHub but your project management lives in Azure DevOps, this gets those alert totals — Dependabot, secret scanning, and code scanning — in front of you without having to switch context.

The extension ships in multiple widget sizes: a 2×1 combined overview, individual 1×1 tiles per alert type, and larger 2×2–4×2 widgets with trend lines, severity pie charts, open/dismissed/fixed status timelines, grouped-by-repo views, and a time-to-close scatterplot to visualise how quickly alerts get resolved.

Available on the Azure DevOps Marketplace.

Jarvis

Repo: rajbos/Jarvis

Jarvis is my personal, locally-hosted AI assistant — an Electron + TypeScript desktop app that lives in the system tray, starts automatically on boot, and sends all prompts to a local Ollama instance so nothing leaves my machine. New capabilities come in via MCP (Model Context Protocol) servers without needing to touch the core app.

The planned feature set reflects my own GitHub-heavy day-to-day:

Guided onboarding to discover local Ollama models and connect GitHub via OAuth
Automated GitHub maintenance tasks (secrets scanning, fork analysis, stale repo detection, dependency audits)
Cross-repo activity tracking with weekly summary generation
Async background jobs with rate-limit awareness
Encrypted local SQLite store for config and conversation history

The project is currently in the first phases — the full architecture specification is in the repo if you want to follow along or contribute ideas.

Where the GitHub Copilot extension points break governance

2026-05-01T00:00:00+00:00

A lot of the recent additions to the GitHub Copilot ecosystem add real value for individual developers, yet they also expand the security surface that an enterprise has to reason about. Most of these new entry points let a developer pull executable instructions, configuration, or full processes from any random repository on the internet, with very little or no central control. This post looks at the five places where I think the gap between “useful for one engineer” and “safe to run across a 5,000 person org” is widest right now.

We’ll look at these topis:

GitHub Copilot CLI plugin marketplace
GitHub Copilot CLI local extensions
Agent Package Manager (APM)
gh skill now in the GitHub CLI
MCP servers across editors
VS Code extensions and the different registries

Photo by Samuel Regan-Asante on Unsplash

GitHub Copilot CLI plugin marketplace

The Copilot CLI lets you register a marketplace of plugins and install plugins from it. The on-ramp is one command:

copilot plugin marketplace add OWNER/REPO
copilot plugin install some-plugin@some-marketplace

A marketplace is just a GitHub repository with a marketplace.json file in .github/plugin/. There is no review, no signing, no central index. Two marketplaces (copilot-plugins and awesome-copilot) are registered by default, but any user can add any other repo, including a personal fork or a newly created account that copies a real plugin name with a small change.

Versioning is the next gap. The CLI plugin reference has a version field in plugin.json, but the copilot plugin install command has no syntax to pin to a version. You install OWNER/REPO, OWNER/REPO:PATH, a Git URL, a local path, or plugin@marketplace, and you get whatever HEAD of the source happens to be at that moment. copilot plugin update NAME pulls latest. There is no lockfile, no SHA pinning of the kind gh skill has, and no provenance attestation. A marketplace can change a plugin’s contents without changing the version field, and the next update ships those changes to every developer who installed it.

Plugins themselves are executable assets.They sit in the directory the marketplace points at, get pulled to the user’s machine, and run in the user’s shell context with whatever permissions the developer has. That is the same context as their git credentials, their cloud CLI sessions, and any local secrets in their environment.

What is missing for an enterprise:

No setting on a GitHub Enterprise or organization to restrict which marketplaces a Copilot CLI user is allowed to add. The MCP private registry policy that exists for VS Code does not cover this. I’d want at least to restrict this to repos under the organizations control.
No way to require signed plugins, or plugins from a verified publisher.
No audit trail on the GitHub side that tells you which plugins your developers installed and from where.
Clear versioning out of the box. preferably with provenance signing build in.

If you compare this to how npm or PyPI are usually handled in a regulated org (a private proxy, an allowlist, a vulnerability scanner in the pipeline), the Copilot CLI plugin story today is roughly where npm was around 2014.

GitHub Copilot CLI local extensions

Beyond the plugin marketplace there is a second, almost entirely undocumented extension surface baked into the Copilot CLI: the .github/extensions/ directory. A detailed reverse-engineering writeup by htek.dev extracted this from the Copilot SDK source, because there is essentially no public documentation for it. The architecture is that the CLI discovers any subdirectory containing an extension.mjs file, forks it as a child Node.js process, and communicates with it over JSON-RPC via stdio. The extension calls joinSession() and gets back a live session object that lets it register custom tools, intercept every agent lifecycle event, rewrite prompts, and make permission decisions.

The lifecycle hooks are the part that matters from a governance angle:

onSessionStart — inject context into every session before the user’s first message
onUserPromptSubmitted — rewrite or augment the user’s prompt before the agent sees it
onPreToolUse — approve, deny, or modify the arguments of any tool call before it executes
onPostToolUse — react after any tool completes, inject feedback the agent acts on
onErrorOccurred — decide whether to retry, skip, or abort on failure

The onPermissionRequest handler in the joinSession() call replaces the standard user confirmation prompt for every tool execution. The SDK ships an approveAll import that does exactly what it sounds like: pass it and the extension silently approves every shell command, file write, and network call without showing the user a prompt.

The discovery paths are what make this a fleet-wide concern:

Project-scoped: .github/extensions/ is committed to the repo. Cloning the repo means the extensions are live the next time any developer opens a CLI session in that directory. No install step. No opt-in. The moment the CLI opens the project, it forks whatever .mjs files are in that folder.
User-scoped: ~/.copilot/extensions/ applies to every repo on the developer’s machine. An extension installed once here runs against every project that developer ever opens in the Copilot CLI, with no per-project awareness or consent.

Project extensions shadow user extensions on name collision, and the load order across extensions is not guaranteed — combined with a known bug where multiple extensions registering hooks results in only the last-loaded extension’s hooks actually firing, this creates silent behavior that is hard to audit even locally.

What is missing for an enterprise:

No org-level allowlist. Any .github/extensions/ directory in any repo a developer clones will have its extensions activated with no central gate.
No signing or publisher verification. An extension is an arbitrary .mjs file on disk.
No audit trail. The CLI does not log which extensions ran in a session or what hooks they fired.
No policy to restrict which onPermissionRequest handlers can suppress user prompts. A malicious or compromised extension can run approveAll and the developer sees nothing different.

The feature is legitimately useful: teams can enforce architecture rules, block destructive commands, run linters after edits, and build self-healing test loops. But those same capabilities — prompt rewriting, permission suppression, tool argument modification — are exactly what a supply-chain attack would want, and right now there is no org-level control surface at all.

Agent Package Manager (APM)

Microsoft APM is a dependency manager for AI agent context. You declare an apm.yml, run apm install, and it pulls instructions, skills, prompts, agents, hooks, plugins, and MCP servers from any git host (GitHub, GitLab, Bitbucket, Azure DevOps, GitHub Enterprise) into every detected agent client on the machine.

The manifest lives in the repo itself (apm.yml and apm.lock.yaml are committed alongside the code, like package.json and package-lock.json). It looks like this:

dependencies:
  apm:
    - anthropics/skills/skills/frontend-design
    - github/awesome-copilot/plugins/context-engineering
    - microsoft/apm-sample-package#v1.0.0
  mcp:
    - name: io.github.github/github-mcp-server
      transport: http

APM does ship a governance story, and it is the most thought-through one in this post. There is apm-policy.yml with tighten-only inheritance from enterprise to org to repo, a published bypass contract, hidden-Unicode scanning on every install, lockfile integrity hashes, and an apm audit --ci mode that you can wire into branch protection.

The catch is that all of this governance is opt-in and lives outside of GitHub itself. A fresh install of apm on a developer laptop has no policy file. Policy is also pull-only: the canonical org policy lives at <org>/.github/apm-policy.yml and the CLI fetches it on demand when a developer or CI job runs apm install or apm audit --ci. There is no push, no agent, and no central enrollment. The fetched policy is cached locally for an hour by default in apm_modules/.policy-cache/, and a fetch_failure: warn|block knob decides what happens when the org repo is unreachable. The default is warn, which means an offline laptop with an empty cache resolves with no policy at all. Repo-local apm-policy.yml files can extends: org and only tighten the parent rules, never relax them. See the Policy Reference and Governance Guide for the full mechanics.

Until your security team writes one, publishes it, and gets it picked up on every machine, an apm install will happily resolve transitive dependencies from any reachable git host. And don’t worry: your CI/CD pipeline will do the same (if the tooling is already installed)!

There is also no auto-install. APM is purely a CLI; it has no editor extension that runs apm install when you open a repo in VS Code. The docs frame it explicitly as “same as npm install after cloning a Node project”, which means the install step relies on the developer running it (or a devcontainer postCreateCommand, or a CI job). The flip side is that the deployed files (under .github/, .claude/, .cursor/, .gemini/) are recommended to be committed, so a teammate who clones the repo gets the agent context immediately, before they ever run apm install. That is convenient and it also means the agent is reading APM-deployed content the moment the editor opens the repo, regardless of whether the local CLI was ever invoked.

APM packages can declare scripts (think npm scripts), and the policy reference exposes manifest.scripts: allow|deny precisely because of this risk. Default is allow. So an attacker who lands a package in your dependency tree can also land scripts, unless your org policy denies them outright.

Versioning is fine on the manifest side: dependencies pin with #tag or #sha, the lockfile records resolved commit SHAs and content hashes, and the org policy can require specific versions with a require_resolution of project-wins, policy-wins, or block. Updates happen on apm install --update, not implicitly. Direct and transitive resolution stay the parts I would worry about: a package you trusted six months ago can pull in a new dependency on its next release, and unless your org policy has a tight dependencies.allow pattern, the new source slips through.

The MCP integration is worth a separate paragraph. apm install --mcp NAME adds an entry under dependencies.mcp in apm.yml and writes the resolved server config straight into the native config file of every detected client (Copilot, Claude, Cursor, Codex, OpenCode, Gemini) on the filesystem, bypassing each client’s own registry or policy layer. The full mechanism is documented in the APM MCP Servers guide. Convenient for a developer; also a clean way around whatever per-client policy exists. You are then relying on the runtime side of those clients to apply policy, and only a few of them do, with workarounds.

`gh skill` now in the GitHub CLI

Note: this is the GitHub CLI, not the GitHub Copilot CLI!

The gh skill command lets you discover, install, manage, and publish Agent Skills from any GitHub repository:

gh skill install github/awesome-copilot documentation-writer
gh skill install some-user/some-repo some-skill --pin v1.0.0

The supply-chain features here are better. Skills can be pinned to a tag (unsafe) or commit SHA, the install records the git tree SHA in the skill’s frontmatter, gh skill update compares local SHAs against the remote, and gh skill publish will offer to enable immutable releases so that even a repo admin cannot rewrite a published version.

Audit tooling is thin. The gh skill manual lists install, preview, publish, search, and update as the only subcommands. There is a gh skill preview to inspect a skill’s content before installing, and gh skill update uses the stored tree SHA to detect drift, but there is no gh skill audit and no org-side audit log of what your developers installed. If you want to know which skills landed on a developer’s laptop, you have to grep the agent host directories yourself.

The thing that is not there is an org-level allowlist.GitHub itself is unusually direct about this in the changelog:

Skills are installed at your own discretion. They are not verified by GitHub and may contain prompt injections, hidden instructions, or malicious scripts. We strongly recommend inspecting the content of skills before installation.

So the tooling around a single skill is solid, the tooling around “which skills is my org allowed to use” is not. A developer can gh skill install from any public repo, and the agent host (Copilot, Claude Code, Cursor, Codex, Gemini, all the CLI options) will pick the skill up the next time it scans the directory. Skills are a first-class extension point for the agent’s behavior, which means a malicious skill is closer to a custom system prompt than to a passive config file.

Dependabot does not help here either: agent skills, MCP servers, APM packages, and Copilot CLI plugins are not on the list of ecosystems Dependabot supports. That means no automatic update PRs, no security advisories wired in, and no scheduled drift detection across these surfaces. You would have to build that yourself.

MCP servers across editors

This is the area where the situation has gotten more confusing rather than less, even though there has been real work on it. Back in March 2025 MCP exploded into the AI world: extensibility from anywhere into anything! Since then, a lot of servers and OSS repos turned out to be playing around with things. The hard part is that a large share of those repos have since been abandoned. Endor Labs covered this in its State of Dependency Management 2025 report (summary on the Endor Labs press release): more than 10,000 MCP servers were created in less than a year, 75% of them by individual developers rather than organizations, around 40% have no license at all, and 82% touch sensitive APIs. Maintenance signals on the long tail are weak, which means the same servers your developers happily installed last year may already be effectively orphaned.

A short summary of where MCP server config lives today:

VS Code: .vscode/mcp.json (workspace), the user-profile mcp.json opened via MCP: Open User Configuration, or contributed by an installed VS Code extension. The full schema is in the VS Code MCP servers docs. APM sits on top of this: it stores MCP servers in the repo’s apm.yml, then writes them into the same .vscode/mcp.json file the editor reads. By default apm install does not overwrite locally-authored entries (that needs --force, per the CLI reference), so the file you end up with is APM’s set plus anything that was already there. If a developer thinks “I’m only running the APM-managed servers”, they are wrong: they are running APM-managed plus whatever they (or another tool) wrote into mcp.json previously.
Cursor, Windsurf, Codex, Claude Code, Gemini CLI, Copilot and other CLI’s: each has its own file in its own location, with its own schema variations.
The remote Copilot agents (Cloud Agent, Spark, Spaces, Review Agent) each have their own configuration surface and can only be configured by a repo admin.

GitHub did ship an MCP private registry policy for Copilot that lets an enterprise restrict which MCP servers Copilot users can connect to. Useful, and a real step forward, but at the time of writing it only applies inside Copilot in VS Code. The same Copilot identity used in JetBrains, Neovim, the CLI, Spark, Spaces, the Cloud Agent, or the Review Agent is not covered by that policy.

Two patterns make the policy easier to bypass than it looks:

Local stdio servers. The VS Code MCP docs describe three config paths: the gallery flow (which the Copilot private registry can gate), the workspace .vscode/mcp.json, and the user-profile mcp.json. The registry policy applies to the gallery flow. A developer who edits either JSON file directly gets a one-time “trust this server” prompt and the server starts. There is a separate VS Code device-management policy that can disable MCP entirely, but it is on/off, not allowlist-aware. See extension runtime security for the surrounding policy surface.
Extension-contributed servers. A VS Code extension can contribute MCP servers through its manifest. If an extension is allowed to install (and most orgs do not gate extensions tightly, see the next section), the MCP servers it contributes inherit the same trust as the extension itself. That sidesteps the registry policy entirely.

Even worse: clone the extension repo from github.com, build it, and just use the compiled VSIX file in VS Code!

So the practical state is: you can get a meaningful slice of governance for Copilot in VS Code if you set up the registry, and almost no governance for any of the other clients on the same laptop, all of which can reach the same internal systems. So we are not there yet, but at least a step in the right direction.

VS Code extensions and the registry split

The extension story is the oldest of the five, and it is the one that has changed shape most recently because of the Cursor and Windsurf-style forks. A few things to be explicit about:

The Microsoft Visual Studio Marketplace is closed to non-Microsoft products by its terms of use. Any VS Code fork (Cursor, Windsurf, VSCodium, Kiro, Antigravity, Positron) cannot legally use it.
Those forks generally point at Open VSX, the Eclipse Foundation registry. Open VSX has a smaller catalog, less aggressive abuse handling historically, and a publish flow that is easier to ride.
On April 21, 2026 the Eclipse Foundation launched the Open VSX Managed Registry as an SLA-backed paid tier (99.95% uptime, defined support tiers), with AWS, Google, and Cursor as initial adopters. The launch numbers paint the scale: 300M+ downloads per month, 200M+ daily requests at peak, 12,000+ extensions, 8,000+ publishers. The community instance was being asked to do the job of always-on critical infrastructure, and the AI editors are most of the reason.

For an org this means the threat model differs by editor, even when the developer thinks they are installing “the same extension”. A name on the Microsoft Marketplace is not necessarily owned by the same publisher on Open VSX. Typosquats and copy-jobs of popular extensions show up regularly on both registries, and an extension is essentially arbitrary code in your editor process with access to your workspace files, your environment, and any tokens the editor holds.

The MCP angle ties back in here: an extension can contribute MCP servers, settings, and language model providers. So an extension that gets past your install policy can reintroduce all the things you tried to gate at the registry layer.

What helps in practice:

The VS Code extensions.allowed and related policies, deployed through your endpoint management, so that only an allowlist of extensions can install at all.
Mirroring Open VSX internally if you support fork editors, with a curated subset rather than a full passthrough.
Treating new extension installs the same way you treat new npm dependencies: review, scan, and budget for the maintenance.

Endpoint protection is the layer that catches what the registries miss. Even the official VS Code documentation on extension runtime security is direct that an extension runs with the user’s full permissions: it can read and write any file the editor can, spawn processes, and make network calls. The Marketplace does scan packages and verify signatures (see the Microsoft post on security and trust in the Visual Studio Marketplace), but malicious extensions and credential-stealing supply chain incidents keep landing (see the Wiz writeup on supply chain risk in VS Code extension marketplaces and Check Point’s report on 45,000+ downloads of malicious extensions). For an org that means the controls have to live below the editor: managed device policy that blocks unsigned binaries, EDR that watches the editor’s process tree the same way it watches a browser, outbound DNS and TLS inspection that can flag the unusual call patterns an extension makes, and a workstation lifecycle that assumes a compromised editor is one of the realistic incidents you respond to. Third-party scanners like ExtensionTotal can give you a per-extension risk score before you allow it, but treat them as an addition to your endpoint stack rather than a substitute.

New: VS Code enterprise policy updates

VS Code shipped a notable batch of new enterprise policies around late April 2026 (documented at code.visualstudio.com/docs/enterprise/policies) that start closing some of the gaps described above. The most relevant additions:

MCP server control — ChatMCP (chat.mcp.access) lets an admin disable access to all installed MCP servers via device policy. This is a blunter but more reliable control than the Copilot private registry alone, because it applies regardless of how the server was registered.

Network filtering for agent tools — ChatAgentNetworkFilter, ChatAgentAllowedNetworkDomains, and ChatAgentDeniedNetworkDomains let you restrict which hosts an agent’s fetch tool and integrated browser can reach. Combined with ChatAgentSandboxEnabled, which runs terminal commands in a sandboxed environment, this starts to limit the blast radius of a compromised or malicious tool.

Agent tool approval — ChatToolsAutoApprove lets you lock down the “YOLO mode” (chat.tools.global.autoApprove) at the org level so individual developers cannot enable it, and ChatToolsEligibleForAutoApproval lets you force specific tools to always require manual confirmation.

Account-gated AI access — ChatApprovedAccountOrganizations blocks all AI features until the user signs into a GitHub account belonging to an approved organization. Useful for contractors, BYOD, and split environments where you need to tie the Copilot seat to an identity your org controls before anything runs.

Linux policy support — VS Code 1.106 added a /etc/vscode/policy.json file for Linux devices, meaning the same policies you deploy on Windows and macOS via ADMX or .mobileconfig can now cover Linux developer workstations through your existing config management tooling (Ansible, Puppet, Chef, Salt).

Policy diagnostics — A new Developer: Policy Diagnostics command generates a Markdown report of which policies are active, what values are in effect, and whether the Account Policy Gate is satisfied or blocked. Useful when you need to prove to an auditor — or a confused developer — exactly what the machine is enforcing.

These additions meaningfully strengthen the VS Code row in the summary table below. The gaps at the Copilot CLI, gh skill, and cross-editor MCP layers remain open.

State of the plugin governance for GitHub Copilot

If I line up the different surfaces by how much org-level governance is actually possible today:

Surface	Org-level allowlist	Provenance / pinning	Notes
Copilot CLI plugin marketplace	None	None	Any GitHub repo can be a marketplace
Copilot CLI local extensions	None	None	Committed to repo; active on clone with no install step
APM	Yes, via `apm-policy.yml`	Lockfile + content hashes	Policy is opt-in, customer-owned
`gh skill`	None	Tag and SHA pinning	GitHub explicitly mentions verification
MCP servers	Limited (Copilot in VS Code only)	None standardized	Local stdio and extension-contributed servers bypass the policy
VS Code extensions	Yes, via VS Code policy	Marketplace + signature	Differs across forks and Open VSX

The pattern across all of them is that the per-developer experience is great, the per-org enforcement is either absent or has to be assembled from policies that live in different places than the feature itself. None of these are unfixable, and APM in particular shows what the right shape looks like, but the gap between “shipped” and “safe to deploy at scale” is wider than the changelog posts suggest.

If you are responsible for any of this in a larger org, the short version of what I would do:

Decide which of these surfaces you want your developers to use at all. Default-allow is a choice that has consequences, not a neutral starting point.
For the ones you allow, pick the strongest available control today (VS Code extension policy, the Copilot MCP registry, an APM policy file) and ship it.
For the ones with no control today (CLI plugins, gh skill), at minimum log and review, and feed back to GitHub and Microsoft that this gap matters. Overall, tighten your grip on endpoint protection and your firewall/proxy configurations.

The features themselves are fine. The missing layer is the one every package ecosystem has had to grow eventually: a place for an org to say which sources it trusts, applied uniformly across every client that can pull from them.

Running GitHub Copilot CLI on local AI inference

2026-04-12T00:00:00+00:00

I’ve been using the GitHub Copilot CLI as my main terminal assistant for a while now. It works great with GitHub-hosted models (Claude, GPT-4) but that means every command, every file you give it context about, every prompt goes over the wire to a cloud provider. And what’s worse: it means that the cloud provider is hosting the beefy LLM model for me, incurring a lot of compute cost. So I wanted to explore what it looks like to run the whole thing locally — both for privacy and just to see how far local models have come. I think I have a good setup on a recent laptop, with two GPU’s and an NPU, so let’s see how far we can go.

The Copilot CLI now supports a “bring your own key” (BYOK) mode where you point it at any OpenAI-compatible API endpoint. That means any local inference engine that exposes a /v1/chat/completions endpoint should work in theory. In practice there’s quite a bit to figure out.

Hardware I was testing on

My machine is a Dell Pro Max 14 MC14250 from Q4, 2025. It has an Intel Core Ultra 7 265H processor in it, with 32GB RAM and a 1TB SSD. The interesting thing about this machine is that it has three separate AI-capable processors:

Intel Arc 140T (GPU 0) — 16 GB VRAM, integrated but a full GPU with dedicated AI acceleration
NVIDIA RTX PRO 500 Blackwell (GPU 1) — 6.1 GB dedicated VRAM (GDDR — fast video memory separate from system RAM)
Intel AI Boost NPU — dedicated neural processing unit built into the CPU

On paper that’s a lot of local AI horsepower. In practice, as I found out, things are more complicated.

How the BYOK configuration works

The Copilot CLI reads four environment variables to switch from cloud to local inference. If you have a local ollama instance running, you can point the CLI at it with:

$env:COPILOT_PROVIDER_BASE_URL          = "http://localhost:11434/v1"
$env:COPILOT_MODEL                      = "qwen2.5:7b-instruct-32k"
$env:COPILOT_PROVIDER_MAX_PROMPT_TOKENS = "32768"
$env:COPILOT_PROVIDER_MAX_OUTPUT_TOKENS = "8192"

Presuming you have downloaded that model into Ollama and it’s running on that port of course. The CLI will then send all requests to that local endpoint instead of the cloud.

You don’t need COPILOT_PROVIDER_TYPE (it defaults to openai, which is compatible with all the local engines below) and you don’t need an API key for local inference. To switch back to cloud, you just remove those variables.

One critical thing I learned early: the context window matters a lot. The Copilot CLI system prompt including all tool definitions is around 21,000 tokens. That’s before you’ve typed a single character. Any model running with less than 32k context will get its prompt truncated and start behaving strangely — either losing tools mid-conversation or giving incoherent responses.

What I tried

Ollama — works, with some gotchas

Ollama is the easiest starting point. Install it, pull a model, and it’s running an OpenAI-compatible API on port 11434. The first thing I ran into was the URL: you need the /v1 suffix (http://localhost:11434/v1), not just http://localhost:11434. The docs example omits it and you get a 404.

The second thing I ran into was the context window. By default most Ollama models run with a 2048 or 4096 context window. With a 21k system prompt you need at least 32k. I had to create a custom Modelfile:

FROM qwen2.5:7b-instruct
PARAMETER num_ctx 32768

Then create it with: ollama create qwen2.5:7b-instruct-32k -f Modelfile

A couple of performance settings also helped, particularly for flash attention and KV cache quantization. The KV cache (key-value cache) stores the intermediate attention values the model computes for each token — with a 32k context window it can consume 1–2 GB of VRAM, so compressing it matters:

[System.Environment]::SetEnvironmentVariable("OLLAMA_FLASH_ATTENTION", "1", "User")
[System.Environment]::SetEnvironmentVariable("OLLAMA_KV_CACHE_TYPE", "q8_0", "User")

Restart Ollama after setting these. The NVIDIA RTX PRO 500 can handle the 32k KV cache with these settings.

On model selection: not all models support tool calling, which the Copilot CLI relies on heavily. From what I tested, gemma3, mistral:7b, and qwen2.5-coder:1.5b-base all fail because they don’t support tool calls. qwen2.5:7b-instruct works well. Llama 3.1 8B also works but quality is mediocre.

Ollama verdict: ✅ Works. qwen2.5:7b-instruct-32k on the NVIDIA GPU is the sweet spot. Slow (~30% GPU utilisation, around 13 tokens/sec) but functional.

Foundry Local — almost entirely blocked because of driver limitations

Microsoft Foundry Local is interesting because it uses OpenVINO and can target all three AI processors: the Arc GPU, the NPU, and CUDA. I installed it with winget install Microsoft.FoundryLocal.

One nice thing about Foundry Local is that it automatically picks the best model variant for your hardware. You just say foundry model run qwen2.5-7b and it figures out whether to download the CUDA, OpenVINO GPU, or NPU variant.

That sounds great, but I hit walls on every path:

NPU (Intel AI Boost): All the NPU-optimised models cap at maxInputTokens=3696. The CLI needs ~21k. That’s a hard fail — not a configuration issue, it’s a hardware architectural limit of how much context the NPU can process in one inference pass. Nothing to be done here until Intel expands the NPU context window in future driver versions.

OpenVINO GPU (Arc 140T): This is the interesting path — the Arc 140T has 16GB and could run 7B models comfortably. But every OpenVINO GPU model I tried threw EPContext node not found. The root cause seems to be a driver version mismatch: the Arc driver I have installed (32.0.101.8509, the Dell OEM version from November 2025) is older than what Foundry Local’s OpenVINO runtime needs (32.0.101.8629, released April 2026). Dell hasn’t published the new driver for the MC14250 yet. Once that driver lands, this path should work well — 7B models on 16GB with no memory pressure. I’ve had some driver issues in the past, so moving over to the raw driver from Intel’s site is a bit scary for now, but it’s on the table if Dell doesn’t update soon.

CUDA (NVIDIA): 7B models crash with Out of Memory (OOM). 6GB VRAM isn’t enough for the model weights plus the KV cache for 28k context. A 1.5B model works but it’s not capable enough for agentic tasks.

Foundry Local verdict: ❌ Blocked by driver version. Worth revisiting once Dell publishes the Arc driver update.

LM Studio — works with the right configuration

LM Studio uses llama.cpp under the hood and exposes an OpenAI-compatible server. It supports Vulkan, which means it can run on Intel Arc without needing OpenVINO or driver-specific binaries.

Install it with:

winget install ElementLabs.LMStudio --scope user

Then there are several things to get right:

Context window: Defaults to 4096. The model load fails with n_keep >= n_ctx until you change it. In the Developer tab, set the context length to 32768 before loading the model.

URL: Use http://127.0.0.1:1234/v1, not http://localhost:1234/v1. LM Studio binds IPv4 only and on some Windows machines localhost resolves to ::1 (IPv6), causing a connection refused.

GPU selection: This was the most complex part. LM Studio’s default is the CUDA backend when a CUDA-capable GPU is present — meaning it runs on the NVIDIA card, not Arc. Getting it onto Arc requires switching the backend to Vulkan, which means editing %USERPROFILE%\.lmstudio\.internal\backend-preferences-v1.json:

[{"model_format":"gguf","name":"llama.cpp-win-x86_64-vulkan-avx2","version":"2.13.0"}]

But even with Vulkan enabled, LM Studio still defaulted to the NVIDIA card — because under Vulkan, the NVIDIA card (Vulkan device 1) is classified as “Discrete” and gets priority over the Arc (Vulkan device 0, classified as “Integrated”). The fix is to go to Settings → Hardware in LM Studio and disable the RTX PRO 500. LM Studio then writes this to hardware-config.json:

{"json":[["llama.cpp-win-x86_64-vulkan-avx2",{"fields":[{"key":"load.gpuSplitConfig","value":{"strategy":"evenly","disabledGpus":[1],"priority":[],"customRatio":[]}}]}]],"meta":{"values":["map"]}}

Arc 140T alone is too slow. This surprised me. The Arc has 16GB but it’s integrated memory — shared system memory (LPDDR5x) at ~68 GB/s bandwidth. Running a 7B model on it saturated the memory bus and froze my entire machine, including the mouse. LLM inference is completely memory-bandwidth-bound, and 68 GB/s shared with the CPU simply isn’t enough for interactive use.

The working configuration ended up being to offload 25 layers of the model to NVIDIA GDDR, the rest on CPU. LM Studio auto-selects this split when both GPUs are enabled. The NVIDIA’s dedicated GDDR bandwidth handles the hot layers and the CPU handles the rest. Inference takes longer than pure NVIDIA CUDA, but it works without freezing the machine.

Model selection for LM Studio: My first download was qwen2.5.1-coder-7b-instruct, which technically works but is mediocre at agentic tasks — it doesn’t proactively explore the repository you’re working in and needs very explicit prompting. The coder fine-tune trades general instruction following for code completion, which hurts agentic behaviour.

Switching to qwen2.5-7b-instruct@q5_k_m (the bartowski build from Discover) gave better results. The @q5_k_m suffix is the quantization level — 5-bit compressed weights, a smaller and faster-to-run version of the model with only a marginal quality trade-off versus the full-precision original. Still not at the quality level of online Claude, but the tool calling works correctly and it handles the CLI’s complex nested JSON schemas for things like asking the user clarifying questions.

I also tried Gemma 4 E4B, which is Google’s newest model with 128K context and native tool calling support. The context window is great (it easily fits the CLI prompt with room to spare) but it consistently generated malformed JSON schemas when calling the ask_user tool — making up property names that don’t exist in the schema. Small models struggle with deeply nested JSON Schema definitions and Gemma 4 E4B isn’t an exception.

The lms load re-load trap. If you call lms load while a model is already running, LM Studio doesn’t replace it — it creates a second instance with a :2 suffix (e.g. qwen2.5-7b-instruct@q4_k_s:2). The original stays loaded with its old context window. Since the Copilot CLI env var points to the plain identifier, it hits the old instance. Always unload first to free up GPU memory and avoid confusion:

lms unload --all
lms load qwen2.5-7b-instruct@q4_k_s --gpu 0.78 --context-length 32768

The --gpu trap with the CLI. After all the above I noticed that loading via lms load qwen2.5-7b-instruct@q4_k_s --gpu max sometimes silently fell back to CPU. The only sign is GPU memory: 1852 MiB = CPU mode; 4914 MiB = GPU split. The --gpu flag takes a 0–1 fraction of layers, not a layer count or “max”. Using --gpu 0.78 reliably puts ~25/32 layers on the NVIDIA GDDR which is the max layer config that fits on my GPU. Full load command:

lms load qwen2.5-7b-instruct@q4_k_s --gpu 0.78 --context-length 32768

Also worth noting: @q4_k_s (4-bit quantization, 4.46 GB) is slightly faster than @q5_k_m (5-bit quantization, 4.78 GB) because the smaller file leaves more GPU headroom for the KV cache. In practice the quality difference between Q4_K_S and Q5_K_M is imperceptible for agentic tasks.

LM Studio verdict: ✅ Works with qwen2.5-7b-instruct@q4_k_s --gpu 0.78, 25 layers on NVIDIA, rest on CPU. Fastest 7B config measured on this machine. NPU is not accessible from LM Studio at all.

What the NPU situation actually is

A quick note on the Intel AI Boost NPU since it kept coming up during my research. The NPU is only accessible through OpenVINO — which means Foundry Local is currently the only local inference engine that can use it from the options I tried. But as covered above, all the NPU-optimised models cap at 3,696 input tokens, which is nowhere near enough for the 21k Copilot CLI system prompt. This isn’t a software configuration problem — it’s the current limit of how the AI Boost NPU processes context in a single pass.

So the NPU is off the table for this use case until Intel ships larger-context NPU models.

What I tried next: vLLM and TGI (HuggingFace inference stack)

After getting Ollama and LM Studio working, I wanted to see if the HuggingFace inference stack could do better. The native PyTorch kernels (AWQ-Marlin, FlashAttention v2) are purpose-built for this hardware and in theory should outperform llama.cpp’s GGUF decode. I tried two options: vLLM and TGI (Text Generation Inference).

The format split: GGUF vs safetensors

Before going further, it’s worth understanding why these are separate ecosystems:

Ollama / LM Studio → llama.cpp → GGUF only
vLLM / TGI → PyTorch → HuggingFace safetensors only

GGUF is the model file format used by llama.cpp — think of it as the packaging format for quantized models in the Ollama/LM Studio ecosystem. Safetensors is HuggingFace’s format, used by PyTorch-based runtimes like vLLM and TGI. Both support “4-bit” quantization but they’re completely different kernel implementations. You can’t load a GGUF file into vLLM or a safetensors file into Ollama.

For Qwen2.5-7B on a 6 GB GPU, the right HuggingFace model is Qwen/Qwen2.5-7B-Instruct-AWQ — an AWQ (Activation-aware Weight Quantization) 4-bit model, about 4.5 GB on disk.

vLLM via Docker

vLLM is the most production-grade HuggingFace inference server. The official Docker image (vllm/vllm-openai:latest) comes with CUDA, FlashAttention, and AWQ-Marlin kernels pre-built.

Getting it to fit on a 6 GB GPU required a few tricks:

docker run -d --runtime nvidia --gpus all --name vllm-server `
    -v hf-cache:/root/.cache/huggingface `
    -p 127.0.0.1:8000:8000 --shm-size 2g `
    vllm/vllm-openai:latest `
    --model Qwen/Qwen2.5-7B-Instruct-AWQ --quantization awq_marlin `
    --max-model-len 32768 --gpu-memory-utilization 0.81 `
    --kv-cache-dtype fp8 --max-num-seqs 1 `
    --cpu-offload-gb 2 --enforce-eager `
    --dtype auto --host 0.0.0.0 --port 8000

Key constraints on a 6 GB card:

Free VRAM after CUDA runtime init is only 4.89 GiB out of 5.97 GiB total — --gpu-memory-utilization 0.81 is the cap
AWQ weights alone need 5.2 GiB on-GPU → --cpu-offload-gb 2 drops that to 3.17 GiB, freeing 1.25 GiB for KV cache
--kv-cache-dtype fp8 halves KV footprint vs fp16 (fp8/fp16 = 8-bit vs 16-bit floating-point precision for stored values; lower precision = smaller memory footprint), required for 32k context
-v hf-cache:/root/.cache/... — use a named Docker volume, not a Windows bind mount. Loading from NTFS through VirtioFS takes 20+ minutes per model shard

Use 127.0.0.1:8000:8000 not 8000:8000 — the unqualified form sometimes fails silently when old wslrelay processes still hold the port.

The logs confirm FlashAttention v2 and AWQ-Marlin kernels loaded, which is great. But the benchmark result was sobering: ~2.6 tok/s. The CPU offload is the bottleneck — those 2 GiB of weight layers go through PCIe every single token. The architecture simply isn’t designed for this.

vLLM verdict: ⚠️ Works but impractical on 6 GB. 2.6 tok/s is too slow for interactive CLI use. Natural fit is GPUs with ≥10 GB where the model fits fully in VRAM.

TGI (Text Generation Inference)

TGI is HuggingFace’s own inference server. It has native AWQ support and the official image is well-maintained. But there’s a hard blocker on 6 GB:

AWQ weights = 5.2 GiB. GPU total = 5.97 GiB. After CUDA init there’s ~4.89 GiB free.
TGI has no --cpu-offload-gb equivalent. It must load the entire model into VRAM.
Result: OOM at KV cache allocation. The weights alone don’t fit, let alone any context.

There’s also a WSL2 Triton linker bug that needs -e LIBRARY_PATH=/usr/local/cuda-12.4/compat to find libcuda.so, but even with that fix the OOM is structurally unavoidable.

TGI verdict: ❌ OOM on 6 GB with this model and context size. TGI has no CPU offload path, so the full model must fit in VRAM. On larger GPUs (≥8 GB free after init) this wouldn’t be an issue.

llama.cpp — fastest raw CUDA inference

After publishing the initial results, I got a tip to try llama.cpp directly — not through LM Studio’s wrapper, but as a standalone server with the native CUDA build. LM Studio uses llama.cpp under the hood but adds process management overhead and restricts some server flags. Running llama.cpp directly gives you tighter control over Flash Attention, KV cache precision, and context window allocation.

Installation (CUDA build on Windows)

winget install ggml.llamacpp installs the Vulkan build, not CUDA. For the CUDA build, download from the llama.cpp GitHub releases directly:

$dest = "$env:LOCALAPPDATA\llama.cpp"
New-Item -ItemType Directory -Force -Path $dest | Out-Null
# CUDA 12.4 build — works fine with CUDA 13.0 driver (backward compatible)
Invoke-WebRequest "https://github.com/ggml-org/llama.cpp/releases/download/b9075/llama-b9075-bin-win-cuda-12.4-x64.zip" -OutFile "$env:TEMP\llama-cuda.zip"
Invoke-WebRequest "https://github.com/ggml-org/llama.cpp/releases/download/b9075/cudart-llama-bin-win-cuda-12.4-x64.zip" -OutFile "$env:TEMP\llama-cudart.zip"
Expand-Archive "$env:TEMP\llama-cuda.zip" -DestinationPath $dest -Force
Expand-Archive "$env:TEMP\llama-cudart.zip" -DestinationPath $dest -Force

Flash Attention syntax changed in build b9075+: --flash-attn is no longer a bare flag. You must write --flash-attn on.

Starting the server

llama.cpp loads GGUF files directly from the LM Studio model cache — no re-download needed:

$model = "$env:USERPROFILE\.lmstudio\models\bartowski\Qwen2.5-7B-Instruct-GGUF\Qwen2.5-7B-Instruct-Q4_K_S.gguf"
& "$env:LOCALAPPDATA\llama.cpp\llama-server.exe" `
    --model $model --alias Qwen2.5-7B-Instruct-Q4_K_S `
    --n-gpu-layers 999 --ctx-size 32768 `
    --cache-type-k q8_0 --cache-type-v q8_0 `
    --flash-attn on `
    --host 127.0.0.1 --port 8088

The --cache-type-k q8_0 --cache-type-v q8_0 flags compress the KV cache to 8-bit precision. At 32k context this saves ~896 MB — exactly the headroom needed to fit Q4_K_S weights alongside a 32k context in 6 GB VRAM:

4.15 GB (weights) + 0.9 GB (q8_0 KV @ 32k) + ~0.4 GB (CUDA overhead) ≈ 5.45 GB total — ~550 MB headroom in 6 GB VRAM.

Copilot CLI configuration

$env:COPILOT_PROVIDER_BASE_URL         = "http://127.0.0.1:8088/v1"
$env:COPILOT_MODEL                      = "Qwen2.5-7B-Instruct-Q4_K_S"
$env:COPILOT_PROVIDER_MAX_PROMPT_TOKENS = "32768"  # must match --ctx-size exactly
$env:COPILOT_PROVIDER_MAX_OUTPUT_TOKENS = "8192"

COPILOT_PROVIDER_MAX_PROMPT_TOKENS must match your --ctx-size exactly. Leaving it at the default 8192 while the server runs a 32k context gives:

400 request (22704 tokens) exceeds the available context size (8192 tokens)

Result: 23.0 tok/s 🏆

23.0 tok/s — faster than Ollama Qwen3-8B and nearly 2× faster than Ollama Qwen2.5-7B (12.3 tok/s). The same GGUF file, the same GPU, just running through llama.cpp’s direct CUDA path instead of Ollama’s management layer — with Flash Attention and 8-bit KV cache enabled.

llama.cpp verdict: ✅ New fastest setup at 23.0 tok/s. It reuses GGUF files already in your LM Studio model cache, so there’s no re-download needed — just the extra setup steps for the CUDA binary.

Bonus: Vulkan tensor split across NVIDIA + Intel Arc

I also tried the Vulkan build to see if offloading layers to the Intel Arc 140T’s 18 GB of shared memory would improve throughput. The device order on this machine:

Vulkan0: Intel Arc 140T  — 18 GB (shared system RAM)
Vulkan1: NVIDIA RTX PRO 500 — 6 GB dedicated GDDR7

Results with --tensor-split 1,4 (≈20% Arc / ≈80% NVIDIA): 22.1 tok/s — essentially identical to NVIDIA-only Vulkan (22.3 tok/s) and slightly below the CUDA build (23.0 tok/s). No throughput improvement.

The reason: “shared memory” on the Arc iGPU is system RAM accessed through the memory controller — roughly 10× slower than GDDR7. The model (4.15 GB) already fits fully in dedicated VRAM, so there are no layers to gain by involving the Arc. Adding inter-GPU synchronisation only adds overhead. The 18 GB number in Task Manager looks tempting, but it only helps when a model doesn’t fit in dedicated VRAM at all.

The PowerShell profile setup

After going through all of this I set up a clean way to switch between providers from any terminal session. The set-copilot-local function gives you an interactive arrow-key menu to pick provider and model. online-copilot clears everything back to defaults.

function Invoke-InteractiveMenu {
    param(
        [string]$Title,
        [string[]]$Options
    )
    $selected = 0
    $count = $Options.Count
    [Console]::CursorVisible = $false
    try {
        Write-Host $Title -ForegroundColor Cyan
        for ($i = 0; $i -lt $count; $i++) {
            if ($i -eq $selected) { Write-Host "  > $($Options[$i])" -ForegroundColor Yellow }
            else { Write-Host "    $($Options[$i])" -ForegroundColor Gray }
        }
        while ($true) {
            $key = $Host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")
            $prev = $selected
            switch ($key.VirtualKeyCode) {
                38 { if ($selected -gt 0) { $selected-- } }              # Up
                40 { if ($selected -lt $count - 1) { $selected++ } }     # Down
                13 { Write-Host ""; return $selected }                   # Enter
            }
            if ($prev -ne $selected) {
                $pos = $Host.UI.RawUI.CursorPosition
                $pos.Y -= $count
                $Host.UI.RawUI.CursorPosition = $pos
                for ($i = 0; $i -lt $count; $i++) {
                    $line  = if ($i -eq $selected) { "  > $($Options[$i])" }
                             else { "    $($Options[$i])" }
                    $color = if ($i -eq $selected) { "Yellow" } else { "Gray" }
                    Write-Host $line.PadRight($Host.UI.RawUI.WindowSize.Width - 1) -ForegroundColor $color
                }
            }
        }
    } finally {
        [Console]::CursorVisible = $true
    }
}

# Launch LM Studio (Vulkan backend pre-configured via hardware-config.json)
function Start-LMStudio {
    Start-Process "$env:LOCALAPPDATA\Programs\LM Studio\LM Studio.exe"
    Write-Host "LM Studio launched (Vulkan/Arc backend configured)" -ForegroundColor Green
}

# Write the hardware-config.json that disables the NVIDIA GPU (Vulkan device 1)
# so llama.cpp Vulkan uses Arc 140T (device 0). Re-run after LM Studio reinstalls.
function Set-LMStudioArcGPU {
    $hwConfig = "$env:USERPROFILE\.lmstudio\.internal\hardware-config.json"
    $json = '{"json":[["llama.cpp-win-x86_64-vulkan-avx2",{"fields":[{"key":"load.gpuSplitConfig","value":{"strategy":"evenly","disabledGpus":[1],"priority":[],"customRatio":[]}}]}]],"meta":{"values":["map"]}}'
    Set-Content $hwConfig $json -Encoding utf8
    Write-Host "LM Studio: NVIDIA disabled, Arc 140T active" -ForegroundColor Green
}

function set-copilot-local {
    $providerIdx = Invoke-InteractiveMenu -Title "Select inference provider:" -Options @(
        "Ollama        (localhost:11434)"
        "Foundry Local (localhost:63839)"
        "LM Studio     (localhost:1234)"
        "vLLM / HuggingFace (127.0.0.1:8000)"
    )

    if ($providerIdx -eq 0) {
        # Ollama — qwen2.5:7b-instruct-32k is a custom Modelfile with num_ctx 32768
        $baseUrl = "http://localhost:11434/v1"
        $models  = @(
            [PSCustomObject]@{ Name = "qwen2.5:7b-instruct-32k"; PromptTokens = 32768;  OutputTokens = 8192 }
            [PSCustomObject]@{ Name = "llama3.1:8b";             PromptTokens = 131072; OutputTokens = 8192 }
            [PSCustomObject]@{ Name = "llama3.2:3b";             PromptTokens = 131072; OutputTokens = 8192 }
        )
    } elseif ($providerIdx -eq 1) {
        # Foundry Local — port is dynamic, detected from service status
        # WARNING: OpenVINO GPU models need Arc driver 32.0.101.8629 (not yet on Dell MC14250)
        $statusOutput = foundry service status 2>&1
        $foundryUrl = ($statusOutput | Select-String -Pattern 'http://[\d.:]+' | ForEach-Object { $_.Matches[0].Value } | Select-Object -First 1)?.TrimEnd('/')
        if (-not $foundryUrl) {
            Write-Host "Foundry Local service is not running. Starting it..." -ForegroundColor Yellow
            $startOutput = foundry service start 2>&1 | Select-String -Pattern 'http://[\d.:]+'
            $foundryUrl = $startOutput?.Matches[0].Value?.TrimEnd('/')
        }
        if (-not $foundryUrl) { $foundryUrl = "http://127.0.0.1:63839" }
        $baseUrl = "$foundryUrl/v1"
        $models  = @(
            [PSCustomObject]@{ Name = "qwen2.5-1.5b-instruct-cuda-gpu:4"; PromptTokens = 28672; OutputTokens = 4096 }
        )
    } elseif ($providerIdx -eq 2) {
        # LM Studio — use 127.0.0.1 not localhost (LM Studio binds IPv4 only)
        # Always unload before load: lms load adds a :2 duplicate if any model is already running.
        # --gpu 0.78 = ~25/32 layers on NVIDIA GDDR (0-1 fraction, NOT layer count).
        $baseUrl = "http://127.0.0.1:1234/v1"
        $models  = @(
            [PSCustomObject]@{ Name = "qwen2.5-7b-instruct@q4_k_s"; PromptTokens = 32768; OutputTokens = 8192 } # 17.2 tok/s — fastest 7B
            [PSCustomObject]@{ Name = "qwen2.5-7b-instruct@q5_k_m"; PromptTokens = 32768; OutputTokens = 8192 } # 13.6 tok/s
            [PSCustomObject]@{ Name = "gemma-4-e4b";                  PromptTokens = 65536; OutputTokens = 8192 }
            [PSCustomObject]@{ Name = "qwen2.5-14b-instruct";         PromptTokens = 32768; OutputTokens = 8192 }
        )
        $lmsExe = "$env:LOCALAPPDATA\Programs\LM Studio\resources\app\.webpack\lms.exe"
        if (Test-Path $lmsExe) {
            Write-Host "  ⏹  Unloading existing models..." -ForegroundColor DarkGray
            & $lmsExe unload --all 2>$null
        }
    } else {
        # vLLM (HuggingFace) — OpenAI-compatible on port 8000
        # Requires a running vLLM container. ~2.6 tok/s on 6 GB (cpu-offload-gb 2). Best on GPUs ≥10 GB.
        # Start the container first (startup takes ~5 min); check logs for "Application startup complete".
        $baseUrl = "http://127.0.0.1:8000/v1"
        $models  = @(
            [PSCustomObject]@{ Name = "Qwen/Qwen2.5-7B-Instruct-AWQ"; PromptTokens = 32768; OutputTokens = 8192 }
        )
        try { $null = Invoke-WebRequest "http://127.0.0.1:8000/v1/models" -TimeoutSec 2 -ErrorAction Stop }
        catch { Write-Warning "vLLM not responding. Start the container first." }
    }

    $modelIdx = Invoke-InteractiveMenu -Title "Select model:" -Options ($models | ForEach-Object { $_.Name })
    $selected = $models[$modelIdx]

    # For LM Studio: load the selected model with correct GPU split and context length
    if ($providerIdx -eq 2) {
        $lmsExe = "$env:LOCALAPPDATA\Programs\LM Studio\resources\app\.webpack\lms.exe"
        if (Test-Path $lmsExe) {
            $ctxLen = if ($selected.PromptTokens -ge 65536) { "65536" } else { "32768" }
            Write-Host "  ▶  Loading $($selected.Name) --gpu 0.78 --context-length $ctxLen ..." -ForegroundColor DarkYellow
            & $lmsExe load $selected.Name --gpu 0.78 --context-length $ctxLen
        }
    }

    $env:COPILOT_PROVIDER_BASE_URL         = $baseUrl
    $env:COPILOT_MODEL                      = $selected.Name
    $env:COPILOT_PROVIDER_MAX_PROMPT_TOKENS = $selected.PromptTokens
    $env:COPILOT_PROVIDER_MAX_OUTPUT_TOKENS = $selected.OutputTokens

    $providerName = @("Ollama", "Foundry Local", "LM Studio", "vLLM")[$providerIdx]
    Write-Host "Copilot provider set to local inference via [$providerName] with model [$($selected.Name)]" -ForegroundColor Green
}

function online-copilot {
    Remove-Item Env:COPILOT_PROVIDER_BASE_URL          -ErrorAction SilentlyContinue
    Remove-Item Env:COPILOT_MODEL                       -ErrorAction SilentlyContinue
    Remove-Item Env:COPILOT_PROVIDER_MAX_PROMPT_TOKENS  -ErrorAction SilentlyContinue
    Remove-Item Env:COPILOT_PROVIDER_MAX_OUTPUT_TOKENS  -ErrorAction SilentlyContinue
    Write-Host "Copilot provider set to online inference (default)" -ForegroundColor Green
}

The main functions that I use here is set-copilot-local for when I want to switch to local inference, and online-copilot to switch back to the default online models in the same terminal session. By default this means Copilot still runs against the cloud models, so I don’t accidentally lose functionality in a random terminal window. But when I want to test local models, it’s just a quick set-copilot-local and a couple of arrow keys to pick the provider and model.

Summary of what works today (April 2026)

Option	Status	Notes
`online-copilot` (GitHub-hosted Claude/GPT)	✅ Best	Default, no setup needed
llama.cpp `qwen2.5-7b-instruct@q4_k_s`, CUDA, 32k, q8_0 KV	✅ Works	23.0 tok/s — fastest local model 🏆
Ollama `qwen3:8b` on NVIDIA	✅ Works	22.1 tok/s — fastest local model before llama.cpp
LM Studio `qwen2.5-7b-instruct@q4_k_s`, `--gpu 0.78`	✅ Works	15.2 tok/s — fastest 7B in LM Studio
LM Studio `qwen2.5-7b-instruct@q5_k_m`, 25 layers NVIDIA + CPU	✅ Works	13.6 tok/s
Ollama `qwen2.5:7b-instruct-32k` on NVIDIA	✅ Works	12.3 tok/s
vLLM `Qwen2.5-7B-Instruct-AWQ` with `--cpu-offload-gb 2`	⚠️ Too slow	2.6 tok/s — CPU offload bottleneck on 6 GB
TGI `Qwen2.5-7B-Instruct-AWQ`	❌ OOM	No CPU offload support; 5.2 GiB weights won’t fit
Foundry Local on Arc 140T (OpenVINO GPU)	❌ Blocked	Needs Arc driver 32.0.101.8629 from Dell
Foundry Local on NPU	❌ Too small	3,696 token limit, CLI needs ~21k
Foundry Local CUDA 7B on NVIDIA	❌ OOM crash	6 GB isn’t enough
LM Studio on Arc 140T only (Vulkan)	❌ Too slow	Saturates shared memory bus, freezes machine

Throughput comparison

Speed matters for interactive use. The Copilot CLI generates structured JSON for tool calls, thinks through multi-step plans, and writes code — all of which require hundreds of tokens of output per turn. Anything below ~8 tok/s starts to feel noticeably slow for a CLI tool.

I measured output throughput (tokens per second) using a standardised benchmark: each provider was asked to generate a ~250-token PowerShell function, at temperature=0.2 with max_tokens=400, repeated 3 times after a warm-up run. The warm-up run loads the model into VRAM and warms the KV cache; only the subsequent timed runs are averaged.

The test prompt is a fixed PowerShell coding task so every provider generates a roughly comparable amount of output. This measures output throughput (decode speed) — which is what you feel as the response streaming in — not prompt processing speed or time-to-first-token.

⚠️ Not a perfect apples-to-apples comparison for all rows. The 1.5B rows are a separate size class — they’re fast because of the smaller model, not the runtime. The Qwen3-8B row uses a newer model architecture; the 7B rows (LM Studio and Ollama) use the same qwen2.5-7b-instruct model in different quant/format variants and are directly comparable. Foundry Local runs qwen2.5-1.5b-instruct — the only model that fits on the hardware today. vLLM uses AWQ format but the same model family.

The script is available as a Gist: Measure-LocalAIThroughput.ps1.

Provider	Model	Avg tok/s	Notes
Foundry Local	`qwen2.5-1.5b-instruct-cuda-gpu:4`	117	NVIDIA RTX PRO 500 CUDA — 1.5B model (not comparable to 7B/8B rows)
Ollama (1.5B)	`qwen2.5:1.5b-instruct-32k`	55.3	NVIDIA RTX PRO 500 CUDA, 32k context — 1.5B model
LM Studio (1.5B)	`qwen2.5-coder-1.5b-instruct`	49.8	NVIDIA RTX PRO 500 CUDA full offload — 1.5B model
llama.cpp (CUDA)	`qwen2.5-7b-instruct@q4_k_s`	23.0	CUDA, Flash Attention, 32k ctx, q8_0 KV cache — fastest 7B overall 🏆
Ollama (Qwen3-8B)	`qwen3:8b`	22.1	NVIDIA RTX PRO 500 CUDA, 32k context
llama.cpp (Vulkan/NVIDIA)	`qwen2.5-7b-instruct@q4_k_s`	22.3	Vulkan build, NVIDIA RTX PRO 500 only
llama.cpp (Vulkan/split)	`qwen2.5-7b-instruct@q4_k_s`	22.1	Vulkan, tensor-split 1:4 across Arc + NVIDIA — no benefit vs NVIDIA-only
LM Studio (7B)	`qwen2.5-7b-instruct@q4_k_s`	15.2	`--gpu 0.78` (25/32 layers NVIDIA GDDR), 32k context
Ollama (7B)	`qwen2.5:7b-instruct-32k`	12.3	NVIDIA RTX PRO 500 CUDA, 32k context
vLLM (Docker)	`Qwen2.5-7B-Instruct-AWQ`	2.5	AWQ-Marlin + FlashAttention v2; `--cpu-offload-gb 2` bottleneck on 6 GB

A few things to note from these numbers:

Foundry Local at 117 tok/s looks spectacular but it’s running a 1.5B model — not capable enough for agentic tasks. Still, it’s a preview of what the numbers could look like once the Arc 140T OpenVINO path is unblocked: running a proper 7B model on 16 GB VRAM with no memory pressure.
llama.cpp CUDA at 23.0 tok/s is the new overall winner. The same Q4_K_S GGUF that LM Studio runs, launched directly via llama.cpp with Flash Attention and 8-bit KV cache — no management overhead, no restricted flags. Nearly 2× faster than Ollama Qwen2.5-7B on the same hardware.
Ollama Qwen3-8B at 22.1 tok/s is the runner-up. Qwen3-8B is a newer architecture than Qwen2.5-7B and fits fully into the 6 GB NVIDIA VRAM (Q4_K_M = 4.68 GB, leaving room for KV cache and overhead). Ollama handles the 128k-capable model natively without needing a custom Modelfile. LM Studio’s Q4_K_M variant fails to load — the 4.68 GB weights plus LM Studio’s per-process overhead pushes it over the 6 GB limit; Q3 quantisation would be needed there.
llama.cpp Vulkan tensor split at 22.1 tok/s is essentially identical to NVIDIA-only Vulkan (22.3 tok/s). The Intel Arc 140T’s “18 GB shared memory” is system RAM — adding it only introduces inter-GPU synchronisation overhead when the model already fits in dedicated VRAM.
LM Studio Q4_K_S at 15.2 tok/s remains the fastest GUI-based option and a solid fallback. The --gpu 0.78 flag is the key — without it, --gpu max silently falls back to CPU when the model + 32k KV cache don’t fit together, dropping to ~2 tok/s.
LM Studio Q5_K_M at 13.6 tok/s and Ollama 7B at 12.3 tok/s are the most stable lower-tier options. Ollama’s OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 settings are important — without them the KV cache alone takes 1.75 GB of VRAM, reducing model layers on GPU and dropping throughput to ~9 tok/s.
vLLM at 2.5 tok/s uses AWQ-Marlin kernels and FlashAttention v2 — theoretically the fastest inference stack available. But on 6 GB you need --cpu-offload-gb 2, which means 2 GiB of weight layers cross the PCIe bus every single token. The framework overhead cancels out the kernel advantage entirely.
You cannot run Ollama and LM Studio at the same time on this machine. Both want to load the model weights into NVIDIA’s 6 GB VRAM. The benchmark script handles this automatically.
For context: online Claude 3.5 Sonnet generates at 80–100+ tok/s. Local 7B/8B models at 12–23 tok/s are usable but noticeably slower for multi-step agentic work.

Fastest local setup for this machine

Based on all the testing, the fastest practical setup for running GitHub Copilot CLI locally on this Dell Pro Max 14 is:

llama.cpp (CUDA) with qwen2.5-7b-instruct@q4_k_s → 23.0 tok/s 🏆

The key steps:

Download the CUDA build from llama.cpp releases and extract to %LOCALAPPDATA%\llama.cpp\
Start the server: llama-server.exe --model <path-to-gguf> --n-gpu-layers 999 --ctx-size 32768 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --host 127.0.0.1 --port 8088
Set environment variables: $env:COPILOT_PROVIDER_BASE_URL = "http://127.0.0.1:8088/v1", $env:COPILOT_MODEL = "Qwen2.5-7B-Instruct-Q4_K_S", $env:COPILOT_PROVIDER_MAX_PROMPT_TOKENS = "32768"

The GGUF file can be reused from your LM Studio model cache — no re-download needed.

Easiest local setup: If you don’t want to manage a separate server process, Ollama with qwen3:8b → 22.1 tok/s is the runner-up with a much simpler setup: winget install Ollama.Ollama + ollama pull qwen3:8b + set-copilot-local. No binary downloads, no server management, and only ~5% slower.

Real-world caveat: These numbers are from a synthetic benchmark (3 runs × 400 tokens, fixed prompt). Real Copilot CLI sessions involve longer prompts, tool calls with JSON parsing, and multi-turn context accumulation — all of which affect throughput differently. I’ll update this section after extended real-world use.

Final verdict on local inference options for GitHub Copilot CLI on this machine:

llama.cpp (CUDA) with qwen2.5-7b-instruct@q4_k_s is the best local option today at 23.0 tok/s — beating Ollama Qwen3-8B (22.1 tok/s) and nearly 2× faster than Ollama Qwen2.5-7B (12.3 tok/s). The key insight is that llama.cpp’s direct CUDA path with Flash Attention and 8-bit KV cache extracts more throughput from the same hardware and model file than any management layer on top. If you prefer a simpler setup, Ollama Qwen3-8B at 22.1 tok/s is an excellent runner-up with a one-line install. LM Studio is a solid GUI fallback at 15.2 tok/s. The HuggingFace stack (vLLM, TGI) is currently impractical on 6 GB due to the need for CPU offload or lack of it, respectively.

Working with this from the Copilot CLI is workable, and of course it cannot compare to the powerful models that the hosting providers can run in the cloud. But for basic agentic tasks, code generation, and tool calling, it’s a workable local setup.

What I’m waiting for

The most promising path — Foundry Local with OpenVINO on the Arc 140T — is blocked by a single driver update. The Arc 140T has 16 GB of dedicated VRAM with Intel’s optimised INT4 kernels via OpenVINO, and the 7B models should run fast and stable with no memory pressure. Once Dell publishes Arc driver 32.0.101.8629 for the MC14250, that’s the first thing I’m testing.

Until then, llama.cpp (CUDA) with qwen2.5-7b-instruct@q4_k_s is the fastest local option at 23.0 tok/s — or Ollama qwen3:8b at 22.1 tok/s if you prefer a simpler setup. And for anything where quality actually matters, online-copilot is still the right call.

Show you Microsoft Certification badges in Credly

2026-02-02T00:00:00+00:00

Microsoft has stopped their partnership with Credly for hosting certification badges and instead centralized everything to the Microsoft Learn platform. However, if you still want to showcase your Microsoft Certification badges in Credly, you can do so by following these steps:

Microsoft Learn steps

Log in to your Microsoft Learn account.
Navigate to the “Credentials” section of your profile.
Open the “view all” link to see all of your certifications in case the new certification is not immediately visible (I have two renewals that took over the list).
So do not use the “Past exams” view, that will not help

Find the certification you want to add to Credly and click on “View certification details”.
Copy the url from the browser.

Credly steps

Log in to your Credly account.
Navigate to your profile in the acccount drop down menu.
Click on “Upload other badges” button.
Use the badge url option at hte bottom of the screen and paste in the url you copied from Microsoft Learn.

Click on “Add Badge” button and wait for the validation to complete.

Wait until the actual bafdge shows up in your Credly profile, takes over an hour in my experience.

GitHub Copilot - Coding Agent Examples Walkthrough

2025-12-20T00:00:00+00:00

In this post, I will walk you through an example of different ways to use the GitHub Copilot Coding Agent to automate a coding task. The Coding Agent is a powerful feature that leverages AI to help you write, review, and refactor code more efficiently. It uses a prompt coming from one of the locations below and will then have a runtime inside of a secured GitHub Actions environment to execute the steps it needs to take to complete your request, with for example a locked down network configuration to prevent it from doing weird things in your name. If you want to know more about the architecture and security model, check out the official documentation: GitHub Copilot Coding Agent.

Places where you can start a Coding Agent session from:

Your editor
Agent Task panel in a repo context
Repository creation through the UI
Chat interface in github.
GitHub Phone app

I will show some examples of these below.

Your editor

When you are in a Copilot Chat conversation you can gather all the context you need and then write the next prompt which can then be “hand off to Cloud Agent”.

Note: At the end of December 2025 when this post was written, this feature was only available in VS Code.

Here is an example screenshot of that experience when staring in Plan Mode and handing it off to Agent Mode:

Here is an example from the chat in Agent Mode itself:

Note: The “Continue in Background” button, which will let you start the same prompt with context in the Copilot CLI, which will execute in the background and notify you when done.

Agent Task panel in a repo context

On github.com you have an “Agent Tasks” tab in the UI, which allows you to start a Coding Agent session in the context of that repository. This is useful when you want to automate tasks related to that specific codebase. I use this all the time to let it work on for example a failed GitHub Actions workflow, where I can be very specific about what I want it to do.

Some example prompts that I regularly use in this context:

1. Fix the failing workflow

Look at the failing workflow here <link to the url for the workflow> or just <name of the workflow or job>. Find out why this is failing first and define if this error makes sense. If we need to fix something in the code or the workflow, do so. If we should handle this differently, explain what we should do and suggest a better approach.

2. Check multiple workflow runs

Look at the last two workflow runs for <name of the workflow>. I notice that the number of executions in the log stays the same, so it seems we are not getting new data. Find out why this is happening and fix the code or workflow to ensure we get new data on each run.    

3. Add extra functionality to the repo:

I have an example repo for hosting your own Private MCP Registry which is needed in especially an Enterprise setting. I got a question from someone how you would add a stdio type of MCP server to the configuration. So I started working on a prompt for that:

We need an example how to register a stdio MCP server. Could you please add a working example for playwright mcp server(stdio MCP server).

Because I know it is helpful to give it some extra context, I also added this to the prompt after doing a quick web search for the relevant documentation:

Here is the normal MCP servers.json config for client side configuration:

{
  "mcpServers": {
    "playwright": {
      "type": "local",
      "command": "npx",
      "tools": [
        "*"
      ],
      "args": [
        "@playwright/mcp@latest"
      ]
    }
  }
}

Since my repository also has a servers.json in it, I wanted to prevent the model from getting confised by that, so I find it a good idea to add just a bit more context by checking the MCP Private Regstry documentation. I quick web search brought me to the GitHub repository for the MCP Registry: modelcontextprotocol/registry. Instead of looking for things myself, I asked the Copilot Web Chat to find the relevant documentation for me:

This brought me to the right file: docs/reference/server-json/generic-server-json.md#L39-L62 with even the relevant line numbers included!

So this extra context was added to the prompt as well:

Use the example from: https://github.com/modelcontextprotocol/registry/blob/9afbaacdfdf8966d73de09a795076fb0386c5c3d/docs/reference/server-json/generic-server-json.md#L39-L62

The result of that PR can be found here: PR link. Note the original prompt is shown in the PR body as well. I apreciate that a lot as it shows how it transformed the PR goal from my initial prompt by leveraging additional information it found during the execution. It can also be a great way to learn how to write better prompts by seeing how it interpreted your original request, or learn how other folks approach this.

Repository creation through the UI

A lot of folks have not seen this yet, but when creating a new repository through the GitHub UI, you can also add a prompt for the Coding Agent to execute right away in the new repository context. The new repo will be created with the things you configure, and then Coding Agent will start working on your prompt right away by creating a new PR. This is super useful for bootstrapping new projects with specific requirements. Do note at the bottom left that you only have 500 characters for the prompt. I use this for initial scaffolding all the time.

Chat interface in github.com

Last but not least, you can also start a Coding Agent session directly from the Chat interface on github.com. This is useful when you want to have a conversation about a specific coding task and then hand it off to the Coding Agent to execute it. You can provide context in the chat and then use the “Continue in Cloud” button to start the Coding Agent session.

I use this either from the Chat buton in the top bar to find information in the repo, setup my context, and then prompt with something like:

Please create a new branch and add a GitHub Actions workflow that does X, Y, and Z. Make sure to follow best practices and include any necessary secrets or configurations.

Or just simply:

Hand off this task to Coding Agent.

This will trigger a background call with the conversation of you current session, and send that into the Coding Agent. Often the UI will either tell you the hand off was successful (not always does it ask for confirmation if it needs to create the task, seems to be dependent on where you trigger this task from), or not.

Also note that the chat window then often shows a direct button to go to that new agent session, but that is not always the case. Seems to be linked to the place where you trigger this from or the prompt itself.

If you do not see that button, you can always go to the “Agent Tasks” button in the top bar to find your new task, or go to the Copilot/Agents page.

GitHub Phone app

The phone app of GitHub Copilot lets you do exactly the same as the web chat interface on github.com. You can have a conversation in the chat, and then hand it off to the Coding Agent to execute your request in the context of the repository you are working with, or just directly start a new Coding Agent session from the Agent Tasks panel on the phone app:

GitHub Copilot Premium Requests

2025-06-17T00:00:00+00:00

Some important changes are happening, which means you will need to start paying for the amount of Generative AI you use with GitHub Copilot. This will finally make the end-user think about the monetary cost of executing a request with a Large Language Model, so they realize this stuff is not running for free. In that sense we have been spoiled, so it is time to take up some ownership here for the end-users. This post will give you an overview of what you need to expect and how you can protect yourself from overspending.

What is Premium Requests for GitHub Copilot ?

Tomorrow the 18th is the date GitHub Copilot Premium Requests will be enforced (see the docs here).

This post gives you an overview to know what you need to expect. Copilot Premium Requests are any request that is made to any model that is not the default (currently GPT-4o and GPT-4.1). Some features against a model will cost 1 Premium Request, some will cost more then 1 Premium Request (using GPT-4.5 will be x50!) and some are less expensive (Gemini 2.0 Flash is the cheapest at 0.25x).

I previously recorded a video on Premium Requests as well, find it here if you prefer to learn things that way:

What is a premium request

Here is an overview of the main features that will consume a premium request:

Chatting with a non default model: 1 premium request per ‘turn’ (every question and answer is a turn)
Every step in the Coding Agent: 1 premium request. The agent can decide to make multiple steps, and will thus consume a premium request for each step. Note: I hope that the agent will also come with a “max requests” setting, so that you can limit the amount of requests it can make in a single conversation and prevent it from overspending.
Requesting a Code Review in a Pull Request: 1 premium request
Agent Mode in an editor: 1 premium request per user initiated request
Copilot Coding Agent: use 1 premium requests per time you trigger it. Starting the task is a trigger. Adding a comment afterwards (and thus starting a follow up task) is a trigger and thus an extra premium request.
From the docs here, Copilot Spaces and Extensions also consume premium requests, but it is not clear how many requests they consume or for what. I can imagine that a Chat Turn against a Copilot Space will consume a premium request each time, and same for Copilot Extensions. For the latter it is also not clear if that is for both local extensions (like the @Azure extension that runs locally inside of VS Code), or also for the remote extensions that are installed as GitHub Apps in your organization.

Free plan

If you are on a Free plan, even the base model will consume a premium request if you are using the Chat feature.

Paid plans

If you are on a paid plan, then you will get a certain amount of premium requests per month. If you go over this amount, you will be charged for the extra requests at $0.04 per premium request. If that request is against a 50x model, then you will be charged $2.00 for that single request! To view the full overview of the different multipliers, see the documentation.

This is the table of the different amount of included premium requests per plan:

Plan	Premium requests	Copilot Chat in IDEs	Code completion
Copilot Free	50 per month	50 messages per month	2000 completions per month
Copilot Pro	300 per month	Unlimited with base model	Unlimited
Copilot Pro+	1500 per month	Unlimited with base model	Unlimited
Copilot Business	300 per user per month	Unlimited with base model	Unlimited
Copilot Enterprise	1000 per user per month	Unlimited with base model	Unlimited

To protect yourself or your users, you can configure a budget for premium requests in your user/organization settings. The default is $0.00, but you can set it to any amount you want. If you reach this budget, then all premium requests will be blocked until the next month.

This is now finally available inside of the new Coding Agent session panel as well:

Finding your own usage info as a User

The changes that where made where showing the multipliers to the different models to the users so they can make decisions on which model to use:

This is now visible for at least Visual Studio Code, Jetbrains IDE’s, and Visual Studio. These editors also show the setup for your account:

If you look closely, you can even see the progress bar in the middle of the screen showing that I have been using some Premium Requests already. Visual Studio shows it in a slightly different way by using the GitHub Copilot icon in the top right corner:

To find your own usage information, you can go to your User –> Settings –> Billing and then get an overview like this:

Analyzing the GitHub Copilot Premium Requests report in your organization / enterprise

Do you need to analyze the GitHub Copilot Premium requests CSV now that they will be enforced? I created a single page application (SPA) with GitHub Spark and GitHub Copilot Coding Agent to display an overview of the Premium Requests CSV that you can currently download (no API yet 😓).

Can be hosted on GitHub Pages: GitHub Copilot Premium Requests Usage Analyzer
Upload the CSV from the enterprise export (Billing and Licenses –> Usage –> Export dropdown right top) Result can be seen here:

See the repo in action here (click on the link on the right side to use your own data): https://github.com/devops-actions/github-copilot-premium-reqs-usage. It’s open source, so feel free to contribute or request features!

GitHub Copilot & Productivity

2025-06-07T00:00:00+00:00

Where the current focus on productivity is wrong

The focus on having more productive engineers is not the right way to go. I see companies struggle all the time defining what productivity even is, and then they still tend to focus on lines of code accepted as a sense of productivity. Us engineers are busy all day with tasks like requirements engineering, architectural work, documentation, and discussions on a way forward. We think about maintainability, readability, and testability of the code we write. And then in most companies we also have all the extra things next to producing code like daily meetings, stand ups, check-ins, and so on. We hope that we get enough time to actually produce some value for our end users! I even met an engineer that starts their day later on purpose, so that they can continue on working when everyone has already left for the day: that is the time they can actually focus on writing code!

We have learned several times over the last decades that just looking at things like lines of code produced is not even close to a good metric to measure productivity. And even worse: the Copilot metrics API only shows the lines of code accepted for suggestions that where accepted as a whole! So if Copilot suggests 10 lines of code and you only accept 5 (or a couple of words), the API will count this as 0 lines of code accepted. So partial acceptance is not even counted in the metrics.

Still, companies want to look at lines of code suggested and accepted, and I even have seen companies that want to focus on driving that acceptance rate up! I think that is the wrong way to go, as it will lead to engineers blindly accepting suggestions without thinking about the code that is being suggested. Looking at a suggestion and thinking if this is the thing you want to do or not, or maybe change the direction you’re going in, is a good thing! GitHub is already showing information in their API to indicate the amount of suggestions that where shown, and the amount of suggestions that where accepted (including partial acceptances). This is already a step forward, and helps you to focus on engagement instead of productivity.

What to focus on instead

I recommend that people instead focus on other things instead of lines of code to see if the use of GitHub Copilot has an impact. This includes looking at the entire Software Development Life Cycle (SDLC):

first of all you have to look at engagement and type of engagement of the engineers. Are they actually using the tool, and what kind of features are they using? I keep on running into engineers that only use the inline suggestions, and they’re not using the Chat feature at all! I find myself living in Agent Mode all day long, as I tend to know how to prompt for the right things to get the right changes, instead of building everything myself. I am thinking in business value, instead of code.
also look at when they use the tool. We keep on thinking that engineers work 8 hours a day, 5 days a week (depending on their schedule), but in reality they are only productive for a couple of hours a day. Sometimes they even struggle to have 2 hours a day of uninterrupted time to work on their tasks! Seriously, ask you engineers how much time they have to work on their tasks. I see team mates with overloaded agendas, having multiple meetings scheduled in parallel all day long. No wonder they cannot focus on their tasks! This fact alone amazes me every time, as businesses are still wanting to define and show the Return on Investment (ROI) of spending time on a tool like GitHub Copilot. The normal Business license is 19$ a month, and the Enterprise license is 49$ a month. If you calculate that you need to give the engineer an improvement and let them save time for around 1 hour a month to break even, you can see that the ROI is easily achieved. Of course this does not include the initial training time (there is quire a learning curve in my opinion!), but still overall, the ROI is easily achieved.
next is taking a look at the downstream changes in the SDLC. Are there more Pull Request created then before? Are the PR’s smaller or larger in size then before? Are there more or less build failures (indicating that users are testing in the pipeline, instead of testing locally)? Do you find more or less bugs in production? Are there more or less incidents in production? All these things indicate what the effects are of using GitHub Copilot in your organization.
we also keep forgetting that having these tools available for your engineers is almost expected these days. It is becoming a normal thing, just like having a decent environment available, together with other tools like a good IDE, a good version control system, and a good CI/CD pipeline. If you do not have these tools available, you will have a hard time attracting new engineers to your organization, and even worse, the existing talent might leave your company because of it. And you can better believe that your competitors are already offering these tools to their engineers! So can you even afford to be falling behind in this area?
next up is that we are supposed to work faster on the boring parts of our jobs, where we can implement default things like boilerplate code easier and faster. The goal here is not to be more productive in my opinion, but to get your valuable time back to work on the more interesting parts of your job! Use the time you save to work on the things that are more interesting to you, or the things you usually do not have time for. I recommend creating initiatives especially during the training phase of using Copilot to focus on those tasks that you never seem to have time for. Let Copilot help in the next two sprints with having better test coverage. Ask Copilot to locate places that can be refactored into more maintainable code, or more readable code. You will be amazed on the things it might find for you. I dare you to use Agent Mode for example and ask it for find all your TODO’s in the codebase and address them one by one (or create Issues from them by using the GitHub MCP server). Triage the low hanging fruit with Copilot and ask it for a fix. Don’t have a linter in the project yet? Add it and ask Copilot to fix all the linting issues it finds. Keep forgetting to update your dependencies? Ask Copilot for a Dependabot config for all the ecosystems you have in your repo. Always wanted to run your linting/unit tests in a CI (Continuous Integration) pipeline and never got to it? Guess what… Copilot can help here! The possibilities are endless, and you can use Copilot to help you with the boring parts of your job.

What’s next?

I jotted down how I think we need to change the way of thinking around productivity in this blogpost. Then we are liberated to start thinking about the way GitHub Copilot changes the way we produce our applications and bring actual value to our end users. And even better, how we can really level up and also enable our non-engineering team members to work more efficiently as well. I have written more about this in this blogpost GitHub Copilot - Change the Narrative.

Lets work in the right direction and embrace the future state of working by leveraging Generative AI more and more, and leave the past behind us. I think we will become more of an orchestrator of AI agents, and the agents can only go faster if we understand how to build a solid foundation to trust on (see that blogpost).

GitHub Copilot - Change the Narrative

2025-04-01T00:00:00+00:00

TL;DR:

Changing the narrative on GitHub Copilot from focus on engineers and productivity to focus on a sturdy (DevOps) foundation to be able to go faster.
Next frontier: the rest of our organization

Premise: current narrative is not helping

In my opinion we need to shift the narrative on enabling engineers to use GitHub Copilot. Currently there is a lot of focus on the engineers that can produce code easier and faster using GitHub Copilot. That leads to companies thinking they finally found a way to create 10x engineers. And even engineers are hyping this up with stories around “vibe coding” with AI: they jump on their keyboards with a prompt and accept every suggestion that is there and then run the application to figure out if their initial problem was solved or not. Eventually this path leads to disappointment: either the code does not work as hoped, or there was crucial information missing and the AI took a wrong turn somewhere. Even worse: we have seen Generative AI following the “scouting rule” where it starts to clean up after itself, changing code that did not need to be changed at all! This can lead to impact in other places in the codebase that can introduce new bugs. Even worse with all the “vibe coding” stories, we see engineers that are not even testing their code before pushing it to production. This is a recipe for disaster and will lead to a lot of frustration and disappointment in the long run.

We also tend to focus on the wrong metrics to see if the use of GitHub Copilot has an impact. I explained more in my follow up blogpost GitHub Copilot & Productivity on this topic. Focusing just on productivity is not the right way to go in my opinion. Let alone that we are not able to define what that even is.

Better narrative: focus on having a sturdy (DevOps) foundation

Instead I recommend to focus on enabling engineers to work on laying the foundation to be able to roll out their applications faster and with more trust that their application works as intended. I’ve always been a DevOps person and I firmly believe in having the basic principles in place:

Automated pipelines and testing
Everything as code or configuration
More eyes principle in place
Enough testing in place to have trust If a deployment fails for any reason, a new test should be added to the pipeline to prevent it from happening again.
Continuous monitoring and feedback loops

Only when a large portion of these fundamentals are in place, can a team of engineers roll out their changes faster, as for example the testing is in place to be able to rely on their deployments. Note that this can be achieved in multiple ways, for example with unit, regression, or integration testing. Use what works for your application.

Generative AI like GitHub Copilot can help to put these foundations in place and works really well for those kind of supporting system. I see a lot of users only focus on coding with Copilot, not knowing it can do so much more: I’ve created all sorts of scripts, pipelines, and even Splunk queries with Copilot and it works really well if you understand how to prompt for the right things.

I see GitHub Copilot as an enabler to give teams the time to get their foundation in order. Since the normal tasks are sped up, the extra time can be spent on initiatives to improve the things that often get pushed to the bottom of the backlog, or at least pushed out of the sprint. I recommend teams always embed these types of technical debt and either include it in their way of working, or specially carve out time in every sprint to improve it. I’ve seen successful teams always having a focus on 10% of their time on technical debt fixing as part of their sprint.

With the foundation in place teams can actually deliver value faster, with increased trust, and with fewer issues in production.

Next level of enlightenment: team efforts

After that we can actually start to think at the other things our teams do. On average an engineer is already lucky if they can focus on writing some code for a period of around two hours a day: ActiveState’s 2019 Developer Survey. The rest of the time is spend on preparations, discussions, architectural work, documentation, etc. And then there are always teams that spend most of their day in meetings, sometimes even overlapping and in parallel during their day. It’s always amazing to see how some organizations have made meeting more important then actual creative work like coding!

The next shift that I see happening is to the way work flows to engineers. We’re constantly busy with describing the changes we want to make to the application, and have to translate user requests into actionable descriptions. From that we need to make sure all team members understand these changes as well as the impact on the application. Teams are working with different stakeholders and perhaps a product owner that funnels the work to the team.

In my opinion we now need to focus on enabling roles like product owners to send in better scoped work. Then review the incoming work descriptions by an engineer that has expertise on the application to add the finishing touches.

When the incoming work has enough clarity AI can be used to suggest the changes translated into code changes. The next iteration of tools like GitHub Copilot has already been announced with project Padawan (blogpost here), where Copilot can try to suggest the changes on its own, rerun all the tests in the pipeline, and reiterate if needed. When the tests succeed and the changes have been implemented, it will submit a pull request for final review.

Conclusion

There will always be work for highly skilled engineers, and yes, I think they will become more of an orchestrator in the future of AI agents that can produce large parts of the work. The engineers will be in charge making sure there is enough trust in the system to proceed with the next steps. It is crucial to then have a good way in place to train new engineers to be proficient in the tools and processes that are in place and understand the impact that changes will have onto the applications they work on. Ignoring to train new engineers will lead to a lot more (downstream) issues and frustration by both the engineers and their stakeholders. Embracing the new way of working will lead to a more efficient way of working, and ultimately to a better experience for the end users of the applications we build.

Really keeping your GitHub Actions usage secure

2025-03-16T00:00:00+00:00

Last Friday what we expected happened: a much used GitHub Action got compromised, read all about it here at the StepSecurity blog where they explain that they detected the issue and jumped into action: Step Security Blog.

So what happened?

What happened? An Action that was used by over 23.000 public repositories (and who knows how many more private repositories) suffered a compromise where an attacker added malicious code to leak out repository secrets. They also updated all version tags to the new commit, so that anyone who is not following the best practices around GitHub Actions was now leaking out sensitive data, out into the open for public repos (as their Action logs are publicly available as well). The CVE (Common Vulnerability Enumerator) logged for this incident can be found here as CVE-2025-30066.

Of course this incident happened on a Friday, just before most people log off for the weekend. There is a good chance your workflow did run in a compromised state, so check your logs!

How to fix this?

StepSecurity stepped into action and published a version of the action that was not compromised in their own Actions organization: https://github.com/step-security/changed-files. They create point in time backups for several actions so that their customers can rely on secured and locked down actions, as they need to follow an approval and validation process before the copy of the action is updated. This process helps finding compromises in updates of the action, and keeps the end user safer.

In the mean time the repository has been purged of the malicious code and is back online. Good change a lot of people did not even notice their repository was leaking out information!

Fixing is nice, but only if you were aware at all

Having a backup of the action for people to switch to can be very helpful, especially since the original GitHub repository got removed leading to errors when the workflows reliant on the action started to run during this period. Knowing about the actual backup is of course then a different challenge. The news about the attack did bubble up through the GitHub Advisories Database, but you will only get a notification if you actual have Dependency Security Alerts enabled on your repository (I have a video on that on LinkedIn Learning).

How do I prevent this from happening to me?

Al this leads me to the information I wanted to share here. Given the way the GitHub Actions ecosystem works, I bet a lot of people will not be aware of what happened during this attack. Having Dependency Security Alerts on is a good way to at least get a notification, but do realize this is happening after the fact and your workflows might already have been compromised.

There are two ways to really protect yourself from these types of attacks:

Use a service from a company like StepSecurity that will validate, secure and store backups of the actions you use. This will prevent downtime in the case that the action gets removed, like it temporarily was in this case.
Setup your own backups of the Actions you (or you company) uses. Doing so gives you a point in time backup of the action code (so you can still run your workflow). An additional benefit is that you can control when the backup gets updated, and add extra reviews and security checks to it!
Next to that it is super important to use tools to know what dependencies you have in your supply chain. You should have GitHub’s Dependency Graph and Security Alerts enabled on everything (available for free for public repos). The paid version also gives you dashboards with an overview of your entire organization / enterprise.

You can also run these inventories yourself, but using for example the devops-actions/load-used-actions that I maintain.

Setting up a your own backup process

Ever since GitHub Actions became available, I have been advocating to take control over the actions you rely on. My earliest blogposts around this topic are from 2021!

This process comes with a couple of steps:

Fork actions to an organization you own. I recommend a completely separate organization so it is clear what information is stored in that location. Take ownership over the updates of the actions by using the GitHub Fork Updater. This will give you a process that allows you to review every single update, before you can use the update in your workflows. You can then add extra security checks in that process as well.
This will lead to setting up an internal marketplace for your organization to use. This will allow you to take inventory over you used actions (and for example easily show you in what workflows a certain actions is used). Additionally you can then setup a request process for actions to be added, so that you can even run all your required security checks up front.

Going through these steps is unfortunately something that every user and company needs to go through to secure their pipelines, which is a key action to take when looking at supply chain security with a security framework like slsa.dev. And do not think your security responsibility stops with taking ownership over the actions repo! When the action pulls in dependencies, you need to secure those dependencies as well! I have seen actions that download binaries or shell scripts on execution and start running them. Or what about docker images that get downloaded just in time?

Next steps

Want to know more? I have presented a session on Using GitHub Actions with Security in Mind several times. A recording from GitHub Universe 2021 can be found here.

I have also co-authored a book on GitHub Actions that also includes these security aspects. You can get that book on Manning.com.

If you want to deep dive into the security tools GitHub has available, then find my course on GitHub Advanced Security on LinkedIn Learning.

DevCon Romania 2024 - Protect yourself against supply chain attacks

2024-11-07T00:00:00+00:00

Today I shared my story on why and how we can protect ourselves against supply chain attacks. This talk was part of the DevCon Romania 2024 conference on the DevOps track.

I find it important to share with folks how to think about our supply chains in the software we deliver, from dependencies we deploy to production, to everything that touches the code before it gets into the production environment. There are so many attack vectors in our pipelines that we need to be aware of and protect against.

We had a nice full room with lots of interested engineers and developers. I tried to inspire them to take a closer look at their pipelines and see where they can improve their security posture.

You can find all of my slides in the PDF here to look at all the links I shared to level up!

Here is the link to pdf of the presentation.

Feel free to connect with me on LinkedIn and ask me questions about this talk or anything else you want to know about!

GitHub Universe 2024 - Successfully scaling GitHub Copilot to thousands of developers

2024-10-30T00:00:00+00:00

This year I got to share my story on how I think you can successfully scale GitHub Copilot to thousands of developers. This talk was part of the GitHub Universe 2024 conference in San Francisco, one of my favorite conferences to go to every year. The vibe of the GitHub community is always so welcoming and inspiring! Every one is very open and approachable, and you never know who you might bump into!

The session has been recorded and can be watched with the viewer below:

You can find the slides of my session here.

Feel free to connect with me on LinkedIn and ask me questions about this talk or anything else you want to know about!

GitHub Models and Inference API

2024-09-29T00:00:00+00:00

Need to use the Azure Inference AI SDK in Python against Azure OpenAI? Then this tip is for you! I ran into an issue converting the default examples to not run against GitHub’s Model endpoint but against an Azure OpenAI endpoint. The code example below says it all: configure your credential the correct way to get this to work.

import os
from azure.ai.inference import ChatCompletionsClient
from azure.ai.inference.models import SystemMessage, UserMessage
from azure.core.credentials import AzureKeyCredential

# Set the runtime to "GITHUB" if you are running this code in GitHub 
# or something else to hit your own Azure OpenAI endpoint
runtime="AZURE"
client = None
if runtime=="GITHUB":
    print("Running in GitHub")
    token = os.environ["GITHUB_TOKEN"]
    ENDPOINT = "https://models.inference.ai.azure.com"

    client = ChatCompletionsClient(
    endpoint=ENDPOINT,
    credential=AzureKeyCredential(token),
)
else:
    print("Running in Azure")
    token = os.environ["AI_TOKEN"]
    ENDPOINT = "https://xms-openai.openai.azure.com/openai/deployments/gpt-4o"

    client = ChatCompletionsClient(
        endpoint=ENDPOINT,
        credential=AzureKeyCredential(""),  # Pass in an empty value here!
        headers={"api-key": token}, # Include your token here
        #api_version="2024-06-01"  # AOAI api-version is not required
    )

GitHub Copilot Extensions

2024-09-14T00:00:00+00:00

GitHub Copilot is a great tool to help you write code. The next phase is starting now by enabling you to write your own extensions for Copilot! This is a great way to extend the capabilities of Copilot to your own needs. You can for example look in your own knowledge store for information, or even call into an API to get the information you need. All from within GitHub Copilot Chat itself!

Some examples of this way of working with Copilot are:

@workspace/#solution - this works inside of your IDE
@github - this works with the GitHub API and you can ask questions like “get all issues for this repo” or “get all PRs for this repo”. Note, only available for GitHub Copilot Enterprise.

Next to that we are seeing some extensions coming out. There are two types of extensions you can build at the moment;

VS Code extensions for Copilot: this is a way to extend the capabilities of Copilot in your IDE and it has access to information in your IDE, like the files you have open. Do note that this extension will only work in VS Code, and in none of the other supported IDE’s for Copilot. An example of this type of extension is the @azure extension (see announcement), that allows you to interact with an Azure knowledge base (currently in private preview). Learn from Matt Olson’s post how to build one of these.
GitHub Copilot Extensions: these live somewhere online (where ever you decide to host them) and can be called from within Copilot Chat in any UI that supports it (currently only VS Code and the GitHub web interface on github.com). That is what this post is on!

Writing a GitHub Copilot Extension

Get started to write your own Copilot extension by reading the announcement and the documentation. From that you can create a GitHub App (public or private to your organization) and go to the Copilot part of the settings (see next screenshot).

After creating the GitHub App, you need to configure the App with a public endpoint that will be called by Copilot. This endpoint should be able to handle the requests from Copilot and return the results in the correct format. The easies way to get started is to create a Codespace and use the Codespace URL when you run your solution as the endpoint for your extension:

The url in the screenshot is the same as the one I am using in the ‘Callback URL’ setting on the General tab of the GitHub App settings.

There are a lot of code examples in the copilot-extensions organization, like the following:

Blackbeard extension (talk like a pirate)
RAG extension (implement ‘retrieval augmented generation’ to build up your response in several steps)
Function calling extension (retrieve a function from the user prompt, like “what is the weather in Amsterdam”)
GitHub Models extension (use the new models API’s to talk to an LLM) These are fine to spelunk in, but as soon as you want to do something else then adding a system prompt and stream back the LLM response, you will find some very rough edges.

Instead I recommend to go straight for the Copilot Extension SDK library. At the time of writing this SDK in in early alpha stage, but it solves several issues that you will run into when you start building your own extension. I’ve been adding some examples to that repository to help you get started.

Writing the extension code

Using the examples in the SDK repo should get you started. The main things you need are in the README.md in the examples folder. Run the npm install and npm run watch commands to get the examples running. Don’t forget to make your Codespace port public (so that Copilot can send its messages to it), and to configure the Codespace port URL in the GitHub App settings.

You need to handle a POST event into your application, and then return text and a createDoneEvent to indicate your response is complete.

If you want to, you can create a createAckEvent that will tell Copilot you have received the request and are working on it. It will display that status to the user and is a nice way to let the user know you are working on their request, especially when you will be going out to an LLM to generate some text for example.

After the acknowledge event, you can add createTextEvent("your text here") to add custom text to the output. What you do next is up to you. There are several options already supported in the SDK:

Call the GitHub Copilot API with the user prompt and get the response back (limited model options)
Call into an API you have access to
Ask the user to confirm something (example later in this post)
Call into GitHub Models to get a response from an LLM with more model options

Running the extension server side in a Codespace

Working on the extension inside of a Codespace is the most easy way to develop an extension: you receive the traffic directly and you can debug all you want. Do note that I’d recommend using a separate GitHub App for this compared to your production version, as ALL the traffic from the GitHub App will be sent your way, which can be confusing to determine what is coming from.

Don’t forget to make the port public in the Codespace settings, and to configure the URL in the GitHub App settings + the Copilot settings section of the GitHub App. If you do not have a Copilot section in the App, then your App is not flagged in for Copilot extensions.

Working with the extension

As soon as you install the GitHub App in an organization (has to have Copilot licenses enabled), every user in that organization can ask it questions. Do note that GitHub Copilot always checks the incoming prompt for the @ symbol, so you need to start your prompt with @your-app-name to trigger the extension. Once you have done so, the entire conversation will be send to your extension, as there is only one extension per thread that can be interacted with. This is probably for security reasons I think, as the extension will get the entire conversation and can do with it what it wants. You can imagine if your extension would be able to read the conversation from another app, sensitive information could be leaked.

You can now try out the extension in every place that supports it! At the time of writing this post, it is available in VS Code (+ Insiders) and the Web UI on github.com. You can ask the extension for something and it will send the prompt to your extension. Do note that GitHub Copilot by itself checks the incoming prompt with its content filter, to prevent harmful messages from being answered. Only when you pass the filter (like for example it needs to be a coding related prompt), the extension will be called. Same goes for the data you send back, it will be checked by GitHub Copilot before it is shown to the user. I first started with just “hi matey” as a response, but that was blocked and thus not shown to the user.

When you first interact with a GitHub Copilot Extension, you get a message asking for consent to share the conversation with the extension. Likewise the extension will get your user information (GitHub handle, Location (approximate)). This is to be able to personalize the response to the user. You can always revoke this consent in the GitHub settings.

Adding system prompts

import { prompt, createAckEvent, createDoneEvent, createTextEvent, parseRequestBody } from "@copilot-extensions/preview-sdk";

const tokenForUser = request.headers['x-github-token'];
const payload = parseRequestBody(body);

let result = await prompt(payload_message.content, {
    messages: payload.messages, // add the current message thread in here for the LLM to use
    token: tokenForUser
});
response.write(createTextEvent(result.message.content));
response.end(createDoneEvent());
console.log('Response sent');

Working with User Confirmation

You also have the option to work with user confirmation in your extension. Copilot Extensions are really shaping up to give you endless possibilities like making it an ‘agent’ that can do things for you. Asking a question about a repository and an API that does not exist? Let your Copilot Extension fill an issue in a repo with the question and the answer from the API. Or ask the user to confirm something, like in the following example:

import { getUserConfirmation } from "@copilot-extensions/preview-sdk";

response.write(
    createConfirmationEvent({
    id: "123",
    title: "Are you sure?",
    message: "Create an issue with the missing API surface.",
    }),
);

Conclusion

GitHub Copilot Extensions are a great way to extend the capabilities of Copilot to your own needs. You can for example look in your own knowledge store for information (API, Database, an LLM with your own data). All from within GitHub Copilot Chat itself! The possibilities are endless and I am looking forward to see what you will build with it.

I am planning to integrate this with our internal knowledge base, starting with something simple like “What is the GitHub handle for Rob Bos?” and then expanding from there. I am also looking forward to see what other people will build with this, as the possibilities are endless.

What ideas do you have to build with this? Let me know in the comments below!

GitHub Copilot Chat - Power User example

2024-06-19T00:00:00+00:00

Learn how I use GitHub Copilot Chat to my benefit and see the end to end flow on creating a script to call into the GitHub API.

I’m showing my prompts and the results, as well where Copilot failed to help me in one go. With spening more time on my propmt up front, I managed to get a much better result that with the first attempt!

Video

In this video I show how I use the chat feature to let Copilot write an entire script for me in the form of calling an API (in a loop on a certain input file), summarize the results based on the response, and calculate both an average inside the loop (per item) as well as an overall average.

GitHub Copilot - Levels of enlightenment

2024-06-07T00:00:00+00:00

I’ve recorded a video series on the lessons I learned to get the most out of GitHub Copilot. This series is called “Levels of Enlightenment” and can be found in this YouTube Playlist.

An overview of these level of enlightenment can be seen in this image:

Overview

Learn about my journey using GitHub Copilot over the last year and a half. Each video in this series shows one of my lessons learned in those “aha” moments and how it helped me getting more value out of it.

Follow along and pick something up that helps you getting to the next level of enlightenment!

1. Explain what you want to achieve

In this video I show how changing the way I write my code helps me get more descriptive with GitHub Copilot so the suggestions I get out of it become a lot better! View the video on YouTube.

2. Know your context

In this video I show you how to think about the context GitHub Copilot uses and how to use that knowledge to get better results out of it! View the video on YouTube.

3. Copy method calls as comments

In this video I show how changing the way I go from a method call to the method implementation speeds up my way of creating code! View the video on YouTube.

4. Top-down programming instead of bottom-up

In this video I show how thinking about my code implementation from top-down instead of bottom up helps me use GitHub Copilot in a better way! View the video on YouTube.

5. Typos in your prompt do not matter

In this video I show you to stop caring (to much) about typos in your prompt. Took me a while to internalize that this is a good way to speed up your prompting! View the video on YouTube.

6. Use the chat

In this video I talk about using the Chat functionality to your benefit: instead of relying on suggestions while you are typing most of the time, I have switched to use the Chat for most of the things, as that will get me faster results so I can focus on what I want to do, instead of creating the right syntax. View the video on YouTube.

7. #file is your biggest friend

In this video I show you how to get the most of of GitHub Copilot by being very descriptive with your prompts and to include specific context by using #file or #selection, instead of relying on the editor to guess what parts of your codebase are important. View the video on YouTube.

8. Accept the truth

In this video I talk about accepting the fact that Copilot is based on Large Language Models, which means it is both non-deterministic as well as not 100% correct. You remain the pilot and are driving the conversation, so you are also the professional guiding the code in a direction. It can take a while to get the code correct :smile: View the video on YouTube.

9. Be smart

In this video I talk about being smart when using GitHub Copilot. Think of novel use cases and how to get more done by Copilot instead of writing the code yourself. Use it for exploration and finding new ways of doing things! View the video on YouTube.

GitHub Copilot Power User example

2024-06-05T00:00:00+00:00

Learn how I use GitHub Copilot to my benefit and see the end to end flow on a code refactoring example where I extract a script from a yaml pipeline (GitHub Actions) into a separate file.

I’m showing my prompts and the results, as well where Copilot failed to help me in one go. With a follow up question and call, I still managed to get the result that I wanted!

Video

Slides

The slides for this video should be self explanatory if you prefer to read instead of watch. You can find them here: GitHub Copilot - Power user example.

GHAS Code Security Configuration

2024-04-27T00:00:00+00:00

GitHub Advanced security has gotten quite a big update in public beta at the moment that helps with the rollout of Advanced Security features across your organization. It is called “Code security configurations” and it allows you to set up a default configuration for some or all repositories in your organization.

Previous situation

Up to now there where only three options during the rollout:

Enable features for new repos only (organization level setting).
Enable GHAS for ALL repos in one go (rather intrusive and needs extensive training up front of all developers).
Enable features on a per repo basis (slower and fits a team-by-team onboarding plan) .

So this means either going slow(er) on a team-by-team basis or going fast and enabling everything at once. That last one is not really an option in my opinion, unless you want to alienate a lot of engineers from the security process. I always recommend giving people training and discuss the expectations that you have of what they will need to do / change when you enable Advanced Security. Features like Secret Scanning with Push Protection are of course low hanging fruit, and can be shared internally without to much training: share a very short video of what will be turned on, why you’re doing that and the consequences of that (blocking secrets from entering the codebase). Then explain what you expect people to do with these alerts, to prevent them of just ignoring everything that comes in and marking it as “closed with will fix later” and then not doing anything about it.

New features in the beta

The new features are available on the organization level (not on Enterprise yet) and are called “Code security configurations”. This allows you to set up a default configuration for some or all repositories in your organization. This is a great way to set up a default configuration for all repositories in your organization, and then tweak it for specific repositories if needed.

New policies

It starts having ‘policies’ that you can deploy to repositories. That matches the administrative terminology that GitHub uses everywhere, so that makes sense.

Since this is new, you only have the ‘GitHub recommended’ policies as well as a ‘legacy’ policy. The legacy policy just contains what settings you had set before to be the default for new repositories.

The GitHub recommended policies will turn on the suggested settings for Dependabot, Secret scanning, and Code scanning. The UI does not really show you what that means at all, and I am very curious to learn what will happen if this recommendation changes in the future.

During testing this turned out to be enabling the following settings on the repo level:

Dependency graph
Dependabot alerts
GitHub Advanced Security on the repo (will claim license seats)
CodeQL analysis with the default configuration, so running in the background (not visible in Actions) and non blocking in a pull request
Autofix for CodeQL is enabled (curious to see how that works with PR’s then! Probably follows the Check runs threshold) based on the alert that is generated, but I also would like to see if it annotates the PR at all.
Secret scanning and Push protection (Great! Low hanging fruit in my opinion)

That least seems like a very sensible default to me, although I wish this would be more visible in the UI. I would like to see a list of what is enabled and what is not, so that you can gauge the consequences are of enabling these features.

Creating a policy

The screen to create a new policy is a dream. It is very clear and easy to use. The different functionalities (or GHAS ‘pillars’ as I like to call them) are grouped together for easy understanding, and then you can still enable all the features we had before. The “advanced security” label also nicely shows which of these features will require GHAS licenses for those repos.

Also note at the bottom the flag ‘Use as default configuration for newly created repositories’. This is a great way to set up a default configuration for all repositories in your organization, and then tweak it for specific repositories if needed. The dropdown on the right hand side has the option to do this for only new public repos, or only for new private and internal repos. I assume the ‘private and internal’ option will also include ‘public’ repos, but I was not sure from the UI, so that could be improved. After testing it proved that this is indeed the case!

Applying a policy

You can apply policies by filtering the repo overview to the repositories you want to target with the new powerful filter bar that is available in certain places in the GitHub UI. Filter to the repos you want to apply the policy at and then click the ‘Apply configuration’ button. This will show you the policies that are available and you can select the one you want to apply.

After applying the policy to a repository, the repository admins will be able see that there has been a policy, although this is so subtle that I looked over it the first three times. I can also imagine an update later on in this same spot that lets repo admins choose from internal policies and adopt them easily during on-boarding.

Repo level admins can still make changes to their repo settings and even lower the security settings if they want to. From an organization management standpoint I’m not sure if I like this or not, as this gives repo admins the option to lower our security standard. I would like to see a way to lock down the settings to the organization level, but I can also see that this means locking repo admins out of certain options. On the organization level you will see it when a repo with an applied policy has removed settings so that the repo does no longer match the policy. This information does not show up in the Audit log at the moment, so you will have to keep an eye on the settings yourself. I assume this is one of those things that will be added before this feature goes GA. With that I can at least monitor if the settings are still in place and act on it if needed.

Do note that the setting “private vulnerability reporting” is not part of the policy settings. It is a feature outside of GHAS so I think that is the reason to not include it. As an admin I would like to see this in the policy settings as well, as it is a security feature that I would like to have enabled by default and promote everywhere. It depends of course on the maturity level of the organization, if they have Enterprise Cloud or Server, or maybe have a different internal process for reporting security vulnerabilities. So far I have seen a lot of companies not even having a policy, so having this on by default makes a lot of sense for them in my opinion.

Limitations

There are at the moment some limitations of using the new policies. This makes sense as it is only a beta, so some updates might come in the future. The limitations are:

Not available in “User space”, so not on your own repos. Which makes sense as this will probably (guessing here) be a paid feature for Teams and higher plans.
For free organizations it is only available for public repos, but not for private repos. It’s great this is free for public repos, it makes it easier to roll this out on all your repos and up your security game!

Summary

Overall I really like the new way to make it faster to rollout defaults in your organization. I can image creating several policies in the form of level 1 to 3 at your company, and graduating teams from level to level:

Level 1: Basic security settings, like secret scanning and push protection, together with the Dependency Graph and Alerts. It makes sense to enable Code Scanning in here to at least get the alerts, but having a communication plan with your engineers is key here to let them know that alerts will be found, and what we expect them to do with them (or not).
Level 2: Include Dependabot security updates and start blocking on PR’s with the Dependency-Review action (not part of the policy options by the way).
Level 3: Also include version updates as well as the CodeQL analysis with blocking alerts in PR’s.

This will really be helpful rolling out GHAS features across an organization, as we can roll this out really easily on a team-by-team basis in one go, instead of having to go repo-by-repo.

Running GHAzDo CodeQL on self-hosted runners

2024-01-22T00:00:00+00:00

When you start running CodeQL scans on your Azure DevOps environment on self-hosted runners, you’ll learn that you have to do one extra step and that is install (and keep up to date!) the CodeQL bundle on your self-hosted runners.

If you don’t do this, you’ll get an error like this:

Following the url in the error will bring you to the docs where you might notice the following three bullets:

Pick the latest CodeQL release bundle from GitHub.
Download and unzip the bundle to the following directory inside the agent tool directory, typically located under _work/_tool: ./CodeQL/0.0.0-[codeql-release-bundle-tag (i.e. codeql-bundle-v2.14.2)]/x64/
Create an empty file at ./CodeQL/0.0.0-[codeql-release-bundle-tag (i.e. codeql-bundle-20221105)]/x64.complete

Getting this configuration right took me longer then I like to admit, so here it is for future reference to get this correct next time:

1. Download the latest version of the bundle

Get the bundle itself for the OS and bitness of the OS the runner is using. In my case I was executing the runner on my Windows 11 laptop, s I needed codeql-bundle-win64.tar.gz:

Note: If you need to automate this, then use the link https://github.com/github/codeql-action/releases/latest to quickly get to the latest version of the bundle.

2. Place the contents of the bundle in the right location on the runner

Go to you runner and get the subfolder codeql into the correct location. This took a couple of tries because the docs are confusing.

The correct location looks like this: runner\_work\_tool\CodeQL\0.0.0-codeql-bundle-v2.15.5

Where I have the following remarks:

runner: location where I have installed the runner service itself. This folder name is for you to choose.
0.0.0-codeql-bundle-v2.15.5: this is the version of the bundle you are using. Since I downloaded v2.15.5 and this bundle is used for all previous versions, this is used in the folder name as well. During testing I found that 0.0.0-v2.15.5 also works.

3. Create the `.complete` file at the correct folder

As the docs state, there needs to be a file with the bitness.complete name in the right location. I made the mistake of placing that into the x64 folder, but it needs to be in the version folder. So in my case it needs to be in 0.0.0-codeql-bundle-v2.15.5.

With that, my folder structure looks like this:

Best viewed blogposts of 2023

2023-12-20T00:00:00+00:00

It’s that time of the year again! Time to look back at the most viewed blogposts of the year. I’m always amazed at the number of views some of these posts get. I’m also amazed at the number of people that find my blogposts useful. I’m glad I can help out!

Here is the overview:

#	Title	Published	Description	Views
1.	2022	GitHub workflows not starting	🙀The most viewed for a reason! When your GitHub workflows are not starting up, there can be quite a few things causing this. Read this post to find out more!	17.2k
2.	2022	GitHub access tokens explained	Personal Access Tokens can be a quick an easy way to test things out, but should not be used in production scenarios for a number of reasons 👺. In this post I explain why and what alternatives there are	6.7k
3.	2020	.NET core: nesting appSettings.json in the IDE	A three year old post that I jotted down for future reference. Seems like other people still have that same need! 🎉	4.5k
4.	2020	GitHub Actions with private runner: deploy to IIS	🔍 Using a private runner for GitHub actions unlocks tons of scenario’s to deploy into your private production environment as well. Some folks thing it’s hard to do so on Windows and a classical setup with IIS. In this post I explain the moving parts in making this happen! IIRC I used the same setup to stop, install, and start Windows Services as well. 🦖🌴	4.2k
5.	2020	Authenticate to Azure DevOps with a Personal access token	Authenticating to Azure DevOps using a personal access token takes a while to figure it out. For future reference I jotted it down. Still helping other folks after 3️⃣ years!	3.5k

Looking at these things is always interesting for me. For the last three years I have been blogging mostly on GitHub, Advanced Security, and some Azure DevOps (mostly on Advanced Security on Azure DevOps). The top post is about a typical scenario that I see a lot of folks struggle with, so I am very happy this post gets found a lot to.

Three of the top five posts are from 2020! That means a lot of folks still have the same need to know this as I did when I wrote the post. This is also the reason why I keep on blogging about these things. It means I can find these things myself, and apparently other folks find them useful as well 🤗.

My top post from 2023 isn’t even in the top 10 of best viewed posts! It’s the post on GitHub Advanced Security on Azure DevOps, where I explain the new features on Azure DevOps that where announced in the beginning of the year. These security features are available for anyone to use since November 2023 (if you need help with it, let me know!).

Blogposts from this year 📝

Looking back at blogging in 2023, I wrote 12 blogposts and shared my lessons learned or how I implemented things for customers. So even though I average a post a month or so, people still are able to find them and use them for their own benefit.

For me the ones I looked at this year that stand out are:

GitHub Advanced Security on Azure DevOps - I’m a fan of Advanced Security, so this functionality becoming available for Azure DevOps is a big win for me.
How to write to the GITHUB_STEP_SUMMARY - Very helpful to have a mental image of the moving parts. Instead of looking at the sparse GitHub docs for this, I refer to my own post almost quarterly at the moment.
GitHub container based action cleanup - I’m a fan of container based actions, but it can be a bit tricky to clean up after yourself in a container based action.
Get alerts from GitHub Advanced Security for Azure DevOps - In the past months I created an extension for Azure DevOps to show alert information in your dashboards, as I was missing that information in the default functionality. This post explains how I you can get those alerts from the REST API yourself (written when the docs were not avaiable yet).

Website stats 📈

For visitor numbers I use Plausible.io and those numbers are publicly available here.
A few of the numbers can be found below:

New LinkedIn Learning course! GitHub Advanced Security for Azure DevOps

2023-12-20T00:00:00+00:00

GitHub Advanced Security for Azure DevOps

My newest LinkedInLearning Course is available now! This course is all about GitHub Advanced Security for Azure DevOps. It’s a great way to learn how to use the GitHub Advanced Security features in your Azure DevOps pipelines, with practical examples. There is even an example repository that you can use to follow along with the course. You can find the course on LinkedIn Learning: GitHub Advanced Security for Azure DevOps.

If you want to read more about GitHub Advanced Security for Azure DevOps you can read my post here. That’ll show you what this new functionality in Azure DevOps is all about.

Course overview

In the course I’ll take you through the three main pillars of Advanced Security for Azure DevOps, that can be used separately and help you shift your security practices left, as is a recommend practice in DevSecOps.

The three pillars of Advanced Security are:

Dependency scanning: know what dependencies you are using, and if they have any known vulnerabilities in the versions you use.
Secret scanning: make sure you don’t accidentally commit secrets to your repository, and if you do, get notified so you can revoke the secret.
Code scanning: scan your code for known vulnerabilities, and get notified if you do.

These features can help you shift your security practices left, and thus help you find and fix security issues earlier in the development process. This can help you greatly reduce the cost of fixing security issues.

Next to explaining how to use the features Advanced Security has, I also show examples of how to get an overview of your progress and alerts in either Defender for Cloud in Azure, or by using my GHAZDo Extension in Azure DevOps.

Keep on learning!

Dependabot alerts triaging in GitHub

2023-10-07T00:00:00+00:00

The GitHub UI displays a couple of helpful tips to use in triaging your Dependabot alerts which are super helpful. Unfortunately the User Interface does not show these filters in the filter bar yet, so I wanted to have a better overview of the filters I could use. I’ve listed them below:

Only show alerts where your code is using the vulnerable calls of the dependency

This is very helpful in triaging the open alerts. Currently in limited preview for certain languages and certain package ecosystems. I hope they will expand this to more languages and package ecosystems.

Instead of wading through all the open alerts where your code might (currently!) not call into the vulnerable part of the dependency code, you can filter down the list of alerts to the things you are calling. Use this filter:

is:open has:vulnerable-calls

Filter on Dependabot auto dismissal

Dependabot now has new functionality to auto dismiss alerts that have low or medium severity. This is a great way to reduce the noise in your alerts list so you can filter on the important issues. You can filter on these alerts with this filter:

is:closed resolution:auto-dismissed

Filter on only runtime dependencies

This is a great way to filter out the development dependencies from the runtime dependencies. This is the case for example when you use NPM as a package manager. You can use this filter to only show the runtime dependencies:

is:open scope:runtime Do make sure that you are not accidentally publishing to production with development dependencies. This is a common mistake that can lead to security issues, as the files are then still on disk, and could be used as an attack vector.

Filter on vulnerable dependencies where a patch is available

This is helpful to focus on quick wins. Since there is a patch available that fixes the vulnerability, getting these dependencies upgraded can lead to fast results. You can use this filter to only show the alerts where a patch is available:

is:open has:patch

Get alerts from GitHub Advanced Security for Azure DevOps

2023-09-02T00:00:00+00:00

GitHub Advanced Security for Azure DevOps (GHAzDo) builds on top of the functionality for GitHub Advanced Security and is giving you extra security tools to embed into your developer way of working. It’s a great way to get started with security in your Azure Pipelines and Azure repos and I’ve written about it before in this blogpost.

Loading the alerts from the API’s

Before starting with the Advanced Security API’s you’ll need to get the ID for the repository you are working with. You’ll need to have the project name and the repository name itself. With that you can make this API call:

https://dev.azure.com/<PROJECT NAME>/_apis/git/repositories?api-version=7.1-preview.1

It will return you the list of repos the token you are using has access to. The repo object will look like this:

{
            "id": "5e5195e1-1b44-4d4b-9310-5d33ee2c4dc",
            "name": "eShopOnWeb",
            "url": "https://dev.azure.com/raj-bos/3651f6f0-74e5-48d7-8ff9-d62ae2464b1/_apis/git/repositories/5e5195e1-1b44-4d4b-9310-5d33ee2c4dc",
            "project": {
                "id": "3651f6f0-74e5-48d7-8ff9-d62ae2464b1",
                "name": "GHAzDo trial",
                "url": "https://dev.azure.com/raj-bos/_apis/projects/3651f6f0-74e5-48d7-8ff9-d62ae2464b1",
                "state": "wellFormed",
                "revision": 141,
                "visibility": "private",
                "lastUpdateTime": "2023-07-26T18:51:26.227Z"
            },
            "defaultBranch": "refs/heads/main",
            "size": 62610330,
            "remoteUrl": "https://raj-bos@dev.azure.com/raj-bos/GHAzDo%20trial/_git/eShopOnWeb",
            "sshUrl": "git@ssh.dev.azure.com:v3/raj-bos/GHAzDo%20trial/eShopOnWeb",
            "webUrl": "https://dev.azure.com/raj-bos/GHAzDo%20trial/_git/eShopOnWeb",
            "isDisabled": false,
            "isInMaintenance": false
        }

From this you need the “id” field of the response. That can then be injected into the next API call:

https://advsec.dev.azure.com/<PROJECT NAME>/<REPO NAME>/_apis/AdvancedSecurity/repositories/<REPO ID>/alerts?top=50&orderBy=severity&alertType=3&ref=refs/heads/main&states=1

The filtering options determine the response you will get back:

Param	Description
top	The number of alerts you want to get back.
orderBy	The field you want to order the results by.
criteria.alertType	The type of alert you want to get back. 1 = Dependency, 2 = Secrets, 3 = Code scanning
criteria.ref	The branch you want to get the alerts for, only needed when looking at code scanning alerts
criteria.states	The state of the alerts you want to get back. 1 = Open, 2 = Closed

You can also leave the alertType away from the url to get all alerts in one go. Do be aware that it will result in a different value for the alertType in the response, instead of the numbers listed in the table above:

Value	Type
code	Code scanning alert
secret	Secret scanning alert
dependency	Dependency alert

Slides for Developer Week '23

2023-06-29T00:00:00+00:00

I had the pleasure of speaking at Developer Week ‘23 in Nuremberg, Germany (link) this year. Below you can find the slides that go with my sessions that have all the links you’ve seen in the session.

GitHub Actions: Beyond CI/CD

With GitHub Actions you can do so much more then just CI/CD! I’ve validated the links on my blogposts, automated my issue management and provided easy configuration of my trainings that sets up entire environments for the attendees!

Join this session for more examples how you can use GitHub Actions to make your life easier!

You will learn:

GitHub Actions can do more then CI/CD
Real-life examples of automation

Download the slides from this link.

Protect yourself against Supply Chain Attacks

As an industry, we are using third party packages and building components for lots of things. In this supply chain, there are lots of places for vulnerabilities. They can then be used to attack your DevOps pipelines!

In this session, I will go over some common attack examples and show you a way to prevent them from happening. There are frameworks available in the industry that guide you through the process of becoming more mature in protecting not only your source code and application but also the packages you use and the pipelines you build them with. I’ll demo some of GitHub’s features that help preventing these types of attacks.

You will learn:

Why do we need to protect ourselves?
What things do we need to think about?
A framework to guide you through improving your security stance

Download the slides from this link.

Protect your Code with GitHub Security Features

Creating modern software has a lot of moving parts. We all build on top of the shoulders of giants by leveraging closed/open source packages or containers that other people have shared. That makes securing our software a lot more complex as well!

In this session, you’ll learn what possible attack vectors you need to look for, how to protect yourself against them and how to leverage GitHub’s features to make your life easier!

You will learn:

Commit signing
Dependabot
Secret scanning
Code scanning using CodeQL

Download the slides from this link.

Cleaning up files changed by a GitHub Action that runs in a container

2023-06-21T00:00:00+00:00

A common issue we see with self-hosted runners is that they can leave behind files that were created or modified by the action. This is because the action runs in a container and the container is using a root user to do its work.

The GitHub documentation says to run the the runner service as root as well, to have the most compatibility with most runners. This is not a good idea, as it can lead to security issues, so a lot of people run the runner service as a non-root user. This can lead to permission issues when the action touches files in the workspace directory that get’s mounted.

Photo by Aakanksha Panwar on Unsplash

Examples

One common example is the super-linter action. Depending on the checks that run, it can touch files on disc that then get owned by the root user.

Another example is running the entire job inside of a container, with using the keyword container on the job level:

jobs:
  build:
    runs-on: self-hosted
    container: ubuntu:22.04
    steps:
      - uses: actions/checkout@v2
      - run: echo "Hello World"

The checkout action will create a .git directory in the workspace directory with the repo contents, which will be owned by the root user as the ubuntu-22.04 container runs as root. The job itself will complete just fine, but the next time you run another job for this repository on the same runner, the checkout action will try to cleanup the $GITHUB_WORKSPACE directory to get a clean starting point, and will fail with a permission error since the job is not running as root, but as the non-root user the runner service is executing under.

There are multiple ways to tackle this issue:

Prevent it from happening: Run the runner service as root and run the job as root as well. This is recommended by GitHub to prevent this from happening, that is how the service has been designed. Most people I talk with do not agree since the user has to much permissions and it can lead to security issues.
Change the action to not run as root, which is not realistic when using actions from the public marketplace.
Get the user to cleanup inside of the container, or add a cleanup action in their job.
Add cleanup configuration on the container level configuration in the runner
Add cleanup configuration on the runner configuration itself
Do not run the container with any persistence: run as ephemeral, where the runner is alive for a single job execution, and then gets completely deleted and cleaned up so there is no reuse.

I’ll go over the last 4 options in this post.

Get the user to cleanup inside of the container, or add a cleanup action in their job.

You could hunt for the jobs that cause this issue, and ‘ask’ the user to add a cleanup action in the job itself. There is for example a cleanup action available in the marketplace that runs in a container that uses root, and that cleans up the workspace directory. This is not a good solution, as it requires a lot of manual work, and you will have to keep track of the jobs that cause this issue. The following options are probably manageable in a better way.

Add cleanup configuration on the container level configuration in the runner

You can configure the runner with specific customization commands that get executed before and after the job, as well as other options:

prepare_job: Called when a job is started.
cleanup_job: Called at the end of a job.
run_container_step: Called once for each container action in the job.
run_script_step: Runs any step that is not a container action. See the entire configuration and examples here. This is rather complex and more suited for configuring sidecars or other complex scenarios. I prefer to use the next option.

Add cleanup configuration on the runner configuration itself

By setting 2 environment variables before the job starts, you can run specific scripts before and after the job starts. The completed job can then be a docker run command that mounts the $GITHUB_WORKSPACE and cleans everything up while running as root. This is the easiest option in my opinion:

ACTIONS_RUNNER_HOOK_JOB_STARTED: The script defined in this environment variable is triggered when a job has been assigned to a runner, but before the job starts running.
ACTIONS_RUNNER_HOOK_JOB_COMPLETED: The script defined in this environment variable is triggered after the job has finished processing. Find more information in the documentation here.

Do not run the container with any persistence

The best advice is to always start runners as ephemeral, where the runner is alive for a single job execution, and then gets completely deleted and cleaned up so there is no reuse. You will need to provide a mechanism to cleanup the Virtual Machine or Container setup where you execute the runner on. The best method I’ve found is to use Actions Runner Controller where the runner is executing inside of a container that gets deleted after the job is done. This will prevent any data to linger around on the runner as well, and gets you a clean execution environment every time.

There might be some valid reasons to still have a persistent runner, but I would recommend to use ephemeral runners as much as possible.

Writing to the $GITHUB_STEP_SUMMARY with the core npm package

2023-06-08T00:00:00+00:00

Every time I need to write to the GITHUB_STEP_SUMMARY in GitHub Actions from the actions/github-script action (or from Typescript), I need to search for the blogpost that announced it’s existence. So I’m writing this blogpost to make it easier for myself to find it a lot easier, including some working examples.

Photo by Markus Winkler on Unsplash

The code for the summaries lives in the actions/core package on npm, but figuring out how to use it can be a bit hard. The only example I’ve seen is in the blogpost I mentioned.

  import * as core from '@actions/core' 
  
  await core.summary
  .addHeading('Test Results')
  .addCodeBlock(generateTestResults(), "js")
  .addTable([
    [{data: 'File', header: true}, {data: 'Result', header: true}],
    ['foo.js', 'Pass ✅'],
    ['bar.js', 'Fail ❌'],
    ['test.js', 'Pass ✅']
  ])
  .addLink('View staging deployment!', 'https://github.com')
  .write()

This does a lot of things at the same time, but we get the general idea that you can:

add headings
add code blocks
add tables
add links And at the end you need to write the summary itself, which will be added to the file in the GITHUB_STEP_SUMMARY environment variable.

Working with the table output

There are no methods to break the table into chunks, like:

Add a header
Add a row

The only method there is, is adding the table in one go, with each row as an array of objects, and some configuration in the first row as that will define if the cell is a header or not. So assuming you have an array of results that you want to show, you can convert that array with properties into an array of rows, with each property value being an item in the row array.

The interesting thing I ran into, is that the row cells must be a string. Sending in integers for example does not work. Take the following example:

await core.summary
            .addHeading('Example')
            .addTable([
                        [{data: 'Topic', header: true}, {data: 'Count', header: true}, {data: 'Public', header: true}],
                        ['foo.js' , "1", "2"],
                        ['bar.js' , '3', '4'],
                        ['test.js', 100, 200]
                      ])
            .write()

In this example, all rows will be added to the summary, and as long as the content is a valid string, it will be shown in the table as well. In this example, the values in the last row are integers, and they will be not visible in the table.

A full example of creating the header array with hardcoded cells, and then adding the rows from an array of objects can be seen below. Here I have an array stored as output in a previous step, so I read that file and map it (as string values!) to an array containing the rows. The next step is to join the two arrays (header + summary) and pass that to the addTable method.

- name: Show information in the GITHUB_STEP_SUMMARY
    uses: actions/github-script@v6
    env:
    summaryFile: $
    with: 
    script: see below in other markup for better readability

        const fs = require('fs')
        const summary = fs.readFileSync(process.env.summaryFile, 'utf8')

        // make the heading array for the core.summary method
        const headingArray = [{data: 'Topic', header: true}, {data: 'Count', header: true}, {data: 'Public', header: true},{data: 'Internal', header: true},{data: 'Private', header: true}]
    
        // convert the summary array into an array that can be passed into the core.summary method
        const summaryArray = JSON.parse(summary).map(t => [t.name, t.count.toString(), t.public.toString(), t.internal.toString(), t.private.toString()])

        // join the two arrays
        const tableArray = [headingArray, ...summaryArray]

        await core.summary
                .addHeading(`Topics used on repos in the [${process.env.org}] organization`)
                .addTable(tableArray)
                .write()

Writing raw text to the summary

If you want to add some lines of text to the summary with this, then let me save you some time on figuring this out (writing this for a friend 🙈):
You are writing the raw text as Markdown, which I often forget. That means that everything has a meaning, especially after a header!

Here is an example of some of my logging:. The end of lines are also needed!

    await core.summary.addHeading("Repo info")
                      .addRaw(``).addEOL()
                      .addRaw(`Total repos: ${repos.length}  `).addEOL()
                      .addRaw(`Large repos: ${largerRepoCount}  `).addEOL()
                      .addRaw(`Gitattributes: ${largerRepoHasGitAttributes}  `).addEOL()
                      .write()

Need to add a mermaid diagram?

Notice the quotes in the exampe below, that’s where I went wrong the first few times:

await core.summary.addHeading("Repo info")
                  .addRaw("```mermaid").addEOL()
                  .addRaw(`pie title Repo size overview`).addEOL()
                  .addRaw(`"Large (${largerRepoCount})": ${largerRepoCount}`).addEOL()
                  .addRaw(`"Small (${normalRepoCount)": ${normalRepoCount}`).addEOL()                          
                  .addRaw("```").addEOL()
                  .write()

Note: Keep in mind that the core.summary.write() method writes the entire buffer to the summary file, but does not clean that buffer. That means that if you use write(), then add more info and write() again, you will get the first buffer as well as the second buffer! I have not found a way to clean the buffer, so make sure you only call the write() only once!

Speaking at GOTO; Aarhus 2023

2023-05-24T00:00:00+00:00

I will be speaking at GOTO; Aarhus link this year. Below you can find the slides that go with my sessions that have all the links you’ve seen in the session.

Protect your Code with GitHub Security Features

In this session, you’ll learn what possible attack vectors you need to look for, how to protect yourself against them and how to leverage GitHub’s features to make your life easier!

You will learn:

Commit signing
Dependabot
Secret scanning
Code scanning using CodeQL

Download the slides from this link here.

Pictures

GitHub Advanced Security for Azure DevOps

2023-05-23T00:00:00+00:00

Microsoft is bringing some of the GitHub Advanced Security tools to Azure DevOps. I have been playing with it for a while and they have presented the latest state at Microsoft Build 2023, which includes a Public Preview!. That means you can try it out yourself, and I can finally share my experiences with you! Since I teach a lot people on how to use this on GitHub, you can find some of the differences between the two implementations in this post as well 😉.

Photo by Adi Goldstein on Unsplash

What is GitHub Advanced Security?

GitHub Advanced Security (GHAS) is a set of tools that help you secure your code and your pipelines. It includes the following tools:

Code scanning
Dependabot / Dependency scanning
Secret scanning

If you want to dive deeper into that functionality, check out my LinkedIn Learning Course here.

Dependabot / Dependency scanning

Scans you repository for known manifest files, like package.json, and checks if there are any vulnerabilities in the packages you are using. It will then create a pull request to update the package to a newer version that does not have the vulnerability. It can also automate creating pull requests for version updates as well.

Secret scanning

Scans your repository for known secrets, like passwords, and notifies you if it finds any. It will also notify you if you push a commit that contains a secret. It will also scan your pull requests for secrets and notify you if it finds any.

Code scanning

Scans your code for known vulnerabilities using CodeQL (or another Static Application Security Tool, aka SAST). It can also scan your pull requests for vulnerabilities and notify you if it finds any.

GitHub Advanced Security for Azure DevOps

Microsoft is bringing the features from GHAS into Azure DevOps as customers have been asking for this for a while. It’s an example of the way they operate both products now, since the new features will get build for GitHub first, and if it makes sense, they might build some of these features into Azure DevOps as well.

You can see the synergy here as well as of course: any known secret scanning patterns from GHAS can also be used in Azure DevOps, since they can share that between the companies.

How to get started with GitHub Advanced Security for Azure DevOps

First you will need to enable Advanced Security on the repository level, for which you will need Project Admin level access of course:

After enabling the feature (on a repo by repo basis, since it will cost you license seats in the future), you will get a new ‘Advanced Security’ tab in your repository menu. This will show you the results of the scans that have been done on your repository.

After enabling the feature you will get some extra menu options left and right, as well as extra tasks for your Azure Pipelines.

Turning on Secret scanning

Secret scanning is turned on automatically when you turn on Advanced Security. It will start a background job that will scan your repository for secrets. It will also scan you repos entire history: all commits and branches will be checked, for known secret patterns. One additional feature is ‘push protection’. This will prevent you from pushing a commit that contains a secret. This is a great feature to prevent secrets from being pushed to your repository and ‘stop the leak’ of new alerts flowing in.

Using dependency scanning

To get started with Dependency scanning you need to inject it into a pipeline. This can be done by adding the following task to your pipeline:

  - task: AdvancedSecurity-Dependency-Scanning@1

The next time this pipeline will run, it will scan you repository for known manifest files, like .csproj for C# projects, or package.json for NodeJS projects. From those manifest files it gathers the information about the packages you are pulling in, and can check the underlying ecosystem for the dependencies of those packages as well (libraries build on other libraries as well). It will then check if there are any vulnerabilities in the packages you are using. You can check each package you use, and all of their dependencies as well, with their version numbers against the National Vulnerability Database (NVD from the NIST institute). Also registered in the NVD is which version is no longer vulnerable (if applicable).

On GitHub that information is used to create a pull request to update the package to a newer version that does not have the vulnerability. It can also automate creating pull requests for version updates as well. This is not available for GitHub Advanced Security for Azure DevOps. I do expect this will be added later, but for now you will have to create the pull requests yourself.

In the alert you will get information on the package version that you use, the first non-vulnerable version, as well as the type of vulnerability, so that you can learn on the type of attack vector for this alert.

Using Code Scanning

For enabling Code Scanning you need to inject CodeQL into an Azure Pipeline as well (other SAST tools will work as well, as long as they upload a SARIF file):

    - task: AdvancedSecurity-Codeql-Init@1
      inputs:
        languages: 'csharp'

    - task: AdvancedSecurity-Codeql-Autobuild@1

    - task: AdvancedSecurity-Codeql-Analyze@1

    - task: AdvancedSecurity-Publish@1

There are a couple of steps to note here:

Init will create a local, empty database that will be filled with all the code paths your code is taking (for example: Method A is calling Method B). You also feed it one of the supported languages (C#, Java, C/C++, Python, JavaScript/TypeScript, Go, or Ruby).
The auto-build step will try to build you application, based on known patterns for the language you have selected. Sometimes the auto build fails for certain projects, you can then use your own build steps in its place.
With Analyze you run all the CodeQL queries against the database that was created in step 1. This will also create a SARIF file that contains all the results of the queries (if a query has a result, it will become an alert).
The Publish step will actually upload the SARIF file to the Advanced Security service, which will then show you the results in the UI.

The queries that will be run can be found in the CodeQL repository on github.com. There is a default set with low noise ratios, but you can also use extended queries by configuring them this way:

    - task: AdvancedSecurity-Codeql-Analyze@1
      inputs:
        querysuite: 'security-and-quality'

Note that in Azure DevOps you configure this on the `Analyze` task where as you do that in GitHub in the `Init` step.

Do be aware that adding more extensive queries will increase the duration of the pipeline, as well as the amount of alerts you will get: the extended queries have a bit more noise and potentially also more false positives. For example from my demo application I went from 4 normal alerts to 7 extra alerts when I added the extended queries from security-and-quality.

The advise is to run these queries at least on a pull request, as well as on a schedule: the community who builds these queries is super active (150~200 pull requests a month in that repository), so you will get new queries that might find vulnerabilities in your code.

When a result of the queries is present, you will see those come by in the logs as well:

From the upload of the SARIF file you will then get the alerts generated in the Advanced Security overview:

When you open up an alert you get more information on the detail page:

From here you can again learn more about the vulnerability found, with links back to the Common Weakness Enumeration number (CWE) on the Mitre site.

Addressing the alerts

To fix the alerts you will need to actually fix the finding it self. There is not yet a way to dismiss the alerts in Azure DevOps (something we do have on the GitHub side).

Fixing secret scanning alerts

For secret scanning the UI currently shows the warning that you need to treat this secret as a leaked value, and fixing can only be done by revoking the secret.

Since there is no way to dismiss this alert through the UI yet, the only way to get rid of this alert is to rewrite the history of the repository so that the commit is no longer in the history. I expect that dismissing alerts will be added in the future, but for now this is the only way to get rid of the alert.

Fixing dependency scanning alerts

To get rid of a dependency scanning alert you will need to create a pull request to upgrade the dependency to a version that is no longer vulnerable. When the detection scan runs again, the alert will get closed. You have a link back to the pipeline that detected that the vulnerable dependency is no longer present:

Addressing code scanning alerts

Fixing Code Scanning alerts is a bit more involved, as you will need to actually fix the code that is causing the alert. After fixing the issue, the next scan will close the alert and will link back to the scan that closed the alert. From it, you can find the commit that fixed the issue as well, so you have that end-to-end traceability.

Note that currently you cannot exclude code from the CodeQL scanning (on GitHub you can). That also means that your unit test suite for example, will get scanned as well, with a good change it finds some issues there as well (I had the same issue with my test project).

Dashboarding

For an overall overview of how you are doing across the entire project or organization, you’ll have to look at Microsoft Defender for DevOps, which is a tool in Microsoft Azure. Since I don’t have Defender integrated yet (I think you need a feature flag to enable that), I don’t have any screenshots yet. I will add them when I have the access!

The integration does make sense from the Azure perspective and integrate it in the tools folks on Azure Cloud use. But if you only use Azure DevOps, then I would expect an overview to find repeat patterns or dependencies in you Azure DevOps Organization as well. I would not be surprised if they add that overview somewhere in the future.

Summarizing

Overall, most of the initial features are already in, with definitely some different choices made comparing it with GitHub’s implementation (for example dashboarding). You can see it is early days, but I expect the rest of the features / finetuning will happen fast, now that the initial part has been integrated (which is often the most work).

We now have good security features integrated into Azure DevOps, right into the developer experience, which is where this needs to land in my opinion: with the folks who can act on the findings and fix them early on in their process. I hope they integrate the dependency-review-action equivalent for Azure DevOps next, so developers will also be enabled to stop the leak (of incoming vulnerable dependencies).

If you want to know more or have questions, use the comments below!

Speaking at VSLive! Nashville 2023

2023-05-17T00:00:00+00:00

I had the pleasure of speaking at VS LIVE! in Nashville link this year. Below you can find the slides that go with my sessions that have all the links you’ve seen in the session.

W19 GitHub Actions: Beyond CI/CD

Join this session for more examples how you can use GitHub Actions to make your life easier!

You will learn:

GitHub Actions can do more then CI/CD
Real-life examples of automation

Download the slides from this link.

W23 Protect yourself against Supply Chain Attacks

You will learn:

Why do we need to protect ourselves?
What things do we need to think about?
A framework to guide you through improving your security stance

Download the slides from this link.

TH05 Protect your Code with GitHub Security Features

In this session, you’ll learn what possible attack vectors you need to look for, how to protect yourself against them and how to leverage GitHub’s features to make your life easier!

You will learn:

Commit signing
Dependabot
Secret scanning
Code scanning using CodeQL

Download the slides from this link.

Photos

Met some nice folks and had a great time! Thanks for having me!

How Copilot/AI helps me in my daily work

2023-04-24T00:00:00+00:00

During an innovation day at work, I needed to generate extra code and a new application. I wanted to check out the newly released Deployment Protection Rules that can help you with protecting when a job in GitHub Actions can roll out your application to an environment.

Deployment protection rules need a new GitHub App that can be triggered when an environment is targeted. That App can then run its own checks and report back the status to GitHub with a callback webhook.

To get started I needed a new GitHub App that can receive the webhook from GitHub. Since I already have an Azure Function App up and running, I thought to add another HttpTrigger function to it, so I could log the payload and learn from that (the payload is not documented yet).

Image generated with Craiyon.ai using the following prompt: `sketched photo with some realism of an AI generating code`.

Usual flow

Normally I would start with searching for the things I need: some documentation for hosting an endpoint to receive the webhook payload, search for creating an app to handle the user login, etc. This can easily take days tinkering around to get things to where it is good enough for a POC. Instead, I used AI tools to help me, and had things up and running in a few hours.

Step 1: Create a new Azure Function

I knew I needed an HttpTrigger and was about to search for it’s declaration, when I remembered I was already testing out GitHub Copilot for Chat, which is in technical preview so I cannot say to much about it. I asked Copilot to write the function definition for me and sure it enough: it proposed the right code, including logging of the incoming payload. Using that code I could publish the function and configure it as a webhook from a new GitHub App.

The logs show that payload sends in the normal information you would expect, like the context for the trigger (in this case workflow_dispatch was triggered by my user account) and the callback URL that can be used to report back the status of the check. The callback has the following form: https://api.github.com/repos/{OWNER}/{REPO}/actions/runs/{RUNID}/deployment_protection_rule.

That means it uses a well defined url, so I can just call that as the GitHub App for testing to see if it works. I’ve configured the Deployment Protection Rule with my new GitHub App based on an existing workflow (Copilot still helped here left and right), and tested it in a workflow to call the callback url.

Step 2: Create a new node.js app for user configuration

An important part for my setup with the GitHub App will be to have the user configure settings for their environment. So that means I need to have a way to host an app that can do the OAuth flow with GitHub and store the settings in a database.

I’m not a JavaScript developer who knows exactly how to get started, but I do have AI tools that can help me with that. The easiest option I have and that is free to use, is the Bing integration in Edge. Here is my prompt: How can I get started with a simple webapplication based on node that uses GitHub authentication for the user to login and then retrieve information as that user:

Here is a part of the resulting app:

It even had the steps in there to store the configuration settings from the GitHub App in the .env file. I could just copy that example and add the code to the app.

There was an error running the app locally, but I could just copy the error message and search for it following the same conversation on Bing. The first result was the right answer, so I could just copy that again (npm install express-session) and add it to the app.

You can see the working redirect for the user authentication below:

Generated code

The generated was not perfect: it picked up some things from the code it was trained on that can be improved. GitHub Copilot should prevent this kind of security issues when generating code, but I used Bing’s AI to generate this part of the code.

This proves the point that even when using AI to help you generate code, setting up security pipelines is still very important. Luckily GitHub has made Advanced Security features available for free for public repos, allowing me to scan the code using CodeQL:

If you want to learn more about GitHub Advanced Security, then check out my LinkedIn Learning Course on it: LinkedIn Learning.

I could then use the generated code to get started, and improve it to make it more secure using the security tools available as well.

Conclusion

And with that, most of my setup (the new Azure Function and the new node.js app) was done with me writing minimal amount of code, and still getting everything to work as I wanted.

Do be aware that I have enough experience with these topics to both know what to look for, and how to spot / find / debug errors in my code. I’m sure that if I was a beginner, I would have had a lot more questions and would have needed to do a lot more research to get to the same result.

I’m really excited to see what the future holds for Copilot and AI in general. I’m sure it will help me and many others to get more done in less time.

Enabling CodeQL on GitHub Enterprise Server

2023-02-07T00:00:00+00:00

To enable CodeQL on GitHub Enterprise Server you need to make sure you have GitHub Actions setup and running, including your own set of self-hosted runners. You can read more about that in my previous post here.

From that point you can get started to enable CodeQL. Of course, you’ll need to have it enabled in your license, and upload that license file to your server as well. Enabling starts at the appliance level, where you need to enable the code scanning and secret scanning features from the management console.

Photo by Barn Images on Unsplash

The default workflow that CodeQL will propose links to the github/codeql-action action, of which a static copy is installed with each Enterprise Server update. This organization is hidden by default, but you can navigate to it with the direct link. The issue here is that these action repos (github/dependabot-action and all repos in the actions org) only have the source code linked and updated with each Enterprise Server update. Since the CodeQL bundle is stored as a release asset, it is missing from the appliance. The bundle contains all CodeQL queries, including the security-extended and security-and-quality query types.

Syncing the CodeQL bundle

Normally we’d use the actions-sync tool to update the actions on the appliance with the latest version on github.com. Unfortunately that tool does not sync any release assets. For syncing the release assets, we need to download the latest release of the codeql-action-sync-tool.

After downloading the codeql-action-sync-tool, you need to make sure you have write access to the github organization. By default you do not have it, so you cannot write to the release assets or anything else in this org. That means we need to promote the user that will execute the syncing to an owner of the github org.

To do so, you need to call the ghe-org-admin-promote command line utility from a remote shell on the appliance:

ghe-org-admin-promote -u USERNAME -o ORGANIZATION

Now we can call the codeql-action-sync-tool to download the latest version of the CodeQL bundle and upload it to the github/codeql-action repo. This tool will also update the github/codeql-action repo with the latest version of the action code.
You can run it with this command:

codeql-action-sync-tool sync --force \
 --destination-url https://enterprise-server-url.com
 --destination-token <PAT> \
 --source-token <PAT> # prevents ratelimiting

Of course, the machine that is running this will need to have access to github.com, as well as the Enterprise Server. The --source-token is optional, but it will prevent you from hitting the rate limit on github.com.

Running CodeQL efficiently

Now that we have the CodeQL bundle on the appliance, we can start using it. The first thing you’ll notice is that the CodeQL bundle is quite large. The Linux zip file alone is 500Mb, which is quite a lot for a GitHub Action. The release asset appropriate for the OS of the runner is downloaded for each run. If you are using ephemeral runners (which you should!), this means that the CodeQL bundle is downloaded for each and every run. Given that this workflow is configured by default to run on every push, pull request as well as on a schedule (once a week), it can take up quite the bandwidth and thus hammer your appliance.

The action follows the normal setup for GitHub Actions to check for the well known folder runner.tool_cache, which is stored in ‘/opt/hostedtoolcache/’. If it can find the bundle in that folder, it will use that. If it cannot find it, it will download the appropriate release asset.

That means that we can prep our runners by copying the CodeQL bundle to this location. This will prevent the bundle from being downloaded for each run. The folder is used by including the CodeQL bundle release version and date: /opt/hostedtoolcache/CodeQL/2.12.1-20230120/x64/codeql/codeql.

Priming that location will significantly lower the download pressure on your appliance, and speed up the execution of the CodeQL workflow.

Note: the version number has changed from calendar versioning to semantic versioning. The version number is now something like v2.15.0, so be sure to handle that change as well.

Making the case for GitHub's Secret scanning

2023-01-22T00:00:00+00:00

After scanning the GitHub Actions Marketplace for the security of those actions (read that post here) I was curious to see what happens if I’d enable Secret Scanning on the forked repositories. I regularly teach classes on using GitHub Advanced Security (where secret scanning is part of) and I always tell my students that they should enable secret scanning on their repositories. I even have a course on LinkedIn Learning about GitHub Advanced Security in case you want to learn more about it.

What is GitHub Secret Scanning?

Secret scanning is part of the GitHub Advanced Security offering if you have a GitHub Enterprise account, but for public repos it is free to use (and enabled by default). GitHub scans the repository (full history) and issues and pull requests contents to see if it detects a secret. The detection happens based on a set of regular expressions shared by GitHub’s secret scanning partners (over 100 and counting). If a secret is detected, GitHub will notify the repository owner as well as the secret scanning partner. Depending on the context (public repo or not for example), the secret scanning partner can decide to revoke the secret immediately, which I think most partners do.

This functionality is so good and fast, that I routinely post my GitHub Personal Access Tokens to GitHub issues during my trainings, to show the power of secret scanning. Usually I already have an email and a revoked token before I finish explaining what is happening in the background.

Photo by Kristina Flour on Unsplash.

Analyzing the actions repositories.

In my GitHub Actions marketplace scan, I have the repositories of 14k actions forked into an organization, so I can enable secret scanning and see what I get back from secret scanning. Since all Action repositories on the marketplace are public, any secret that is found has been found before, so I expect all these secrets to already have been revoked before.

Overall results: Found [1353] secrets for the organization in [1110] repositories (out of 13954 repos scanned). Here is a top 15 of most found secrets to see what is being found:

Secret scanning alerts

Alert type	Count
GitHub App Installation Access Token	692
Azure Storage Account Access Key	155
GitHub Personal Access Token	120
Amazon AWS Secret Access Key	50
Plivo Auth ID	40
Amazon AWS Access Key ID	40
Google API Key	34
Slack API Token	31
Slack Incoming Webhook URL	27
Atlassian API Token	22
Plivo Auth Token	16
GitHub SSH Private Key	12
Amazon AWS Session Token	12
HashiCorp Vault Service Token	11
PyPI API Token	10

With all the news recently about credential leaking and malicious actors using these secrets to do bad things, I think it is very important to enable secret scanning on your repositories! Having this data really shows the power of GitHub and its secret scanning partners.

Analyzing the results

I wanted to get these results to get a feel for the amount of things secret scanning would find. I my opinion, the maintainers of these actions have a high level of understanding Git and GitHub, so you’d expect a relative low number of secrets being found. Still, 1110 repositories out of 13954 repos is 7.9% off all repos where secrets have been found. This shows how easy it is to accidentally commit secrets to a repository. Even in my own repos, a secret was found (a GitHub Personal Access Token even) that I accidentally committed in an environment file! And that while I teach people on these things!

I think this is a good number to show to my students and customers to make the case for enabling secret scanning on their repositories. Even folks with a high level of understanding, will still make mistakes and secret scanning will help by finding them for you, every time you make a change to your repository.

Improving your GitHub repositories security setup by adding the OSSF scorecard action

2022-12-08T00:00:00+00:00

Recently I’ve started to add the OSSF scorecard action to my (action) repositories. This is a GitHub action that will run the OSSF scorecard checks against your repository to see if you are following best practices, like having a security policy, using a code scanning tool, etc. Using this badge can give your users a quick overview of the security of your repository. OSSF stands for ‘Open Source Security Foundation’ and is an initiative to improve the security of open source repositories.

Scorecard action

The scorecard action will do three things (if you use the default settings):

Analyze your repository for the checks
Upload the results to the GitHub Security tab (using a SARIF file)
Upload the results to the OSSF API (A web API that you can even host yourself) With the data uploaded to the OSSF API you can then retrieve a badge with your latest score and show that on your repository:

Scoring

The action checks your repository against several checks and then calculates a score for each check on a scale of 0 to 10. Based on the risk for that check, the score is then multiplied with a weight. The final score is the average of all the weighted scores. The higher the score, the better.

Some of the checks that are executed are:

Branch-Protection: Is branch protection enabled for the default branch?
Are GitHub Action versions pinned to SHA hashes, following best practices?
Is the repository using a code scanning tool? Note that this check currently only supports CodeQL and SonarCloud
Is there a security policy defined?
Is there a definition file for Dependabot?

Setting it up

Setting is up is as easy as going to the OSSF scorecard action repo and copy the workflow example. Add that to you a new workflow file in your repo and run it from the default branch (usually main). After the first run, you can add a link to your README.md file to show the badge.

The result will be a badge showing the latest score:

Code scanning alerts

The default workflow also uploads the check results as a SARIF file to the Code Scanning Alerts that you can find on the Security tab of your repository. Each check can give one or more results, so you will get an alert for each result. That way you can fix the issues one by one.

Even better is that most of these alerts will give you actionable suggestions on how to fix the issue. For example, the alert above will give you a link to the Step Security Application that can analyze your repository and give you a pull request with the changes to fix the issue. They focus on three things:

Restrict permission for the GITHUB_TOKEN
Add security agent for GitHub hosted runner (Ubuntu only)
Pin actions to a full SHA hash

Restrict permission for the GITHUB_TOKEN

One of the best practices is to always indicate what permissions you want to allow for the GitHub token that will be used in your workflow. The default setting is to permissive: it is read and write for everything in your repo. I’ve been advocating that people always make this read-only at the organization level.

Since you cannot see what the organization permissions are, it’s a best practice to always specify the rights. You can add it to the top of your workflow file:

permissions: read-all

And now you can override it at the job level if you need to. For example, if you need to push to a branch, you can add write to the contents permission:

jobs:
  build:
    permissions:
        contents: write
    runs-on: ubuntu-latest
    steps:
     # etc

Add security agent for GitHub hosted runner (Ubuntu only)

This setting is something I want to look into more. It seems that you can add a security agent to your GitHub hosted runner (available only for Ubuntu). It’s an extra product from Step Security which is available for free. It will analyze the network traffic in the job and log that in their system. After a couple of runs they know what kind of traffic is to be expected, and then you can convert that to basically a firewall rule. Any violations will then be logged and blocked. This can give you a better perimeter defense (as by default all traffic is allowed).

Definitely something to look into more!

Pin actions to a full SHA hash

The last option is to pin your actions to a full SHA hash. This is a best practice that I’ve been advocating for a while. It’s a bit more work, but it will make sure that your workflow will not be affected by a malicious actor pushing a new version of an action. What I really like is that this setting will analyze your workflow file (either pasted in or just all workflows in your public repo) and then suggest the SHA hashes of the actions, by looking at their latest version! Even better, they store the latest version in the comment next to the SHA hash, so you can understand where the hash comes from. Even Dependabot understands that comment and will update it in their version update PRs!

Examples for calling the GitHub GraphQL API (with ProjectsV2)

2022-11-28T00:00:00+00:00

Recently we had to call the GitHub GraphQL API for creating a new GitHub Project (with V2). Since this can only be done with the GraphQL API, we had to figure out how to do this. We found little bits and pieces of information, but no complete example. So we decided to write one ourselves. I hope this helps you as well.

ProjectsV2 GitHub GraphQL API

The new GitHub Projects simply do not have a REST API at the moment, so you are forced to use the GraphQL API. As an extra challenge, the GraphQL API requires a new set of ID’s for these calls.

Logging in

If you are not careful, then the first hurdle will be to login with the right scopes. I’m going to use the GitHub CLI in this post, but you will need to make sure you have the right scopes for this with manual API calls as well. Logging in with just the normal gh auth login call is not enough in this case. Add the project scopes to your call:

gh auth login --scopes "project"

Getting the new ID’s

Now we can start making the calls to the GraphQL API. Since this API with ProjectsV2 needs the new ID’s, make all calls like this to get the new ID’s:

# powershell:
$response = $(gh api graphql -H "X-Github-Next-Global-ID: 1" -F query=$query | ConvertFrom-Json)

Now we can call the API with the current login to get the new ID for this login:

$query = "query { viewer { login, id } }"
$response = $(gh api graphql -H "X-Github-Next-Global-ID: 1" -F query=$query | ConvertFrom-Json)

Need to get the ID from an organization? Use this query:

$query="query { organization(login: ""$organizationName"") { id, name } }"

Working with ProjectsV2

Now we can start loading the existing projects for an organization:

$query="query { organization(login: ""$organizationName"") { projectsV2(first: 100) { edges { node { id } } } } }"

And we can create a new project:

$query="mutation (`$ownerId: ID!, `$title: String!) { createProjectV2(input: { ownerId: `$ownerId, title: `$title }) { projectV2 { id } } }"    

If you want to see the full script, you can find it here.

Working with GitHub secrets without admin rights

2022-11-02T00:00:00+00:00

I was giving a training today on GitHub Actions and learned something new! One of the attendees asked about being able to read and write to Repository Secrets without having admin rights. I had never tried this before, but it turns out it is possible!

The premise:

To be able to create actions on the repository you need to have Admin access to the repository: otherwise the UI will not be visible, since it is under the repository settings. For organization level secrets you need Admin access to the organization level.

That also means that team members that do not have Admin access, cannot SEE the repository secrets in the UI. They always had to rely on the examples in other workflow files to see what type of secrets were available. Today I learned that any user with write access to the repository can also create, update, and delete a secret. Even if they cannot see it in the UI, they can use the API to manage the secrets. It would be great if GitHub updates the UI to make these kinds of information available to all users that need it (write and maintain). Something similar happens with available GitHub Self-hosted Runners (information is not available for non-admins).

Photo by Markus Winkler on Unsplash

The solution:

If you have write access to the repository, you can use the API to create, update, and delete secrets. You can use the GitHub CLI to do this, or you can use the GitHub REST API.

There is also an API for Environment secrets that work the same way: GitHub REST API.

The GitHub CLI is a great tool to use, since it is easy to use and it is cross-platform. You can install it on Windows, Linux, and Mac. For maintaining the repository secrets, there is a native call in the CLI that you can use: link.

  # list all secrets for the current repo
  gh secret list

For environment secrets there is no native call in the CLI, but you can use the REST API to list the secrets. You can use the following command to list the environment secrets:

    # list the environments on the repository:
    gh api repos/<OWNER>/<REPO>/environments

    # list secrets for that environment:
    gh api repos/<OWNER>/<REPO>/environments/<ENVIRONMENT>/secrets

Output of the CLI calls:

From then you get options to set (create), delete or update the value of the secret. Retrieving the value of the secret is not possible, since there are no API calls for that, which makes sense since they are secrets.

Examples of setting a variable:

# Paste secret value for the current repository in an interactive prompt
$ gh secret set MYSECRET

# Read secret value from an environment variable
$ gh secret set MYSECRET --body "$ENV_VALUE"

# Read secret value from a file
$ gh secret set MYSECRET < myfile.txt

Figuring out environments and secrets

The environment secrets can be listed by users with write access, and they can create environments directly with the API.

For creating an environment you need to have write access to an repo, and the repo needs to be in an Organization. User space repos do not work with this write API.

    # list secrets for that environment:
    gh api repos/<OWNER>/<REPO>/environments/<ENVIRONMENT>/secrets

For creating environments in your own user space, you have Admin access so you can use this API call. If you add a body you can the timeout rule, specify protection rules (who needs to approve the job that targets the environment) as well as the deployment branch rule (which branch is allowed to target that environment).

 gh api -X PUT /repos/<OWNER>/<REPO>/environments/NEW_ENVIRONMENT

Results:

If you are invited as a Collaborator to another user’s repo, you cannot create environments with the API. Trying to make the call you can get two results:

404: Not found (repo is private)
403: Forbidden (public repo, but you do not have admin access)

As long as you do have read access to the repo, you can list the environments:

    # list the environments on the repository:
    gh api repos/<OWNER>/<REPO>/environments

💡 As you have read access to all public repos, you can list the environments on any public repo as well. Since this information is already available from the workflow files (and they are public), having the name of the environment is not a big deal. Unfortunately the API returns all there is to know about that environment, including the deployment branch and protection rules. The feedback I got from GitHub is that this is ‘by design’. You can get similar information from the branches API.

Closing thoughts:

The only thing that is not available this way is retrieving a list of Organization level secrets: the API calls for that need admin:org scope, which a user with write access to the repository does not have by default. That is still a big missing feature, since these kinds of secrets are often used to set some large scope (often read-only) secrets for the entire organization, or to have 1 ‘team’ secret that gets added to the repos that team owns.

So, to list all the Repo secrets you can run:

gh secret list

And for finding the Environments and their secrets:

    # list the environments on the repository:
    gh api repos/<OWNER>/<REPO>/environments

    # list secrets for that environment:
    gh api repos/<OWNER>/<REPO>/environments/<ENVIRONMENT>/secrets

LinkedIn Learning: GitHub Advanced Security

2022-10-19T00:00:00+00:00

My LinkedIn Learning course on GitHub Advanced Security (GHAS) has been released! In this course I teach you all about the features of Advanced Security:

Dependabot
Code scanning
Secret scanning

You can watch it with a LinkedIn Learning account (30 day trial is available) with this link: GHAS on LinkedIn Learning.

Teaser for the training:

How GitHub Actions versioning system works

2022-10-19T00:00:00+00:00

TL;DR

The runner just downloads what you specified, by getting it from the tag
The runner does not do SemVer at all. It’s up to the maintainer
Even GitHub does not update (or create) all SemVer versions, so @v3 is not necessarily the latest thing for v3!
The marketplace shows releases, not tags. If the maintainer does not actually release, it’s not visible
It’s more secure to use a SHA hash instead of a tag: read more info here

Semantic versioning

When using GitHub Actions, the default is to use the Semantic Versions for which the actions where released. Semantic versioning (SemVer) is an industry wide standard of giving meaning to the version number. SemVer always follows this setup:

MAJOR.MINOR.PATCH

Given a version number you increment the:

MAJOR version when you make incompatible API changes
MINOR version when you add functionality in a backwards compatible manner
PATCH version when you make backwards compatible bug fixes Optionally you can use any suffix label you want to indicate special versions like alpha, beta, release candidate, etc:
1.2.5-alpha.1
1.2.5-RC.1

The goal of using SemVer is that you can then specify what you as a user of a library want to use. You can be very specific and say you want to use version 9.1.4, but also be less specific and say you want to use version 9.1. Following SemVer this means you want to use any version that matches the 9.1 MAYOR.MINOR version. This means you will get the latest version of 9.1 that is available. This is very useful when you want to use the latest version of a library, but you don’t want to have to update your code every time a new version is released. You can just specify the MAYOR.MINOR version and you will get the latest version of that version. Effectively that means that you’re saying 9.1.*.

If version 9.1.3 is the current latest PATCH version, your package manager will download that version. If version 9.1.4 is released, your package manager will download that version. If version 9.2.0 is released, your package manager will not download that version, because it does not match the MAYOR.MINOR version you specified.

The same setup goes for the MINOR version. If you specify version 9, you will get the latest version of that MAYOR version. This means that you will get the latest version of 9.1 and 9.2 and 9.3 and so on. Effectively that means that you’re saying 9.*.*.

What the runner does with semantic versioning of using GitHub Actions

The runner that executes GitHub Actions for us is open source. You can check the source code for it here. If you dive into it, you can find that the runner tries to download the version from the Action you specified:

- uses: actions/checkout@v2     --> will download the repo with TAG = v2
- uses: actions/checkout@v3.1.0 --> will download the repo with TAG = v3.1.0
- uses: actions/checkout@main   --> will download the repo with BRANCH = main
- uses: actions/checkout@e2f20e631ae6d7dd3b768f56a5d2af784dd54791 --> will download the repo with COMMIT = e2f20e631ae6d7dd3b768f56a5d2af784dd54791 (SHA hash)

You can find the code with it makes the download link here. The runner then calls the REST API to download a tarball of the repo (or on a non-Windows host the zipfile of the repo).

If the TAG (or branch or SHA hash) you have specified is not available in the repo, the runner will give an error saying it cannot find that version.

Using SemVer for Actions

Now the interesting thing is what happens if you specify a semantic versioning pattern for an action. For example:

- uses: actions/checkout@v3.1

Following semantic versioning, you’d expect that this example would work. Configuring actions with a version (or tag, or hash) is required these days so the runner can find the correct reference to download. That means that the maintainer of the action MUST follow SemVer when releasing their actions, as also described in the GitHub Actions documentation. If the maintainer does not follow SemVer, the runner will not be able to find the correct version to download.

Now guess again what happens when you specify version v3.1 for the checkout action. The runner will try to download the repo with TAG = v3.1. But that tag does not exist! The runner will then give an error saying it cannot find that version. So the runner does not check for matching versions by itself!

- uses: actions/checkout@v3.1 <-- this will fail!

You can find the tags for this action here and see that v3.1 is missing! So even the most used action, does not follow GitHub’s best practices!!!

Even better, the GitHub Actions Marketplace actually does not show the version from the tags: it only shows the information from Releases in the repo! That means you could be missing out on tags that where not released as a GitHub Release.

Check the tag list versus the release list shown in the marketplace:

Make your GitHub Actions usage more secure

I’ve been telling people that tags are not secure: the maintainer of the action can update the tag to point to a different commit. That means that your workflow could be using a commit you verified (that should always be step 1!), but all of a sudden the maintainer of the action updates the tag to point to a different commit. That means that your workflow is now using different code, which you did not verify! Read more on becoming more secure with your Actions usage in this blogpost.

The way to fix this and make your setup more secure, is to use the SHA hash of the commit you want to use. That way you can verify the code yourself and you know that the code you’re using is the code you verified. Incoming changes can be send as notifications by setting up Dependabot for the github-actions ecosystem.

Summary

We keep telling people to follow SemVer for a reason, but for GitHub Actions the honus is on the maintainer of the action to actually re-tag all matching versions of their action with the correct SemVer version. And apparently, even GitHub doesn’t follow along with this.

So when you have a current release v4.2.5, you also need to re-tag that commit with the v4.2 tag, as well as the v4 tag. This way the runner can find the correct version to download.

Example:

jobs:
  job1:
    runs-on: ubuntu-latest
    steps:
    # works, as this is an actual tag in the repo
    - uses: actions/checkout@v3

  job2:
    runs-on: ubuntu-latest
    steps:
    # works, as this is an actual tag in the repo
    - uses: actions/checkout@v3.1.0

  job3:
    runs-on: ubuntu-latest
    steps:
    # does not work, as this is NOT an actual tag in the repo
    - uses: actions/checkout@v3.1

Enabling GitHub Actions on Enterprise Server: Common gotcha's

2022-10-08T00:00:00+00:00

When customers start using GitHub Enterprise with Actions and private runners, there are some common gotcha’s you can run into. In this post I’m sharing the ones I have encountered so far. Even Dependabot comes along, since that runs on Actions as well for GitHub Enterprise Server.

List of topics:

First of all: Don’t use self signed certificates on GitHub Enterprise
The default actions in will download the binaries from github.com
Actions org will be cleaned up with each major/minor update
Mismatches in the settings UI (harder for admins)
Dependabot runners need to be created
Log storage is external to the server
GitHub Connect is mandatory
Think about GitHub Actions and using them securely

The default actions in the /actions organization, will download the binaries from github.com

Be aware that the default actions in the /actions organization (setup-node, setup-go, etc.), will download the binaries they need from github.com. This means the runner will download those files without any authentication and will be rate limited after 60 downloads/hour/ip-address. To get around that, you will need to create your own version of these actions and download the releases from something like GitHub Releases or an internal source like Artifactory, or store them as GitHub Releases in a repository.

Actions org will be cleaned up with each major/minor update

This was not such a pleasant surprise: we have had to customize the default actions, since setup-node and setup-go will download the binaries from github.com (see above). Our runners do not have internet access for security reasons. We have created our own versions of these actions that download the binaries from our internal Artifactory instance and uploaded them to our GitHub Enterprise instance. This worked fine for a while, but when we upgraded from 3.1 to 3.2, the actions org was cleaned up and our custom actions were gone. This was not a problem for us, since we had copies of the source code on several machines, but it was a surprise. So be aware: the entire org gets cleaned up with each major/minor update!

Mismatches in settings UI: it’s all over the place

There are 2 different idioms in the settings, and if you have admin rights (either Enterprise, Org or repo level), you will see both.

UI where you make changes and need to scroll down to save them
UI where any changes trigger a post back to the server and are stored (and in effect immediately)!

For the second one, here is an example. There is a save button in here, so…. what do you thing will happen if you change the All repositories setting in the drop down?

This setting posts back to the server immediately, so the change is stored and in effect immediately! There is a small page reload, but if you are not paying attention, you could miss it. Guess how I turned off GitHub Actions for all users in our Production Environment? I took a bit more then half an hour before users started calling us (my team maintains it), and a check on this page to learn that the setting was changed…. That was not the intention, because I didn’t press the save button!

Dependabot runners need to be created

When you want to enable Dependabot, and it’s version updates, you need to do this at several levels: first enable it on the appliance. Then change the settings for the Organization level to allow it to be used.

Then you can configure it per repo:

Enable the Software Composition Analysis (SCA) feature (that’s Dependabot starting point)
Configure the Dependabot.yml file to run with updates
Be aware that Dependabot runs on actions and uses the default setup action to configure the ecosystem (more below)
Notice that the Dependabot runs use a specific label to target runners with (more below)
Create runners for Dependabot to use (more below)

Dependabot runs on actions

Dependabot for Enterprise Server runs on actions and are visible from the ‘Actions’ tab for the repo. That way you can monitor their runs. If you enable all the features, then runs will be scheduled. Only until you actually go and check those runs, you’ll learn that it is using a ‘dependabot’ label to target the runner with. This is of course so that you can control where these runs happen, and how many runners can be bothered with these jobs. That way Dependabot runs will not just randomly start hogging all of your runners that are labeled with ‘self-hosted’. So, you do need to create runners with that label, before anything starts to happen.

Hint: checkout the Dependabot Version updates workflow run logs, that magically will appear in the ‘actions’ tab!

Because there is no API for Dependabot to monitor it’s runs, or a way to see queued runs and alert on them if they are queued for more then 30 minutes (for example), there is no good way to learn about this until you start searching around.

Finding out about this happened because I went looking for the settings and found the stuck in queue workflow run. Looking at the workflow definition told me what I needed to know. This is quite a bit hidden in the docs: if you know where to find it, and dig through 4 different pages of information on enabling Dependabot, then eventually you find this reference to it, hidden under step 3. I made a suggestion to make it more clear you need to do this here (update: PR accepted!).

Dependabot uses a preconfigured workflow definition

This definition pulls down the setup-node from your actions organization on the server. If you overwrite them with something custom (as we had to do since our runners are not allowed to download things from the internet), these runs can easily fail (for us they did). I have not found a way to override this workflow file, so you need to make sure this works. We currently are working on setting things up for us, which is halting our adoption of Dependabot at the moment ☹️.

Log storage is external to the server

GitHub Actions on the SaaS version (github.com) has been created with running on Azure in mind. All of those logs are stored then on cloud storage. Currently Azure Blob storage and AWS S3 Storage is supported. If you cannot push the logs to one of those, there is a Min.io container setup that has the exact API surface of an S3 bucket that can be used, to layer that on top of you own storage solution. You will need to create and configure that storage to be used before you can continue.

GitHub Connect is mandatory

To be able to use GitHub Actions, you need to have GitHub Connect enabled. This is a feature that allows you to connect your GitHub Enterprise Server instance to GitHub.com. This is needed to be able to use the GitHub Marketplace. I’m not really sure why this is the case, because the runners will use their own internet connection to download the actions from github.com. The only reason I see would be the UI in the workflow editor, that allows you to search the public marketplace for actions.

Think about GitHub Actions and using them securely

I written about how to use GitHub Actions with a secure setup here and you can view a recording of my conference session at Code Europe 2022 here.

You really need to check the actions and vet them, since the security model is based on trust. Read more on the state of the GitHub Actions Marketplace here.

Techorama NL: Protect yourself against supply chain attacks

2022-10-05T00:00:00+00:00

Placeholder for sharing the slide deck for Techorama: session link

Abstract:

Attacks against your pipelines are more and more common these days. We’ll go over the attack vectors you need to be aware of and how someone could potentially misuse a simple setting to hijack your environment, with very large consequences. From breaking out of your shell scripts in the CI/CD pipeline, misusing typo’s in third party packages or even squatting your internal package names on a public repository: there are lots of ways to get into your pipeline!

Slides:

Presentation dotnetsheff - Protect your code with GitHub security features

2022-09-21T00:00:00+00:00

I have the pleasure of virtually speaking at dotnetsheff and these are the slides for it:

In this session you’ll learn what possible attack vectors you need to look for, how to protect yourself against them and how to leverage GitHub’s features to make your life easier!

Topics:

Signed Commits
Dependabot updates
Dependency scanning for known vulnerabilities
Secret scanning (and revoking) out of the box
Using CodeQL

You can download the slides here.

Questions:

Some questions that came up during the presentation: Q: Can we configure Dependabot to use conventional commits? A: No you can’t. There are several issues on the repo that ask for this, but they have not included it. What you can do, is work with prefixes in the commit message. Read more on this from the docs

Q: Can you run Dependabot locally? A: You can through a local install or from a Docker container, for example using this project. But for a feature rich experience, you should use the GitHub setup.

GitHub Advanced Security

Want to learn more about these GitHub Advanced Security features?
Check out the LinkedIn Learning course I made for it:

Click the image to go to LinkedIn Learning

Analyzing the GitHub marketplace - Dependency security is a big issue

2022-09-18T00:00:00+00:00

I have been a fan of GitHub Actions since the beta in the end of 2019. And the more I use them and create my own, the more I have this growing itch to see how these actions are made, how active the community is, and what we can do to improve this ecosystem. So I decided to do some research and see what I could find out. I already have a (now inactive) Twitter bot that scrapes the GitHub Actions Marketplace and stores that info for later use (unfortunately, the marketplace has no API to use for this).

Photo by Ashes Sitoula on Unsplash

I am also fascinated with the security aspects of using GitHub Actions for my workloads. My first conference session on this topic was at NDC London in January, 2021 and I have been advocating on these learnings ever since. That is why I also decided to run my usual security checks on the entire marketplace, starting with forking them so I can enable Dependabot on the forked repositories.

The Marketplace shows us almost 15 thousand actions that are available for us to use 😱. That means there is lots of community engagement for creating these actions for us, but also lots of potential for malicious actors to create actions that can be used to compromise our systems. Do be aware that in this post I’ll only be taking actions into account that have been published to the marketplace. Since any public repo with an action.yml in the root directory (and a bit more options) can be used inside of a workflow, there are many more actions that are available to us that are not part of this research.

Note: I’ve updated the analysis to disregard the DevDependencies after that information was added to the API. This means that the numbers are lower than the ones in the original post: a large portion of the alerts where coming from DevDependencies.

Analysis of the actions from the GitHub Actions Marketplace

I created a new repo to run these checks using GitHub Actions by scheduling a workflow that runs every hour and checks the dataset for new actions that have not been forked to my validation organization yet. If you have more checks or type of information you would like to see, definitely let me know and I’ll add them to the workflow.

Some caveats up front:

I could only load the information for 10.5 thousand actions. All the others have issues that makes it that I cannot find them anywhere. These are not included in the dataset for this analysis.
Some have been archived by their maintainer, but still show up in the Marketplace. These are of course older and have more security issues in them. The actions are included in this analysis. I’m planning to remove these when the Marketplace doesn’t show them anymore.
There are some actions where I could not parse the definition file (if used), often because of duplicate keys in their definition file. I’ve reached out to some of the maintainers to get those fixed, but also want to improve my method of loading these kinds of files. Currently the library I use for this does not support duplicate keys and throws unrecoverable errors when it finds them.

I’ve reported this information back to GitHub and they are planning to improve the freshness of the data in the Marketplace. Still, this is a good two thirds of the actions that are available in the marketplace, so this is a representable dataset to look at. Examples of actions that show up in the marketplace but will 404 when you want to load the detail information for them:

Additionally all this analysis is done on the default branch for the repository. I have one action for example that uses a Dockerfile in the main branch, but I am working on converting it to Node in another branch. This number should be small enough to have no significant impact on the overall analysis.

Overall stats - Action type

My first interest was analyzing the overall makeup of the actions. For example, you can define the actions in multiple ways: By the ecosystem they use: Node based (Typescript or JavaScript), Docker based or it can be a composite action.

Type of Action	Count	Percentage
Node based	4,7k	45%
Docker based	3.7k	35%
Composite	1.6k	16%

This tells us that there is a lot of use for Docker based actions! That means that a lot of these actions do something more complex than you can do in a Node based action. They made a choice to have a slower action, since the Docker image needs to be build or downloaded, and then needs to be booted up. Compare that with a Node base action, that immediately starts to run. But, you can use whatever language and ecosystem that fits your action the closest.

Action definition setup

Next up was analyzing the setup of the action itself. You can define it in multiple ways in the root of your repository:

action.yml - This is the default way to define an action. It is a YAML file that contains all the information about the action.
action.yaml - This is an alternate way to define an action that was available at the beginning of GitHub Actions and is therefor still supported.
Dockerfile - This is another way to define a Docker based action. If you don’t provide an action.yml or action.yaml file, this one will be picked up. This Dockerfile is used by the runner and will be build on the execution environment when the action is used. An alternative way is to define the Dockerfile link in the action.yml (or yaml) file. I’ve not made a specific overview that indicates how many Docker based actions where defined in the action.yml file and how many have a separate Dockerfile.

Definition of the Action	Count	Percentage
action.yml	9.2k	89%
action.yaml	700	7%
Dockerfile	144	1%
Unknown	300	3%

This tells us that almost 90% of the actions have been defined with the action.yml file, which is indeed the way that is show in most docs and demos.

Docker based actions

For the Docker based actions I was interested to see how many actions use a pre-build image from a container registry and how many do not. If they use a local Dockerfile, it will have a potentially big impact on the startup time of the action. Even worse, those are harder to support on GitHub Enterprise Server, since those environments are usually locked down from the internet, for valid security reasons. Supporting one of these actions means also supporting the ecosystems in use, starting with the base container image (run security scans on them folks!) and then anything it uses: apt-get, yum, yarn/npm, pip, etc. This will bring in some significant overhead for the support team. I’d rather recommend a maintainer to use a remote image, preferably the GitHub Container Registry, since that is directly associated with the repo and can then be downloaded without any (significant) rate limiting.

Docker based action	Count	Percentage
Remote image	550	15%
Local Dockerfile	3.1k	85%

From that we learn that on-boarding and validating these Docker based actions will likely be a lot of work and will have significant impact on your runner setup: each time it is used, the base image will be downloaded and then the action will be build on top of that. This is a lot of overhead for the runner and will slow down the execution of the workflow. It also costs unnecessary bandwidth on the runner as well as extra compute power, which has an impact on the environment. All this is rather unnecessary if the maintainer would use a remote image, where the runner host can cache that image locally. Be extra aware of this if you are hosting ephemeral runners, that get deleted after they have executed a workflow job.

One of the next steps here is to analyze the remotely hosted images on where these are hosted.

Repo age

The next thing I wanted to know was the repository age: what can we tell about the last time the repo had seen (any!) updates? If the action is not maintained at all, there is a good chance there might be something wrong with it (either in functionality, security or adding in new features).

I need to dive deeper to see if there are any patterns in the repo age:

Do the 267!!! archived repos bring down the average age a lot?
Do the most vulnerable repos match with the oldest repos?
What about the last release that was made? All demos show that you need to include the version in the uses statements, so the last time the action has seen a release might be significant as well.

Security alerts for dependencies of the Actions

I’ve also forked over the action repos to my own organization and enabled Dependabot on them to get a sense of the vulnerable dependencies they have in use. Some caveats to this analysis are:

Not every dependency will end up in the action itself, so a high alert from Dependabot will point to a ‘possibly’ vulnerable action. Since this is not something you can track automatically and see if this would be the case, we cannot be sure that the action itself is vulnerable.
This only works for the Node based actions, which is 4.7k, so almost 50% of the analyzed actions. Dependabot does not support Docker at the moment.
I’m only loading the vulnerable alerts back from Dependabot that have a severity of High or Critical.

I’m planning to add something like a Trivy container scan to the setup so that we get some insights from this as well.

Security results

Of the 10488 scanned actions, 3130 of them have a high or critical alert 😱. And that is with only scanning the Node based and Composite actions, since the Docker based actions are not scanned yet. This is a way higher than I even expected and very scary! If your dependencies already are not up to date and thus have security issues in them, how can we expect your action to be secure? That calculates to 30s% of the actions that have one or more high or critical alert on their Dependencies.

To be complete: I have not filtered down the alerts to a specific ecosystem. Since GitHub Actions is one of the ecosystems Dependabot alerts on, there is a change these alerts come from a dependency on a vulnerable action for example, which would be unfair (since these will not end up in the action I am checking). Since there are only 3 actions in the GitHub Advisories Database, I expect this to be of zero significance, but still: good to mention.

Diving into the security results

I’ve also logged the repos with more 10 (high + critical) alerts to a separate report file and that file contains more than 600 actions!

The highest number of high alerts in one singe action, is 58. Since that repo happens to be Archived, it should not be in the actions marketplace at all, as well as the fact that this should not be used at all. Luckily it is only used by a small number of workflows. I’d rather see that the runner would at least add warnings to the logs for calling actions that are archived.

The highest number of critical alerts in one singe action, is 16. This repo is also only used by less then 10 other repos, so it is not a big impact. Since there is no API for finding the dependents that Dependabot finds, I cannot easily find out how many workflows are impacted by this.

I’ve checked some of the repos with a lot of alerts and found one example, that has 14 high severity alerts and 2 critical alerts. This action is used by 34 different public repos (so private could even be more!). One of these dependents is a repo with 425 stars and another has 6015 stars. That last one is producing a serverless CMS that will be delivered as 48 different packages into the NPM ecosystem. One of those packages sees more than a 1000 downloads a week! This is a lot of impact for a single action that could be prevented by enabling Dependabot. Of course, more analysis is needed for this case to see if the alerts are actually relevant for the action. This depends on what the action does and how it uses the dependencies.

Overview

In short, this is a top level overview of the security results:

So for all action repos I could scan, 30% have at least 1 vulnerability alert with a severity of high or critical.

Node based actions

Filtering this down to only the Node action types, this becomes a lot scarier:
That is 58% of the Node actions that have at least 1 vulnerability alert with a severity of high or critical! And all demos and docs still indicate you can just use the actions as is and only hint at the security implications of that!

Want to learn how to improve your security stance fur using actions? Check out this guide I made: GitHub Actions Maturity Levels.

Conclusion

There is a lot of improvement for the actions ecosystem to be made. I would like to see GitHub take a more active role in this, by for example:

Enforce certain best practices before you can publish an action to the marketplace
Cleanup the marketplace when an action’s repo gets archived (work for this is underway)
Add a security score to the marketplace, so that users can see how secure an action is, run at least these type of scans on the action repo and report it back to the end user
Add a check that validates you also pushed a new release of the action to prevent maintainers to add Dependabot and keep their (vulnerable) dependencies up to date, but not actually release a new version of the action.
Add API’s to not only the marketplace, but also Dependabot Of course, as maintainers of actions we are also in this together! It’s our responsibility to make sure our actions are secure and that we keep them up to date. I hope this post will help you understand why to do that!

Appendix

Maybe I can write a scanner action that looks at this information somehow. and inject that into the load used actions action that I created.

I’ll need to add this type of information for example to the Private Actions Marketplace setup for example.

Do you have ideas on what to add to the scans or how we can improve the Actions ecosystem? Please drop a comment below!

My GitHub Actions workflows are not starting

2022-08-12T00:00:00+00:00

Check the GitHub status!

Every once in a while there is an outage on e.g. GitHub Actions, and I see a lot of influx of users on this blogpost. So before you start reading this, check the GitHub status page to see if there is an outage. If there is, you can wait until it is resolved. If there is not, you can continue reading this post to see if you can find the cause of your issue. Outages can always happen in any SaaS (Software as a Service) environment and keeping a large scale environment in a healthy state is not an easy task. So if you are experiencing an outage, be empathic with the engineers who are solving the issue 🤗.

Caveats, some triggers only work on the default branch

Some default caveats that new GitHub Actions users run into is that their workflows are not being triggered or that the UI to do so is missing. In the beginning everyone starts with the on: push trigger but there will come a time that you only want to execute some workflows on the default (main) branch. So you limit the on: push trigger to that branch:

on:
  push:
    branches:
      - main

When you follow best practices you want to implement a Pull Request based process to prevent a single person from making changes to a repository. That means that you are making your changes to workflows in a feature branch and not in your main branch.

What happens next? See in the steps below. You can start at the top (scheduled run not starting) and then work your way down to the more specific examples. This also represents the order in which a lot of times these questions will occur.

Photo by Sora Sagano on Unsplash

First things first: file location

I have ran into this myself and looked for 15 minutes before I saw what was wrong: check the location of your workflow file! When you are creating the first workflow in a repo, you might learn you have configured the wrong directory because the UI does not show the workflow file if it is in the wrong folder. That can be a hint 😉.

It needs to be stored in the following folder: /.github/workflows/. And that folder is workflows (plural) and not workflow (singular).

Next, pushing with a user?

If you make changes within a workflow and did not set any user information (a Personal Access Token or with a GitHub App, more info in this post), then you are using the GitHub Actions Bot (a GitHub App under the covers), that creates a default GITHUB_TOKEN that can have write access to your repository. GitHub has built in that this token never triggers a workflow run, to prevent an endless loop of actions triggering workflows, that write files, create issues, or something else, and then trigger another actions run.

Scheduled runs not starting

When you add your first schedule for a daily run, you might be surprised that it does not start at the schedule you have set. You might scratch you head and wait for a couple of days, and nothing will happen.

on:
  schedule:
    - cron:  '30 5,17 * * *'

The cause of this is that scheduled runs only trigger from the default branch (main). Several triggers behave this way, like a Pull Request (+Status) trigger, the issue / label / comment triggers, etc.

So if you have a schedule in your workflow and you are not on the default branch, the workflow will not start. This is a security measure to prevent someone from creating a workflow that runs on a schedule and then creating a pull request to a repository that has a scheduled workflow. The scheduled workflow would then run on the pull request and the attacker could do something malicious. This is a good thing, but it can be confusing when you are just starting out. The solution is to make sure that you are on the default branch when you create your scheduled workflow, so that means that you need to merge in your changes for it to start based on your schedule.

Other reasons for schedules to not trigger

There are other reasons why a schedule might not trigger:

The schedule event can be delayed during periods of high loads of GitHub Actions workflow runs. High load times include the start of every hour. To decrease the chance of delay, schedule your workflow to run at a different time of the hour.
The workflow might have been disabled. On forks all workflows get disabled and you need to manually enabled them (makes sense of course). Additionally: in a public repository, scheduled workflows are automatically disabled when no repository activity has occurred in 60 days. The account that last changed the workflow will get an email notification after around 23 days of inactivity in a repository:

Manual runs (workflow_dispatch) UI is not visible

This is the common next step when the schedule does not start: you just add a workflow dispatch trigger to the workflow to trigger it manually. But since this is a new workflow that has not existed yet, the UI for it to trigger is not visible! This is the same as creating a new workflow file with this trigger in one go.

For a manual trigger, the UI is only available from the default branch. You can choose which branch to trigger the run from then, and have the inputs available from the default branch. But the file and the trigger has to be on the default branch for the UI to be visible.

You have two options to proceed and trigger the workflow:

Go to the next step and use ‘on: push’
Trigger the new workflow, from the branch by using the API

Triggering the workflow with the API

You can trigger a workflow dispatch (as well as a repository dispatch for that matter) using the UI and even trigger it from a branch.

You need to make an (authenticated) call to the url for your workflow:
‘https://api.github.com/repos/{OWNER}/{REPO}/actions/workflows/{WORKFLOW_ID}/dispatches’
The workflows actually have an ID under the covers, but you can also use the filename, which is easier to read. So when the workflow is named get-action-data.yml and it lives in the repo rajbos/actions-marketplace it becomes this url: https://api.github.com/repos/rajbos/actions-marketplace/actions/workflows/get-action-data.yml/dispatches. Include a JSON payload with the branch you are referencing in a “ref” property (see Postman screenshot below).

Note: do not include a slash at the end of the url, GitHub’s API’s do not accept that and will return errors.

My tool of choice for this is Postman, because I can store my requests in it and it lives in its own window. This makes it super easy to navigate to and hit CTRL+ENTER to trigger the call, which is helpful when you are creating the workflows.

You can also use the GitHub CLI to trigger the workflow, by running the following command from the repository folder:

  gh workflow run get-action-data.yml --ref rajbos-patch-1

Do not forget to include the ref here if you want to run from a branch. Otherwise it will run from the default branch. Also be aware that this will execute in the context of the repo content that is ‘on GitHub, so not your local content!

On: push then?

The last option you have is to just trigger the workflow whenever someone pushed data into the repository. You can decide if that should happen on certain branches but the best tip here is to include a path filter (see the docs here).

on:
  push:
    branches:
      - main
      - my-test-branch
    paths:
      - 'src/**'
      - '.github/workflows/my-workflow-file.yml'

I often run the workflow on at least my test branch, but then only when the relevant files for that workflow have been edited. That usually is the workflow file itself and maybe certain source files in the repo that are used: whenever there is a change in those files: execute the workflow. This is especially helpful during the development of the workflow: if you push a change in it, it is a good change that you want to trigger the workflow 😄.

Filtering the trigger with a globbing pattern?

When you are using for example the paths, tags or branches filter, you can use a globbing pattern to match the files you want to trigger the workflow on. Through a quirk in the yaml spec, when your pattern starts or ends with a star, you have to encapsulate the pattern in quotes.

So this setup does not work:

on:
  push:
    branches:
      - main
      - my-test-branch
    paths:
      - *

And has to be written like this:

on:
  push:
    branches:
      - main
      - my-test-branch
    paths:
      - '*'

Big changes might prevent the workflow from being triggered as well

Workflows do not start when you push a large amount of files with changes to GitHub. The documentation mentions this on the diff section: Diffs are limited to 300 files. If there are files changed that aren’t matched in the first 300 files returned by the filter, the workflow will not run. You may need to create more specific filters so that the workflow will run automatically.

So if you have a filter setup on the paths of a file, and the first 300 files do not match anything in the filter, the workflow will not trigger. Recommendation is to always make your changeset as small as possible!.

Targeting a wrong label

If you target a runner that you do not have access to (check the runner group permissions) or a label for which no runner exists, your job will hang around until the timeout of 24 hours has passed. Then it will get a timeout error and the job will be cancelled. Double check the label you are using (this typo in the screenshot always gets me!) and fix it, or check the permissions on the runner group.

Final remark

Note that this all will not help when you have specific use cases you want to test, like when someone creates a comment, a pull request. There are other ways to deal with that, but that is for another post.

Creating a GitHub Action

2022-06-01T00:00:00+00:00

I wanted to describe how my flow usually works for creating a GitHub Actions. People often struggle to think of something to build because they start with an empty canvas: the action itself. That is not how I build up the action. For me the process is as follows:

Have a need for something straightforward: like calling the GitHub API in a certain way.
Create a small github-script for it to see if this works.
Move from the inline script to an actual script file, since that is easier to debug.
Have a fully working script, and realize this could be helpful for others
Create a new action repo for it and move the code there
Add unit tests to validate everything works
Add a local workflow that tests the action
Use the action repo in the place where it started, validate it works there as well
Publish the action

Photo by Mika Baumeister on Unsplash.

Have a need for something straightforward

I work a lot with GitHub Enterprise server these days, and we have self hosted runners. I wanted to load the number of available runners by trait and alert me when the number of a certain trait is beneath 3. You can call the API at the organization level to retrieve that information: GET /repos/{owner}/{repo}/actions/runners.

Create a small `github-script` for it to see if this works.

With the github-script you get an authenticated GitHub client to use that can make calls to the API. Since this is an organization level API, we need to give it an access token for it to use. Since I like to use a GitHub App for that, I use that to get a token an pass it in.

    # example of loading a token from an GitHub App
    - name: Get Token
        id: get_workflow_token
        uses: peter-murray/workflow-application-token-action@v1
        with:
            application_id: $
            application_private_key: $

Then I call the API with the client:

  const { data } = await octokit.request("GET /repos/{owner}/{repo}/actions/runners", {
        owner: organization,
        repo
    })
  console.log(`Found ${data.total_count} runners at the repo level`)
        

The data object will have the response of the call and I can start working on manipulating it to get the representation I want, like grouping them per label:

const groups: group[] = []
  data.runners.forEach((runner: any) => {
        runner.labels.forEach((label: any) => {
            const index = groups.findIndex((g: any) => g.name === label.name)
            const status = runner.status === 'online' ? 1 : 0
            if (index > -1) {
                // existing group                
                groups[index].counter = groups[index].counter + 1
                groups[index].status = groups[index].status + status
            }
            else {
                // new group
                groups.push({name: label.name, counter: 1, status: status})
            }
    })
  })

I can now search for a certain group and alert if the count for that group is less then the number I want.

Move from the inline script to an actual script file, since that is easier to debug.

The hard part of using the github-script is that it is very hard to debug. Your script is inlined somewhere so you get an error message stating there is a parsing issue with the inline script, which will be something vague like: error parsing ‘ on line 57. You can try to count the line number in your inline script and search for it that way, but that is very brittle and just sucks. To improve on this, we can move the entire script into a separate file, which we can then call from the github-script:

New file –> Move the code to a separate file. I often save the file in the .github/workflows folder since that it is what it is for. Don’t forget to checkout the repo since now we need to have the file on disk before we can execute it:

  uses: actions/checkout@v3

  uses: github-script@v5
  with:
    script: |
       const script = require('./path/to/script.js')
       console.log(script({github, context}))

If there are parsing errors, the parsing engine will actually tell you: there is an error in you script.js on line 25. Much better!

Have a fully working script, and realize this could be helpful for others

Now that we have a fully working script, with the alerts, I can start thinking about how this could be useful for others. Some people might just want to get the grouped information, some might want to generate alerts. An alert could be a failed workflow, and then use the GitHub notification system to alert the owner. Or the alert might be send into a Teams channel, or a Slack message. Since that could be anything, I want to make only the building block needed to get the info we need. It’s then up to the user to figure out what they want to do with it: they can choose to make an alert and then configure where to send it.

As a good practice, you try to keep your action as minimal as possible. I have actions that only get the data I need, and return it as JSON. What happens with the JSON is up to them. I have cases where I save the JSON back into a repo, or where I load the JSON and then filter it. The point is: leave that up to the user: they can build on you building block.

Create a new action repo for it and move the code there

Since I determined this might be useful for others, I start creating a new action repository. A good starting point is always the Typescript template repository. Use it as a template and it will give you the skeleton setup you need. Only thing that currently doesn’t work on my Windows machine is the building of it, since the setup they use just fails on Windows. Switching that to esbuild does work, so I often convert those calls.

Add unit tests to validate everything works

Since there is some intelligence in the code, like a function that handles the grouping, I start creating unit tests for that. This way I can test the code locally, which saves me a lot of time. Use .env files to load an access token and you can even execute all the API calls and verify that things work.

Add a local workflow that tests the action

After validating the unit tests (and integration tests if you use the .env file setup), it is time to test the action inside of an workflow. You can call the local action in a workflow, pass in the needed params and then execute it:

  uses: ./
    with: 
      access_token: $

We can store the results in a JSON file for example, and pick that up in the next step in the job. Then parse it and validate it contains the data we expect.

Use the action repo in the place where it started, validate it works there as well

As a final validation I replace the script file in the first step, with a call to this new action. If it still works as expected, it is time for the next step. If not, we can fix it before telling the world about a new action 😉.

Publish the action

Now it is time to publish the action. We can automate almost everything: creating a release from a tag is where it starts. GitHub Actions use Git Tags for the version number. I use a workflow that is triggered whenever I push a new tag to the repo: that will create the release and add the repo into it.

The only thing is: publishing the action to the marketplace can currently only be done with the UI 😲. There is a checkbox that needs to be set, and that is not available from the API. As a workaround, I create the release from the workflow file, and then push a message into my own Slack channel with a link to the release. I can then follow the link, edit the release and set that checkbox.

Tell the world about your new action

A quick shoutout on social media is the least we can do 😄.

Note:

To see the resulting action at work, and to the source code I used for all the setup, check out this action: load-runner-info.

Code Europe: Protect your code with GitHub's security features

2022-05-30T00:00:00+00:00

I have the pleasure of speaking at Code Europe and these are the slides for it:

In this session you’ll learn what possible attack vectors you need to look for, how to protect yourself against them and how to leverage GitHub’s features to make your life easier!

Topics:

Signed Commits
Dependabot updates
Dependency scanning for known vulnerabilities
Secret scanning (and revoking) out of the box
Using CodeQL

You can download the slides here.

GitHub Advanced Security

Want to learn more about these GitHub Advanced Security features?
Check out the LinkedIn Learning course I made for it:

Click the image to go to LinkedIn Learning

Speaking at NDC Security (Oslo)

2022-04-06T00:00:00+00:00

I had the pleasure of giving 3 sessions at NDC Security in Oslo on April 6th and 7th. After 2,5 years of not being at a conference in real person, this was the first time speaking in front of an audience and sitting in on sessions in real life! Super weird to do in this time of turmoil in the world, but when you sign up to speak, the conference organizers are relying on you to show up of course!

For me a nice way to get back to public speaking in front of a group: I’ve done my session on “How to use GitHub Actions with security in mind” a lot last year, but everything was online due to corona! The atmosphere at a conference and in front of a real live audience is completely different from doing this from your home office. Sitting in to hour long sessions is really hard work if you do this from home, and as speaker you often do get little to no feedback during the sessions from home. Honestly, the less professionally a virtual session has been arranged, the more engagement and fun I had in them! A simple meetup for example just uses a Zoom/Teams session where people can chat and ask questions. Often they also turn on their camera’s and that really helps to keep you sane. Having a feeling of talking to the wall really does not help when the conference is virtual. That interaction and connection with the audience was often missing in “professionally managed” sessions.

Getting back to public speaking in front of an audience was something I was curious to see what it would do to me: will I get overwhelmed with stress, do I freak out on stage? I have gotten a lot more confident on virtual sessions over the last years, but will that pay out on stage as well?

Before getting started with my session, the last 10-5 minutes before getting on the stage I felt stress building up. Some of it lingered on through the intro part of the session, but left me relatively quickly after that. I had some people in the room that I met earlier that day and that already helps as a focus point during the session. That really kept me on track.

Having a full room with 60+ people was awesome: there where actual people, nodding along, frowning if they couldn’t follow it and even asking questions in between! Having that interaction is key to deliver a good session for me, I need to have that connection I have learned.

Afterwards people actually hung around and asked questions! More engagement happened during the ‘hallway track’, where people randomly come to you, tell you something on how they experience the session and some things they picked up from it. AWESOME! I got some great feedback on the things that stuck with them and learned that I still like doing this. That means I will be sending in some of these session to CFP’s (Call For Papers) again, because I wasn’t sure before.

Sessions, slides and links:

The sessions where also recorded for people attending at home and the recordings should be shared afterwards as well! I’ll update this post when they become available, so you can watch them if you are interested.

Session & Slides	Link	Recording
Protect your code with GitHub Security features	Link	YouTube
Protect yourself against supply chain attacks through your pipeline	Link	YouTube
Using GitHub Actions with Security in Mind	Link	YouTube

GitHub Advanced Security

Want to learn more about these GitHub Advanced Security features?
Check out the LinkedIn Learning course I made for it:

Click the image to go to LinkedIn Learning

GitHub Notification Settings

2022-03-12T00:00:00+00:00

I notice a lot of people getting lost in their GitHub notifications. Here is what you can do to get some control back! The default settings send you emails for everything. A lot of people then create an email rule to move all those emails to a specific folder, which means they will never look at those emails again! With some tweaking you can make the notifications work for you.

If you want to see this in action, watch the video I created:

Intro

There are a couple of things at play when using notifications:

settings for how and where you want to receive the notifications (email, web, etc.)
reasons for getting a notification

Telling GitHub to send you a notification can happen in a few different ways. You can subscribe to updates, either at the organization level (bad idea: way to noisy), at the repository level or for a specific issue, pull request, discussion. People can also @mention you or a team that you are in, which can then trigger a notification for you.

Step 1: check you settings on where you want to receive the notifications

Go to github.com/settings/notifications

Automatic watching

Under this section you find two options, both of which I would like to control for each organization I am in. Unfortunately you cannot at the moment.
By default these checkboxes are checked, which means that you will receive notifications for all the repositories you have write settings to. If you are an admin, or like me a trainer: you will get a lot of notifications, so it’s often a good idea to turn this off. In my case, I am one of the trainers who use an organization during these trainings and give our trainees a repository each to work in. Since we are admins on the org, we would get write access to each repo, meaning a notification for each issue and pull request that happens in it! Given that one of the exercises is to enable Dependabot on a test repository, that is a lot of noise for me!

The other setting in here is for all the teams that you are added to. If your team uses discussions in GitHub to reach out to each other, this setting is often very useful to leave on.

Participating and watching

These settings define the medium used for notifying you: email, or through the UI. Since the emails do not work for these (you can use the notifications as a personal backlog if you use the UI and not through mail), if always configure this to only use the UI (web and mobile). For triaging on the go I usually use the mobile app and then the things I really need to work on stay in the notifications list. Every once in a while I go through them and act where needed.

Dependabot alerts

With this setting the places where the notifications from Dependabot alerts (‘you have a vulnerable dependency’) are visible. You can then still configure it to also send you an email:

when the vulnerability alert is created
and / or once a week as a digest

Actions

With these settings you can tell GitHub how to send you notifications for your workflows. Then you can tune that down a bit to only send notifications for failed workflows. I really wish they will improve this feature so that it cooperates with me looking in the UI to see if I have already watched the workflow fail: often during the creation of the workflow, you are already working in the UI to check the logs and fix them. After a while you find 40 notifications, for all the failed runs you have already looked at. I’d rather only have one notification for each failed run that I have not yet seen. ![Notification settings - Actions](/images/2022/20220312/20220312_04.png]

Organization alerts

Not a lot of people will have these rights, but you can toggle of the notifications for when someone creates a deploy key.

Email notification preferences

With these settings you can pick for which type of event your notifications emails will be send and to which one of your verified email addresses.

Custom routing

With custom routing you can configure the preferred email address for each organization. For example you could have a single account to login to your personal account and your work organization. Then you might prefer to send the email notifications for your work organization to your work mailbox.

Step 2: Tune down the amount of notifications!

On github.com/settings/notifications you can use the filter options to find out why you are getting them: you might be ‘following’ (called watching) the repo (and thus get a notifications for anything that happens in that repo), or have subscribed to specific pull requests or issues.

In the lower left hand side of the notifications view there is a small link ‘manage notifications’ which will give you this popup:

Watched repositories and subscribed to PR’s and issues

With ‘watched repositories’ and ‘subscribed’ you can see which repositories you are watching and which pull requests or issues you are subscribed to. Since the watched repositories give the most notifications, you want to start here:

Go through the list of repositories and figure out if you really need to be watching for every thing that happens in it. Often you want to be notified on participating and @mentions. To turn them off, use the ‘Ignore’ settings for that repository.

The power option here is the ‘Custom’ setting:

Now you can really configure for what things you want to be notified. So this is somewhere between the ‘All activity’ and ‘Participating and watching’ settings in.

Why did I get this notification?

You can go to the notification and click on it. That will bring you to PR / issue / discussion that was the reason the notification was sent. The UI tries to help you figure out why you are receiving it:

Hope this helps making your GitHub notifications more manageable. If you have any follow up questions, leave them in the comments below!

Configuration as Code for the GitHub platform

2022-03-12T00:00:00+00:00

I am slowly diving into ‘Configuration as code’ for the GitHub Platform: all the things you want to automate with as few steps as possible, making big impact. Some of these things also fall under ‘GitOps’ in my opinion: if you store it into a repo and on changes you make, the automation will make it happen.

The plan is to have this post as a central starting point for people searching to achieve a similar setup. There are loads of people who blog on how to make this happen and what works for them, but often there is no actual implementation they can share. I want to give the you the examples and give you (a copy of) the code used as well.

Note: Since most of these items need to be running in a separate org, I created them on my robs-tests org.

Scenario:1 Automate user onboarding and repository creation

For on of our trainings we invite all trainees into and organization and create teams and repositories for them. Doing this with the UI is cumbersome and error-prone. We want to automate this process where someone can edit a yaml file, lint it and through a pull request approve it so that you always need another person to verify the incoming changes. After merging a workflow starts that make the new situation happen.

The steps that I want to have are as follows:

Add new users and teams to the users.yml file
Create a PR
The workflow pull_request.yml workflow checks if:
- The user file is valid yaml
- The users is a valid GitHub handle
- The user is already a member of the organization
After merging, the user-management.yml workflow runs and:
- Create the team if needed
- Add the user to the org
- Create repository attendee-
- Add the user to the repo
- Add the user to the team
- Add the team to the repo

Link to repo: robs-tests/user-test-repo

Step 1: define a yaml structure and parse it

In this example I want to start a simple structure, parse it, and then loop through the results. You can do this in any format you want. I thought of doing this in json for example, but that gives you a lot of overhead with all the extra double quotes and it is a bit harder to read. It would be easy to link a user to the wrong team for example. I knew I can parse yaml with a library, and that is what I went with: it will be compact, no extra characters around the content. Since for our trainings we usually have a max of 20 people in the group, the entire team and their users will fit on the screen without scrolling.

yaml format:

This is the format I settled on for now: there is a list of teams and in each team there is a list of users.

teams:
  - name: team01
    users: 
      - rajbos
      - Maxine-Chambers

Reading that from parsing the yaml gives me two loops to create: for each team and then for each user, do ‘x’.

Step 2: define the way you want to build your workflow

For most languages there will be an library available to parse the yaml, so it becomes a choice on what you would like to use for the automation. It depends on what your team already knows and how easy you want to be able to test this. These days I skip having a hard to manage setup and go to something really simple: github-script. This is a JavaScript Action that you can give your own script file and it will execute that for you. Inside your script you then get access to the GitHub contexts with authenticated clients and calls for all the API’s you need. You can even run it with an access token from a GitHub App (since we only use the organization level API’s here, this will work).

github-script also helps with having your code in JavaScript already: if you later want to make this a building block and make an action out of it, you are ready to go!

Step 3: the PR workflow

In the PR workflow I want to verify that:

the yaml file is valid yaml
the handles in the file are valid GitHub handles (we sometimes get e-mail addresses or typos in the handles)

    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2
        with:
          node-version: 14
          
      - run: npm install yaml
      - uses: actions/github-script@v5
        name: Run scripts
        with: 
          github-token: ${{ secrets.GH_PAT }}
          script: |  
            const yaml = require('yaml')
            
            const repo = "${{ github.repository }}".split('/')[1]
            const owner = "${{ github.repository_owner }}"
            const userFile = "users.yml"
            
            const script = require('${{ github.workspace }}/src/check-pr.js')
            const result = await script({github, context, owner, repo, userFile, yaml})
            console.log(``)
            console.log(`End of workflow step`)

In this example you see that we are:

checking out the repository
setup node
so that we can install a node package that can parse the yaml
then we use the github-script action to execute the script
load the file check-pr.js that will do the work
we pass in the info needed for the script to have all the context it needs

In the check-pr.js file you can see that we are:

loading the contents of the yaml file in the current branch: github.rest.repos.getContents({owner, repo, path, ref})
get the list of current teams: let existingTeams = await getExistingTeams(owner)
parse the content of the yaml file: const parsed = yaml.parse(content)

And then we can loop through each element in the arrays:

  for each team:
    for each user in team:
       check if user exists

Verifying if a user exists can be done with a call to this API: https://api.github.com/users/${userHandle} Checking if that user then already is a member of the org can be done with a call to this API: https://api.github.com/orgs/${orgName}/members/${userHandle.login}

Step 4: the user-management workflow

When the PR is merged to main, another workflow executes: user-management.yml. This has the same setup: install the npm packages, load the script and run it.

From load-users.js:

// send an invite to the user on the org level:
await addUserToOrganization(user, organization)

// create a new repository for this user:
const repoName = `attendee-${user.login}`
await createUserRepo(organization, repoName)

// give the user admin acccess to the repo:
await addUserToRepo(user, organization, repoName)

// add the user to the team for the day of the training:
await addUserToTeam(user, organization, team)

// add the team to the repo (so that the rest of the team can help with PR's):
await addTeamToRepo( organization, repoName, team)

Final thoughts

With this setup, you now have a complete example how you can use GitHub Actions to automate the process of adding users to an organization and preparing things like teams and repositories for them. You can build on top of this with for example:

a setup that has a folder structure that defines the hierarchy of teams and users
each folder could then have different code owners that defines who needs to approve the PR (give the team itself self-service on who to add)
add more properties to the users.yml in case you need it

GitHub Codespaces creation with the CLI

2022-01-26T00:00:00+00:00

This one took me some time to figure out, so I wanted to have something for the next time I need it :-). I needed to create repos for a team and then create a codespace for them. I already was using the GitHub CLI for my automation, so I wanted to use it as well to create the codespace. The documentation states that you can call gh codespace create flags and you are good to go.

Well, this command in the CLI works differently then you might think: First of, most of the commands work and pickup the repo that you are currently in. This command does not: you need to pass in the -r parameter and specify the repo using the <owner/repo> notation.

Next up is the -m parameter to specify the machine you want to have as the compute option for the Codespace. In the REST API you do not need to specify this parameter and it will give you the type of machine in the response:

"machine": {
    "name": "standardLinux",
    "display_name": "4 cores, 8 GB RAM, 64 GB storage",
    "operating_system": "linux",
    "storage_in_bytes": 68719476736,
    "memory_in_bytes": 8589934592,
    "cpus": 4
},

The help for the CLI command doesn’t give any hints as well.

So, I tried to be smart and send in this json as the value, thinking they’ve missed that wrapper in the CLI. That fails with no extra information, probably due to some wrong string escaping on my end.

Then I entered ‘standardLinux’ as the value, thinking this will never work, since I want to create it with a certain size, and all flavors are Linux based….

This is then the response:

 gh codespace create -b main -s -r "globaldevopsbootcamp/team01" -m "standardLinux"

error getting machine type: there is no such machine for the repository: standardLinux
Available machines: [basicLinux32gb standardLinux32gb premiumLinux largePremiumLinux]

Finally some hints of the expected values! (even better: these are the expected values 😄).

Here you can see how those values match to a machine type:

Tips

Keep in mind that the API for Codespaces is still in beta.
Do note that for automating these API calls you cannot use a GitHub App: the scope for the Apps is not available (yet?). You can make this work with an OAuth App or a Personal Access Token (PAT).

Automating my home setup: turning on the lights when the camera is in use

2022-01-17T00:00:00+00:00

I have a nice working from home setup that allows me to use a great camera, lights and microphone. I have so many stuff, that I wanted to automate some of it to detect if I am working or not and then toggle them all on or of.

I already use Home Assistant to remotely toggle loads of stuff in the house, so why not integrate everything?

Home Assistant Scenes

For this I have create some Home assistant scenes and a script to automate these actions. Next I want to toggle based on the state of things on my laptop.

Scene 1 - Office lights

When this one is triggered, my office lights (small desk lamp), my ‘Do Epic Shit’ signal and my speakers will turn on: everything I need to start working (laptop and monitor have their own flow and can be considered as ‘Always on’). I’ve set this up as with Shelly Plug S and an extension cord that powers all three devices. Wrapped it in a scene in Home Assistant together with my Elgato Light Strip for easy switching it on and off.

Devices:

Shelly Plug S (extension cord, Speakers, Epic Shit signal)
Elgato Light Strip

Scene 2: Camera On (to be automated)

When the camera is on (I have two different ones to use), I switch on this scene to turn on the 2 Elgato Key Light Airs that I have, so that people can actually see me (check my blogpost on my setup here). When I am done with the call, I switch off the scene and the lights turn off.

Devices:

Elgato Key Light Air Left
Elgato Key Light Air Right

Scene 3: Office Leave

When I stop working, everything I have automated needs to turn off again, so this scene switches of the desk lamp, the Light strip, the ‘Do epic shit’ signal, the speakers and the Key Lights.

Devices

Shelly Plug S (extension cord, Speakers, Epic Shit signal)
Elgato Light Strip
Elgato Key Light Air Left
Elgato Key Light Air Right

Trigger - Starting to work

When I logon to my machine, I want to trigger the Office Lights & Speaker setup (Scene 1). For this I need to send the Office Lights On signal to Home Assistant. For this purpose it has a REST API that you can configure to securely send the signal to your Home Assistant.

Since I am running on a Windows Laptop, I can leverage the Windows Task Scheduler to automate this. I have a Windows Task is triggered on ‘workstation unlock’ and run a PowerShell script that sends the signal to Home Assistant.

Note: Make sure to use the `pwsh` command as the program to start the script, otherwise it fails because of missing modules/parameters on PowerShell

To make sure this only runs when I am at home, I configured the Windows Task to only run when the network connection from home is available.

To make sure this only runs when I am at my office space (that means that my peripherals are plugged in), I have a check in the script to verify if there are more than one monitors available by making this call:

Get-CimInstance -Namespace root\wmi -ClassName WmiMonitorBasicDisplayParams

This prevents the lights from turning on while I am using the laptop at the couch 😁.

Camera on/off

I also want to toggle the lights on or off depending if I am using the camera or not. Since I have multiple camera’s setup and LOADS of video conference applications in use (I always think I have them all, until something obscure pops up), I want to do this automatically by detecting if a camera is in use.

Looking around you can find solutions like Presence Light, that only look at the status in the Office365 API. Since I also use different tools, I needed something more.

Searching around I had to put some things together and I created this PowerShell script that checks ALL cameras that are connected to the machine and then checks if any process is using them.

Step 1: Find all camera’s on the machine

You can get all the connected devices from the Get-PnpDevice (plug and play) commandlet. Give it the classes ‘camera’ and ‘image’ to get all devices that can be used as a camera.

Get-PnpDevice -Class Camera,Image

If you have ever used a camera at a different location (I connect to a standalone camera at our office space for example), you’ll see that those devices are stored there as well.

Step 2: For each camera, see if they are connected

Only for connected camera’s, you get a Physical Device Object Name (PDON) back from the call below that you can use to check if the camera is in use. If the result is empty, the device is not connected so you don’t need to check if it is use.

Get-PnpDeviceProperty -InstanceId $device.InstanceId -KeyName "DEVPKEY_Device_PDOName"

This call also returns the current file that can be used to see if a process has a handle on that file. It will look something like this: ‘\Device\000000ac’. See the logs below for more examples.

Step 3: Check if any process is using the camera

That file can be used to check if any process is using the camera by checking the handles. For this you can use the handle tool from SysInternals.

Tie it all together

For starting a loop that checks all camera’s and sees if they are in use, I use a Windows Scheduled Task again: If connected to the home wi-fi, when I unlock my laptop, start running a PowerShell script that continuously checks if any camera is in use. If so, turn on the lights. If not, turn them off. I run this check every minute, since finding all handles is a bit of a computational expensive operation.

Output example

Below you can find an example of the output of the script. You can see that it found 5 camera devices:

the internal camera is in the list twice, no idea why
I have two USB camera’s connected
The Elgato Facecam is the camera from our office space

After scanning I wait for the remainder of the minute to not continue check for cameras in use and make the fans fly off 😀.

13:31 Searching for camera devices
13:32 Found [5] camera devices
13:32 1.  No PDON found for [Elgato Facecam] so skipping it
13:32 2.  Checking handles in use for [Logitech BRIO] and PDON [\Device\000000ac]
13:38   - Found [2] handles on Logitech BRIO
13:38   - Found process [Teams.exe] that has a handle on [Logitech BRIO]
13:38 Found active camera device
13:38 Running action to make the state [True]
13:38 Sleeping for 54 seconds

Tested with these video conferencing applications

So far I have tested this with the following applications:

Zoom
Teams
OBS Studio
Calls through Edge browser with Teams and Zoom
Slack

Run with highest privileges

For some of these tools the preview window (before you join a meeting) runs in a svchost.exe process. That means you need to run the handle calls with admin privileges. You can set this checkmark in the main window of the Windows Task.

Prevent the PowerShell window from popping up

If you run with the setting ‘Run only when the user is logged on’ on your Windows Task, you’ll see the PowerShell window pop up. To prevent this, check the ‘Run whether the user is logged in or not’ box.

Repo with the scripts

You can find the repo with all my setup on GitHub.

GitHub Access Tokens explained

2022-01-03T00:00:00+00:00

There is a lot of confusion of what GitHub (access) tokens are and how you should use them for automating things inside of GitHub. There are three main types of tokens:

Personal Access Tokens (PATs)
The GITHUB_TOKEN environment variable (explainer here)
An access token created from a GitHub App (explainer here)

You can use these tokens to authenticate to GitHub and perform actions with it, like cloning repositories, making API calls, etc.

Photo by James Sutton on Unsplash

1. Personal Access Tokens (PATs)

This type of token is often the first thing that people start to use when automating things. It is the same for different CI/CD systems like Azure DevOps that I’ve used before. A personal access token is inked to a user account and can be set to automatically expire. These days they are prefixed with ghp so that the secret scanner can detect them more easily.

You can use the PAT to do anything the user account that created them can do (as long as it is given the appropriate scopes for it):

Create repositories
Create issues
Create pull requests
Read security alerts
etc.

The downsides of Personal Access Tokens

This token can actually do anything the user can do, but for anything the user has access to! If the user has access to repos, organizations or enterprises that have internal/private repos, the PAT has access to it! The only thing you can do is limit the scope it has, but not organizations or repos. (note: this is planned on the roadmap and very needed).

Because of this, using a PAT poses a big security risk. Do not hand them out to any random action in your workflows: if they store the PAT somewhere, they can read all your private repos (and more!). I recommend to stay away from using PATs if you can help it.

Additionally, they are linked to a user, so when that user leaves the company (and the user account is disabled): all automations using their PAT will stop working!

Note: the only valid reason for using the GitHub Token is for accessing things in the API from the Enterprise level, in case you need to automate things like the creation of users, which should be done with SCIM as a best practice. So still, most of this reason is not valid 😏.

2. The GITHUB_TOKEN environment variable

The GITHUB_TOKEN is an environment variable you can use in your workflows by injecting it wherever you need it: $. The token is autogenerated and is not stored anywhere. It is a token that is only valid for the duration of the workflow it was created for. By default it has read and write access to the repository the workflow is running for. That means you cannot use this token for another repository. You can use it to do things like create issues, create pull requests, or comment on them. Depending on the context, it might get less permissions, for example: when running on a Pull Request coming from a fork, it will only have read access to the repository.

You can also give it a specific scope by setting that inside the workflow:

permissions:
  contents: read
  pull-requests: write

This way, the token cannot be used for creating an issue for example, so even if the action you use tries to create an issue, the token doesn’t have the permissions to do so and thus it will fail.

If you want to learn more on the GITHUB_TOKEN, I have explained its use and how to limit what it can do in this short video.

3. An access token created from a GitHub App

You can use a GitHub App to have very specific access to one or more repositories. I use this for example for setting up a (lot of) Jenkins connections: each team at my customer has their own set of Jenkins jobs that should trigger when someone pushes a new commit to their GitHub repository. By giving each team their own GitHub App, I can limit the access the App has to only the repos that are relevant to them (by installing the App on only their repositories and not all repos in the organization). Since this token is tied to the installation of the App, we call this an installation token.

Note: on github.com you can only create 100 apps per organization. On GitHub Enterprise Server this restriction does not apply.

With a GitHub App, you get an AppId and a private key in the PEM format. You can create a GitHub App on a website and then install it on the repositories (or organizations) you want to use it for. You can then use the AppId and the private key to create an access token that can be used to access the repositories.

There are multiple options to get the token, depending where you want to use it:

Normal shell scripts
Use an action
Use a library (can be included as an extension to the GitHub CLI)

Note: this token is only valid for 1 hour, after that you need to refresh it or your calls will fail

For an explainer of how to create a GitHub App and then use it in for example a GitHub workflow, check my video on it here:

Get an access token for an App with shell scripts

I have a GitHub Gist that shows you how to get an access token from a GitHub App from shell scripts (I have an example how to call the shell script from PowerShell as well).

The steps are:

Get the AppId and the private key from the GitHub App you created
Generate a signed JWT token with the AppId and the private key
With the JWT toke, get the installations of the app (you need the installation id to create the token)
Create a token for the installation
Use the token to access the repositories

You can then use this token for any API call you want to make (and that is accessible for an App. Creating codespaces is not for example, since you can only create a codespace for your own user account).

In the gist you also find an example of using git config rewriting to overwrite the global git config with a rule that maps a SSH setup to a HTTPS setup that includes the access token. This can be helpful to help custom scripts / tools that can are preconfigured with only SSH support.

Get an access token for an App with an action in a workflow

Inside of a workflow I always use the action from Peter Murray. It takes an application_id and an application_private_key and generates a token that can be used to access the repositories. You can even give the token it creates a scope by sending in the permissions parameter.

Get an access token for an App with a library

My buddy Bassem Dghaidi has created a bash library that does the work for you, and can be included in the GitHub CLI. You can find it in this repo with all the setup needed. You can for example use this easily in a Docker container when you are automating things.

Note: for installation as an extension in the CLI you need an authenticated CLI session first, so that is not helpful for automation purposes.

The downside of using GitHub Apps

GitHub Apps are great for automating things: they have more access then the GITHUB_TOKEN (across repositories for example) and do not have the issue that they operate from a user account and have access to everything the user has access to (like a PAT).

There are some downsides as well:

‘Only 100 apps’ per organization on github.com (so public repos or GitHub Enterprise Cloud: GHEC).
Only minimal options to automate the setup of the apps themselves: you can use a manifest to create them, but installing them on repositories (even the ones you own), is not possible with an API.
Little to no support on external tools: most of them want to have the token and cannot generate that token from the AppId and the private key, so you need to do that yourself (and be aware of the 1 hour expiration!).

Some things I have used GitHub Apps for:

Installing and configuring self hosted runners.
Configuring access from Jenkins pipelines to GitHub repos.
Inside workflows everywhere: I like the limited access the App has and do not want to give anything my PAT as it has way to much access!
Automating creation of repos with defined content for Global DevOps Bootcamp.

Create a GitHub App from a manifest

2021-12-27T00:00:00+00:00

At my customer we have the need to create a lot of GitHub Apps. In this specific case we use GitHub Apps as an integration point between GitHub and Jenkins: the code is moving to GitHub, and we still want to trigger our existing Jenkins jobs on code changes. We have over a 100 teams in Jenkins, all with their own pipelines. We have a security requirement that teams that connect to their code in a Jenkins pipeline only can see their own code, and not the repos from other teams. That means that each team has to have its own GitHub credential set up for them.

Photo by Hello I’m Nik on Unsplash

I consider it a bad practice to use Personal Access Tokens (they have way to much scope, which should improve in the future). Instead we use a GitHub App: we can create and install the GitHub App the organization level and give it access to certain repositories. This way we can use the GitHub App to trigger our Jenkins jobs and give it only access to the repositories it need access to.

At the moment there are no API’s in GitHub to handle this situation for you: you have to create the App manually, get the credentials and install it in the organization for the repositories that you need the App to have access to. Creating the App has quite a few steps, like setting up a name and a description, adding webhook to trigger, configuring all the permissions you need and the events that belong to those permissions.

Given that we have a lot of teams, this is a repeating task and prone to errors when you follow the internal documentation for setting this up. To improve this, we have created a manifest that we can use to create the App.

Note: you can also do the same with a request with request parameters.

Flow

With a manifest, we can use a flow in an internal webapplication that will create the App with all the default settings, get the credentials and store the credentials somewhere safe. Since this flow requires a user to be logged in, we need to give them a webpage that set things up (the manifest) and then redirect to the page on our GitHub environment (Can be Enterprise Cloud or Enterprise Server), that will do the authentication and authorization to check if the user has the rights to setup the App. Unfortunately that is also as far as we can go, since there are no API’s to install the App into repositories.

In the overview above we can see that there are six steps to create the App with a manifest:

Send the user to our index page where we setup the manifest (we let them pick the environment this app needs to be created in) and redirect to the GitHub App creation page with the manifest.
GitHub handles the authentication and authorization.
Redirects the user back to our redirect page (you pass this in with the manifest).
Our redirect page loads a specific code from the redirect (this code is only valid for 1 hour).
And posts that code back to GitHub to get the App information it just created.
We store the AppId and the private key (PEM) for the App somewhere safe, so we can continue with the next steps.

Example files for this flow can be found here. You need to host them on your own server, but redirecting to localhost works just fine. (next step for us will be hosting that in a NodeJS app that can run either locally or be hosted in a Docker container). It also includes a 5 minute video that explains the whole flow.

Next steps:

Setting up the manifest is straightforward, but only part of the setup we need. To be able to configure this on the client side (in this case Jenkins), we need to go through the following steps:

Install the new App in the repositories it needs to have access to (you give it access to those repos from the org level).
Add then new set of Credentials (AppId + private key) in Jenkins at the correct folder / team level

Future work:

Following DevOps best practices, we want to setup key rotation for the app’s as well. To do this there are also no API’s available at the moment, and when you generate a new private key, the old one is revoked immediately 😒.

Maturity levels of using GitHub Actions Securely

2021-12-11T00:00:00+00:00

I’ve been discussing using GitHub Actions in a secure way for a while now (see here), and I got a question on how to improve your usage of actions. I wanted to capture that info in an easy to follow set of steps, so here we go:

Default demo examples: version pinning or by branch
Review the source code and trust the publisher / action
SHA hashes
Dependabot for actions
Fork the repo and take control
github-fork-updater
Internal marketplace
Request actions process

Photo by Markus Spiske on Unsplash

I’ll go over each of these below and give some more context. With that you should be able to determine where you or your company is at this scale. I’d love to what your next step will be, so please let me know on LinkedIn!

1) Default demo examples: version pinning or by branch

This is currently where all the demos start: use version pinning (now required) or by branch:

    - uses: actions/checkout@v1
    - uses: actions/checkout@main

The engine and UI now force you to use one of these ways, or fail to start the workflow or save it.

The first line is referencing a version which was used to publish the action as a release. You can find all the released version in the action repository itself: link. Often the release will have release notes and a list of the commits that made it to the new release. Actions should be following semantic versioning which means that you can use v2 to always use the latest compatible version with v2. So if the publisher releases v2.14.17, the runner will use that version.

The second line references a specific branch in the actions’ repository.

For both options, the runners will download the entire repository by calling either git checkout (if git is installed on the runner) or downloading the status of the repository as a tarball. The runner is open source, so you can follow along with the steps it is taking.

The issue with using both, is that you are pulling in arbitrary code from the internet! Even if you follow best practices, you should look at what the action is doing for you. GitHub has no documented process for publishing an action or a security check on them: anyone can set up a public repository with the right content and then everyone can use it. Very helpful to get started fast, but security is not part of that picture!

Why is this first level so bad?

As I said, the methods above always use a version of the action as is. The branching method uses whatever was pushed last to that branch. So even if you have just reviewed it and start using it, a newer version might just have been pushed to that branch. This happens without you knowing of it at all! Anything can happen between you reviewing it and you using it. A vulnerability in a package the action is using might be found, or the maintainer decides to export all environment variables to their own server. Or maybe even the maintainer hands over the code to a third party so they can do the maintenance. You never know, but might be using an action in production that is not what you intended.

For version pinning the same principle applies: you might even pin a specific version, say v2.1.4 and think you’re now safe: you are not! The version of a release comes from a git tag. Git tags can be reused! They are flexible by design, so the maintainer can tag code and push that as v2.1.4. Then, even months later decide to add some code, maybe introduce a vulnerability and reuse the same tag! Then the runner will download the version of the repository as it is with that tag linked to it and you are running code in your workflow that you never intended.

2) Review the source code and trust the publisher / action

The first step in being better at using actions in a secure way is to review the source code: know what the action will be doing on your behalf! The repo is open source, so go to the repo:

Did you know you can even use the action call it self to find the repo? The uses statement can be partially copied behind github.com to have the direct link to the repo! So this uses: actions/checkout@v1 becomes this github.com/actions/checkout!

Now find the action.yml in the root and find it’s entry point under runs, for example the checkout action:

runs:
  using: node12
  main: dist/index.js
  post: dist/index.js

The using indicates that this action runs under node, so it is JavaScript based. This can be compiled TypeScript, so the next step is to look for the entry point and how it got there. Often you find a src folder and the TypeScript files in there (*.ts). Usually the starting file is then main.ts or index.ts.

In this case we see a main property that indicates the starting point when this action starts, so index.js in the dist folder. This already is a good chance this action is build with TypeScript. You often can find where it actually starts by checking the packages.json file. For this action we find "main": "lib/main.js" in there, so we can find where it actually starts 😄. For TypeScript the file names just have the .ts` extension.

We also have a post definition here, which means that part of the action will also run at the end of a run:

To summarize, there is no verification or security label before someone can use an action, so you need to do your own due diligence! Check what the action is doing and make an informed decision if you can trust the action.

3) SHA hashes

Now that you have reviewed the actions’ source code, you need to make sure you always use that specific version of the action. This is where the SHA hashes come in. Each git commit gets its own unique SHA hash, based on the contents of the commit. That SHA hash is unique! Generating the same SHA hash with different contents is possible, but very hard to do. Having the contents of the repo as a pre-specified setup (with an ‘action.yml’ file for example), makes it very unlikely that you will get the same SHA hash for different code (we call that SHA collisions). That means we can use the SHA hash to indicate a specific version of the action we will use. The runner will detect the SHA hash and use that to checkout the actions repository at runtime. Since adding or changing the code in the repo will mean a new SHA hash, you will always use the version you have reviewed.

You can find the SHA hashes in the history of the repo, by going to the commit history. Don’t use the short version of the hash, that was insecure (collisions more likely) and does no longer work.

You can then find the full SHA hash here:

4) Dependabot for actions

Dependabot also supports actions as an ecosystem. It can run on a schedule and check all workflows in your repository for the actions you use. If you use version 2.1.5 of an action and the publisher has released version 2.1.6, Dependabot will create a Pull Request for you that updates the action to the latest version (major/minor updates configurable).

Dependabot supports all usage options for the actions that indicate a version:

owner/action@pinned-version
owner/action@SHA-hash

version: 2
updates:
  - package-ecosystem: "github-actions"
    directory: ".github/workflows/"
    schedule:
      # Check for updates to GitHub Actions every weekday
      interval: "weekly"

The only downside is that in the Dependabot Pull Request, you only see the version number (or SHA hash) change in the PR. If the publisher has added commits or release notes, that will be available on the ‘Conversation’ tab of the PR. It’s up to you to check the actual code changes!

5) Fork the repo and take control

Now that you have at least a safer way of using actions (by reviewing their code and pinning it to the version you have reviewed), you are ready for the next step: taking full control over the action and any updates to them. This is one of the best practices for using actions and was always the initial guidance for using them: fork the repo!

By forking the action you have a full copy of it, so in case something happens (newer versions, or the maintainer deletes the repo), you workflows will continue to do their work happily. You are now also in control of any incoming updates. More on that in the next level. I always use an organization called org-name-actions for this, so that I have a single location for all the actions we use and an easy way to limit the use of ALL public actions (see next paragraph). Especially in an work setting, you do not want your GitHub users to just run any action from the public internet, you need to check these things first!

The nice thing is that you now can enforce that the users in your organization can only use the actions under your control. Go to your organization –> permissions –> actions permissions and select ‘Allow select actions’:

You can now enter the actions you want to allow for the selected repositories in your organization (you can also do this on the repo level). If someone triggers a workflow that does not adhere to these limitations, it will give an error and will not start. No action repos will even be downloaded on a runner.

You can go completely crazy in this list:

Setting	Description
`owner/*`	All actions from this owner are allowed
`owner/action@*`	The action in the repo ‘action’ from this owner are allowed for all versions and all branches
`owner/action@main`	The main branch of this action is allowed
`owner/action@v2`	All versions for v2 are allowed for this action of this owner
`owner/action@SHA`	Only this version is allowed for this action of this owner

Note: ‘owner’ can be either an organization or a user account here.

6) github-fork-updater

Now that you have a fork of the repo, you need to maintain this fork. The best way of doing that is sharing any changes you make back to the publisher of the action (yeah for Open Source) by clearly communicating what you changed and why. Then send in a Pull Request.

You also want to incorporate incoming changes from the publisher back into the fork. You can wait until someone spots the message on the repo that says ‘you are x commits behind the parent repo’, but that doesn’t scale of course. There is also a big button on that banner that lets you pull in all the updates automatically.

DO NOT USE THAT BUTTON! Use the compare in the drop down first!

You’d be blindly pulling all the changes from the parent repo! I think the explanation from level 2 where clear: review what the action is doing for you! That means you need to check the incoming changes!

To scale this process, I have created the github-fork-updater repository that centralizes the process for you. It checks all forks on a schedule and will create an issue for you when there are updates. You can then review the incoming changes and decide to update the fork. More is explained in this blogpost.

7) Internal marketplace

The next maturity level is having a setup to let the users in your GitHub organization find the actions you have available in your actions-org. They can search the organization of course with the normal search options, but that means searching in all code in the repos, trying to find something that the action should do. You only want to search in the action.yml and the readme. Having a better search experience therefor is a nice way to send your users to a central location: the internal marketplace. Having all that data in a single location also has additional benefits. More on that later.

The internal marketplace groups all your (internal/private or public) actions in one place, with the information from the action.yml and the readme for users to search on. That way you can send your users here to find the actions already approved within your organization. Read more on the internal marketplace here.

Note: this is still work in progress, one of the things I still want to add is adding links to the internal usages of the actions in case a vulnerability is found or for implementation examples.

8) Request actions process

The last maturity level is setting up a good governance process to add actions to the internal marketplace. More information can be found in my blogpost on it here. We have created a repo for it where we can go and:

Users create an issue to request a public action to be added to the internal marketplace
An engineer with a security mindset (and training!) does a preliminary manual check on the actions’ source code.
The engineer can request an automated scan
The scan forks the action over to a test organization and enables Dependabot for the vulnerability alerts
Runs a CodeQL scan
Checks the actions repo for its own Dependabot and CodeQL setup
Runs a Trivy scan on any containers in use
Drops the results of the scans back into the request issue
The engineer can make an informed decision to decide if they want to take on the risk of using the action
On approval, the action gets automatically forked over to the org-actions organizations, where it will be useable and be picked up by the internal marketplace

Summary

The eight maturity levels is something that each action user hopefully goes through when they are told about the risks of using actions out of the box. For more insights on using actions with security in mind you can watch my GitHub Universe session on this exact topic.

I’d love to hear where you are on the maturity scale and how you intent to improve you security: let me know on Twitter or LinkedIn!

dev.to entry: Using GitHub Actions to setup a Marketplace

2021-12-04T00:00:00+00:00

This post is for sharing my dev.to entry for the 2021 GitHub Actions Hackathon. This entry shows how I have setup the workflow(s) for the GitHub Actions Marketplace. I wanted to have a long form post detailing the steps and reasoning behind each the setup as an entry point for people building these automations.

Workflow:

The starting workflow for this setup can be found here. This workflow goes through all repositories in a user or organization and checks if they contain action definitions. If so, it adds the information about the action and the repository it lives in to a data file that then can be used. In this case the data is used to display an Internal GitHub Action Marketplace so our organization users. We lock our main organization down so that only the actions in our internal marketplace can be used as is a common best practice.

jobs:
  get-action-data:
    runs-on: ubuntu-latest

    steps:
    - uses: devops-actions/load-available-actions@v1.2.12
      name: Load available actions
      id: load-actions
      with:
        PAT: ${{ secrets.PAT }}
        user: ${{ github.repository_owner }}

    - name: Store json file
      run: echo '${{ steps.load-actions.outputs.actions }}' > 'actions-data.json'

    - name: Upload result file as artefact
      uses: actions/upload-artifact@v2
      with:
        name: actions
        path: actions-data.json

    - name: Upload json to this repository
      uses: rajbos-actions/github-upload-action@v0.2.0
      with:
        access-token: ${{ secrets.PAT }}
        file-path: actions-data.json
        owner: ${{ github.repository_owner }}
        repo: actions-marketplace
        branch-name: gh-pages

Workflow steps

Below I listed the steps in the workflow that do the actual work.

Load available actions from an organization

I created the load-available-actions action just for this purpose. It needs a token to access the GitHub API and will load all repositories it can find from either the user or the organization you give it. It loops through all the repos, scans for either an action.yml or actions.yaml in the root of the repository and adds the information to the output of this step so that it can be used later on in the workflow.

Store the outcome of the previous step as json

We store the output of the load-available-actions action in a file so we can upload it more easily to both the artefacts for this run (handy for testing and validation) and to the GitHub Actions Marketplace repo.

Upload the json as an artefact for easy testing

To view the results in the workflow run, I upload the json file as an artefact so I can download and check it if needed.

Upload the json to the GitHub Actions Marketplace

This is the important step for later usage of the data: store it inside the repository that holds the Internal GitHub Marketplace. For this I use this GitHub Action where I have added the option to upload the files into a branch: Since the GitHub Pages website from this repo runs on a gh-pages branch, I needed to upload the file in the same branch.

Open Source FTW!

Actions have to be Open Source to be able to use them by default (you can use private repos, but that is a bit more work), So if you miss something, you can start a conversation with the maintainer (use issues or discussions if they have that enabled) and even send in a Pull Request to propose the fix! The upload-action was missing some features that I needed for this project. I have added the following features:

Internal GitHub Actions Marketplace

You can read more information about the Internal GitHub Actions Marketplace I created in this blog post. The goal is to give our users a place where they can find the supported actions internally, since we block the use of public actions in our production organization.

Next steps:

Next to prettifying the marketplace and adding search, I want to add more information about the actions being used inside of our production environment. This way my users can see examples of their usages and I can easily find the implementations in case there is something wrong with the action, like a security vulnerability or newer features being implemented.

Loading the information is done by another action that loads all repositories’ workflow files and outputs a list of all the actions in use in them, with the information of which workflow uses with action and version of it. The action can be found in devops-actions/load-used-actions.

GitHubs magic files

2021-11-26T00:00:00+00:00

I keep coming across files in GitHub that have some mystic magic feeling to them. There’s always a small incantation to come with them: the have to have the right name, the right extension and have to be stored in the right directory. I wanted to have an overview of all these spells for myself, so here we are 😉.

Photo by Artem Maltsev on Unsplash

Overview

A list of all the magic files / links that I came across in GitHub. I also created a LinkedIn Learning Course for ~25 of these files, with more detail how to use them. You can find that course on LinkedIn Learning.

Filename	Location	.github repo support	Description	Docs
CNAME	root	no	Alias for the GitHub Pages site	Docs
CONTRIBUTING.md	root, /docs or /.github	yes	How to contribute to a project	Guidelines
CODE_OF_CONDUCT.md	root, /docs or /.github	yes	Code of conduct	How to behave for this project Code of Conduct
CODEOWNERS	root, /docs or /.github		List of people who can make changes to the files or folders	Code owners info
CITATION.cff, CITATION.md, and others	root or inst/CITATION	no	Let others know how to citate your work	cff
LICENSE or LICENSE.md or LICENSE.txt or LICENSE.rst	root	no		License
FUNDING.yml	.github folder	yes	Display a Sponsor button in your repo and send people to platforms where they can fund your development	Docs
SECURITY.md	root, .github or docs folder	yes	Instructions for how to report a security vulnerability	Security policy
SUPPORT.md	root, .github or docs folder	yes	Tell people how to get help for the code in the repo	Docs
workflow.yml	workflow-templates	only available in .github repo	Store starter workflows for your organizations	Starter workflow templates
dependabot.yml	.github/		Dependabot configuration file	Dependabot configuration
codeql-config.yml	.github/codeql/codeql-config.yml (convention, not required)	sort of	CodeQL configuration file. Can also be stored in an external repository (hence .github repo works). If using external repo, referencing can by done by using `owner/repository/filename@branch`	CodeQL config
secret_scanning.yml	.github/secret_scanning.yml		Secret scanning configuration file	Secret scanning
README.md	.github, root, or docs directory	yes, see below	Project readme, also used on marketplace if the repo is published to the marketplace	About readme’s
README.md	.github/username/username		Profile readme	About readme’s
README.md	organizations .github repo or .github-private repo: profile/README.md		Organization readme	Organization readme
release.yml	.github	Automatically generated release notes		Automatically generated release notes
workflow.yml	.github/workflows/			Workflows
action.yml/action.yaml	root		Configuration file for an actions repository
dependency-review-config.yml	.github	no	Dependency review configuration file	Dependency review
$GITHUB_STEP_SUMMARY	workflow		Job summary output in markdown	Job summary

GitHub Copilot Files

With the rise of AI-powered development tools, GitHub Copilot has introduced its own set of magic files to help customize the AI experience for your specific repository context. These files help Copilot understand your project better and provide more relevant suggestions.

Filename	Location	.github repo support	Description	Docs
copilot-instructions.md	.github/	yes	Repository-wide custom instructions providing context and coding guidelines to GitHub Copilot for all requests in the repository	Custom Instructions
NAME.instructions.md	.github/instructions/		Path-specific custom instructions that apply to files matching the `applyTo` glob pattern defined in the file’s YAML frontmatter. Both repository-wide and path-specific instructions are used when both apply	Custom Instructions
AGENTS.md	anywhere in the repository		Agent instructions for Copilot coding agent. The nearest file in the directory tree takes precedence. `CLAUDE.md` and `GEMINI.md` at the repository root are also supported as alternatives for other AI agents	Custom Instructions
NAME.prompt.md	.github/prompts/		Reusable prompts for specific and repetitive tasks that can be invoked in Copilot Chat. Supports YAML frontmatter for metadata like description and which tools to use	Prompt Files
NAME.agent.md	.github/agents/	yes	Custom agent profiles with YAML frontmatter defining the agent’s name, description, available tools, and MCP server configurations. Allows creating specialized agents with tailored expertise for specific development tasks. Available on GitHub.com, VS Code, JetBrains, Eclipse, and Xcode	Custom Agents
settings.json	.github-private/.github/copilot/	no	Enterprise managed plugin standards for GitHub Copilot CLI. Defines a plugin marketplace, auto-installed plugins, MCP server configurations, and hooks that are distributed and applied automatically for all enterprise users with Copilot Business or Copilot Enterprise	Enterprise managed client settings

Note: content exclusion (preventing Copilot from accessing certain files) is not configured via a file - it is set up through your repository or organization settings on GitHub.com. See the content exclusion docs for more details.

These files help you customize the AI experience by:

Providing repository-specific context and coding guidelines through custom instructions
Applying specific instructions to certain file types or directories
Guiding Copilot’s coding agents with information about your project conventions
Creating reusable prompts for common development tasks in your project
Building specialized custom agents with their own tools and MCP server configurations
Distributing enterprise-wide plugin standards, MCP configs, and hooks to all Copilot CLI users automatically (now in public preview)

Just like the other magic files, these need to be named exactly right and placed in the correct directories to work their magic ✨.

Then there is a whole list of templates you can configure for issues / pull requests / discussion:

Filename	Location	.github repo support	Description	Docs
FORM-NAME.yml	.github/ISSUE_TEMPLATE/		Issue templates with forms (in Beta for github.com, not available for GHES)	Templates
config.yml	.github/ISSUE_TEMPLATE/		Issue templates configuration settings	Template chooser
issue_template.md	.github/ISSUE_TEMPLATE/	yes	Issue template	Template
Url query	In the url link		Create an issue with certain fields filled in with values	Create issue with url query
pull_request_template.md	root, /docs, /.github or in the PULL_REQUEST_TEMPLATE directory	yes	Create the default body for a Pull Request	Using a PR template
Discussion category templates	/.github/DISCUSSION_CATEGORY_TEMPLATES	?	Create discussion category templates	Create discussion category forms

Some of these are extra tricky, like for example the organization profile lives in a different directory and repo then the user profile readme: .github or in .github-private repo in the org and then in a folder named profile: README.md.

Magic links

There are also some magic links that can be super useful.

Link setup	Description	Documentation
github.com/OWNER/REPO/releases/latest	Permalink to the latest release	Permalink to latest release
github.com/userhandle.keys	Get the public part of a users SSH key
github.com/userhandle.gpg	Get the public part of a users GPG key
github.com/userhandle.png	Get the profile picture of a user
avatars.githubusercontent.com/userhandle?s=32	Easy method to show user profile pictures anywhere. The `s` parameter is the size. Example output:
github.com/owner/repo#readme	Scroll the repo link to open up with the README text on the page. Since GitHub shows the file content of the repo first, this can be helpful to push you users down the page into the README section. This works because the README is based on a header in the page, so this is just normal HTML behaviour.

Atom feeds

A lot of things have atom feeds enabled. The things in all caps need to be configured:

Link setup	Description
github.com/OWNER/REPO/commits.atom	Get an RSS feed for the commits in that repo
github.com/OWNER/REPO/commits/BRANCH.atom	Get an RSS feed for all the commits in that branch
github.com/OWNER/REPO/wiki.atom	Feed for the wiki in that repo
github.com/OWNER/REPO/discussions.atom	Get an RSS feed for the discussions in that repo
github.com/OWNER/REPO/releases.atom	Get an RSS feed for the releases in that repo
github.com/USER.atom	Get an RSS feed for the user’s public activity
github.com/security-advisories	Get an RSS feed for ALL the security advisories

There should also be a feed for issues, but I continuously get HTTP:406 errors on github.com/OWNER/REPO/issues.atom. Other the user specific feeds can be loaded by making an authenticated call to https://api.github.com/feeds.

You can get an entire firehose of ALL issues on the GitHub platform if you want to: - https://github.com/issues?q=.

Personal views

github.com/issues - get a list of issues that are either created by you, or assigned to you
github.com/pulls - get a list of pull requests that are either created by you, or assigned to you
github.com/discussions - get a list of discussions that are either created by you, or assigned to you

Using separate git configurations with SSH for the same user

You can add a - to the ssh url, to have different ssh configs on you machine and use the right one for the right repo. Exampe: git@github.com-myworkaccount:devops-actions/load-used-actions. This boils down to using a separate hostname (github.com-myworkaccount) to use for the configured repo (devops-actions/load-used-actions).

GitHub Universe session: How to use GitHub Actions with security in mind

2021-10-27T00:00:00+00:00

My session on GitHub Universe is now available here. You can watch it for free!

Follow up questions

Let me know if there are any follow up questions. You can tag me on the discussions created for the event by using my GitHub handle: rajbos or comment on the YouTube video. I’ll add them below for future reference!

Setting up an internal marketplace

A blogpost on the topic to setup an internal marketplace can be found here. It contains the full setup from having a process for your users to running security checks on the actions before you fork them over to be used (last past is still a work in progress).

Slides

For anyone interested in the slides with all the links I mention, you can find them here.

Setup an internal GitHub Actions Marketplace

2021-10-14T00:00:00+00:00

One of the best practices of using GitHub Actions is to fork all actions that you want to use to your internal actions organization. If often use organizationname-actions for that, just like I am doing for my own personal setup here: rajbos-actions.

After forking the repositories I always get the question:

What now?
How do we handle internal discovery?
How can we have a process that gives our engineers control over the actions that we use?
How can we do all this in a secure way?
Can we automate this process? How do I stay up to date with the parent repository?

This post describes my way of working, and how I set up a GitHub Actions Marketplace.

The reasons for forking are plentiful, for example:

Take control over the Actions as a backup for your production organization (since they are downloaded just-in-time by the runner)
Have a formal moment in your organization that marks the end of a security check on the actions’ source code (very important!)
Have a central location for all the actions that can be used inside your production organization (combines nicely with the next item)
Block actions from the marketplace from being used in your production organization (see item above)

Want to know more? Check out a previous user group session on it here or my 2021 session on GitHub Universe 2021!

After setting the internal marketplace up (see below) that will host the ‘blessed’ actions, we also need to prevent any other actions from being used in our production organization. You have control over this in the organization settings:

Internal Actions Marketplace

The reason for having an internal actions marketplace is to have a central location for all the actions that can be used inside our production organization. This is to prevent any actions from the public marketplace from being used in our production organization, without being checked for security risks first.

Guidelines for the actions organization

DevOps Engineers own the actions when they fork them
Before forking actions there is a full security review
Requests for adding actions go through an internal repository by adding issues
Setup an internal marketplace to discover the internal actions
Have a communication plan on new actions, for example by having an inner source sharing platform (newsletter?)

Full marketplace process

The process for adding actions to the marketplace is as follows:

User finds an action on the marketplace
Request through an issue to include it
Manual security validation
Issue gets labeled security-check
Security validation on the issue (scan the source repo)
Decide on the risk of forking the action and using it
Sign off by security team [optional, can be handled by the next step]
Issue gets labeled security-approved
Fork it and own the maintenance
Share fixes back to the parent repo

Request through an issue to include it

I’ve setup a (start of a) example project github-actions-requests that is used to request actions to be added to the internal marketplace.

To do so:

a user needs to create an issue in a specifically setup repository, describing the action and the reason for requesting it. It helps if they already have an example repo for which they would like to use it.
a DevOps engineer with a security mindset / background checks out the actions repository and reviews the code (see more below)
after the initial and manual check, the engineer labels the issue with security-check
an automated security check is done on the actions repository, with communication in the issue (security scores for example)

Manual security validation

A DevOps engineer can check out the actions repository and review the code. This is done by:

looking at the setup of the action: is it JavaScript, Typescript or using a Docker container?
is the action only doing what it is meant to do?
does the action read files from disk outside of the work folder? (e.g. your ssh keys)
does the action read any environment variables?
what does it do with all the information it has access to? (e.g. does it sent is out to an external endpoint?)
does it use the GitHub Token for anything? If so, is it safe?
does it support GitHub Enterprise Server (GHES) if you need it? (e.g. does it use the github.com domain for anything, or does it use the GITHUB_API_URL environment variable?)

Security validation

For the security validation you can use your own internal setup. You need something that runs a Software Composition Analysis (SCA) scan and from that get security alerts for any dependencies that have them.

There are lots of tools available for this, for example Black Duck or White Source. GitHub already has Dependabot available that can be used for free on public repos. Since a fork is already a public repo, you can use it to scan the actions source code as well.

For our final setup I want to have it automated as much as we can, so I’ll be describing that process here. After the initial validation is completed and satisfies internal requirements, I want to label the repository with security-validation and run automated security validation on it. More on that below, but I am setting that up as well in the example repo here. The results from the checks will be added to the issue as badges from the different systems, with deep links into those systems to check the analysis. That can then be used for the final checks.

Dependabot

For Software Composition Analysis, we can use Dependabot to scan the actions source code for the packages that it uses. You can see the results of one of my actions in the repositories Dependency Graph:

Here you find the direct dependencies (in this case from the packages.json file of the repository) and all the ‘transient’ dependencies. A transient dependency is a dependency used by one of your direct dependencies. And since the transient dependency can have its own dependencies… it can be a long list. This is why some research shows that 70% of the code you deploy, was never created by you, but pulled in through a dependency. This is of course why a Software Composition Analysis (SCA) is so important to know about the dependencies that you have and match them to a known vulnerability database, also called a ‘Common Vulnerabilities and Exposures’ or ‘CVE’. Some examples of these databases are the National Vulnerability Database from NIST or the CVE database from Mitre.

GitHub has its own GitHub Advisory Database as well, with lots of vulnerabilities listed in it:

Security Advisories

After Dependabot has scanned the actions source code, it knows which dependencies are being used. Next, it will generate a list of security advisories for the packages that it found. It can even generate pull requests to the repository to update the vulnerable packages to a non-vulnerable version. Since we’re using it here on a public fork, we don’t use it ourselves. The action publisher should use it on their end to fix the issues found. You could of course use it on your fork and then send a Pull Request to the parent repo to fix the issues, and let the publisher know that they can use these features as well.

In this setup we can use the findings from the Dependabot scan to validate if we can use the action without large vulnerabilities in the packages used.

CodeQL

CodeQL is a tool that can be used to scan the actions source code for vulnerabilities and available for free on public repositories, which our forks are. You configure it as a workflow like I’ve done here. It will use action minutes for its execution.

Be aware that it by default scans the entire repository. In my case, I have a Typescript based action. That means that the Typescript is transpiled to JavaScript and then uploaded to the repository. So CodeQL will then scan everything. This meant that I found an issue in moment.js through the JavaScript in the repository. Only checking the Typescript code doesn’t find that for example of course, since it is not in that part of the code.

The CodeQL workflow will upload its scan results to GitHub, that can be found by going to ‘Security’ and then ‘Code scanning alerts’:

You can then review the vulnerability listed and check the recommendation.

Note: finding the source of the code vulnerability in the JavaScript code can be very hard. Transpile your Typescript with the inlineSourceMap setting to make it easier to find the actual Typescript / dependency source.

Container scanning

Since the Actions can also be run as a container, we need to check those dependencies as well. That means scanning the container from the image setting in the action.yaml. This setting can also refer to a Dockerfile in the root of the repository:

# action.yml
name: 'Hello World'
description: 'Greet someone and record the time'
inputs:
  who-to-greet:  # id of input
    description: 'Who to greet'
    required: true
    default: 'World'
outputs:
  time: # id of output
    description: 'The time we greeted you'
runs:
  using: 'docker'
  image: 'Dockerfile'
  args:
    - $

Scanning the containers can be done by using something like Trivy or Anchore.

Security approved

When you have done all the security checks, we can do a formal approval of the action and on board it to our own actions organization on GitHub. Labeling the issue as security-approved will trigger a workflow that:

Forks the repository to the organization-actions organization
Updates the issue with that information
Closes the issue since the request has been fulfilled

Fork it and own the maintenance

Now that we have forked the action, it’s up to us to maintain it, update it with the latest changes from the parent repo and fix any issues that might arise (and send those back to the parent repo!). Keeping everything up to date with incoming changes from the parent repo is something that I blogged about earlier here.

Note: We also need a process to handle new code scanning alerts on the repository, and a way to keep the CodeQL workflow running, since it is automatically stopped after 90 days of no new code changes in the repository. Then we also need to handle any new security alerts from CodeQL as well.

Internal marketplace

Now that we have all that setup, we need to have a good way to discover the actions that are available within our actions organization. We can use the default repo overview page, but that doesn’t feel very user friendly. We want a searchable list of actions, with more information then the default repository description. Since there is nothing available out of the box, I created something myself.

With the setup from actions-marketplace I’ve created an actions marketplace out of the box: you can fork it, configure it and use GitHub Pages to host your website. With it your internal users have a central place to search for internal actions. I also want to include links in it to the internal workflows that use the actions, so you can find examples easily. For the Marketplace maintainers, this will also give them a way to track the actions internal usage: if the action is no longer used in any workflow, you might want to remove it from the Marketplace and save you some maintenance work.

Setup the Marketplace

The marketplace repo has been setup with three main components:

Gathering the available actions repositories in an organization
Gather the used actions from an organization
Host the marketplace with GitHub Actions to display the info from the previous steps

Gathering the available actions repositories in an organization

The get-action-data.yml workflow loads all the repositories from an organization it has access to, checks the root directory for an action.yml or action.yaml file and parses it for information. The result will be a json file stored in the target repository with a specific branch named gh-pages.

Gather the used actions from an organization

The get-action-usages.yml workflow loads all the repositories from an organization (can be a different one then the other step) it has access to, checks the workflows directory for all files with a .yml extension and parses it for information. The result will be a json file stored in the target repository with a specific branch named gh-pages.

Summary

In this post I’ve given you a way to get started with an Internal Marketplace for GitHub Actions, to take back responsibility for the usage and maintenance of the actions. I also shown how to incorporate some security checks and some examples of setting all this up.

Got any feedback or more questions on how to set things up? Please let me know!

Note: the github-actions-requests example repository that I am setting up currently only does the first few steps. I’m still building the rest of the workflow 😄.

Current status:

Currently the github-actions-request:

is triggered by labeling the issue
searches for a uses statement in the last comment of the issue
if found, it forks the repository to the organization-actions organization (hardcoded in the workflow at the moment)
adds a CodeQL workflow file to the forked repository

Since the ‘organization-actions’ organization has been setup to enable the Dependency Graph and Dependabot alerts, I don’t need to do that in the workflow. I just need a way (after a certain amount of time) to check if there are any alerts and included that back into the issue information.

GitHub Actions: Convert from PowerShell to Typescript

2021-09-12T00:00:00+00:00

I am an avid PowerShell user and have been using it for a while now. Together with C# it is my main development experience these days. That is why I created my first GitHub Actions in PowerShell. Using PowerShell in you actions is possible by running the scripts in a container with PowerShell installed.

I wanted to create the same workflow in Typescript but am not that versed in the language. Together with finding the running in a container somewhat cumbersome (it takes time to pull the image and testing locally needs an extra script). I also found running in a container poses an extra step to use it in a GitHub Enterprise Environment (you probably need to mirror the image internally).

With an Xpirit Innovation Day coming up I had all the reasons I needed (and time available 😅) to create a Typescript version of the same workflow. I started the yak shaving earlier and made sure I had the basics under control so I would not have to find any of that out during the innovation day itself:

have a running example of a typical GitHub Action
with a way to test it in my IDE (VsCode)
with a way to read inputs and a way to inject them for testing in the IDE
have logging under control, so I can see what is happening while running
have a way to output the results so they can be used in the next action in a workflow
know how Typescript pass stuff like arrays back and forth between methods and how to loop over them

This post will be about my learnings moving from PowerShell to Typescript.

Getting started

To get started I recommend looking at the example repository actions/typescript-action. It has been setup as a template, which means you can click on the use this template button and you are good to go.

Learnings: Typescript

Converting to Typescript definitely took some time for me. The language is locked down regarding types and usages of things you declare:

If you declare a variable and don’t use it: you’ll get a compiler error.
If it cannot figure out what type an object is, you’ll get a compiler error.
If a variable could be null or undefined, you’ll get a compiler error using it. When you add a null check first, the compiler error goes away. All in all it reminds me much of C# 8’s nullable reference types: you are forced to make sure you are handing the nullable objects correctly which is nice, but can add a bit of a burden when you are new to the language. I also found out my head already declares things that I’ll need later on and then the IDE starts complaining that it isn’t used yet. It also seems to stop compiling further ahead of the error, which I don’t like.

The template also has ESLint configured, which I think is a good thing. It is a linter that checks for some common mistakes and helps you to fix them. It also has a code style checker that helps you to fix the code style. The style checker feels a bit enforced and gets in the way of typing: you cannot see if you have a compiler error at hand or that it is a linting violation 😢.

Typescript needs compilation to run

GitHub Actions in the end runs a NodeJS script. To enable Typescript to run, you need to compile the script to a JavaScript file for things to work. For this you can use @vercel/ncc to compile the script into the dist folder. Then you need to add the dist folder to the repository and push it to GitHub for things to work. Adding compiled code to the repository off course goes against my DevOps heart, but this is how it works for Actions: it could directly be checked out by adding a commit SHA or the branch name to your uses statement, as you should and then it is expected to still work.

Octokit

The Octokit library is very useful to make the GitHub API calls. It wraps the GitHub API by leveraging the official REST API Description for GitHub. For example connecting to the API with authentication is very straight forward:

In PowerShell I had to wrap all calls to use my own authentication setup and convert the PAT to a Base64 encoded header. It’s working, but Octokit saves me all that trouble 😄. Similarly handling API rate-limiting is handled for you, so you don’t have to worry about it.

Octokit and inputs

You can declare inputs in your action that the workflow can inject as parameters for your action. Getting the values of the parameters at runtime is build into Octokit, but injecting them during debugging is not. You can use the environment variables from the NodeJS process to still load them.

Octokit and pagination

I already have more than 30 repositories in my user account, which means I have to make calls to getRepos already implementing pagination to get all the results. In PowerShell this meant wrapping my calls, check for the pagination headers and handling them properly. Then I needed to stitch all the results into one list and return that to the caller.

In Octokit this is done by added octokit.paginate() around your call and you are done!

Octokit and debugging/logging

During debugging locally, I would like to see the messages being logged for the user during the execution, to get an idea of the context we’re working in. The library has concepts for logging with different levels:

core.info: for general information
core.warning: for warnings
core.setFailed: for logging an error and stopping execution The core calls work nicely during the execution of a workflow, but they don’t show up in the debug console! That is not helpful. The only thing that you can do to still get output in your debug console is to fall back to console.log.

Summary

Overall the experience was smooth: the main issues I had was with using Typescript instead of PowerShell. Luckily adopting a new programming language is made easier if you have a good grasp of the core concepts, so you can search for how to do a for loop in a new language, or using Promises, immutable data structures, etc.

The actions/typescript-action template is very complete and can be used out of the box. I also noticed the execution of the action is much faster when using Typescript. Of course, the container doesn’t have to be pulled, but also all my handling in PowerShell with authentication, pagination and checking if we’re not hitting the rate limit is much faster.

Iterating through 36 repos (at the time), checking all repos for an action.yml or action.yaml file, parsing the yaml and storing the information in an array when from 35 seconds (PowerShell) to 6 seconds (Typescript).

Using the Typescript template also had unit tests installed, which I could do in PowerShell as well, but never did. Let’s see if I’ll use them now 😄. Still, the ultimate test is including a workflow in your repository that will run your local action and check if it works.

GitHub Actions: Run PowerShell in Container

2021-09-12T00:00:00+00:00

You can create GitHub Actions running in a container, which allows you to execute ‘anything’ in an action that can be run inside a container, including PowerShell, my favorite scripting language. To get started, you can use the available template repo to create a new repository filled with the contents of the template.

The moving parts of the template are as follows:

action.yml
Dockerfile
entrypoint.sh

action.yml

The action.yml contains the setup for your action repository when it is used in a GitHub Actions workflow as well as the listing of it on the GitHub Actions Marketplace. In it are the description of the action, the author, the version, and the list of inputs and outputs.

It also tells the workflow runner (the thing that executes the action), how to run the action. In this case we use the using property to tell the engine to use a Docker image to run in. The image property tells the engine which Docker image to use. If you use Dockerfile as its value, it will use the Docker file in the root of the current repository. In the args section you tell it the arguments to pass into the container when it starts. These are then added in the order the are added to this list. So the array has index 0 for the first argument, 1 for the second, and so on. That will also be the order in which the arguments are passed to the entrypoint file in the container.

Dockerfile

The ‘Dockerfile’ will be build on the spot when the action is executed. With the ENTRYPOINT you tell it what script to run when the container starts. This is the point where you can put your PowerShell script. Remember that the parameters are injected into the script as variables.

FROM ghcr.io/rajbos/actions-marketplace/powershell:7

COPY /Src/PowerShell/*.ps1 /src/

ENTRYPOINT ["pwsh", "/src/entrypoint.ps1"]

PowerShell

The PowerShell script is in the entrypoint.ps1 file. I declare the parameters that PowerShell will then map in the order the script was started with. Remember that the order is determined in the action.yml file. Which is nice, since that means you are in control of the order and the user cannot make mistakes with it. If the arguments are optional, they will still be injected but with the default or empty value for them.

You can leverage that feature like I do below. I test for the values to see if they are empty and if so, I use a default environment variable to run in the current GitHub context (a user account or an organization). If not all minimal values have been set, I still fail the action.

param (
    [string] $organization,
    [string] $PAT
)

if ($null -eq $organization -or "" -eq $organization) {
    Write-Host "Using default for organization: [$($env:GITHUB_REPOSITORY_OWNER)]"
    $organization = $($env:GITHUB_REPOSITORY_OWNER)
}

if ($null -eq $PAT -or "" -eq $PAT) {
    Write-Error "No value given for input PAT: Use at least [GITHUB_TOKEN]"
    throw
}

$actions = (.\load-used-actions.ps1 -orgName $organization -PAT $PAT)

Note: failing the action can be done by exiting the entrypoint script with a non-zero exit code. I use this setup for it:

try {
    # always run in the correct location, where our scripts are located:
    Set-Location $PSScriptRoot

    # call main function:
    main

    # return the container with the exit code = Ok:
    exit 0
}
catch {
    # return the container with an erroneous exit code:
    Write-Error "Error loading the actions:"
    Write-Error $_
    # return the container with an erroneous exit code:
    exit 1
}

PowerShell container image

For full reference you can find the Dockerfile I used below:

FROM ubuntu:20.04

# install powershell for Ubuntu 20.04
RUN apt-get update \
    && apt-get install -y wget apt-transport-https software-properties-common \
    && wget -q https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb \
    && apt-get update \
    # Register the Microsoft repository GPG keys
    && dpkg -i packages-microsoft-prod.deb \
    # Update the list of products
    && add-apt-repository universe \
    && apt-get update \
    && apt-get install libssl-dev -y \
    && apt-get install gss-ntlmssp -y \
    && apt-get install powershell -y \
    && apt-get install curl -y \
    && apt-get install git -y

# install the module we need
RUN ["pwsh", "-Command", "install-module -name powershell-yaml -Force -Repository PSGallery"]
SHELL ["pwsh"]

Hosting GitHub runners on Kubernetes

2021-08-06T00:00:00+00:00

If you need to host your own GitHub Actions runners that will execute your workflow, you have multiple options: you can go the traditional route and host them on your own VM’s. You can even host multiple runners on the same VM, if you want to increase the density and reuse your CPU or RAM capacity on your VM. The first downside of this is that VM’s are harder to scale and often a bit more costly to run then other options. Even more important reason not to use VM’s because it is not a great option for having an environment that is clean for every run: GitHub Actions will leave several files on disk for both reuse (the actions downloaded and used for example, or the docker images you run on). Even the checkout action will only cleanup the source code when it is executed, to make sure that the latest changes are checked out. You can include a cleanup action at the end, but often that is not added.

Even worse are the potential security pitfalls that come from reusing an environment between runs of a workflow, or different workflows in different repositories: the first run could leave some files behind, like from a package manager you use, or overwrite a local docker image for example. Subsequent runs on that machine will look in the local cache first, and use the (potentially) compromised files. These are some of the examples of supply chain attacks that are more and more common these days.

To combat those risks, you want to have ephemeral runners: the environment only exists during the execution of the workflow: after it is done, everything is cleaned up. The does mean that caching things becomes a little harder. There are ways to combat that with for example a proxy close by your environment that can do the caching for your (note: this is still a potential risk!).

Photo by ian dooley on Unsplash

Ephemeral runners

GitHub does not have any support for hosting your runner inside a container with some ‘bring your own compute’ options. It uses that setup on the GitHub hosted runners, where a runner environment is created just for your run and destroyed afterwards, but hasn’t released anything for their customers. When you start looking for options, you will find a community curated lists of awesome runners that have been made by the community to host your runners inside k8s, AWS EC2, AWS Lambda’s, Docker, GKE, OpenShift or Azure VM’s (at the time of writing 😄).

Actions runner controller

The one that got recommended to me was the actions-runner-controller: it had an active community (82 contributors, including me now! lots of stars) with a lot of communication on the issues and discussions list.

Hosting in Azure Kubernetes Service

For testing to see if I could get things to working with my bare minimum of k8s knowledge, I created an Azure Kubernetes Service cluster with all the defaults and installed the actions runner controller in it, with all the information in the readme of the project. Even used a GitHub app for authentication and things worked straight out of the box. You can just run one runner for quick testing, or use the build in scaling options to scale up and down between your limits (for example 0 runners when there is nothing to run and 100 runners as a maximum), or scale up and down based on time windows you define (so scale up to 30 runners at 7AM on workdays, if your company still folows tradditional time slots people are working on).

A remark on scaling

During testing I found that the scaling options for actions-runner-controller have a downside: it can only look at the current queue of workflows and not the set of jobs inside those workflows. That is because GitHub currently does not support loading the queue based on jobs inside workflows. There is development being done on that side, but I have not seen any progress yet.

Hosting on internal Rancher server

My customer that wanted this setup for their own GitHub Enterprise Server (GHE) to have a local runner as well. The security team also wanted to do a security check on the setup and didn’t want to use AKS for that (and notify Microsoft of active pen testing activities on the cluster). They had an internal Rancher setup available that they wanted me to use. The thing is, that this cluster was already tight down a lot: it only could pull internal Docker images, and it had a lot of other restrictions, like using a proxy for all its traffic. This is where things got a little bit more complicated. Their internal images host was a private registry hosted on Artifactory and pulling from public container registries was not available.

Configuring images used by the controller manager

The first thing I ran into was that our Rancher setup sat behind an internal proxy / load balancer that forced all images to be downloaded from an internal Artifactory registry. Depending on the original registry a lookup was done in the allow listed images in Artifactory. These where the images we added to the allow list in Artifactory:

summerwind/actions-runner-controller:v0.18.2
summerwind/actions-runner:v0.18.2
quay.io/brancz/kube-rbac-proxy:v0.8.0
docker:dind

The only image that could not be pulled transparently with our setup was the one from quay.io: this registry was not mirrored transparently, which meant that the label was different in Artifactory. As an initial fix I choose to override the image name manually after it has been deployed, using this command:

kubectl set image deployment/controller-manager kube-rbac-proxy=registry.artifactory.mydomain.com/brancz/kube-rbac-proxy:v0.8.0 -n actions-runner-system

This means that the controller-manager deployment gets a new image assigned that has the name kube-rbac-proxy and it will reload that container. After that, things actually started running and I could the runner to be available on either the organization or repository level.

Docker in docker (DinD) with internal certs

Our Rancher setup used an internal proxy to pull our images from an Artifactory server that was signed with an internal certificate (without a full trust chain). This meant that the Docker client used to pull the images had to be configured to trust the internal certificate as well or you only got pull errors with an untrusted cert. To accomplish this I build a new DinD container on our runners that where running on a VM and had the certificates installed locally.

Action.yml that build the image:

    - uses: actions/checkout@v2
    - name: Build
      run: |
        cd dind
        # certs on RHEL are found here:
        cp -R /etc/pki/ca-trust/source/anchors/ certificates/
        docker build -t ${DIND_NAME}:${TAG} -f Dockerfile --build-arg http_proxy="$http_proxy" --build-arg https_proxy="$http_proxy" --build-arg no_proxy="$no_proxy" .

So that we could load the local certificates folder into the image (note that you can also used the commented RUN command to hardcode a specific certificate):

FROM docker:dind

# Add the certs from the VM we are running on to this container for secured communication with Artifactory
COPY /certificates /etc/ssl/certs/

# add the crt to the local certs in the image and call system update on it:
#RUN cat /usr/local/share/ca-certificates/docker_registry.crt >> /etc/ssl/certs/ca-certificates.crt
RUN update-ca-certificates

Note: tested with `$DOCKER_TLS_CERTDIR` as well: didn’t work

# Add the certs to this image for secured communication with Artifactory
COPY docker_registry.crt $DOCKER_TLS_CERTDIR # Docker should load the certs from here, didn't work

Note: tested with daemon.json as well: didn’t work

I also tested by adding a daemon.json

{
    "insecure-registries" : []
}

And then copied that json file over:

COPY daemon.json /etc/docker/daemon.json # see https://docs.docker.com/registry/insecure/,  didn't work

Somehow this file seemed to be ignored and pulling the images from the internal registry still failed.

Loading the new DinD image

Loading the new DinD image could not be done by the same ‘hack’ as used for the image from quay.io. After reaching out to the community they helped me with actually overwriting the controller-manager deployment with both the image from quay.io and the new DinD image:

# controller-manager.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    control-plane: controller-manager
  name: controller-manager
  namespace: actions-runner-system
spec:
  replicas: 1
  selector:
    matchLabels:
      control-plane: controller-manager
  template:
    metadata:
      labels:
        control-plane: controller-manager
    spec:
      containers:
      - args:
        - --metrics-addr=127.0.0.1:8080
        - --enable-leader-election
        - --sync-period=10m
        - --docker-image=registry.artifactory.mydomain.com/actions-runner-dind:latest
        command:
        - /manager
        env:
        - name: GITHUB_TOKEN
          valueFrom:
            secretKeyRef:
              key: github_token
              name: controller-manager
              optional: true
        - name: GITHUB_APP_ID
          valueFrom:
            secretKeyRef:
              key: github_app_id
              name: controller-manager
              optional: true
        - name: GITHUB_APP_INSTALLATION_ID
          valueFrom:
            secretKeyRef:
              key: github_app_installation_id
              name: controller-manager
              optional: true
        - name: GITHUB_APP_PRIVATE_KEY
          value: /etc/actions-runner-controller/github_app_private_key
        image: summerwind/actions-runner-controller:v0.18.2
        name: manager
        ports:
        - containerPort: 9443
          name: webhook-server
          protocol: TCP
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 20Mi
        volumeMounts:
        - mountPath: /tmp/k8s-webhook-server/serving-certs
          name: cert
          readOnly: true
        - mountPath: /etc/actions-runner-controller
          name: controller-manager
          readOnly: true
      - args:
        - --secure-listen-address=0.0.0.0:8443
        - --upstream=http://127.0.0.1:8080/
        - --logtostderr=true
        - --v=10
        image: registry.artifactory.mydomain.com/brancz/kube-rbac-proxy:v0.8.0
        name: kube-rbac-proxy
        ports:
        - containerPort: 8443
          name: https
      terminationGracePeriodSeconds: 10
      volumes:
      - name: cert
        secret:
          defaultMode: 420
          secretName: webhook-server-cert
      - name: controller-manager
        secret:
          secretName: controller-manager

After that, we could deploy our own Single-runner.yaml, which has an option to specify the image to use for the runner:

# runner.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: Runner
metadata:
  name: gh-runner
spec:
  repository: robbos/testing-grounds
  image: registry.artifactory.mydomain.com/actions-runner # overwriting the runner to use our own runner image, the DinD runner comes from the controller

Runner fix

I actually had to patch the runner image as well, because our setup had a tmp folder on a different device, which was causing errors during the bootup of the container. I copied the container definition over from the actions-runner-controller repo and fixed the script, published our own image and told the runner deployment to use it by setting the spec.image as in the example above.

Other observations

Namespace

Something that took me a while to figure out: the namespace actions-runner-system is hardcoded in all deployment files, so you cannot change it (easily). Keep that in mind if you want to land in a pre existing namespace with internal pod security policies for example.

Community

The community creating these runner setups is active on both creating the solutions and helping people out. Given that the used setup is actively maintained and they supported me with my questions is a great sign of a good community. Without the community, I would not have been able to get this done, so thanks a lot!

Photo by Tim Mossholder on Unsplash

Azure DevOps: enable project functionality

2021-05-31T00:00:00+00:00

Sometimes you spot interesting things online that you have to figure out 😎. This time it was a tweet from Martin Ehrnst:

@AzureDevOps i'm using your API to create new projects. However, I would like to provision these without services like boards. I cannot find any way to to do this. Doesn't the API support this?
— Martin Ehrnst ☁️ (@ehrnst) May 21, 2021

In Azure DevOps you can enable or disable features on a per-project basis:

After some reverse engineering I found out that you can request and set the state of these features by calling into the API.

Getting the feature state

POST: https://dev.azure.com/{ORGANIZATION}/_apis/FeatureManagement/FeatureStatesQuery/host/project/{PROJECTID}?api-version=4.1-preview.1 Indicating in the body what you want to know. If you only pass in 1 featureId, you only get the state for that single feature.

{
    "featureIds": [
        "ms.vss-work.agile",
        "ms.vss-code.version-control",
        "ms.vss-build.pipelines",
        "ms.vss-test-web.test",
        "ms.feed.feed"
    ],
    "featureStates": {},
    "scopeValues": {
        "project": "{PROJECTID}"
    }
}

Setting the feature state

You can set the state of a feature by sending in a PATCH request for a single feature. Posting for multiple features in one go doesn’t seem to work. This makes sense since the API also does it on a per feature basis. PATCH: https://dev.azure.com/{ORGANIZATON}/_apis/FeatureManagement/FeatureStates/host/project/{PROJECTID}/{featureId}?api-version=4.1-preview.1 With body:

{
    "featureId": "ms.feed.feed",
    "scope": {
        "userScoped": false,
        "settingScope": "project"
    },
    "state": 1
}

Where featureId from the body needs to match the featureId in the URL (find the list in the example above).

Observations

To bad you cannot pass in these settings through the API when you create a project. At least there is another way to get things working 😄.

Solidify: Using GitHub Actions Securely

2021-05-28T00:00:00+00:00

Today I got to deliver my session “Using GitHub Actions Securely” at the Solidify show, hosted by our friends at Solidify. A nice virtual community session during lunch in my time zone (CEST) with people joining in, even from Kuala Lumpur!

I got a couple of questions during the session that I wanted to dive deeper into and address them here, as well as sharing the slides and the recording of it.

The session has been recorded and you can re-watch it on YouTube or here:

Slide deck

I you want to look up some things from the slides or visit one of the many links in there, you can look at the slide deck here.

Question: Can you prevent a reviewer to approve their own changes on a Pull Request?

Full question: “I would like to enforce that at least two people are involved with every change made to a protected branch. GitHub by default allows a reviewer to approve their own changes to a PR, which makes it possible to use any open PR to merge changes possibly for making an attack on the production environment. Do you have any guidance how to avoid such a scenario?”

You can use branch policies to set up rules that need to be met before a Pull Request be merged:

With the default setup people with higher access levels can still merge the pull request: I see two options to still enforce that someone else needs to review the PR:

Add a minimum of 2 reviewers: then the PR author always needs to review the PR their self as well as an extra reviewer.
Add a check that verifies that there was an extra reviewer, next to the PR author. You can create that check with a workflow that posts back the status after a reviewer has approved the PR.

Yes! As much as you can: that gives you a nice extra security boundary for the stuff you actions are doing: instead of running those processes on the runner process itself, it will run them in a container. This is also a good way to prevent stuff from being installed on the runner machine: for example, you can use the OpenJDK image to build your JAVA project, without installing JAVA on the host machine. You get an extra security layer on top of it as well. Breaking out of a container can probably still be done, but is a bit harder to do. This also helps with anything else remaining on the machine, like files on disk, stuff in the environment variables. You can read more in the ephemeral runners section below.

Question: Is it possible to run the GH Actions in Self Managed Kubernetes cluster in Cloud?

Yes there is! There are multiple community projects available to run the runners in a scalable, self hosted way. The default is installing the agent your self on a VM of your choosing. If you want to add autoscaling into the mix, and get ephemeral runners as an extra benefit, you can look into this list of curated community solutions for hosting them on your own infrastructure, for example Kubernetes. I’ve used one option myself that I’ll add the experiences below.

Lets first explain ephemeral runners:

An ephemeral runner is only spun up for a specific workflow run, and only exists for the duration of the run. Afterwards it is deleted and cannot be used anymore. This is of course an ideal setup to use with Kubernetes: when you need capacity (a workflow is scheduled): spin up a container with the workflow runner inside of it, register it with GitHub, and execute the workflow. Doing this with a Virtual Machine is a lot more time consuming and difficult to do, even with the cloud. If you really have the need, you could use it that way.

Using an ephemeral runner also means that there is no option for state (files stored on disk, environment variables) to linger on the environment that could potentially be picked up by another workflow that executes on the same runner.

Actions runner controller

The solution I’ve been testing things out is the Actions runner controller. This one is spun up as a controller on Kubernetes (I used AKS for it) that periodically checks with the GitHub API if there are jobs waiting in the queue. If so, it will spin up a new container with the runner inside of it, register it with GitHub and let it do the work for you. After the job is done, the container will be removed and deleted. The container will execute the workflow itself, or leverage Docker in docker to create a container if your actions need them.

This works really nicely: I’ve hammered it with 6 workflows in the same repository that spun up over 20 jobs at the same time: scaling kicked in and created multiple runners for me. After the completion there is a cool off period, where the runners are being removed completely after 10 minutes (the default setting).

Pester tests: moving from v4 to v5

2021-05-25T00:00:00+00:00

This one took me way to many trials and searches to figure out, so I wanted to store it here in case I need it later on. Maybe someone else will find this useful as well 😄.

The premise

We have a pipeline for GDBC from June 2019 that uses Pester tests written in PowerShell to verify the outcome of our pipeline: we create (a lot of) resources in Azure and Azure DevOps and want to check if they actually exists.

We run the tests inside a PowerShell task in Azure DevOps and install the Pester module with this:

Install-Module -Name Pester -Force -SkipPublisherCheck
Import-Module Pester

This of course installs the latest version. After moving things to a different environment, I skipped the tests for a while (booh!) and yesterday decided to add them back. And lo and behold: things where not working anymore.

The tricky part was getting things to work with a Pester file that holds parameters that we need to pass into it.

Pester file has parameters

param (
    [string] $region,
    [string] $pathToJson,
    [string] $runDirectory
)

Parameters we need:

To set things up, we need to set the parameters with some values to use:

$region="LocalDevOpsBootcamp"
$dataFilePath="c:\temp"
$datafilename="data-999.json"

Pester 4.0

With the previous version of Pester we called Pester and added a Data object to pass in the variable values.

Invoke-Pester
 -Script 'GDBC-AzureDevopsProvisioning.Tests.ps1'
 -Data = @{
     region = '$region';
     pathToJson = '$dataFilePath/$datafilename';
     runDirectory = 'AzureDevOps-provisioning' }
 -OutputFile Test-Pester.XML
 -OutputFormat NUnitXML

Error running the v4 setup against v5

This gave the following warning / error with version 5:

WARNING: You are using Legacy parameter set that adapts Pester 5 syntax to Pester 4 syntax. This parameter set is deprecated, and does not work 100%. The -Strict and -PesterOption parameters are ignored, and providing advanced configuration to -Path (-Script), and -CodeCoverage via a hash table does not work. Please refer to https://github.com/pester/Pester/releases/tag/5.0.1#legacy-parameter-set for more information.

System.Management.Automation.RuntimeException: No test files were found and no scriptblocks were provided.

 at Invoke-Pester<End>, C:\Program Files\WindowsPowerShell\Modules\Pester\5.2.1\Pester.psm1: line 5082
 at <ScriptBlock>, D:\a\_temp\272537fd-8fdd-42fc-b176-803d9ca859d6.ps1: line 7
 at <ScriptBlock>, <No file>: line 1

Pester 5.*

This is the part that took me way to long to figure out. You can run Pester with a container by calling Invoke-Pester -Container $container and add the parameters to pass along to the test. That is step 1.

If you also want to add settings, you need to wrap the container in a configuration!

So the steps are:

Create a container
Add the parameters and the testfile(s) to the container
Create a configuration
Add your settings on the configuration
Set the container as the run in the configuration
Run Pester with the configuration

Example

# create a new container that will be executed
$container = New-PesterContainer
 -Path './gdbc2019-provisioning/AzureDevOps-provisioning/GDBC-AzureDevopsProvisioning.Tests.ps1'
 # include the parameters to pass into the Pester file
 -Data @{
     region = '$region';
     pathToJson =
     "$dataFilePath/$datafilename";
     runDirectory = './gdbc2019-provisioning/AzureDevOps-provisioning'
     }

# create a new configuration with our settings
$config = New-PesterConfiguration
$config.TestResult.OutputFormat = "NUnitXML"
$config.TestResult.OutputPath = "Test-Pester.XML"
$config.TestResult.Enabled = $True

# configure the run with the new container from step 1
$config.Run.Container = $container

# actually call Pester
Invoke-Pester -Configuration $config

More information can be found here:

Techorama: Using GitHub Actions Securely

2021-05-18T00:00:00+00:00

Today I got to deliver my session “Using GitHub Actions Securely” at Techorama, my favorite conference. I could feel the pressure at the start of the session: this is Techorama, so you need to deliver this one top notch!

I think I had some viewers (wasn’t visible to me) and I got a couple of questions during the session that I wanted to dive deeper into and address them here.

I think the sessions have been recorded, but will only be shared with the attendees of the conference. So if you weren’t there…

Slide deck

I you want to look up some things from the slides or visit one of the many links in there, you can look at the slide deck here: here.

Question: Running SAST tools on pull requests from forks

Full question: Secrets are not shared with forks and Action runs from forks cannot use secrets from your repo. How do you run SAST (SonarCloud) on pull requests from forks? Answer: This was around the part where I was talking about the dangers of running a workflow on a Fork from a Pull Request on a public repository. If you want to know more about this topic, follow that link to https://xpir.it/gh-pwn-request.

You can still run a SAST tool on a pull request. Just don’t blindly run it on a pull request from a forked repo: you don’t now what changes someone made before creating the PR.

What you can do, is run a basic workflow with the pull_request trigger that does the security scans on any dependency you want to verify. If that one succeeds (add enough checks to have a feeling of trusting the changes), add a label to the Pull Request. That label can then trigger a secondary workflow that does the execution of the SAST tool. This secondary workflow can then run with the pull_request_target trigger AND the existence of the label. Since adding a label can only be done by a maintainer of the repository, so that is a good safety measure to take. Additionally you can use code owners in the repository to trigger an additional notification to for example your security engineer in the team to have an extra look. For example when the PR changes something in your .github folder, like a workflow file.

Question: Is a reputation check (comment/starts) a good enough validation for GitHub Actions

Answer: you can try to use the reputation check together with the name of the publisher of an action to gauge some things around a GitHub Action. For example, you can see the community involvement with that Action. But really checking if the Action is being maintained can only be done by looking into issues and following the discussions there. It’s probably good to check if the Action has been created by the vendor of the tool it’s targeting. If the vendor themselves is maintaining it, they have their name attached to it and hopefully they see that as a responsibility to keep their action up to date and as a way to engage the community to enable people to use their product more. So the cloud Actions from Google, AWS and Microsoft Azure are probably good enough to use.

The problem here is that still, you are pulling in code from the internet and you should check what that code is actually doing. Will you really trust the publisher on their blue eyes that they will not screw things up? And after all, we are still humans, so a mishap is a regular thing 😄.

So in my opinion: trust is good, but verification is better.

Don't use self signed certificates on GitHub Enterprise

2021-05-16T00:00:00+00:00

Often you come across an organization that has a policy to use self signed certificates on internal services: as long as you control the workstations used to connect to them, that is a solution that might work. Sometime you still run into issues from them and they usually have a workaround available. Maybe IT-services likes to be in control who can create and hand out certificates that way. In a real DevOps environment I’d want the team to have control over them, and even automate their deployment and refresh them automatically in a regular interval.

GitHub Enterprise Server

For GitHub Enterprise server, this has so many downsides that I really recommend not using self signed certificates on that: get a proper cert with a full certification chain on it. Let’s go over the things that you will run into when you install a self signed certificate:

Users cannot connect to GitHub Enterprise

Browsers will give you errors if the certificate on the site is not trusted on the machine. This happens with a self signed cert that does not have a full certification chain. Fixing this one is easy: if IT already has self signed certificates setup throughout the internal services, they will also have a way to add a new certificate to the internal trust stores. Those are distributed to the workstations and servers and therefor all clients connecting will have no issues doing so.

GitHub Actions on Virtual Machines

When you start using actions in GitHub Enterprise (available in version 3.0), you’ll probably start with self hosted runners as well: there was a reason for hosting your own GitHub Enterprise server, so internal runners will come next. For starters lets assume you are running them on a virtual machine (VM).

When you create the VM, you now also need to trust the self signed certificate into the VM. If you don’t do that, then you cannot use the checkout action on the VM: when it does a Git Checkout it will use the the remote url to get the code from the repository: that will use the self signed certificate that it doesn’t trust. If there is no Git installed on the VM, it will execute an HTTPS fetch from the repository, that will need the self signed certificate that it doesn’t trust.

Alright, this also has a straightforward fix: trust the self signed certificate on the runners VM (please automate the creation of the VM’s and the installation of the runner 😉).

You can load the local Linux cert directory if the action is running on a VM by adding this setting to the action:

SSL_CERT_DIR: /etc/ssl/certs

Or use the node settings to load a specific cert if the action is build on node:

- name: Checkout
    uses: actions/checkout@v2
    env:
      NODE_EXTRA_CA_CERTS: /etc/ssl/certs/ca-bundle.crt

The downside of this is that you need to add these workaround into each and every workflow that users create (and tell them they need to).

GitHub Actions running Docker containers

A good security practice to protect your runners, is to use Docker containers to run Actions you don’t (want to) trust: run them inside a container as an extra security boundary to limit the access the runner user has on your VM. If you have setup the runner with least-privileges then the container boundary is hard to break out of.

If you start running Actions or jobs in a container, you are now bound to the certificate trust chain of the container and you are no longer running in the context of the runner that reads it’s certificate trust chain from the host. That means you will find it hard to use the self signed certificate. From something like a call to an internal NPM endpoint (with a self signed certificate as well), you could go all-in and just ignore the full SSL certificate completely by adding NODE_TLS_REJECT_UNAUTHORIZED = 0 to your environment. This is highly insecure and will open up your environment to a person-in-the-middle attack. I’m sure that is not what you want?

You could also add the file onto the container (volume mount it or download it as an artefact) and add it to NPM so it can find it:

npm config set cafile "<path to your certificate file>"

What you now have accomplished is that each and every developer who wants to use your internal service with a self signed certificate needs to add extra steps to the workflows to get them to work, adding a lot of extra friction. If you had added a certificate with a full trust chain, things would work out of the box, without any workarounds.

Start the runner with skip validation setting

You can also start the workflow runner with --sslskipcertvalidation which is another bad practice opening you up to a person-in-the-middle attack. Please don’t use it!

Mount the certificate store:

Another option is to mount the cert store in the job that is running the container:

jobs:
  build:
    runs-on: self-hosted
    container:
      image: node:10.16-jessie
      volumes:
        - /etc/ssl/
    name: Checkout will happen inside the container
    steps:
    - name: Checkout current repo
      uses: actions/checkout@v2

This will still mean your DevOps engineers will need to remember to think about the volume mount when using Actions. Even worse: depending on the base-image they are using, the volume to mount even might be different!

Make your own base images

A better (and safer option all around in my opinion) would be to create your own base images that the DevOps engineers can use: lock down and harden those images any way you want, and then install your own certificate in the image as well!

Conclusion:

You can find workarounds for the GitHub runner / action or runner to be able to get things working. A lot of workarounds have impacts that for example mean you need to include them in each and every workflow you want to run. The easier way then is to get a proper certificate setup on your GitHub Enterprise server: now things will work as intended and you don’t have to tell your users to add the workaround to their workflows.

My working / presenting from home setup

2021-05-13T00:00:00+00:00

I got some questions recently on my office setup at home, so it might be helpful to share it here as well. I am lucky to have the means to invest in my setup, even with a supporting employer who helps out with some of the financials since we all need to work from home these days instead of being at the customer’s office. Some time ago I figured we’d be working this way for a while (due to the COVID pandemic) and it seemed like a good idea to invest in the place I work the most in: my home office. Even after the pandemic is over (hopefully) and people have more freedom to go into their work offices, the plan is to still work 2 or 3 days from home: I really don’t have the wish to go back to the major traffic congestions we have in The Netherlands. That means that these investments will be used for a while and that helps with some of the decisions that where made. Even my partner agreed on that, hence why some of this stuff is over the top awesome 😁.

The image on the left displays my working from home setup somewhere in April 2020: we just moved to a new house and I was not planning to have an office space at all. Then we went into the first lockdown and I was forced to set something up. So, a couple of moving boxes, a piece of board and a tablecloth was what I made my makeshift desk with. Our offices where so kind to lend a good chair and an extra screen if you needed it.

The gear

Let’s talk about the gear you see in the image 😄.

Monitor

The monitor is a Dell U3419W. It’s a 34 inch ultrawide curved screen with Quad HD (1440p). This is one of the more expensive items: I was searching for a better solution then my laptop on a stand and an extra screen. When I was a developer I remembered being very fond of having 3 21 inch monitors side to side, and having the different resolutions of my laptop and an extra screen from the office weren’t good enough. I researched monitors and found an option of around €500, since that was the amount of money I felt comfortable spending. When talking things over with my wife of the extra stuff I wanted, she told me to get something good since that is what you are looking at for over eight hours a day. I still felt a little uncomfortable spending this much on a monitor. After setting it up I am very happy with it! Really glad I could spend the extra money on it and have an awesome setup. You don’t notice the curve and the resolution is excellent (3440 x 1440).

Sidenote: couldn’t find a link on the Dell site itself so I used this Dutch web shop that I like.

I am using this monitor with a Picture-By-Picture (PBP) setup: that means that I can plugin two cables to my laptop and the laptop thinks it is working with two different screens. I really wanted this setup that way since I work remotely in such a way that I often share my screen. Screensharing an ultrawide with a resolution of 3440 x 1440 can be done, but the other side of the video chat will not be able to read anything you are displaying, unless they have a similar resolution available on their end. You can of course just share just a small window, but as a hands-on DevOps consultant I often share my entire screen so I can use different windows on it. Having this PBP option enables me to have two screens available with each a resolution of 1720 x 1440. It is a rather strange resolution because of the height. Luckily this works well enough to not be a real hassle in my day to day usage.

Lights

The lights are Elgato KeyLight Airs. They have options for their brightness and you can set the color from ultrabright white to really yellowish:

I need them so people can actually see me during a video meeting, usually the brightness is set really low (15%), just good enough to balance out the light coming from the sky light in the roof. The color temperature has been set to 5150 and I usually don’t change that.

The photo’s below are taken on a cloudy day. When the sun is out (and especially in the mornings are flooding in through the window) I really need an extra light. So from left to right: without any lights (somewhat okay today), with only one light on, and then with both lights on.

I have them both setup a bit higher on my screen to prevent to much eye strain from having a bright light in my eyes all day. That is also the reason why I have the brightness on 15%: I find that is good enough on most days and still doesn’t bother me to much. Keep in mind that I sometimes have them on for hours in a row, with some breaks in between.

Camera’s

During the start and summer of 2020 good webcams where very hard to find, since there was a lot going on with production (of both chips and camera’s) being stalled due to the pandemic. When I finally was able to grab on, I picked the Logitech C922 Pro both for it’s price and the fact that it did 1080p at 60fps. Additional benefit was that it has a great microphone. Since the only thing I had available at that time was the one on my headsets (Sony WH-1000XM2) or the standard iPhone headset) or 🤢 the build in one on my laptop (Dell XPS9500), this was a very welcome addition at the time.

Later on I got a Logitech Brio. This one is a full 4K webcam, has HDR, can be zoomed (with the right tools) and even has Windows Hello support!

Do note that the Brio has a worse microphone then the C922: the later has an awesome stereo microphone that really performs better then the simple one on the Brio. Since I am not using either at the moment, it doesn’t matter for me right now.

The cool thing I have now, is that I can use two different viewpoints by leveraging the dual camera’s that I have: the Brio sits on top of my monitor for normal video conferencing, the C922 is to my left and shows off the microphone better, together with my cool GitHub Popfilter 😍 (was a gift). I use the OBS Virtual Camera setup to switch between camera’s.

Headset

Music is very important for me. Often I’ve got something playing during the day, in meetings and even presentations. I really can get a lot of energy from music.

I actually have multiple headsets to use. A couple of years ago I treated myself to a Sony WH1000XM2: over the ear headset. It gives excellent audio and has one of the best (at that time, get the fourth generation now) noise-cancelation. The battery duration is great as well. I think I’m easily using that for more then a week these days before I need to charge it (which does take four hours btw). Only thing that this second generation misses is the dual connection option: the fourth generation has it, where you can connect to multiple devices at once, like for example a phone and a laptop at the same time. Currently I sometimes miss phone calls, since the noise cancellation is that good 😁.

Also note that after a couple of hours, my ears get really warm and I don’t like that. So I usually switch to a different output option.

As a backup I have a Jabra Elite 25e. This one is in-ear, has good noise-cancelation but still lets you hear some environment noise. The only thing I don’t like on these is that the plugs that are included are to large and pressure my ears to much.It comes with 3 different sizes and even the smallest one still hurts my (large) ears after an hour or so. Still, it is wireless, blue tooth and has good audio.

As the ultimate backup, I have the original iPhone headset with an earphone jack. That’s not wireless of course, but still has good audio and I can wear it for a couple of hours.

So in the event something isn’t working or a battery is empty, I can still work.

For the last few weeks I have been using the build-in speakers of my monitor more and more, if the kids are not at home and I don’t need the noise-cancellation I like that my ears are free. (it also doesn’t help that my new laptop is on Windows Insider (Dev Channel) and currently has issues connection any blue-tooth headset 😲) Together with a great microphone I can still communicate without to much interference being picked up from the speakers. The speakers do a well enough job, although the sound on my Sony is a lot better 😀.

Microphone

As mentioned before, I used the microphone on my C922 for a couple off months. During that period I really wanted a better one that I could move closer to my mouth for that deeper radio voice 🎤. I had to wait for some months to actually be able to order the one I wanted, since that happened to be out of stock at the time as well 😱. I finally landed on a set of both microphone and flexible arm in one: the yeticaster. It is the yeti microphone, the Radius III (shockmount) and Compass (flexible arm). The arm is what drew me in the most: a lot of flexible arms have an open construction with the springs visible. I don’t like the look of that and I’ve read that those springs can wear out (which can then still be replaced). This arm uses internal springs with friction control on the parts where it bends. It looks very slick and smooth and even adjusts really well.

Desk

At the beginning of the pandemic I needed a desk fast and wasn’t expecting to need it for a long time. I still liked the standing desks we have at the office, so I really wanted a standing desk. Look around and most of the shops where closed and only Ikea had the option to remotely order one and pick it up. So I picked an Ikea Skarsta desk, with a manual lever since I found the premium for the electric version to much. How often do you stand during the day anyway? For me that was once or twice, and I could use the two minutes of exercise 😀. What was I wrong on this one! The desk wobbles a lot and I don’t stand often enough because of this: lowering and raising the table makes everything wobble. Some of the placement of the lamps doesn’t really help currently as well: since I am very tall, the desk needs to be raised to its full length (1.20m) and the left lamp hits the sloped ceiling. I need some more space on the right to shift things over, but I am still looking for a better way to stash my laptop somewhere: I have it either on my desk or on a stack of stuff on the right of my desk.

The reason for this is that the laptop produces a lot of heat and having it flush on the desk is not giving it enough air to breathe. I’d rather have it hanging under my desk or in a good open laptop drawer. I bought a great drawer, but made a mistake by reading the inner size and measuring that out. Unfortunately the outer sizes don’t find under the desk 🙁. If you have a better recommendation: please let me know! I’d rather not see it at all, so under the desk would be great. Another issue with that is that I haven’t managed to get my Dell XPS 9500 to sleep and then wake up with something. I’ve tried the on-power trigger (doesn’t work since I use the 90W power over USB-C coming off of the monitor. Dell thinks that is not a power supply, and underwatted even since it wants 130W).

Led strips

The led strip is mostly for myself and is wrapped around the edges of my desk. It is an Elgato Light Strip. You can set them from ultra-bright to fully dimmable, choose both the lumen and the RGB color you want to use. The control center for it is awesome and works within the boundaries of my WiFi network, which is good from a privacy standpoint. It also has support for a phone app, which I use all the time.

Windows Control Center:

I bought this one to try it out and light the wall behind me, since it has the flexibility and it can be shortened. I’m a fan of Elgato stuff, since they take good care of designing their products to look smooth, work great and even take a lot of care around packaging of them as well.

For the wall behind me it is a little bit short and the LED’s are to large: you would probably see them in my video feed. So now I have it wrapped around the desk (it handles a 90 degree bend perfectly) and light up the wall behind my desk. It curls a bit at the end that doesn’t have the power supply, but a small nail, pin or some extra glue will probably fix that (it doesn’t curl any further right now and I don’t see it, so it doesn’t bother me at all).

Keyboard / Mouse

For my keyboard and mouse I use the same one I got 3 years ago for my consultancy setup: I (used to) travel a lot and still wanted to have an ergonomic keyboard. That’s why I still use a Microsoft Sculpt Ergonomic Desktop. I don’t like the sculped mouse that is in the set, that’s why I only got the keyboard itself. I do use the numpad these days as an easy toggle in OBS for its Virtual Camera. This way I can easily switch in any meeting tool that we happen to use at that time. (Seriously, I think I have most of them installed 😱).

Laptop

As a laptop I use the one our company provides us with. They let us choose a new one every 2 - 2,5 years and you have all options with a couple of manufacturers. When I got started at Xpirit I choose a Dell XPS 9500, with 32 gigs of RAM and a 1 Terabyte SSD. This was one with the up-your-nose-camera at the bottom of the screen. Always fun during video meetings 😄.

Just recently I got to order a new one and now I have a Dell XPS 9700, again with 32 gigs of RAM and a Terabyte SSD. Luckily this one has the camera on the top of the screen and even has Windows Hello support, both with the camera and a fingerprint reader.

Headset stand

I was doubting between a wireless charger to place my phone on and a separate headset stand to place my headset on. I didn’t want a separate charging pad on my desk and wanted to stop laying the headset flat on the desk everyday. So this seemed like a good compromise. It works really great! It’s a URGOODS headset stand.

Update 2022

In the beginning of 2022 I started automating some things in my office to have less things to turn on and off 😁. Writing the repo readme with all the setup triggered me to update this post with the new stuff. (Yes, lot’s of stuff will turn on and off ‘by itself’ now).

Speakers

In December of 2021 I’ve added a set of speakers to the setup: I didn’t like the ones from the monitor, especially on calls there was something annoying with the base from it. It supports Bluetooth, dual RCA and a normal headphone jack (which I use for my laptop). After looking around (and asking colleagues who had opinions 😁) I settled on the Edifier R1700BT 2.0 Pc Speaker. It is a really nice set for this price, and is not to big on the desk as well. I also like that they are very forgiving for shutting on and off by just switching off the power through the power cable (instead of the switch on the back). That helped with turning them on and off from some home automation scripts!

Do Epic Shit signal

We have an internal motto at Xpirit that tells us go do out and ‘Do Epic Shit’ (which is whatever you deem epic), with a logo, T-shirts and since the last holidays an ‘Do Epic Shit’ signal as well! This is linked to an Azure IOT setup with a service bus that can push messages to us (by using the light as morse code signals). Fun thing to have, and it lights up the place!

What is next?

What I still really want is a good way to have the laptop running somewhere, with enough air to breathe. As long as it is not on the desk but hidden somewhere, preferably with a remote boot option, so I don’t need to physically press the power button.

Next I’d like a good docking station, to support both of my laptops in case I need to switch (I currently mostly remote into the old one if I need something). Since I’m currently working for a customer that has company issued laptops, I often need to switch all my gear between two laptops. That currently is a lot of cable wrangling (both of them have different ports setup, so I have different USB hubs for them). Ideally I’d want one that can drive the two screens on my monitor, supports a lot of USB/USB-C plugs (I have eight or so in use right now), and can connect (and power) my laptops with just one USB-C cable. That way switching between my own laptop and the one from the customer is just switching one cable.

The last thing that I’d like is something to light up the wall behind me with different colors. Something like the Nanoleaf would be really nice!

Example of the nanoleaf on a wall behind a monitor:

GitHub and Azure DevOps: best of both worlds

2021-04-23T00:00:00+00:00

Ever since Microsoft acquired GitHub we have been looking at how this will play out in the long run with their Azure DevOps offering. We’ve seen a lot of people move from the Azure DevOps team to GitHub by looking at their LinkedIn or other social network updates, so it only makes sense to wonder about the future. We know that several very large teams inside Microsoft are using Azure DevOps, together with some really large customers, which means Azure DevOps will stick around for a long time and will still receive regular updates to its functionality. Azure DevOps has a very strong offering, from ‘everything in one suite’ to mix and match with third party tools, to a good hosting model (data sovereignty is a big thing for us Europeans) and excellent Azure support with even integrated billing on the marketplace.

GitHub on the other hand is the largest dev-community in the world, with some 56 million plus developers on it. If you want to entice developers to work at your company, doing so with the tools they already know and love is an excellent selling point. GitHub is made by developers, for developers. This also gives them a lot of information from open source communities into their activity, languages used and even a great host of information they can pull from the open source code stored on the platform. They are now even hosting security advisories and enable scanning for vulnerability patterns in your codebase with CodeQL.

Exploring best of both worlds

In this post I want to share what are the strengths of both offerings and explain what parts I’d use of each of them and why. In my opinion both products have some unique features that the other doesn’t have (yet), so using the right tool for the right job is only logical from that. Luckily the integration between both is very good, so mixing and matching is a great option here.

I’ve split things up in the following sections:

Source control
Work tracking
Pipelines
Artifacts
Security features

Source control

In the basis, source control on both platforms is very similar: Azure DevOps still has TFVC support, but I really don’t recommend using that at all: Git is the standard these days and for good reasons (that I’ll skip over for this post 😁). Both platforms have branch tracking, pull requests, a branch protection system, a good CLI to work with, so choosing what to use here is comes down to the extra features on top of the basics. That is where GitHub has the better options in my opinion because it offers these features:

GPG commit signing
Branch protection rules with GPG commit signing
Out of the box security features like code scanning

GPG Commit signing

Commit signing is an extra layer of security on top of basic authentication. Without using commit signing it is rather easy to make some code changes for you, commit them locally to the repository and then push them upstream. I’ve automated parts of this lots of times know, so I know how easy it is to create. Running the software on you machine, scanning the repos and adding some malicious stuff to your projects is rather easy and you would not even know that it was happening right under your nose 😄. What’s even worse, I could even run this on my own machine and push in code changes with your email address and user name attached to it. The configuration in Git is just meta data that is not verified anywhere in the chain: GitHub only checks if the user pushing the changes has write access to the repository and then lets the changes in. This is of course the standard for all source control systems using Git.

With commit signing, you need to sign the commit with your private key that only you should have. By default, that key can only be used together with a passphrase (you can create it without a passphrase, but that would defeat its purpose). Adding the passphrase means that when you sign the commit, you have to type in the passphrase. There is no way to automatically fill in the passphrase for you (that I know of). This also means you cannot sign the commits automatically, which means someone cannot spoof your signage on your behalf.

The server side (GitHub in this case) will store the public part of your GPG key that they use to verify the signed commits: those are made by users with GPG signatures AND those signatures have been verified with their accompanying public key. This enables you to lock down the repository or specific branches in the repository to only allow verified commits. .

Work tracking

Work tracking has always been great in Azure DevOps and I think it still is the better offering. GitHub has several options to enable some of the same setup, with great integration into source control with issues and pull requests. You can even add subtasks to an issue with task lists, although I like the Epic setup with subtasks better from Azure Boards.

Available in both offerings

Let’s look into what is available in both offerings:

Rich text setup (with markdown) that includes emoji, gifs and even video’s in most fields.
Easy way to create new items (even with CLI and API’s)
@Mention other users, issues or pull requests in the text of a work item / issue
Discussing on the work that needs to be done
Notification system for items that mention you
Labeling and searching on them

Note: what GitHub doesn’t have is the whole formal setup with approvals, adding in extra fields, adding story points (and summarizing them). Some of it can be replicated if you want. Personally I don’t like all that extra stuff, it’s unnecessary process ceremony in an effort to feel in control over the agile process, while it only brings extra gates to jump through without adding the correct value. That might be a different blog post 😄.

GitHub’s Projects vs Azure Boards

GitHub has created project boards in an effort to give us more control over issues then just the list it is by itself. It basically adds a kanban setup to track the issues through different stages. By default this is just ‘To do’, ‘In progress’ and ‘Done’. This works great in an OSS project where you don’t operate in regular sprints (who knows when you have the time and energy to work on things). If you are working in an organization that has sprints, you need something to define them. That currently isn’t available in GitHub: what they’ve created for it is Milestones.

By adding an issue to a milestone, you can track their state over a longer time period and view how much work is still left before the milestone is reached. The general idea comes again from the OSS where this works great: you create issues for things you find or features you want to add, mark them as part of a milestone and work together towards that goal.

You can use them for sprints as well, where a milestone can be seen as a sprint and (open) issues would be added to a milestone. That way they can be tracked over time and be used for planning. Only thing I don’t like about this setup is the naming: a milestone is not a sprint: usually what we see as a milestone can easily take multiple sprints to achieve.

On the other hand: I more and more dislike the idea of sprints: we humans are not meant to keep on sprinting all the time: from that we experience time like a treadmill where we run until we cannot run any longer. I feel like we should be working towards a milestone, with short iterations, delivering updates to production as soon as they are done and then enable the new features to the user when we feel like it is working ‘good enough’.

In conclusion: we should be able to work with Milestones if have a simpler process, skip things like story points and think differently about how we work as a team.

Azure Pipelines vs GitHub Actions

Automation for Continuous Integration (CI) and Continuous Delivery (CD) is available in both platforms. Azure DevOps has the benefit of being available for years now, whereas GitHub Actions only went GA for GitHub Enterprise in the beginning of 2021. GitHub Actions is catching up fast, but it is still catching up.

Azure DevOps has a robust and powerful setup for CI/CD: you can use the classic pipelines or yaml pipelines (preferred) and mix and match after your own preference. You can have a CI pipeline for fast validation on a branch, and a more extensive one for creating the artefact that you want to deploy. There are lots of extensions available on the marketplace, so you don’t have to reinvent the wheel. I just checked my Twitter bot @AzDoMarketNews and there are currently 1630 extensions available, with new ones still being added every once in a while.

Photo by Danil Shostak on Unsplash

Big things that are currently missing on GitHub Actions are:

Templates
Security scans on GitHub Actions
Better CI/CD story

Missing in GitHub Actions: Templates

The ability to use a template to group steps so you don’t have to repeat them everywhere. Think of a common template to use for an NPM pipeline for example: often you want to run the same steps in there, like dependency scanning, linting, building and running tests. GitHub has a setup for composite actions, but you cannot include other actions in them, only shell scripts. Same thing goes for defining something like a ‘stage’ in a deployment pipeline: I want to deploy my artefact on Dev/Test in the exact same way as to Production, with only some parameter changes targeting the correct cloud environment: currently, you have to repeat the same steps in both stages of the deployment. Adding a step in between means adding them in multiple places, or you risk drift between the environments. Azure DevOps has the better story here because they support templates in your yaml pipelines. This helps you with eliminating duplicate code from your pipeline.

Missing in GitHub Actions: Security scans on GitHub Actions

Currently there is no real way to trust the actions you use, other then trusting their publisher. In Azure DevOps there was a process to get an action published on the marketplace that at least included some protective scans before the extension was available. You also had to create a publisher account and only if you had the correct permissions you could install the extension into your tenant, where it would become available for everyone in the organization to use.

For GitHub Actions the story is a lot simpler, which also makes it less secure. The work to verify what the extension is doing, and keeping up to date with them (using the latest version is against the best practice) can take up time for your team. Anyone with write access to the repository can include new actions in the workflows (the thing that executes your actions). Anyone with a public GitHub repo can create an action.yml file in their repository and you can use it in your workflow. Without any validation by GitHub. Therefore it is really important to lock down your organization and think about the actions you want to allow. It is critical to have a process around GitHub Actions to still enable the DevOps engineers to run them, but in the safest way possible.

Note: I’m still looking for a way to leverage a CodeQL scan on the action repository with all the incoming changes included as a check on the issue. I’ve seen that GitHub can run them on all actions in their system and then send in Pull Requests fixing any issues it found. I would really want to run that as an extra security scan as well.

Missing in GitHub Actions: Better CI/CD story

With GitHub Actions you have two options for CI/CD:

Create one yaml file with both CI/CD in them
Create a yaml file for CI and a separate file for CD Both options have the same issue: especially on the CD part: I want to use the same steps to deploy to all my environments. Maybe insert something additionally in a specific test environment, or have an option to run a load test separately, but mostly I want to use the same deployment. Currently this cannot be done in a straightforward manner. You have the option to add an extra workflow for a new environment, and you could clone an existing one, but that is mostly it. I’d want something like Azure DevOps templates or what GitLab has. In GitLab you can include a separate yml file as a template, or use anchors to reference complete scripts and have a better way to reuse them between multiple stages.

Artifacts

In terms of artefacts I’m somewhat split:Azure DevOps has a great story around artefacts: the support for having different feeds and even different views on a feed, where you can upload a package to a Beta feed and then promote them to production is really great. GitHub doesn’t have something like that.

Same story around Universal packages: you can story any file in them, even publish a build script in them for later reuse outside of Azure DevOps for example. GitHub doesn’t have something like that. There is the concept of Releases but that will always download the release as either a zip file or an exe or msi file. Sometimes you need to download the files themselves, without any additional wrapping. There is an item on the roadmap to support this, but it is not there just yet.

Docker images

Where GitHub currently shines, is the option to have your Docker images stored in GitHub Packages. This works out of the box (note: I’ve only used this to publish public container images, not sure if you can lock that down easy enough): if someone pulls an image from this feed, it will just work. The automation flow to push images is simple as well: you can do it from a workflow, as I am doing here where I am pushing my images into ghcr.io/rajbos/local-dependency-updates. It’s using the GitHub Container Registry for it, with my GitHub user tag as the image publisher.

Security features

This is the killer feature for GitHub at the moment: I really recommend checking this one out and start moving over your code into GitHub as soon as you can. You can still use Azure Boards and Azure Pipelines if you want to: the integration between the products is wonderful and just works™️. You can still commit your code and tag issues on either platform: through the integration it knows where to look and will render the correct links in the UI so your users can seamlessly switch to where they need to work.

Some parts of the security features in GitHub can be replicated inside of Azure DevOps, but only with a lot of elbow grease. GitHub has them available in the product itself. Some of these are free for public repos and not for private repos. You can find more information on that on the pricing page. If you are running GitHub Enterprise server, then some of these are available through GitHub Advanced Security.

Dependabot

First off is Dependabot. This a tool that can scan your code and the third party dependencies that you are using. It has support for a lot of package managers and even can update the Action versions you use in your workflows! You can set it up to run periodically and it will scan your repository for you. If it finds any updates it will gather the release notes from them and create a new Pull Request for every package. If there already is an open Pull Request for the package, it will update that PR with the latest version, or close the old one and create a new PR. This is a very easy way to keep your dependencies up to date (and believe me it is: I’ve created my own setup here). If you have good checks in your pipelines, you can even enable auto merge on your repository and automatically close the PR when all checks are passing. This way you’re always up to date with the latest versions of your dependencies, saving you a lot of update and security headaches in the process.

Security scanning

When you have Dependabot enabled, you can also enable vulnerability scanning. With this feature, Dependabot keeps track of your dependencies and the versions that you are using. If there is a new vulnerability found and published on their own advisory, Dependabot will alert you of the vulnerability and create a Pull Request on the repo with the issue. It will even tell you if there are multiple repositories with the same vulnerability in them.

Secret scanning

Another great feature that GitHub launched in March 2021 (GA) is Secret Scanning. They have a long list of partners for which they can find secrets (think passwords or access codes) from in your repository and even automatically revoke those secrets. How awesome is that?! You still need to be aware of secrets and could include a process in your pipelines to detect them. This feature automates that and helps you with the biggest issue of committing your secrets into your repository: they should be considered as leaked: you can use Git BFG to clean the secrets from your repository, but someone might still have an old copy of the repository somewhere and still have the old secrets available. It’s considered much better to revoke those secrets immediately and create new ones.

Code scanning with CodeQL

Code Scanning with CodeQL is a recent addition to the GitHub offering as well. It only supports a couple of languages right now (see below). It can do very powerful things, like finding user input that is not sanitized before you use it in your code, even if it’s only used three layers deep in your codebase!

C/C++
C#
Go
Java
JavaScript/TypeScript
Python

There are a lot of CodeQL queries available in the public repository that you can run on your code. There is even one that checks you code for traces of Solorigate and other backdoors. You set it up by adding a CodeQL Analysis workflow in your repository.

By running it in your workflows, you get notified if it finds problems in your repository. Those notifications are added as additional checks and they can be found as annotations on incoming Pull Requests as well.

All in all, very powerful stuff!

Bash cheat-sheet for PowerShell devs

2021-04-11T00:00:00+00:00

Coming from years of Windows dev experience to using a Bash shell took a while to know some of the differences between the two. Since I often still run into these things, I thought it would help me and perhaps other people as well if I wrote some of it down. I suspect this will be a work in progress, so this page will receive some updates over time 😁.

Working with variables

Working with variables is a bit different in Bash. Setting a variable is done by using its name:

if it doesn’t exists, it will be created
if it already exists, its value is updated

Set variable example

myVar=12

Using the variable example:

To use the variable, you need to add the $ sign in front of it. This means you want to use the value in the variable. Watch out though! The shell will try to execute whatever is in the variable. So using the same command you are used to in PowerShell will not work:

$myVar # error here: the shell wants to execute the value in the variable, which is 12 in this case.

Example and output: In order to set the variable and then echo it’s value to the prompt, you can use this setup:

myVar=12
echo "$myVar" # handle the value as a string and output it

String interpolation in Bash

String interpolation works differently as well. Injecting the variable into a string using double quotes will interpret it and inject the value in to the string:

echo "this is my value $myVar" # this wil inject the value of myVar as a string

Wrapping the string in single quote will inject the object into the string and not the value:

echo "this is my value $myVar" # this wil inject the object myVar

Especially when someone is using the single quotes in a script, this has bitten me once or twice: it took some time to figure out why my value wasn’t visible.

Fileheaders

Add a file header (first line) to the file contents to indicate how the shell should run this file. Example:

#!/bin/sh

This will indicate it needs to feed this file to the sh command inside the bin folder. It will be used when you execute the file from the command line like this:

.\myShellscript

Outputting to a file

Sending some output to a file is similar to the use in PowerShell:

"my string value" > myShellScript # this wil always create a new file and set its content
"my string value" >> myShellScript # this wil add the string to its content

The thing that always gets me, is that dang string interpolation difference:

cls command

In bash there is no cls command shortcut. Always use clear to have the same result.

LastExitCode and returning it from a script

Just like in PowerShell you can use $? to check the last exit codes. In the example below I’m deploying a CDK stack to AWS (Infrastructure as code). And saving the exit code of the npm command so I can check it, log it and return it to the caller of this shell script.

npm --silent run cdk -- deploy --require-approval never --context env=${env} --context tag=${TAG} ${stack_name}
exitCode=$?
if [ $exitCode != "0" ]
then
  echo "NPM deploy cdk step failed: $exitCode"
  exit $exitCode
fi

GitHub Actions Create: Dockerfile symlinks error

2021-04-09T00:00:00+00:00

I was building a new GitHub Action today with a Dockerfile and got a strange error… `unable to prepare context: path “/action” not found.

I was using a Dockerfile in a sub directory, but the documentation indicates this is supported. So, what gives? Is it a mix-up of the file path? I’m testing it on Windows and can build the Dockerfile itself just fine 🤔. Tested with a backslash instead of a slash, but still an error: unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /path/to/Dockerfile

The fix

On a hunch (this has happened before!), I changed the file ending of the Dockerfile from CRLF into LF (created the file on Windows of-course!) and… 🎉.

For future reference

Check the line endings on your Dockerfile!
You can run an image in your GitHub Action in a subdirectory by using the correct path to it. It needs to be relative to the action.yml file!

# action.yml
name: 'Octogotchi'
description: 'Octogotchi'
branding:
  icon : alert-circle
  color: blue
runs:
  using: 'docker'
  image: 'action/Dockerfile'

GitHub Actions & Security: Best practices - One workflow per runner

2021-03-07T00:00:00+00:00

One important best practice for any Continuous Integration / Continuous Deployment setup is thinking about attack vectors for your setup. One of those vectors is the way you download your third party dependencies. Whether you are using Docker containers or libraries to build your code upon, these dependencies are external to your system. Usually these are pulled in either through a Container Registry (for Docker images) or through a Package Manager. You are downloading those dependencies at build or deploy time from an external source, usually from the internet.

There are several ways that a malicious actor can try to leverage these dependencies in an effort to compromise your system.

This post is part of a series on best practices for using GitHub Actions in a secure way. You can find the other posts here:

Photo by Georg Bommeli on Unsplash

One workflow per runner

The best practice here is to only use a runner for one single workflow. You might be tempted to setup a set of machines that are dedicated for all your pipelines and install multiple runners on the same machine. Or to maximize the utilization of your machines by letting all repositories use the same runners throughout your organization.

Attack vectors

Some example attack vectors that you need to think about with this setup are:

Data Theft
Data Integrity Breaches
Availability

Data Theft

An action could be siphoning off your data to a third party. This can be anything: from your host information (OS, IP-address, network information, etc.), your software installation, to locally installed certificates, API-keys or SSH keys. For this reason you need to severely limit the user rights the runner is operating with. Under no circumstance give it root access or network administrator access. Limit it to a service account with access to only the bare minimum it needs: does it really need any internal network access, or is an outgoing https connection enough to do its job? Don’t give it access to more of your local disk then the folder it needs to operate on. Only give it write access there and even limit the read access to most of the rest of the disk.

This is a reason to not blindly use 1 runner and give it access to all your workflows: each workflow should only have access to its own data, and not that of a different workflow. The user the runner is operating on, will have access to everything inside of its workfolder, with all consequences from that access (keep reading below).

Data Integrity Breaches

We all use open source package managers or container images to build on top of. Most package managers and container setups use a local cache to prevent a new download each time you need a package version. If you reuse the machine, you enable the cache to be used by multiple workflows. What if workflow A runs a malicious script or action and overwrites some of your cache? Workflow B (and C and so on) will be using a package from the cache that has been overwritten with perhaps some malicious content. Solorigate was a prime example of this attack vector: one assembly overwritten without anyone noticing with the huge consequences from it.

Availability

Having something compromise your runner does not specifically have to mean stealing your data or overwriting it. Another option can be that the malicious actors want to achieve is hurting your availability to use the workflows altogether. One scenario might be that a zero-day attack is found against your application that can be used to do harm to your company or its users. What if an attacker then has the option to flip a switch somewhere on your runner and all of a sudden you cannot deploy the fix for the zero-day? You’re probably very dependent of your pipelines doing the deployment and might not have ‘break the glass’ setup to deploy an update to production.

Summary

All examples above are reasons to not reuse your runners for multiple workflows. Limit their access to your machines and network as much as you can and consider setting something up where each execution always gets a new and clean environment, to prevent security issues from it. Using the hosted runners from GitHub might even be a better option: GitHub always gives you a new clean instance

SonarQube Community Edition - Change default branch name

2021-02-12T00:00:00+00:00

Small tidbit so I could potentially find this later on: If you are running SonarQube yourself with the Community Edition (for a POC for example, otherwise invest in yourself by getting a higher edition!) then you might find this useful.

Default Branch name in Community Edition

The default branch in SonarQube Community Edition is still locked to master. If you want to change that in SonarQube then there is no user interface option available, as is in higher editions under Administration –> Branches and Pull Requests.

You can use this url to get to the same page for your project and change the name there: https://sonarqube.server.url/project/branches?id=<your project name>

GitHub Actions & Security: Best practices - Private Runners

2021-02-07T00:00:00+00:00

In this post I want to look into Private Runners for your GitHub Workflows and show you some best practices for them. GitHub Workflows can run on either a GitHub Hosted runner or on your own private runner. For the private runner you can install it on a machine of your choice and you maintain everything on that machine: the tools that you pre-install, the network stack the runner has access too and any storage it is using during its runs.

This post is part of a series on best practices for using GitHub Actions in a secure way. You can find the other posts here:

Photo by Sherise VD on Unsplash

Why use a private runner?

Some organizations have a policy that their builds (or deploys) always need to run in their own (hosted) datacenter. Another reason might be that you need to use some licensed software that you can run only on your own environment. Or maybe you need a GPU setup in your workflow (those aren’t available from GitHub Hosted Runners as far as I know).

The runner itself is open source so you can look at the way it works and contribute updates to it. Installation is made very easy: you can go to your repository or organization or enterprise. Go to Settings –> Actions and at the bottom you can find a button to add a new runner.

If you add the runner at the organization level, it will be available to all repositories in that organization. If you add the runner at the enterprise level, it will be available to all repositories in all organizations in that enterprise.

Installation consists of executing these steps:

Download the latest zipped version of the runner at https://github.com/actions/runner/releases
Unzip it
Run the config.cmd script with the url of the level to add the runner and a specific token (note you can also get this token trough the GitHub API). The token will be valid for 1 hour
Start the runner

The runner will run with the privileges you provided during installation and will start a long polling outgoing connection over HTTPS to GitHub. It’s an outgoing connection for safety reasons (e.g. corporate firewalls) and goes out over HTTPS to make it as secure as possible. The runner will periodically ask GitHub (I think every minute) if there is work to do, and if not, back off for another minute.

Best practices in this post:

Limit the access of your private runner
Do not use a runner for more than one repository (on an persistent runner)
Never use a private runner for your public repositories

If you want to learn more about hardening your runner environments, you can read the GitHub documentation here.

Limit the access of your private runner

Limit the access of the private runner to an absolute minimum. Think of all the things the processes the runner itself or the actions it will run for your have access to. Never install it with network admin or root rights. Give it just enough rights to only do what it was meant to do.

Does the runner really need to have access to your database cluster? If not, limit the network access for it (segment the network for example).

Does the runner really need access to your file share? What could it do with all those files? What if an action scans the share and start sending the interesting bits to an external server? What if the action executes your code, downloads a third party dependency that has been compromised and that starts to do something on your environment?

What happens if the action breaks the runner sandbox? Could it elevate itself to a more privileged account?

Best practice: give the runner the absolute least amount of access it needs to the machine it runs on.

Do not use a persistent runner for more than one repository

This best practice comes from the same practice as before and mostly comes from the methods used in supply chain attacks like for example the recent Solorigate attack. Both the action that is being executed and the third party dependencies it uses (NPM or NuGet packages or all the layers in the Docker image it uses) can store things on the runner machine. Even the runner itself could store these things, although a compromise of the runner is less likely, it could still happen (we’re all humans after all). Context here matters: this is not so relevant for what we call an ephemeral runner: a runner that only exists for the duration of one job (not workflow!). Having data persisted on an ephemeral runner is not possible, since the runner will be deleted afterwards.

This topic is of course super important for persistent runners: think of a Virtual Machine that is constantly running.

What if a first run downloads a compromised package and stores that in your local package cache? A second run that needs that package, finds it in the package cache and uses that version instead of retrieving the correct one? As in the Solorigate attack, only 1 assembly was overwritten as a starting point of the compromise.

This problem could even be enlarged by using the same runner for multiple repositories: Package cache poisoning in one repository could be subsequently be used by all other workflows that are executed on the same runner.

Even worse: installing multiple runners on the same machine in an effort to enable more efficient use of the available resources on the machine: what if they all use the same package cache. NuGet uses a centrally stored cache on the machine level for example…

So as a best practice: if you really need a runner, create at least a new runner on a new machine for each repo and give it the minimum rights it needs to do its work.

An even better option is to create a new runner for each run. For example use an Action that creates a new runner just for that specific run, or host your runner on AWS Fargate or even setup auto-scaling yourself.

Note: the options above all use AWS for hosting. I haven’t seen any examples for this setup on Azure yet. If you have an Azure Example, let me know: I will add it to the list above.

Never use a private runner for your public repositories

Also mentioned in the guidance from GitHub is that you should not use private runners for public repositories. Depending on your setup, your workflows will be triggered by new commits being pushed to the repository, or by an incoming pull request.

What if someone with ill-contempt forks your repository, adds code or a dependency that will compromise your setup (or even the workflow itself) and create a new Pull Request on your repository? The workflow will be triggered (hey, they can even add the trigger for you!) and the malicious setup will run on your machine with all the access the runner has. This is considered very dangerous for obvious reasons.

GitHub Actions & Security: Best practices

2021-02-06T00:00:00+00:00

I’ve been diving into the security aspects of using GitHub Actions and wanted to share some best practices in one place. If you like to get an overview through a presentation setting instead of a blog, you can also find one of my conference sessions on it here.

Photo by Jon Moore on Unsplash

Setting up an internal marketplace for GitHub Actions

All posts eventually lead up to setting up an internal marketplace for GitHub Actions: something that your team can control, prevents random actions from being used and gives you the process to run actual security checks on the actions in use. Learn how to set one up in your organization here.

Forking action repositories

In the post on Forking action repositories I show these best practices:

Verify the code the Action is executing
Pinning versions
Forking repositories
Keeping your forks up to date

Additionally (and especially in an enterprise setting), you’ll want to create your own internal market place for actions. How to set it up and have a good security process around it can be found here.

Keeping your forks up to date

After forking all the actions you want to use, you also have to own the maintenance. I’ve described a good way of keeping your forks up to date here, by making sure you review the incoming changes before you merge them.

Secure your private runners

In the post on Private runners I explain these best practices:

Limit the access of your private runner
Do not use a runner for more than one repository
Never use a private runner for you public repositories

Do not reuse a runner, ever!

One runner, one workflow

Run your own action in a container

To have an additional boundary for your action you can run it inside a container. This also enables you to use something in your container that doesn’t have to be installed on the runner itself: Run your action in a container.

Run you runners in a Kubernetes cluster

To mitigate a lot of attack vectors from running your runners on a virtual machine (e.g. disk / network access), you can host your self-hosted runners in a Kubernetes cluster. Then you have ‘ephemeral’ runners that only exist during the execution of your workflow and then are cleaned up.

Untrusted input

An overview from GitHub on untrusted input, from the issue title to the commit message, if you act upon them (even just echoing them to the output!), they can be misused and therefor are an attack vector.

Preventing pwn requests

It used to be the case that you would trigger your workflow by using the pull_request in the trigger definition. That scope had to much rights, so GitHub has dialed that down. The scope with more rights (to your access token with some write permissions for example) has now been created to be pull_request_target. You need to be really careful with using that scope. Best practice here is to use a label for the pull request so you can manually check the PR and authorize its actual execution.

You need to be very careful with incoming Pull Requests from any fork: potentially they are changing scripts/commands in your workflow or even the workflow itself. There have been examples where a PR was send in that updated the workflow and made good use of the 10 concurrent jobs available for open source projects to generate some bitcoin for the attacker. This is why we can’t have nice things 😲!

GitHub Actions & Security: Best practices - Forking Action repositories

2021-02-06T00:00:00+00:00

I’ve been diving into the security aspects of using GitHub Actions and wanted to share some best practices in one place. If you want to have an overview how and why you need this, you check checkout a session I have on this topic from a user group recording here.

From the beginning, GitHub always indicated that the best way to use GitHub Actions is to fork the repository containing them and take control over the code that you are running. That way you can review what the Action is doing before you start using it. You don’t want to run just any random code from the internet, would you?

This post is part of a series on best practices for using GitHub Actions in a secure way. You can find the other posts here:

Photo by Anita Jankovic on Unsplash

Always fork the Actions repository and use that!

These days, most GitHub Actions demos only show you how to use them, straight from the source repository. That is scary as heck! We should always mention the best practice of forking them!

This post will go over the following ways to add more security when you are using GitHub Actions:

Why should you verify the code the Action is executing?
Pinning versions
Forking repositories
Keeping your forks up to date

Why should you verify the code the Action is executing?

The power of GitHub Actions is that anyone can create a GitHub Action in a public repository for others to use. That also means that there is no real trust between the user that wants to include the Action in their workflow and the maintainer of the Action repository. In the past, there are numerous examples of bad actors taking over respected repositories and changing the code to do something malicious.

That also means the onus of checking what the Action’s code is doing is up to you. GitHub has some methods of limiting the Actions you or your organization can use, but these have some issues as well. Read more on this below. GitHub’s security guidance for Actions can be found here

You can also look at the maintainer of the Action and how many stars, forks and issues the Action has to gauge if the Action is widely used and regularly maintained.

Only allow Actions from GitHub

Your first option to limit Actions being used is to use the setting to only allow Actions made by GitHub. That means that you put your trust into GitHub that the maintainers of the Actions repositories have setup sufficient repository security (limiting who can make changes to the Actions) and always verify incoming changes before publishing new versions of the Actions.

Only allow Actions from Verified Creators

The second option is to limit Actions from verified creators. Unfortunately there is no process (that I could find) to become a verified creator. GitHub only states that the maintainer is a partner that GitHub has worked closely with.

Only allow specific Actions

The third option you have is to limit Actions to the specific ones you list, either directly with the full path to the Action repository, of by using a minimatch filter that limits the Actions to a specific organization.

Note that you can also use specific commit SHA’s here as well, more on that later

Pinning versions

After you have checked the Action itself, you can make the decision to start using them. One of the first options you have is to pin the Action version. By adding the minor or major version of the Action. This will make sure you are always using the same version and (hopefully) prevent any breaking changes from messing up your use of the Action later on.

Example pinning the version

uses: gaurav-nelson/github-Action-markdown-link-check@v1
uses: gaurav-nelson/github-Action-markdown-link-check@v1.0.1

In the example, I’m pinning the major version in the first line to v1 and pinning the minor version to v1.0.1. This works as long as the Action author is using semantic versioning.

The downside of this method is that there is no way to guarantee that the source code of that version has been altered after you pin the version: an author can create a new release (with new code) and use the same version tag for the release. That means that the code that is running in your pipeline could have been altered, without you knowing about it!

Safely pinning version by using the commit SHA

The best practice for pinning Action to the version you have reviewed, is by pinning it using the commit SHA: this value is created for each commit and is immutable: meaning that this value cannot be changed without changing the code and that any change to the code will generate a new commit SHA.

Example pinning the commit SHA

uses: gaurav-nelson/github-Action-markdown-link-check@f0656de48f62c1777d073db4a5816eba1dcc1364

In the example you see a full length commit SHA that will run for this Action. You can find the commit SHA from the source repository by going to the commit itself with this link:

Note: don’t use the short 8 character SHA: this is much less secure and will be deprecated as of February the 15th, 2021 (announcement). Always use the full SHA.

Forking repositories

The best best practice is to completely limit your organization to only use Actions from an organization you control yourself and then fork all Actions to that organization.

I recommend creating a specific organization that only has Actions repositories in them. That way I have a central location to manage all my Actions and I can limit the Actions that are allowed in any other organization I have.

Benefits of forking

An overview of the benefits of forking the Actions repositories:

This gives you full control over the Actions you or your organization can use.
You have a copy of the Action in case something happens with the maintainer or the repo of the Action (this should not be underestimated).
You can review the Action’s code before you start using it in production (meaning your own workflows).
You know for sure that the source code of the Action does not change between your own runs.
You have full control over updates and when to review and incorporate them or not.

Don’t block DevOps teams

Keep in mind that only allowing Actions from your own organization will possibly block your DevOps teams from finding, testing and then incorporating new Actions in their daily work.

Follow the DevOps culture and empower teams to review new Actions and have a process to incorporate the new Actions in their normal workflows. This also allows them to take up ownership of the Action by reviewing the Actions source code before forking them to your Action’s organization.

To unblock them I recommend documenting the process and having a separate organization where they are free to test new Actions. Fork the Action repository there first, test them out and after diligently vetting the Action, fork them again into the production organization allowed in your normal workflows.

Keeping your forks up to date

If you are following the best best practice of forking the Actions you want to use, the you find another problem: how do you keep your forked repositories up to date? You can be on the lookout in the GitHub user interface for messages that your fork is a number of commits behind the parent repository, but when (if ever) will you come to that specific page?

Looking around I could not find a great way of keeping all Actions in your organization up to date AND giving you an option to review the incoming changes before you start using them, so I created my own 😁.

Keeping your forked (GitHub Actions) repositories up to date

I wanted a process that would update my forked repositories with these requirements:

work for all forked repositories in my organization (so not just for one repo at a time)
allow me to review the incoming changes
update the forked repository with minimal manual intervention
enable others to use this method as easy as possible

For this purpose I created this repository: github.com/rajbos/github-fork-updater. The information to start using it can be found in the readme and I will list them here as well:

Fork the repo into your Actions organization.
Enable issues (these are off by default for forks)
Enable the workflows in the fork and enable the schedule on the check-workflow (off for security reasons)
Trigger the check-workflow manually to get going (or wait for the schedule to be triggered, which is on workdays at 07:00)
Check any new issues being created
Add a secret to the forked repository with the name PAT_GITHUB and the rights to push changes to the repositories in your organization
Review the incoming changes for the fork you want to update
Label the issue with update-fork
Wait for the magic to happen
Your fork is updated
The issue is closed
Sit back and wait for new notifications the next time the fork is out of date

You can also watch the demo video I created here:

Summary

This will make your life a lot easier. The default GitHub notification methods are used to notify you of new issues. All issues are always in the GitHub Fork Updater repository, instead of all over the place. And you can choose when to review the incoming changes and have an easy way to update the fork.

GitHub Actions & Security - NDC London

2021-01-28T00:00:00+00:00

Today I delivered my GitHub Actions & Security session at NDC London for the first time (both for the session and NDC London 😁).

NDC London was a lot of fun! The organizers really went out of their way to enable speakers to even see the attendees, welcome you into the room and guide you through things.

Presentation

You can find the presentation on SlideShare.

Keep note of this page, as I will start adding new blogpost discussion security measures when using GitHub Actions here with the learnings from this session.

How to secure your GitHub Actions

When working in the real world with continuous integration / continuous deployment, you have to take care of your pipelines. - Who can push to an environment? - Who could change the connection strings to the database? - Who can create new resources in your cloud environment? - Do you trust your third party extensions? I’ll go over each of these aspects of your GitHub Actions Workflows and show you what to look for and how to improve your security stance without locking every DevOps engineer out.

More info about this session can be found here.

GitHub Container Registry

2021-01-06T00:00:00+00:00

I wanted to use the GitHub Container Registry to host an image for me and had some issues setting things up. To save me some time the next time I need this, and hopefully for someone else as well, I wanted to document how this process works.

Photo by Evgeni Tcherkasski on Unsplash

Beta period

During the beta, the container registry will be free to use. Open-source and public repositories are always entirely free to use, but private repositories will fall under the standard billing rates for GitHub Packages after the beta is over. The free tier of that includes 500 MB of storage and 1 GB of transfer every month.

Enable GitHub Container Registry

Currently, the registry is in Beta, so you’ll need to enable the beta feature on your profile or on the organization level you want to use it on. To do so, go to your profile (or organization) and go to Feature preview where you can toggle the feature. You’ll notice a new ‘Packages’ tab on you profile page as well.

Preparing to push image to the registry

Currently the only way to authenticate with GitHub Container Registry is to use a GitHub Personal Access Token (PAT). GitHub already knows this is an issue because the PAT can be used in the entire account it is created and will change that later. For now the advisory is to create a specific PAT with only rights to the registry and use that.

These are the scopes you need to enable for the PAT:

read:packages scope to download container images and read their metadata.
write:packages scope to download and upload container images and read and write their metadata.

If you want to delete the packages, also use this scope:

delete:packages scope to delete container images.

Using a GitHub workflow to build and push an image

To push a new image from a workflow, use the complete example below.

The steps used are as follows:

Get the source code with the docker file and anything you need to build the image. ``` yaml
- uses: actions/checkout@v1 ```
Build the image ``` yaml
- name: Build the Docker image run: docker build -t ghcr.io/«ACCOUNT NAME»/«IMAGE NAME»:«VERSION» . ```
The normal docker build step where I am tagging the image with the tag I want to push to the registry ``` yaml
- name: Build the Docker image run: docker build -t ghcr.io/«ACCOUNT NAME»/«IMAGE NAME»:«VERSION» . ```
Authenticate with the GitHub Container Registry Using the recommended setup from GitHub) for safety ``` yaml
- name: Setup GitHub Container Registry run: echo “${{ secrets.GH_PAT }}” | docker login https://ghcr.io -u ${{ github.actor }} –password-stdin

5. The normal `docker push` step to push the container
``` yaml
    - name: push to GitHub Container Registry
      run:  docker push ghcr.io/<<ACCOUNT NAME>>/<<IMAGE NAME>>:<<VERSION>>

Full workflow example

name: Build and Push Docker container

on:
  push:
    branches:
      - main

jobs:
  build-and-push:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v1

    - name: Build the Docker image
      run: docker build -t ghcr.io/<<ACCOUNT NAME>>/<<IMAGE NAME>>:<<VERSION>> .

    - name: Setup GitHub Container Registry
      run: echo "$" | docker login https://ghcr.io -u $ --password-stdin

    - name: push to GitHub Container Registry
      run:  docker push ghcr.io/<<ACCOUNT NAME>>/<<IMAGE NAME>>:<<VERSION>>

Secret names

Do note that I am using secrets.GH_PAT to inject the PAT token I’m using into the workflow. You cannot use GITHUB as a prefix for the secret name, so you need to change that. The secrets user interface doesn’t tell you about that in a great way, which I have sent GitHub feedback on through the GitHub Community.

Consuming the new image

By default the images are kept behind a login, so if you want to make the image publicly available you need to do that for each package.

To use the image behind the login, you’ll need to authenticate with the registry first:

echo "$env:GH_PAT" | docker login https://ghcr.io -u USERNAME --password-stdin

Enable the public registry

Note: this is a one way trip: it cannot be made private after making it publicly available

To change this setting: go to the package and to its settings:

And make the image publicly available:

DevOps Maturity Levels - Continuous Deployment

2020-12-31T00:00:00+00:00

If you have a good Continuous Integration process in place, you can start using the generated artefact to deploy that to an environment as the next state of enlightenment in your DevOps way of working. Check that link for posts on the other topics.

Note: in this case I specifically mention an environment: any place you can roll out your artefact is part of your Continuous Deployment strategy. Where specifically is part of forming your strategy and there are plentiful ways of doing so. Different strategies will be discussed further down in this post. For now, lets take rolling your application out onto a test environment as a starting point: the goal is to roll out the application so you can verify if things are working as expected. You typically do so on a test environment: something that is not production 😄.

Note: Application can be anything, but I’ll use a web-application as an example here

Note: Environment in this case will then be a hosting environment to run the application, so something of a webserver in this example

Note: I’ll use the application as running in the cloud as an example, some points can be made about other platforms

Continuous Deployment (CD)

There are some differentiation one can make about a difference between Continuous Delivery (deliver the artefact to a server) and Continuous Deployment (enabling the changes to an end-user). I’m not making that distinction in this post: the goal is to roll out a new version of the application to our test environment. As far as DevOps enlightenment goes I think the following steps can be separated out:

Deploy to an existing environment

When you start here, you’ll probably start with an existing environment. Perhaps you are already using a tool to deploy to that environment manually: time to step up your game and deliver the updates automatically. With DevOps tooling you can automate this process to deploy the update with each new artefact being created: you want to automatically trigger the start of your deployment when this happens and the CD pipeline should do the rest. In the cloud example you’d use these steps:

Trigger the CD pipeline on a new artefact
Download the artefact
Connect to the hosting environment
Deploy to the environment
Signal the development team a deployment was successfully done

Infrastructure as Code (IaC)

A next step could be to start creating the environment you want to deploy to automatically as well. This enables you to:

create a clean testing environment whenever you need, for example for a Pull Request verification build
recover from issues with you environment for example when somebody accidentally deletes something
rollout the environment in different cloud regions: when automating the creation of the environment this becomes very easy to do

This step does not necessarily come in this order and could be skipped in favor of the following two steps. You’ll probably find the need to start working with IaC soon thereafter, and adding it here usually saves you pain later on.

Nothing broke with the deployment checks

After rolling new version out automatically, it is good to add a step in the CD pipeline to verify that the application still is working as expected. You don’t want to rollout every change and find out from your users (or stakeholders!) that the test environment is not working! Better to add a step and verify yourself.

At first it could be a basic test: if I navigate to the web-application, do I get a HTTP 200 Ok response back:

url="https://devopsjournal.io"
status=`curl --silent --head $url | head -1 | cut -f 2 -d' '`

if [ "$status" != "200" ]
then
    echo "status was other than '200': was '$status'"
    exit 1
fi

This should at least tell you if your application can be loaded by the hosting environment and can load the start page. It doesn’t really tell you anything else.

A next step could be to add the additional tests you need to get more confidence in your application that indicate that it still is working as expected. You could go all out with this and start creating end-to-end tests for the application checking each and every screen and functionality. Unfortunately these tests are hard to build, maintain and slow to execute.

You can also add just the tests you need to verify that the basics are there: test the index page, maybe login with a test user and check if some mayor parts of the application can be loaded. Then you already have several end-to-end integration points you are testing as part of your deployment: the application itself, some user authentication and perhaps authorization and maybe even the way you build up your pages or a menu for example. Try to make sure you can run these test fast and in parallel to gain even more speed.

Automatic Rollback options

What do you want to happen when you detect that a new deployment fails? In DevOps we usually say we want to roll forward: find the issue, fix it and roll the new out to the environments. When starting to do Continuous Deployment, this is not always the first thing you think about. Perhaps rolling out doesn’t give you enough confidence yet. Or perhaps your rollout is not fast enough for this. And there is always the case where you want to investigate what is going wrong at all, before you can fix it. Finding the issue can take some time. In the meanwhile, you want to get back to a known state. This can be hard, depending on the way you rollout updates. For example, think of database upgrades here: if you are changing data in the update, perhaps the previous version of the application cannot handle the new database schema. There are ways to remediate that, giving you the opportunity to rollback: re-deploy the last known good version, enabling you to find the issue with the erroneous version.

Often, the systems you use for your CD pipeline has an option for rolling back to a previous version: this can either be triggered manually or automatically.

Canary deployments

Alright, when you have your setup so far that you can deploy to any environment, with confidence that the new version is still working and with a rollback option if needed, you can start thinking about Canary Deployments: a deployment to a subset of your users. This is a method to limit the scope of the changes to only a small part of your users in an effort to limit the possible issues that the new version has. This way you can monitor the changes and from that decide if the changes are behaving good enough to roll them out to a larger set of your users.

There are lots of ways to roll out to a subset of your users: you can use a percentage of the incoming requests and direct them to a new server and slowly ramp that up, or let the users set a setting that they want to be alpha/beta users (even for just one functionality). This enables you to monitor the environment and roll out with more confidence.

Deploy without downtime

Most deployments to a server involve some downtime. Overwriting files on a web server has issues with locks when files are in use. Some servers make your stop the running website, overwrite the files and then start serving incoming request again.

This stage goes well with the previous one: if you have the ability to get an environment and split the traffic up between the old and the new version, you have found the way to deploy without any downtime: the basis is that both the old and new version are running at the same time and you control where the request are going to.

DevOps Maturity Levels - Source control

2020-12-31T00:00:00+00:00

One of the first things to get in order when improving your DevOps way of working is having proper version control of your source code. Source code in this case means anything: from application source code that you can build and deploy, to scripts you use to do the deployment. In my opinion: anything around your team that can be saved as text, should end up in source control.

Source Control in order

I still come across teams that don’t have any source control for some parts of their tools, or have ancient systems that don’t have support anymore. Or even worse: things live on a specific server, stuffed away beneath someone’s desk 😱. If that ‘server’ fails, everything could be gone. If a copy is stored on someone else’s machine, then maybe getting things back in working order is ‘just’ a few days of diligent work.

At a previous company something like this happened on Christmas Eve and I was very lucky to only find out about this after the holiday period, since someone else had taken up the chore to get a new server (a remarkable feat around the holidays!) AND restore everything back to working order for everyone in the new year.

Git: distributed version control

For source control you can pick any tool you like, I’d only recommend it to be Git based and skip the other ones (like SVN, TFVC). Git is the de-facto standard these days: investing in learning it will give you a solid base that can be reused across different organizations.

One of the reasons Git has become the standard is that it is distributed version control: this means everyone that is using it, has the entire code history on their system (up until the last time they loaded the last changes): you get a local copy to work with. So in the case something happens with the system you labeled ‘server’: all developers will have a local copy as well. Getting back to a working environment can then be super easy.

Common offerings for Git (that I have worked with) are GitHub / GitLab / Azure DevOps. Each of these has on premises offerings (meaning you install them in your own network) or hosted versions where the hosting is handled for you. I’d recommend getting a hosted version when using it in production, since those companies can host it a lot better then you can yourself 😁.

Git: Getting started

Looking around online gives you a lot of options to get started with Git. You can find the tools and documentations on the Git Source Control Management website where they even host introduction video’s to get you started or use GitHub’s training environment to get started with a hosted solution for free (as long as your code is made available publicly).

After getting to know how you can work with Git and push changes to a central location, do make sure your team knows about using branches and start using feature branches to work in isolation.

Want to know more? Reach out to me with any question you have.

DevOps Maturity Levels

2020-12-31T00:00:00+00:00

I was thinking about the teams I’ve been helping out in my professional work life and suddenly noticed that there seem to be different stages that each team goes through in an effort to improve something in their day-to-day work. Usually I start at a new assignment with a specific question and together we evolve my assignment from there.

The entry point often is something specific, like for example:

Migrate our on premise source control to something in the cloud (usually much newer)
Quality assessments of the code base
Way of working to get changes into production

After the initial question is handled (or planned to be handled), I often start helping the teams with additional steps they can include to improve certain aspects of their way of working and we go from there.

Since my expertise is DevOps, all these things are around those topics. I’ve done a lot of things around the DevOps cycle, from explaining Git as a way to version your code to monitoring in production and deployment into a cloud environment. Note that these items are mostly based on using technology to improve things. The world of people and culture in an organization is for another time 😄.

DevOps - States of enlightenment

Looking back at my assignments, I found different states where the teams where in their DevOps way of working and from that you can find the next thing that would probably help to improve their environment. With environment I mean everything they do to keep the application running in production. Of course, some teams already have (part of) these topics handled, so you can check other states as well to find possible improvements.

I’ve tried to set these in a logical order in the image below and will (try to) create a blog post for each stage to describe what this means and ways to implement this in your setup. You can always dive deeper into a specific item no matter where your team is in the improvement process or skip things and come back later. In my mind, this is an order that makes sense to me:

I’ve color coded the states indicating their main point of view and if this is Developer driven (blue) or Operations driven (green), although this is not always as clear cut of course.

Dev - Source control in order

From a developer perspective you need to have your source control in order. I still come across code that is not version controlled at all 🙀, sometimes it just sits on a user machine, a share or even worse: on the shared server used for CI/CD (continuous integration and deployment)! Link 👉 DevOps - Source control in order.

Dev - CI (continuous integration) pipelines

From a developer driven wish they usually start implementing something of a continuous pipeline, that at least builds the code on a different machine then the developer who wrote that code. This state will be split up into different maturity levels since there are a lot of different improvements to be made. Link 👉 DevOps - Continuous Integration.

Ops - CD (continuous deployment / delivery) pipelines

Often times the people who are deploying the application to an environment have the need to automate these actions into a repeatable process that can be executed without manual interactions. I’ve still seen teams that where orchestrating this process with six designated team members, each having to wait for someone else was done to start the next step in the deployment by running their own batch file (non versioned of course!) with custom settings for that environment. This state also can be split up into multiple maturity levels. Link 👉 DevOps - Continuous Deployment

DevOps - Pre and post deployment gates with actual checks

This is the first common scenario where both dev and ops have a stake to get things done: operations usually has an interest in checking if things are still working after a deployment and this often is the last step for the developers when they thing their work is complete.

Ops - Beginning with monitoring

Heavily ops driven of course, although (finally) devs are more and more looking at the actual running application to see how things are going. To get more insights, monitoring is an important topic to handle. When devs start to tackle this, they often take a different starting point then someone with an operations background.

Ops - Alerting

When monitoring is setup, it would really help a lot if you also get alerts from your monitoring setup. Often you start here and you get too many alerts. Getting this right is hard, yet very important. From alerting you can also start with a good recovery mechanism, like adding playbooks/runbooks to start a recovery procedure (this could even be scaling things up/out).

Ops - Observability

Often triggered by something from monitoring (hence ops), the need arises to have actual insights into what the application is actually doing. Developers usually have the insight in where you can add additional logging in the application to get that information out, hence the color gradient in the image: it starts with ops (often) and then becomes a joint effort to get the information out.

DevOps - Feature flags

After having more information about the application through basic logging and observability, you can get on the way with proper feature flags: toggling a flag to enable/disable something in the application/environment to for example enable new features, or disable a recommendation engine on black Friday to have a more reliable application. This is often a joint effort between dev and ops minded people.

DevOps - Rapid A/B testing with user cohorts

When you have feature flags in your system, often you then have another light bulb go off: you can start using these to test in production! Why not give certain users a label like ‘internal’, ‘beta-tester’, ‘customer’ and have only those groups test new features? This enables canary releases (a subset of users get the new feature) or even actual A/B testing to see which application flow gets more customers to buy a product.

DevOps Maturity Levels - Continuous Integration

2020-12-31T00:00:00+00:00

If you have proper source control in place, you have a central location that holds the source code. From that location you can start with Continuous Integration as a next state of enlightenment in your DevOps way of working. Check that link for posts on the other topics.

Continuous Integration (CI)

With continuous integration we mean doing something with each incoming change to the central source code location that verifies that things still ‘work’ as intended. For an application, this could mean building the source code and verify that you have everything you need to build it. This prevents ‘it works on my machine’ that you hear in teams that don’t have a central build environment to do these checks. Finding out what is causing an issue between a deployment and your local developer environment can be hard that way.

By having a central location (or server) that starts doing things to verify code changes you prevent a lot of issues: you make sure all changes needed to create a ‘build’ are checked in and that the code is not just working because it could be build on a developers machine.

There are multiple aspects of Continuous Integration that can be viewed as different maturity levels for a teams way of working. For me, the logical process to add them is listed below:

Create a build on changes

The first thing to create when improving your DevOps way of working is to create a CI build: the idea of a build is to create an artefact: this can be a zip file that can be deployed onto a webserver for example.

By automating the steps to create the build you are already moving a way from the ‘works on my machine’ syndrome some developers have 😀. Setting the CI build up on a server designated specifically for builds will make sure that any changes made to the source code can actually be build.

At the end of the build, the results are stored on the build server (or uploaded to the cloud, or a release location) and labeled with a version number (usually the date with a incremental number for that date or by using date + time).

We call this process a build pipeline: a set of tasks that can be run to execute the steps to verify the build.

Only deploy build artifacts

The next step is to agree with you team that the only code that is being deployed is coming from the build server as artifacts! No more copy and pasting files from a developers machine. (yes, I’ve seen teams doing that, with all the consequences of not having reliable deployments since you could not find with what code that file was created).

Build for any change

Make sure you run the build on any change that comes in! You can have different builds: one for verification (the code can be build) and one for creating the actual artefact to be deployed.

If you are using feature branches, build the incoming changes on those branches as well as on the main branch of your code. Of course, you can setup the artefact creation to only run on the main branch, to prevent someone from deploying an artefact from a feature branch. Later on, you could create a new test environment based on a feature branch, to further verify the changes, but we are not there yet.

Automatic testing

When there is a build process in place, you can start adding extra tasks to it in an attempt to shift left (the process of finding bugs earlier in the CI/CD cycle). You want to have more and more confidence that your code is still working as expecting. To verify that, there are multiple ways of doing so.

Dependency checks

An important check to add to your CI pipeline is to check your dependencies for issues. Looking at a modern application, you learn that 99% of code isn’t yours. We are using dependencies everywhere because someone else has already solved a problem for us, and we don’t want to reinvent the wheel: we need to focus on adding value to our product.

The thing is: there are so many dependencies it is hard to keep them up to date. And why would you if they are working as they currently are? Well, sometimes issues are found with the version you are using. Something could have happened to the dependency where a vulnerability was found, or maybe even misused and someone injected additional software in the dependency.

This is where dependency scanning comes in. There are a lot of different options and offerings around. GitHub has Dependabot that scans your code for versions of the dependency and it’s known issues against the CVE database (Common Vulnerability and Exposures). There are container scanners that offer the same for your container dependencies. You can include WhiteSource, BlackDuck and other tools in your pipeline to at least check the dependencies for know issues.

Most of the offerings also include a license check against the dependency. Depending on the application you are working on, a license like the GPL might be a big (legal) issue in your organization to use. You really want to include a step in your CI pipeline that checks those items and fails the build when a dependency is used with a license you don’t want to support in your organization.

Dependency updates

After adding dependency scanning to your pipeline, a next step is to also update those dependencies regularly. Some teams have resolved to someone who runs all the updates every second Wednesday of the month. I like to wake up and see that an update was already checked for, a new branch for it created and already pushed to my repository, with a Pull Request to boot that already has ran all automatic checks to verify that:

the build still works
tests automation still works (so no regressions)
there is no known vulnerability in the dependencies The only thing left to do is approving the Pull Request, which some teams even do automatically: how easy can you make it on yourself?

Tools like NuKeeper already have it available for the hosted versions of GitLab, Azure DevOps, GitHub and BitBucket for the .NET (Core) framework. I’m working on getting the update part as automated as well and sharing that setup here, with for starters support for NPM and NuGet updates against a private GitLab environment.

Static Code Analysis

After automating your build another step to take is including Static Code Analysis. Tools like SonarQube, HPE Fortify or GitHub’s CodeQL are very well known and used products.

With Static Code Analysis you run an analyzer on your code to scan for known patterns. Most products have a set of patterns to use that can be extended when needed. The scan will find violations against the patterns and alert them to you. Most of them can then fail the build and additionally annotate a Pull Request for you, in an effort to prevent those violations to even become part of your code base (shift left).

Missing stages in Azure DevOps YAML Pipelines

2020-12-06T00:00:00+00:00

Alt. title: approval to an environment blocks the whole pipeline

Sometimes you find out about something and feel rather stupid. This one is one of the reasons YAML pipelines often feel like you need a magic incantation to get things working the correct way. Since this took me way to long to figure out, I’m writing about it here to hopefully safe someone else a lot of time (probably my future self 😁). I have bumped into these incantations before: using different CI/CD systems seems to make me forget them 🤔.

Photo by Dan Meyers on Unsplash

The premise

I was working on a multi-stage pipeline in Azure DevOps using YAML files. In the beginning I only had the Build part of my pipeline to build the solution. Wanting to deploy it, I added deployment phases to my setup. To get them working, I made the mistake to see them as specific jobs. [TLDR: the are not]:

jobs:
 - job: Build
   steps:
     - task: SomeTask

 - deployment: DeployTest
   displayName: Deploy to test
   environment: Test
   dependsOn: Build

 - deployment: DeployUAT
   displayName: Deploy to UAT
   environment: UAT
   dependsOn: DeployTest

You can see that I needed to add dependsOn to get the to work sequentially: not doing this would trigger all jobs at the same time (and subsequently fail the deploys, since the artifacts where not available).

Missing stages

Looking back, the first clue was that I didn’t get multiple stages in any overview page:

My confusion came from this screen and not making the connection. The different ‘stages’ of my pipeline where showing here, so why not in the overview page?

Finding a solution

Skipping the confusion part (I assumed I must have done something wrong or Azure DevOps changed something since I last checked), I continued with adding approvals to the environments my ‘stages’ where creating.

Environment approval blocks all jobs

Running the pipeline with an approval on one of the environments, blocked the whole pipeline! Not just the job targeting that specific environment, as I was expecting. Searching the web, I came across other people having sort of the same issue, like for example this uservoice. An almost two year old feature request to prevent this scenario from happening?!? Surely this is not happening in Azure DevOps?

Eventually it was this Stack Overflow answer that finally pushed me in the right direction…

Some more hints: Stage picker when starting a pipeline

I learned about a new feature that lets you pick the the stages to run when starting a pipeline run: This leads to this message, not making it that much more clear (oh hindsight):

The fix

This is where I felt stupid, shocked and reinforced the ‘magic incantation’ part of the YAML pipelines… The jobs relate to the same jobs in a classic pipeline and you need to wrap them in stages to have an actual multi-stage pipeline!

stages:

- stage: Build
  jobs:
  - job: Build
    steps:
    - task: SomeTask

- stage: DeployTest
  jobs:
  - deployment: DeployTest
    displayName: Deploy to test
    environment: Test

- stage: DeployUAT
  jobs:
  - deployment: DeployUAT
    displayName: Deploy to UAT
    environment: UAT

Now, the pipeline overview actually shows the correct number of stages:

And the approval works as planned:

So: wondering why an approval on an environment is blocking your whole pipeline? Or wondering why you only see one ‘stage’ indicator in the pipeline runs overview? Know you know how to fix it.

Request:

Found this page with mostly the same searching around I was doing? Please let me know! Finding out these posts help someone else really makes my day. That also makes me feel less stupid 😁.

Use GitHub Actions with a private runner to deploy to IIS

2020-11-24T00:00:00+00:00

Recently I got asked if you could use GitHub Actions to deploy to an IIS web application which of course I had to test :grin:.

TL;DR It runs the same as you would with a PowerShell script

Example

For testing this I used an example application in this repo (you can find the actions there as well). It’s based on the following dotnet command:

dotnet new webapp

Flow

Since this is a .NET Core application, the workflow for GitHub Actions has these steps:

Checkout the repo
Set the correct .NET Core version
dotnet build
dotnet publish
deploy to IIS
Run a smoketest
Run the webtests (added as an extra example, keep on reading for more info)

You can find that workflow here. 💡 If you want to see the workflow for pushing the application to an Azure App Service, check the dotnetcore.yml file next to it.

Setup

For running the IIS commands I’ve used the most simple example, other command line options will work as well:

Stop the website (or the entire webserver in this case)
Overwrite all files
Start the website again Using WebDeploy or a remote PowerShell session will work as well. Find more explanation on remoting in this blogpost as well.

Action

The actual actions that ‘deploy’ the application are as follows.

- name: Deploy to IIS
      run: |
        iisreset /stop
        Copy-Item ./dotnetcorewebapp/* C:/inetpub/wwwroot/dotnetcore-webapp -Recurse -Force
        iisreset /start

Note: running these steps requires Admin level access rights, so you’ll need to run the self-hosted runner with that access level. This stems from the AppExec commands that it fires that require that level of access (still an unfortunate thing).

Private GitHub Action Runner

To enable the deployment of the application on a Windows box, you’ll have to use a private GitHub action runner since the cloud hosted runners will not have access to that machine (they shouldn’t!). You can install them like a normal runner like for example Azure DevOps. Luckily the list of URL’s you need to add to your proxy/allow list is a lot shorter than the Azure DevOps list.

The runner runs on demand or as a Windows Service and will periodically open a long polling connection to GitHub, asking if there is work to do. The connection is always outgoing and on port 443.

Installing a runner can be done from a repository, team or organization level from the website. Go to “Settings” –> Actions and scroll down to Self-hosted runners:

Adding a runner is made very easy, all the steps are listed right in the screen, even including the temporary token it uses for a one time authentication process:

Bonus

The next question that came up was if you could run a Selenium WebTest (as I call that type of end-to-end test) with such a runner and if that would also work with a hosted runner. Long story short: it just works.

In both workflows I’ve added the last step ‘Run Web Test’ that runs the unit tests in the WebTest project that use a Selenium Driver to talk to the installed Chrome instance on the runner. You can find all the preinstalled software on the hosted runner here.

Azure Repos: Authenticate Git with a PAT

2020-11-08T00:00:00+00:00

Sometimes you have these weird things you run into, and I’m sure I will not be able to find this one if I don’t store it here.

Photo by Hakan Aldrin

Configuring Git with a PAT token with Azure DevOps

Usually in Windows I use the Windows Credential Manager for storing authentication against remote Git repositories. You can also use the SSH setup that Azure DevOps supports as a widely used alternative.

This time I was setting things up for a user with a Docker container and didn’t want to setup any of those options: I was already using a Personal Access Token for accessing the REST API and wanted to reuse that for the Git repository as well.

Searching around took quite a while until I found an obscure reference within the Git LFS repo that indicated you could setup Git with an extra authorization header with the PAT token in it. Seriously: can’t even find the repo I found this in.

Solution (repo based)

After some messing around I got things working, so here is the solution for future reference.

function SetupAuthentication {
    param (
        [string] $organization
        [string] $project
        [string] $repoName,
        [string] $userName,
        [string] $PAT
    )

    # convert the Personal Access Token to a Base64 encoded string
    $B64Pat = [Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes(":$PAT"))

    # store the extra header for git to use
    git config --global --add http.https://$userName@dev.azure.com/$organization/$project/_git/$repoName.extraHeader "AUTHORIZATION: Basic $B64Pat"
}

Note that this is specific for the repository you are using.

Solution (project based)

If you want to skip configuring this for every repo, you can also leave the repo name off this setting like the example below:

    # store the extra header for git to use
    git config --global --add http.https://$userName@dev.azure.com/$organization/$project/_git/.extraHeader "AUTHORIZATION: Basic $B64Pat"

Note that this is specific for the project you are using.

Solution (organization based)

If you want to skip configuring this for every repo, you can also leave the repo name off this setting like the example below:

    # store the extra header for git to use
    git config --global --add http.https://$userName@dev.azure.com/$organization/_git/.extraHeader "AUTHORIZATION: Basic $B64Pat"

Note that this is specific for the organization you are using.

Use NuKeeper 'manually' to update NuGet packages

2020-11-03T00:00:00+00:00

While building up a scheduled pipeline for updating our NuGet dependencies I found NuKeeper to help with automatically updating the packages we use. If you don’t know NuKeeper, it is an open source project that will go through your .NET solution and check with the configured NuGet sources to see if there are any packages that have available updates. If there are updates, it can update your project (or solution) and even generate a Pull Request for you.

With the DevOps mindset of handling these updates as technical debt, I wanted to run this check automatically each day, get any updates if possible and start our build pipeline to verify if it works.

Side note: it uses the normal NuGet flow to check for updates and uses that flow for authenticated/private package sources as well.

NuKeeper can handle multiple platforms as you can see below. Currently I was running this on GitLab, so the examples given here might need some small tweaks to push the Pull Request for a different platform. The manual check with NuKeeper will work regardless of the platform.

Flow

With NuKeeper the normal flow is usually just a one line command:

nukeeper repo <repo url> <token>

The issue

The setup I’m working with has an on-premises GitLab instance. Even thought NuKeepers documentation states that they support GitLab, it is only the cloud version hosted by GitLab.com. When you use the nukeeper repo commands, you’ll get an error that the url is not a known platform. After adding a parameter indicating it is a GitLab environment, it will tell you that it cannot execute against that url.

Solution

To still get everything working, I needed to set things up manually with this flow:

Check for updates on the project/solution
If there are no updates, stop the run
If there are updates, pull them in
Create a new branch
Commit and push the changes to origin
Create a Pull Request

Since we already have a continuous integration pipeline setup, the pull request automatically starts a pipeline that will build and test the solution, so that will make sure the incoming changes don’t break anything.

Checking with NuKeeper manually

I wanted to run NuKeeper and then find out if there are updates available. Currently this is not possible to do in one call: there are some output parameters available, but nothing that is easily usable out of the box.

I used the output to csv parameter, that at least returns all output as an array that I can loop through

    # get update info
    $updates = .\nukeeper inspect --outputformat csv

Then you can do a (most ugly) search through the results to see if there are any updates found by NuKeeper:

    # since the update info is in csv, we'll need to search
    $updatesFound = $false
    foreach ($row in $updates) {
        if ($row.IndexOf("possible updates") -gt -1) {
            Write-Host "Found updates row [$row]";
            if ($row.IndexOf("0 possible updates") -gt -1) {
                Write-Host "There are no updates"
            }
            else {
                Write-Host "There are updates"
                $updatesFound = $true
                break
            }
        }
    }

When updates where found, I can create the new branch and create a Pull Request by hand:

    $branchName = CreateNewBranch
    UpdatePackages
    CommitAndPushBranch -branchName $branchName
    CreateMergeRequest -branchName $branchName -branchPrefix $branchPrefix -gitLabProjectId $gitLabProjectId

In UpdatePackages I use the NuKeeper command to update the solution/project:

function UpdatePackages {
    .\nukeeper update
}

The full code is available in this GitHub Gist. I’ve split this up between the NuKeeper code (nukeeper.ps1) and the GitLab code (gitlab.ps1) for easier reuse.

Git clone error: Filename too long on Windows 10

2020-09-23T00:00:00+00:00

Today I ran into an issue that I tried to clone a Git repository with large filenames/folder paths in it.

fatal: cannot create directory at 'src/Modules/<long path here>': Filename too long
warning: Clone succeeded, but checkout failed.

The folder path display was only 195 characters long, but adding my root folder with 38 characters got awfully close to the known 260 characters limit in Windows.

Fixing the issue

To fix this you need to do two things:

Tell Windows to support long file paths
Tell Git to support long file paths

Configure Windows for long file paths:

You can do this either by updating the local Group Policy Setting through the Editor:

1. Windows Run --> gpedit.msc
2. Computer Configuration > Administrative Templates > System > Filesystem > Enable Win32 long paths

Or by using the registry editor:

1. Windows Run --> regedit
2. Path:  HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem
Key name: LongPathsEnabled
Value: 1

After updating this setting you either need to Sign Out and back on, or reboot the machine.

Configure Git for long file paths

Git doesn’t know about the changes in Windows and is installed by default without the LongPath setup we need. Enable it from the command line:

git config --system core.longpaths true

Now you can clone the repository again.

Fixing .NET Core tool update issues

2020-09-18T00:00:00+00:00

I had an issue running an EntityFramework command after updating the NuGet packages to the latest version and searching got me to multiple steps and sites to get things fixed. Grouping them here for future reference.

If it helps you as well, please let me know! Always nice to see these posts helping someone else as well.

The trigger message

This was the initial message that got me on this path:

The EF Core tools version '2.2.0-rtm-35687' is older than that of the runtime '3.1.8'. Update the tools for the latest features and bug fixes.

Photo by Clay Banks on Unsplash

Use the search Luke!

The number one skill you need as a developer / engineer / person who needs to do this, is asking for help. Learning how to use your google skills to find solution is one way to do it. Even after ~15 years of programming I do this all day long.

For this one, I had to use multiple sources to find a fix.

.NET Core Tools

A .NET Core tool is a special NuGet package that contains a console application and can be installed in several ways:

As a global tool
As a global tool in a custom location
As a local tool (from version 3.0 and up) In this case I am using the default, so a global tool.

Updating .NET Core tools.

Installing or updating any global .NET Core tool can be done with the same command:

> dotnet tool update --global dotnet-ef

Do note: you can use `donet tool install` as well, but the update command does the same thing and will install the tool if it is not installed yet.

Errors during the update:

Unfortunately running the update resulted in the following error. Seems like we are going down the rabbit hole again…

> dotnet tool update --global dotnet-ef
Tool 'dotnet-ef' has multiple versions installed and cannot be updated.

Fixing the errors

The .NET Core tools are installed in your user folder by default. There is no centralized storage of what versions are installed, the commands will be found if they can be found in this folder. So, you can look what it contains:

Each tool has its own folder and in that are folders for each version. Since there is no centralized storage (the tools folder can be seen as storage of course), deleting the folder for the tool can be used as a first step. You can try to remove a specific version in this folder, but I decided to delete it completely.

Note: I had a preview version in here, that might have been the cause of things.

del C:\Users\<USERNAME>\.dotnet\tools\.store\dotnet-ef

After deleting that folder and running the EntityFramework command again, you probably get the same error as I had:

Failed to create shell shim for tool 'dotnet-ef': Command 'dotnet-ef' conflicts with an existing command from another tool.
Tool 'dotnet-ef' failed to install.

That folder holds information about the version of the tool installed and not the console application itself. Turns out you also need to remove the executable itself. Makes sense when you find that out, but it was not that intuitive the first time.

So remove the executable:

del C:\Users\<USERNAME>\.dotnet\tools\dotnet-ef.exe

Now you can run the update again. You can specify a version if you want to or leave it off to get the latest.

> dotnet tool update --global dotnet-ef --version 3.1.8

ADAL error in Azure DevOps API interaction

2020-08-07T00:00:00+00:00

Today I encountered an issue while interacting with the Azure DevOps API. In the end this is not an issue with the API but with the user authentication and verification of tokens. Since it took me a while to figure out what was happening, I’m documenting it here.

Error message ‘Failed to obtain an access token of identity …, The refresh token has expired due to inactivity’

It matched the current project that I was working on: it had been a while since I used that project (and Azure DevOps organization for that matter) last: at least 5 months ago. The weird thing is, other calls to the REST API where working, using the same Personal Access Token (PAT) to setup projects, create repositories and other calls. The script that failed was adding new users from the AAD to our project, calling this endpoint with a body that holds the full username (username@domain.tld) with Invoke-RestMethod from PowerShell:

https://vssps.dev.azure.com/<<organization>>/_apis/graph/users?api-version=5.0-preview.1"`

Photo by Stephen Leonardi

Full error message

Microsoft.VisualStudio.Services.Aad.AadAccessSilentException:
GetUserAccessToken: Failed to obtain an access token of identity ea9d0e15-1e40-7030-b823-2612135e0b31. AAD returned silent failure.
Microsoft.IdentityModel.Clients.ActiveDirectory.AdalSilentTokenAcquisitionException:
Failed to acquire token silently as no token was found in the cache.

Call method AcquireToken
Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException: AADSTS700082: The refresh token has expired due to inactivity.
The token was issued on 2020-02-29T21:32:27.8019754Z and was inactive for 90.00:00:00.
Trace ID: << GUID >>
Correlation ID: << GUID >>
Timestamp: 2020-08-06 07:39:38Z
 Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException: Response status code does not indicate success: 400 (BadRequest)

The interesting parts are below:

Failed to obtain an access token of identity << ID >>. AAD returned silent failure
The refresh token has expired due to inactivity.
The token was issued on 2020-02-29T21:32:27.8019754Z and was inactive for 90.00:00:00.

The cause

From the error there seems to be something wrong with the user we are using for talking to the REST API, but what? Seems like that user hasn’t logged on in the last 90 days. It could not be my own account, because I’m using that to verify the projects are created correctly! Is it something with my PAT? I was able to find and add users from the AAD through the UI, so my account seemed fine. I even checked the link with the AAD to make sure nothing had happened with that connection.

Finding the user

I’m not sure how to find back the user from the PAT token or that error: searching around in the AAD and Azure DevOps (even using the users API) didn’t track back the specific user. If you happen to know a way to do this, please let me know! I’ve only found that specific Id in the network log of the ‘Manage User’ functionality in Azure DevOps:

Luckily it happened to be that I could be using the PAT from two users in this project: either my own, or the one from the service user we created for all the automation. I also happened to have logged the specific PAT Token in my local KeePass file that I use for these things. I’ve even learned to note down the creation or ‘valid until’ date for them, so I have something to cross reference in case of things like this!

The fix

After finding the user, the quick fix was easy: log in with that user!

The only long term fix that I can think of will be periodically logging in with that user to the Azure DevOps organization, just to keep that account alive.

This also helps with keeping scheduled builds for that user alive and prevent them from no longer starting. That is another common error people run into with this 90 day period.

Register a Startup or Shutdown script on Windows with PowerShell

2020-07-15T00:00:00+00:00

Today I got asked how you could register a Startup and/or Shutdown script on Windows through PowerShell. My colleague already had a setup for creating a VM, but wanted this extra step as well.

Searching the web revealed some bits and pieces, so I’m logging it here for future reference.

Photo by Daniel Öberg

Gist

I’ve created this gist with the registration script and an example file with a script to execute on Startup or Shutdown of the VM. It will then log the date/time to a text file for easy testing:

The trick is to set up (quite a lot of registry keys) and a windows folder that has to be present to get things working.

Tested

Tested on a Windows 10 VM (1909).

Caveat:

You need to run the register script with an elevated session because you need to have access to at least the %SYSTEMROOT% directory.

Missing wiki repo in Azure DevOps

2020-07-14T00:00:00+00:00

Today someone in the Azure DevOps Club slack asked a question about finding the repo from the default wiki in Azure DevOps. This used to be available if you knew what to do, so you could clone the repo and add pages programmatically for example. Weirdly enough, we couldn’t find how to get the repo to be visual so we could use it. In this case, the person asking the question wanted to add branch policies on the wiki repo so they can enforce Pull Requests on incoming changes. Of course, I was intrigued and started to search: this functionality was always there before, so surely this will still be available?

Photo by Matthew Henry)

TL;DR

Short version: it is not available anywhere, but you can ‘guess’ the correct URL and clone it: git clone https://dev.azure.com/<organization>/<project>/_git/<name of wiki>.wiki

Wiki types

In Azure DevOps there is a distinction between two ways to setup your wiki:

The default (old) way of creating a wiki. It is a Git repo under the covers
Publish one or more repositories as a wiki. You can find more documentation here

History

If you created a wiki a couple of years ago, you have the first wiki type. You could get the git URL to clone the repo and when you made changes to it, the repository would become visible on the Repos overview. This is no longer the case.

Currently when you create a new team project, you get the option to choose between the two types, although it is not very clear to see the difference between the two.

Note: I find the two types very confusing for the user and there is not a clear way to use them the same way. It would be better if a Project Admin could choose to include the repository in the normal overview or not. If the team using it is mature enough to use it as a normal repository, then why limit its use?

Current Wiki creation

When you create a new project and navigate to the wiki page, you are now greeted with this screen. Already confusing and the Learn more link tries to make the difference more clear, but doesn’t really make it clear it will always be a repo underneath.

I mean, if you need this large a matrix to try and make the differences clear, while under the covers it is the same setup, why not make your product easier to use?

Searching for the repo

Testing things out, I created a new team project [Demo] and created a new project wiki for it. Then I started searching for the wiki in the repos overview, but it only shows the default, empty project repository, not the wiki repo:

If you go to the wiki, it shows the name of the wiki:

Even the dropdown or extra button (the three dots) doesn’t some more information. You can find the git URL for cloning you need, but not how to get to the repository to set up branch policies for example…

REST API

Azure DevOps has an awesome REST API you can use to automate almost everything in Azure DevOps, so let’s see what it returns.

If you update the URL in your browser, you can test the API with normal GET request without setting up to much stuff. Go to https://dev.azure.com/raj-bos/Demo/_apis/git/repositories (so organization/project/_apis/git/repositories) and you get a list of repositories.

Note this does not include the wiki repo, only the default empty repo with the same name as the project.

Include Hidden repositories

If you include the query string includeHidden=True as can be found in the API docs, you see that the wiki repo is visible:

Conclusion: it is a repo, but a hidden one!

I’ve searched and tested some options, but I didn’t manage to update the repo and make it not hidden anymore.

Finding some really old posts and a GitHub issue that requested the hidden repo to be visible (that got redirected to a UserVoice request that was closed due to inactivity 😧), I figured that this old URL might still be working… And luckily it is!

The fix

If you check the name of your wiki repository, you can enter it in the URL of a normal repository (use the repo selection dropdown first for the correct URL to appear for easy changing): https://dev.azure.com/<organization>/<project>/_git/<name of wiki>.wiki

The most amazing part: the UI will now remember the last repo you have viewed, so if you use the menu to navigate to branches, it will enable you to set branch policies. You can add the repo to the URL here by hand as well of course, if need be.

Hopefully the Azure DevOps team improves on this omission (in my opinion) soon.

Summary

So, in conclusion: if you can’t find the wiki repo in Azure DevOps, you now have a way to get to it, even when the UI doesn’t give you an option for it.

Using EntityFramework Core tooling with .NET Standard

2020-04-23T00:00:00+00:00

I want to target .NET Standard so I can always use my libraries in any project later on, independently of its target framework (as long as it supports the .NET Standard version I’m targeting).

Photo by Hans Vivek

Today I had an issue with using the Entity Framework Core tools in a .NET Standard Library: the EF Core tools don’t support the .NET Standard framework: they can only target .NET Core or .NET Classic (Full framework).

This means that when you use a .NET Standard project to host your database setup in, you will get a nice error message when you run dotnet ef migrations add InitialCreate:

Startup project 'Provisioning.DataLibrary.csproj' targets framework '.NETStandard'. There is no runtime associated with this framework, and projects targeting it cannot be executed directly. To use the Entity Framework Core .NET Command-line Tools with this project, add an executable project targeting .NET Core or .NET Framework that references this project, and set it as the startup project using --startup-project; or, update this project to cross-target .NET Core or .NET Framework. For more information on using the EF Core Tools with .NET Standard projects, see https://go.microsoft.com/fwlink/?linkid=2034781

The documentation link does point you in the right direction, but it wasn’t as easy to find. Next time I run into this issue, I should be able to find the solution quicker 😄.

Do note: I happen to still have the version ‘2.2.0-rtm-35687’ of the EF Core tools installed, not sure how this behavior is with the newer versions.

Solution setup

The docs already indicate to create a dummy project with a dependency on the .NET Standard Library. What they don’t clearly explain, is that you then need to do some extra steps to get the EF Core tooling (like migrations) working.

I’ve setup my solution like above:

Provisioning.DataLibrary holds the DbContext with all of its models and targets .NET Standard 2.0 in this case.
Provisioning.ConsoleApp is the dummy project with a dependency on the DataLibrary and targets .NET Core 3.1.

You need to add the EF Core designer package to the dummy project for it to get all the commands you want to use:

cd Provisioning.ConsoleApp
dotnet add package Microsoft.EntityFrameworkCore.Design

Also note that I am using the appsettings.json to read the connection string to the database to use for EntityFramework, so I’ve included them to the console app: that is what EF Core will be using for running everything, so it needs to find those files as well (don’t forget to mark them as Copy always to actually get them in the correct bin folder.).

I usually open the Package Manager Console to execute actions like calling the EF Core Tools, just to stay in the same window. If you open it, you will start in the folder of the solution file. To use the EF Core tools, you’d normally change into the correct folder: in this case Provisioning.DataLibary ad then run the tools, like the migration actions:

cd Provisioning.DataLibrary
dotnet ef migrations add InitialCreate

This will give you the error above: Provisioning.DataLibrary is a .NET Standard library and the tools cannot analyze this.

Correct commands

To get the EF Core tools to work, you can stay in the main (solution) folder and indicate everything the tools need to find the correct references:

dotnet ef migrations add InitialCreate --project Provisioning.DataLibrary --startup-project Provisioning.ConsoleApp

So, with --project Provisioning.DataLibrary it knows where it needs to create the migrations + folders for it. And with --project Provisioning.ConsoleApp it can find a .NET Core project to target.

In the end it’s not that complicated, but certainly not intuitive.

Azure Active Directory Authentication Issue with .NET Core

2020-04-23T00:00:00+00:00

Today I faced an issue with Azure Active Directory authentication that was interesting enough to not this down for later reference 😁.

AADSTS50011: The reply URL specified in the request does not match the reply URLs configured for the application

I’ve got this issue in our (new) web application:

With the help from this blogpost from Antti I’ve learned that the url you’ve entered to redirect to after the authentication is done, has to match exactly with the URL you send in with the Authentication Request itself.

To verify your own setup, go to the App Registration Setup and find the URL you are using. I my case, we where using OpenIdConnect middleware that listens on a specific url for the callback that you can specify yourself (so you can match it with the App Registration). To make it clear where we are coming from, I’m using signin-microsoft.

OpenIdConnect .NET Core middleware

In the image below you can find the place where we configure this callback path (we load it from the configuration here). Do note that the middleware doesn’t want the root path here, so /signin-microsoft will do the trick in this case.

Azure DevOps: Update release variables across stages

2020-04-17T00:00:00+00:00

In Azure DevOps I needed to determine a variable in one deployment stage, and use it in another. I remember finding and implementing this solution before but couldn’t figure out how I did things, so this post is for me to find it easier next time 😉.

For example, in a stage I want to set a variable with the name ReleaseVariableName to a value. Searching online points you to an example on how to do this with for example the PowerShell command below. You first create a variable in the variable tab and then set/overwrite its value:

Write-Host "##vso[task.setvariable variable=ReleaseVariableName;]Value1.0"

Note that you don’t necessarily need to create variable, Azure DevOps does that for you. But it helps in figuring out what is happening later on.

Photo from Melissa Guzzetta

The issue

Testing the code above will prove that this works, but that the variable values are reset in a new Agent Job or another Stage. This stems from the fact that each job or stage can be run on a different Agent (and even in parallel) and that the values are not synced across.

The fix: use the REST API for Azure DevOps

The only way I found to update the variable value is to use the REST API for Azure DevOps, find the current release we’re in and then overwrite the variable value there. Then the next Stage / Job will pick up the new value and you can continue.

Do note that this updated value will not be available with this in the same stage as you’re updating it in! Handle that separately.

#region variables
$ReleaseVariableName = 'StageVar'
$releaseurl = ('{0}{1}/_apis/release/releases/{2}?api-version=5.0' -f $($env:SYSTEM_TEAMFOUNDATIONSERVERURI), $($env:SYSTEM_TEAMPROJECTID), $($env:RELEASE_RELEASEID)  )
#endregion

#region Get Release Definition
Write-Host "URL: $releaseurl"
$Release = Invoke-RestMethod -Uri $releaseurl -Headers @{
    Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
#endregion

#region Output current Release Pipeline
Write-Output ('Release Pipeline variables output: {0}' -f $($Release.variables | ConvertTo-Json -Depth 10))
#endregion

#region Update StageVar with new value
Write-Host "Updating release variable with name [$(ReleaseVariableName)] with new value [$(ReleaseVariableValue)]"
$release.variables.$(ReleaseVariableName).value = "$(ReleaseVariableValue)"
#endregion

#region update release pipeline
Write-Output ('Updating Release Definition')
$json = @($release) | ConvertTo-Json -Depth 99
Invoke-RestMethod -Uri $releaseurl -Method Put -Body $json -ContentType "application/json" -Headers @{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN" }
#endregion

#region Get updated Release Definition
Write-Output ('Get updated Release Definition')
Write-Host "URL: $releaseurl"
$Release = Invoke-RestMethod -Uri $releaseurl -Headers @{
    Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
#endregion

#region Output Updated Release Pipeline
Write-Output ('Updated Release Pipeline variables output: {0}' -f $($Release.variables | ConvertTo-Json -Depth 10))
#endregion

Note: you will need to set the Job you are running this in to have access to the OAuth Access Token:

Even easier, download the Task Group definition

Making implementing this even easier, you can download my exported Task Group here and import it (after reviewing it for security issues of course!) into your own environment.

Authorization for the Service account you are using

Good to note that you need Manage Releases with the service you are running the deployment pipeline with, otherwise you will run into an error like this:

VS402904: Access denied: User Project Collection Build Service (AzDoServiceAccountName) does not have manage releases permission. Contact your  release manager.

Future Azure DevOps update

There is a new update for Azure DevOps on its way to make this even easier as noted by the Azure DevOps team here. You can see that the initial issues was created in 2017 and the solution is rolling out in 2020 😄.

Update for yaml pipelines

After reading this blog post, Sebastian Schütze knew about another way to fix this issue in a yaml pipeline: you have the option there to upload an artefact from the pipeline that can be downloaded in any subsequent stage/job.

Just created a new blog post: Variables Cross Stage in Azure DevOps with YAML #AzureDevOps #DevOps #AzurePipelines #AzureDevOps #Cross-Stage #Variables #YAML AzureDevOps AzureDevOps https://t.co/CI94l1K8yG
— Sebastian Schütze 🚀☁️ (@RazorSPoint) April 18, 2020

You can read his post here: www.razorspoint.com.

Recreation in Classic Pipeline

I wanted to check to see if I could replicate the behavior in a classic pipeline and it all seemed good: there is a Publish Pipeline Artifact task available that is meant just for cases like this.

docs

You can then retrieve the file in the next stage/job and read it back in…. Or so was the plan:

The Upload Artifact task cannot be run in a release pipeline! 😠💩 It has been added to the documentation, but why they then show the task as being available and all, is beyond me. There have been more people who want this to work, as you can find in this GitHub issue.

There is an option to upload a file to the release pipeline, but then you cannot download it again:

Write-Host "##vso[task.uploadfile]$($file.FullName)"

Note: you can then download this file with the logs for the release pipeline.

Install an Azure DevOps Agent behind a proxy

2020-04-16T00:00:00+00:00

Sometimes you need to run the Azure DevOps Agent behind a proxy. If you search around you can find a lot of posts regarding this, and I wanted to have my own overview of all the things you need to keep in mind. At least I’ve tested this list myself 😁.

To run the Azure DevOps agent behind a proxy, the proxy must be updated with the url’s below in the allow-list. The origination of this list comes from Microsoft and Jesse Houwing.

Proxy allow-list

To be able to install the agent we have requested these url’s to be added to the allow-list:

Organization specific:

Replace ORGANIZATIONNAME with the name of your organization of course.

https://ORGANIZATIONNAME.visualstudio.com
https://ORGANIZATIONNAME.vsrm.visualstudio.com
https://ORGANIZATIONNAME.pkgs.visualstudio.com
https://ORGANIZATIONNAME.vssps.visualstudio.com
http://sso.ORGANIZATIONNAME.com
https://sso.ORGANIZATIONNAME.com
https://login.ORGANIZATIONNAME.com

Generic

https://login.microsoftonline.com
https://app.vssps.visualstudio.com
https://login.live.com
https://auth.gfx.ms
https://app.vsspsext.visualstudio.com
https://*.ods.opinsights.azure.com
https://*.oms.opinsights.azure.com
https://ods.systemcenteradvisor.com
https://secure.aadcdn.microsoftonline-p.com
https://*.dev.azure.com
https://dev.azure.com
https://login.microsoftonline.com
https://management.core.windows.net
https://api.nuget.org
https://login.microsoftonline.com
https://microsoftonline.com
https://go.microsoft.com
https://microsoft.com
https://app.vssps.dev.azure.com
https://dev.azure.com
https://aex.dev.azure.com
https://vstsagentpackage.azureedge.net
https://graph.windows.net
https://aadcdn.msauth.net
https://aadcdn.msftauth.net
https://windows.net
https://visualstudio.com
https://live.com
https://app.vssps.visualstudio.com
https://cdn.vsassets.io
https://gallerycdn.vsassets.io
https://static2.sharepointonline.com
https://spsprodweu2.vssps.visualstudio.com
https://vssps.dev.azure.com
https://vsrm.dev.azure.com

Start the installation by making it aware of the proxy:

The installation will then do the rest.

.\config.cmd --proxyurl "https://fqdn.url.of.your.proxy:3128"

If there are any errors, it will be logged in the _diag folder.

.NET Core: logging the configured URLs during starting of the application

2020-04-16T00:00:00+00:00

Sometimes you want to log information during the startup of the web application. In our case we wanted to log some generic information about the server to see where we are in the process. Doing so proved a little more complicated than expected, so I needed to document this in case I need this in the future 😉.

Usually you can only find out the incoming URLs if you have a request you are handling, but during the startup phase in .NET core this is not possible. In ASP.NET you could do this with this call, but this is not available in .NET Core.

HttpRuntime.AppDomainAppVirtualPath

The solution

Below is the full setup we started using. We are using Serilog for structured logging and sending it to an ElasticSearch endpoint. Key part is of course in the call to the ServerFeatures: host.ServerFeatures.Get<IServerAddressesFeature>();

Full code example

Note: this code was tested with .NET Core 2.2.

using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Hosting.Server.Features;

public static void Main(string[] args)
{
    Log.Logger = new LoggerConfiguration()
        .ReadFrom.Configuration(ConfigurationService.Configuration)
        .Enrich.WithElasticApmCorrelationInfo()
        .CreateLogger();

    try
    {
        Log.Information(
            "Starting {assembly} on {environment}",
            Assembly.GetCallingAssembly().GetName().Name,
            Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT")
        );

        var host = CreateWebHostBuilder(args).Build();
        var webAddresses = host.ServerFeatures.Get<IServerAddressesFeature>();
        foreach (var webAddress in webAddresses.Addresses)
        {
            Log.Information("WebAddress: {webAddress}", webAddress);
        }

        host.Run();
    }
    catch (Exception ex)
    {
        Log.Fatal(ex, "Application start-up failed");
    }
    finally
    {
        Log.CloseAndFlush();
    }
}

Nesting .NET Core appsettings.json

2020-04-05T00:00:00+00:00

I was working on a new .NET Core Unit/Integration Test project in a solution using Visual Studio and need to load some setting from the configuration. Naturally I wanted to use the same setup for retrieving those settings as in the real project, so I added a new file appsettings.json. Next up I wanted to add appsettings.Development.json just like we use in normal projects. Somehow I expected it to be nested beneath appsettings.json, like in the normal project. Of course, it didn’t 😄. While searching for a solution I noticed a lot of screenshots with the same issue: the files where not nested. Here is how to fix it.

The solution

The solution is updating your csproj file with a (new) item group that indicates the behavior you want:

 <ItemGroup>
    <None Update="appsettings.Development.json">
      <DependentUpon>appsettings.json</DependentUpon>
    </None>
  </ItemGroup>

With this you tell Visual Studio how you want this file to be displayed!

Visual Studio Code

Note: unfortunately the solution above will not work for Visual Studio code! For adding the same setup in Visual Studio Code you need to add the following to your .vscode/settings.json file:

"explorer.fileNesting.enabled": true,
    "explorer.fileNesting.patterns": {
        "appsettings.json": "appSettings.*.json"
    },

Deploy local IIS application on Windows with Azure DevOps

2020-03-29T00:00:00+00:00

Sometimes you need to deploy an application on a machine, but there is no option to use PowerShell remoting from the outside. In that case, you can deploy an Azure DevOps agent on that machine and use that for local deployments.

These are the steps to deploy an application with Azure DevOps on the localhost of the agent. As an example I’m using IIS to deploy a web application to for this.

Alex Holyoake

Local Windows User

First, create a local windows user to run the agent with. We need to give the account some extra rights later on, and we should not do that with the default service account the Azure DevOps Agent uses.

We need a Windows user for the agent to run with
Make a local service user to run the agent with
Make sure it has run as service rights (use the Group Policy Editor with gpoedit.msc in the ‘Run’ command)
Store the password for the user somewhere safe.

IIS deployments specific

The user needs to be added to the local Administrator group to be able to execute AppExec commands, used for administrative tasks in IIS, like creating a website. I’ve checked, but couldn’t find better ways to do this with a less privileged account.

Windows Remote Management (WinRM)

The default deploy task in Azure DevOps use PowerShell with remote management to do the administrative tasks through AppCmd. You can enable to use this from a remote host, but you can also use this on the local host!

Check locally trusted hosts:

Check the existing trusted hosts to see if the local host is already in the list: Get-Item WSMan:\localhost\Client\TrustedHosts By default it is not added to the list.

Set them open for localhost and a specific DNS name

If you don’t find the localhost in the trusted list, you’ll need to add them:

Set-Item WSMan:\localhost\Client\TrustedHosts -Value 127.0.0.1
Set-Item WSMan:\localhost\Client\TrustedHosts -Value servername.FQDNdomain.com

I add them with both the local IP-address and the fully qualified domain name so it will work with both, so if we later do get an option to use it from the outside, it will still work.

Test WinRM sessions

You can then test the connectivity with the created user with a PowerShell script (run it on the VM).

Test the session

Get credential from a windows popup and set up a session:

$USER = Get-Credential
Enter-PSSession -ComputerName 127.0.0.1 -Credential $USER

Or you can skip the popup for easier testing, remove the password from the file when done!

$Username = '\AzurePipelinesSVC'
$Password = '1234512345'
$pass = ConvertTo-SecureString -AsPlainText $Password -Force

$SecureString = $pass

And create a session with the user and credentials you’ve stored:

$USER = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $Username, $SecureString

# Entering the session, after that you can for example use $env:computername to see where that session is opened.
Enter-PSSession -ComputerName 127.0.0.1 -Credential $USER

Oops, you've worked on the master branch and cannot push to the remote

2020-03-26T00:00:00+00:00

We’ve all been there, what happens when you commit changes to the master branch and during the sync to the remote you get an error.

Starting point: you didn’t check the branch you where committing to: So during the sync you get an error:

In the output window this message is shown:

Pushing to https://**********.visualstudio.com/DefaultCollection/CICD/_git/DemoRepo
To https://**********.visualstudio.com/DefaultCollection/CICD/_git/DemoRepo
Error: failed to push some refs to 'https://**********.visualstudio.com/DefaultCollection/CICD/_git/DemoRepo'
Error encountered while pushing to the remote repository: rejected master -> master (TF402455: Pushes to this branch are not permitted; you must use a pull request to update this branch.)
This means the remote branch is protected and that you can only update it via a Pull Request.

You changes will not be lost! Here’s what to do: Step 1. Move your changes to a new branch, by creating a new branch from here: (note the master branch as the source)!

Now you can push that branch to the remote and create a Pull Request with you changes.

Step 2. If you switch back to the master branch, you’ll still see those commits on it and you still cannot push…

We need to reset the commit from the master branch. There is no way to do this with Visual Studio, so we need to use the command line for that.

Open a command line and go to your repository folder. I’m using Windows Terminal here.

Run git status in the folder to check if you are in the correct location and state:

To reset the last N commits, use this command: git reset --hard HEAD^N

As an example: to reset the last 2 commits: git reset --hard HEAD^2

Be aware: this is a destructive operation: data from the master branch in those commits will be lost! That’s why we saved them to another branch.

Azure SQL Database Firewall

2020-02-12T00:00:00+00:00

Did you know you have more than one option to set the SQL firewall settings on an Azure SQL Database? Most people know you can set firewall rules on the server level:

Example settings

Do note that toggling the setting for `Allow Azure services and resources to access this server` opens up connection from anywhere inside of the Azure Cloud: this is not limited to your own subscription!

Firewall settings on the database level

There is one extra option, which is very helpful if you want more control per database, instead of opening the firewall for all databases on the same server.

You can set the firewall rules on the database level as well! These will be checked after the server level settings, so you can mix and match.

Do note that the server level settings are evaluated after the database level settings have been checked. So if you have deleted an IP-address from the database level but it is still active on the server level, the IP-address can still access the database. See the docs here for more information.

There is no editor available, you’ll need to edit the settings in the database itself, for example with the Query editor or through a query or table editor with the SQL Server Management Studio or Azure Data Studio

You can use the table sys.database_firewall_rules to find the current settings and then use the stored procedures sp_set_database_firewall_rule to set a rule or sp_delete_database_firewall_rule to delete them.

Opening up the connection for Azure Cloud resources

To open the connection from other Azure resources (note: not limited to your subscriptions), you can toggle the setting for IP-address 0.0.0.0.

Other options

You can set the server level firewall rules with the Azure CLI, with a login on the master database or through PowerShell. For database level, your only option is the SQL statements.

Using CD with Application Insights Status Monitor on IIS

2020-01-27T00:00:00+00:00

Recently I found an IIS hosted web application that we couldn’t instrument with Application Insights. As it is running in IIS, it is possible to start monitoring it with Application Insights through the Web platform Installer. You can find how to do that for example here.

Installation and configuration is rather straightforward.

Photo by Dominik Schröder

What this post is about, is the files you need to take care of when you are deploying your web application with a Continuous Deployment pipeline, as we where doing. In that case you’ll find, that the connection with the Application Insights instance keeps getting lost.

Figuring out what was happening took some time and meticulously checking before and after of directories and file contents.

Solution for the disconnect

After taking care of the default settings in the ApplicationInsights.config (instrumentation key) and web.config (handlers), we found out that we needed to include the binaries in the bin folder that the Status Monitor sneakily added to our application directory:

After adding these files (for example in an internal NuGet package) and including them during our deployment, the Status Monitor now stays connected and sending telemetry to you Application Insights instance.

Azure Repos Policy Settings Issue

2020-01-21T00:00:00+00:00

I ran into an issue where I could not change Azure Repositories Policy settings, even while my account was in the the Project Administrator group. Somehow I could set the policy, but not change it later on! This posts explains how I got here and how I finally found out why this happened. In the end it was a simple fix, only rather difficult to find the reason behind this 😄.

Adding an email validation policy

It all started at a new organization where I wanted to make sure that everyone was pushing changes into the repositories connected with their work e-mail address. To do that, you can set up a ‘Commit author email validation’ policy on each repository. That can be a lot of work if you have a lot of repo’s 😀. Fortunately for us, a feature has been added to set those policies on the project level.

Side note: you can even set those policies on the company level, but only with the api.

Side note: you can have multiple patterns here as well, just split them with a semicolon in between.

You can see here, that I have set a policy on the Git repositories level, easy as that. From now on, everyone that tries to push something into any of these repositories with an email that doesn’t match this policy, will get an error telling them to fix the setting.

Side note: find out how to setup the commit email address from the Git documentation.

Policy Inheritance

The tricky part for me was that these policies are now inherited by all the repositories. It’s not that clear in the UI what happens with the inheritance: are the settings from the top level extended with the settings on the lower level? I’ve seen strange things happen here. I’ll come back to that later.

You can see how the inheritance looks here: Note the highlighted area and the text.

Blocking Policy Editing

Here is the catch that took me some time to figure out. I wanted to prevent team members to change any of these policies, so that they cannot circumvent them by disabling them. (yes, there is a note to make about trusting your team to do the right thing 😄).

So I created a policy that blocked members of the Contributor group from changing the policies. To do that, I set the rights for that group to Deny.

Result on the organization level

If you go back to the e-mail policy on the top level, you can see that the message here is still the same: the setting is inherited from the project level. Except for that it is now disabled!

Cause of the issue

In hindsight, it is clear that this is because my account is also part of the Contributors group. Deny settings have a higher priority then Allow settings! Fixing it was changing the Deny setting to Not set: that means that the contributors cannot change the settings, but this setting is not as explicit and people with the correct rights can still update policies.

Policy Inheritance questions

As noted before, it’s not clear if the policies are overwritten on the lower level if you set additional data on it. This is not clear from the documentation

For example, you can add an extra domain to use validation. It will get stored and stays in the input if you refresh. If you remove the initial domain but add a different one, then only that new domain is stored. Is the original rule now active? I need to test this out 😄.

If you remove all text from the policy, it will show that it stored an empty value. On refresh, the policy from the top-level has been filled in again!

I suspect that the entries at the lower level is additive and the top level value always gets verified.

Update Azure CLI Extensions

2020-01-09T00:00:00+00:00

After getting a message that a command I wanted to use was not available in my local installation, I needed to update an Azure CLI extension. Finding that information was scattered around the internet, and took me to long to find. So for future reference, I’ll document it here 😄. Should be way easier to find next time 🙈.

If you know the name of the extension, you can find it version by using the show command:

az extension show --name azure-devops --output table

In the result you can find out the version that you have installed:

If you don’t know the name of the extension you can list ALL extensions and try to find it. For example I didn’t know the name of the Azure DevOps extension: the command always is az devops ***, but the extension name is azure-devops. I understand why they kept the command name short, but you need to know how to look and where to figure out what is going on.

 az extension list-available --output table

You can then update the extension with the following command.

 az extension update --name azure-devops

Do note, that the command will tell you nothing that displays if the update was successful or not! Apparently the decision was made to not show you any feedback during default execution. If you want more information, you can add --verbose to the end of the command.

Updating the Azure CLI itself

Do note, that you cannot update the CLI itself by using any commands for it. You need to download and install the latest version from Microsoft.

Deploy .NET Core web application using GitHub Actions

2019-10-26T00:00:00+00:00

I wanted to try out using GitHub Actions to deploy a .NET Core web application to Azure. Microsoft already has some actions available to accomplish this, so this should be rather straightforward 😄. I haven’t really played with GitHub Actions yet, so this should be rather informative 😁. Usually I do this using Azure DevOps, so this will be a nice way to check the other side of the fence.

Photo by Emre Karataş

Starting off

For starters, I want to have a small web application that would be similar to something I could already have running in a regular Azure App Service with a CI/CD pipeline in Azure DevOps. To create something I used these steps:

Create a new repository on GitHub link.
Clone it to a local folder on my laptop
Move into the new directory
Run dotnet new webapp
Update the .gitignore file
Commit the first version of the code
Push into the remote on GitHub

For testing how the new application worked and if it could be run I used these commands:

dotnet restore
dotnet build
dotnet run

GitHub Actions

To get started with GitHub Actions (do scroll on that site, or you will miss a lot of good stuff!) I had to join the Beta program and from then on I had an ‘Actions’ tab available on all my repositories. Note: the beta for GitHub Actions ends somewhere in November 2019.

Clicking on ‘Actions’ on a repository on GitHub analyzes your repo and proposes a starter yaml file to use, based on the code they found in the repo. On mine, they correctly found some .NET code:

There are a lot of actions available already, as are other setup possibilities to get you started, regardless of your stack:

The yaml file that you create is a set of actions to run after each other (some parallelization is possible) and is called a workflow.

.NET Core set up

I picked the suggested setup since it matched the .NET Core project I wanted to deploy into Azure. You can see that there are steps to set up the Action runner with the tooling it needs to build a .NET Core project and then we start a build with the dotnet commands.

After saving the yaml file it will be committed to your repository and this will kick off a run, since it the trigger for this ‘workflow’ is on: push. You can see the commit ID that is associated with the push, as well as the branch and the account that pushed the changes:

First run

The first run I start reported an error. See the screenshot below for the message detail. This is already a good start: it clearly states what is wrong, as well as where! I am running the latest [].NET Core 3.0](https://dotnet.microsoft.com/) bits on my machine and the default yaml template uses the 2.2 version! I naively changed that to use 3.0, but that run failed as well: you need to use the full SDK version, so 3.0.100 did the trick.

First working run result: nice and fast!

Also some things that stand out: the GitHub site doesn’t automatically show new runs on pushes and you cannot get back to the list of runs to switch to a previous run for example: you can click the ‘Actions’ tab again to start on the top-level of the ‘Workflows’ overview: it is possible to have multiple workflows in the same repository (handy!). The rest of the UI does look great though!

Publishing to Azure App Service

To be able to publish the web app to an Azure App Service I create a new App Service on an existing App Service Plan. I found some documentation on using GitHub actions from the App Service team, so I tried that out first. The examples are specific to running a docker container on the App Service, or a Java, Javascript or Python application. Surely, this will work for .NET Core as well?

To prepare your repo, download the publish profile for the App Service and store it in a GitHub Secret. Here is some strange UI as well: I used the secret name from the documentation I was following and it contained a dash between the words. This is not a valid secret name, so GitHub will show you an error. Adding the correct name (I changed it to an underscore) shows a green notification. This UI can be confusing: which message belongs to which values? Maybe the could add the secret name that you where trying or at least a timestamp to it?

Just adding this step in the yaml file ran the workflow again and indicated a warning: this space is moving so fast as it is just a couple of months old! The warning states that the action I am using is already deprecated: I need to use a different one!

##[warning]This action is moved to azure/webapps-deploy repository, update your workflows to use those actions instead.

To bad this warning doesn’t include a link to the correct repository. Luckily the name of the action is also its path after GitHub.com: https://github.com/azure/webapps-deploy. Scrolling through the repository shows even more example yaml files to use! For .NET Core the example can be found here.

Do note that this also adds a publish step: I forgot that one initially!

By now my mail application has caught up as well: on each failed run, GitHub sends you an email message to make sure you know something is not right 😄.

Do note you can change that behavior on your profiles settings, but not on a per project basis.

Complete workflow

By now I have the current workflow that is running as I want it to:

name: .NET Core

on: [push]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v1
    - name: Setup .NET Core
      uses: actions/setup-dotnet@v1
      with:
        dotnet-version: 3.0.100

    # dotnet build and publish
    - name: Build with dotnet
      run: dotnet build --configuration Release
    - name: dotnet publish
      run: |
        dotnet publish -c Release -o dotnetcorewebapp
    - name: 'Run Azure webapp deploy action using publish profile credentials'
      uses: azure/webapps-deploy@v1
      with:
        app-name: dotnetcorewebapp19 # Replace with your app name
        publish-profile: $ # Define secret variable in repository settings as per action documentation
        package: './dotnetcorewebapp'

And with that setup I have my first successful run!

You can now browse to the website and see if it is running:

To be really sure, the files now exist in the Kudu editor:

Summary

There you go, a complete example of setting up your first GitHub Actions Workflow for pushing a .NET Core application into an Azure App Service. The set of actions I used also has options for deploying to a deployment slot as well, witch makes it complete! If this is the first time you use yaml, this can look a bit daunting, but after reading what is happening, I figured everything out as well!

Next up is seeing if we can parameterize this yaml file and deploy to different web applications (for example staging) with the same template. That’s something for another time!

Deploy Windows Service using Azure DevOps

2019-10-23T00:00:00+00:00

Ever needed to deploy a Windows Service onto a machine with Azure DevOps? It turns out this is really easy (some caveat’s apply 😄, see section at the bottom)!

There is an extension on the Azure DevOps Marketplace that is a wrapper around the SC tool from Windows:

Add the extension and perform the tasks that you need:

In this case, I use these tasks:

delete the existing service (will fail on the first try when the service is not available!)
copy the files to the destination path
set up the parameters for that server
install the service on the remote server

Caveats

There are some small caveats to get this to work:

WinRM needs to be enabled on the target server.
This extension cannot use the standard service user accounts to run the service as. This means you are tied to local user accounts on the target server and cannot use Local System or Network Service.
Only works for X64 server and installed agent.

Parallelizing a long Stryker Run in Azure DevOps

2019-10-11T00:00:00+00:00

I’ve been working on a Stryker run for a larger .NET solution (115 projects and counting) and wanted to document on the final setup in Azure DevOps.

You can find more information on what Stryker is and how this can be used on a .NET project with an example on this previous blog post.

In this post you can find how I got to this point: link.

Photo by Dallas Reedy

Final Setup in Azure DevOps

You can find the repository with all the scripts on GitHub. I call these scripts in the Azure DevOps pipelines.

High level overview

See the screenshot below for the final setup. Jobs 1 & 2 will run in parallel since they aren’t linked to any other job as a dependency. This enables us to run multiple Stryker jobs (each with multiple projects!) at the same time. If there are enough eligible agents in the build pool, we can fan out this rather extensive task. Depending on the number of unit tests and the code that is being tested, mutation testing can easily take a while.

Task 3 does have a dependency on the other tasks, so it will run when task 1 and 2 are completed successfully.

Running Stryker on a set of projects

Job 1 and 2 have the same taskgroup that will run.

Note: read my previous post on the set up I use to run Stryker on a set of projects. This job runs Stryker on all projects in a given folder by creating a specific configuration file for that part of the solution. The next task then uploads all the json files back into Azure DevOps so they will be available for downloading in a later step. This is needed because each agent job will/can run on a different agent and therefore the json files will be generated in a different folder.

Because each job can run on a different agent, we need to build the solution on each agent to make sure that Stryker can run. I’ve asked the Stryker team why they need this method and to see if we can do that once instead of in each job here. Still, the tradeoff with the ability to run on different agents is worth it. If I can change this setup, I will update this post.

After building the solution I need to make sure that the agent has the Stryker tooling installed. I don’t like to do that on an agent by hand or by baking tools like this into an agent image. I’d rather have the tool installation available in the build itself. That enables us to add new agents to the pool when needed, without us having to do something to make sure all our tooling works. Checking for a .NET Core tools on a server can be done with the code in this GitHub Gist. Using PowerShell I then download the files from my GitHub repository that has all the code we need to run Stryker on multiple projects and join their results. Those results are then copied into the Artifact directory. That way I can pick them up and upload them to the Artifacts linked to this build.

I update the settingsfile here to correct the hard-coded paths in the configuration.json so the tools can find them in the agents source code directory.

Joining the Stryker results

This job will first download all the artifacts from the other jobs so we have them available. By using the same code in my GitHub repository, I can now join those json files and create a new report from it.

As a final step I upload the generated HTML file (self contained btw, very nice) to the artifacts for the build so they can be downloaded and analyzed.

Todo

What I haven’t done yet, is failing the build on a low mutation score. I’m not sure what is helpful here: I could store the results from each run and use the lowest score to verify it against a threshold, or I could try to calculate an overall score from the results files. Unfortunately that information is not stored in the results json, so that is currently not possible. Still, having a (regular) checkup on you unit tests is already a nice improvement to have!

SonarQube analysis on a Java project - fixing error 'Project was never analyzed'

2019-10-07T00:00:00+00:00

Today I was configuring a SonarQube Analysis in Azure DevOps on a Java project. Following the documentation I still got this error:

[ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.7.0.1746:sonar (default-cli) on project 'prefix-project' Project was never analyzed. A regular analysis is required before a branch analysis -> [Help 1]

Given the error message says it is an error for the project prefix-project I guessed that the plugin wanted to link everything to a project with that key in SonarQube. Since I was creating this in a different Project Team, there might be an issue with the new service connection (those are tied to the team project). So to test, I created a project in SonarQube with the same key.

Running another build resulted in the same error. Running a build on the master branch, just to make sure it wasn’t related to a different branch and I need to run it first on on the master branch: same error.

In the pom file I noticed that there was an extra groupId added:

 <groupId>com.organizationname</groupId>
 <artifactId>prefix-project</artifactId>

Fixed project key

And running the build in debug mode showed that it was using that combination as a project key for the SonarQube project. So, changed the project key in SonarQube to use that: com.organizationname:prefix-project

Tasks

Using that project key with the default tasks now works.

Prepare step before running with Maven:
Update Maven build tasks to push to SonarQube:

Running Stryker on a large .NET / core solution

2019-10-04T00:00:00+00:00

TL;DR

Stryker cannot run for an entire solution with multiple test projects (YET), so we need to help it a little and run each project by itself and then join the results. Finding a way to do so started by checking in with the Stryker team on GitHub. I ❤️ OSS! They are working at making this part easier, so I checked out their code to see if I could help with that. That proved to be rather hard! There is a lot going on under the covers. Still want to help, but first I tried to get a quick fix that I can bring back to the team that wants to run Stryker for their entire solution file, with 112 projects in it (and only growing 😱). Yes, they should see what else needs fixing 😁.

Using Stryker in .NET code

I previously posted about using Stryker for .NET in your Azure Pipeline (find it here). After running it for one project, I now want to run it for a lot of projects, in the same solution. Currently, this is not available in the Stryker tooling. After reaching out to Stryker team, I had confirmation that the way I wanted to do things, seemed like the correct way to go:

Run Stryker on each project
Gather the output files from each run
Join all files together
Run the full report with the new file

I created all code to do this in a PowerShell file and have shared that on GitHub. All information on how to do things is in de readme of that repository.

Files

I’ve used these file to set things up:

Run Stryker.ps1
Stryker.data.json
StrykerReportEmpty.html

The PowerShell file is the entry point of course. It will use the json file for configuration so we can push in multiple project files to mutate. All the generated json will be copied over to a new output folder and will then be joined together, From that, I can paste it into the HTML file and store that as a new result file. That gives us the final html file that is self contained.

Next step

The next step is to update the code in the repo so I can do things like running multiple projects in multiple runs through this script (for example in Azure DevOps, where this can be parallelized), and join the resulting json files back together with an extra call at the end. That means I need to make the current code a little bit more modularized 😄.

For now, this is already a working solution, so: Happy mutating!

Update:

For the final setup with all steps running in parallel in Azure DevOps you can find the end result here.

Installing .NET Core tools: Preventing errors in your Azure Pipelines

2019-09-25T00:00:00+00:00

I ran into a weird thing in .NET Core Global tools: If you try to install the tools while they are already installed on that system, .NET Core will throw an error and exit with a non-zero exit code.

This is not helpful in a Continuous Integration (CI) scenario!

Expected installation

Normally you expect that you run an install command like this:

dotnet tool install dotnet-stryker -g

And the tool would install. This is correct if you run it once.

In Azure DevOps you would call the command from a .NET Core task, as I previously described here.

If you run the pipeline again, or the command locally, you get an error indicating that the tool is already installed!

This behavior seems rather odd to me: if the tool is already installed, then great. Perform a no-op (no operation) and go ahead with the next command.

In this issue on GitHub this behavior is described. The team has made a choice about doing it this way, even though they are not sure that this was the right choice. The current debate seems to be if the decision can be reverted (that would mean a breaking change from the old behavior) or adding a new parameter that would enforce the expected behavior.

Workaround

The work around for this issue is almost as weird as the issue it self. You run the update command for the tool:

dotnet tool update dotnet-stryker -g

Outcome:

if the tool is not installed, it will install the latest version!

Seems strange behavior to me, but ok… At least it will return an exit code of 0 and we can move along.

If you run the command again, the tool will try to upgrade to the latest version, which is what I would expect. The logs do seem to indicate an interesting story… It seems to reinstall that same version (given there is no newer version available)…!

Strange way of implementing this functionality, at least there is a work around to prevent the CI build from failing.

Update for multiple parallel executions

I needed to run a dotnet tool parallel on multiple machines in the same pipeline and then this method does not work. Seemed like a conflict when running the update command during the same time window.

To circumvent this, I had to create a small PowerShell script to check the output of the dotnet tool list -g command and first manually check if the tool was installed. You can find the code for it in this Gist.

Note: Not sure why the last exitcode is not 0 when running the regular .NET Core command in an Azure DevOps pipeline, so I had to enforce a normal exit code.

Use Stryker for .NET code in Azure DevOps

2019-09-04T00:00:00+00:00

Recently I was at a customer where they where testing running test mutation with Stryker. Mutation testing is a concept where you change the code in your System Under Test (SUT) to see if your unit test would actually fail. If they don’t, your unit tests aren’t specific enough for the SUT and should be re-evaluated. Since Stryker changes your code, they call it mutations and they check if they survive with the unit tests or not. Nice play on words there 😄.

Of course this triggered me to see how this works with .NET code and if we can integrate this in Azure DevOps!

Photo by Suzanne D. Williams

Setting up an example

To have a SUT and a set of unit tests for this, I have set up a small C# library with .NET Core and some unit tests to run. I created a solution that only contains what we need:

To have something to unit test I added a simple class that would be instantiated with a string and we load that as a boolean value into a property of the class.

To check if my class setup works, I created two unit tests as an example for both the parameter values ‘True’ and ‘False’. You can probably spot the first issue here: I am using different casings in my unit tests, but they still pass the test runs.

I’ve already set up an Azure DevOps build to trigger on any pushes to the repo, with the default .NET Core template to restore the NuGet packages, run a Build and then run the Unit Tests:

The Azure DevOps build is green with the current set of tests:

Using Stryker for .NET can by done on the CLI by installing the dotnet tool with this command:

dotnet tool install -g dotnet-stryker

I’ve ran into some configuration issues and an older version of .NET Core that I have documented here.

When you run the tool from the CLI, you can see the results immediately and also see the changes Stryker has made to your code.

In the screenshot above you can see that by default, running Stryker in your solution folder will not work. Stryker wants to be run from the folder containing the UnitTests project and will pick it up automatically. It will also find the solution and the project containing the code it needs to mutate. If you need, you can help Stryker find all this by adding some parameters that have been documented here.

Running Stryker in the CLI

Running Stryker from the unit test project directory will start mutating your code and running the unit tests on it again. It will try out all the mutations it can find and then track if it survived all the unit tests (meaning that there was at least one unit test that failed when running against the mutation). The results are visible inline.

dotnet stryker --reporters "['cleartext', 'html']"

Mutations

As you can see in the screenshot above, Stryker searches the original code for boolean expressions, strings and other things it can ‘mutate’.

The first mutation in this run was changing the line if (isOpen == "true") into if (isOpen == "") (a string mutation). This mutation is caught by the first unit test and therefore marked as ‘killed’.

Stryker report

Adding a html report parameter to the Stryker command will write a html file to your disk that can be used for finding the mutations that either survived and where killed.

Summary view

Details view

Adding Stryker to your Azure DevOps pipeline

Now you are ready to include a Stryker run into your Azure DevOps build pipeline. To do so, you can include calls to the dotnet tool commands using the normal .NET Core task. If you need help figuring out how to set up the custom commands, read my blogpost about Running dotnet tools in Azure DevOps. Do note the specific arguments I pass into the Stryker command here: my mutation tests where scoring on 54%, so I needed custom thresholds to actually fail the build.

Failed build result

Running Stryker on my current set of tests will actually fail the build because of the custom threshold. This way you can validate your unit tests and actually check to see if there are any outliers that you missed while creating the tests for your code.

Note: mutating your code and running the unit tests again means that your tests will run multiple times. This can add up to quite some additional time that your build needs to run!

Next step

The next step is to include the html report in you build pipeline and upload it as an artifact. You can then download it if you need to check it.

The Stryker team seems to be working on an extension for Azure DevOps to enable the build results to show an extra tab that would open that artefact file, but it seems that this is not yet ready. Keep up to date on this by watching this repository.

Update

Read more information about the setup in Azure DevOps in my follow up post here.

For the final setup with all steps running in parallel in Azure DevOps you can find the end result here.

Fixing error in .NET Core tool installation

2019-09-03T00:00:00+00:00

Last week I was testing some .NET tooling and wanted to install a tool locally instead of globally. To do so you run this command:

dotnet tool install dotnet-stryker

While running (either locally or in an Azure DevOps task) I got this error message:

[command]"C:\Program Files\dotnet\dotnet.exe" tool install dotnet-stryker
Cannot find any manifests file. Searched:
C:\Apps\TFSAgent\_work\7\s\StrykerDemo.UnitTests\.config\dotnet-tools.json
C:\Apps\TFSAgent\_work\7\s\StrykerDemo.UnitTests\dotnet-tools.json
C:\Apps\TFSAgent\_work\7\s\.config\dotnet-tools.json
C:\Apps\TFSAgent\_work\7\s\dotnet-tools.json
C:\Apps\TFSAgent\_work\7\.config\dotnet-tools.json
C:\Apps\TFSAgent\_work\7\dotnet-tools.json
C:\Apps\TFSAgent\_work\.config\dotnet-tools.json
C:\Apps\TFSAgent\_work\dotnet-tools.json
C:\Apps\TFSAgent\.config\dotnet-tools.json
C:\Apps\TFSAgent\dotnet-tools.json
C:\Apps\.config\dotnet-tools.json
C:\Apps\dotnet-tools.json
C:\.config\dotnet-tools.json
C:\dotnet-tools.json

Searching around on the internet for the file it is searched (throughout the whole folder tree), I found that you need to run this command to create a local manifest:

dotnet new tool-manifest

Yet doing so resulted in the following error message and the default prompt to choose a template to run.

No templates matched the input template name

Apparently the command to generate the manifest wasn’t available on my machine. Further searching lead to this GitHub issue that pointed out this was recently added in .NET Core 3.0, so it seemed that it could be coming from an old preview version?

Checking the runtimes I had installed with dotnet --list-runtimes, pointed out that I was indeed running on on older version of the preview for .NET Core 3.0.

Microsoft.NETCore.App 2.2.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.0.0-preview-27113-06 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.0.0-preview-27114-01 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Downloading and installing the latest preview (3.0.0-preview8-28405-07 at the time of writing) fixed the issue and I could carry on with figuring out my other steps that I was actually working on 😄.

Running .NET Core tools in Azure DevOps

2019-09-03T00:00:00+00:00

I wanted to run .NET Core tools in Azure DevOps and ran into some installation issues. I tried installing the tool I needed globally, yet the agent could not find it.

Local tools to the rescue

In the latest versions of .NET Core 3.0 (currently still in preview), you can install tools locally. This means that you can install the tool in the current folder, with its own version and thus independent from other tools or versions on your machine. More information can be read here.

Calling the installation of the tool in Azure DevOps

To actually install the tool (locally) through the .NET Core tasks you need to run the command in a specific way. This took quite some testing to figure this out. I wish this was documented a little better, so here it is for myself in the future 😁:

Do note that the custom command to run is just tool and the parameter input gets the name of the action and the tool.

In this case I am installing Stryker to start with mutation testing.

Error: ‘Cannot find any manifests file’

Just running the installation in the work folder will give the error below. .NET Core wants a config file for local tools. Find more information about that here.

Cannot find any manifests file. Searched:
C:\Apps\TFSAgent\_work\7\s\.config\dotnet-tools.json
C:\Apps\TFSAgent\_work\7\s\dotnet-tools.json
C:\Apps\TFSAgent\_work\7\.config\dotnet-tools.json
C:\Apps\TFSAgent\_work\7\dotnet-tools.json

To get a new manifest, add an extra .NET Core task and run this custom command:

dotnet new tool-manifest

Run the .NET Core Tool

After setting up a manifest and the installation itself, you can now run the .NET Core tool itself by using a custom command again:

Lets Encrypt: Manually get a certificate on Windows for an Azure App Service

2019-08-27T00:00:00+00:00

Recently I had to refresh a Let’s Encrypt certificate for an Azure App Service after the first certificate had expired. Of course, refreshing a certificate should be done by some tooling, either in a CI/CD pipeline or another service. I tried setting up the Lets Encrypt Extension on the App Service, but could not get it to work. Eventually I even ran into the Let’s Encrypt rule that you can only try to get a certificate 5 times a week for a production environment, after which they blocked me. Therefor I decided to update the certificate by hand, because that should not be too hard at all…. Unfortunately this was not as straight forward as I wanted it, so I decided to document the process here for later referral when I run into this again. Hopefully I’ve remembered to have automated this the next time!

There will be steps in here that can be executed easier. If you have any tips, please let me know!

Photo by Johnny Chen

Let’s encrypt process

To be able to get a Let’s Encrypt certificate you first need to prove that the domain you are using is actually owned by you. Most identity challenges do this by requiring a specific txt-record in the root of the domain so they can request that from the DNS server. The more modern way to do this is by setting up a specific well known route on the webserver for this specific use. It seems that the industry standard is moving to the same setup. Let’s encrypt sends a request to a sub-url on the domain you are validating to \.well-known\acme-challenge\unique_file_name. It checks to see if a specific set of characters is in the file. If you can set that information up on the domain, it proves to Let’s Encrypt that you are the domain owner and they can generate a certificate for you.

Windows Subsystem for Linux!

To get that filename and its contents you can use the Certbot, that is available for a couple of different Linux distro’s. Since I have Ubuntu running on my Windows machine inside Windows Subsystem for Linux (WSL), I wanted to use that. I followed the installation steps from the documentation. After the installation I can now run the certbot with:

sudo certbot certonly --manual

The bot will then ask you a couple of questions, like the domain(s) you want to get the certificate for, your email address so they can e-mail you when the certificate is about to expire and if you are OK with logging your IP-address.

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/www.gdbc-challenges.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/www.gdbc-challenges.com/privkey.pem
   Your cert will expire on 2019-11-25. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Uploading the challenge files to an App Service

In the screenshot above you can see the name of the file and the contents Let’s Encrypt expects in the challenge. In the Windows Terminal I am using you can select the text with the left mouse button and then copy it to the clipboard with the right mouse button. Trying to do this with the keyboard will send a key to the shell and will be seen as a ‘return’ key-press! When this happens, Let’s Encrypt will try to validate the file and you can start over again! Luckily it will rotate the expected file and content, so after two or three tries you will be back at the initial values 😉.

Create a new file in the Azure App Service with the correct name through Kudu: You can only do this with the editor in Kudu, since the App Service Editor will only enable you to create or edit files within the /site/ folder. The acme-challenge lives outside of it.

Extracting the Let’s Encrypt files

After Let’s Encrypt validates the domain, the CertBot will write down a couple of files that you can use for the certificate. It will tell you that it wrote the files in the following location: etc\letsencrypt\live\your_domain_here. To get to those files from Windows, you need to find out where WSL saves its local files. The root the file store is %userprofile%\AppData\Local\Packages. As I am using Ubuntu, the folder for that subsystem is CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc from where I can navigate to LocalState\rootfs\etc\ to find the root file system and then the rest of the path.

Be aware the files in this directory only contain links to an archive folder! Get the actual files from that path, you can see they are all suffixed with the number one in the filename.

Converting the pem files to a pfx

Azure App service wants to have a pfx file instead of the pem file that was generated. After all, it is IIS behind the covers, so it behaves the same way as IIS. This means we need to convert the pem file to a pfx file. You can do that in several ways, but the OpenSSL tooling that was mentioned in this Stack Overflow question seemed rather straightforward. Unfortunately OpenSSL does not provide the binaries for the different platforms anymore. You can only download the Git repository and try to build it from there. Luckily I found the binaries hosted here and I used them to execute the next steps.

Navigate to the OpenSSL path and execute this command to generate a pfx based from the pem files Let’s Encrypt generated:

.\openssl pkcs12 -inkey "C:\Users\RobBos\Desktop\GDBC Challenges\privkey1.pem" -in "C:\Users\RobBos\Desktop\GDBC Challenges\fullchain1.pem" -certfile "C:\Users\RobBos\Desktop\GDBC Challenges\cert1.pem" -export -out "C:\Users\RobBos\Desktop\GDBC Challenges\gdbc_challenges.pfx"

It will request a password that can be left empty for usage in Windows itself, but Azure App Service requires a password on it.

Uploading the new certificate

Uploading a certificate to Azure App Service can be done in just a few steps. Upload the new certificate and bind it with an SNI Binding to the correct domain.

Run .NET Core programs in Azure DevOps

2019-08-17T00:00:00+00:00

Recently I wanted to build and run a .NET core console application in Azure DevOps and found out you cannot do that with the default .NET core tasks.

Photo by Sam Truong Dan

The default tasks in Azure DevOps and tutorials are more geared towards web-development and publishing a zip file that can be used with a WebDeploy command.

For an application,I would have thought that you could run the compiled assembly by calling dotnet run path-to-assembly on it. Turns out that the run command is used to run the code from a project, not from a compiled assembly (see the docs).

You can just call dotnet path-to-assembly, but the .NET core tasks in Azure DevOps will not let you do that: you can select a custom command, but you cannot leave that command empty for example.

Option 1: Publish the application to self-contained

Here’s how to go around that limitation: publish the application for the platform(s) you want to run: that way you’ll have an executable that can be executed with a PowerShell task. I choose the Windows platform as a target and published the files to a separate publish folder. .

You can then run it in a release. The release just consists of extracting the build artefact, overwriting the application settings with an Azure DevOps Extension and running the executable.

Option 2: Run the assembly

An even easier way to run the assembly is to call the dotnet command on the assembly itself, just do it in a PowerShell task:

Azure DevOps Marketplace News - Or Imposter Syndrome for developers?

2019-08-16T00:00:00+00:00

I’ve been developing software for over 16 years now and every now and then I come across someone who thinks developers do something magic that they can never learn to do. Maybe they are even afraid to ask us something because they don’t understand something. As a consultant my role has often meant that I could take the time and explain to someone more functional oriented the reasoning behind some decisions a developer could make.

Photo by Clem Onojeghuo on Unsplash

So this post is meant as an example battling against imposter syndrome and to openly document the development process and some key decisions along the way and give an insight about the stuff I need to look up (hint: it is a lot!).

The git repository with the source for the tool is open source and can be found on Github.

All in all I have spend around 6 hours in an editor (measured with the WakaTime extension) over the course of 2 weeks to get a working twitter account that checks something every three hours and tweets about it if needed.

Reason this project exists

The reason I started with this project is that I always wondered if there could be a way to stay up to date on Azure DevOps extensions on the Marketplace: there are many extensions available in it and if you don’t check the marketplace regularly, you can easily miss some gems. Of course, when you encounter a specific problem, you will probably find the extensions when you need them, but I thought this was a fun thing to do.

Searching around did seemed to prove there isn’t a good way to stay up to date on the extensions: there is no RSS feed, Twitter bot, or any other option than to regularly check the “Most recent” feed. Since I still am a developer, I thought I could probably make something myself and that this should be a fun thing to do that should not cost to much time to make!

Functional requirements

My own functional requirements where quite easy:

Check the marketplace data periodically for changes.
Tweet about them on a new account when found.

Seems straightforward and not that hard!

Research

To check if this is even possible with the API, I opened up the developer tools in Chrome and checked the calls that the marketplace page makes to the back-end. After finding the call that seemed to response with actual JSON data that listed all the extensions, I checked that I could make the same call in Postman and actually get some results: that way I knew that I didn’t need to login to an API or send in some magic cookies or something.

Seeing results in Postman meant that this whole idea is feasible at all an convinced me to get started. I needed a tool around the calls, somewhere to store the current data, diff any changes and then tweet about new or updated extensions. So just a couple of components 😄.

Starting point

I didn’t want to start exploring fun new technologies to get all the necessary steps working: I just wanted to be notified when there was a new extension available! Therefor I decided to play my strengths and stick with what I have been using for more than five years:

Git repository
C# with .NET core
Azure DevOps

First real commit

Starting in Visual Studio with File –> New –> .NET Core Console Application after creating a new Git repo I quickly created an outline to load the information from the API.

In the first commit you can see the thinking process and coding style that I use when building a MVP (minimal viable product).

Everything is still in the Program class with a lot of to-do’s in it. Some data classes like ExtensionDataResult.cs are separated and just filled with the awesome Visual Studio feature “Paste JSON as Classes”.

At this point I was mostly trying to get the download working and store the results in a list so I could easily start a diff method next.

I haven’t found it easy to do any TDD around my code, especially not at first, and because I was the only developer I skipped all those best practices 😁. Seeing this as a fun side project supported that decision. My head still seems to work in a different way and I like to see the results as soon as possible.

Searching for code

To give a feel for the amount of searching around I did to actually create a tool around my own needs:

Working with HttpClient to send calls to the Marketplace API: looked it up in a different project.
Serializing the result with Newtonsoft.Json: googled it.
Sending a tweet through the Twitter API: found an example on Stack Overflow
Setting up a developer account on Twitter so that I can tweet: googled it.
Exporting a list to CSV: googled it.
Storing the data in an Azure Storage Account using blobs: looked it up in a different project.

Flow

After getting the JSON results and de-serializing them, I wanted to see what overall information was in the dataset, if there where any duplicates and other stuff that stood out. One of the quickest ways to do this for me is in good old Excel: just export a CSV and load it in, create a pivot table on it and go to town. From that I learned that I did need to deduplicate the list and that there are tags that could mean that the extension was pushed to the marketplace but was not made public yet. Good checks to add to the application.

Running locally

This is still a console application that I run in debug mode to see if everything works. Next step was storing the data to disk, so I could create a method to diff the lists between the previous check and the current one. When I found changes in the next day (remember that this took two weeks to complete?), I searched around for some code on how to create tweets from the changes. I found an example on Stack Overflow that looked okay and just worked, so I’ve included that in a separate class. I haven’t even added a real secret store yet: I am just running it from appSettings.json with an optional load of appSettings.secrets.json! Yes, quick and dirty 😛. I did make sure to add that file to the .gitignore before committing, so that is fine for my single developer use case.

I think I ran with this setup for at least a week, running it multiple times a day, right from Visual Studio. I added logging on any exceptions that occurred, added extra information to the tweets, like the name of the publisher, using the tags as Twitter hashtags, etc.

After running for a couple of days I got some feedback about some of the tags that seemed to be tags that the marketplace website uses to determine for instance if the extension is paid or if it has a trial period attached to it. That has been added in as well.

I also thought it would be nice to add the publishers to the tweet themselves if I could find their Twitter accounts. The first publishers have been added to a hard-coded list.

See how gradually this has progressed? Everything has been added when I actually had a need, and refactored into separate classes when those parts felt ready.

Unit tests

Only recently I added my first unit test to the project, because I wanted to test if the Azure DevOps publish tags would work correctly: that is a comma separated string in the JSON object and I created some tests to make sure those would work.

Automated runs

The next step was to run the tool somewhere in an automated fashion. I could do something fancy and go to a serverless solution, but in the current iteration had enough value in it for myself, so I just need it to run somewhere. I could have just added a local Windows Task that ran the tool on an interval or something, but that was not good enough: I wanted to run every couple of hours, even when my laptop was not running. I also did not want to add all kinds of exception handling in the case the tool was running and my WiFi lost its connection or the laptop was being shutdown. Basically I needed a scheduler that could run my .NET Core tool.

Preparation

Before I could run the tool on a different machine, I needed to make sure that the data file that I was saving locally on each run, would not interfere with a local file for the Azure DevOps Agent. Time to start storing that file in an Azure Storage Account as a blob! Only now I’ve added the code to do this. If you check the code, you can find that all I’ve changed is downloading and uploading the file from blob storage to a local copy. The rest of the code has not been changed, because it worked. No need to go to an in-memory solution without writing the file to disk, the current code works and speed is not an issue if I run once every three hours.

Azure Pipelines

As a consultant I currently live and breathe Azure DevOps on a daily basis, so this tool is my big scheduling hammer: I don’t need any other fancy stuff, just run it every three hours and let me know if something happens. I have an MSDN account that I can use for these test projects and as Azure DevOps provides unlimited pipeline minutes on open source projects, this fits perfectly.

Building the solution

The first step is building the solution. I could run it from the build but that seems like overkill: the solution should not change that much that often (after the first few iterations) and adding a release for something in .NET should not be that much work.

I started with the default ASP.NET core web template. That uses the flag Publish Web Projects to publish a zip file with a WebDeploy package in it. Since we cannot use that I changed the publish step.

Later on I found out that the release cannot handle the normal .NET core DLL that is generated instead of a .NET executable: the .NET core tasks do not support executing the dll with the .NET run command: it wants the .NET core project folder to first build and then run the solution. I had to work around it by publishing a full .NET core app targeting the Windows platform. That way I have an executable that I can trigger with a PowerShell task.

Release or run pipeline

The release just consists of extracting the build artefact, overwriting the application settings with an Azure DevOps Extension.

Full circle

I started out with a Git repo, pushed that to GitHub, build and run in Azure DevOps and then report the status of the build and release through badges in Azure DevOps and included those in the readme of the repository:

Step	Latest execution
Build
Release/Run status

By the way, I am using Azure Pipelines from the GitHub marketplace to run the CI/CD triggers for the project.

For the scheduling part I am using a scheduled trigger that will run the release definition every three hours. Somewhat irksome to add so many (no cron notation), but if it works….

Conclusion

I hope this gives an insight to the development process to someone that is curious and maybe helps to lower some of the imposter syndrome that some developers seem to have (myself included!). With a good mindset you can figure things out or just reach out and ask for help. With that, you can get anything done!

Using Chrome Personas to split identities

2019-07-13T00:00:00+00:00

As a consultant, I get to work at a lot of different settings and environments. For most of my customers these days, that means working on my own laptop and in the cloud with SaaS application.

Logging in to all those customers can be a messy thing: I’ve seen people having a identity picker in Azure (or any other Azure Active Directory backed system) that they have to scroll through to get to the identity they want to use 😱. Seriously!

Note: It did get better in the last years, previously you actually had to log out and in as a different identity to make this work, but still… we can do this better!

Jesse Orrico on Unsplash

Solution

I have a helpful way of keeping all those personas separately from each other, with some additional benefits.

First and foremost, I do this to keep everything nice and tidy: separating them helps my mind in compartmentalizing the status and context of the tasks that I am doing.

Chrome Personas

I have use Chrome personas for several years to accomplish this, after learning this feature was available in FireFox and Google had copied that functionality. It is a feature that lets you separate parts of your browser into it’s own (hosted) process and keep everything in that compartment. I noticed that Chrome does this with:

Browsing history
Tab history
Plugins
Stored Passwords

Benefits

So by doing this, I can separate my personas and with it my identity! I make a new persona for every customer that I get to (and others for personal account separation). I let Chrome store the URLs, tabs and passwords that I need for that customer, and when I leave, I just remove the persona from my system!

In each persona I have logged in to different Azure accounts for example, together with different Azure DevOps accounts, Office365 logins and other services that I need (tooling, CRM’s, other SaaS offerings). This saves a lot of switching.

Some note’s to this:

I don’t sync the personas currently, as I mainly use my 1 laptop), but you can if you want to. This is a persona by persona setting.
Any account information that I need for later is stored inside of a KeePass file backed up in my Office365 OneDrive folder.

Cool! Now how do you do this?

To get started, click on the User circle on the top right. You get a flyout with all you persona’s. Mine is quite long:

Noticed the persona Faebook? I wanted a persona with my FaceBook account in it, to prevent my personal account leaking around everywhere when I don’t want to. I use that product only on my phone for family stuff, so I did not want to login to it on any of my persona’s. Unfortunately you cannot renamed them (can’t find it in the UI anyway)!

Manage people

Click on ‘manage people’ to open a window where you can create new accounts and delete existing ones:

Notice that you can use a lot of different icons and images to link to your persona. I usually use the companies color for my customers persona and something else for personal stuff.

Now you can use the new persona. Notice how it opens a new Chrome window that’s also separated on the taskbar? That means you can pin it!

My most active accounts are always within reach.

Edge Dev

You can do the same thing in Edge Dev, since the functionality is in the Chromium underpinnings that both Edge Dev and Chrome build on top off.

I wanted to use Edge Dev for this because of one big benefit: This means that you no longer have to copy and paste links into a different browser/persona window to use a link! You can just use the build in functionality.

The main reason I haven’t started moving my personas there is listed below in the Downside section below. I was a little disappointed by this and got sidetracked with other stuff. Writing about this is getting me to think about doing it again…

Downside

The only real downside I have with this setup, is that Chrome does not let me pin my main persona (the one that I did not create) to the taskbar. Edge Dev has the same issue, so I suspect that this stems from the Chromium underpinnings.

Main persona

The main persona seems to be the one that you logged in with to Chrome: I use that as my Gmail account and let that persona sync across other devices (iPad and iPhone). It is the user account on Googles back-end, so to say.

The issue with the main persona

The main persona is attached to the first Chrome pin on the taskbar. When you switch to that persona in Chrome, suddenly it is updated with the image for that persona:

Note the difference with the screenshot before, when that persona was not loaded. Notice the 6th icon here? It was empty before:

When you try to pin that persona, it’s pinned as the default process! There is no way to have that persona pinned on the taskbar. If you close that session, the icon is clear again. If you click on it it will open the last opened persona!

Help me out

If you have a way to fix this (If have even tried to mess around in C:\Users\RobBos\AppData\Roaming\Microsoft\Internet Explorer\Quick Launch\User Pinned\ with manual shortcuts, but none of it worked), please reach out!

That would solve the only issue that I have with this setup: it could be complete by fixing this!

Update: also tested with launching Chrome manually

After a tip from Jasper I tried to see if I can launch Chrome.exe from the commandline with the correct persona. You can do so by providing it an extra parameter: .\chrome.exe --profile-directory="Profile 1" Maybe this can be included in a shortcut and launch it that way?

To see how Chrome names the personas, you can check your user directory here: C:\Users\RobBos\AppData\Local\Google\Chrome\User Data\Guest Profile

Or look in the registry editor:

I’ve tried them all before finding that the Default profile is indeed called…. the Default profile! Unfortunately does the same thing. I also found to extra personas that might have been deleted at some point. After launching them, they are also visible again in the Chrome UI!

By the way, storing the shortcut like this on the desktop and launching it does work. But I don’t like to switch to the desktop each time I want to launch into this profile.

Fixing GitHub Pages Syntax Highlighting

2019-07-12T00:00:00+00:00

Today I noticed that my syntax highlighting was not working on this blog. Here is how I fixed it!

I am using Jekyll on GitHub pages as I wrote before.

Photo by Zach Reiner

Looking at the generated HTML indicated that there was some parsing done during the build of the page, but there were no CSS classes available to them:

I tried searching for documentation about this issue and found some basic stuff. This hinted that I needed to set up a highlighter in my _config.yml:

highlighter: rouge

There is another highlighter (Pygments), but this is not supported. It even seems that rouge is just the default, so you do not need to set it at all!

I found a Stack Overflow question that indicated I needed to include a CSS file with the highlighting I want myself.

Lazy as I am, I searched around and found a gist with a SCSS setup in it. I modified that to be just CSS and added it to my head.html like so:

<link href="/css/syntax.css" rel="stylesheet">

I also needed to add the syntax name that I am using in lowercase to get it all to work: so powershell instead of PowerShell.

Cool feature of GitHub Pages

GitHub pages is already cool by itself, but did you now they actually send you an e-mail if there is an issue with you setup that prevents the yml-build from working?

Using Azure CLI with PowerShell: error handling explained

2019-07-12T00:00:00+00:00

I found myself searching the internet again on how to use the Azure CLI from PowerShell so that I can use it in Azure Pipelines to create new Azure resources. The reason I want to do this with PowerShell is twofold:

Azure Pipelines has a task for using the Azure CLI, but this only has the options to use the command line (.cmd or .com files), or from bash (.sh). I don’t like them that much, I want to use PowerShell (Personal preference)!
Running the Azure CLI from PowerShell has the issue that it was not created specifically for use with PowerShell. You’ll need to do some extra work.

I’ve fixed it before, but it took a while to find it again. That is why I am documenting it here, to save me some yak shaving in the future.

The issue in PowerShell

Running this Azure CLI command in PowerShell will result in an error, because storage accounts cannot have a dash or capitals in the name:

$StorageAccountName = "testRb-001"
$ResourceGroup = "myResourceGroup"
$location = "westeurope"
az storage account create -n $StorageAccountName -g $ResourceGroup -l $location --sku Standard_LRS

Result:

Seems like an error, what’s the issue then? Well, adding error handling like you’d expect from PowerShell will not work!

$StorageAccountName = "testRb-001"
$ResourceGroup = "myResourceGroup"
$location = "westeurope"
try {
  az storage account create -n $StorageAccountName -g $ResourceGroup -l $location --sku Standard_LRS
  Write-Host "Just continues"
}
catch {
    Write-Host "An error occurred!"
}

You can see that PowerShell doesn’t notice the error and just continues:

Even adding -ErrorAction will not work.

How to add error handling yourself

The Azure CLI runs on JSON: it will try to give you JSON results on every call, so we can use that to see if we got any data back from the call. After converting the result, we can test to see if it has data:

$StorageAccountName = "testRb-001"
$ResourceGroup = "myResourceGroup"
$location = "westeurope"
$output = az storage account create -n $StorageAccountName -g $ResourceGroup -l $location --sku Standard_LRS | ConvertFrom-Json
if (!$output) {
    Write-Error "Error creating storage account"
    return
}

Do remember to wrap every call you need to run with this setup, and return to prevent PowerShell to continue with the next statement.

Writing the error to the output helps with:

Displaying the error correctly
Blocking the release in Azure DevOps, which is were I needed this the most.

Shorthand

There is a shorthand version of this code that you can use, if you don’t care about the information in the output (Thanks Rasťo !). You can use PowerShells about automatic variable: $? to just check if the result was successful or not.

$StorageAccountName = "testRb-001"
$ResourceGroup = "myResourceGroup"
$location = "westeurope"
$output = az storage account create -n $StorageAccountName -g $ResourceGroup -l $location --sku Standard_LRS | ConvertFrom-Json
$LastExitCode
if (!$?) {
    Write-Error "Error creating storage account"
    return
}

As stated in the documentation:

Contains the execution status of the last operation. It contains TRUE if the last operation succeeded and FALSE if it failed.

I am just not sure yet as to why this works. I suspect $? is looking at the $LastExitCode, because this will print out 1:

$StorageAccountName = "testRb-001"
$ResourceGroup = "myResourceGroup"
$location = "westeurope"
$output = az storage account create -n $StorageAccountName -g $ResourceGroup -l $location --sku Standard_LRS | ConvertFrom-Json

$LastExitCode

Why am I using the Azure CLI?

After posting this, I got asked why I am using the CLI to do this at all? Surely, Azure PowerShell or ARM Templates would be sufficient.

Here is why:

Azure PowerShell is not idempotent, so not so great to use in Azure Pipelines.
CLI is much terser than ARM, although it feels like you need to do a little more work, linking resources together.
Read Pascal Naber ‘s post: Stop using ARM templates! Use the Azure CLI instead.
I am looking into doing this with Terraform, because the declaration is a lot shorter as well. It is a new paradigm to learn, and needs installation in your Azure Pipeline. The CLI was already available.

Why not use bash?

The reasons for not using bash is that:

It will not work on a Windows Azure Pipelines Agent (and that is what I am using here).
You need to include JQ as a library to be able to parse the JSON. This seems like extra work to me. I also find the JQ syntax not that straightforward.

Here is a shell example to make sure you are connected to the correct Azure Subscription, to be complete:

# Switch to the correct subscription
az account set --subscription ${SUBSCRIPTION_ID}
output=$(az account show | jq '.')
[[ -z "$output" ]] && printf "${FAILURE}Error using subscriptionId, halting execution${NEUTRAL}\n" && exit 1

subscriptionId=$(echo $output | jq -r '.id')

DevOps Principles series

2019-07-10T00:00:00+00:00

I have started recording a series of video’s explaining some of the principles and sayings I use when I talk about DevOps. There are teams that I meet that have no idea what DevOps is and why we are doing some of these things for it. I’ve found myself referencing something like ‘Shift Left’ and then having to explain it to the teams and their management that I am helping with something that for me, is related to something in the DevOps culture.

I also want share that information with the rest of the team members that weren’t there at that time. So for these cases, I thought it would be nice to have a set of small video’s available that I can share and explain those things to everyone.

Creating these video’s helps me getting a little more comfortable speaking to an audience, even if it is through a camera lens and with the ability to stop and do it again 😄. It also helps that I have some time to think through the message I’m trying to explain to the team. From there I am trying to get my intonation and articulation under control. This stuff is all new to me, so I am figuring this all out live on the job. I’m learning tons of stuff I didn’t really think about before. This will be explained below. Somehow I like to challenge myself and do this stuff out in the open 😁.

I have a different post explaining my setup and learning process on recording: link.

You can find the entire playlist here. I try to keep them as short as possible: bite-size!

Some of the topics explained are:

What is DevOps?

I couldn’t just start and not talk about the origination and basic idea’s behind DevOps. This is a starting point for the series so I took a little extra time to explain why I started creating these video’s. You can find this episode here or watch it below.

Shift Left

The first principle I explained is ‘Shift Left’, a common theme when we talk about DevOps. The basic goal here is to have a fast feedback cycle on anything in the development process. Watch this episode here.

T-Shaped engineers

In this episode I explained the reasoning behind ‘T-Shaped engineers’, because we move from traditional role based work like ‘developer’, ‘tester’, ‘systems engineer’ to a more generic ‘engineer’: someone with a broad skill sets in multiple aspects of the traditional roles and deep knowledge about one (or more) of them. Find the episode here.

If it hurts, do it more often

On of the DevOps things that I say a lot: if it hurts, do it more often! If you only update production twice a year, it will be a scary activity to do: lot’s of things can go wrong! To make this activity a non-event, we need to do it as often as we can. We will learn how to do it and we can fix the things that go wrong and improve on them. Watch the video.

You build it, you run it

If you follow this principle, running the software in production becomes a team effort. Instead of throwing the software over a wall, you handle it as a team. This also means setting up monitoring and alerts and thinking about observability during the development process. Find this video in the link.

Four eyes principle

Making sure you work on the software and embed the knowledge about it in the team can be done by following the four eyes principle. This makes sure that no-one is working on things by themselves. This will help on other things as well, like the team spirit for example.

Automate everything

Automating everything is a bit large as a saying. It’s not about automating everything in an effort to just blindly automate everything: it’s about thinking of automating every step in the development process: could we benefit if we automate this step and is it worth the time we take to automate it? Watch the video.

Final thoughts

Creating these video’s and sharing them online has been an interesting journey, I learn something new every time 😄.

If there are any topics that you feel have to be included, please let me know!

DevOps Principles series - Recording setup

2019-07-10T00:00:00+00:00

Read more on why I created short video’s wherein I explain some of the DevOps principles and practices here.

Setup

Let’s start with the basic setup. As I am trying to find out how this all works and where to go next, I am using the hard- and software I have available. As a laptop I happen to have a Dell XPS 15. The XPS range has a couple of issues in their design. The biggest one relevant to this post is the position of the webcam. Apparently the good people at Dell found the slim bezel on the top and sides of the screen more important than the actual usability of the webcam. So up the nose shots it will be, unless you try to overcome that (more on that later).

Software

I’ve tried a couple of different software downloads that are freely available, for example with a trial version. I know a couple of colleagues use Camtasia for this stuff, but I am not ready to commit to a € 260+ yearly purchase for a small series of video’s. Surely there must be something available that is sufficient for my objectives?

Windows 10 already has some options build in, like the default camera application. Unfortunately that will not let you record the screen as well, so only audio and the video through your webcam. What I want to achieve, is a PowerPoint presentation, with audio and my webcam in the lower right corner, taking about the subject at hand.

I once came across Soapbox, a Chrome plugin that can record the screen and impose your webcam video on top of it. This seemed like a nice solution, except that I found out that you need to pay to export the video! My goal is to create a playlist of all the video’s on my YouTube account and then share the video’s as needed. I am all for paying for the product if it adds value, yet I find that I am not ready to commit to anything yet.

PowerPoint!

Finally I came across a blog post that had a top 10 of video tools on Windows and it mentioned that PowerPoint has some functionality built in to record yourself while giving a presentation. This is to help the presenter with their timings, articulation and other things they need to see while rehearsing the presentation. Sounds like a good option, so I went ahead with it!

Height

Trying to prevent the weird shots, I’ve been testing with different setups. The goal is to find a setup where the camera is around my eye level and where I can sit comfortably talking over a presentation.

Next up is trying out to record the videos while trying to look AT the camera.

Light

I found out that light in your recording setup is really important, especially if you use the in-laptop webcam for it. More light is better! I tried recording with all the lights on in my living room, in front of a window and behind a large lamp in the kitchen. None of these were adequate enough for me: I need to invest in a good camera that I can freely position on the top of my laptop for a better view.

Sound

For sound, I recorded the first videos with the onboard microphone. You can hear the poor quality in the first three videos! As different options, I’ve used my iPhone headset (better), my Sony WH-1000XM2 Headset (bluetooth of the Dell XPS is crap, it flunks out after 30 seconds or so) and finally got my hands on a Trust Mantis GXT232 Streaming Microphone.

This is the extra microphone that I needed! I even recorded with this one in a hotel, because of it’s size, it can easily be taken with me on-the-go.

Recording

After using PowerPoint I found that I needed a way to suppress the fans of my laptop: they do start to make a sound that you can hear in the first video’s: as soon as I start recording, everything goes over the video card and the fans start spinning 🙁. To overcome this I now use Streamlabs OBS. This open source streaming tool can also record and has the key-feature that I needed: noise filtering! Getting the level of the filter setup is still a search, so in some of the video’s the sound is rather soft. I trust that this will improve with more experience.

Note: Streamlabs OBS is an adaptation of another open source tool: [OBS](https://obsproject.com/).
OBS itself doesn't have great HiDPI support that I really needed on my 4K screen. It's not doable without it. Streamlabs has better support for it and the navigation is a lot easier to use.

Playlist:

You can find the entire playlist here. Some of the topics explained are:

What is DevOps?
If it hurts, do it more often
T-Shaped engineers
Shift Left

I try to keep them as short as possible.

GDBC: Link overview

2019-07-07T00:00:00+00:00

Last month we got the opportunity to organize the Global DevOps Bootcamp (link) and it was a blast!

I wanted to create an overview of all blogposts that I could find about the event, so here it is.

Links

Pre-event registration

Jasper Gilhuis wrote down how he handled the pre-event registration of venues and enable them to register the attendees. Read about it here.

Azure Learnings

A post by myself about all the stuff I learned while creating the automation to roll out all the resources we needed in Azure: link.

Monitoring the event

Posted by Michiel van Oudheusden on the things he did for all the monitoring of the infrastructure for the event.

48 hours running the global event

A post by myself about the day leading up to the event and during the day itself. We ran all the infrastructure we needed as a team. You can read it here.

Application Insights

Michiel van Oudheusden shares how he enabled Application Insights to track dependecies in our infrastructure here.

Challenges

The GDBC team has opened up the challenges website we used during the event itself so everyone can go through the challenges (and behind the scenes videos) to keep on learning! Available under the creative commons (non-commercial) license on gdbc-challenges.com

Looking back at the event day

There are a lot of people who blogged about their day at the event, how they found out about it at all and those posts are awesome to read.

Here is an overview:

Thomas Rümmler helped organize an event in Stuttgart, Germany and wrote about the experience here.
In Stockholm Sweden Antariksh Mistry was at a venue organized by Soldify. This was his first GDBC and you can read about his experience here.
Some venues even created a video of their day! This one is from a venue in Bogotá, Colombia and this one is from Vancouver, Canada. This one is from Quebec, Canada.
Dmitry Larionov walked into a venue without really knowing anything about GDBC! Find out here what he thought of it :smile:.
In Vancouver, Canada Willy-Peter wrote down their feedback.
Hannupekka Sormunen visited a venue in Helsinki, Finland and posted his whole dairy here.
In Zaragoza (Spain), Veronica Rivas helped to organize their second GDBC an wrote about it here.
Read here about the event in Sofia, Bulgaria, which they organized for the third time (perfect score!).
The organizers in Toronto, Canada blogged here about the event.
David Gardiner helped organizing the third edition in Adelaide, Australia. Read about their day here.

Behind the scenes

If you want to see what the team created to run the event, you can find all the behind the scenes video’s here: https://xpir.it/GDBC2019.

Keynotes

Here are some of the keynotes that have been published online.

Year	Speaker	Topic	Link
2017	Donovan Brown	What is DevOps	No video online
2018	Buck Hodges	How Microsoft does DevOps when creating Azure DevOps	YouTube
2019	Niall Murphy	Azure SRE Practices	YouTube

GDBC: Azure learnings from running at scale

2019-06-23T00:00:00+00:00

On the 15th of June we got the opportunity to organize the Global DevOps Bootcamp edition of 2019 (see link) and we had a blast!

For the 2018 edition we created challenges for the attendees to setup their CI/CD pipelines to push a web application into Azure. You can read up on the setup for that edition here.

Next level

Since we need to create something next level for each new edition we had a brainstorm session somewhere in November of 2018. How can we improve the experience for 2019?? We got quite some feedback that for the attendees that had no previous experience with Azure that some steps (e.g. creating a Service Principal) where to hard for them, both to understand and to execute. Although that story has improved a lot with for example the release of the Azure CLI, we wanted to make it even easier: what if we setup everything for the teams?

Team setup

So we would start them with a working web application, running in Azure including a database connection and application insights to boot. We also wanted to set them up with a complete Azure DevOps setup, so everything from a Git repository, CI/CD pipelines, working service connections, package management, the whole deal.

Find out more

If you want to know more about the whole setup, checkout the YouTube playlist the team has made around the event here. I helped creating the Azure DevOps scripts (together with René, Jasper and Sofie) and created the Azure setup myself. You can also look at this video explaining the Azure DevOps and Azure parts:

If you really want to find out more, I would love to come talk about all of it on conferences or meetups! You can find my session here. I am sending this session in for conferences, either alone or together with René.

Size

We decided to cap the venue count this year to a 100 venues. This would give us a limit in the amount of resources we needed to create and we could calculate an indication of the costs of those resources. We got 4 sponsored Azure subscriptions from Microsoft with limited budgets on them. From the tickets the venue organizers added in Eventbrite we could guess the scale we needed.

In the end we created:

1340 App Services,
96 SQL servers,
1340 SQL databases,
1340 Azure DevOps Teams in 7 Azure DevOps organizations (each with a full setup)
96 AAD Venue Team groups and users,
1340 AAD Team groups and users,
1340 AAD Team Service Principals
1340 team role assignments in 1340 + 96 resource groups

Learnings

Running at a scale like this means we where going to hit some weird stuff in the API’s we used. Here is what I learned by provisioning the Azure resources. There where some small things like the fact that not all resource providers are registered in new subscriptions. I knew that this was the case, but totally forgot about it. When I hit this issue, I was surprised that this even happens for the most ‘basic’ resource providers, like Application Insights and KeyVault! My colleague Pascal Naber has a blogpost about how to enable all the providers for a subscription. In our case I decided to not use that, and enable the providers we needed by hand in the four subscriptions we had. The main reason behind this was that we needed to make all the venue organizers and the team accounts Owner on the resource groups we would create for them, so they could actually do the things we wanted them to do. That would give them the rights to spin up everything they need, including the potentially more expensive things like ASE, DataBricks clusters or just VM’s. Just to be on the safe side I choose to leave those possibilities disabled.

Azure learning: Cloud resources are not always finite

I found out that Azure has limits on the amount of available resources you can create in each region. I already experienced that when I was attending Microsoft’s AI Bootcamp in January, when the proctors there indicated limited amounts of DataScience VM’s in each region. I figured this was a limitation based on the compute resources needed and the usual ‘standard’ resources would not have this limitation.

Imaging my surprise when I tested with creating SQL Servers in India:

Even switching to one of the other two regions in India didn’t help! Apparently some quota is set on Azure’s backend and you cannot create those resources 😲. Even testing a couple of weeks later still indicated it wasn’t possible! Eventually we moved those teams to Southeast Asia.

Azure learning: Portal search is not great

When you have a large Azure Subscription with a lot of resources in it, the search just is not that great. Searching for a wildcard in the beginning of a name isn’t possible, so searching for %brisbane% was not possible. That meant a lot of copy and pasting if I needed to find the teams resources to check them for something.

Azure learning: Azure SQL server limits

As I always want to segment users and teams from each other as much as possible, I wanted to create an Azure SQL Server per team at first. I knew there are soft limits and hard limits in Azure for some resource providers, like for example the amount of CPU cores you can allocate. There is a support process around it where you can ask to increase those quotas.

That reminded me to check the quotas for SQL Servers, Databases and App Service Plans that we needed to create. Luckily I did: there is a soft limit of 20 Azure SQL Servers per region in a subscription that can be increased with support tickets and a hard limit of 200 Azure SQL Servers per subscription! That would not work for the 1200+ teams that we had planned by the time I found out about this limit! We decided to switch to provision 1 SQL Server per venue (planning on max 100) and then create the database per team. That would mean that the teams could see and control the databases of ALL the teams in their venue, but we expected the teams to not screw up the databases of other teams. I also made sure the venue organizer would get a VenueAdmin account that could help out the teams in case of need. Since I wanted the attendees to be able to monitor their database for at least the Percentage DTU consumed, it meant that they needed to have those rights on the server, since you cannot set ACL’s on the database level. Note: we did create SQL Server User accounts for the databases as well, so we had different passwords per team.

Azure learning: Application Insights not available in all regions

Application Insights resource provider is not available in Central US! For me, Application Insights is a base provider that I’d use in any application I create or consult for. That made me expect that this provider would be available world wide, or at least at what I would consider standard or big regions. This is not the case! So the advise is: always check! We got tipped beforehand by an observant venue organizer in the last week before the event and we moved those web applications to East US.

Azure learning: 2000 role assignments per subscription

At first, I was making role assignments on a user level, based on the fact that I had that call working. There is a limit on how many assignments you can make per subscription! Read more about it in the docs.

If you get the error message "No more role assignments can be created (code: RoleAssignmentLimitExceeded)" when you try to assign a role, try to reduce the number of role assignments by assigning roles to groups instead. Azure supports up to 2000 role assignments per subscription.

Azure learning: portal only shows the first 2000 resources in any pane

This only makes sense and is also present in the REST API, but the portal ‘only’ shows 2000 resources in the listings. This is at least the case in the All resources view.

Azure learning: always check the defaults or pick your own

This one was something I had not thought of for a long time: I was creating the Azure SQL databases with the default during testing. This meant that the teams got a S0 database which would cost us €0.0171/hour/database (remember that we needed 1350 databases!). Right in the period that we ran the full set of resources, the default changed!! Azure now picked a Gen 5 with 4 vCores that costed €0.9021/hour/hour/database! I ran through € 5.000 in less than a day, because of this!! 😱😱😱😱. Lucky that I found this, I ran an updated on all of them to resize them back to an S0. Also updated the code so I picked my own default and set that to an S0 as well. You can find out more information about pricing here. I could have also used a SQL Elastic Pool per venue and place the database in it, that would just have costed less euro’s and more time to build. I skipped it because I needed to do a lot of things in that last week 😉. You can read more about the last 48 hours of the event in another post.

Azure learning:

There are some calls that you can make that can fail in a weird way. I had several calls fail where I created both the Active Directory Application and the Service Principal to go along with it. Given that I made 1350 calls to this API and > 20 of them failed, this is quite interesting. Anyway: on failure: the Application would be created and the Service Principal would not! I quickly added a check on it that would just delete the Application so the next run would recreate it. Not sure if this is caused by the Fluent SDK that I was using.

servicePrincipal = await AzureClientsFactory.AADManagementClient.ServicePrincipals
                        .Define(spnName)
                        .WithNewApplication(spnName)
                        .DefinePasswordCredential("ServicePrincipalPassword")
                        .WithPasswordValue(spnPassword)
                        .Attach()
                        .CreateAsync().ConfigureAwait(true);

Azure DevOps learnings

I encountered the Azure DevOps REST API’s again and as in previous times, I am very grateful for all the experience and knowledge my colleagues René, Jasper and Jesse have with them! These API’s are documented but we seem to always need some exotic variant or a specific thing that is hard to find, for example setting default repository permissions on your Azure DevOps Organization, as documented by Jesse here. We knew that we needed to setup 1200+ team projects and the GDBC team has excellent contacts with the Azure DevOps product group, so we arranged 7 different Azure DevOps organizations. This would enable us to:

spread the load on the service, we hit some weird quota’s and functionality issues last year,
keep the latency low for the teams, since we could place them to an organization closest to them,
run the CI/CD pipelines that would create the App Services and other resources as close as possible to the start of the event, so we could keep the costs low.

DOSsing the Azure DevOps service

Since we had 7 different organizations, René created a nice pipeline to start all the CI/CD pipelines after we created all the other setup (Git repo’s, Service connections, etc.).

Getting the sponsoring from Microsoft also meant an increase in available Microsoft Hosted Pipelines. I learned later that you cannot even buy these numbers of pipelines as a regular customer! This meant that we either had 100 0r 160 hosted pipelines available (depending on the estimated size of the organization we would set up), so we could kick off all +/- 400 pipelines in each organization when needed.

This would mean scheduling all 400 CI builds when we wanted, witch would all kick off their CD release on completion. This eventually meant rapid scaling of the pipelines in the region the Azure DevOps organization was linked to. Two off these regions had some serious issues handling that load!

Eventually this was all sorted out by SRE’s in the Azure DevOps team, with great, direct support for us. I think the learned something about their own service in the end!

Azure DevOps functionality View in-progress jobs (bug)

Regarding the fly-out of the ‘View in-progress jobs’: there is a call to the backend to load all the running jobs. The fly-out will only show after that callback has completed. With a lot of team projects, this can take a long time to actually do the flying out. During that wait time, you can click the link multiple times: you will then get the fly-out multiple times and you can then close them one by one :grin:.

A refresh button on the fly-out would be nice as well, since this was one of the places I could check to see how the load on the organization was going and if we could start the next stage of our pipeline: right now you need to close the fly-out and then request it again :smirk:.

Another small issue I have with this fly-out is that in more and more places, this fly-out can be closed by clicking on the area outside of it (for example in the build progress view). This fly-out hasn’t gotten this treatment yet.

Before you know, it is in production

2019-06-19T00:00:00+00:00

When I am working on something, usually software, I know from experience that a simple tool to test something out (e.g. a POC, Proof of Concept), can be in production in no-time. That is when I start to focus on everything we start to ignore:

don’t write unit tests, it is only a POC;
we don’t need to make this resilient, it is only to proof this will work;
just name the project yyyyMMddd.TestProject.exe, we will never need to find it again, if this works;
we don’t need to make this scalable yet, we’ll figure that out in the future.

I am just as guilty of this, hence this post for later referral :smile::

Guilty as charged!

Case in point

Recently I was a member of the core team for running the infrastructure for Global DevOps Bootcamp.

When we started, Jasper Gilhuis was trying to automate everything we needed to create the accounts and pages for the venues in Eventbrite (read more about his experience here) and he could not find an API for making the venue organizers a Co-admin in our Eventbrite event.

We needed this because we wanted to have the venue organizers have visibility into their registered attendees and be able to send them in-mails from the platform. First we created the global event and added the local venues as a sub-event. Then we wanted to add their accounts as a co-admin. Do check out Jasper’s post for the why’s behind this setup.

Every tool is a hammer

I believe that every tool is a hammer: even if you know you should do this, you know you can do it with your tool: before you know it, you are hammering in a nail with a phone (check YouTube, it happens!).

Selenium is on of my hammers

Seeing that the Eventbrite API was not able to do what we wanted, and that the flow in the website didn’t seem that hard, I made a new console application for this to use Selenium to click through the website. My first contribution to GDBC this year! As this was a tool to help with the Eventbrite automation, this ended up in the Eventbrite repository.

We need to store some state!

After a while we needed to store the state of the venue registrations: which venues where already mailed and checking into our Slack channel, etc.? We tried to do this with a Table in an Azure Storage Account, but found out the hard way that you need to update the full document, otherwise you will only see the columns you just updated: the rest will be gone. So, us having a lot of experience with SQL server used the tool we knew that would work Azure SQL (another hammer!). Switching to that immediately triggered me to start with Entity Framework to make the communication as easy for me as could be (hammertime!).

We need to show the venues on the website!

A team member asked to use the data we had to update the website and show the locations, registration urls and the number of venues on our main (website). Since this was already in the ConsoleApp, I added this as an extra startup action: Run the application like this and it will update a blobstorage container with the latest version of a data file that the website could then use to show the data 👇.

ConsoleApp.exe -exp

Since I did not want to run this application by hand but every night, I found … another tool to do so!

Provisioning Azure Resources

After a couple of weeks we needed to start rolling out Azure Infrastructure (for more information, check out this YouTube video where I explain what we did. All the resources we needed to create had a link back to the venue and teams that where in the database…. And so I added just another startup action to the tool to start running the steps to provision everything in Azure, based on the information in the database. Can you see where this is going???

Startup actions for when you want to run the tool locally

Initially this started out to be a tool so me or Jasper could run it locally, see all the Eventbrite screens fly by (later headless) and then do some other actions where added.

Today, this looks like this (what a mess!): I wanted to have multiple options for the user to choose from and be able to run action 2 first and if that was executed correctly, run action 3 next.

Running the action in a pipeline

After I added more and more actions for the tool to do, I needed to make it run inside of a pipeline, read some parameters that where passed in and then close the application. For this I always fall back to the Mono.Options NuGet package.

This is way those settings look like today:

So is this still the same application!? Unfortunately, it is… 😲. Every decision in this application was a combination of:

YAGNI (You ain’t going to need it),
This is just to run once,
Lets make this work quickly

In the end, it got the job done, as it is included in every stage of our main provisioning pipeline:

Conclusion

Even if it is just a tool to test: write the code and name things like it will end up in production.
Every tool is a hammer and will be used as a hammer a some point.

Check out the video from this one day build on YouTube

GDBC: 48 hours in the life of a team member

2019-06-18T00:00:00+00:00

Last weekend we got the opportunity to organize the Global DevOps Bootcamp (link) and it was a blast!

Thanks to René van Osnabrugge, Marcel de Vries and Mathias Olausson for coming up with the idea to create GDBC and sticking with the team to get this idea of the ground! Without them and our sponsors (Xpirit, Solidify, Microsoft) we could not have started with the event!

Team work!

To set everything up we send out a call to everyone who helped last year and also to their friends. In the end we had a team with around 15 members, each picking up tasks they could handle (or try something new!). Without all that countless effort of them we would not have been able to pull this off!

The week leading up to GDBC

During the last few months, we had a Monday call at 9:00 PM with the entire core team and of course René, Marcel and Mathias. In it, we discussed the progress, challenges we where facing and asking for help if needed. On and off there where more people involved (special mentions for Sofie and Niels, who supported us heavily in the last week!!!). Usually we spend half an hour to an hour keeping each other up to date.

We all do have day jobs to feed our families, so we actually worked as much as we could during the regular working days and then switched to GDBC when we could. For most of us that meant as soon as the kids or our partners where asleep. I’ve seen a lot of input, commits and feedback after midnight, so everyone was fully committed to this cause.

We lived the event through Slack, where the whole team would communicate with everyone involved: from sponsors, core team to the local venue organizers.

Wednesday and Thursday

Although we got a sponsorship from Microsoft, our funds where limited, so we started provisioning all the infrastructure we needed for the event on Wednesday. I will dive into that setup later on, but you can see my explanation here:

Azure DevOps Pipelines:

Resources for Azure Subscription 1:

This didn’t go as easy as we hoped, luckily we anticipated throttling, availability issues of resources in the Azure regions we needed, etcetera, so we had plenty of time to do so. I will blog about the lessons we learned here separately and update this post with a link to it. Fortunately we had four different Azure Subscriptions to use so we could spread the load!

As most of our infrastructure got available, we enabled the venue organizers to pick on of the teams available for them and start testing our infrastructure end-to-end with it. We already did that with some specific demo venues, so we got some confidence that this would mostly work. We needed their feedback, specifically because there is no place to test on, then production!

Thursday call

On Thursday we had an extra call: around 24 hours before the event would start, so we could cross the t’s and dot the last i’s. There where some last minute challenges that some of us where facing, like the flaky connection that had issues with the way the Parts Unlimited website now used the connection strings and a call to Azure DevOps that wasn’t tested with 1300 team projects :smile: .

Firewall rules for challenge

To handle the firewall rules changes I’ve spend the rest of the evening with Jakob testing and updating all the SQL servers that we had provisioned so that the challenge could work.

More teams needed!

During the day an organizer checked their setup and asked us for more teams! They checked the participants registrations, did the calculations and where worried they would not have enough resources to place everybody in the group size they had in mind. Only other options was making the team bigger. We had been discussing that we would communicate a stop on any changes, but decided to help this organizer out. We communicated the stop on changes a couple of minutes later to prevent us from scrambling at the last minute with changes.

A team is missing!

During the call that evening another organizer pinged us letting us know that he missed team-05 out of the 16 teams we provisioned for them. In Azure the resource group and the team users where created, but the team project was not available in Azure DevOps. As they also indicated they had sold out all the tickets, we needed to fix this. So I planned in a new run for that venue as well.

If it hurts, do it more often

It is really true: if it hurts, do it more often. Especially in a DevOps team, do the things that hurt more often you’ll get good at it and you have the change to automate them! I already pressed René on this last year: we needed to have pipelines setup for as much as we could, so we wouldn’t be locked to a specific developer and their laptop to kick off updates. Since we where creating a lot of stuff, these would also take up to hours of runtime before they’d be finished, so this wasn’t helpful. This meant that I had to setup pipelines for everything I did in Azure, otherwise René would not stop bothering me this year to do so, as he should! :kissing_heart:! Him, Jasper and Sofie spend their time setting up a pipeline for Azure DevOps, so we had that part automated from front to back.

That meant that I could run through these steps in 15 minutes:

Update the database with teams we needed to create or update
Create everything for them in Azure
Create everything for them in Azure DevOps
Kick off their CI/CD pipelines, so the web shops would be deployed
Update their App Services to set up the correct DNS entries and SSL Certificates

48 hours: running the infrastructure for the event

After having a late night on Thursday I had trouble getting a sleep. The following morning I was up at 5:30 AM still rushing with excitement: This was the big day! Last year was phenomenal, hopefully this year would be as awesome as then! I checked the last results for the pipelines that where running to verify that every resource we needed was in place (all green!).

My oldest son (9 years old) had a presentation to give at his school and he wanted to practice it one more time, so I spend the hours before school with him so he could do so. After that I drove to the office to check in with the team there, for the final checks. I left at 4:00 PM to be home early, spend that time with the kids until they where a sleep and went to bed at 9:15 PM tried to sleep for some hours. My alarm clock woke me at 10:45 PM: time to login and join the Call Bridge we set up for it in Microsoft Teams!

During the first checks my youngest son (4 years old) woke up and I tried to get him back to sleep. He needed a hug an a stuffed animal to keep him company.

11 PM: Christchurch New Zealand starts the event!

There was a venue in Christchurch that would start the event officially and we decided to be online for them to see if everything was running as expected and offer help if needed. They had to go through the keynote presentations first, select the teams and then start logging in to the challenges website with their account we created. The team that decided to be on stand by was me, René and Michiel.

Btw: the challenges has been open sources under the Creative Commons - Non Commercial license here.

We already pinged them to let them know we where available and asked them to keep us up to date on how the first hours went.

The keynote was finished at 12:00 AM and we watched the infrastructure with hawk eyes! And … everything worked! Crazy thing was that some organizers in Mexico and Redmond where actively testing with one of their teams, so they where on the scoreboard! Luckily Geert, Niels and Chris left us documentation in our wiki on how to remove them.

The teams in Australia where running by then and still everything look great on our end. Besides looking for issues there wasn’t really anything to do but check the social media channels and respond there as well (we got sponsoring from Walls_io to host a tweet wall here).

2:00 AM Crashing team members!

At 2:00 AM both René and Michiel really needed to go to bed and sleep to be worth anything in the morning. I was still rushing with excitement and stayed online if anything came up. I managed to do that until 4:00 AM:

So I messaged that to the venue organizers:

Gave a last status update to the team: And went to bed, to find my youngest son laying in my side of the bed! Seems like he needed more than just a hug and a stuffed animal to hug and my wife had laid him in our bed.

2019/06/15 Global DevOps Bootcamp (CEST)

I actually managed to sleep until 6:30AM, even with the youngest kid taking up all the available space in bed (my side only :grinning:!). Quick shower and off to the office:

In Hilversum we planned to set up the Command Center / War Room for the event. Everything was planned around this because we knew from last years that West-Europe and the America’s would take up the most resources and chatter to handle. Luckily for us: Jasper had taken it upon himself to host our own venue there. As usual, our hospitality team has the perfect setup to host a venue like this.

At the office, we have a Ripple maker to print on your cappuccino’s and Jesse needed a laptop to upload new GDBC images for it, so I helped him out with them:

We had 10 colleagues from Xpirit in the office to help during the day with receiving guests, proctoring the teams and running the venue in Hilversum.

9:30 AM Started the day with attendees arriving

10:00 AM Keynote started

We started the keynote in Hilversum! After a welcome note from René, Marcel and Mathias it was time to learn something about DevOps and SRE from Niall Murphy, Director of Engineering for Azure Cloud Services and Site Reliability Engineering.

We later found out his reaction about speaking in front of 10.000 attendees:

We kept monitoring during the keynote:

Image courtesy by Jesse Houwing, find more here

And during the rest of the day

Image courtesy by Jesse Houwing, find more here

Closing off in Hilversum with part of the team

At 4:00 PM we closed the venue in Hilversum and send our own attendees home. Some of us decided to stay at the office for a while and help the rest of the teams in the America’s that where still busy with their day. We needed to stay around our laptops and decided to get some pizza delivered instead of going to a restaurant. After the pizza we all drove home to check our feeds from there.

9:20 PM Closing off

I stayed online for a while at home until I could no longer. Niels volunteered to shut down the lights and man the fort until the last venue was done. My hero! After 7 hours asleep in 48 hours I really needed to get some sleep!!!

SonarQube setup on Azure App Service

2019-04-29T00:00:00+00:00

As noted in a previous post, you can host a SonarQube on an Azure App Service, thanks to Nathan Vanderby, a Premier Field Engineer from Microsoft. He created an ARM template to run the SonarQube installation behind an Azure App Service with a Java host. This saves you a lot of steps mentioned above! You can find the scripts for it on GitHub or deploy it from the big deploy on Azure button.

Photo by Vikram Sundaramoorthy

Several benefits you get from hosting SonarQube on an App Service:

You don’t have to manage a Virtual Machine anymore.
Setting up SSL becomes easier, thanks to the Azure App Service SSL being simpler (and free if you run on *.azurewebsites.net).
Using a KeyVault for your secrets is now an option, you can inject those values as environment options.
If you already have an App Service Plan, you can use that for hosting and cut down on your resources and cost (although, running on a burstable VM isn’t that expensive)

The biggest benefit is that the App Service already has Java installed! So no more downloading the JRE by hand, installing it manually and starting the SonarQube service by hand!

Setting up a new SonarQube server this way is a breeze. Updating it should be easier as well and that is the another big plus: you now have the ability to run the installation on a Deployment Slot, let it update the database and then switch to the slot. No manual updating anymore!

Set up

After creating the basic SonarQube App Service from GitHub or the ARM template, you need to create a new SQL database. Do note that you need to set the database to the correct collation for it to work: SQL_Latin1_General_CP1_CS_AS.

Creating a user

There are two options to create a new user in the database. You can create one like you would do in a full MSSQL server installation, by using the master database. Running this way on a Azure SQL Db has a couple of downsides:

You now have a dependency on the master database. Every new connection will have to do a lookup in the master database and then route you through to the database you want to connect to. This will create a potential bottleneck from the master database, if you are running a lot of connections or a lot of databases on the server.
Moving the database to another server takes more configuration, since the user configuration is not inside the database (see option two), but in the server itself.

Creating a contained user

A Contained user is a user account created inside the database itself, making it easier to move if needed. Run this statement in a query editor connected to your database. Of course, you can make the user a data_reader, data_writer or something else. Since I need the application using this connection also creating the tables, stored procedures, etc, I gave it DBO rights in this database.

CREATE USER [MyUser] WITH PASSWORD = 'Secret';
EXEC SP_ADDROLEMEMBER N'db_owner', N'MyUser'

Creating a regular MSSQL user

To create a regular MSSQL user if you do not want to have it contained in the database (see above), you can create a new SQL user by running this on the Master Database:

CREATE LOGIN SonarQubeUI WITH password='<make a new secure password here>';

Then create a login from the new user account by running this statement on the new database (you cannot use a use database_name statement in Azure SQL database, so you need to switch to it in the UI of your query editor):

CREATE USER SonarQubeUI FROM LOGIN SonarQubeUI

Settings

Copy these settings from the previous steps for next use:

SQL Server address (mysqlserver.database.windows.net)
DatabaseName
MSSQL Username and password

App Service update

In the Azure Portal, find your new App Service and navigate to Advanced Tools –> Kudu

Open a debug console:

And navigate to the configuration folder of your SonarQube and open the file sonar.properties:

D:\home\site\wwwroot\sonarqube-7.7\conf>

Update these properties:

sonar.jdbc.username=SonarQubeUI
sonar.jdbc.password=<your sql server user password>
sonar.jdbc.url=jdbc:sqlserver://mysqlserver.database.windows.net:1433;databaseName=SonarQubeDb

Note the portnumber and the property of databaseName instead of database (this was changed in SonarQube > 5.3). Also note that the database name your enter is CASE-SENSITIVE! SonarQube is running on Java, hence the sensitivity.

After setting these properties. Stop and Start the App Service again: SonarQube only reads in this file at startup and then initializes the new database.

Set an admin password!

By default, the admin login is admin/admin, so you want to change that ASAP (after setting up the database).

Errors?

If the SonarQube server doesn’t load after the changes, you can find the errors in the SonarQube folder here

/logs/sonar.log
/logs/web.log

Remember to restart the App Service after changing anything in the config!

Azure Active Directory

Setting up authentication for the users with Azure Active Directory is very easy, thanks to the work of the ALM Rangers. Follow the setup here.

Do note the Server base URL in SonarQube, I missed it the first time. By default, this is empty!

Go to Administration, Configuration, General –> Server base URL to set the url to match the url of the App Service.

Blocking anonymous users

If you don’t want to show the state of your projects to the entire world, don’t forget to change the default setting for this. Since the free edition is available for open source projects, this setting is on by default. You can find it under Administration:

Webservice Plan Scaling in Azure Machine Learning Studio

2019-02-25T00:00:00+00:00

I recently found that I had a web service plan running for my Machine Learning Studio (MLS) workspace in Azure. I was hosting some test webservices on it from a research session earlier on. The web service plan was not doing anything for me, but I did incur some costs running it. Since the default tier it picks during is already an S1, this can build up if you are paying the subscription yourself.

Photo by Agê Barros

Finding that web service plan started by looking at the resourcegroup the MLS workspace was created in.

The Azure Resource Group looks like this:

You can see the plan, but when you select it you have no extra options. No insights into the cost and no option to scale.

To do any of this, you need to look inside of the workspace in MLS. To find the web services you deployed, open Web Services:

Here, you can only see the actual deployed services from the experiments you made (I don’t have any running here). You still cannot see the web service plan!

The Trick

To actually find the web service plan and act on it, you need to go to a different interface for Azure ML: Go to: https://services.azureml.net/ to manage your deployed services, then go to “Plans” and there you can administer your plans.

Select the plan and click on Upgrade/Downgrade plan: Now, you can actually scale the plan to your needs.

If you aren’t using the service that much, there even is a Free Plan!

Fixing Azure Function Error RunResolvePublishAssemblies

2019-02-21T00:00:00+00:00

I ran into an issue with a new Azure Function I created: when I tried to run it, I got an error message about a RunResolvePublishAssemblies setting.

The target “RunResolvePublishAssemblies” does not exist in the project

Digging around the internet did not give an indication where to look. Most examples pointed to years old issues that indicated this message for dotnet core version 1.0. I am running a preview version of 3.0, so that could be the issue.

Testing creating another Function but with Visual Studio did have the same result: the error occurs there as well.

Finding the issue

Eventually I grabbed another working project with Azure Functions that we where running for a couple of months and went through all its settings to figure out what the issue could be.

Checked items:

The project SDK had been set correctly to Microsoft.NET.Sdk
The host.json pointed to a correct function host version (2.0)

Eventually I found the culprit! Because the global.json isn’t present, the dotnet core version wasn’t fixed to any version and that is why it uses a version that doesn’t work.

The fix

Add a global.json file into the root of the project folder with this content, pinning the sdk version to something working for Azure Functions, and build the project again.

{
  "sdk": {
    "version": "2.1.502"
  }
}

Setting up Azure Monitor to trigger your Azure Function

2019-01-21T00:00:00+00:00

I wanted to trigger an Azure Function based on changes in the Azure Subscription(s) we where monitoring. The incoming data can than be used to do interesting things with: keeping track of who does what, see new resources being deployed or old ones being deleted, etc. Back when I started working on this, there was no Event Grid option to use in Azure Functions, so I started with linking it to Azure Monitor events. I haven’t checked the current options, so I cannot compare them yet.

In this blog I wanted to show how you can do this, both by using the Azure Portal and the Azure CLI.

Architecture

To get Azure Monitor to send in the changes that we need to see, I use this architecture:

Steps

To configure Azure Monitor to send all activities into an EventHub and then into our Function, you’ll need to execute several steps.

Create a new event hub
Configure the Activity Monitor to send the changes into the event hub
Configure the Event Hub to send the messages to the changes function

Use the Azure Portal

How to do this manually via the Azure Portal is described below.

Create a new event hub to send the Activity Log to:

The default setting of 1 throughput unit is enough for this setup, as mentioned by Microsoft’s documentation.

Create the export of the Activity Log

Go to Activity Log, hit the export button:

PICK ALL REGIONS!! Most activities we want to see are GLOBAL and those would be missed otherwise.

Configure the event hub to send the messages to the Azure Function

Choose the EventHub entity you picked for the Azure Monitor to export the activities to:

You can add a consumer group to it, named ‘$default` by default.

Get the Access Policy

Go to Shared Access Policies and create a policy. We only need to have the Listen rights so we can listen to incoming events.

Copy either on of the Connection strings and configure the Azure Function host with it:

No need to restart the function app: the platform does that for you.

Use the Azure CLI

How to do this via the Azure CLI is described below.

Step 1: Create an Event Hub

First we need to check if there already is an Event Hub and if we could use it. Use these commands to list the available namespaces in the current subscription:

az login # login with an account that has the correct access rights to deploy resources on subscription level
az eventhubs namespace list

To search for the namespaces in a specific resource group and subscription, you can also add those parameters:

$resourceGroup = "<resourceGroupName>"
$subscription = "<subscriptionId>"
az eventhubs namespace list --resource-group $resourceGroup --subscription $subscription

You’ll get back [] as an empty JSON array indicating that there are no namespaces in that list.

If there is an existing namespace, you can use that to create the Log Profile with. Of course, you’ll need to check how this namespace is being used and where it sends the messages!

Create a new namespace with the CLI

Use this command to create a new namespace:

az eventhubs namespace create --resource-group $resourceGroup --subscription $subscription --name EventHubNameSpaceCLI

This will take a couple of minutes to complete and will return the newly created namespace. You can save the JSON response in an object with this command:

$namespace = az eventhubs namespace show --resource-group $resourceGroup --subscription $subscription --name EventHubNameSpaceCLI | ConvertFrom-JSON

Create a new EventHub in the namespace

$eventhub = az eventhubs eventhub create --subscription $subscription --resource-group $resourceGroup --namespace-name $namespace.Name --name EventHubCLI | ConvertFrom-Json

Create a new authorization rule in the EventHub

To be able to connect to the new EventHub, we need an authorization rule. That will have an Id that will be used as a connection string.

$ruleName = "<authorization rule name>"
$rule = az eventhubs eventhub authorization-rule create --resource-group $resourceGroup --namespace-name $namespace.Name --eventhub-name $eventhub.Name --subscription $subscription --name ListenRule --rights Listen | ConvertFrom-Json
# The Id of the rule that you need:
$rule.Id

Step 2: Create an Azure Monitor Log Profile

Reconnaissance

First, connect the CLI and check if there already is a profile available

az login # login with an account that has the correct access rights to deploy resources on subscription level
az monitor log-profiles list # list the current log-profiles

Note: this will run against the currently selected (or default) subscription. You could get one of two results:

No profile set for this subscription: []
There already is a profile for this subscription:

If there already is a profile, carefully read through the results to see if it contains everything we need. Note: there can only be one profile per subscription. If there is only one subscription with a profile and that is set to export to the correct EventHub, that is fine. From top to bottom this is the information you see in the image above:

Profile categories: We needs all of these, so if necessary, add the missing categories.
The SubscriptionId for this profile
The locations for which this profile will export activities. We needs at least Global and the regions that your resources have been deployed to.
The ServiceBusRuleId will indicate the namespace and authorization rule the activities will be send to. The name of this setting comes from the fact that EventHubs are build on top of Azure ServiceBusses. We needs this to be set correctly to send the activities to the EventHub that the function will listen to.
The StorageAccount indicate if this profile also exports the activities to a StorageAccount. This can be used for accountability reporting for instance: it contains all the information to find out what account executed what actions within the Azure Subscription.

Existing profile

If there already is an export configured to a Storage Account (but no ServiceBusRuleId), you can check the locations and categories, if those are sufficient, you can update the profile to also have a ServiceBusRuleId set to send the data into the EventHub you need.

If the locations or categories are not sufficient, you need to check the Storage Account that is being used if to see if it hurts if you send more information there. This depends on the setup that is used on top of that information.

Cannot update the existing profile?

If changing this profile is an issue, you are toast. You can create a new profile on a different subscription, but all activities on the default subscription will only be sent to the profile on that subscription!

No profile

If there is no profile setup, you can create a new profile with the necessary settings like this:

az monitor log-profiles create --name "default" --location null --locations "global" "eastus" "westus" "westeu" "northeu" --categories "Delete" "Write" "Action"  --enabled true --days 1 --service-bus-rule-id "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.EventHub/namespaces/<nameSpaceName>/authorizationrules/RootManageSharedAccessKey"

Important parameters for us:

Locations: the locations you want to send the activities for: always include global
Categories: what are the activity categories you want to receive?
Service-bus-rule-id: the EventHub rule to send the activities to.

Finding the service bus rule

az login # login to the account
az account set --subscription "<SubscriptionId you want to use>" # switch to the correct subscription
$resourceGroup = "<resource group name>" # name of the resourcegroup the EventHub is in
az eventhubs namespace list -g $resourceGroup # list all the eventhub namespaces in the given resourcegroup

Find the name of the namespace you want to use and use that in the next set of commands

$EventHub = az eventhubs namespace show -g $resourceGroup --name "<name of the namespace>" | ConvertFrom-Json
az eventhubs eventhub list -g $resourceGroup --namespace-name $EventHub.Name

Find the name of the event hub in the list and use that in the next set of commands:

$NameSpace = az eventhubs eventhub show -g $resourceGroup --namespace-name $EventHub.Name --name "<name of the eventhub>" | ConvertFrom-Json
# find all authorization rules
az eventhubs eventhub authorization-rule list --resource-group $resourceGroup --namespace-name $NameSpace.Name --eventhub-name $EventHub.Name
$AuthRule = az eventhubs eventhub authorization-rule show --resource-group $resourceGroup --namespace-name $EventHub.Name --eventhub-name $NameSpace.Name | ConvertFrom-Json
# search for the rule with the name you want
$ruleName = "<authorization rule name>"
$rule = az eventhubs eventhub authorization-rule show --resource-group $resourceGroup --namespace-name $NameSpace.Name --eventhub-name $EventHub.Name --name $ruleName | ConvertFrom-Json

Linking the EventHub connection to the Azure Function

The Azure Function needs to be told about the connection to the EventHub that it needs to listen on. For that, you need the Id of the authorization rule that we created in the previous section.

You can retrieve that Id from the rule we saved:

# The Id of the rule that you need:
$rule.Id

And then you can set that into the corresponding app setting for it:

az functionapp config appsettings set --resource-group $resourcegroup --name $functionapp --settings EventHubConnectionAppSetting=$rule.Id

The name of the setting we are changing here (EventHubConnectionAppSetting) needs to match the name of the Connection you gave the parameter on the function:

[FunctionName("EventHubActivitiesFunction"))]
        public static async Task Run(
            [EventHubTrigger("insights-operational-Logs", Connection = "EventHubConnectionAppSetting")]
            EventData eventHubMessage,
            ILogger log)
        { ... }

For more documentation about the Event Hub binding, check docs.microsoft.com.

Docker for Windows: fix unauthorized errors

2019-01-16T00:00:00+00:00

After installing Docker for Windows (recently renamed to Docker Desktop) I could not get the basic command docker run hello-world working. I checked my install, read more docs, got confused if it was in my Hyper-V setup, the networking stack in it, or something else. Finally a light bulb went off and I found the solution!

The issue

After installation Docker present you with a login screen. Since that login worked just fine, I’d expect that everything is setup correctly.

The next step is then to verify that you can run at least hello-world:

C:\Users\RobBos> docker run hello-world
Unable to find image 'hello-world:latest' locally
C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: Get https://registry-1.docker.io/v2/library/hello-world/manifests/latest: unauthorized: incorrect username or password.
See 'C:\Program Files\Docker\Docker\Resources\bin\docker.exe run --help'.

That message: ‘Error response from daemon: Get https://registry-1.docker.io/v2/library/hello-world/manifests/latest: unauthorized: incorrect username or password.’ can send you down a wild goose-chase to figure out what is wrong!!!

Since it tells you the call back to the docker registry is not authorized, you think you need to login again, even though Docker tells you, you are logged in…. Hmm, already strange, let’s try that nonetheless.

Logging in again

If I run docker login, it shows me the currently logged in user, which I did via the login interface after the installation:

Notice that I am using my e-mail address here. The login is just fine:

Still, calling docker run hello-world gives me the same error:

C:\Users\RobBos> docker run hello-world
Unable to find image 'hello-world:latest' locally
C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: Get https://registry-1.docker.io/v2/library/hello-world/manifests/latest unauthorized: incorrect username or password.
See 'C:\Program Files\Docker\Docker\Resources\bin\docker.exe run --help'.

After testing all sorts of stuff, reinstalling Docker (reboots required) and searching around some more, I finally got to a comment somewhere on another issue… And the issue is…. I am logged in with my e-mail address and not with my “insert curse” username!

Fixing it

Switching to use the username in Docker will switch the login for the session, and all of a sudden it just works!

Very odd that docker does accept both for the login here, while the e-mail address is not working. Somebody probably enabled that for the web front-end and didn’t test the CLI.

Maybe someone else will find this post when hitting the same issue and it saves them a lot of time!

Missing Azure Functions Logs

2019-01-13T00:00:00+00:00

I was testing with our Azure Function and had set the cron expression on the timer trigger to "0 0 */2 * * *", based on the example from the Microsoft documentation. When I went to the logs a day later, I noticed that some of the runs weren’t there! Photo by Emily Morter

Missing logs ??

I added a red line were I noticed some of the logs missing.

At first, I thought that the trigger wasn’t firing, or maybe something was wrong with my cron expression. I tested several other expressions and redeployed the function, but to no avail.

The search

Eventually, I found a comment deep down in a GitHub issue that actually pointed me in the right direction!

Application Insights Sampling

The logs you see in an Azure function are provided by Application Insights. Due to large data ingestion during our testing period (we had the trigger fire every minute to test with), I had enabled sampling on the Application Insights instance! That change was made after seeing a bill going upwards of € 500,- during the testing period 😄.

Finding the sampling setting

To correct the sampling settings (running once every two hours is significantly less data then every minute!), you need to go to the Application Insights instance.

Go to Usage and estimated costs and click on the data sampling button:

You can now change the sampling setting:

Wait for a couple of runs to execute and you can verify that it now shows all the logs again:

Azure Functions Connection Monitoring

2018-10-24T00:00:00+00:00

Last week I noticed our Azure Function wasn’t running anymore and I got a pop-up in the Azure Portal indicating that we reached the limit on our open connections. The popup message contains something like Azure Host thresholds exceeded: [Connections] and links to this documentation page. The documentation already hints at the usual suspects: HttpClient holds on to the connections longer then you’ll usually need. Since the whole Azure Functions sandbox has several hard limits, usage of an HttpClient in the default pattern is a common way to hit the Connection Count limit. The documentation also notes an example for a DocumentClient and SqlClient, although the latter already uses connection pooling.

NICK SELIVERSTOV

Searching around

When searching for this pattern, you can find a lot of examples that show you how the HttpClient does things and how to fix it (declare the client as static so it isn’t disposed on every function execution). You can even find examples from Troy Hunt and from the ALM Rangers.

Monitoring

Since we are using the Azure Fluent SDK to retrieve information from an Azure Subscription, and that instantiates several HttpClients inside of it, I wanted to start monitoring the connection count first, to see if it was a gradual ramp up (first time finding it was 24 hours after deployment of a change), or something else.

Monitoring would give a better idea overall about the issue and it’s part of the DevOps practice: you cannot improve things if you aren’t monitoring them first.

Searching around for monitoring

Since I couldn’t directly find how to monitor the connection count, I even opened an issue on the documentation repository to see if someone could help. Sure enough, someone responded with the location where to find the metric.

Since it took me a lot of searching and overlooking the first selection option, I am documenting the full process here.

Finding the metric

Of course, the information is only available when you are running Application Insights.

Go to the Application Insights instance connected to the Function
Go to ‘metrics’
Change the resource from your ‘Application Insights’ instance to ‘App Service’ to connect to the App Service that is hosting the function.
There is only one ‘metric namespace’ to choose from, and it is already selected.
Select the ‘Connections’ metric:

Fixing the used connection count

After monitoring we changed the instantiation of the clients we used from the Azure Fluent SDK to static instances and you can see that the connection count has improved a lot:

SonarQube setup for Azure DevOps

2018-10-20T00:00:00+00:00

During installation and setting up a SonarQube server for usage in an Azure DevOps Build I found some things that I didn’t remember from previous installation and wanted to log that in this post, so the next time I have somewhere to find these things in one place.

Updated 5-1-2019

Read the last section of this post if you want to use an even easier way of setting up and maintaining SonarQube: run it behind an Azure App Service!

Zach Lucero

Installation on Azure

For installation on an Azure environment I used the same Azure QuickStart ARM template I used before. Somehow, each time I need this template, something has changed underneath and I get to fix the template. This time the download URL for the SonarQube installer was changed as well as a new version got released. This has now been included in the template: because it is open source, I could find the places that needed to be updated and send in a pull request to Microsoft with the fix. I love open source! Such a pleasure to find an issue, look at the code and present a fix to the repository maintainer!

You can follow the usual steps from the ARM template: download and install the Java Development Kit on the SonarQube server, restart the SonarQube service and you’re up and running with the server side.

Things to know for next time

There are a couple of things that you need to think of when starting an installation yourself. The ARM template is already a great help in it, but you need to think of some other things. Those are mostly client side, so on the Azure DevOps Agent.

Bring a valid certificate

As noted before, the template uses a self signed certificate, which will not work with Azure DevOps: the tasks from the marketplace need a valid certificate that it trusts for the connection with the server. Therefore you need to provide a valid certificate and setup a DNS entry to match the URL in the certificate.

Download or install the SonarQube extension

Go to the marketplace and download or request installation in your Azure DevOps subscription.

The ‘Run Analysis Task’ has java as a requirement

This was a gotcha that I forgot this time: the Run Analysis Task has a demand requirement that it needs Java (specifically the Java Runtime Environment 8.0) on the agent. This also means that you cannot run it on a hosted agent: those do not have the JRE installed! Only the JDK is installed, which doesn’t add support for the java demand. I raised an issue on the hosted agent with a request for it.

Building on a new agent?

When you have a new agent you could install Visual Studio Community edition on it, that will provide you with the msbuild capability on the agent. This does not give you the tools to run the unit tests on the server, which is needed for the Run unit test task. These tools will provide SonarQube with the necessary information it needs to do, well basically, anything! You can install a licensed Visual Studio Enterprise on it, but then you need to update that license every once in a while. You probably don’t want that, because of the occasional error it will give you, for which you need to login to the server.

To help fix this, you can use the Visual Studio Test Platform Installer task to install all the tools VSTest needs to run.

Want CSS analysis from SonarQube

Out of the box, SonarQube can scan your CSS files for issues with over 180 available rules. To do this, the agent needs to have Node.js installed. This is already available on a hosted agent, but you cannot use that yet because of the dependency on JRE!

SonarQube CSS issue on large solution

Currently there is an issue on SonarQube with larger solutions or CSS files. The process seems to run out of memory somewhere. In the Azure DevOps Build log, you’ll see these as the last steps being logged:

INFO: Quality profile for cs: Sonar way
INFO: Quality profile for css: Sonar way
INFO: Quality profile for js: Sonar way
INFO: Quality profile for xml: Sonar way
INFO: Sensor SonarCSS Metrics [cssfamily]
WARNING: WARN: Metric 'comment_lines_data' is deprecated. Provided value is ignored.
INFO: Sensor SonarCSS Metrics [cssfamily] (done) | time=1937ms
INFO: Sensor SonarCSS Rules [cssfamily]

For now the recommended fix is: do not use the CSS analysis, which isn’t great, but better then the alternative: currently the Run Analysis Task just hangs until the maximum runtime of your build has been reached. Al that time, your build server will run at 100% CPU (if you have 1 CPU available, 2 CPU’s got me 50% utilization)! It took me quite some searching around to find this one, so it’s good to document it here.

The current fix is to start the analysis task with a parameter that redirects the CSS files to a non-existing analyzer:
d:sonar.css.file.suffixes=.foo or do that globally for your entire SonarQube server via the settings on the CSS analysis there (which would be easier if you have multiple projects with this issue).

Update (05-01-2019) Run it in an Azure App Service!

Nathan Vanderby, a Premier Field Engineer from Microsoft, has created an ARM template to run the SonarQube installation behind an Azure App Service with a Java host. This saves you a lot of steps mentioned above! You can find the scripts for it onGitHub.

Several benefits you get this way:

You don’t have to manage a Virtual Machine anymore.
Setting up SSL becomes easier, thanks to the Azure App Service SSL being simpler.
Using a KeyVault for your secrets is now an option, you can inject those values as environment options.
If you already have an App Service Plan, you can use that for hosting and cut down on your resources and cost (although, running on a burstable VM isn’t that expensive)

The biggest benefit is that the App Service already has Java installed! So no more downloading the JRE by hand, installing it manually and starting the SonarQube service by hand!

Azure DevOps Pipeline for GitHub Open Source Projects

2018-09-10T00:00:00+00:00

Microsoft announced today that they have a ‘new’ product: Azure DevOps! With that announcement came another one: Azure DevOps pipelines for GitHub open source projects with unlimited minutes! I wanted to see what the integration with GitHub would look like, so I tried it out.

Note: of course, you could already create pipelines for GitHub repo’s, but only inside of a VSTS account and not with unlimited build/release minutes! If you had you own private agent installation, you could build with that.

Now all open source projects can utilize the Azure DevOps pipelines!

Here are the steps you’d take to add a new pipeline from a GitHub repository.

From your GitHub repo, go to MarketPlace and search for Azure. The marketplace entry is named Azure Pipelines:

Click on Set up a plan:

Choose Free:

Choose to what repositories you want to enable a pipeline for. I have picked a .NET tool that I created to talk to VSTS.

One more authentication in GitHub to make sure you have the rights to enable the market place integration and you get send to Azure DevOps.

You are now in an editor to setup a new Azure DevOps account. If you already have one, you can choose that:
Next, Azure DevOps starts creating your project.

It will ask you which repository you want to use.

It detects the kind of project in the repository and gives you a default option based on that. You can choose other options, but I’ll just go with the .NET desktop build for this project, because that is what this project is.

A yaml file is created for your (named azure-pipelines.yml) which will be saved inside of your repository: that also means you now have a two-way binding between GitHub and Azure DevOps: it can also write changes back to the GitHub repository.

Save and run does exactly what is says, and if left with the default setting, will commit the new yml file to the chosen repository:

Now we have a running pipeline (free of charge! how nice is that!).

After the build is done and it succeeded, you get a summary page:

Status badge

If you want to show your build status on a project page, you can also get the build status badge for the project, right from Azure DevOps.
Note: you could do that for any project before.

Go to the build definition, find the menu, go to status badge:

And copy the necessary code/markdown into the place you want to show it.

And the end result inside the readme of my repository:

Email notification

Because I have an email address connected to my GitHub account, I also receive pipeline notifications by email:

Triggers

You have all the regular options for your pipeline available. For example: the new Azure DevOps project has been provisioned with a continuous integration trigger: this way, each commit will trigger a new build and will let the committer know how the build went.

Want more?

Since you now have an Azure DevOps project available, you can also enable other features for this project. This way you can leverage the powerful options Azure DevOps gives you.

Go to Project settings–> Services and enable the services you like to use.

Reload the page and the enabled services are available. Here I enabled the Boards service:

Azure error setting up export from Activity Log to Event Hub

2018-09-05T00:00:00+00:00

While working to setup an export from Activity Log to an Event Hub I got no response on a save action. This took some time to figure out why this happened, so I thought it could be helpful for someone else.

Photo by Adam Solomon on Unsplash

Issue when saving

When saving the export setting via this blade:

I got this error:

After scratching my head a little I checked the browsers console log:

Well, what do you know! Apparently the resource provider microsoft.insights hasn’t been registered yet! Would have been a nice message inside of the Portal itself, but at least now I can fix it!

The fix using the portal

Go to your subscriptions, pick the correct one and navigate to resource providers:

The fix using PowerShell

You can also fix this via PowerShell, as you can read on Pascal Nabers blog.

GDBC DevOps pipelines in VSTS

2018-09-02T00:00:00+00:00

Global DevOps BootCamp

In June 2018 I was part of the team behind Global DevOps BootCamp (or GDBC in short). The goal of the boot camp is to create a world wide event where everyone could get a taste of DevOps on the Microsoft Stack. It is an amazing combination between getting your hands dirty and sharing experience and knowledge around VSTS, Azure, DevOps with other community members. This years edition was great, with 75 registered venues worldwide and 8.000 registered attendees! The feedback we’ve got was wonderful: a true community event where everybody was asking for more!

Photo by rawpixel on Unsplash

Challenges

During the GDBC participants could try to complete 60+ challenges that the GDBC team had created. Those challenges where created by a team of more than 15 people (global organizers and venue organizers helped out), so they had quite a lot of changes during the preparation period. If you want to see and dive into the challenges, you can! The challenges have been open sourced for the community and can be found here: GitHub/XpiritBV/GDBC2018-Challenges.

The challenges would be created as work items inside a VSTS team, where participants would be assigned to, so they could collaborate on the challenges there. By dragging the work items to the “done” column, they’d indicated that they finished the challenge and received points for it, which they could follow on a leaderboard (more info in this post.

Pipelines for the Challenges

Early on, it was decided to save all the challenges in a git repository in VSTS, practicing what we wanted the participants to use (and because we are nerds!). The challenge definitions where saved as markdown files, with a yaml header to indicate if the challenge was a bonus challenge, if the challenge had additional help available and other properties. In 2017, the team only found out issues with the templates when the organizers ran their scripts to create the actual work items in VSTS.

To help the team with this, I started setting up a build and release pipeline in VSTS. Since there were a lot of moving parts with a complete team helping out, it would be helpful to run several checks during the changes to make sure all challenges where setup correctly.

Eventually I even created another pipeline to push the changes into a VSTS team, so the challenge maintainers could see the end result as soon as possible. All tThis really helped fixing issues quickly.

Pipeline on push

We had a tool available to convert the markdown, read the yaml headers, parse all links and images in the markdown and eventually create work items from that. After adding additional checks to it, it could be used to check the challenges for completeness. The tool, some PowerShell scripts, the challenges all had there own git repositories, so we could plug them in their own build pipelines: The tool was .NET core, the challenges would be published as artifacts and the same for the PowerShell scripts. You can see them linked as dependencies in the screenshot below.
In the environment was just one task: run the checks from the .NET core tooling:
The trigger for this was a pull request (PR) being created for the challenges repository, so the person creating the PR would get an email indicating the level of correctness of the challenges.

Pipeline to production

After the pull request was checked by the pipeline described above, then merged into master, the second pipeline would be triggered. This time a couple more steps where involved.

The artifacts where the same:

The tasks where as follows:

Convert the markdown file to json
This is the same tool that does perform the checks and saves a json for easy formatting.
Zip up the help directories
The help needs to be a link to a single file that the participants could request and open when they needed it.
Save the zip files into a DropBox share and get unique links for them. The unique link couldn’t be guessed, to prevent anyone from cheating.
Update the database for the scoreboard, with the correct points and help links for the challenges.
Clear the test team’s sprint from previous versions of the work items.
Create new work items based on the new challenge content.

After this pipeline completed, the whole team could check the end result inside of VSTS, by checking the setup for the test team.

These pipelines helped the team find issues early on, so we could make sure the quality would be where we wanted it to be.

Hopefully this post gives you more insights into some of the work we did to help a team out. If there are parts of this pipeline that you’d want more detail of, please let me know.

Retrieving AppSettings for an App Service with the Fluent SDK

2018-08-23T00:00:00+00:00

I am using the Azure Fluent SDK to retrieve information about the Azure setup and I wanted to retrieve the AppSettings from an App Service (or function app, or logic app). The simple solution didn’t work and searching around didn’t reveal any information about it. Finding something that did work (initial testing with a different service principle didn’t change the results), so here we are…

Photo by Osman Rana on Unsplash

Expected simple solution

Because all other information I needed came from a WebApp, retrieved by the default call, I expected that loading the AppSettings from the SiteConfig would deliver all settings:

IAzure AzureConnection = GetAzureConnection();
var webApp = await AzureConnection.AppServices.WebApps.GetByIdAsync(webAppResourceId);

var settings = webApp.Inner.SiteConfig?.AppSettings;
// settings is null because SiteConfig is null

Unfortunately, the SiteConfig stays completely empty. As mentioned I checked and tested it with another service principle with more rights, but that wasn’t the issue. Searching around on the web didn’t yield any new information.

Research the Http traffic

So next I checked the Http traffic being send to see if the information came back from the HttpCall. [Note: the Fluent SDK is a wrapper around a generated HttpClient against the REST api’s]

Traffic from the FluentSDK calls

First up was seeing what Http calls where being made. For this I used Fiddler.

HttpRequests:

GET /subscriptions/{SubscriptionId}/resourceGroups/{ResourceGroupName}/providers/Microsoft.Web/sites/WebAppName
GET /subscriptions/{SubscriptionId}/resourceGroups/{ResourceGroupName}/providers/Microsoft.Web/sites/{WebAppName}/config/web

You can see the loading of the WebApp information and next loading of the config for that WebApp.

PowerShell

Next I wanted to see if I could load the settings from a different stack, like e.g. PowerShell. Lo and behold, the PowerShell call did yield AppSettings!!!! WTF?!

From PowerShell:

$webApp = Get-AzureRmwebApp -ResourceGroupName $resourceGroup -Name $webAppName
$webAppSettings = $webApp.SiteConfig.AppSettings

HttpRequests:

GET  /subscriptions/{SubscriptionId}/resourceGroups/{ResourceGroupName}/providers/Microsoft.Web/sites/WebAppName
GET  /subscriptions/{SubscriptionId}/resourceGroups/{ResourceGroupName}/providers/Microsoft.Web/sites/WebAppName/config/web
POST /subscriptions/{SubscriptionId}/resourceGroups/{ResourceGroupName}/providers/Microsoft.Web/sites/WebAppName/config/appsettings/list
POST /subscriptions/{SubscriptionId}/resourceGroups/{ResourceGroupName}/providers/Microsoft.Web/sites/WebAppName/config/connectionstrings/list

Here you can see the same calls to the WebApp ad the Config, but then all of a sudden there are two other calls into the config endpoint! You can see it’s loading both the AppSettings and the ConnectionStrings.

Luckily the Fluent SDK is open source (hurray for Microsoft!) so I started searching for the class that would handle the json that was being retrieved from the REST call.

Solution

Searching in the Fluent SDK’s source code if eventually stumbled onto a new ManagementClient: the WebSiteManagementClient, specifically helps with the settings for those websites!

So the final code came down to this:

var WebSiteManagementClient = new WebSiteManagementClient(AzureCredentials) { SubscriptionId = SubscriptionId};
var settings = await WebSiteManagementClient.WebApps.ListApplicationSettingsAsync(resourceGroupName, appServiceName);

var appSettings = settings.Properties;

Using a self signed certificate on a SonarQube server with VSTS/TFS

2018-08-12T00:00:00+00:00

Recently I got a customer request to help them with provisioning a SonarQube server hosted in Azure. Fortunately there is an ARM template available for it: link.

I ran into some issues with the ARM template at first and then tried to use the new SonarQube server within VSTS.

TL;DR

I didn’t manage to get the SonarQube VSTS Tasks working with the self-signed certificate. I think it’s probably possible, but you’ll be much easier off

ARM template issues

At my first go with it, it took some time to figure out that the reason we couldn’t connect to it came from the way the self-signed certificate was created: the template didn’t create the certificate with a fully qualified domain name. A couple of years ago that would’ve worked, but with the tighter security rules in browsers that doesn’t work anymore. Luckily I changed that with a small adjustment in the script and a pull request later that problem has been fixed.

A little later I found out that SonarQube updated their download links, deprecating the older TLS 1.2 versions which gave another issue. Another pull request later and that is also fixed.

Java installation

Now that those issues have been handled, you’ll probably find that you’ve missed the comments in the readme: the ARM template cannot provision the Java JDK needed for installation, because Oracle will not let you download it from an open folder (like SonarQube does)!

For this step you’ll need to RDP into the server and install the JDK by hand.
Do note: don’t forget to change the default passwords for logging in to the SonarQube installation!!

After that you’ll find out that the template provisions an IIS installation to host the SSL certificate and then be the proxy for the SonarQube server.

Using the SonarQube server in VSTS / TFS

When you have the server up and running, you’ll want to use it in VSTS. If you start adding the necessary steps to your build (find out more about it here, you’ll find out that the builds will fail with some obscure messages connecting to the SonarQube server. If you are using a private agent, you can log into the server and try to remediate these issues.

First, you’ll hit it in the “Prepare analysis on SonarQube” step. Thinking it runs on the agent server on Windows, you can trust the server’s certificate in your local certificate store, using the certificate snap-in. Double check the user the agent is running on or trust the certificate machine-wide. Don’t forget to check your proxy configuration if you have one in between!

Unfortunately: this doesn’t work! Nest, you double check and find out a part of this step actually creates a local Java Virtual Machine JVM that has its own version of the local certificate store. To add your own certificate in it, you can follow the steps from here.

Next, you’ll find that NodeJS is used to send the requests to the SonarQube server! Great, now that also has its own version of the trust chain setup…

End result

And this is where we couldn’t figure out how to include the certificate here. Given the time that it already cost to get here and the expectation that a hosted agent could still be needed (where you cannot trust your own certificates and need to have an official one), we stopped searching around for the solution and got an actual certificate. That prevented all these errors and also enables the usage of a hosted agent.

VSTS Personal Access Token for an Agent: Revoke after use

2018-08-03T00:00:00+00:00

Today I was listening to RadioTFS episode 163 on my commute, with guests Wouter de Kort and Henry Been. During the show Wouter mentioned that he always revoked his VSTS Personal Access Token after using it, especially when used for a Build Agent.

Apparently the PAT is only used for the initial authentication to VSTS/TFS and after that it isn’t needed anymore! That indeed means that you can revoke the token after it has been used and that you don’t need to keep the token around. Until this day, I had always copied the token into keepass for keepsake, but I don’t need to do that anymore. This will also save me from seeing the expiration date and wondering if I need to update this token!

If I ever need to register another agent, I can always create a new one. Revoking the old one also means that it cannot be used anymore, so that’s also less of a security concern then :-).

Off course I had to test it first to see if it actually works, but it did :-). So wanted to save this for later in this post.

Chocolatey Server in Azure

2018-07-20T00:00:00+00:00

Recently I wanted to demo an example of how you can rollout Chocolatey packages via your own choco server. Sometimes we cannot save every binary in VSTS to use it in a pipeline as an artifact and therefor I needed a different artifact server. Chocolatey provides a NuGet wrapper around binaries that you can easily track different versions with.

Since that worked out an I now have a local document with the necessary steps to do so, I wanted to share that for later reuse.

Of course, Microsoft just announced that they started working on a different artifact server in VSTS (called Universal Package Management), next to the already available NuGet / npm / Maven package management.

Since I needed something to show, I started researching how you can do this with your own Chocoserver. Unfortunately, the installation for Choco.Server on Windows consists of multiple steps that can take up some time. That just doesn’t feel right, so I wanted to see if I could wrap that inside of a PowerShell script being triggered from an ARM template and Azure Automation DSC. I couldn’t seem to find the time to really start on it, but fortunately enough, my colleague Reinier van Maanen needed a bit of a challenge and picked this up. You can find his ARM template here, together with the necessary PowerShell script.

Beneath you can find the steps to get a working Choco.Server and Client up and running with a first basic package.

Install a Choco.Server

First, you’ll need a server to host the packages. You can host that with a couple of steps thanks to Reinier.

Git clone https://github.com/rvanmaanen/arm.chocolateyserver.git
cd arm.chocolateyserver
Connect-AzureRmAccount
Open parameters.json 
 - change dnsNameForPublicIP (max. 15 characters!)
 - change allowRdpFromThisIpAddress
.\Deploy-AzureResourceGroup.ps1 -ResourceGroupLocation "West Europe" -StorageAccountName "ChocoARM"

~~Do note that the ARM template will report failure on the DSC step. To still get a working server you’ll need to log in to the new VM and execute the last 5 lines by hand.~~ Update: thanks to Reinier this is now fixed! The ARM template now uses 2 steps to make sure the IIS management cmdlets are available for the second step. You can read how he fixed that here. To check it, you can navigate to http://localhost/. We are aware of the use of http, we need still to add that step to the script. We decided we could live with it for a small demo and blocking it from your own IP-address.

Create package on different machine

Install choco (elevated PowerShell) with this command:

Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

You are now ready to create new packages and push it to the Choco.Server. I did this on my own laptop, so therefor I am using the public IP address of the Azure server. To wrap the files Chocolatey uses the well known NuSpec file, known from its use in NuGet.

choco new testpackage creates the default nuspec in that folder
change it, add notepad.exe into the \tools directory
remove ps1 files from the tools directory
choco pack on the folder containing the nuspec file.
choco push testpackage.{your.version.number.here}.nupkg --source http://{public.ip.of.chocoserver}/chocolatey --api-key={please update your key in the web.config} -force

Note: the -force is necessary because we are using http instead of https.

Client machine (in the same resourcegroup/nsg/network!)

For demo purposes I created another Azure Windows Server in the same resource group, connected to the same network so the servers could easily connect to each other.
On that extra server, verify that it can see the Choco.Server by navigating to http://local.ip.of.chocoserver/chocolatey to see if you get a valid result.
Then you can install the chocolatey client, link you own server as a source and use that server to install your own package.

Steps:

Install choco with the PowerShell command (see 'create package')
choco source add --name=internal_machine --source=http://local.ip.of.chocoserver/chocolatey
disable Internet Explorer Enhanced Security Configuration (IE ESC) to prevent the http page from loading (you are on a server). This step should be simpler when you are using https (as we should).
choco install testpackage
Files are copied into C:\ProgramData\chocolatey\bin\ by default

All in all, I found the process pretty straight forward. Now to start using the Choco.Server and push new versions to it!

GDBC DevOps on the Leaderboard

2018-06-16T00:00:00+00:00

Global DevOps Bootcamp

On the 16th of June 2018, Xpirit and Solidify organized a global event around the topic of DevOps and improving your release cadence. It is an ‘out of the box’ event with a lot of self organization where people around the global gathered on their free Saturdays to learn something new about DevOps.

People interested in hosting a local venue went to the site https://globaldevopsbootcamp.com/ and started from there. Anybody, anywhere could host an event. Eventually, 76 venues registered with 8.000 participants!

The teams from Xpirit and Solidify provided completely configured VSTS accounts, with challenges, webhooks, users and a filled git repository:

Showing how we worked

We got several questions during the event how we organized the leaderboard application and some participants where astonished we used the same tools for this as they had been working on today! That’s why I wanted to share some of the stuff we did and what happened during the day!

Getting points for work items

Every time a workitem’s state changed, a preconfigured webhook was triggered. When the team moved a work item to state ‘done’, they would get points for that work item. Points were depending on the amount of work necessary to complete the challenge. The team could also request help, by adding a tag to the work item. Doing so would cost them half of the points for that work item, but also provided a link to a zipfile containing step by step instructions.

Leaderboard application

To organize all this, Peter Groenewegen and Geert van de Cruijsen created a leaderboard application for last years event. You can find the source for it on GitHub: github.com/xpiritbv

We build on that for this years version, where Peter has added the webhook callback so VSTS could tell us when a workitem changed.

Off course, this .NET core application is hosted in Azure on an App Service instance backed by an Azure SQL Database. The code is on GitHub and we created a build and release pipeline in VSTS:

This build would trigger when a pull request got merged into master and after successfully running all unit tests would trigger a release.

DevOps for the leaderboard application

To check the application during the event, we created a dashboard to monitor the performance of the application and the database.

During the event

The event started everywhere at 10:00 AM local time, so New Zealand and Australia got to be the first to use the application. We were a sleep during most of that timeframe, but we checked during the start to see if there where no errors. Luckily, that wasn’t the case!

Checking issues together with Geert, image by Jesse Houwing.

EMEA region starting with the challenges

When we started in Hilversum, The Netherlands, 10:00 AM CET, we noticed the average page load time climbing up and up. Apparently, a lot of venues in the EMEA region where using the leaderboard and where updating the workitems, causing some load on the webhook as well!

We quickly scaled the App Service Plan and the Azure SQL Database to fix the page loads. This was important, because the webhooks where also on the same endpoint. When a webhook fails a couple of times, it will be disabled! That would mean teams not getting new points for the challenges they would complete!

Thanks to the power of Azure and our team being enabled to fix things while running, we mitigated the issue.

Failing webhooks

A couple of hours later, someone spotted errors in Application Insights for the calls into the webhook. Checking the callstacks and exception messages, we found the culprit. We made sure we checked the workitems coming in to find their correct tags so we could find the points they where gathering, but we didn’t anticipated our participants creating their own workitems and tasks to distribute the work between them! This meant the webhook was being called with workitems that didn’t have any tags! So: a simple edge case we missed in our unit tests!

A commit, push, pull request, review and merge later, the CI/CD pipeline we created kicked in and the application was pushed to production! Just like the teams where learning to use today!

Other issues

Somehow, some teams managed to trigger the webhook in such a way that we got a duplicate record in the database. We found out about this, again through Application Insights, and fixed the issue quickly. How they managed to trigger this, is something we will look into before using the application again.

Can you spot where we scaled the database?

Usage throughout the day

There was a noticeable bump for the period EMEA region was live:

Closing

All in all, I think we managed to keep the leaderboard and the webhooks in the air without our users noticing much of these issues. Great to see what a team can do when the have control over their entire pipeline, even into production!

That’s why we need to keep repeating the message: empower your teams!

DevOps and Telemetry: Support on the supporting systems

2018-04-02T00:00:00+00:00

Note This is part 3 in a series of posts about DevOps and the role of telemetry in it. In part one I described the reasoning behind the series and explained how I started with logging (usage) telemetry for a SaaS application. You can read part 1 in the series here. In this post I want to explain about the next step: logging information about the systems that support the application: servers, database, storage and anything that comes with that.

Series overview

Part 1 - My journey with telemetry and starting with logging
Part 2 - Supporting systems and how to gather that information (this post)
Part 3 - Supporting the support systems

SSL certificates - validity

SSL certificates: industry standards on encryption

SSLLabs

Security headers

https://securityheaders.com/

DevOps and Telemetry: Supporting systems

2018-03-22T00:00:00+00:00

Note This is part 2 in a series of posts about DevOps and the role of telemetry in it. In part one I described the reasoning behind the series and explained how I started with logging (usage) telemetry for a SaaS application. You can read part 1 in the series here. In this post I want to explain about the next step: logging information about the systems that support the application: servers, database, storage and anything that comes with that.

Series overview

Part 1 - My journey with telemetry and starting with logging
Part 2 - Supporting systems and how to gather that information (this post)
Part 3 - Supporting the support systems

Telemetry on supporting systems:

For a SaaS application running on an Azure Web Service Plan, you can use a lot of components, so I’ll focus on stuff I have used in the past and are most commonly used:

App Service Plan
SQL Database
Blob Storage
This should be enough to give a broad overview of the standard monitoring options.

Available information from Application Insights

I’ll start at the point from the previous post where I added Application Insights to the application: I had (request) telemetry available from that, with visibility into all calls to depending services. From this, I started creating a dashboard to bring all this information together in one view, so that it could be shared with the team responsible for building and running the application. From Application Insights, I always add the application map to the main telemetry dashboard for the operations team. In that map, you’ll get a quick insight into ALL aspects of the application, including the parts that are suffering (performance wise).

App Service Plan

The default telemetry on an App Service Plan level are pretty basic. You can see CPU and Memory usage of the hosted web server, with data in and out.
I do check this ever so often, but not on a daily basis. If you have more than one app service on the same plan, then you can check to see if there is an application that is hogging resources. To do so, navigate to: App Service Plan –> Diagnose and solve problems –> Tools - Metrics per instance. Note this menu item keeps moving around. I think this is the third location I’ve seen this item appear. Search around if you cannot find it.

App Service

On the App Service level itself, you get most of the same information as on a Plan level, but only for the application you are viewing. Additionally, you have information available like: number of requests, erroneous requests (can be filtered per status code (4xx and 5xx)) and CPU utilization of this app service. See the information about the metrics from Application Insights for more (and better in my opinion) informational data.

Database

One example of the additional telemetry data that I got from Application Insights was for the Azure SqL Database that was used. Thanks to Application Insights, you’ll get the following (very handy!) information:

Query duration
Number of times a query has been run against the database
Where that query has been called from (e.g. in your application code) Some of this information is also visible from the Azure SQL Database itself, but I think Application Insights provides nicer reporting on it.
This came in very handy when searching for performance issues throughout the application. I made this information available in a separate dashboard to be used for hunting down those issues, because I didn’t need them for the operational overview that I made sure to have always visible on a separate screen.
Note: some of this information is also available from Query Insights.

Database Transaction Units (DTU)

Next up is the standard database telemetry that Azure already logged for us: the most interesting parts are database size, maximum database size and even more important: DTU measurements! For a more in dept explanation about DTU’s, check Microsoft’s explanation here.

Picking a good service tier for your database can be tricky and depends heavily on you usage patterns. If you have spikey traffic (think long running transactions or daily ETL tasks), you can consider changing the database tier up front or over provisioning (can be costly). Of course, you can think of other tactics to prevent running into the Max DTU for your service level, like caching (always tricky!), offloading heavy writes & updates to a different storage mechanism (or sticking it in a queue!).

Anyway: make sure to test with a representative load before running it in production! Seeing a chart link above is not great in production.
Changing the service tier is possible, but you really do not want to do this during a long running transaction. This will only make your performance issue last longer :-).

Blob Storage

On blob storage you’ll want to start with monitoring these metrics: Ingress (traffic into the storage account, so uploads), Egress (traffic out of the storage account, so downloads). Next up will be the latency and number of requests.
Depending on the layout of your storage account, you might want to dig deeper in the used ‘folders’ or ‘containers’ inside of the storage account. I had once a multitenant SaaS application that billed the tenants for (parts of) the used storage, so I talked to the storage API to get file and size measurements per tenant. To start retrieving that information, you can look here.

Next up:

The next post in this series will be: ‘Part 3 - Supporting of the support systems’. I think it could be helpful to have a (start of) a list of items I started tracking that come up when checking the support systems like blob storage and SSL certificates. Sometimes these parts of the system are forgotten, until it is to late! I’ll dive into that in the next post.

I’ll update this post with a link when Part 3 is available.

DevOps and Telemetry: Insights into your application

2018-02-23T00:00:00+00:00

I like to make work and performance visible for teams by giving them the necessary information to gauge the performance of a system, both on a daily basis as well as over time. This will help them to find less performant parts of the system, or checking engagement statistics. Sometimes I feel this part of the process is forgotten when talking about DevOps these days. People will focus on bringing the different teams together, create vertically oriented teams and changing the static mindset into ‘the DevOps mindset’.

In my opinion, telemetry Is a very important part into the DevOps journey. Without it, it’s hard to prove anything: whether it is the effectiveness of your organization into rolling faster, or checking usage statistics on new features. Step one should be to get a solid understanding of (a lot of different aspects of) your application.

This series of posts will go through my own journey in this aspect of DevOps.

When I was the team lead for a multi-tenant SaaS product we hosted for our customers, I made sure we enabled our product owner, management and the dev team to get important insights in the availability, performance and usage of our systems.
As we followed the adagio: ‘You build it, you run it’, I made sure it was possible to view the current (near real-time) performance on a dashboard and more business information on a daily basis, like the number of (new/recurring) users inside a Business Intelligence product. We even tracked supporting systems to make sure everything was up to our standards (think e.g. of SSL certificates via SSLLabs for validity and key strength), SAS Key validity and more.

Series

Part 1 - My journey with telemetry and starting with logging
Part 2 - Supporting systems and how to gather that information (this post)
Part 3 - Supporting the support systems

Part 1 in the series

In this first post I’m exploring the information I found necessary to have and will follow this up in another post with several methods to gather that information. The closing part of this series will be an overview of tools that could be use to gather the necessary information and display inside of a dashboard.

System

I’ll base all examples in this series on a multi-tenant SaaS application, which was build with ASP.NET MVC. We hosted that application in Azure on an App Service Plan, making use of SQL Database, blob storage (blob/table and queues), key vault and more.

Part 1

What information do you need to check?

Are we up?

Maybe the first information we started gathering was information about the incoming requests to the web application: raw number of requests and request errors, together with information like server side duration, user agent, request path, tenantid and userid.
At first we gathered this information by just logging it into the database after a request was ended, but after a while we learned that this was putting quite a load on our database, locking tables on inserts and live checking them in production is not a great idea: performance will tank if you have a small database tier and > 800k entries in the table per month. We could start to shard that table, but even then it would still be a performance issue with our growing customer and user base. Remember, inserting new records in that table will lock it and new insert will have to wait.

We decided to test and later implemented logging that same information into table storage, with a sensible keying strategy that would effectively shard that information inside the table storage. That way, inserting and reading that data would no longer hit our database performance.

This information could be loaded into our reporting engine without drawing any performance on our database, which was a big plus for us. Since loading the data from table storage was also fast, we could load it on demand to generate new reports & check performance issues on the fly (e.g. to see if it was an issue for one user, one tenant or all users).

Issue with this method

Regularly loading and testing this information proved us that the system was running in a normal way, but we only saw that the application wasn’t available to the end user if there was limited or no data available: it will then take a while to see what the cause is and perhaps you didn’t even notice that the information was lacking if you aren’t checking for the volume of new data. You might even NOT notice anything, because you might think that there are less or no active users using the application!

Next step - rolling our own solution

We then started with the search for a monitoring platform to test the availability of the application by testing the login page. Since the login page (usually) doesn’t actually hit the database, we even implemented a specific endpoint in the application that would check to see if the database was available and up to date with the latest schema. Of course, all other elements of the application (blob storage with its own parts, key vault, etc.) would also have to be checked, which could lead us into creating a performance bottleneck in our own system if we’d hit that endpoint a lot. Nevertheless, because we already had such a monitoring solution in place, we added a web job to it that we could configure with url’s to check for at least a HTTP OK (200) result on that endpoint. When a couple of checks would fail, the system would trigger an email to the admins, telling them to start their analysis on the system. We figured that having a solution in place would be better than none. In the mean time, we’d search for a better way to monitor the system.

A better way (for us)

Since rolling your own solution can take quite some time to get things right, we also looked into other available ways to get the necessary insights into our application. Since we where running on Azure, and even with ASP.NET, implementing Application Insights was a low friction step. These days, you can start by adding Application Insights as an extension to your web app, so without even changing the code, but back than we had to just implement the NuGet package and add our telemetry key to our configuration. Adding two or three lines of code in both ASP.NET and a central JavaScript location and we where up and running.

After the next rollout, we were gathering the (basic) data so that Application Insights could give us:

Easy and high level tracking of the performance of our application.
Finding errors in both our own requests, but also to our dependencies like SQL server, blob storage, key vault, etc.
A very easy way to initiate alerts to our admins: even adding sending an email to a distribution group is a simple way to get started.

With this information available, we created dashboards inside the Azure portal that we’d share between the team members that had access to the Azure subscription.

Since we didn’t add any additional data into Application Insights like tenantId or userId for better correlation to those levels, we needed to rely on Application Insights way of calculation user and session counts. Since these are gathered based on application insights own methods, we saw a difference with our own unique user counts. Application Insights would see a user logging into a different browser or device as a new user/session and count it thusly. Without a userId, it couldn’t relate the information back to the same user (of course!).

Next step: log additional information

The next step would be to add that missing information into Application Insights, as well as any specific activities we would like to track. You can think about actions like adding/deleting a product to a basket, (not) paying for the items, etc. These items can then be used inside reporting. Information about adding this information can be found here.

Next step: long term data retention

After getting some trust in the information (checking it with our own logs), we can then start using the information of Application Insights into our Business Intelligence platform (just pick one that’s available to you and you can actually work in) and provide information about our application over a longer period of time. The Application Insights dashboards inside of the Azure portal would usually be shown for the last hour and the last week. If you want to track usage over a longer period of time, you’ll need to export it and use it in another place. One way to do so is exporting the logs into a SQL database and then consuming that data into a dashboarding solution of your choosing.

A way to do so into Power BI is described here.

Next up:

The next post in this series will be: ‘Part 2 - Supporting systems and how to gather that information’. I think it could be helpful to have a (start of) a list of items I started tracking about supporting systems. What parts of your infrastructure does an ‘Ops’ team really care about? I’ll dive into that in the next post, which you can find here.

VSTS Bulk Change WorkItemType

2018-02-23T00:00:00+00:00

Update process templates

Recently I had a customer request to update their process definition in Visual Studio Team Services (VSTS). They had 30+ different processes migrated from TFS (Team Foundation Server), so they were all Hosted XML processes.

Somehow they had the process setup like this:

Epic –> Product Backlog Item

Which they requested me to convert to this:

Epic –> Feature –> Product Backlog Item

In TFS you would grab witadmin and change the process in a pretty straigthforward manner (once you’ve figured out all places you need to update!). Unfortunately, you cannot run the change commands in witadmin against VSTS, only the list and export commands will work: see here. Microsoft is working on a REST api to perform administrative tasks against VSTS. Luckily for me, they have also wrapped the REST calls in a nice C# NuGet package (link)!

Making changes to the process template isn’t available (yet?), although there is a strange method available named ‘UpdateWorkItemTypeDefinitionAsync’ in the ‘WorkItemTrackingClient’. The only info I can find about this is here, which seems to indicate that you can only update (maybe add) a specific Work Item Type. Since I also needed to update the tree structure in the ProcessConfiguration.xml file, I still need to export the process, make the necessary changes in the xml files, zip it back up and upload the file back into VSTS.

Updating work items to the new type

After doing so, the request was to convert all the old Epics to the new Features. Off course, you can do this by using a query on the Epic work item type and using the UI to change them to Features, but this would take a lot of manual actions to do.

Luckily you can do this with the TeamFoundationServer.Client NuGet package! When you start looking into this, I can highly recommend using Microsoft’s GitHub repo containing a lot of samples here.

It took me some figuring out to get a good workflow in an application, so I have made the tools source available on my GitHub account.

Using excerpts in Jekyll

2017-12-29T00:00:00+00:00

I wanted to include at least some more information in the index page of my blog instead of just the publish date and title, so I searched around for some help to include an excerpt in Jekyll and found some help on this blog.

The solution was very straightforward, but I’ll include it here for future reference.

Index page

In the index page, you can search the content of a post, check for specific tags and use the text between them:

<!-- index.html -->
<p class="post-excerpt">

</p>

Note: if the specified tags aren’t found in the content, the first 20 words will be used.

Posts

In a post, you can now include the excerpt tags to add a specific excerpt:

<!-- _posts/some-random-post.html -->
<p>
Here's all my content, and <!--excerpt.start-->here's where I want my summary to begin, and this is where I want it to end<!--excerpt.end-->.
</p>

Trying out Jekyll

2017-12-17T00:00:00+00:00

Trying out Jekyll on top of GitHub pages as a new blogging platform. For know, I just needed a simple way to create posts, but add some stuff I am missing on my current method (WithKnown), like RSS and Google Analytics.

So far I like the easy setup (like: no installation whatsoever!) and the fact that it uses Jekyll to generate static pages.

I started with the excelent guide of Jonathan MCGlone.

Azure App Service - a quick way to take your app Offline

2017-11-19T00:00:00+00:00

After searching for the third time on how to do this, I thought it would be time to write about this here 😬.

If you have an Azure App Service that for some reason should just display a message to the user, indicating that it isn’t available, you can do this.

I have had several reasons to do this:

single app service host, without a deployment slot and a big db update ( > 10 minutes db hitting a spending limit and no wish to update the limit moving dns names and certs between app service plans (were recreated with a better name)

App_offline.htm

Create or place a new file in your webroot (wwwroot) named “app_offline.htm”. I usually create it using kudu. The appservice will see this and online serve this file, as long as it’s there. Note: the contents of the file can be anything, including css and images from anywhere else in your wwwroot.

DotNetCore: Adding HTTPS to your MVC webapp

2016-08-20T00:00:00+00:00

I wanted to use https in my dotnetcore application (v. 1.0.0-rc2-final) and had to dig around the web quite a bit to find the most recent and working method to accomplish this. Eventually a link in the MVC github site lead to an example how to fix this (link).

First, the most easy way I've found to do this, is to add some custom middleware for redirecting all http requests to https:

public class HttpsRedirectMiddleware
    {
        readonly RequestDelegate _next;
        
        public HttpsRedirectMiddleware(RequestDelegate next)
        {
            _next = next;
         }

        public async Task Invoke(HttpContext context)
        {
            if (!context.Request.IsHttps)
            {
                HandleNonHttpsRequest(context);
            }
            else
            {
                await _next(context);
            }
        }
        
        void HandleNonHttpsRequest(HttpContext context)
        {
            // only redirect for GET requests, otherwise the browser might not propagate the verb and request
            // body correctly.
            if (!string.Equals(context.Request.HttpContext.Request.Method, "GET", StringComparison.OrdinalIgnoreCase))
            {
                context.Response.StatusCode = 403;
            }
            else
            {
                var newUrl = string.Concat(
                    "https://",
                    context.Request.Host.ToUriComponent(),
                    context.Request.PathBase.ToUriComponent(),
                    context.Request.Path.ToUriComponent(),
                    context.Request.QueryString.ToUriComponent());
                context.Response.Redirect(newUrl, permanent: true);
            }
        }
    }

I've added this right before the Mvc call:

services.AddMvc();

Links to Visual Studio Extensions

2016-06-04T00:00:00+00:00

Some links to important Visual Studio extensions for later reference:

Open Bin folder Visual Studio Extension: Visual Studio Gallery
Wakatime: wakatime.com
SlowCheetah Visual Studio Gallery
SonarLint sonarlint.org
T4MVC github.io

Not geting new windows 10 preview builds after reverting to an older build?

2016-03-03T00:00:00+00:00

If you previously had a new Windows 10 preview build installed in your computer and then you reverted back to an older build, you could lose access to the new build where the new build is no longer offered as an upgrade option. If you want to install that build again, deleting that build number from the list in the Registry Editor will restore the ability to upgrade.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WindowsSelfHost\Applicability\RecoveredFrom 

Source

Integrate SonarQube with TFS 2015 update 1

2016-01-23T00:00:00+00:00

While migrating CI stuff from Jenkins into TFS 2015 SP1 I ran into this blog post from Microsoft explaining how to include SonarQube runs information in the TFS Build Tasks. We have been running SonarQube on our projects for about a year now to gain some insights into Code Coverage and basic code smells. I sure don’t want to lose the information Sonar gives us.

The problem was that we weren’t getting any results into SonarQube. The default way TFS runs the UnitTests is that it generates a trx file with coverage information in it. Unfortunately,when you have a VS Professional license, it will not generate the code coverage. You’d need a Enterprise license for it!

In Jenkins, we used OpenCover to generate the coverage data, which integrates into the calls to the SonarQube runner. Its open source, which is a big plus.

Step 1:

Create a new location on the server to place the coverage reports into. I’ve used C:\OpenCover\ for this, and let OpenCover generate a file for each assembly we’re testing, using the name of the assembly as the filename for the xml report.

Step 2:

Add arguments to the SonarQube build step to tell Sonar where to find the coverage report. This needs to be done in the start step:

Add this argument to ‘Additional Settings’:

/d:sonar.cs.opencover.reportsPaths="C:\OpenCover\SolutionName.AssemblyName.UnitTests.opencover.xml": Sonar settings

Step 3:

Add in an additional Command Line build task to start the OpenCover tools. As you can see, I’ve added this after the build step. I think Sonar needs the build step for some data, but I need to test if it still works when we move that step out of the Sonar Start and End steps. Running the OpenCover tool should be enough, but I’m not sure of that.

Add these arguments:

-output:"C:\OpenCover\SolutionName.AssemblyName.opencover.xml"

-register:user

-target:"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe"

-targetargs:" \"{path_to_tfs_workfolder}\bin\Release\SolutionName.AssemblyName.UnitTests.dll"

Step 4:

Run the build and see some coverage results!

Windows 10: Enable 8.1 fly-out style WiFi / VPN menu

2016-01-09T00:00:00+00:00

I’m not the only one who is annoyed by the new Windows 10 way to connect to VPN connections. The new route takes a lot of new clicks, just to connect to a VPN! The old Windows 8 style was a lot faster. Since I frequently change connections at work, It’s a recurring annoyance everyday :-(.

Today, I’ve found out that there is a simple registry setting to revert the dialogs to the old Windows 8 style.

Registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Control Panel\Settings\Network\ReplaceVan

Values:

- Default
- Network settings in settings app
- Windows 8 style

Note: you’ll need admini

strator rights to change the settings, or use a tool like RegOwnershipEx to change the settings. Take ownership of the folder and then open the register editor to change the value.

Thanks to Nick Craver, for mentioning it was possible to change it.

Visual Studio 2015 update 1 - Service update 1

2016-01-03T00:00:00+00:00

Just putting this out here for future reference: there is a service update for VS2015 update 1 to fix some issues. I needed this update to fix an error in VS with T4MVC.

Knowledge base article: Link

WakaTime

2015-05-19T00:00:00+00:00

https://wakatime.com/dashboard

Plugin for Visual Studio (and other editors) to log hours spend in the editor. Free account only retains the information for a couple of weeks and gives you an overview of time per project/solution and per language. Really neat to see those stats.

Currently I have this extension installed on both my laptop, pc and in a VM designated for SharePoint development.

ASP.NET Demoproject started

2015-05-19T00:00:00+00:00

Tonight, I had the idea of starting a new project to test and demo new area’s and features of ASP.NET. Initially it will be a project to test ASP.NET MVC 6 from Visual Studio 2015. I’ll set the project on GitHub for future reference and to get more familiar with git.

GitHub: https://github.com/rajbos/dotnetcore-webapp.

Rob's Blog

AI Engineering Fluency: tracking your AI coding habits

What it does

Token tracking and usage charts

Multi-model usage insights

Environmental impact

The Fluency Score

Supported editors

Getting started

The AI subsidy era is ending: time to talk business value

The trough we’re sliding into

What the numbers actually look like

My own usage hockey stick

A concrete case: $57 in tokens for a JetBrains extension

Foundation first, then speed

Then the real prize: speed of opportunity

Look at the current spent to avoid surprises, then shift the conversation to business value

GitHub Copilot App is now in Technical Preview

First look: introducing the app

Using it for repo maintenance

Shooting yourself in the foot with AI

Figuring out where this was coming from

Side note: Copilot CLI extensions!

My Open Source Projects

DevEx Metrics

AI Engineering Fluency

Alternative GitHub Actions Marketplace

devops-actions Organisation

GHAzDoWidget

Jarvis

Where the GitHub Copilot extension points break governance

Photo by Samuel Regan-Asante on Unsplash

GitHub Copilot CLI plugin marketplace

GitHub Copilot CLI local extensions

Agent Package Manager (APM)

gh skill now in the GitHub CLI

MCP servers across editors

VS Code extensions and the registry split

New: VS Code enterprise policy updates

State of the plugin governance for GitHub Copilot

Running GitHub Copilot CLI on local AI inference

Hardware I was testing on

How the BYOK configuration works

What I tried

Ollama — works, with some gotchas

Foundry Local — almost entirely blocked because of driver limitations

LM Studio — works with the right configuration

What the NPU situation actually is

What I tried next: vLLM and TGI (HuggingFace inference stack)

The format split: GGUF vs safetensors

vLLM via Docker

TGI (Text Generation Inference)

llama.cpp — fastest raw CUDA inference

Installation (CUDA build on Windows)

Starting the server

Copilot CLI configuration

Result: 23.0 tok/s 🏆

Bonus: Vulkan tensor split across NVIDIA + Intel Arc

The PowerShell profile setup

Summary of what works today (April 2026)

Throughput comparison

Fastest local setup for this machine

Final verdict on local inference options for GitHub Copilot CLI on this machine:

What I’m waiting for

Show you Microsoft Certification badges in Credly

Microsoft Learn steps

Credly steps

GitHub Copilot - Coding Agent Examples Walkthrough

Your editor

Agent Task panel in a repo context

1. Fix the failing workflow

2. Check multiple workflow runs

3. Add extra functionality to the repo:

Repository creation through the UI

Chat interface in github.com

GitHub Phone app

GitHub Copilot Premium Requests

What is Premium Requests for GitHub Copilot ?

What is a premium request

Free plan

`gh skill` now in the GitHub CLI

3. Create the `.complete` file at the correct folder