Software Engineer Intern

May 2020 - Present | Remote

Since May 2020, I have been working as a software engineering intern at Sourcegraph on the Distribution team. The Distribution team is responsible for making Sourcegraph easy to deploy, scale, monitor, and debug, solving challenging problems that our customers face when they deploy and scale Sourcegraph on-premise in a variety of environments, and that we face when we deploy and scale Sourcegraph Cloud (the largest Sourcegraph installation in the world).

My work as an intern has had several areas of focus: building out the monitoring stack that ships with Sourcegraph, improving the process for creating Sourcegraph releases to on-premise deployments with new capabilities, and experimenting with changes to the pipelines that help us roll out Sourcegraph changes to the various deployments we manage ourselves.

Most of the company’s work is open-source, so you can see my pull requests for Sourcegraph on GitHub! If you poke around, you might spot me chiming in on a variety of other pull requests and issue discussions as well.

# Monitoring at Sourcegraph

During my time at Sourcegraph, a major part of my focus has been on expanding the capabilities of Sourcegraph’s built-in monitoring stack and improving the experience for:

Administrators of Sourcegraph deployments, by making it easy to configure alerts and provide diagnostics to help us triage their issues
Sourcegraph engineers, by improving the flexibility of our tooling for adding monitoring for their features and services, and adding container monitoring using cAdvisor to all our deployments
Sourcegraph support, by unifying and updating our existing Prometheus queries, improving the generated solution documentations for alerts, and integrating team ownership for easier triage of support requests and paging

A diagram of the monitoring stack that we ship to customers as part of each Sourcegraph deployment. Learn more about it in our handbook entry that I helped write!

Some specific examples of the work I did to enable this include:

I created a new sidecar service to ship with the Sourcegraph Prometheus image, which I wrote a bit about in this blog post. It allowed us to build integrations with alerting configuration directly into the Sourcegraph application, as well as monitoring features such as the ability to include recent alerts data in bug reports. I eventually used this sidecar to implement a proposal for routing alerts to the teams that own them.
I made a variety of contributions to our monitoring generator, which generates the Grafana dashboards, Prometheus rules and alerts definitions, documentation, and more that ship with Sourcegraph from a custom monitoring specification that teams use to declare monitoring relevant to their services. I also drove cross-team discussions to overhaul the principles that drive our work on this tooling to help guide the future of monitoring at Sourcegraph.

# Sourcegraph Releases

Previously, creating Sourcegraph releases was a lengthy, complex process that involved a large number of manual steps that would frequently delay our monthly releases.

My work in this area includes:

Extensive improvements to the Sourcegraph release tool, which handles automation of release tasks such as generating multi-repository changes, creating tags, setting up tracking issues, adding calendar events, making announcements, and more
Improving our integration and regression testing suite by introducing the capability to directly leverage candidate images in tests, generalising test setup tooling, and adding automated upgrade tests to ensure compatibility

The long-term vision of this work is to enable releases to be handled by any engineer at Sourcegraph, as seamlessly and painlessly as possible, improving the pace at which we can confidently ship releases to our customers.

A generated release campaign providing release captains an overview of the changes required for a Sourcegraph release to happen. Sourcegraph campaigns was a feature undergoing extensive development by another team, and I made this fun integration to build to check out their work and help out our own team's management of releases!

# Deployment Pipelines

Sourcegraph maintains a variety of Sourcegraph instances in addition to Sourcegraph Cloud. Deployment at Sourcegraph generally consists of two distinct steps:

Building and publishing images
Propagating published images

You can read more about this in the handbook page about instances.

I worked on making adjustments to our build and publish pipelines, such as enabling direct integration testing of candidate images and making it easier to build tooling that interacts with our images.

Deployment methodology varies from instance to instance, but when I first joined Sourcegraph we did not have any instance that was kept closely up to date synchronously with both the state of our monorepo, sourcegraph/sourcegraph, and the state of our primary method of distributing Sourcegraph, sourcegraph/deploy-sourcegraph. To amend this, I built a trigger-based pipeline that would keep deploy-sourcegraph in sync with the latest images, and immediately propagate changes in deploy-sourcegraph to an internal dogfood instance.

# About Sourcegraph

Sourcegraph provides code search and intelligence on the web across massive collections of codebases. Their long-term vision is to make it so everyone, in every community, in every country, and in every industry — not just the ones working at the half-dozen dominant tech companies — can create products using the best technology. Sourcegraph is a fully distributed company with employees across the world.

Robert Lin

Enthusiastic eater of lunches. Please feel free to reach out to me at robert@bobheadxi.dev!