What’s new in ZenHub On-Premise

In last year’s blog post, we discussed our efforts towards migrating all of ZenHub’s cloud architecture over to Kubernetes. At the end of that post, we promised that one day we would deliver these innovations to our On-Premise customers as well. We’re pleased to announce that this day has arrived!

On July 19, we launched ZenHub Enterprise 3.2.0 (ZHE 3.2) for general release and it brings all of the enhancements, features, and architectural improvements we’ve made in Cloud to On-Premise. In today’s post, I’ll share the technical details and challenges we’ve encountered along the way.

At the start of this journey, we knew we had two very distinct types of customers:

1. Customers with existing Kubernetes experience and infrastructure who wanted to be able to run ZenHub inside their existing Kubernetes cluster, and
2. Customers with little (or no) Kubernetes experience or infrastructure who wanted to continue to leverage VM deployments

To address both concerns, we put together a roadmap consisting of three distinct phases.

1. Phase 1: In April, we launched ZHE 3.0, which allowed customers with existing Kubernetes clusters to run an instance of ZenHub as a fully managed service without the need for any VMs
2. Phase 2: In May, we followed that up with the launch of ZHE 3.1, which gave our VM customers more flexibility and power by embedding an instance of Kubernetes inside our VM offering (via a tool called k3s)
3. Phase 3: This month, we shipped the conclusion of this trilogy, ZHE 3.2, which unifies the two platforms under a single code base and includes all of the latest and greatest product features from our cloud offering

With ZHE 3.2, our entire application infrastructure is unified under a single technology stack which means we are able to quickly deliver new features to both our Cloud and On-Premise customers with ease and on a predictable schedule. Let’s take a look under the hood at the technical details of each of the two groups of system administrators we support.

For Kubernetes Experts

Many ZenHub On-Premise customers are large enterprises with dedicated operations teams and technical administrators who have existing Kubernetes experience. We wanted to create a seamless user experience for these customers by giving them as much control over the entire system and upgrade process as possible.

We open-sourced our configuration and management toolset and embedded the relevant documentation directly into the package. The new public repository provides instructions for a number of highly requested features, which are now available with ZHE 3:

• The ability to run externally managed databases. Many of our customers requested the ability to host the ZenHub databases outside the main VM to improve reliability and integrity, simplify the backup & restore process, and have greater control over the company’s personal data. By moving to a Kubernetes architecture, this is now as easy as specifying a custom postgres_url string in the configuration file pointing to your external database. Not just databases too; all ZenHub services can now be managed externally.
• The ability to easily scale resources as the ZenHub user base grows. Many of our customers kick-off an initial On-Premise trial with just a handful of users (maybe a single dev team). Since ZenHub usually ends up solving a lot of pain points for those users, more and more teams at the company start to adopt ZenHub and the load on the infrastructure starts to increase. With our older ZHE 2 VM-based solution it was tricky to scale the VM without any downtime and with predictability. With ZHE 3 it can be as simple as updating a single value in the configuration.
• The ability to upgrade the application without downtime. The system administrators who are often managing the ZenHub application are under immense pressure. Their users want the latest and greatest ZenHub features, but the existing infrastructure doesn’t make it easy to perform an upgrade without first going through a number of lengthy steps (ie. backups, maintenance windows, etc…). With ZHE 3 installing the latest application images is just a few kubectl commands away, and a rolling restart ensures no downtime or interruptions.

Finally, to provide greater visibility into the internal architecture of our application, the zenhub-enterprise repository now also includes a service diagram to illustrate the various dependencies and network layers ZenHub uses. We hope this gives system administrators a clear view of what’s happening under-the-hood and allows them to configure and optimize their deployments with confidence.

For VM Customers

Despite all the advantages mentioned above, we know that the majority of our customers won’t have the resources or time to adopt Kubernetes in their tool stack, and would prefer to skip the hassle and simply run an out-of-the-box VM installation of ZenHub (similar to ZHE 2). This is still fully supported, and further enhanced. At launch, ZHE 3 supports the following VM platforms: AWS EC2, VMWave, Hyper-V and Azure. We’ve refined our documentation for VM customers with additional information about hardware sizing, ports, access and network configuration, and SSL/TLS certificate setup.

By leveraging k3s in the VM, we’re able to mirror a nearly identical configuration for VM customers as our Kubernetes customers. This gives the ZenHub dev team the power and confidence to deliver new application features and versions with a single unified deployment architecture.

Release Cadence & Security

We’re ashamed to admit that between our 2.44 release and the 3.2 release is a gigantic 1 year gap. That’s an entire year worth of features, bug fixes and enhancements that our On-Premise customers were desperately waiting for. Features like Workflow Automation, Sprint Planning, performance improvements, and many many more. This is far too long and with the release of 3.2 we’re not only finally bringing these features to On-Premise, but we’re making a promise to deliver future application updates with more predictability and frequency. Moving forward we expect to be able to deliver new major versions every 3 months. Critical security patches and bug fixes will be released more frequently as well.

With security in mind, ZHE 3.2 also includes a number of security improvements:

• Update to a newer version of MongoDB and PostgreSQL
• Update to a newer version of embedded Kubernetes (k3s)
• Disable TLS 1.0 and 1.1 for VM customers
• Prevent search engine indexing of ZenHub Enterprise pages
• Added limits on SSH retries
• Moved the ZenHub admin portal to a separate domain (or port) so its access can be locked down at a network level

To help automate our releases and builds we’ve adopted the use of GitHub Actions. Our Actions workflows are now handling the building, tagging and pushing of Docker images, generation of downloadable image bundles, building our various multi-environment VM images (VMWare, KVM, AWS, GCP, Azure) as well as running our automated test suites. We’ve been developing a number of innovations when it comes to managing Kubernetes jobs and resources via GitHub Actions. If this is something your team is actively working with as well, let us know. We’d love to partner with you and perhaps kick off an Open Source project in this space.

A New Era for ZenHub

ZHE 3 represents a major milestone for the engineering team at ZenHub, and we’re thrilled to be able to share the results with you today. We have many new and exciting innovations planned for our users in the coming months. With the conclusion of this chapter, we’re already starting research into further improvements to the user experience for our On-Premise users. Specifically, we’re looking to simplify the distribution process of our browser extensions, simplifying update checks, expanding the list of supported deployment targets, and considering leveraging Replicated to help us deliver a more refined administration panel for system operators.

If you haven’t already grabbed the ZHE 3 upgrade bundle, get started today! Looking forward to seeing you join us on this journey.